Tavily Company Search Benchmark

Business AI agents need accurate, up-to-date company information to make informed decisions. Company research supports use cases such as CRM enrichment, prospect qualification, onboarding, underwriting, and KYB/KYC by providing key information about what a company does, who it serves, and who owns it. That context changes as businesses grow, enter new markets, change ownership, and expand their operations, and the latest information is spread across company websites, filings, news, and other sources.

The Tavily Company Search Benchmark evaluates a foundational step in company research: how much a search engine helps an agent complete the task at hand. It is built around real production query patterns, and each query comes with a structured scoring card that describes what a useful, accurate answer should contain, including how recent its sources need to be.

For the motivation behind the benchmark and the results of the provider comparison, read the blog post: Your Agent Has the Company Name. How Much Can it Actually Tell You About It?

This release is a snapshot based on queries from recent weeks, intended to illustrate the approach. Internally, the evaluation data is refreshed regularly, including daily updates to the most time-sensitive slice.

Dataset at a glance

• 1,000 anonymized and obfuscated queries reflecting real production patterns, each with a scoring card.
• Categories: profiles (494), news_and_markets (333), financials (134), other (39).
• Scoring cards built as of 2026-10-06. Company facts change, so the cards reflect the evidence available on that day.

Core concepts

• Wording matters. Queries reflect how customers' agents phrase their searches in production, because how a question is asked affects what a search engine finds.
• Today's questions need today's data. The dataset draws on recent production patterns to capture what agents are researching now.
• Company facts change. A fresh scoring card is built for each query on the day of evaluation, recording verified facts and defining what a useful answer should cover based on the evidence available at that time.
• A sensitive measure of search quality. The benchmark measures how much a single search response helps answer the research question, rather than whether an agent can eventually find the answer through repeated searches.
• An agent's perspective matters. It measures what an agent can learn from a response and use to complete its research task, not just what information the response contains.

How the dataset was built

Starting with the query, not a predefined answer

The benchmark starts from queries that follow patterns observed in production traffic — the entities being researched, the intent behind each query and its wording — rather than from a predefined ground-truth answer sourced from public registries, news feeds or official websites. This preserves the complexity agents face in production. Some queries are very difficult, and some may not have a complete answer on the public web; they are not excluded, because removing them would filter out the hardest production tasks and measure quality only where the answer is easy to establish.

Some queries contain an incorrect premise or don't quite fit the company they name. This matches production traffic: agents sometimes start from a wrong assumption, and a good search response helps them correct it. The scoring cards check each query's premise against the evidence.

Scoring cards

A short ground-truth sentence or a single number is often not enough. For each query, a Deep Research Agent builds a shared scoring card, drawing on all search providers in the comparison as well as extraction tools, to capture the widest picture of what the web shows about the query at that moment. It can reformulate queries, investigate supporting details, follow leads and cross-check findings. Financial figures are additionally human-verified.

The scoring card states the query's intent and target company, checks whether the query's assumptions are supported by evidence, and records the verified facts that answer it. It defines what the best answer supported by available web evidence should contain: which aspects it must cover, what counts as useful partial information, and what makes a response irrelevant or misleading. Together with the relevance-level definitions below, this gives the criteria for Complete, Partial, Related and Irrelevant answers.

Data fields

Every scoring card contains these sections:

• QUERY INTENT: what a precise answer must contain.
• AS OF: the date the card was built.
• COMPLETENESS CRITERIA: yes/no questions that a complete answer must satisfy.
• PARTIALNESS CRITERIA: yes/no questions describing useful but incomplete answers. An answer that relates to the query but meets none of these is judged Related.
• IRRELEVANCE CRITERIA: yes/no questions describing answers about the wrong entity, the wrong period, or that accept a false premise.

Some cards also contain:

• TEMPORAL INTENT (the 259 cards with recency_aware = true, mostly news_and_markets): how much recency matters. time-sensitive means the answer needs current sources, historical means it is about a specific past event or edition, and evergreen means publication date does not matter.
• ANSWERABLE, FALSE PREMISE and REFERENCE ANSWER (134 cards, mostly financials): for queries asking for a specific figure, whether the web supports an answer, whether the query's premise holds, and the verified answer.

Evaluation protocol

1. Search. The same query is sent to each search provider.

2. Answer generation. The same LLM, the Answer Generator, produces an answer for every provider from the query and the returned search results only. This simulates answering from a single search query. The generator never sees the reference answer, the scoring card or the provider's identity, and is instructed to rely only on the retrieved evidence and preserve factual details.

   What the Answer Generator sees. For each provider, up to the top 10 results, each with its URL, title and the text the provider's API returns for that result. Depending on the API, that text is called content, snippet, highlights, excerpts or description; when a provider returns several fragments for one result, they are joined. The Answer Generator prompts call this text a snippet; in the Judge prompt, document is the generated answer being graded. The only information taken from the pages themselves is the publication date: for recency-aware queries, the generator also sees the publication date of each retrieved URL, extracted by an independent analytical tool from the full page content (algorithm), plus the query's time-relevance label (time-sensitive, historical or evergreen), and cites each fact with its date and source.

   We recommend that your production agents account for publication dates as well. If your search provider returns a date field and its meaning matches your application's requirements, you can use it directly. However, providers may use different date extraction logic, and some do not return dates at all. For our evaluation, we therefore extract publication dates independently, using the same tool across all providers to give the Answer Generator a consistent basis for assessing freshness.

3. Judging. A Judge Model rates each generated answer against the query and its scoring card, including the query's intent, freshness requirements and criteria. Cards with a REFERENCE ANSWER are judged on how closely the answer matches it. For recency-aware queries, the judge also checks the answer's sources against the publication dates. It assigns one of four relevance levels:

The reported score is the average weight across queries. It measures whether the returned information lets the Answer Generator produce a useful, correct answer from the search response alone. It does not directly score ranking within the top results or the number of duplicates.

Prompts and methods

• Answer Generator: one prompt for general queries and one for recency-aware queries, chosen by recency_aware.
• Judge: one prompt whose applicable section is chosen by the scoring card: general company research queries, recency-aware queries, or financial queries with a REFERENCE ANSWER.
• Publication-date extraction (LaTeX source): the algorithm that extracts the publication date of each retrieved URL for recency-aware queries.

Intended use

• Explore the query patterns seen in production company research and the scoring criteria behind them.
• Check how closely these queries match your own workload before relying on any benchmark to choose a search tool.
• Use the approach to inform your own evaluations. The most useful test of a search tool is how well it supports the questions your agents actually ask, so evaluate on your own queries with the model you plan to deploy in production.

Limitations

• Point-in-time. Scoring cards in this release reflect web evidence as of 2026-10-06. Facts change and so does their availability, so some criteria will go out of date; internally the cards are rebuilt on each evaluation day.
• Single-query. The benchmark measures one search response, not multi-step agent workflows, in which an agent can compensate for weak individual responses.