Benchmark Disclosure Standards for Database Vendor Claims

A buyer looking at a database vendor's launch page sees a table: a benchmark name, a number, maybe a percentage gain over a competitor. What that buyer cannot do, in almost every case, is trace that number back to a test they could run themselves and get the same result. That is the actual failure here, and it sits apart from the question of whether any vendor is lying.

Why database benchmark claims are structurally unverifiable by buyers

Start with the test a buyer can actually run: open the vendor's own launch page, find the benchmark number, and try to identify exactly which test produced it. Not whether the number is plausible. Not whether the vendor seems trustworthy. Just: can you point to the specific version of the benchmark, the specific subset of tasks, the specific tool used to run the evaluation, and the specific settings applied? For most published numbers, the answer is no, and an August 2026 audit of 42 benchmark rows published by nine vendors put a figure on how often that happens: 29 of the 42 rows were missing at least one applicable disclosure field, and not a single row was both fully disclosed and independently confirmed.

The audit did not go looking for fabricated numbers. It checked four fields that any reproducible benchmark claim needs before anyone even gets to the question of re-running the test: benchmark version, task subset, evaluation harness, and reasoning-effort setting. These are the coordinates of a test, not its result. Skipping any one of them leaves a buyer with a number that has no address, nothing to type into a terminal or hand to an independent lab. Of the 41 rows that could be scored on this basis, only 10 named the evaluation harness used to produce the number, and those 10 came from just three of the nine vendors in the sample.

None of this requires assuming bad faith. A vendor can report a real score, on a real test, run honestly, and still leave the buyer with nothing they can check. A score only the vendor has published is not proof the score is false, but it is proof the buyer has no way to find out either way, and that gap, not any accusation, is the subject of everything that follows.

How the database industry created this problem in the 1980s and partially solved it with TPC

Benchmark disclosure gaps are not a new failure mode.

TPC's 1988 formation by eight vendors was the industry's answer: mandatory full disclosure reports, ACID compliance requirements, and realistic pricing rules. Under TPC rules, a vendor cannot just publish a number. The full benchmark run gets submitted with a disclosure report covering system configuration, setup, testing conditions, and complete results, and an independent auditor certified by TPC has to validate the whole package before the number can be called a TPC result.

TPC also built a standard for benchmarks it does not administer itself, called the Fair Use policy, which asks for schema definition, query text, transaction definitions, durability and data accessibility guarantees, run methodology, and primary metrics, all disclosed in enough detail that an outside party could judge whether the comparison is fair. That list reads almost like a checklist for everything most vendor benchmarks skip today.

TPC's own compliance has thinned out over the decades. Headline TPC figures can still mask real architectural trade-offs, and the modern analytics and AI vendor landscape operates almost entirely outside TPC's jurisdiction, so it publishes numbers with none of the independent auditing that made TPC numbers meaningful in the first place. The incentive that produced aggressive benchmark marketing in the database wars of 1985 hasn't gone anywhere. It's producing the same behavior in managed database and AI benchmarks in 2026, except now it runs at a larger scale and with fewer institutions positioned to check it.

The DeWitt Clause and Vendor-Built Benchmark Tools

Even if a buyer wants to run an independent test, hire an outside lab, publish the results and settle the question for everyone, they run into legal and technical obstacles built for that exact purpose. Many vendor end-user license agreements contain a so-called DeWitt Clause: a provision that bars the customer from publishing independent benchmark results without the vendor's permission. A BenchANT analysis found that a significant share of major vector database vendors include exactly this restriction. The practical effect is that the party with the least incentive to find an unflattering number, the vendor, is often the only party legally permitted to publish one.

Where independent testing is allowed, the tooling itself frequently comes from the vendor whose product it measures. VectorDBBench, maintained by Zilliz, rewards distributed scaling patterns and penalizes single-node deployments. Benchmark suites associated with Redis and Qdrant weight in-memory operations heavily and leave out the total cost of ownership that comes with memory-heavy architectures. Even ANN-Benchmarks, which circulates as an academic, vendor-neutral standard, relies on datasets like SIFT and GIST built around low-dimensional vectors that don't resemble the dimensionality of modern LLM embeddings. The benchmark itself is methodologically sound. It measures the wrong workload for how these systems get used now.

A second, quieter distortion runs through nearly all of these tools: they measure performance at what engineers call Time Zero, the moment right after data finishes loading and before any live updates begin. Production systems don't have a Time Zero. Data keeps arriving, indexes keep needing to re-optimize, and the Milvus engineering team has said as much directly: benchmarks test the system right after ingestion completes, while real deployments never stop ingesting.

The OLAP world has its own version of this same gap, and its name is ClickBench. ClickBench measures scan-heavy aggregation queries against a single flat, append-only table, a workload that looks nothing like the join-heavy, mixed traffic that real production systems handle. ClickBench is a narrow benchmark, built to answer a specific question about scan performance, and the trouble starts when managed OLAP vendors cite it as a general cross-vendor performance comparison rather than the narrow test it actually is.

The specific omissions that make a benchmark number unreproducible

Set aside legal restrictions and vendor tooling for a moment and ask a simpler question: even with a published number sitting in front of a buyer, what specific pieces of information would need to be filled in before that number could be trusted or reproduced? The list is shorter than it might seem, and it repeats across almost every category of benchmark.

Hardware and compiler configuration sit at the top of it. SPEC CPU 2026, released in May 2026, added an explicit Compiler Category field to its run rules, and the reason is concrete: a community compiler like GCC running with standard flags produces different numbers than a vendor-specific toolchain like Intel oneAPI running aggressive optimization flags such as -O3 -ipo -unroll. But those differences don't carry over to a production environment running a standard distribution. Published SPEC CPU 2026 results show this in practice: the gap between base and peak metrics, driven by compiler choices and BIOS tuning, varies a great deal even across platforms running identical processors, and peak numbers achieved through vendor toolchains and non-default BIOS profiles have little in common with what a standard deployment will see. TPC built a rule against exactly this pattern decades ago: it banned so-called benchmark specials, configurations tuned narrowly to win a test rather than reflect real deployment. That practice was common before TPC existed, and it is still common in the much larger space of benchmarks that TPC doesn't govern.

Cost accounting is the second recurring gap. A benchmark that reports query speed without also reporting per-query egress and request costs is leaving out the part of the bill that matters most for a system running around the clock.

Query workload and tuning scope make up the third gap. A benchmark built around average latency hides what users actually feel during real traffic, because tail latency, specifically P99, is what shows up under concurrent load, and Reddit's engineering team, running hundreds of millions of vectors in production, found that metadata filtering under concurrency was their actual bottleneck. A single-client benchmark never surfaces that kind of problem, because it never creates the concurrency that causes it.

The fourth gap sits outside the benchmark entirely: operational cost. Real-time OLAP systems carry cluster management and ingestion pipeline work that no throughput number captures. A benchmark can report the fastest peak query performance in its category and say nothing at all about the engineering hours it took to reach that peak and keep the system there.

Each one is simply information a reproducible benchmark has to include and most published numbers don't.

Same Benchmark Name, Different Numbers

Suppose a buyer finds the same benchmark name on two different vendor pages, and both pages seem to cite the same evaluation. Are the numbers actually comparable? Not necessarily, and the reason has nothing to do with hardware, cost, or tuning. It comes down to version and subset ambiguity, a failure mode that disclosure fields exist to close.

The August 2026 audit caught a clean example of this. One vendor's own page reported a score of 26% on Terminal-Bench v3.0 for a given model. An independent evaluator tested the same model separately and measured a far higher score, but on the older Terminal-Bench v2.1. Both come from a benchmark called Terminal-Bench. They are not the same test, and nothing on either page made that distinction obvious without digging.

This is not an isolated incident in the audit's findings. Benchmark version was among the most frequently omitted disclosure fields across the full dataset, and the ambiguity compounds whenever a vendor cites a benchmark name that has gone through multiple version releases with different task subsets. TPC's insistence on naming benchmark version, task subset, and harness traces back to this same problem, which appeared decades earlier in database benchmarking, where the same benchmark name covered meaningfully different tests depending on who was running it.

What closes this gap doesn't require an independent audit or a legal fight over a DeWitt Clause. It requires plain labeling. The audit found cases of exactly that: vendors marking an internal test set as internal, marking a private benchmark as private, stating what the number does and doesn't represent. That's a different category from concealment, and it costs a vendor nothing beyond the discipline of writing it down. Kimi K3's page was the most fully disclosed row in the dataset, which is a useful data point in itself: full disclosure is achievable today, by a vendor willing to do it.

Agentic Query Patterns and the Benchmark Gap

Every benchmark discussed so far assumes a particular kind of user: a human analyst who considers a question, writes one query, and reads the result. That assumption is now out of date for a growing share of real traffic, because AI agents don't query databases that way.

An agent given a short description of a database doesn't write one query and stop. It requests a list of tables, inspects columns, runs an initial SQL query, revises it, and sometimes switches databases entirely before settling on a final answer. That sequence generates schema discovery traffic, incremental query refinements, and speculative reads, and a human analyst produces none of this in the course of answering the same question. A throughput or latency figure measured against single-shot human queries doesn't just miss this pattern. It actively understates the real query volume an agentic workload generates and overstates the effective throughput the system will deliver once agents are the ones asking.

ClaimDB, a fact-verification benchmark built by researchers at the University of Washington, gives a sense of what testing this pattern actually looks like. The researchers used Google's MCP Toolbox for Databases to let evaluation agents run arbitrary SQL queries, iteratively, against databases spanning millions of records across multiple tables. That scale and that query pattern don't appear in any existing OLAP throughput benchmark. More than half of the 30 LLMs evaluated scored below a modest accuracy threshold on structured fact verification at this scale, and the models struggled specifically with abstention, recognizing when the evidence in front of them wasn't enough to answer confidently, a real question for anyone planning to put an agent in front of a production database, separate from any question about query throughput.

The volume is no longer hypothetical, either. AppsFlyer's AI Assistant generated 37,000 queries between January and May 2026, and thousands of companies connected to MCP over that same stretch. That's a concurrency and volume pattern that no published benchmark currently covers. The gap documented across hardware, cost, version, and workload in the sections above was already serious for benchmarks measuring yesterday's query patterns. It gets wider, not narrower, as the thing being measured keeps changing underneath it.