Controlled Variable Methodology for Managed Database Comparisons

A managed database comparison is only as good as the conditions it holds still. Two vendors can publish honest numbers, run honest tests, and still produce a comparison that tells a reader nothing, because the hardware, the schema, the cache state, or the pricing tier for each number was never equalized in the first place. In database benchmarking, a new engine often gets tested against a competitor running an under-tuned, default-config, cold-cache setup, and the "10x faster" headline is real in the sense that the test ran and produced that number. It's meaningless in the sense that it measured a mismatch, not a database.

Before trusting any benchmark, ask not "is this number real," but "what was held constant, and what was allowed to drift."" It's "what was held constant, and what was allowed to drift." That's the actual decision facing anyone trying to evaluate or produce a comparison that survives scrutiny from someone who knows the systems involved.

The eight variables a managed database benchmark must disclose and equalize

Eight things have to be pinned down before a latency or cost number means anything. Leaving any one of them loose is enough to sink the whole comparison on its own.

Hardware tier and compute: instance size, vCPU count, RAM, all need to match across vendors, named explicitly, not implied. Dataset size, shape, and cardinality. Row count, column count, the distribution of values inside those columns. None of it can shift between runs. Query shape. Aggregation-heavy workloads, point lookups, and full scans each stress different parts of a system, so mixing them without labeling which is which corrupts the comparison from the inside. Concurrency level. QPS and simultaneous connections affect whether a benchmark reflects production conditions, because a benchmark run by one user at a time isn't testing anything close to a production system. Cache state, cold versus warm, is probably the most quietly abused variable in vendor marketing. Managed-service tier has to match too: comparing a $67-a-month development SKU against a production deployment is a mismatch dressed up as a controlled experiment. Network topology, same-region versus cross-region, changes both latency and cost through egress paths. And schema design quality rounds out the list, tuned independently rather than copied over naively from whatever system came before.

Here's a fast filter for reading someone else's benchmark: scan the methodology section for these eight disclosures. If they're not there, the number attached to the headline isn't provable, and there's no obligation to believe it just because it's specific.

Schema design is not setup, it is the variable with the largest performance range

Most engineers treat schema design as a task you finish once, early on, then forget about. That's wrong in a way that costs real money over time. Schema decisions don't sit still. Their effects compound across months of query traffic, and a bad choice made on day one can quietly cost 10 to 100 times the performance a well-designed schema would deliver, depending on the workload.

Take the ORDER BY key in a columnar analytical database. It's arguably the single highest-leverage decision in the whole schema, because it determines how much data the engine has to scan versus how much it can skip. Put a high-cardinality timestamp first, something like ORDER BY (event_time, country, event_type), and index pruning falls apart, because nearly every value in that first column is unique, so there's no meaningful way to narrow down which parts of the data even need to be touched. Flipping the order so low-cardinality filters come first and time comes last, something closer to ORDER BY (country, event_type, user_id, event_time), allows the engine to skip enormous chunks of data before it ever gets to scanning rows.

This is why porting a schema over from a relational database, without rethinking it, tends to produce embarrassingly slow benchmark results. A table designed with Postgres assumptions baked in, no low-cardinality encoding, joins modeled the way you'd model them in a row store, timestamp-first ordering, measures how badly the schema was translated rather than the database engine's ceiling. It's measuring how badly the schema was translated. Any comparison that doesn't disclose how much tuning went into each side's schema is comparing engineering effort, not database performance.

What cache state and storage architecture determine at query time

Whether a query hits warm cache or has to fetch cold data from object storage isn't a footnote, it's often the single biggest swing factor in the whole test. The same query, against the same dataset, on the same hardware, can differ by an order of magnitude depending purely on whether the relevant data was already sitting in memory. A benchmark that doesn't say which one it measured is a number without a label. It's a number without a label.

Storage architecture makes this worse, not better, once systems separate compute from storage. Using object storage like S3 as the backing layer cuts per-gigabyte cost dramatically compared to keeping everything on local NVMe drives. But that savings comes with a catch: any query that can't be served out of local cache has to pull native data files from object storage on demand, and at that point, latency depends not on the query engine's cleverness but on how fast that storage layer can hand back bytes. Lower storage bills, but higher risk of latency spikes once the working set of "hot" data outgrows whatever's cached locally.

That architecture carries operational weight beyond raw speed, too. Compute-storage separation introduces coordination complexity that can produce unexpected behavior across replicas. On February 19, 2026, a routine mutation meant to materialize an index tripped a bug in zero-copy replication logic, and it convinced one replica that every data part it held was unreferenced and safe to drop. The incident affected PostHog. The incident that affected PostHog is what "compute-storage separation saves money" looks like when it goes wrong in production. It's what "compute-storage separation saves money" looks like when it goes wrong in production.

Pricing models that make cost comparisons structurally incomparable

Three pricing models dominate managed database offerings today, and they don't translate into each other cleanly. That mismatch is why cost comparisons across vendors go sideways so often. Instance-based pricing charges an hourly rate for fixed vCPU and RAM, with storage and I/O billed on top, separately. Usage-based or serverless pricing charges for whatever compute, storage, I/O, and data transfer actually got consumed, so cost moves with real usage instead of staying fixed. Tiered plans bundle everything into a flat price from a specialized provider, predictable, but with less room to flex as workloads change.

The nominal sticker price rarely reflects what actually lands on the invoice. A "cheap" small instance running Multi-AZ with 100 GB of storage and moderate I/O can climb to $80 to $100 a month, well past whatever number appeared in the pricing table. Running the math on a real production setup makes the gap get bigger fast: a Multi-AZ PostgreSQL instance sized for production work runs somewhere around $350 a month for compute alone. Adding 2 TB of storage costs roughly another $230 a month. Add backup storage, another $46 or so. Still running a Postgres version past its end-of-life date on AWS? Extended Support tacks on $0.20 per vCPU-hour, on top of everything else. High availability alone can roughly double the compute bill: a modestly sized instance in the Mumbai region goes from around $88.79 a month running single-instance to $177.57 a month once a standby gets added.

None of these numbers are dishonest on their own. The problem occurs when someone compares an instance-based number to a usage-based number, or a development-tier price to a production-tier price, and presents the gap as an apples-to-apples finding. It isn't one, unless both sides are stated at the same tier, same HA configuration, same region, and same billing model.

How concurrency and query shape expose hidden bottlenecks that single-user benchmarks miss

Most published benchmarks run one query at a time, or close to it. Production systems don't work that way: dashboards, data pipelines, and now autonomous agents all hit the same database simultaneously, and that concurrency is where bottlenecks that never appear in a single-user test start to surface.

There are rough thresholds worth knowing, because they reveal whether a benchmark reflects anything close to real usage. Dashboard-style workloads should stay under 1 second at the 95th percentile with throughput above 200 queries per second; a dashboard query that takes more than 3 seconds to come back has failed, functionally, whatever the average looked like. Ad-hoc analytical queries have more room, under 30 seconds for complex work, but if memory usage climbs past 80% of what's available, that's usually a sign of a partitioning problem or a poorly shaped query, not just a heavy workload. On the ingestion side, streaming and ETL pipelines should sustain more than 10,000 rows per second, and batch inserts taking longer than 500 milliseconds usually point to issues with the partition key or how data parts are merging. When simple aggregations can't clear 100 queries per second, that's a signal of resource contention or an index that isn't doing its job, and background merges that stretch past an hour usually mean data parts have grown too large.

Query shape matters just as much as concurrency, and the two compound each other. Aggregation-heavy dashboard queries, point lookups for operational systems, and full-scan batch jobs each stress different parts of the system, memory, I/O, and the main processing hardware, in different proportions. A benchmark that averages across all three without labeling which is which produces a number that doesn't describe any real workload. High write volume adds another wrinkle: heavy ingestion triggers merge activity that competes directly with read queries for I/O bandwidth, so a test that only measures reads, or only measures writes, but never both running at once, is testing half of a production system. It's testing half of one.

Agentic workloads introduce a concurrency pattern that no existing benchmark framework was designed to measure

Something genuinely new is happening in how databases get queried, and it's outrunning the benchmark tools built to measure it. Agentic data access describes AI agents that identify, retrieve, and use business data on their own, choosing which sources to query and how to compose the query, without a person in the loop making those calls.

The scale involved breaks assumptions baked into human-centric benchmark design. Autonomous agents can query a database millions of times a day, at a pace no human-driven access pattern comes close to matching. When a single person spins up a database, the cost barely registers. When an agent creates a large number of database objects in a single day because that's what the task called for, the cost becomes existential for whoever's paying the bill.

Agents also tend to over-explore before they commit to a query, a more subtle problem than the raw scale itself. Research published in 2026, titled "Sophrosyne: Agentic Exploration of Relational Data Systems Needs Moderation" (arXiv:2605.30862), found that text-to-SQL agents make a series of tool calls to explore a data system's schema and contents before they ever write the actual query. The tools available for that exploration tend to be either coarse-grained or fine-grained, and picking between them forces a real tradeoff: the choice between them involves real tradeoffs between cost and accuracy.

That behavior doesn't look anything like a conventional query benchmark. It includes bursts of schema introspection, exploratory probing queries that never touch the actual business question, and parallel sequences of calls that no existing benchmark harness was built to record, let alone measure fairly. Whatever methodology comes next for evaluating databases under automated, model-driven access is going to need to account for a query pattern that doesn't resemble a human sitting at a terminal typing SQL.

Running a controlled comparison: what to disclose, equalize, and report

Putting the pieces together, a workable methodology starts to take shape, one built around disclosure first, equalization second, and honest reporting last.

Disclosure means stating all eight variables covered earlier: hardware tier, dataset shape, query shape, concurrency level, cache state, service tier, network topology, and schema tuning level. Equalization means matching those variables across every system under test, not approximately, but as close to exactly as the vendors' offerings allow. Where an exact match isn't possible (say, because pricing models differ structurally between instance-based and usage-based billing) the gap itself needs to be named, not smoothed over or ignored.

Reporting means separating what a benchmark measured from what a system could have measured under different conditions. A cold-cache result and a warm-cache result are both valid data points. They are not the same data point, and they don't belong on the same chart labeled with a single number. A benchmark that mixes aggregation queries with point lookups needs to report them separately, because averaging across query shapes produces an average that describes nothing.

A benchmark's value comes from what it holds constant, not from how large the resulting number looks. A speedup measured against a naive schema and a cold cache says something about the naive schema and the cold cache. It doesn't say much about the database. Anyone evaluating a managed database comparison, or producing one, does well to ask the same question at every step: what changed here, and what stayed the same? If the answer isn't in the methodology, the number that follows isn't worth much either.