Concurrency Stress Testing Methodology for OLAP Databases

Meaningful concurrency testing for OLAP systems means finding the exact point where a database stops behaving well under real, simultaneous load, before production finds that point for you. It's about finding the exact point where a database stops behaving well under real, simultaneous load, before production finds that point for you. Most teams never get there, because the way they test doesn't resemble the way their system actually gets used.

The gap: a single-user benchmark with a warm cache and no background writes tells you almost nothing about what happens when a large crowd of dashboard users hit the same cluster at 9 a.m. on a Monday. Those are two different systems, even if the underlying hardware is identical. The rest of this piece walks through a methodology for closing that gap: how to model real query load, build a query mix that isn't a toy, ramp concurrency instead of spiking it, separate cold reads from warm ones, and read latency in a way that actually tells you something.

What an OLAP engine's concurrency model allows, and where it can break

Column-oriented analytical engines built on an append-only, log-structured merge model don't block reads against writes at the storage layer. Inserts create new immutable data parts; queries read whatever set of parts is currently visible. Reads and writes don't fight over the same lock: inserts on an append-only, log-structured merge model don't block reads against writes at the storage layer. That's the theory, and it's mostly true in practice too.

But background merges still have to happen somewhere. Heavy ingestion triggers merge activity that competes for I/O bandwidth and memory, and that competition doesn't disappear just because the storage engine is clever about avoiding locks. Under enough concurrent load, merge pressure raises latency, even though nothing in the architecture is technically "blocked.""

Concurrency is governed by admission control settings, which are configurable and often left at defaults that nobody has stress-tested. It's governed by admission control settings, which are configurable and often left at defaults that nobody has stress-tested. A self-managed deployment typically caps simultaneous queries around 100 by default. A managed cloud deployment might default to something like 1,000 concurrent queries per replica. Neither number is a law of physics. Both are settings somebody chose, and both can be raised, lowered, or exceeded depending on what the workload demands.

What happens when a query arrives after the limit is already full? Without queuing enabled, it gets rejected outright, something like a "too many simultaneous queries" error. Some engines support a queuing mechanism instead, where a query waits for a free slot rather than failing immediately, up to a configurable wait time (often around 5 seconds by default). If a test never pushes hard enough to hit that queue or that rejection, the test hasn't actually measured overload. It's measured comfort.

Vendor documentation for these systems sometimes quotes eye-catching ceilings, like sub-10-millisecond queries at extremely high throughput on petabyte-scale data. Those numbers are real under ideal conditions, and they say something true about how far the architecture can scale. But a ceiling under ideal lab conditions isn't the same as a floor guarantee for a specific deployment running on specific hardware with a specific query mix. The methodology question is never "can the architecture scale," it's whether a given test setup can actually find where this deployment's limits sit, on this hardware, with this data." It's whether a given test setup can actually find where this deployment's limits sit, on this hardware, with this data.

Turning active user counts into real query load, before writing a single test

Active users is a marketing number, not a test input. It's a marketing number. Before designing any load test, that number has to get translated into something a database engineer can actually work with: requests per second, concurrent query count, and query shape.

Here's why the translation matters so much: 10,000 active users might produce only 50 simultaneous queries at any given moment, if usage is spread out and each session issues one query every few minutes. Or 500 users loading a single dashboard that fans out into 20 queries per page load could produce a burst of 10,000 requests in the same window. Same user count, wildly different load profile. Without doing this math first, sizing a system, or testing it, is close to arbitrary.

A few numbers need to be nailed down before test design starts:

• Peak requests per second actually hitting the database
• How many queries a single dashboard or page load fans out into
• How many queries are running at the same moment, not just arriving over time
• The mix of query types (filters, aggregations, joins, sorts, exports) and how often each occurs
• Target latency at p50, p95, and p99 for each query class separately
• How big the result sets get at peak
• Insert rate and batch size happening during peak query traffic
• What capacity looks like if a replica goes down mid-peak

At 1,000 queries per second with an average execution time of 50 milliseconds, the system is running roughly 50 queries at once on average, which is the kind of check Little's Law gives quickly. At 1,000 queries per second with an average execution time of 50 milliseconds, the system is running roughly 50 queries at once on average. Bump execution time to 500 milliseconds at the same request rate, and now it's running roughly 500 queries concurrently, ten times the pressure, from the same request rate. Execution time per query matters just as much as request volume in determining how much concurrent load a system is actually under.

One mistake occurs constantly: using end-to-end request latency, the number that includes application-level queues and network round trips, instead of the database's own execution time. That inflates the apparent concurrency estimate. Teams end up over-provisioning compute in the wrong place while missing the actual database bottleneck entirely. The fix is to measure execution time at the query engine, not at the browser or the API gateway.

The output of this whole exercise should be a written document: peak QPS, concurrent query count, and a breakdown of query classes. That document becomes the test spec. Skip it, and the load test is just guessing with extra steps.

Building a query mix that looks like production, not a leaderboard

Standard industry benchmarks (a common one runs 43 analytical queries against a 100-million-row web analytics dataset, around 15 GB compressed) exist for a good reason: they're reproducible, and they let you compare versions or vendors on equal footing. That's genuinely valuable. It's also a different job from testing whether a specific deployment can survive a specific production workload.

What that kind of benchmark doesn't cover: multiple query types running at the same time and competing for resources, mixed read and write traffic happening together, high-cardinality group-bys that stress memory through hash table growth, or workloads where multiple tenants share a cluster and need isolation from each other. Every one of those is a normal Tuesday for a production analytical system.

A query mix that actually resembles production usually has several distinct classes running simultaneously:

• Selective point lookups, fast, touching few data blocks, run at high concurrency

Each class needs individual attention because each one fails differently. Adaptive aggregation strategies and hash table optimizations can handle high-cardinality group-bys reasonably well in isolation, when memory spill settings are tuned correctly. Run that same query next to three others competing for the same memory pool, though, and the behavior shifts in ways a single-query test will never reveal.

Schema decisions affect performance directly: a per-tenant query that doesn't have the tenant identifier as the leading column in the sort order turns into a full table scan under the hood. A per-tenant query that doesn't have the tenant identifier as the leading column in the sort order turns into a full table scan under the hood. Nullable columns carry a hidden cost too, an extra mask that has to get loaded and checked on every read. None of this appears when one user runs one query against a warm cache. It appears as latency once twenty queries are competing for the same resources at once.

The practical fix: pull real query patterns from the system's own query log, or from a pilot workload that's actually representative. Classify by shape (selective lookup, aggregation, export), weight by how often each shape occurs, and build the test file around those real proportions. Many benchmarking tools support randomizing query order from an input file. Use that option for a multi-query test, otherwise sequential patterns create artificially warm cache conditions that flatter the results.

Ramping load in steps, not spikes, to find where things break

Every system looks healthy at low concurrency. Every system looks healthy at low concurrency, but that's not a compliment; it's just a fact about how little stress a handful of queries puts on anything. What matters is finding where the cliff sits, the point where adding one more concurrent query class causes tail latency to fall apart or admission control to start rejecting requests. Jumping straight to peak load skips right past that point instead of finding it.

A staged ramp works better: start with a handful of simultaneous queries, step concurrency up gradually, and at each step measure p50, p95, and p99 latency separately per query class, not averaged across all of them. Averaging is where important signal disappears. A fast selective lookup running at 5 milliseconds will drag the average down and hide a wide aggregation query that's degrading badly at the same concurrency level. Only looking at each class's own tail latency will actually catch that.

At every ramp step, watch a specific set of things:

• p99 latency, tracked per query class
• Active query count, current and peak memory use, peak thread count
• Whether queries start queuing for a free slot, or getting rejected outright
• Background merge activity, and whether ingest I/O starts competing with read I/O as concurrency climbs

Ingestion has to be running during every one of these ramp steps. A concurrency test with zero writes happening isn't simulating production, it's simulating a demo. Run inserts at the actual measured production rate and batch size throughout the whole ramp, not just at the start.

It's also worth checking whether insert traffic and query traffic stay properly isolated from each other as load rises. Systems that support a separate concurrent-insert limit are trying to stop heavy write traffic from eating all the available query slots. That isolation needs to hold under real pressure, not just when the system is sitting idle. Confirming it holds at idle and assuming it holds at peak is exactly the kind of gap this whole methodology exists to close.

Cold cache versus warm cache: the difference that decides whether a benchmark means anything

Run the same query twice, and the second run is going to look fast. That's not a real result; it's just cache doing its job. Operating system page cache, engine-level mark cache, and uncompressed data cache all get populated by that first run, so query number two measures memory speed, not disk speed. Any benchmark that only reports warm numbers is quietly reporting the best-case scenario for a query nobody's actually running for the first time.

Testing this properly means clearing all three cache layers before every timed run: dropping the OS page cache, then clearing the engine's mark cache and uncompressed cache, then running the query immediately after. Only then does the result reflect a genuinely cold read.

Cold cache numbers tell you the worst case: what a first-time or infrequent query looks like, what the I/O ceiling actually is, and what happens right after a restart or when a new replica joins a cluster with nothing cached yet. Warm cache numbers tell you the opposite: steady-state performance for the dashboard queries that get run over and over, and how much a sparse index (something compact enough to cover even petabyte-scale tables with only a few thousand entries, small enough to sit comfortably in memory) is actually buying in terms of speed.

Both numbers need to get reported, labeled clearly, side by side. A benchmark that shows off warm-cache results while the production system serves a constant mix of first-time and repeat queries isn't lying exactly, but it's not telling the whole story either. Latency stability at p95 and p99 under concurrent load is the standard that actually matters, and that standard can't be evaluated from warm numbers alone.

This gap is largest on deployments where storage sits behind an object store, since a cold read there has to make a full round trip to that storage layer. A caching layer that keeps hot data closer to compute narrows that gap and keeps p99 more stable as concurrency rises, and cold versus warm testing is exactly the method that reveals how much narrower.

Run the cold pass first at each concurrency step, then the warm pass right after. The ratio between the two numbers, at that specific concurrency level, quantifies exactly how much cache residency is worth under that load. That ratio tends to shift as concurrency rises, which is itself useful information.

Why p99 is the number that governs everything else, and how to read it

p50 is comfortable to report and mostly useless for judging an analytical system under load. It shows the median experience while hiding everything that happens in the tail. A system where half the queries return quickly but a small fraction take many seconds will still report a great median. That system is not fine. Anyone running a customer-facing dashboard on top of it is going to hear about the 30-second queries, not the fast median.

p99 describes the experience a meaningful slice of real users actually run into. For a product serving a few hundred simultaneous dashboard sessions, by definition roughly 1 in 100 queries is experiencing at least that latency. That's not a rare edge case; it's a regular occurrence at scale. A target like sub-second latency at high concurrency, somewhere in the range of 50 to 100-plus queries per second, is really a p99 target dressed up as a p95 target. It's the honest, tighter version of the same constraint.

p99 surfaces things p50 simply can't:

• The concurrency cliff itself, since p99 can climb while p50 remains largely stable
• Interference between workloads, where a greedy aggregation eats resources and drags down a selective lookup class that still looks fine on p50
• The first sign of admission control kicking in, since the marginal queries that start queuing or getting rejected show up in the tail first
• The real cost of a cache miss, which a warm-only benchmark simply erases

Reading p99 properly at scale usually means going straight to the query log: average duration, minimum, maximum, and count, grouped by query shape, across the whole test window. The maximum value is effectively p100, and it gives useful context for interpreting where p99 sits relative to the absolute worst case. Filtering to only completed queries, separate from ones that failed or got rejected, keeps the latency numbers honest rather than mixed in with errors.

The pass criterion that actually means something: p99 per query class stays flat, or within a defined and acceptable range, as concurrency climbs through the ramp. The moment it stops staying flat and starts curving upward marks the ceiling. Not the vendor's ceiling, not the one from the documentation. The one that belongs to this deployment, this hardware, this query mix, found before a real dashboard user ever has to feel it.