RTO and RPO Benchmarks for ReplicatedMergeTree Failure Scenarios

RTO and RPO benchmarks for ReplicatedMergeTree must come from timed recovery exercises. A clean replica failure, Keeper outage, or replication log limit breach each has its own recovery path. Each needs separate evidence of when service returned and which acknowledged writes survived.

Why RTO and RPO must be proven separately for ReplicatedMergeTree, not assumed from architecture

A replica that promotes in five minutes may have data older than the RPO window allows. A backup that has every committed record may take hours to restore. Neither result proves both objectives were met. RTO needs a timed recovery path that ends with proof that service is back, while RPO needs a durable recovery point and data reconciliation. Replication lag helps explain the latter, but cannot replace it.

ReplicatedMergeTree replicates between nodes and uses Keeper for coordination, but those mechanisms do not resolve every failure in the same way. What happens when the data survives but coordination fails? What happens when a replica returns after its recovery history has expired? A valid benchmark names the failure boundary rather than treating every outage as a node replacement.

What a failure matrix for ReplicatedMergeTree actually looks like

A failure matrix lists the recovery path for each scenario and the evidence needed for RPO and RTO. For ReplicatedMergeTree, it needs separate rows for single-replica loss, full-shard loss, Keeper unavailability, replication log limit breach, and broader control-plane loss. Each scenario requires a distinct test. Full-shard loss requires routing changes at the Distributed table layer and may leave service partially degraded, while credential, DNS, or orchestration recovery can block data recovery altogether.

Replica count belongs in the scenario assumptions. Two replicas per shard tolerate a single node failure but leave no margin for simultaneous issues. Three tolerate one failure while still satisfying quorum-based insert acknowledgment. Those starting conditions determine what a successful recovery can demonstrate. A result obtained with surviving peers is not evidence for a scenario without them.

The matrix also needs a common event clock. T-failure marks the first impact on eligible service, T-detected an actionable monitoring signal, and T-declared the recovery decision. T-contained records fencing of old writers and containment of the failure. T-data marks when restored or promoted data is available, and T-service marks when critical service can be reached. The endpoint is T-accepted, when a synthetic business transaction is durable and reconciled.

Single-replica loss: what the best-case RTO and RPO actually look like when measured

In a clean single-replica failure, the replicas that survive keep serving reads and writes. Monitoring detects the failed node, traffic is rerouted by the load balancer or Distributed table, and a replacement pulls missing parts from peers. This is the favorable baseline for recovery testing. Even here, the detection gap must be measured, and recovery of service must be distinguished from recovery of redundancy.

Service-level recovery can take seconds from detection to traffic rerouting. That is not the entire failure-to-acceptance interval. Restoring the full replica count takes longer because the replacement must fetch data, and how long that takes depends on the volume and throughput. Reporting both clocks shows how long service was affected and how long the cluster remained exposed at reduced redundancy.

If every acknowledged write reached at least one replica that survived the failure, RPO is effectively zero. Replication queue depth helps measure how much data was at risk when the failure happened, but reconciliation must show which writes survived. A short queue is not a data-integrity verdict. The exercise therefore needs both the replication signal and the recovered data result.

Keeper unavailability: how a control-plane failure changes the recovery clock

Keeper unavailability blocks writes even when the data replicas remain intact. Replicated tables enter read-only mode, so moving traffic between those replicas does not solve the failure. The recovery dependency has moved to the control plane. The relevant interval begins when writes are blocked and ends when a post-reconnect write is durable and reconciled.

The system tries to reconnect from time to time, and simple session-expiry cases may recover on their own. Other cases require SYSTEM RESTART REPLICA, or SYSTEM RESTORE REPLICA when metadata has been lost, after Keeper becomes reachable. Keeper’s snapshots and Raft logs allow state reconstruction, but restoring the ensemble must precede writable tables. DNS, certificates, and network paths can extend that prerequisite work. Their recovery belongs inside the measured path.

Already replicated data has zero RPO exposure from this outage alone. In-flight or buffered writes need to be checked separately, because missing writes count as loss based on whether they were durably acknowledged. This is not a data-volume problem by default. The rehearsal must prove control-plane recovery and then reconcile writes, rather than substitute a replica-copy benchmark for the time needed to restore write availability.

The replication log limit: the silent cliff that invalidates a replica after extended downtime

Extended downtime introduces a different boundary through merge_tree_replicated_max_replicated_logs_to_keep. A returning replica reads its replication queue from Keeper and attempts to fetch missing parts. If the retained replication log no longer covers its recovery needs, it can return in a “lost” state. Automatic catch-up is no longer the recovery path. Explicit operator intervention is required.

The timing of discovery makes this boundary easy to miss. Operators may learn that the replica crossed the limit only when it reconnects, after expecting ordinary synchronization. Recovery requires SYSTEM RESTORE REPLICA to rebuild its state from a peer. The resulting duration includes diagnosis, the manual step, and data synchronization at the available throughput. A plan that budgets only for automatic catch-up omits part of the work.

The cluster’s RPO can still be zero if surviving replicas are intact and contain all acknowledged writes. The immediate exposure is reduced service margin. Where no further replica margin remains, another failure has no fallback. Monitoring replication queue depth against the configured limit provides early warning, while a rehearsal beyond that boundary establishes how long explicit recovery takes.

Four documented production failure cases that changed measured RTO and RPO

GitHub issue #115549 documents a power-loss ALTER failure in which local metadata gets ahead of Keeper’s record, leaving a torn commit. Server restart, SYSTEM RESTART REPLICA, flags/force_restore_data, and SYSTEM RESTORE REPLICA all leave the table read-only. The reported exits are SYSTEM DROP REPLICA followed by re-creation, or hand-editing the .sql file to match ZooKeeper. Standard recovery commands do not complete the recovery. The case exposes an RTO gap involving disruptive intervention and was reported open as of late August 2026.

GitHub issue #116432, reported in August 2026, documents force_restore_data resurrecting data that a peer had TRUNCATEd. The recovered replica becomes writable but serves the removed rows, and SELECT count() differs between replicas. The empty peer has nothing to fetch that would overwrite the resurrected data. Write availability therefore masks a failed reconciliation. The RPO defect was reproducible on versions 26.9.1.1 and 26.8.1.1 and reported open as of late August 2026.

GitHub issue #118002 documents replication queues jammed after multiple crash-recovery runs in 2026. Large numbers of GET_PART exceptions retried indefinitely. Some entries cycled through DUPLICATE_DATA_PART, while others failed with NO_REPLICA_HAS_PART. The queue did not clear through automatic recovery. Operators had to intervene, extending RTO, and the supplied case record gives no subsequent resolution status.

GitHub issue #117413 documents silent automatic MergeTree → ReplicatedMergeTree conversion failure in 2026 for databases created with lazy_load_tables = 1. On-disk metadata changes, but ZooKeeper metadata is never restored and the convert_to_replicated flag remains. The table stays read-only without an error or warning. Diagnosis becomes a prerequisite to selecting a useful recovery path. 26.10 and later include the fix.

How the recovery clock actually starts: the gap between failure and diagnosis

These cases show why restore throughput alone is a poor predictor of recovery time. A read-only table can be a sign of Keeper loss, torn metadata, or failed conversion, but each needs a different response. An alert saying only “replica offline” does not distinguish them. The time spent identifying the mechanism belongs in RTO, including time spent on commands that fail to resolve it.

The early clock events make that work visible. The time from T-failure to T-detected shows the monitoring gap, while the time from T-detected to T-declared shows the decision gap. The time from T-declared to T-contained includes the diagnosis needed to choose a recovery path and start it safely. Escalation, access approval, and failover authority also consume time. Starting measurement after those decisions conceals part of the outage.

Commands should be paired with a diagnostic decision tree. A rehearsal record should state the scenario and exclusions, record monotonic timestamps with sub-second precision, and distinguish automatic events from human decisions. It should also preserve diagnostic commands and their outputs. Those details help explain delays. Final reconciliation then establishes whether the chosen path restored correct data.

Point-in-time recovery gaps and where the RPO promise breaks down

When replication cannot self-recover, point-in-time recovery has a different evidence base. ReplicatedMergeTree has no native continuous-log mechanism for fine-grained PITR. Available patterns include re-consuming Kafka or a similar source of truth from a known offset, incremental backups, partition-level restores, ALTER TABLE FREEZE snapshots, and recovery from a lagging replica. None inherits the clean-failure replication result. For an intact backup chain, incremental backup cadence sets the maximum data-loss window.

Chain integrity can make the effective exposure worse. An S3 lifecycle rule that deletes a base backup that is still needed breaks the incremental chain that depends on it, even if later backup jobs succeed. Moving the base to Glacier instead adds a retrieval prerequisite. The object must return to Standard before restoration, adding hours to RTO.

The length of the chain affects how long recovery takes even when every object is still available. If restoring the full chain exceeds the target, the available responses are shorter chains, parallel restore by partition, or a continuously replicated standby cluster. Throughput must be measured on the production backup path, including decryption and decompression.

Keeper persistence must be part of the same recovery chain. Losing its state without a backup can damage the cluster’s coordination, so data backups alone are insufficient evidence. RPO measurement also needs durable boundaries, including the recovered point, source cutoff, and recovered cutoff. A marker stream written through the real application path gives evidence for reconciliation. That test answers which acknowledged data survived rather than inferring survival from backup age or replication lag.

What scenario-specific rehearsals must cover to produce valid RTO and RPO verdicts

The next step is to turn the failure matrix into separate rehearsals. A rehearsal of single-replica loss should measure failure-to-acceptance time and how long it takes to restore redundancy, and prove that the replicas that survive keep accepting writes. Keeper loss should measure recovery through the first durable post-reconnect write and verify that acknowledged writes survived. The control-plane instructions must be available during the failure. Neither exercise substitutes for full-shard or broader control-plane recovery.

The log-limit rehearsal should deliberately leave a replica offline beyond the configured boundary, then time the SYSTEM RESTORE REPLICA path end to end. It should also establish whether monitoring warned before the threshold was crossed. Torn-metadata testing should measure time to correct diagnosis and verify peer state before force_restore_data is applied. Queue jams and conversion failures need their own diagnostic paths. A clean node-loss drill does not test those mechanisms.

Each rehearsal should produce an evidence packet. The packet needs the scenario, assumptions, exclusions, target objectives, event timestamps, human decisions, measured restore throughput, and reconciliation result. The clock stops at T-accepted when monitoring shows that behavior is stable. Where declaration starts a separate clock, both intervals should be reported to preserve the time spent detecting and deciding.

Infrastructure changes the scope of that obligation. Compute-and-storage separation with a hot-data cache mesh can sidestep several self-managed recovery paths, particularly incremental-chain restore throughput and lifecycle exposure. Isolated read-only compute, query logging, and sandboxed branches can support rehearsals against real data without touching production state. Those advantages do not supply the verdict. Teams still need to build the matrix, run the exercises for each scenario, and keep the reconciled evidence that backs each RTO and RPO claim.