Eliminating the Batch Layer in Real-Time Risk Engines

Eliminating the batch layer in real-time risk engines is about making current data trustworthy at the moment of decision. Lambda’s split between fast approximations and later corrections cannot provide that guarantee. A single streaming pipeline is the better design for this workload, provided its replay, serving, recovery, and access controls uphold accuracy and latency together.

Why Lambda architecture creates a structural correctness problem in risk engines

Lambda architecture, introduced by Nathan Marz, sends data through two parallel paths whose outputs meet in a serving layer. The batch layer recomputes views over the full historical dataset on a schedule. The speed layer supplies approximate, low-latency results until that recomputation catches up. For a risk engine, that distinction reaches directly into the decision it must make now.

A transaction, position change, or exposure update arriving after the last batch run exists only in the speed layer’s approximate view. A fraud signal that depends on a full cross-account pattern may be absent from that incremental view altogether. What, then, does a real-time exposure query actually return? It returns a merged result that is accurate for history and approximate for recent activity, without a structural guarantee about where the approximation ends. A view fixed hours later cannot undo a decision on a transaction that was already made using stale information.

The logic-duplication problem compounds the correctness gap

Even when both layers have received the same data, their answers can differ because Lambda requires business logic to be implemented twice. A transformation written in Spark for batch processing and rewritten in Flink for streaming can produce different outputs because of floating-point handling, windowing semantics, or state management. Identical inputs do not guarantee identical results. Where a batch VaR calculation and an intraday exposure estimate are meant to converge, the architecture supplies no automatic way to decide which implementation is correct. Scheduled reconciliation arrives after the decision that needed agreement.

Feature engineering makes this divergence especially hard to detect. A model can be trained on a batch feature and served a streaming feature with the same label but a different mathematical definition. The label conceals the mismatch. That error may remain invisible until predictions are audited against ground truth. A simple count leaves little room for the answers to differ, while a multi-step, time-windowed aggregate with exclusions and late-arrival handling leaves many more ways for them to differ. In regulated risk workloads, this is an undetected consistency risk.

What Kappa architecture actually eliminates and what it requires instead

Kappa architecture, proposed by Jay Kreps, makes the stream processor authoritative for both live and historical computation through a single codebase. All data passes through a durable, replayable log, typically Kafka. To recompute history, earlier events run through the same streaming job, with no separate batch pipeline. This removes one specific source of disagreement. Recent and historical results no longer depend on independently maintained implementations of the same rule.

With a single code path, the fixed logic runs over the history affected by the fault, and the corrected output takes the place of the old results. That mechanism has real storage and compute costs. The log must keep enough history to reprocess what is needed, perhaps forever, and the processor must be able to run large historical replays. Long retention windows can be expensive or impractical. Some teams therefore keep stream output in columnar storage so they can query history at a lower cost through a separate query engine. That hybrid preserves Kappa’s principle when it does not create a second implementation of the operational risk logic.

Where Lambda’s defenders have a point — and why risk is not that case

Lambda earns its complexity when batch and streaming perform intentionally different computations. Machine learning training can use all the data in the historical corpus while a speed layer serves the model it produces for inference. Those jobs are different by design. Likewise, IoT analytics can pair immediate alerts with long-range trends over months of sensor history that would be impractical to replay. Recommendation engines can combine batch-computed collaborative filtering with session-based personalization without pretending that the two calculations are interchangeable.

Two paths can serve different purposes or compete to produce the same answer. An exposure aggregate or fraud feature belongs to the latter case. Its consumer needs one authoritative value. Overnight model training for fraud detection still has a place as a batch workload, but training is separate from transaction scoring. The scoring path should be stream-native and single-codebase because its latency and accuracy requirements apply to the same decision.

What the serving layer of a stream-native risk engine must deliver

Once the batch layer is gone, the serving layer must answer sub-second analytical queries over high-volume streams that keep being updated, without falling back on an approximate speed-layer answer. It must ingest at event rate without buffering writes into another source of delay. Freshness cannot stop at the stream processor. Rolling windows, cumulative exposure, and cross-instrument Greeks must also stay accurate as new events come in. A fast query over a lagging or incorrect aggregate fails the requirement just as surely as a slow query.

Ingestion-time pre-aggregation can reduce scan volume for predictable queries by computing rollups at write time. Exposure by counterparty and VaR by instrument class fit that approach well. An investigation is less predictable than a dashboard. A request for all transactions matching a pattern across the past 72 hours may fall outside the rollup schema and require a full scan. Raw columnar data with aggressive segment indexing and vectorized execution preserves that flexibility, at the cost of more compute per query. The serving layer must support both predefined operational views and unplanned investigations, rather than meeting its latency target only for known dashboards.

Why routing serving-layer queries through object storage undermines the whole design

A fast streaming pipeline loses the speed it gains if every serving query has to wait on object storage. S3 read latency, request costs, and rate limits become part of the operational query path. Query spikes make that dependency particularly awkward. During market stress, degraded bucket performance can delay the very requests the risk engine needs to answer promptly. Batching reads to amortize request costs brings waiting back into the system, only this time downstream of the stream processor.

The design response is to separate durability from query speed. S3 or other object storage that does the same job holds the immutable log and long-term snapshots, while a cache keeps hot data in memory and on fast local NVMe storage. Operational queries do not have to read from object storage. Current positions, recent transactions, live market data, and active feature windows form the working set that must be available there. As retained history grows into hundreds of terabytes, that separation is intended to keep latency flat. When the stream processor and serving layer run on different cloud providers, the added egress cost and latency make queries that depend on object storage cost still more.

Per-query pricing as a hidden tax on risk engine behavior

Infrastructure cost also shapes how often a risk engine is allowed to ask questions. These systems continuously fan out queries across instruments, counterparties, and time windows, so per-query pricing charges directly for investigative activity. That pressure can alter the design itself. Teams may throttle query frequency, defer incident investigations whose cost is unpredictable, or batch feature refreshes that would benefit from continuous computation. The architecture can lose its real-time behavior without adding a formal batch layer.

Agents amplify that pressure because one anomaly investigation may involve dozens of exploratory queries before reaching a finding. Each step adds a charge, even though all the steps are part of one investigation. Compute-hour pricing changes that incentive. It charges for capacity rather than adding a toll to each question, making frequent probing economically more reasonable within the capacity provisioned.

Agentic access to risk data requires the same guarantees as human access, but the failure modes are different

Agents reach the same risk data through a different access pattern. They invoke tools at machine speed and can launch exploratory queries in parallel across shared production data. Their stopping conditions need explicit engineering controls. Without rate limits and sandboxing, an investigation can consume compute needed by operational queries. An agent also cannot be relied on to recognize that a query touches PII or regulated data in the way a human analyst might.

Anthropic introduced Model Context Protocol, or MCP, as an open standard in November 2024. It provides a connectivity layer for structured tool calls to databases and external systems without requiring direct SQL access. Connectivity alone does not supply regulatory accountability. A database MCP server can offer tools for schema description, query execution, and pipeline validation, but the database must have governance in place for how those tools are used.

For risk access, every agent query must be logged in full so an audit can reconstruct the query history behind a decision. Sampling or summarizing that history is not enough. Agents need isolated, read-only compute over shared data, with writes and schema mutations structurally impossible. Sandboxed branches provide a separate place for investigative experiments against real data. Those controls let an agent explore without allowing its actions to propagate into the live serving layer.

Operational resilience in a stream-native design — replacing what the batch layer used to provide

Lambda can recompute results from immutable raw data when errors reach the serving layer. That capability needs an explicit replacement in Kappa. The event log becomes the system of record, while the serving layer holds results that can be regenerated. After a logic fix, replay replaces incorrect state instead of leaving a competing batch view to be reconciled later.

The distinction is not that replay makes correction instantaneous. Retention must go back to every point from which the business may need to reprocess, and backfill compute must be able to handle that much history. Point-in-time recovery at the serving layer adds the ability to return to a known-good state without replaying the entire log. Replication, high availability, and automatic recovery remain baseline requirements because a single fast node is not an acceptable production dependency.

For risk decisions, the correction deadline is the next transaction that depends on the affected state. Waiting for scheduled batch recomputation does not meet that requirement. Replay can make these corrections without a second codebase, but it must have the throughput and recovery provisions to finish by the deadline.

The integrated design — what a batch-free risk engine looks like end to end

The complete design begins with Kafka or an equivalent durable, replayable event log as the system of record. Events are written once and retained indefinitely, with history available for replay at any offset. The log feeds the stream processor and object storage for long-term durability. A single Flink job, or equivalent, applies feature computation, windowed aggregates, and model feature generation. That same job processes live arrivals and historical replay, then writes continuously into the serving layer.

A columnar OLAP engine takes in those results and keeps current positions, recent events, and active feature windows in its hot-data cache. Operational reads come from memory and NVMe, while object storage keeps the data needed for retention and recovery. Sub-second p99 latency must hold as the amount of data kept grows. Replication and recovery protect availability without reinstating a second source of risk calculations. Agents get to shared serving data through MCP tools that use isolated read-only compute, keep full query logs, and have sandboxed branches for investigations.

Compute-hour billing supports frequent operational queries and exploratory fan-out without a separate price for every question. The resulting architecture is more demanding than simply deleting a component from Lambda. Each layer must preserve the same contract through ingestion, computation, retrieval, and access. Its required output is one number, correct and current, because a risk engine cannot postpone either property until after the decision.