Arrow Database Connectivity for OLAP Query Engines

OLAP engines store and compute in columns. JDBC and ODBC move data in rows. That mismatch forces every result set through two full conversions: the server flattens its columnar data into rows for transit, and the client rebuilds those rows back into columns once they arrive. Neither conversion is optional, and neither is a bug that a better driver can patch out.

The cost appears in the decoding of individual values. Every field in every row gets decoded one at a time through accessor APIs built for that purpose. If a result set has millions of rows, it becomes millions of separate per-value parses rather than one buffer operation moving at memory speed, each parse paying a small tax that adds up fast once the result set gets large.

The reason this persists is historical. JDBC and ODBC were built for row-based OLTP databases, running over bandwidth-scarce networks, read by workstations with little memory to spare. Rows made sense for that world. The assumption that a result set arrives as rows is written into the protocol itself, not a setting an engineer can flip.

Columnar engines have only multiplied since those protocols were designed: analytical databases, in-process query engines, stream processors, all producing data in column form. Each one of these systems still has to flatten its native format into rows just to hand results to a client, then watch the client undo that work immediately. The more columnar the engine underneath, the more wasted motion this round trip creates. That's the gap ADBC was built to close.

What ADBC is in the Arrow ecosystem

Three pieces of the Arrow project get bundled together in conversation, but they're separate layers doing separate jobs. Arrow Flight is a columnar transport built over gRPC: it moves batches of Arrow data, not rows. Arrow Flight SQL sits on top of Flight as a database wire protocol, and it handles query submission, result streaming, and catalog operations. ADBC is the client API on top of that: a standard interface for getting Arrow data back from a database, independent of what's doing the work underneath.

A useful comparison: Flight SQL is like a web server such as nginx, handling requests and responses over a defined protocol. ADBC is the client, like curl, issuing requests against that server. What comes back is zero-copy columnar data, ready to use without parsing.

These layers are independent of each other, and that independence is what gives ADBC its reach. A JDBC driver can speak Flight SQL without touching ADBC. Conversely, ADBC doesn't require Flight SQL underneath it. The API surface an application sees stays identical either way.

Arrow itself dates back to 2016, built as an in-memory columnar format. ADBC is the subproject that extends that format into a connectivity standard, giving database access the same columnar consistency Arrow already gave in-memory computation. The ecosystem around it includes pandas, Polars, PyIceberg, Apache Spark, and several major cloud warehouses, all participating as Arrow-native tools.

How ADBC removes the conversion overhead, step by step

ADBC doesn't make the JDBC conversion path faster. It skips that path. The server produces Arrow RecordBatches directly. The transport carries those batches as-is. The client receives Arrow arrays that point straight at the memory it just received, with no per-value parsing anywhere in the chain.

The mechanism behind that zero-copy receive works as follows. Arrow IPC buffers are contiguous blocks of memory, described by a Flatbuffers header that tells the receiver exactly where each buffer begins and ends. The receiving side reads that header, finds the buffer offsets, and constructs Arrow arrays pointing directly at the memory it already has. So if a batch holds many rows and many columns, it reduces to a small, fixed number of buffer operations. It never becomes a per-field decode repeated per row.

The Go ADBC package documentation lays out the object model this runs on. A Database holds connection configuration and driver state. A Connection, opened from that Database, carries session state for the duration of its use. A Statement manages the lifecycle of a query: SetSqlQuery to define it, Prepare to ready it, Bind to attach parameters, ExecuteQuery to run it. That last call returns an Arrow RecordBatch reader, not a row cursor. The distinction matters: nothing downstream of ExecuteQuery expects rows, because nothing upstream of it produced any.

Bulk loading gets the same treatment. IngestStream, added in adbc v1.7.0, collapses what used to be a five-step ingestion process into a single call, and it runs much faster than building large INSERT statements by hand. The columnar-to-row serialization that made bulk JDBC ingestion slow simply isn't present in this path. Parameterized queries follow the same logic: Arrow records bind directly as parameters, with no string interpolation and no per-parameter boxing through a row-shaped API. The Go package documentation shows this as a working interface, already structured for production use.

Partitioned execution: how ADBC distributes query results across workers

A single client pulling one result stream is only part of what ADBC supports. For distributed OLAP, the more consequential feature is AdbcStatementExecutePartitions. Instead of handing back one stream, it returns a set of opaque partition descriptors. An application can hand those descriptors out across threads, processes, or separate machines, and each worker then opens its own independent reader against its own partition.

In the Flight SQL driver, each of those descriptors carries a serialized FlightInfo, which itself holds one of the original FlightEndpoint messages from the server. A client that reads that information can see where the data actually lives and schedule its workers accordingly, placing compute near the data instead of pulling everything across a network link it didn't need to cross.

What this does is turn parallel result consumption into something the protocol supports directly, rather than something an application bolts on afterward. The server decides where the partition boundaries fall. The client decides where its workers run. So every worker gets the same zero-copy columnar batches a single-stream client would get, no matter which partition it's reading. Nothing about correctness or format changes when the work gets split across machines, only where it runs.

Known failure modes engineers must plan for before deploying ADBC

ADBC is ready for production use, but it isn't without rough edges, and the honest version of this story includes them. The failure modes that matter cluster into three areas: message size limits, type system mismatches, and incomplete metadata.

Message size is the most immediate one. The Flight SQL driver defaults to a 16 MiB maximum for incoming gRPC messages. A single RecordBatch that exceeds that limit fails with an internal error. Wide tables with large string columns hit this ceiling faster than engineers expect, and the fix requires raising the limit explicitly at connection time, not something the driver handles on its own.

Type systems are the second area, and they're subtler. Arrow's type system overlaps with any given database's type system, but the overlap isn't total. Decimal precision and scale, timestamp units and time zones, null-typed arrays: all of these need explicit handling when binding parameters or reading results back. JDBC drivers are used to silently coercing mismatched types into something that works. ADBC doesn't do that coercion automatically, so an assumption that held under JDBC can fail silently, or incorrectly, under ADBC if it isn't checked first.

Metadata is the third gap. The Flight SQL driver doesn't populate column constraint information, primary keys or foreign keys, when an application calls AdbcConnectionGetObjects. Catalog filters in that same call are evaluated as plain string matches. Any tooling built around catalog introspection for schema discovery needs to account for both limits directly.

None of these are reasons to avoid ADBC. They're work items to budget for before go-live, alongside the hygiene that solid driver implementations already practice: handling empty results properly (returning one zero-row chunk that still carries the query schema, instead of returning nothing) and releasing the database handle on every failure path, not only the one that succeeds.

Observability: how ADBC query telemetry integrates with existing OTel pipelines

The failure modes above are the cost side of the ledger. Observability is where ADBC pays some of that cost back. The Go-based Flight SQL driver emits OpenTelemetry traces for connection and statement activity, which puts ADBC-routed queries into the same observability pipeline an organization already runs for the rest of its infrastructure.

Setup follows the conventions teams already know. Traces are controlled through adbc.telemetry.traces_exporter, or through the standard OTEL_TRACES_EXPORTER environment variable. Neither requires writing ADBC-specific instrumentation into the application. ADBC Libraries 24, released 2026-07-28, added tracing for core Flight SQL operations and ships four exporter options: none, otlp, console, and adbcfile. The adbcfile option writes rotated JSON Lines files to a platform-specific directory, which is useful for local development or for air-gapped environments where you have no collector to send traces to.

The practical payoff is straightforward. A team debugging a slow dashboard refresh can trace a single query from the application layer, through the ADBC driver, into the Flight SQL endpoint, all inside one trace context. There's no separate query log to maintain and no custom middleware layer to build and keep in sync. For an OLAP pipeline serving real-time analytics or, increasingly, AI agents issuing their own queries, the ability to audit every query in a standard trace format is a baseline requirement for running the system responsibly.

Arrow ecosystem adoption of ADBC in production systems

Adoption has moved well past early experimentation. ADBC now shows up in enterprise data platforms, open-source query tooling, and geospatial infrastructure, and each of those cases tests a different part of the architectural argument made above.

Microsoft Fabric's June 2026 feature update highlighted its Google BigQuery V2 connector, built on ADBC and first introduced in May 2025. Microsoft describes it as "a cloud-native standard optimized for high-throughput analytical data transfer." That V2 connector covers Power BI, Dataflows Gen2, and other Fabric-hosted Power Query experiences, which makes it a validation of ADBC as a production data movement standard at Microsoft's scale.

Databricks, at Data+AI Summit 2026, framed ADBC as the emerging universal connector for agentic data workflows. The argument it made: agents are first-class users of the data stack now, and they inherit the same slow row-oriented APIs, heavy SDKs, and brittle system-specific connectors that have always burdened human-facing tools. ADBC's vendor-neutral Arrow results were presented as the correct architectural answer for agents making fast tool calls across systems.

GDAL added an ADBC vector driver in version 3.11, supporting SQLite3, PostgreSQL, Snowflake, DuckDB, BigQuery, and Flight SQL as backends. That driver exposes ADBC as both an OGR feature source and an ArrowArrayStream, which carries the zero-copy argument into geospatial data pipelines, a domain that doesn't usually get mentioned alongside OLAP query protocols. The project itself keeps expanding rather than settling: the ADBC Libraries 25 pre-release, dated 2026-09-21, adds a Presto driver and a chdb driver to the official drivers table and raises the Go toolchain requirement to Go 1.26.

Why AI agents make ADBC's architectural properties more valuable

Agents don't query the way people do. They issue queries in bursts, often in parallel, and they explore data without a human pausing to review each step before the next one fires. That changes the math on everything described above. So per-query overhead from row conversion adds up faster when nothing slows the query rate down. Because agents fire bursts of parallel queries without a human pacing them, parallel result consumption becomes a requirement. Full query auditability stops being optional, because nobody is reviewing each query by hand before it runs.

ADBC's partitioned execution model fits this pattern directly. An agent orchestrator can hand partition descriptors out across isolated worker threads the same way a distributed query engine already does, and the protocol supports that without any agent-specific extension bolted on.

GetObjects exposes statistics, row counts, min and max values, so an agent's query planner gets the same data-skipping ability a human analyst gets from a BI tool's metadata layer. The agent doesn't need to issue a separate exploratory query just to find out whether a table is worth scanning. That information is already sitting in the metadata it can read before running anything.

If an agent issues a query, full OTel trace emission logs it with the same fidelity as any other infrastructure event. Autonomous access doesn't have to mean unaccountable access. ADBC's telemetry model enforces that without requiring custom instrumentation built specifically to watch what agents are doing.

Access control follows the same pattern. You get the read-only connection option, ADBC_CONNECTION_OPTION_READ_ONLY in the ADBC API, along with standard transaction isolation levels, so platform teams can sandbox an agent's connection at the driver level. What an agent can and can't do gets scoped at the point of connection, not bolted on as application-layer enforcement after the fact. None of these properties were designed exclusively for agents. They were built for distributed, auditable, parallel analytics generally, and agents just happen to need that same combination.

Evaluating ADBC for an OLAP pipeline

Adopting ADBC is a commitment to moving data in columnar form from end to end, a commitment that touches query planning, result handling, type mapping, telemetry configuration, and, for agentic workloads, access control design as well.

Language coverage is uneven enough to require some planning. In the current pre-release, C, C++, GLib, Go, Python, and Ruby ship version 1.13.0, while C#, Java, R, and Rust ship version 0.25.0. Arrow's ABI stability makes reusing drivers across languages practical, but you still need explicit dependency management across that split, not an assumption that everything lines up.

Before you move production traffic, you need to identify gRPC message size limits for the target workload and configure them at connection time, rather than discover them when a RecordBatch fails partway through a migration. Every decimal, timestamp, and nullable type crossing the ADBC boundary needs a type mapping audit to confirm round-trip fidelity, since ADBC won't silently coerce mismatches the way a JDBC driver might. Where an application depends on primary or foreign key information from GetObjects, that dependency needs to be supplemented from a schema registry or database-native introspection, since the Flight SQL driver doesn't populate it. And an OTel exporter, whether otlp, console, or adbcfile, needs to be chosen and validated before go-live so that ADBC traces show up in the same pipeline as the rest of the application's traces.

If a team already runs columnar OLAP engines that keep hot data in memory and route queries without hitting object storage on every read, then ADBC's zero-copy result path compounds an advantage their architecture already has. The protocol tax that JDBC reintroduces at the application boundary has been, up to now, the one conversion overhead left standing in an otherwise columnar pipeline. ADBC is what removes it.