ORC vs Avro vs Parquet for S3-Backed Analytical Pipelines

Choosing ORC, Avro, or Parquet for an S3-backed analytical pipeline is chiefly a decision about I/O. The amount of data each query must pull from object storage matters. That lens puts Avro on the ingestion path, makes Parquet the usual analytical default, and gives ORC a specific role where selective filters and Hive integration reward its stripe indexes.

Why the ORC vs Avro vs Parquet decision is really an S3 I/O decision

A format is chosen at design time, but its access pattern shapes the S3 bill on every query. S3 is not a local disk. An engine cannot treat remote reads like cheap seeks on a local filesystem, because it retrieves byte ranges through HTTP GET requests. The file’s internal layout therefore governs both how many requests the engine issues and how much data those requests retrieve.

Which format compresses best is only part of that question. Before decompression begins, column pruning and predicate pushdown determine which parts of a file need reading at all. Physical layout establishes what can be read separately, while embedded statistics identify regions that can be skipped. Compression then reduces the bytes needed for whatever remains.

How columnar layout produces column pruning and why row-based layout cannot replicate it

Parquet and ORC group values by column within their internal storage units, so an engine can read the regions for the columns it needs without fetching the rest. Avro instead stores each record’s fields together. Reading one field from Avro still means reading through records in which the other fields are interleaved, so narrowing the query projection does not produce the S3 traffic reduction available from columnar storage.

The counterintuitive illustration is a read in which asking for two columns was slower than asking for all eleven. A row-based layout cannot turn a narrow projection into separate byte ranges for the fields the query asks for, even when it returns fewer values. No configuration change removes that structural constraint from an analytical scan. This makes Avro’s placement in the pipeline consequential.

Avro's legitimate role: the write path, streaming, and schema evolution

Avro’s record layout fits work that arrives one event at a time. Writing an event is a single contiguous append, without the columnar buffering and statistics computation needed to prepare analytical files. That works well for streaming ingestion and change data capture.

Avro also stores its schema as JSON with the data, allowing consumers to decode records without a separate catalog call. Producers and consumers need not evolve together. Its writer and reader schemas can differ, and readers can ignore unknown fields when producers add them. AWS positions Avro for streaming and serialization, including integration with Apache Kafka for real-time streams and schema handling.

Keeping settled data in Avro gives up column pruning, and the supplied comparison also found Parquet with zstd meaningfully smaller than Avro with deflate at equivalent row counts. Columnar layout gives compression runs of similar values, though a comparison with different codecs cannot show how much of the gain comes from that layout. The production handoff is therefore deliberate, with Avro for ingestion and event streaming, followed by ORC or Parquet for analytics.

What ORC's stripe architecture does for predicate pushdown on S3

ORC divides files into stripes, typically around 64 MB by default, with embedded indexes containing min/max values, row positions, and bloom filters. These let an engine test whether a region could contain rows matching a query predicate before retrieving its data. The metadata must still be read. Once it has been evaluated, however, the engine can omit ranged GETs for stripes that cannot match and retrieve only the relevant regions of those that can.

This is how skipping becomes an S3 benefit rather than merely an engine optimization. A query over time-series data sorted by timestamp can reject stripes outside its requested time range without fetching their data bytes. Column pruning can also cut how much the engine reads within the stripes it selects. ORC therefore works well when highly selective filters act on physically ordered data, especially when the table is large enough that whole-stripe skipping cuts the number of requests by a lot.

That read efficiency has an ingestion cost. Building 64–256 MB stripes can mean buffering millions of rows, and statistics and encodings use CPU. Compared with row-based ingestion, this preparation can move ingest latency from seconds to minutes. ORC’s case is strongest in Hive-centric stacks, where its indexes and predicate pushdown are deeply integrated, rather than in an abstract comparison that assumes every engine uses them equally well.

Parquet's row-group model, its compression benchmark result, and its ecosystem position

Parquet starts the read decision at the footer, which identifies row-group offsets and column-chunk byte ranges. Row groups are typically 128–512 MB, with data divided into column chunks and statistics including minimum, maximum, and null count, plus optional bloom filters. An engine evaluates those statistics before requesting relevant ranges. The footer provides the map for selective reads. Unneeded columns generate no data GET requests, while row-group statistics can also exclude regions that cannot satisfy the filter.

A controlled benchmark on a 3-million-row, 11-column table found Parquet with Snappy meaningfully smaller than ORC with Snappy, and Parquet with DEFLATE meaningfully smaller than ORC with DEFLATE. Parquet with zstd was the smallest combination tested and the fastest on full scans and pruned reads. That result has a clear boundary, however. It came from one mixed-schema dataset.

The same analytical shape appears in the DuckDB comparison, where a filtered query touching 4 of 11 columns ran dramatically faster on Parquet with zstd than on CSV. Spark, Trino, cloud warehouses, and the major open table formats use Parquet as a default underneath. Amazon S3 Tables initially supported Parquet before adding ORC and Avro, and its support for new Iceberg V3 types further favors Parquet.

The compression claim that most comparisons get wrong

ORC’s compression reputation grew from Hive workloads using numerical and flat data, where tuned encodings can work particularly well. The matched-codec results above show why that claim does not hold for mixed schemas. A format’s reputation is not a measurement of the table an engineer is about to store.

Codec choice can also obscure the comparison. Putting Parquet with Snappy against ORC with Zlib changes both format and compression, so the resulting size difference cannot be attributed to format alone. Matching codecs is the minimum useful control. Separately, the benchmark found that ORC with zlib wrote significantly slower than Parquet with zstd while still producing a larger file, illustrating why operational defaults deserve scrutiny even when the comparison concerns complete configurations.

For S3, getting the size wrong affects how much data is stored for the retention period and how many bytes are read as queries run again and again. The practical response is to measure actual data, schema, codec, and write toolchain before using compression as a deciding argument. A fixed seed, published script, and raw results in a repository give the team a way to test whether the apparent advantage holds under its own conditions.

How the open table format layer (Iceberg) changes what the file format decision controls

Apache Iceberg places a metadata layer above the files in S3, managing transactions, schema, partitions, and time travel through manifests and table metadata. Choosing a file format no longer has to settle every question about how the table will evolve. It still determines column pruning, region skipping, compression, write throughput, and compatibility outside the catalog.

Schema evolution makes that separation especially useful. Iceberg tracks columns by unique IDs rather than names or positions, so adding, dropping, renaming, reordering, or widening columns can be metadata-only operations without rewriting data files. Reusing a dropped column’s name does not resurrect its old values, and reordering columns does not reassign them. Avro’s streaming schema capabilities remain useful at ingestion, but they are no longer a reason to accept row-based scans in an Iceberg analytical table.

Iceberg V3 adds deletion vectors and row lineage, supporting lower compaction overhead and identifying changed rows without full-table scans. On S3 Tables, the new variant, nanosecond timestamp, geometry, and geography types currently have Parquet-only support. The specification itself defines ORC mappings for geometry and geography, so the service restriction should not be confused with a blanket specification limit. S3 Tables also defaults compaction output to Parquet, another signal of the ecosystem’s direction.

Matching format mechanics to S3 access patterns: a decision map for real pipeline architectures

For large time-series or event tables dominated by narrow time-window queries, ORC is a candidate when data is sorted or clustered on the filter column. Without that ordering, stripe ranges may let the engine rule out little of the data. In a Hive or Hive-adjacent stack, the write cost of building stripes and statistics can be spread across many selective reads. A team taking this path should check its engine’s predicate-pushdown support instead of assuming that it will make full use of the file’s indexes.

For mixed-schema analytical tables whose queries touch varying subsets of columns, Parquet with zstd is the safe greenfield default supported by the supplied measurements. Column pruning works even when filters rule out few rows. This branch also favors Parquet for unpredictable lakehouse workloads and teams without an existing Hive investment, because broad engine support and access to S3 Tables’ Iceberg V3 features reduce the reasons to specialize around ORC.

For high-throughput streaming or change data capture, Avro belongs at the ingestion boundary when write latency and decoupled schema evolution take priority. The settled data can then be converted or compacted into Parquet, or ORC for a suitable Hive stack. S3 Tables can perform that conversion through automatic compaction within an Iceberg table, with Parquet as the default target. The boundary between ingestion and analytics earns its place because the two stages ask different things of storage.

Before committing, a team should measure file size with matched codecs on a sample of actual data, pruned read time for its common projection, and write throughput at the target ingest rate. S3 GET count and bytes per query should be measured in the target engine as well. Those readings connect format mechanics to operating conditions. They can justify ORC for a filter-heavy Hive workload or reveal that a presumed advantage does not survive the actual query pattern.

How AI agents querying S3-backed tables change the format calculus

AI agents intensify the same I/O problem through frequent queries with narrow projections and hard byte-scan limits. AWS reference architectures for agentic Athena access instruct agents to retrieve schema before constructing SQL, including columns, data types, partition keys, and storage details. Schema discovery becomes recurring work. Agents using Iceberg catalogs can obtain schema there, while Parquet and ORC footers provide a small metadata read when file-level introspection is needed.

Per-query byte-scan limits can stop an agent’s request before it completes, so Parquet’s column pruning helps queries stay within those limits and finish sooner. ORC’s pruning and stripe skipping can serve the same broader goal where its access pattern fits. Human analysts may run tens of queries per hour, while agents can run hundreds in parallel, amplifying small differences in retrieved bytes across many slices of shared data.

That scale also requires controls beyond the file format. Agents that access shared S3-backed tables need read-only, isolated compute and full query logging so their autonomous activity can be accounted for. Schema legibility helps the engine interpret requests and supports more precise cost attribution, but it does not replace those controls.