BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News QCon San Francisco 2026: The Future of Apache Iceberg

QCon San Francisco 2026: The Future of Apache Iceberg

Listen to this article -  0:00

Apache Iceberg V4 proposes a redesign of table metadata to improve scalability and support more frequent writes, alongside new capabilities for data management and schema evolution. Daniel Weeks, co-creator of Apache Iceberg and member of technical staff at Databricks, presented a session at QCon San Francisco on the fourth version of the Apache Iceberg open table format specification.

Iceberg V4 will address the limitations of managing very large tables while expanding the format to support new data types and workloads. The speaker described the new version as a major reimagining of the table format rather than an incremental step, with changes including a new metadata tree, more efficient commits, typed column statistics, relative file paths, and support for file and vector data types. While earlier versions left most of the metadata structure intact, V4 addresses problems that emerge at large scale and adds support for new use cases. Weeks told the audience in San Francisco:

V4 is a major reimagining of a table format. We didn't get everything right in V3 and prior, so now we're taking the opportunity to fix some of the biggest things that are challenges as we scale up to even larger data sets and we bring on new use cases.

Apache Iceberg is an open table format for managing large analytical datasets across different data processing engines. Originally developed at Netflix in 2017, it was open-sourced in 2018 and became an Apache top-level project in 2020. The main alternatives among open table formats are Delta Lake, created by Databricks and now a Linux Foundation project, Apache Hudi, originally developed at Uber, and Apache Paimon, which grew out of the Flink community and gained adoption for streaming-oriented workloads. However, interoperability tools are increasingly blurring the lines between them.

In his talk, Weeks covered three areas of the proposed Iceberg V4 changes: metadata and scalability, including the Adaptive Metadata Tree, single-file commits, improved column statistics, relative paths, and column-level updates; new table features, such as expression-based default values and constraints; and new data types, particularly file and vector types, which support additional workloads such as AI and search. Regarding faster commits, Weeks explained:

This gets us into use cases where we can stream data much faster. We can do commits at the 2nd level or subsecond level, where right now we largely recommend that you don't do commits under 2 minutes.

In V3, adding a data file requires at least four I/O operations, including manifest creation and rewriting. V4 simplifies this process by allowing the metadata tree's root to point directly to data files, reducing the operation to a single metadata write. According to Weeks, this could enable commits every second or even at subsecond intervals, compared with the current recommendation of at least two minutes between commits. Iceberg V3, introduced in 2024, added deletion vectors, row lineage, new data types, encryption, and default values.

V4 replaces V3's binary column statistics with structured, typed statistics, improving projection and schema evolution. Furthermore, V4 introduces expression-based default values, check, unique, and primary key constraints, and statistics-only fields that support partitioning, sorting, and filtering without storing additional data. It will also track constraint validation status, allowing engines to determine when further checks are needed.

In August, Weeks started an initial V4 scope discussion on the Apache Iceberg developer mailing list, with Steven Wu suggesting adding monotonic snapshot and row-level update timestamps, while Manu Zhang discussed efficient column updates and the removal of equality deletes.

During the Q&A, Weeks explicitly confirmed that V4 metadata will not be backward compatible with older readers. Existing V3 metadata can remain in place during an upgrade, but older readers cannot consume the new representation.

About the Author

Rate this Article

Adoption
Style

BT