From Ledgers to Intelligence Part 10: The Lakehouse Delta Lake, Apache Iceberg, and the Convergence of Lake and Warehouse

Digital Transformation | June 2026

The data lake had solved the storage problem cheap, scalable, format-agnostic object storage that could hold everything. What it had not solved was the reliability problem: the absence of transactions, schema enforcement, and consistent reads that made relational databases trustworthy for decades. Data lakes were large; they were not dependable in the ways that enterprise analytics required.

The lakehouse architecture emerged between 2019 and 2021 as the synthesis of these two paradigms: the economics and scalability of the data lake combined with the reliability and performance of the data warehouse. It was enabled by a new category of technology the open table format that brought ACID transaction guarantees to files stored in object storage.

Delta Lake: ACID Transactions on Object Storage

Delta Lake was developed by Databricks and open-sourced in April 2019. It is not a storage format but a table format a layer of metadata and protocol on top of Parquet files in object storage that provides transactional guarantees. The core innovation is the Delta transaction log: a sequence of JSON files recording every operation performed on the table (inserts, updates, deletes, schema changes). Because all changes are recorded atomically in the transaction log, Delta Lake can provide ACID guarantees even though the underlying storage (S3 or ADLS) is eventually consistent.

The practical implications were significant. ACID transactions meant that concurrent writes to a Delta table were safe: two jobs writing to the same table simultaneously would not corrupt each other’s data. Schema enforcement meant that a record with a missing required column or an incompatible data type would be rejected at write time, not silently accepted and discovered later in a query. Time travel meant that any previous version of the table could be queried by timestamp or version number enabling point-in-time analysis, audit trails, and rollback of accidental writes.

Delta Lake also introduced MERGE semantics the ability to upsert (update if exists, insert if not) records based on a matching key which had been impossible with plain Parquet on object storage. MERGE was the operation required for CDC (Change Data Capture) pipelines that needed to apply incremental changes from operational systems to the lake. Its absence had been a major limitation of the raw data lake architecture; Delta Lake’s MERGE was the solution.

Apache Iceberg and Apache Hudi

Delta Lake was not the only open table format to emerge. Apache Iceberg, originally developed at Netflix and donated to the Apache Software Foundation, solved similar problems with a different design philosophy. Where Delta Lake’s transaction log is a sequence of JSON commit files, Iceberg uses a tree of metadata files (manifest lists, manifests, and data file references) that enables extremely efficient planning of large-scale queries Iceberg can determine which data files are relevant to a query without scanning the entire table, making it particularly performant for tables with many billions of rows partitioned across thousands of files.

Iceberg’s design also prioritised engine-neutrality from the start: any engine implementing the Iceberg spec Spark, Flink, Trino, Presto, Snowflake, BigQuery, Athena could read and write Iceberg tables, with no lock-in to any particular compute platform. This open-standard approach was philosophically different from Delta Lake, which was closely associated with Databricks (though also open-sourced and supported by multiple engines).

Apache Hudi (Hadoop Upserts Deletes and Incrementals), developed at Uber, focused specifically on incremental processing the need to efficiently process only records that had changed since the last pipeline run. Hudi’s architecture included built-in support for record-level indexing that made upserts and deletes dramatically faster than rewriting entire partitions, making it the preferred format for pipelines processing high-frequency CDC streams.

The Lakehouse Concept: Armbrust et al. (2021)

The academic formalisation of the lakehouse architecture came in a 2021 paper by Michael Armbrust, Ali Ghodsi, Reynold Xin, and Matei Zaharia the core researchers behind both Spark and Delta Lake. Their paper defined the lakehouse as a data management system that “combines the key benefits of data lakes and data warehouses: open direct-access data formats, elastic compute, support for diverse workloads including ML, and the performance and reliability that business intelligence applications require.”

The paper’s key architectural insight was that the performance gap between data lakes and data warehouses had been closed by three innovations: (1) caching of hot data in fast SSD storage closer to compute; (2) advanced query optimisation including data skipping (using table statistics to skip irrelevant files without reading them); and (3) vectorised query execution processing data in SIMD-friendly columnar batches rather than row by row. Together, these optimisations delivered sub-second analytical query performance on files stored in object storage comparable to dedicated columnar warehouses at a fraction of the cost.

Why Open Formats Matter

The emergence of multiple competing open table formats and the support for all of them by multiple compute engines represented a fundamental architectural shift: the decoupling of data storage from query execution. In the on-premise warehouse era, storage and compute were inseparable your Teradata nodes stored the data and processed it. In the cloud data warehouse era, storage was separated from compute but remained within a single vendor’s platform (Snowflake’s proprietary format in Snowflake’s object storage, readable only by Snowflake’s query engine).

The lakehouse with open table formats goes further: data stored in Delta Lake or Iceberg format in your own S3 bucket can be queried by Spark, by Trino, by Snowflake (via external tables), by Athena, by BigQuery Omni, and by any other engine implementing the format spec. Vendor lock-in becomes structural the data itself is portable, and the choice of compute engine is separable from the choice of where to store data.

---

References

1. Armbrust, M. et al. (2021). Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics. Proceedings of CIDR 2021.
2. Databricks (2019). Delta Lake: Open Source Reliability for Data Lakes. Databricks Blog.
3. Apache Iceberg (2018). Apache Iceberg: A Table Format for Huge Analytic Datasets. Netflix Tech Blog.
4. Zaharia, M. et al. (2016). Apache Spark: A Unified Engine for Big Data Processing. Communications of the ACM, 59(11), 56–65.
5. Bae, J. et al. (2021). Apache Hudi: The Data Lake Platform. Uber Engineering Blog.
6. Ramakrishnan, R. et al. (2017). Azure Data Lake Store: A Hyperscale Distributed File Service for Big Data Analytics. Proceedings of SIGMOD 2017.
7. Sai, S. et al. (2023). Format Wars: A Comparison of Open Table Formats. Databricks Engineering Blog.