How DuckLake compares to Databricks, IceBerg, BigQuery and Snowflake?

Update on the 19th of April 2026 regarding DuckLake specification 1.0, found on thegerister.com

DuckDB proposed DuckLake as a standard for metadata and catalogs, claims it simplifies lakehouses by using a standard SQL database for all metadata, instead of complex file-based systems, while still storing data in open formats like Parquet. It will be seen, if this makes them more reliable, faster, and easier to manage.

The corresponding blog post is worth reading, because it gives a systematic view on the high level data architectures of DuckLake, Databricks, IceBerg, BigQuery and Snowflake.

Innovative data systems like BigQuery and Snowflake have shown that disconnecting storage and compute is a great idea in a time where storage is a virtualized commodity. That way, both storage and compute can scale independently and we don't have to buy expensive database machines just to store tables we will never read.

Let us start with the main differences between data files and collections of data files, Lakehouses and Database like systems:

A singe data file is usually some file with structured data, which can be used via libraries and modules in a programing language or be used with some standard software. The problem here is, that each file has to be handled individually and there are many different file formats around all with their corresponding libriaries, modules, API's etc.
From the user perspective the main idea behind Lakehouses is to bring all the different types of data (files) together and use them just like all of them are in one place and with all of them usable in the same (standard) way. This is mainly done with technical metadata stored along the data files. This technical metadata then is also used to optimize the data retrievial.
If you add some table catalog information to this Lakehouse idea you will basically get a complete database like data storage.

Based on this definitions we can now answer the question, how DuckLake data architecture compares to Databricks, IceBerg, BigQuery and Snowflake:

Data architectural layers	Databricks	Iceberg	DuckLake	BigQuery	Snowflake
Catalog	Unity Catatog in Databricks, Hive catalog in DeltaLake	Iceberg Catalog	Schema in SQL Database for both catalog and table metadata	Spanner	FoundationDB
Metadata layer	DeltaLake Metadata files	Iceberg metadata files (manifest list, manifest files)	Schema in SQL Database for both catalog and table metadata	Spanner	FoundationDB
Data layer	Data files (Parquet etc.)	Data files (Parquet etc.)	Data files (Parquet etc.)	Data files	Data files
Caching	in memory DB engine (Spark, SQL Warehouse, Photon)	mostly in memory DB engine used	in memory DB Engine (DuckDB)	in memory DB engine (BigQuery)	in memory DB engine (Snowflake)
Buffering			batching "small changes" in chunks first stored in a RDBMS and then written in a single bulk write to cloud object storage

Please refer also to the following articles for more background:

Abbreviations and terms used in this post:

DB = database
DBMS = database management system
RDBMS = relational database management system
SQL = structured query language
Metadata = some data describing a data object, such us created timestamp, last modified timestamp, owner and information about the structure of the data object
Catalog = a set of medadata entries corresponding to each other and/or belonging together
RAM = random access memory
SRAM = static random access memory (usually used for caching in processors)
DRAM = dynamic random access memory (usually used for storing running software together the data needed)
SSD = solid state disk
Cache = a technique used by a DBMS where a subset of always, frequently or often needed information, data and metadata is stored in a faster memory or storage subsystem, than the usually used (for example random reading and writing of data to RAM is much faster than to an SSD, which are usually faster than hard disks, where the latter are faster than tape drives). Caching is not restricted to a predefined use scenario, it can be multipurpose.
Buffer = a technique used in software design to make frequent smaller data reads and/or writes faster by using a data object in a faster memory or storage subsystem, such as RAM, and to do expensive and/or slower data read or write operations to a slower (data) storage system only when the read, write or read/write buffer is full. Buffering is usually used to smooth data transfer operations or to make things faster while reading or writing data into a persistent data storage system.

Related posts