DuckDB Internals: storage and the query pipeline

Category: Engineering post
Status: discussion
Sources: greybeam.ai, discussion
Summary: A deep-dive on DuckDB internals (dated 2026-05-04, part one of three) resurfaced on the HN front page (431 points). It walks the in-process execution architecture, columnar storage with compression and zone maps (min, max, and null count per row group), row groups of up to 122,880 rows, the 256 KB default block size, the native file format with checksum verification, pipeline breakers in the query pipeline, and dynamic join-filter pushdown that turns into an IN list when the build side has fewer than 50 distinct join keys.
Why it matters: Concrete storage-layout and query-pipeline detail helps engineers reason about when DuckDB's columnar, in-process design fits an analytics workload.