Notes from an Iceberg migration
Lessons from moving 12 PB of data to Apache Iceberg without downtime — and what we'd do differently.
Twelve petabytes is not a large migration by hyperscaler standards, but it is large enough that anything you get wrong will hurt for a long time.
What worked: dual-writing to Iceberg alongside the legacy Hive tables for six weeks, with automated diffing on every hourly batch. This let us build confidence without risking user-visible incidents.
What we underestimated: the operational surface of table maintenance. Compaction, snapshot expiration, and orphan file cleanup are all first-class concerns. Budget an engineer's time to own them, or you will discover the hard way that your query performance quietly regresses over months.
What we'd do differently: start the catalog decision earlier. Iceberg is a table format, not a catalog, and the catalog you pick — Nessie, Polaris, AWS Glue, Unity — will shape your governance story for years.
Priya leads Dataamps' data platform practice, with 15 years shipping lakehouse and streaming systems inside Fortune 500 enterprises.
More from PriyaRelated articles
FinOps for the modern lakehouse
The five levers that actually move data platform cost, ranked by impact — and the three that don't.
Data contracts are boring — and that's the point
Why the least glamorous idea in the modern data stack is quietly the highest-leverage one for teams tired of 3AM pages.
Why your warehouse is slow (it's probably not the warehouse)
Nine times out of ten, warehouse performance issues are model-design issues. Here's how to tell.