Netflix created a new solution for incremental processing in its data platform, leveraging Maestro workflow engine and Apache Iceberg to improve data freshness and accuracy. The solution enables incremental processing of added/updated records, reduces the cost of computing resources and execution time, and provides managed backfill capabilities. The IPS solution implemented a lightweight approach using an additional Iceberg table called ICDC table, which stores references from the original table without copying data files. Data engineers can integrate the incremental processing approach into existing Maestro workflows, resulting in new patterns and optimizing data transformation logic. The team at Netflix plans to improve IPS to support more complicated cases and multiple Iceberg table change types.
