Netflix created a new solution for incremental processing in its data platform. The incremental approach reduces the cost of computing resources and execution time significantly as it avoids processing complete datasets. The company used its Maestro workflow engine and Apache Iceberg to improve data freshness and accuracy and plans to provide managed backfill capabilities.
Netflix has migrated all its data processing to Maestro, the latest generation of home-grown workflow orchestration platform catering to various use cases, including ETL pipelines, ML model training, batch jobs, etc. At the same time, the company was looking to address some common challenges with data pipeline execution.
Firstly, the platform lacked effective support for low-latency batch processing use cases. This led to a few internal solutions, like Psyberg, which work only for specific use cases and introduce tight coupling to the business logic. Secondly, late-arriving data forced workflows to use a lookback window, increasing execution time and costs. Lastly, backfilling datasets required a fair amount of engineering effort as dedicated backfill workflows had to be created, executed, and validated.
The solution that addresses these challenges enables incremental processing of added/updated records and leverages rich features of Apache Iceberg, a high-performance format for massive analytics tables that supports expressive SQL queries, schema evolution, and time travel and rollback, among other things.
Incremental Processing Solution Using Apache Iceberg (Source: Netflix Technology Blog)
The IPS solution implemented a lightweight approach to create an additional Iceberg table (called ICDC table) that only stores references from the original table without copying any data files, which is efficient and cost-effective. Additionally, the new table captures the change range for specified data fields. Data engineers can integrate the incremental processing approach into existing Maestro workflows by using a new workflow step type or/and a new incremental trigger mechanism.
The introduction of the incremental processing capability resulted in new patterns. The most basic one is using only the ICDC table in the workflow pipeline if no other data sources are needed, which significantly simplifies the workflow execution and removes the need to use the lookback window completely. Other patterns involve using incremental change data or captured change range information to optimize data transformation logic to limit the scope of reprocessing when the workflow uses many source tables.
Jun He, the staff software engineer at Netflix, shared the overview of capabilities the team is planning to work on next:
We will improve IPS to support more complicated cases beyond append-only cases. IPS will be able to keep track of the progress of the table changes and support multiple Iceberg table change types (e.g. append, overwrite, etc.). We will also add managed backfill support into IPS to help users to build, monitor, and validate the backfill.