Netflix is moving toward the open-source Apache Flink Autoscaler for more than 30,000 streaming jobs across multiple AWS regions, after finding that its cluster-level approach was less effective for complex, stateful pipelines with operators that have different processing requirements. Netflix reports that one team reduced annualized Flink compute expenditure by 58%, saving approximately $1.1 million annually.
Comparison of Netflix’s two Flink Autoscalers (Source: Netflix Blog Post)
Netflix has run Apache Flink since 2017 and built its first autoscaler around 2019. The system ran on Mantis and consumed cluster-level telemetry from Atlas, including CPU, network utilization, Kafka lag, input rate, and consume rate. It adjusted the total number of TaskManagers and reduced resource usage by 25% to 45% across thousands of pipelines.
The limitation was the scaling unit. Because the original autoscaler reasoned about the cluster rather than individual operators, all operators in a job effectively shared the same scaling decision. That became less suitable for stateful pipelines containing branches, joins, and terabytes of state, where different parts of the dataflow can have different processing requirements.
The Apache Flink Autoscaler instead uses metrics exposed by the running job to estimate each operator’s true processing rate from throughput and busy time. It then walks the job graph and calculates required parallelism for individual vertices. The approach is described in FLIP-271, which addresses autoscaling for heterogeneous streaming jobs and the cost of rescaling stateful applications.
The technique also builds on research from the DS2 project. Vasiliki Kalavri, a systems researcher involved in the work, said the project initially explored critical path analysis before adopting True Processing Rate as a simpler baseline. It turned out that this very simple idea worked really well. The approach subsequently became part of Flink’s autoscaling work.
Netflix integrated the autoscaler with its internal control plane rather than deploying it directly through the Flink Kubernetes Operator. A Spring Boot service uses Temporal workflows to isolate autoscaling decisions for individual jobs. Netflix also modified JobManager metric collection to support jobs with up to 3,000 subtasks, added server-side metric filtering, preserved forward connected subgraphs during scaling, and added handling for sink backpressure.
The OSS-based Flink Autoscaler architecture with Temporal workflows (Source: Netflix Blog Post)
The forward connection issue has also appeared in Apache Flink developer discussions. Changing parallelism across a FORWARD connection can require redistribution, while Netflix’s implementation keeps forward connected operators together. An open FLINK-38538 issue separately highlights cases where busy operators can be affected by output ratio-based scaling decisions.
Flink Job DAG Showing Current and Desired Parallelism Per Vertex. (Source: Netflix Blog Post)
Unlike generic event-driven autoscalers such as KEDA, which scale workloads from external metrics or events, Flink’s autoscaler reasons about the internal dataflow graph and operator capacity. Netflix currently uses a 0.45 utilization target, below Flink’s 0.7 community default, to reduce aggressive rescaling of large stateful jobs. The company plans to migrate its remaining internal autoscaling use cases to the open-source implementation, while investigating Flink 2’s disaggregated state architecture to address the cost of state recovery during rescaling.