BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News LinkedIn Cuts Daily Spark Shuffle Fetch Failures from 750 to 20 with Apache Celeborn

LinkedIn Cuts Daily Spark Shuffle Fetch Failures from 750 to 20 with Apache Celeborn

Listen to this article -  0:00

LinkedIn has deployed Apache Celeborn as a shared remote shuffle platform across the majority of its Apache Spark and Apache Flink workloads, separating intermediate data storage from compute infrastructure. The deployment reduces daily shuffle fetch failures from approximately 750 to 20, lowers compute costs by 10%, and improves application runtimes by 14%, compared with its previous Magnet implementation.

Daily shuffle fetch failures decreased from approximately 750 to 20 (Source: LinkedIn Blog Post)

LinkedIn's offline data platform executes more than 300,000 Spark jobs daily across thousands of production nodes, consuming approximately 200 petabyte-hours of compute. As the company moves toward a unified Flink architecture for real-time and historical processing, large-scale backfills introduce additional shuffle workloads. Shuffle moves intermediate data between processing stages, potentially stressing networks, disks, and application runtimes.

LinkedIn previously addressed shuffle read inefficiencies with Magnet, its push-based shuffle implementation contributed to Apache Spark 3.2. Magnet aggregates shuffle blocks by reducer partition into larger files, reducing random disk reads. However, shuffle storage remained within the compute cluster, where External Shuffle Services shared CPU, memory, disk I/O, and network resources with compute workloads.

With Apache Celeborn, map tasks push intermediate data to dedicated workers, which aggregate and store partitions independently of compute executors. Reducers fetch the data directly from these workers, allowing shuffle storage to scale independently of compute and reducing dependence on executor lifecycles. Unlike Magnet, which uses merger External Shuffle Services within the Spark compute cluster, Celeborn runs as a separate shuffle service with dedicated workers and a replicated control plane. This separation reduces resource contention between compute and shuffle workloads.

Celeborn architecture at LinkedIn (Source: LinkedIn Blog Post)

Aravind Patnam, a staff software engineer at LinkedIn, explained,

Operating it reliably across thousands of nodes and hundreds of thousands of daily applications required substantial engineering beyond the upstream project.

The deployment uses a five-node Raft-based control plane distributed across availability zones, with Apache Ratis providing metadata replication, leader election, and failover. LinkedIn also introduced network-aware replication and integrated Celeborn with its host health platform to deprioritize workers approaching maintenance and exclude degraded workers. These changes contributed to a 20-fold reduction in task failures during infrastructure disruptions compared with Magnet.

LinkedIn further optimized the data plane by moving shuffle storage to NVMe SSDs, reducing overall shuffle duration by 40%. Using Zstandard compression instead of LZ4 reduced the required shuffle worker fleet by approximately 35%. The company also provisioned workers with 100 Gbps network interfaces and improved slot selection time sixfold through JVM and data-plane optimizations. Across the fleet, shuffle cost decreased by nearly 90% per task, while the proportion of executor runtime spent waiting on shuffle operations fell by approximately 10–15% from its peak.

The converged Flink platform processes hundreds of millions of events per hour in historical backfills, completing multi-day workloads in single-digit hours. Thousands of pipelines now use unified Flink computations for nearline processing and historical corrections, replacing parallel Spark implementations. LinkedIn is exploring caching and faster upstream upgrades while continuing to contribute to Celeborn.

About the Author

Rate this Article

Adoption
Style

BT