InfoQ Homepage Apache Kafka Content on InfoQ
-
Sleeping at Scale - Delivering 10k Timers per Second per Node with Rust, Tokio, Kafka, and Scylla
Lily Mara and Hunter Laine walk through the design of a system, its performance characteristics, and how they scaled it.
/presentations/scale-rust-kafka/en/mediumimage/MaraLaine-medium-1713348578894.jpg)
-
How to Build a Reliable Kafka Data Processing Pipeline, Focusing on Contention, Uptime and Latency
Lily Mara shares how OneSignal improved the performance and maintainability of its highest-throughput HTTP endpoints (backed by a Kafka consumer in Rust) by making it an asynchronous system.
/presentations/reliable-kafka-data-processing-pipeline/en/mediumimage/LilyMara-medium-1702973251200.jpg)
-
Tales of Kafka @Cloudflare: Lessons Learnt on the Way to 1 Trillion Messages
Andrea Medda and Matt Boyle discuss Kafka on the way to one trillion messages, and the internal tools used to ease adoption as well as improve resiliency.
/presentations/kafka-cloudflare/en/mediumimage/Matt-Boyle-Andrea-Medda-medium-1698936435255.jpg)
-
Streaming from Apache Iceberg - Building Low-Latency and Cost-Effective Data Pipelines
Steven Wu discusses the design of the Flink Iceberg, comparing the Kafka and Iceberg sources for streaming and how the Iceberg streaming source can power many common stream processing use cases.
/presentations/apache-iceberg-streaming/en/mediumimage/Steven-Wu-medium-1689934544144.jpg)
-
Co-Designing Raft + Thread-per-Core Execution Model for the Kafka-API
Alex Gallego discusses the lessons learned building a new storage engine from scratch with no virtual memory, no page cache, with purpose-built read-ahead and write-behind strategies.
/presentations/raft-kafka-api/en/mediumimage/Alex-Gallego-m-1637914189671.jpg)
-
Building and Scaling a Control Plane for 1000s of Kafka Clusters
Gwen Shapira and Vivek Sharma discuss some architectural highlights of building, evolving and scaling a control plane for thousands of Kafka clusters, and some challenges encountered.
/presentations/kafka-clusters-control-plane/en/mediumimage/Vivbig-1636122905463.jpg)
-
From Mainframe to Microservices with Pivotal Platform and Kafka: Bridging the Data Divide
Dmitry Milman and Ankur Kaneria showcase how Pivotal and Apache Kafka are leveraged within Express Scripts’ transformation from mainframe to a microservices-based ecosystem, ensuring data integrity.
/presentations/express-scripts-pivotal-kafka/en/mediumimage/Doibig-1594908860212.JPG)
-
Everything You Wanted to Know about Apache Kafka But Were Too Afraid to Ask!
Ricardo Ferreira explains what a streaming platform such as Apache Kafka is and some of the use cases and design patterns around its use.
/presentations/streaming-platform-kafka/en/mediumimage/Ricbig-1594572536079.JPG)
-
Kafka: A Modern Distributed System
Tim Berglund covers Kafka's distributed system fundamentals: the role of the Controller, the mechanics of leader election, and the role of Zookeeper today and in the future.
/presentations/kafka-controller-zookeeper/en/mediumimage/Timbig-1588674373241.JPG)
-
Monitoring and Tracing @Netflix Streaming Data Infrastructure
Allen Wang talks about the design and implementation details of the dev/ops tools used by Netflix and highlights the critical roles they play in operating their data infrastructure.
/presentations/netflix-streaming-data-infrastructure/en/mediumimage/Albig-1584722318863.JPG)
-
Streaming with Spring Cloud Stream and Apache Kafka
Oleg Zhurakousky and Soby Chacko explore how Spring Cloud Stream and Apache Kafka can streamline the process of developing event-driven microservices that use Apache Kafka.
/presentations/spring-cloud-stream-kafka-microservices/en/mediumimage/Doibig-1583367224904.JPG)
-
Kafka Needs No Keeper
Colin McCabe talks about the ongoing effort to replace the use of Zookeeper in Kafka: why they want to do it and how it will work.
/presentations/kafka-zookeeper/en/mediumimage/Colbig-1583280959599.JPG)