InfoQ Homepage Apache Kafka Content on InfoQ
-
/articles/beyond-offset-lag-kafka-apache-hudi/en/smallimage/beyond-offset-lag-kafka-apache-hudi-thumbnail-1787577734265.jpg)
Beyond Offset Lag: Computing Time in Queue for Apache Hudi Data Lake Pipelines at Petabyte Scale
In this article, author Srikanth Mamidala discusses the data lake architecture used for analytics, reporting, and machine learning and shows how to manage the consumer lag metrics when using Kafka and Apache Hudi.
-
/articles/tradeoffs-event-driven-design/en/smallimage/tradeoffs-event-driven-design-thumbnail-1782458803116.jpg)
Scaling Java-Based Real-Time Systems: the Hidden Tradeoffs of Event-Driven Design
Event-driven architecture promises scalability, but in Java-based real-time systems the tradeoffs only surface in production. Drawing on a Java/Kafka contact center platform handling 80k BHCC across 10k agents, this article details where the design breaks down—state management, partition limits, deduplication, JVM tuning, cascading consumer failures—and the Redis-backed patterns that fixed each.
-
/articles/architecting-cloud-native-kafka/en/smallimage/architecting-cloud-native-kafka-thumbnail-1779433382367.jpg)
Architecting Cloud-Native Kafka: from Tiered Storage towards a Diskless Future
This article explores Kafka's transition toward a cloud-native architecture, examining how tiered storage, FinOps telemetry, elastic consumer scaling, virtual clusters, and Share Groups reshape the operational and economic model of event streaming platforms. It also analyzes emerging diskless-storage proposals and their architectural trade-offs.
-
/articles/schema-proliferation-problem/en/smallimage/schema-proliferation-problem-thumbnail-1779270222602.jpg)
The Schema Proliferation Problem in Kafka and Flink Pipelines: How to Solve It
Schema proliferation builds slowly and gets expensive fast. One schema per event type feels right until there are ten tables, union queries spanning all of them, and a single field rename touching every schema. Discriminator-based schema consolidation collapses that to two tables, turning multi-table unions into a single query, while new variants are additive and don't break existing consumers.
-
/articles/apache-kafka-stretch-cluster-failures/en/smallimage/apache-kafka-stretch-cluster-failures-thumbnail-1750152098922.jpg)
Analyzing Apache Kafka Stretch Clusters: WAN Disruptions, Failure Scenarios, and DR Strategies
Proficient in analyzing the dynamics of Apache Kafka Stretch Clusters, I assess WAN disruptions and devise effective Disaster Recovery (DR) strategies. With deep expertise, I ensure high availability and data integrity across multi-region deployments. My insights optimize operational resilience, safeguarding vital services against service level agreement violations.
-
/articles/choosing-message-broker/en/smallimage/beyond-trends-thumbnail-1741943360849.jpg)
Beyond Trends: A Practical Guide to Choosing the Right Message Broker
Choosing the right message broker for your application requires matching the appropriate technology with the messaging patterns needed. Message brokers can be broadly categorized as either stream-based or queue-based, each offering unique strengths and trade-offs.
-
/articles/managing-memberships-netflix/en/smallimage/Managing238M-1710832361735.jpg)
Managing 238M Memberships at Netflix
In this article Surabhi Diwan shared how the Netflix membership team does distributed systems: the architecture bets, technology choices, and operational semantics that serve the needs of Netflix’s ever-growing member base.
-
/articles/kafkaflow-dotnet-framework/en/smallimage/building-kafka-event-driven-applications-with-KafkaFlow-small-1693903734363.jpg)
Building Kafka Event-Driven Applications with KafkaFlow
KafkaFlow, a .NET open-source project, simplifies Kafka-based event-driven app development with features like middleware for message processing, enhancing maintainability, customization potential, and allowing developers to prioritize business logic.
-
/articles/kafka-clusters-cloudflare/en/smallimage/tales-of-Kafka-at-Cloudflare-small-1684768928637.jpg)
Tales of Kafka at Cloudflare: Lessons Learnt on the Way to 1 Trillion Messages
Cloudflare uses Kafka clusters to decouple microservices and communicate the creation, change or deletion of various resources via protobuf, a common data format in a fault-tolerant manner. The authors suggest investing in metrics for problem detection, prioritizing clear SDK documentation, and balancing flexibility and simplicity for standardized pipelines.
-
/articles/series-apache-kafka-streams/en/smallimage/apache-kafka-article-series-logo-small-1673455781546.jpg)
Article Series: Developing Apache Kafka applications on Kubernetes
Apache Kafka has integrations with most of the languages used these days, but in this article series, we cover its integration with Java. In this series, we also discuss how to provision, configure and secure an Apache Kafka cluster on a Kubernetes cluster.
-
/articles/secure-kafka-cluster-strimzi/en/smallimage/securing-a-kafka-cluster-in-kubernetes-using-strimzi-small-1671647644310.jpg)
Securing a Kafka Cluster in Kubernetes Using Strimzi
Deploying an Apache Kafka cluster to Kubernetes is easy if you use Strimzi, but that’s only the first step; you need to secure the communication between Kafka and the consumers and producers, provide RBAC to access topics, spread the secrets correctly to Kafka Connect components and all using a Kubernetes GitOps way.
-
/articles/high-fidelity-data-streams/en/smallimage/building-and-operating-high-fidelity-data-streams-small-logo-1664284454024.jpg)
Building & Operating High-Fidelity Data Streams
At QCon Plus 2021 last November, Sid Anand, chief architect at Datazoom and PMC Member at Apache Airflow, presented on building high-fidelity nearline data streams as a service within a lean team. In this talk, Anand provides a master class on building high-fidelity data streams from the ground up.