InfoQ Homepage Kafka Streams Content on InfoQ
Articles
RSS Feed-
/articles/kafka-clusters-cloudflare/en/smallimage/tales-of-Kafka-at-Cloudflare-small-1684768928637.jpg)
Tales of Kafka at Cloudflare: Lessons Learnt on the Way to 1 Trillion Messages
Cloudflare uses Kafka clusters to decouple microservices and communicate the creation, change or deletion of various resources via protobuf, a common data format in a fault-tolerant manner. The authors suggest investing in metrics for problem detection, prioritizing clear SDK documentation, and balancing flexibility and simplicity for standardized pipelines.
-
/articles/series-apache-kafka-streams/en/smallimage/apache-kafka-article-series-logo-small-1673455781546.jpg)
Article Series: Developing Apache Kafka applications on Kubernetes
Apache Kafka has integrations with most of the languages used these days, but in this article series, we cover its integration with Java. In this series, we also discuss how to provision, configure and secure an Apache Kafka cluster on a Kubernetes cluster.
-
/articles/secure-kafka-cluster-strimzi/en/smallimage/securing-a-kafka-cluster-in-kubernetes-using-strimzi-small-1671647644310.jpg)
Securing a Kafka Cluster in Kubernetes Using Strimzi
Deploying an Apache Kafka cluster to Kubernetes is easy if you use Strimzi, but that’s only the first step; you need to secure the communication between Kafka and the consumers and producers, provide RBAC to access topics, spread the secrets correctly to Kafka Connect components and all using a Kubernetes GitOps way.
-
/articles/high-fidelity-data-streams/en/smallimage/building-and-operating-high-fidelity-data-streams-small-logo-1664284454024.jpg)
Building & Operating High-Fidelity Data Streams
At QCon Plus 2021 last November, Sid Anand, chief architect at Datazoom and PMC Member at Apache Airflow, presented on building high-fidelity nearline data streams as a service within a lean team. In this talk, Anand provides a master class on building high-fidelity data streams from the ground up.
-
/articles/change-data-capture-debezium/en/smallimage/logo-1660294927422.jpg)
Debezium and Quarkus: Change Data Capture Patterns to Avoid Dual-Writes Problems
It’s common in microservices to write data in two places, a database and then send the content to another microservice. One approach to tackle this problem is dual writes, but you may lose data because of concurrent writes. Debezium is an open-source project for change data capture using the log scanner approach to avoid dual writes and communicate persisted data correctly between services.
-
/articles/quarkus-with-kafka-streams/en/smallimage/kafka-streams-and-quarkus-real-time-processing-events-small-1656083680635.jpg)
Kafka Streams and Quarkus: Real-Time Processing Events
Consuming Kafka messages is simple; you get them as long as they are produced, but nothing more. But if you need real-time processing of the data (filtering, joining, or manipulating events), just using the Kafka-consuming API might not be the best approach as the resulting code becomes complex. Kafka Streams and Quarkus are the perfect matches to start processing Kafka events in real-time.
-
/articles/real-time-api-kafka/en/smallimage/real-time-api-kafka-s-1602600487985.jpeg)
Real Time APIs in the Context of Apache Kafka
Events offer a Goldilocks-style approach in which real-time APIs can be used as the foundation for applications which is flexible yet performant; loosely-coupled yet efficient. Apache Kafka offers a scalable event streaming platform with which you can build applications around the powerful concept of events.
-
/articles/building-scalable-iot-application-with-apache-kafka/en/smallimage/The-Kongo-Problem-Building-a-Scalable-IoT-Application-with-Apache-Kafka-logo-1581595522263.jpg)
The Kongo Problem: Building a Scalable IoT Application with Apache Kafka
In this article, author Paul Brebner discusses the best practices for developing IoT projects using Apache Kafka and Kafka Streams technologies and how to maximize Kafka scalability.