InfoQ Homepage Apache Kafka Content on InfoQ

Articles

RSS Feed

Newer Older

AI, ML & Data Engineering

Beyond the Database, and beyond the Stream Processor: What's the Next Step for Data Management?

Databases have been around forever with the same shape: you make a request to your data and then you receive an answer. Now, stream processors came along with a different approach: data isn’t locked up, it is in motion. Understand how stream processors and databases relate and why there is an emerging new category of databases that focus on data that stays in place as well as data that moves.

Ben Stopford
on Nov 16, 2020
Architecture & Design

Real Time APIs in the Context of Apache Kafka

Events offer a Goldilocks-style approach in which real-time APIs can be used as the foundation for applications which is flexible yet performant; loosely-coupled yet efficient. Apache Kafka offers a scalable event streaming platform with which you can build applications around the powerful concept of events.

Robin Moffatt
on Oct 16, 2020
AI, ML & Data Engineering

Applied Probability - Counting Large Set of Unstructured Events with Theta Sketches

In this article, author Ronen Cohen discusses the solution to processing the event data using Theta Sketches and technologies like HBase and Kafka.

Ronen Cohen
on Jun 29, 2020
AI, ML & Data Engineering

The Kongo Problem: Building a Scalable IoT Application with Apache Kafka

In this article, author Paul Brebner discusses the best practices for developing IoT projects using Apache Kafka and Kafka Streams technologies and how to maximize Kafka scalability.

Paul Brebner
on Feb 15, 2020
AI, ML & Data Engineering

How to Use Open Source Prometheus to Monitor Applications at Scale

In this article, the author discusses how to collect metrics and achieve anomaly detection from streaming data using Prometheus, Apache Kafka and Apache Cassandra technologies.

Paul Brebner
on Jun 20, 2019
AI, ML & Data Engineering

Apache Kafka: Ten Best Practices to Optimize Your Deployment

Author Ben Bromhead discusses the latest Kafka best practices for developers to manage the data streaming platform more effectively. Best practices include log configuration, proper hardware usage, Zookeeper configuration, replication factor, and partition count.

Ben Bromhead
on Oct 19, 2018
AI, ML & Data Engineering

Democratizing Stream Processing with Apache Kafka® and KSQL - Part 2

In this article, author Robin Moffatt shows how to use Apache Kafka and KSQL to build data integration and processing applications with the help of an e-commerce sample application. Three use cases discussed: customer operations, operational dashboard, and ad-hoc analytics.

Robin Moffatt
on Sep 07, 2018
AI, ML & Data Engineering

Democratizing Stream Processing with Apache Kafka and KSQL - Part 1

In this article, author Michael Noll discusses the stream processing with KSQL, the streaming SQL engine for Apache Kafka. Topics covered include challenges of stateful stream processing and how KSQL addresses them, and how KSQL helps to bridge the world of streams and databases through streams and tables.

Michael Noll
on Jun 15, 2018
AI, ML & Data Engineering

Traffic Data Monitoring Using IoT, Kafka and Spark Streaming

Internet of Things (IoT) is an emerging disruptive technology and becoming an increasing topic of interest. One of the areas of IoT application is the connected vehicles. In this article we'll use Apache Spark and Kafka technologies to analyse and process IoT connected vehicle's data and send the processed data to real time traffic monitoring dashboard.

Amit Baghel
on Sep 28, 2016
AI, ML & Data Engineering

Chris Fregly on the PANCAKE STACK Workshop and Data Pipelines

InfoQ Interviews Chris Fregly, organizer for the 4000+ member Advanced Spark and TensorFlow Meetup about the PANCAKE STACK workshop, Spark and building data pipelines for a machine learning pipeline

Dylan Raithel
on Aug 29, 2016