InfoQ Homepage Event Stream Processing Content on InfoQ

Articles

RSS Feed

Newer Older

AI, ML & Data Engineering

Beyond the Database, and beyond the Stream Processor: What's the Next Step for Data Management?

Databases have been around forever with the same shape: you make a request to your data and then you receive an answer. Now, stream processors came along with a different approach: data isn’t locked up, it is in motion. Understand how stream processors and databases relate and why there is an emerging new category of databases that focus on data that stays in place as well as data that moves.

Ben Stopford
on Nov 16, 2020 1
Architecture & Design

Real Time APIs in the Context of Apache Kafka

Events offer a Goldilocks-style approach in which real-time APIs can be used as the foundation for applications which is flexible yet performant; loosely-coupled yet efficient. Apache Kafka offers a scalable event streaming platform with which you can build applications around the powerful concept of events.

Robin Moffatt
on Oct 16, 2020
Architecture & Design

The Challenges of Building a Reliable Real-Time Event-Driven Ecosystem

Globally, there is an increasing appetite for data delivered in real time; we are witnessing the emergence of the real time API. When it comes to event-driven APIs engineers can choose between multiple different protocols. In addition to choosing a protocol, engineers also have to think about subscription models, too: server-initiated (push-based) or client-initiated (pull-based).

Matthew O'Riordan
on Jul 30, 2020
AI, ML & Data Engineering

Applied Probability - Counting Large Set of Unstructured Events with Theta Sketches

In this article, author Ronen Cohen discusses the solution to processing the event data using Theta Sketches and technologies like HBase and Kafka.

Ronen Cohen
on Jun 29, 2020
Cloud

Is Edge Computing a Thing?

Edge Computing is definitely a thing, but the computing need not occur at the edge. Instead what is needed is an ability to compute (anywhere) on streaming data from large numbers of dynamically changing devices, in the edge environment. This in turn demands an architectural pattern for stateful, distributed computing.

Simon Crosby
on Apr 17, 2020
AI, ML & Data Engineering

The Kongo Problem: Building a Scalable IoT Application with Apache Kafka

In this article, author Paul Brebner discusses the best practices for developing IoT projects using Apache Kafka and Kafka Streams technologies and how to maximize Kafka scalability.

Paul Brebner
on Feb 15, 2020
AI, ML & Data Engineering

Rethinking Flink’s APIs for a Unified Data Processing Framework

Since its very early days, Apache Flink has followed the philosophy of taking a unified approach to batch and streaming. The core building block is the “continuous processing of unbounded data streams, with batch as a special, bounded set of those streams.” Recent updates to the Flink APIs include architectural designs by the community to support batch and streaming unification in Apache Flink.

Aljoscha Krettek
on Sep 16, 2019
Architecture & Design

Increasing the Quality of Patient Care through Stream Processing

Today’s healthcare technology landscape is disaggregated and siloed. Physicians analyse patient data streams from different systems without much correlation. Even though health-tech domain is mature and rich with data, the value of it is not directed towards increasing the quality of patient care. This article presents a stream processing solution in which streams are co-related.

Himasha Guruge Nuwan Bandara Nadeeshan Gimhana
on Jan 17, 2019
AI, ML & Data Engineering

Scaling a Distributed Stream Processor in a Containerized Environment

The article presents our experience of scaling a distributed stream processor in Kubernetes. The stream processor should provide support for maintaining the optimal level of parallelism. However, adding more resources incurs additional cost and also it does not guarantee performance improvements. Instead, the stream processor should identify the level of resource requirement and scale accordingly.

Sarangan Janakan Miyuru Dayarathna
on Jan 13, 2019
AI, ML & Data Engineering

Apache Kafka: Ten Best Practices to Optimize Your Deployment

Author Ben Bromhead discusses the latest Kafka best practices for developers to manage the data streaming platform more effectively. Best practices include log configuration, proper hardware usage, Zookeeper configuration, replication factor, and partition count.

Ben Bromhead
on Oct 19, 2018 3
AI, ML & Data Engineering

Democratizing Stream Processing with Apache Kafka® and KSQL - Part 2

In this article, author Robin Moffatt shows how to use Apache Kafka and KSQL to build data integration and processing applications with the help of an e-commerce sample application. Three use cases discussed: customer operations, operational dashboard, and ad-hoc analytics.

Robin Moffatt
on Sep 07, 2018
Architecture & Design

A Critique of Resizable Hash Tables: Riak Core & Random Slicing

This fall, Wallaroo Labs will be releasing a large new feature set to our distributed data stream processing framework, Wallaroo. One of the new features requires a size-adjustable, distributed data structure to support growing & shrinking of compute clusters. It might be a good idea to use a distributed hash table to support the new feature, but what distributed hash algorithm should we choose?

Scott Lystig Fritchie
on Aug 26, 2018