InfoQ Homepage Streaming Content on InfoQ
-
/articles/storage-cloud-apache-bookkeeper/en/smallimage/logo-1619468572224.jpg)
Indestructible Storage in the Cloud with Apache Bookkeeper
At Salesforce, we required a storage system that could work with two kinds of streams, one stream for write-ahead logs and one for data. But we have competing requirements from both of the streams. Being the pioneers in cloud computing, we also required our storage system to be cloud-aware as the requirements of availability and durability are ever more increasing.
-
/articles/future-data-engineering-riccomini/en/smallimage/the-future-of-data-engineering-logo-small-1613120173588.jpg)
The Future of Data Engineering
Chris Riccomini examines the current and future states of the art in data pipelines, data streaming, and data warehousing. He presents a six-stage evolution that data ecosystems follow, from a simple monolith to a complex data-microwarehouse architecture as the data engineers who manage them solve problems and clarify their roles as infrastructure engineers, rather than data stewards.
-
/articles/pulsar-customer-engagement-platform/en/smallimage/pulsar-customer-engagement-platform-s-1604947819956.jpeg)
How Apache Pulsar is Helping Iterable Scale its Customer Engagement Platform
In this article, author Greg Methvin discusses his experience implementing a distributed messaging platform based on Apache Pulsar.
-
/articles/whats-the-next-step-for-data-management/en/smallimage/database-1605276684430.jpeg)
Beyond the Database, and beyond the Stream Processor: What's the Next Step for Data Management?
Databases have been around forever with the same shape: you make a request to your data and then you receive an answer. Now, stream processors came along with a different approach: data isn’t locked up, it is in motion. Understand how stream processors and databases relate and why there is an emerging new category of databases that focus on data that stays in place as well as data that moves.
-
/articles/real-time-api-kafka/en/smallimage/real-time-api-kafka-s-1602600487985.jpeg)
Real Time APIs in the Context of Apache Kafka
Events offer a Goldilocks-style approach in which real-time APIs can be used as the foundation for applications which is flexible yet performant; loosely-coupled yet efficient. Apache Kafka offers a scalable event streaming platform with which you can build applications around the powerful concept of events.
-
/articles/realtime-event-driven-ecosystem/en/smallimage/realtime-api-logo-1595963690562.jpg)
The Challenges of Building a Reliable Real-Time Event-Driven Ecosystem
Globally, there is an increasing appetite for data delivered in real time; we are witnessing the emergence of the real time API. When it comes to event-driven APIs engineers can choose between multiple different protocols. In addition to choosing a protocol, engineers also have to think about subscription models, too: server-initiated (push-based) or client-initiated (pull-based).
-
/articles/applied-probability-unstructured-events-theta-sketches/en/smallimage/applied-probability-unstructured-events-theta-sketches-s-1593199758266.jpg)
Applied Probability - Counting Large Set of Unstructured Events with Theta Sketches
In this article, author Ronen Cohen discusses the solution to processing the event data using Theta Sketches and technologies like HBase and Kafka.
-
/articles/edge-computing-streaming-data/en/smallimage/edge-computing-streaming-data-s-1586865838285.jpg)
Is Edge Computing a Thing?
Edge Computing is definitely a thing, but the computing need not occur at the edge. Instead what is needed is an ability to compute (anywhere) on streaming data from large numbers of dynamically changing devices, in the edge environment. This in turn demands an architectural pattern for stateful, distributed computing.
-
/articles/building-scalable-iot-application-with-apache-kafka/en/smallimage/The-Kongo-Problem-Building-a-Scalable-IoT-Application-with-Apache-Kafka-logo-1581595522263.jpg)
The Kongo Problem: Building a Scalable IoT Application with Apache Kafka
In this article, author Paul Brebner discusses the best practices for developing IoT projects using Apache Kafka and Kafka Streams technologies and how to maximize Kafka scalability.
-
/articles/rethinking-flink-api/en/smallimage/Rethinking-Flinks-APIs-logo-1568358627523.jpg)
Rethinking Flink’s APIs for a Unified Data Processing Framework
Since its very early days, Apache Flink has followed the philosophy of taking a unified approach to batch and streaming. The core building block is the “continuous processing of unbounded data streams, with batch as a special, bounded set of those streams.” Recent updates to the Flink APIs include architectural designs by the community to support batch and streaming unification in Apache Flink.
-
/articles/azure-data-lake-analytics-usql/en/smallimage/azure-data-logo-1568103168665.jpg)
Azure Data Lake Analytics and U-SQL
In this article, the author shows how to use big data query and processing language U-SQL on Azure Data Lake Analytics platform. U-SQL combines the concepts and constructs both of SQL and C#. It combines the simplicity and declarative nature of SQL with the programmatic power of C# including rich types and expressions.
-
/articles/anamoly-detection-yurita/en/smallimage/logo-1562313713564.jpg)
Stream Processing Anomaly Detection Using Yurita Framework
In this article, author Guy Gerson discusses the stream processing anomaly detection framework they developed by PayPal, called Yurita. The framework is based on Spark Structured Streaming.
CONTENT IN THIS BOX PROVIDED BY OUR SPONSOR
Featured Content
[On Demand Webinar] Get the Most Out of Kubernetes with NGINX
Learn about the latest Kubernetes trends and how NGINX solves common challenges to securing applications in Kubernetes. Watch now.
Connect, Secure, and Scale Microservices
Learn how Buzzfeed built a microservices request router using NGINX Plus. Try NGINX Plus for Free.
Microservices Up and Running
Step-by-step guidance for building an effective microservices architecture, complements of O’Reilly Media and NGINX. Download now.
Web Application Security
Available compliments of NGINX, this O’Reilly Media eBook features practical security tips and advice that your development and security teams can use right away. Download now.