InfoQ Homepage Streaming Content on InfoQ
-
Experience Running Spotify’s Event Delivery System in the Cloud
Event delivery is a key component at Spotify; the events contain important data about users, actions they take, and operational logs. After running the event delivery system in the cloud for 2 ½ years, Bartosz Janota and Robert Stephenson have written a blog post discussing what they have achieved and how they have been able to evolve and simplify the system by moving up the stack in the cloud.
-
Pull Queries and Connector Management Added to ksqlDB (KSQL) Event Streaming Database for Kafka
The new release of KSQL, an event streaming database for Kafka, includes pull queries to allow for data to be read at a specific point in time using a SQL syntax, and connector management that enables direct control and execution of connectors built to work with Kafka Connect. The Confluent team behind KSQL believes it's a significant release, and has decided to rename the tool to ksqlDB.
-
Recap of AWS re:Invent 2019
Last week in Las Vegas, AWS held their annual re:Invent conference and unveiled a slew of new products, while updating many existing ones. Here's a review of announcements impacting compute, data and storage, app integration, networking, machine learning, identity management, enterprise services, and development.
-
Ahead of re:Invent, Amazon Updates AWS Lambda
A series of updates to AWS Lambda aim to improve how the function-as-a-service platform handles asynchronous workflows and processes data streams. These newly announced features arrived the week before the annual mega-conference, AWS re:Invent.
-
Jay Kreps: Events, Event Streams and Their Importance in a Digital Business
Organizations are moving more and more processes into software, Jay Kreps notes in a blog post, and adds that in an accompanying change businesses are increasingly defined in software – the core processes are specified and executed in software. To support this transition, he believes we have to move away from traditional databases into working with the concepts of events and events streams.
-
Delta – a Data Synchronization and Enrichment Platform by Netflix
Large systems often utilize numerous data stores. There is sometimes a need to keep some of these data stores in sync, and to enrich data in a store by calling external services. To address these needs, Netflix has created Delta, an eventual consistent, event-driven data synchronization and enrichment platform. In a blog post, the team behind Delta gives an overview of their design.
-
Oleg Zhurakousky on Spring Based Event-Driven Microservices
Spring Cloud Stream and Spring Cloud Function technologies support the implementation of event-driven microservices for diverse use cases, ranging from application integration to data streaming. Oleg Zhurakousky, project lead for Spring Cloud Stream and Spring Cloud Function, spoke at SpringOne Platform 2019 Conference about the recent developments and features in both of these projects.
-
Jagadish Venkatraman on LinkedIn's Journey to Samza 1.0
At the recent ApacheCon North America, Jagadish Venkatraman spoke about how LinkedIn developed Apache Samza 1.0 to handle stream processing at scale. He described LinkedIn's use cases involving trillions of events and petabytes of data, then highlighted the features added for the 1.0 release, including: stateful processing, high-level APIs, and a flexible deployment model.
-
Data Engineering in Badoo: Handling 20 Billion Events Per Day
Badoo is a dating social network that currently handles billions of events per day, explains Vladimir Kazanov, data platform engineering lead. At Skills Matter, he talked through some of the challenges of operating at this scale, and what tooling Badoo uses in order to process and report on this data.
-
Celia Kung on LinkedIn's Brooklin Data Streaming Service
Celia Kung from LinkedIn's team spoke at the QCon New York 2019 Conference last week about Brooklin data streaming service that supports pluggable sources and destinations. These can be data stores or messaging systems making the solution flexible and extensible. Brooklin is part of the streams infrastructure platform developed at LinkedIn.
-
Amazon Managed Kafka Aims to Simplify Kafka Streaming Setup and Use
Introduced as a public preview at AWS re:invent 2018, Amazon Managed Streaming for Kafka (MSK) is now generally available. Amazon MSK aims to make it easy to build and run streaming applications based on Kafka.
-
Experiences Going from Event-Driven to Event Sourcing: Fangel and Ingerslev at MicroCPH
At MicroCPH 2019, Thomas Bøgh Fangel and Emil Krog Ingerslev, both at Lunar Way, a fintech company, described how after building a monolithic Rails application they decided to migrate to an event-driven microservices architecture. During the migration they found some design issues and decided to move to event sourcing. In their presentation they discuss the problems and how they solved them.
-
Event Streams and Workflow Engines – Kafka and Zeebe
In a recent joint meeting in Amsterdam, Kai Waehner described features of Apache Kafka, a distributed streaming platform, and how it fits in an Event-Driven Architecture. Bernd Rücker described how workflow engines can handle complex business processes, and discussed how Zeebe, a new highly scalable workflow engine, can be used with Kafka.
-
In a Serverless World, We Still Need State
Today there is a large push in enterprise architecture towards serverless architecture. Jonas Bonér argues that although he strongly believes in the serverless movement, the programming model should not focus only on stateless functions. We must also have a focus on state, allowing us to benefit from the advantages of serverless while building distributed general-purpose applications.
-
Creating Events from Databases Using Change Data Capture: Gunnar Morling at MicroXchg Berlin
When you store data in a database, you often also want to put the same data in a cache and a search engine. The challenge is how to keep all data in sync without distributed transactions and dual writes. One way is to use a change data capture (CDC) tool that captures all changes made. In a presentation at MicroXchg Berlin, Gunnar Morling described Debezium, an implementation of CDC using Kafka.