InfoQ Homepage Streaming Content on InfoQ
-
Ahead of re:Invent, Amazon Updates AWS Lambda
A series of updates to AWS Lambda aim to improve how the function-as-a-service platform handles asynchronous workflows and processes data streams. These newly announced features arrived the week before the annual mega-conference, AWS re:Invent.
-
Jay Kreps: Events, Event Streams and Their Importance in a Digital Business
Organizations are moving more and more processes into software, Jay Kreps notes in a blog post, and adds that in an accompanying change businesses are increasingly defined in software – the core processes are specified and executed in software. To support this transition, he believes we have to move away from traditional databases into working with the concepts of events and events streams.
-
Delta – a Data Synchronization and Enrichment Platform by Netflix
Large systems often utilize numerous data stores. There is sometimes a need to keep some of these data stores in sync, and to enrich data in a store by calling external services. To address these needs, Netflix has created Delta, an eventual consistent, event-driven data synchronization and enrichment platform. In a blog post, the team behind Delta gives an overview of their design.
-
Oleg Zhurakousky on Spring Based Event-Driven Microservices
Spring Cloud Stream and Spring Cloud Function technologies support the implementation of event-driven microservices for diverse use cases, ranging from application integration to data streaming. Oleg Zhurakousky, project lead for Spring Cloud Stream and Spring Cloud Function, spoke at SpringOne Platform 2019 Conference about the recent developments and features in both of these projects.
-
Jagadish Venkatraman on LinkedIn's Journey to Samza 1.0
At the recent ApacheCon North America, Jagadish Venkatraman spoke about how LinkedIn developed Apache Samza 1.0 to handle stream processing at scale. He described LinkedIn's use cases involving trillions of events and petabytes of data, then highlighted the features added for the 1.0 release, including: stateful processing, high-level APIs, and a flexible deployment model.
-
Data Engineering in Badoo: Handling 20 Billion Events Per Day
Badoo is a dating social network that currently handles billions of events per day, explains Vladimir Kazanov, data platform engineering lead. At Skills Matter, he talked through some of the challenges of operating at this scale, and what tooling Badoo uses in order to process and report on this data.
-
Celia Kung on LinkedIn's Brooklin Data Streaming Service
Celia Kung from LinkedIn's team spoke at the QCon New York 2019 Conference last week about Brooklin data streaming service that supports pluggable sources and destinations. These can be data stores or messaging systems making the solution flexible and extensible. Brooklin is part of the streams infrastructure platform developed at LinkedIn.
-
Amazon Managed Kafka Aims to Simplify Kafka Streaming Setup and Use
Introduced as a public preview at AWS re:invent 2018, Amazon Managed Streaming for Kafka (MSK) is now generally available. Amazon MSK aims to make it easy to build and run streaming applications based on Kafka.
-
Experiences Going from Event-Driven to Event Sourcing: Fangel and Ingerslev at MicroCPH
At MicroCPH 2019, Thomas Bøgh Fangel and Emil Krog Ingerslev, both at Lunar Way, a fintech company, described how after building a monolithic Rails application they decided to migrate to an event-driven microservices architecture. During the migration they found some design issues and decided to move to event sourcing. In their presentation they discuss the problems and how they solved them.
-
Event Streams and Workflow Engines – Kafka and Zeebe
In a recent joint meeting in Amsterdam, Kai Waehner described features of Apache Kafka, a distributed streaming platform, and how it fits in an Event-Driven Architecture. Bernd Rücker described how workflow engines can handle complex business processes, and discussed how Zeebe, a new highly scalable workflow engine, can be used with Kafka.
-
In a Serverless World, We Still Need State
Today there is a large push in enterprise architecture towards serverless architecture. Jonas Bonér argues that although he strongly believes in the serverless movement, the programming model should not focus only on stateless functions. We must also have a focus on state, allowing us to benefit from the advantages of serverless while building distributed general-purpose applications.
-
Creating Events from Databases Using Change Data Capture: Gunnar Morling at MicroXchg Berlin
When you store data in a database, you often also want to put the same data in a cache and a search engine. The challenge is how to keep all data in sync without distributed transactions and dual writes. One way is to use a change data capture (CDC) tool that captures all changes made. In a presentation at MicroXchg Berlin, Gunnar Morling described Debezium, an implementation of CDC using Kafka.
-
Swim Open Sources Platform That Challenges Conventional Wisdom in Distributed Computing
If one reviews the talks at a modern conference, it'd be reasonable to assume that many of today's software systems are made up of stateless compute, distributed databases, and high-throughput message brokers. Swim recently open-sourced their platform that uses stateful "digital twins" to analyze streaming data in real time without depending on databases, message brokers, app servers, or storage.
-
The Importance of Event-First Thinking
For global businesses to meet today’s architectural challenges with constant change and extreme scale, we need to go back to the basic principles of system design. The common element in the problems we face is the notion of events driving both actions and reactions, Neil Avery writes in a series of blog posts describing why events are so important and the advantages of an event-first approach .
-
Amazon Announces Managed Streaming for Kafka in Public Preview
At the recent AWS re:Invent 2018 event, Amazon announced a new fully managed service that makes it easy for customers to build and run applications that use Apache Kafka to process streaming data. This new service is called Amazon Managed Streaming for Kafka, Amazon MSK for short, and is now in public preview.