InfoQ Homepage Streaming Content on InfoQ
-
Jagadish Venkatraman on LinkedIn's Journey to Samza 1.0
At the recent ApacheCon North America, Jagadish Venkatraman spoke about how LinkedIn developed Apache Samza 1.0 to handle stream processing at scale. He described LinkedIn's use cases involving trillions of events and petabytes of data, then highlighted the features added for the 1.0 release, including: stateful processing, high-level APIs, and a flexible deployment model.
-
Data Engineering in Badoo: Handling 20 Billion Events Per Day
Badoo is a dating social network that currently handles billions of events per day, explains Vladimir Kazanov, data platform engineering lead. At Skills Matter, he talked through some of the challenges of operating at this scale, and what tooling Badoo uses in order to process and report on this data.
-
Celia Kung on LinkedIn's Brooklin Data Streaming Service
Celia Kung from LinkedIn's team spoke at the QCon New York 2019 Conference last week about Brooklin data streaming service that supports pluggable sources and destinations. These can be data stores or messaging systems making the solution flexible and extensible. Brooklin is part of the streams infrastructure platform developed at LinkedIn.
-
Amazon Managed Kafka Aims to Simplify Kafka Streaming Setup and Use
Introduced as a public preview at AWS re:invent 2018, Amazon Managed Streaming for Kafka (MSK) is now generally available. Amazon MSK aims to make it easy to build and run streaming applications based on Kafka.
-
Experiences Going from Event-Driven to Event Sourcing: Fangel and Ingerslev at MicroCPH
At MicroCPH 2019, Thomas Bøgh Fangel and Emil Krog Ingerslev, both at Lunar Way, a fintech company, described how after building a monolithic Rails application they decided to migrate to an event-driven microservices architecture. During the migration they found some design issues and decided to move to event sourcing. In their presentation they discuss the problems and how they solved them.
-
Event Streams and Workflow Engines – Kafka and Zeebe
In a recent joint meeting in Amsterdam, Kai Waehner described features of Apache Kafka, a distributed streaming platform, and how it fits in an Event-Driven Architecture. Bernd Rücker described how workflow engines can handle complex business processes, and discussed how Zeebe, a new highly scalable workflow engine, can be used with Kafka.
-
In a Serverless World, We Still Need State
Today there is a large push in enterprise architecture towards serverless architecture. Jonas Bonér argues that although he strongly believes in the serverless movement, the programming model should not focus only on stateless functions. We must also have a focus on state, allowing us to benefit from the advantages of serverless while building distributed general-purpose applications.
-
Creating Events from Databases Using Change Data Capture: Gunnar Morling at MicroXchg Berlin
When you store data in a database, you often also want to put the same data in a cache and a search engine. The challenge is how to keep all data in sync without distributed transactions and dual writes. One way is to use a change data capture (CDC) tool that captures all changes made. In a presentation at MicroXchg Berlin, Gunnar Morling described Debezium, an implementation of CDC using Kafka.
-
Swim Open Sources Platform That Challenges Conventional Wisdom in Distributed Computing
If one reviews the talks at a modern conference, it'd be reasonable to assume that many of today's software systems are made up of stateless compute, distributed databases, and high-throughput message brokers. Swim recently open-sourced their platform that uses stateful "digital twins" to analyze streaming data in real time without depending on databases, message brokers, app servers, or storage.
-
The Importance of Event-First Thinking
For global businesses to meet today’s architectural challenges with constant change and extreme scale, we need to go back to the basic principles of system design. The common element in the problems we face is the notion of events driving both actions and reactions, Neil Avery writes in a series of blog posts describing why events are so important and the advantages of an event-first approach .
-
Amazon Announces Managed Streaming for Kafka in Public Preview
At the recent AWS re:Invent 2018 event, Amazon announced a new fully managed service that makes it easy for customers to build and run applications that use Apache Kafka to process streaming data. This new service is called Amazon Managed Streaming for Kafka, Amazon MSK for short, and is now in public preview.
-
Microsoft Announces the General Availability of Azure Event Hubs for Apache Kafka
Recently, Microsoft announced the general availability of Azure Event Hubs for Apache Kafka. Customers will get the best of both worlds—the ecosystem and tools of Kafka, along with Azure’s security and global scale.
-
Event Sourcing to the Cloud at HomeAway
Adam Haines, Data Architect at HomeAway, recently spoke at the Data Architecture Summit 2018 Conference about how his team leverages event sourcing cloud design pattern to accelerate the big data initiatives in their organization.
-
William McKnight on Data Platforms and Creating a Modern Data Architecture
William McKnight gave a keynote presentation last week at Data Architecture Summit 2018 Conference on creating a modern data architecture using different data platforms.
-
Netflix Keystone Real-Time Stream Processing Platform
Netflix recently published a post in their tech blog discussing the design considerations and insights of Keystone, their Real-time stream processing platform. Keystone has been operational since December 2015 and has grown significantly over the years as Netflix subscribers have grown from 65 to over 130 million in the past 3 years. This article follows on the latest state of Keystone platform...