InfoQ Homepage Streaming Content on InfoQ
-
Event Streams and Workflow Engines – Kafka and Zeebe
In a recent joint meeting in Amsterdam, Kai Waehner described features of Apache Kafka, a distributed streaming platform, and how it fits in an Event-Driven Architecture. Bernd Rücker described how workflow engines can handle complex business processes, and discussed how Zeebe, a new highly scalable workflow engine, can be used with Kafka.
-
In a Serverless World, We Still Need State
Today there is a large push in enterprise architecture towards serverless architecture. Jonas Bonér argues that although he strongly believes in the serverless movement, the programming model should not focus only on stateless functions. We must also have a focus on state, allowing us to benefit from the advantages of serverless while building distributed general-purpose applications.
-
Creating Events from Databases Using Change Data Capture: Gunnar Morling at MicroXchg Berlin
When you store data in a database, you often also want to put the same data in a cache and a search engine. The challenge is how to keep all data in sync without distributed transactions and dual writes. One way is to use a change data capture (CDC) tool that captures all changes made. In a presentation at MicroXchg Berlin, Gunnar Morling described Debezium, an implementation of CDC using Kafka.
-
Swim Open Sources Platform That Challenges Conventional Wisdom in Distributed Computing
If one reviews the talks at a modern conference, it'd be reasonable to assume that many of today's software systems are made up of stateless compute, distributed databases, and high-throughput message brokers. Swim recently open-sourced their platform that uses stateful "digital twins" to analyze streaming data in real time without depending on databases, message brokers, app servers, or storage.
-
The Importance of Event-First Thinking
For global businesses to meet today’s architectural challenges with constant change and extreme scale, we need to go back to the basic principles of system design. The common element in the problems we face is the notion of events driving both actions and reactions, Neil Avery writes in a series of blog posts describing why events are so important and the advantages of an event-first approach .
-
Amazon Announces Managed Streaming for Kafka in Public Preview
At the recent AWS re:Invent 2018 event, Amazon announced a new fully managed service that makes it easy for customers to build and run applications that use Apache Kafka to process streaming data. This new service is called Amazon Managed Streaming for Kafka, Amazon MSK for short, and is now in public preview.
-
Microsoft Announces the General Availability of Azure Event Hubs for Apache Kafka
Recently, Microsoft announced the general availability of Azure Event Hubs for Apache Kafka. Customers will get the best of both worlds—the ecosystem and tools of Kafka, along with Azure’s security and global scale.
-
Event Sourcing to the Cloud at HomeAway
Adam Haines, Data Architect at HomeAway, recently spoke at the Data Architecture Summit 2018 Conference about how his team leverages event sourcing cloud design pattern to accelerate the big data initiatives in their organization.
-
William McKnight on Data Platforms and Creating a Modern Data Architecture
William McKnight gave a keynote presentation last week at Data Architecture Summit 2018 Conference on creating a modern data architecture using different data platforms.
-
Netflix Keystone Real-Time Stream Processing Platform
Netflix recently published a post in their tech blog discussing the design considerations and insights of Keystone, their Real-time stream processing platform. Keystone has been operational since December 2015 and has grown significantly over the years as Netflix subscribers have grown from 65 to over 130 million in the past 3 years. This article follows on the latest state of Keystone platform...
-
Data Artisans Announces Serializable ACID Transactions on Streaming Data
Data Artisans has announced the general availability of Streaming Ledger, which extends Apache Flink with capabilities to perform serializable ACID transactions across tables, keys, and event streams. The patent-pending technology is a proprietary add-on for Flink and allows going beyond the current standard where operations could only consistently work on a single key at a time.
-
NATS Messaging System Gets Kafka-Like Log API via Liftbridge
Joining the hot event-driven technology space is Liftbridge, an open-source project that extends the NATS messaging system with a scalable, Kafka-like log API. InfoQ spoke to creator Tyler Treat to learn more about the project, and the changing nature of data integration.
-
Axon Framework 3.3 with a Subscription Query API and Kafka Support
Version 3.3 of the Axon framework was recently released with a subscription query API for subscribing to query model updates, a manager for scheduling the publishing of deadline messages, and an Axon-Kafka module allowing for the use of Kafka to send and receive events. An updated version, 3.3.2, has also been released, and for those on version 3.3 an upgrade is strongly recommended.
-
QCon NY: Jonas Bonér on Designing Events-First Microservices
Events-first domain driven design (DDD) and event streaming are critical in developing a resilient and scalable microservices architecture. Jonas Bonér from LightBend engineering team spoke at QCon New York 2018 Conference last week about the events-first design.
-
Experiences from Building an Event-Sourced System with Kafka Streams
At the recent JEEConf conference in Kiev, Amitay Horwitz described how he and his team implemented an event-sourced invoice system, the challenges they experienced after running in production for 2 ½ years, and how they implemented a new design using Kafka Streams. The new design is still under assessment, but they do heavily use Kafka in production.