InfoQ Homepage Apache Kafka Content on InfoQ
-
Netflix Reworks Conductor for 420 Million Monthly Workflow Executions and 10X Larger Workflows
Netflix has reworked its Conductor workflow orchestration engine to handle larger workloads, increasing supported workflow size from about 2,500 to 30,000 tasks and reducing p99 workflow evaluation latency by about 40%. Conductor 4.0 separates workflow metadata from task data, moves evaluation to asynchronous processing, and introduces dynamic worker allocation and concurrency controls.
-
Netflix Moves toward Open Source Flink Autoscaler for 30,000+ Streaming Jobs
Netflix is moving toward the open-source Apache Flink Autoscaler for more than 30,000 streaming jobs across multiple AWS regions. The operator-level approach addresses limitations of Netflix’s cluster level autoscaler for complex, stateful pipelines. Netflix reports a 58% reduction in annualized Flink compute expenditure for one team, saving approximately $1.1 million annually.
-
Spring News Roundup: First Milestone Releases for Boot, Framework, Data, Security, Modulith, Batch
After a 10-week hiatus since the last batch of Spring ecosystem releases, there was a flurry of activity during the week of August 17th, 2026, highlighting first milestone releases of: Spring Boot, Spring Framework, Spring Data, Spring Security, Spring Integration, Spring HATEOAS, Spring Modulith, Spring Batch, Spring AMQP and Spring for Apache Kafka.
-
How Netflix Scaled its Real-Time Service Map
Netflix has described how it redesigned the streaming pipeline behind Service Topology, its real-time service dependencies map, to support production scale. The system uses three stages to separate intermediary resolution from enrichment and persistence, propagates backpressure to Kafka rather than dropping records, and uses server-sent events instead of gRPC for high-volume internal transfers.
-
DoorDash Uses Envoy and Valkey for a 1.5M RPS Proxy Cache with 99.99999% Availability
DoorDash has developed Entity Cache, a transparent proxy caching platform built on Envoy and Valkey to reduce redundant service-to-service requests across its microservices architecture. Operating within DoorDash’s service mesh, the platform serves over 1.5M requests per second with 99.99999% availability through caching, event-driven invalidation, failure handling, and performance optimizations.
-
Cloud Native Infrastructure Emerges as the Foundation for Trustworthy Agentic AI
A new technical analysis published by the Cloud Native Computing Foundation (CNCF) argues that the future of agentic AI will be built not on entirely new infrastructure, but on the mature cloud-native ecosystem that already powers modern distributed applications
-
Inside Atlassian’s Forge Billing Architecture for Distributed Usage Tracking at Scale
Atlassian details the Forge billing platform built for usage-based pricing across its cloud ecosystem. It processes large-scale usage events with correct attribution, deduplication, and aggregation using a streaming pipeline, idempotent processing, and layered storage to enable accurate billing, near real-time visibility, and reliable reconciliation across distributed services.
-
Spring News Roundup: Point Releases of Boot, Security, Integration, Modulith and Spring AI 2.0
There was a flurry of activity in the Spring ecosystem during the week of June 8th, 2026, highlighting point releases of: Spring Boot, Spring Security, Spring Session, Spring Integration, Spring Modulith, Spring AMQP and Spring Vault; and GA releases of Spring AI 2.0 and Spring Data 2026.0.0.
-
Java News Roundup: WildFly, Micronaut, Spring AI, Apache Fory, GlassFish Plugin, Open Liberty
This week's Java roundup for May 18th, 2026, features news highlighting: GA releases of WildFly 40, Micronaut 5.0, Maven Embedded GlassFish Plugin 8.0 and Apache Fory 1.0; the May 2026 edition of Open Liberty; point releases of Gatherers4j, Apache and Kafka; and the seventh milestone release of Spring AI 2.0.
-
Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance
Confluent introduces a new approach in Apache Kafka that moves schema IDs from message payloads to record headers, aiming to simplify schema governance and evolution. The update integrates with Schema Registry, improves compatibility across serialization formats, and reduces coupling between data and metadata in event-driven architectures.
-
Spring News Roundup: First Release Candidates of Boot, Security, Integration, Modulith, AMQP
There was a flurry of activity in the Spring ecosystem during the week of April 20th, 2026, highlighting the first release candidates of: Spring Boot, Spring Security, Spring Integration, Spring Modulith, Spring AMQP, Spring for Apache Kafka and Spring Vault.
-
Grafana Rearchitects Loki with Kafka and Ships a CLI to Bring Observability into Coding Agent
At GrafanaCON 2026 in Barcelona, Grafana Labs announced Grafana 13 with the new Loki Kafka-backed architecture at the ingestion layer and the AI Observability in Grafana Cloud to monitor and evaluate AI systems in real time. In particular, the new CLI called GCX was announced, designed to surface Grafana Cloud data inside agentic development environments.
-
Uber Launches IngestionNext: Streaming-First Data Lake Cuts Latency and Compute by 25%
Uber launches IngestionNext, a streaming-first data lake ingestion platform that reduces data latency from hours to minutes and cuts compute usage by 25%. Built on Kafka, Flink, and Apache Hudi, it supports thousands of datasets, enabling faster analytics, experimentation, and machine learning workloads globally.
-
Spring News Roundup: Third Milestone Releases of Boot, Security, Integration, AI and AMQP
There was a flurry of activity in the Spring ecosystem during the week of March 16th, 2026, highlighting the third milestone releases of: Spring Boot, Spring Security, Spring Integration, Spring AI and Spring AMQP; along with the second milestone releases of Spring Data and Spring for Apache Kafka.
-
QCon London 2026: Introducing Tansu.io — Rethinking Kafka for Lean Operations
Peter Morgan introduced Tansu at QCon London, an open-source, Kafka-compatible, stateless, leaderless broker that scales to zero, with pluggable storage (S3, SQLite, Postgres), broker-side schema validation, and direct writes to Iceberg and Delta Lake. Written in Rust, it uses 20MB of RAM and starts in 10 milliseconds.