InfoQ Homepage Apache Flink Content on InfoQ
-
Stream All the Things: Patterns of Effective Data Stream Processing Explored by Adi Polak at QCon SF
Adi Polak, Director of Advocacy and Developer Experience Engineering at Confluent, illuminated the complexities of data streaming in her QCon San Francisco presentation. She outlined key design patterns for robust pipelines, emphasizing reliability, scalability, and data integrity.
-
Confluent Cloud for Apache Flink is Now Generally Available with AI Features
Confluent announced last month the general availability (GA) of Confluent Cloud for Apache Flink. This fully-managed service enables real-time data processing and the creation of high-quality, reusable data streams. The service is available across Amazon Web Services (AWS), Google Cloud, and Microsoft Azure.
-
Yelp Overhauls Its Streaming Architecture with Apache Beam and Apache Flink
Yelp reworked its data streaming architecture by employing Apache Beam and Apache Flink. The company replaced a fragmented set of data pipelines for streaming transactional data into its analytical systems, like Amazon Redshift and in-house data lake, using Apache data streaming projects to create a unified and flexible solution.
-
QCon London: Lessons Learned from Building LinkedIn’s AI/ML Data Platform
At the QCon London 2024 conference, Félix GV from LinkedIn discussed the AI/ML platform powering the company’s products. He specifically delved into Venice DB, the NoSQL data store used for feature persistence. The presenter shared the lessons learned from evolving and operating the platform, including cluster management and library versioning.
-
Pinterest Open-Sources a Production-Ready PubSub Java Client for Kafka, Flink, and MemQ
Pinterest open-sourced its generic PubSub client library, PSC, which has been heavily used in production for a year and a half. The library helped the engineering teams by increasing developer velocity, and the scalability and stability of services using it. Over 90% of Java applications have migrated to PSC with minimal changes.
-
Instacart Creates Real-Time Item Availability Architecture with ML and Event Processing
Instacart combined machine learning with event-based processing to create an architecture that provides customers with an indication of item availability in near real-time. The new solution helped to improve user satisfaction and retention by reducing order cancellations due to out-of-stock items. The team also created a multi-model experimentation framework to help enhance model quality.
-
DoorDash Develops New Sessionization Platform with Flink to Improve Notification Delivery Timeliness
DoorDash has significantly enhanced its user engagement by leveraging Apache Flink for real-time session detection and notification delivery. This move marks a substantial advancement in user interaction and cart conversion rates.
-
Goldsky’s Streaming-First Architecture for Blockchain Data with Flink, Redpanda and Kubernetes
Goldsky created a platform for the real-time processing of blockchain data. The platform allows clients to extract data from blockchains into their own databases to support product features, but without running the data pipeline infrastructure. The event-driven architecture (EDA) of Goldsky leverages Apache Flink, Redpanda, Kubernetes, and cloud provider services.
-
Reddit Unveils REV2: Modernised Rule-Execution with Kubernetes, Kafka, and Flink Stateful Functions
Reddit's Safety Engineering team recently published how it modernised its Rule-Execution system, which detects and acts on policy-violating content in real time. The new architecture includes improvements like transitioning from legacy EC2-based systems to Kubernetes, better rule version control with Github and S3 storage, and the capability to scale more efficiently with Flink Stateful Functions.
-
Confluent Announces Apache Flink on Confluent Cloud in Open Preview
Confluent recently announced the open preview of Apache Flink on Confluent Cloud as a fully-managed service for stream processing. The company claims that the managed service will make it easier for companies to filter, join, and enrich data streams with Flink.
-
Running Apache Flink Applications on AWS KDA: Lessons Learnt at Deliveroo
Deliveroo introduced Apache Flink into its technology stack for enriching and merging events consumed from Apache Kafka or Kinesis Streams. The company opted to use AWS Kinesis Data Analytics (KDA) service to manage Apache Flink clusters on AWS and shared its experiences from running Flink applications on KDA.
-
Instacart Creates a Self-Serve Apache Flink Platform on Kubernetes
Instacart moved their Apache Flink workloads from AWS EMR to Kubernetes to meet the high demand for data processing use cases using Flink within the organization, as using EMR became problematic for many teams with different requirements. As a result, they made the platform easier to use and reduced their operational and infrastructure costs.
-
Uber Freight Near-Real-Time Analytics Architecture
Uber Freight is the Uber platform dedicated to connecting shippers with carriers. Providing reliable service to shippers is crucial for Uber Freight. This is why the Carrier Scorecard was developed, with several metrics including on-time pickup/delivery, tracking automation, and late cancellations.
-
Apache InLong: Integration Framework for Massive Data
Apache InLong, an integration framework designed for massive data, was originally built at Tencent, where it was used in production for more than eight years, to support massive data reporting services in big data scenarios. The project officially graduated as an Apache top-level project three years after the introduction of the project in the Apache Incubator.
-
Netflix Builds a Custom High-Throughput Priority Queue Backed by Redis, Kafka and Elasticsearch
Netflix recently published how it built Timestone, a custom high-throughput, low-latency priority queueing system. They built it using open-source components such as Redis, Apache Kafka, Apache Flink and Elasticsearch. Engineers state that they made Timestone since they could not find an off-the-shelf solution that met all of its requirements.