InfoQ Homepage Apache Flink Content on InfoQ
Articles
RSS Feed-
/articles/schema-proliferation-problem/en/smallimage/schema-proliferation-problem-thumbnail-1779270222602.jpg)
The Schema Proliferation Problem in Kafka and Flink Pipelines: How to Solve It
Schema proliferation builds slowly and gets expensive fast. One schema per event type feels right until there are ten tables, union queries spanning all of them, and a single field rename touching every schema. Discriminator-based schema consolidation collapses that to two tables, turning multi-table unions into a single query, while new variants are additive and don't break existing consumers.
-
/articles/wonders-of-postgres-logical-decoding-messages/en/smallimage/logo-1677840597750.jpg)
The Wonders of Postgres Logical Decoding Messages
In this article, author Gunnar Morling discusses Postgres database's logical decoding function to retrieve the messages from write-ahead log, process them, and relay them to external consumers, with help of use cases like outbox, audit logs and replication slots.
-
/articles/rethinking-flink-api/en/smallimage/Rethinking-Flinks-APIs-logo-1568358627523.jpg)
Rethinking Flink’s APIs for a Unified Data Processing Framework
Since its very early days, Apache Flink has followed the philosophy of taking a unified approach to batch and streaming. The core building block is the “continuous processing of unbounded data streams, with batch as a special, bounded set of those streams.” Recent updates to the Flink APIs include architectural designs by the community to support batch and streaming unification in Apache Flink.
-
/articles/beam-perry/en/smallimage/apachebeam-small.jpg)
Apache Beam Interview with Frances Perry
InfoQ Interviews Apache Beam's Frances Perry about the impetus for using Beam and the future of the top-level open source project and covers the thoughts behind the programming model as well as some of the touch-points in integration with other data engineering tools like Apache Spark and Flink.