InfoQ Homepage Data Lake Content on InfoQ
Articles
RSS Feed-
/articles/ai-changes-data-interface/en/smallimage/thumbnail-1792487239242.jpg)
AI Changes the Data Interface, Not the Query Engine
This article presents a disaster-recovery analytics workflow in which an AI agent generates constrained SQL or Python, DuckDB computes reproducible facts from Parquet, and the agent interprets the results. It explains the determinism split, replayable analysis, and four signals for deciding when an embedded engine is sufficient or a managed lakehouse is justified.
-
/articles/liquid-clustering-lakehouse-layout/en/smallimage/liquid-clustering-lakehouse-layout-thumbnail-1791465984987.jpg)
When Should You Use Liquid Clustering? Benchmarking Lakehouse Layout Strategies
Physical data layout is one of the most consequential decisions in lakehouse engineering. The author compares Liquid Clustering, ZOrdering, and partitioning across four workload types on Delta Lake, uncovering two undocumented failure modes that likely affect production systems today and offering a decision framework for choosing the right layout strategy.
-
/articles/beyond-offset-lag-kafka-apache-hudi/en/smallimage/beyond-offset-lag-kafka-apache-hudi-thumbnail-1787577734265.jpg)
Beyond Offset Lag: Computing Time in Queue for Apache Hudi Data Lake Pipelines at Petabyte Scale
In this article, author Srikanth Mamidala discusses the data lake architecture used for analytics, reporting, and machine learning and shows how to manage the consumer lag metrics when using Kafka and Apache Hudi.
-
/articles/schema-proliferation-problem/en/smallimage/schema-proliferation-problem-thumbnail-1779270222602.jpg)
The Schema Proliferation Problem in Kafka and Flink Pipelines: How to Solve It
Schema proliferation builds slowly and gets expensive fast. One schema per event type feels right until there are ten tables, union queries spanning all of them, and a single field rename touching every schema. Discriminator-based schema consolidation collapses that to two tables, turning multi-table unions into a single query, while new variants are additive and don't break existing consumers.
-
/articles/lakehouse-sql-identifier-rules/en/smallimage/lakehouse-sql-identifier-rules-thumbnail-1776241856705.jpg)
Lakehouse Tower of Babel: Handling Identifier Resolution Rules across Database Engines
Lakehouse architectures enable multiple engines to operate on shared data using open table formats such as Apache Iceberg. However, differences in SQL identifier resolution and catalog naming rules create interoperability failures. This article examines these behaviors and explains why enforcing consistent naming conventions and cross-engine validation is critical.
-
/articles/reproducible-ml-iceberg/en/smallimage/reproducible-ml-iceberg-thumbnail-1753341474504.jpg)
Building Reproducible ML Systems with Apache Iceberg and SparkSQL: Open Source Foundations
Traditional data lakes are great for storing massive amounts of stuff, but they're terrible at the transactional guarantees and versioning that ML workloads desperately need. Apache Iceberg and SparkSQL bring database-like reliability to your data lake. Time travel, schema evolution, and ACID transactions help support reproducible machine learning experiments.
-
/articles/rethinking-medallion-architecture/en/smallimage/problem-multi-hop-data-logo-small-1737975004889.jpg)
The End of the Bronze Age: Rethinking the Medallion Architecture
A shift left approach to data processing relies on data products that form the basis of data communication across the business. This addresses many flaws in traditional data processing and makes data more relevant, complete, and trustworthy.
-
/articles/data-leadership-book-review-interview/en/smallimage/Data-leadership-s-1595453339836.jpg)
Data Leadership Book Review and Interview
Data Leadership book, authored by Anthony Algmin, covers the data leadership topic and how data leaders should manage and govern the data management programs in their organizations. Data Leadership is how organizations choose to apply their energy and resources toward creating data capabilities to influence their business.
-
/articles/cazena-big-data-as-a-service/en/smallimage/data-bases-logo2.jpg)
Data Lake-as-a-Service: Big Data Processing and Analytics in the Cloud
Data Lake-as-a-Service solutions provide big data processing in the cloud for faster business outcomes in a very cost effective way. InfoQ spoke with Lovan Chetty and Hannah Smalltree from Cazena team about how Data Lake as a Service works.