InfoQ Homepage Data Lake Content on InfoQ
-
Harper Argues Against the Multi-System Stack and Releases 5.2
The database platform Harper advocates for a single-runtime architecture that keeps application code and data together, with its benchmark against a Vercel-based stack reporting significantly better performance on live, personalized-data workloads. Harper recently released version 5.2, with a new record cache and more throughput per node.
-
Spotify Builds External Index to Enable Low Latency Point Queries on its Data Lake
Spotify introduced external indexing architecture for Apache Parquet data lakes that enables low-latency point queries without replicating datasets into operational databases. The approach maps lookup keys to Parquet files and row locations, allowing targeted reads from cloud object storage while supporting analytics, machine learning, AI applications, and online services from the same datasets.
-
Version Controlled SQL Database Dolt Releases 2.0 with Automatic Storage Cleanup and Compression
DoltHub has recently released Dolt 2.0, a major update to the open source version-controlled SQL database. The latest major version adds automatic storage optimization, including garbage collection and compression, along with improved support for large and vector data types.
-
Cloudflare Details Unified Data Platform Where Billing Workloads Account for 53% of Queries
Cloudflare details Town Lake, an internal unified data platform, and Skipper, an AI analytics agent unifying access to operational, billing, security, and business data. The platform processed ~91K billing queries, with billing forming majority usage. Built on a lakehouse architecture using Trino, Iceberg, R2, and DataHub, it enables governed cross-system analytics and natural language access.
-
Anthropic Reports Claude Now Handles 95% of Internal Analytics Queries
Anthropic recently reported that Claude now handles around 95% of its internal analytics requests, letting employees query business data independently instead of relying on data teams. The company attributes this result less to advances in models and more to data governance, semantic definitions, and operational discipline.
-
Pinecone Brings AI Agents Directly to Enterprise Data with Microsoft OneLake Integration
Pinecone has announced a new integration between its Nexus knowledge engine and Microsoft OneLake, aiming to fundamentally change how enterprise AI agents access and reason over corporate data.
-
Google Cloud Introduces Cross-Engine Iceberg Support in BigQuery
At the Apache Iceberg Summit last month, Google announced new interoperability features for Apache Iceberg in BigQuery. The preview of the serverless Iceberg REST catalog lets teams create, update, and query the same Apache Iceberg tables in BigQuery and in engines like Spark, Flink, and Trino without duplicating data.
-
DuckLake 1.0: Data Lake Format with SQL Catalog Metadata
DuckDB Labs recently released DuckLake 1.0, a data lake format that stores table metadata in a SQL database rather than across many files in object storage. The first implementation is available as a DuckDB extension and includes catalog-stored small updates, improved sorting and partitioning options, and compatibility with Iceberg-style data features.
-
Uber Launches IngestionNext: Streaming-First Data Lake Cuts Latency and Compute by 25%
Uber launches IngestionNext, a streaming-first data lake ingestion platform that reduces data latency from hours to minutes and cuts compute usage by 25%. Built on Kafka, Flink, and Apache Hudi, it supports thousands of datasets, enabling faster analytics, experimentation, and machine learning workloads globally.
-
Hybrid Cloud Data at Uber: How Engineers Solved Extreme-Scale Replication Challenges
Uber’s HiveSync team optimized Hadoop Distcp to handle multi-petabyte replication across hybrid cloud and on-premise data lakes. Enhancements include task parallelization, Uber jobs for small transfers, and improved observability, enabling 5x replication capacity and seamless on-premise-to-cloud migration.
-
Databricks Introduces Lakebase, a PostgreSQL Database for AI Workloads
Databricks has recently announced the general availability of Lakebase, a serverless, PostgreSQL-based OLTP database that scales compute and storage independently. Lakebase is designed to integrate with the Databricks platform, providing a hybrid solution that combines both transactional and analytical capabilities.
-
350PB, Millions of Events, One System: inside Uber’s Cross-Region Data Lake and Disaster Recovery
Uber’s HiveSync is a sharded, cross-region batch replication system keeping Hive/HDFS data consistent across multiple regions. Handling 5M daily Hive events and 8PB of data replication, it uses event-driven jobs, hybrid RPC and DistCp strategies, DAG-based orchestration, and dynamic sharding, enabling disaster recovery, horizontal scaling, and 99.99% cross-region data accuracy.
-
Cloudflare Introduces Data Platform with Zero Egress Fees
Cloudflare has recently announced the open beta of Cloudflare Data Platform, a managed solution for ingesting, storing, and querying analytical data tables using open standards such as Apache Iceberg.
-
How Netflix is Reimagining Data Engineering for Video, Audio, and Text
Netflix has introduced a new engineering specialization—Media ML Data Engineering, alongside a Media Data Lake designed to handle video, audio, text, and image assets at scale. Early results include richer ML models trained on standardized media, faster evaluation cycles, and deeper insights into creative workflows.
-
Apache Hudi 1.0 Now Generally Available
The Apache Software Foundation has recently announced the general availability of Apache Hudi 1.0, the transactional data lake platform with support for near real-time analytics. Initially introduced in 2017, Apache Hudi provides an open table format optimized for efficient writes in incremental data pipelines and fast query performance.