InfoQ Homepage Apache Iceberg Content on InfoQ
-
Cloudflare Makes Basin Serverless Analytics Generally Available
Cloudflare has announced the general availability of Basin, a serverless analytics platform built on Apache Iceberg and R2 Object Storage. The managed service supports open data formats and integrates with tools such as DuckDB, Snowflake, and Apache Spark, with no data egress fees.
-
Pinterest Uses Event Time Watermarks to Finalize Petabyte Scale Data Partitions
Pinterest has introduced event time-based partition finalization for its database ingestion framework. The mechanism uses Apache Flink to track event time and Apache Iceberg metadata to publish a non-regressing watermark, helping downstream jobs determine when continuously ingested CDC data is sufficiently complete for processing.
-
Amazon Acquires DuckLabs and Integrates DuckDB in Amazon Aurora PostgreSQL
Amazon has acquired DuckLabs, the company behind the analytical database DuckDB. Co-founders Hannes Mühleisen and Mark Raasveldt will continue to lead the team under AWS. The acquisition aims to enhance the development and reach of DuckDB while maintaining its open-source status. Amazon Aurora PostgreSQL gets its first DuckDB-powered feature following the acquisition.
-
Netflix Reworks Conductor for 420 Million Monthly Workflow Executions and 10X Larger Workflows
Netflix has reworked its Conductor workflow orchestration engine to handle larger workloads, increasing supported workflow size from about 2,500 to 30,000 tasks and reducing p99 workflow evaluation latency by about 40%. Conductor 4.0 separates workflow metadata from task data, moves evaluation to asynchronous processing, and introduces dynamic worker allocation and concurrency controls.
-
Spotify Builds External Index to Enable Low Latency Point Queries on its Data Lake
Spotify introduced external indexing architecture for Apache Parquet data lakes that enables low-latency point queries without replicating datasets into operational databases. The approach maps lookup keys to Parquet files and row locations, allowing targeted reads from cloud object storage while supporting analytics, machine learning, AI applications, and online services from the same datasets.
-
AWS Introduces Amazon S3 Annotations
AWS recently announced Amazon S3 Annotations, a feature that lets teams attach rich, searchable context such as summaries, classifications, compliance data, or AI-generated insights directly to S3 objects. Annotations can be updated independently of the object and queried across datasets, reducing the need for separate metadata systems.
-
Cloudflare Details Unified Data Platform Where Billing Workloads Account for 53% of Queries
Cloudflare details Town Lake, an internal unified data platform, and Skipper, an AI analytics agent unifying access to operational, billing, security, and business data. The platform processed ~91K billing queries, with billing forming majority usage. Built on a lakehouse architecture using Trino, Iceberg, R2, and DataHub, it enables governed cross-system analytics and natural language access.
-
Google Cloud Introduces Cross-Engine Iceberg Support in BigQuery
At the Apache Iceberg Summit last month, Google announced new interoperability features for Apache Iceberg in BigQuery. The preview of the serverless Iceberg REST catalog lets teams create, update, and query the same Apache Iceberg tables in BigQuery and in engines like Spark, Flink, and Trino without duplicating data.
-
DuckLake 1.0: Data Lake Format with SQL Catalog Metadata
DuckDB Labs recently released DuckLake 1.0, a data lake format that stores table metadata in a SQL database rather than across many files in object storage. The first implementation is available as a DuckDB extension and includes catalog-stored small updates, improved sorting and partitioning options, and compatibility with Iceberg-style data features.
-
Pinterest’s CDC-Powered Ingestion Slashes Database Latency from 24 Hours to 15 Minutes
Pinterest launched a next-generation CDC-based database ingestion framework using Kafka, Flink, Spark, and Iceberg. The system reduces data availability latency from 24+ hours to 15 minutes, processes only changed records, supports incremental updates and deletions, and scales to petabyte-level data across thousands of pipelines, optimizing cost and efficiency.
-
Etleap Launches Iceberg Pipeline Platform to Simplify Enterprise Adoption of Apache Iceberg
Etleap has recently launched the Iceberg pipeline platform, a new managed data pipeline layer designed to let enterprises adopt Apache Iceberg without building or maintaining a complex custom stack.
-
AWS CloudWatch Evolves into Unified Observability Platform with Apache Iceberg Support
AWS has expanded Amazon CloudWatch to unify log management across operational and security use cases. By integrating native OCSF normalization and Apache Iceberg-compatible storage via S3 Tables, the update aims to eliminate data silos and enable Zero-ETL analytics across multiple AWS accounts and regions.
-
AWS Adds Intelligent-Tiering and Replication for S3 Tables
AWS has introduced Intelligent-Tiering and cross-region replication for S3 Tables to automate cost optimization and data availability for Apache Iceberg workloads. These features allow data to transition to lower-cost storage tiers based on access patterns while maintaining consistent, read-only table replicas across regions and accounts without manual synchronization.
-
DuckDB's WebAssembly Client Allows Querying Iceberg Datasets in the Browser
DuckDB has recently introduced end-to-end interaction with Iceberg REST Catalogs directly within a browser tab, requiring no infrastructure setup. The new feature leverages DuckDB-Wasm, a WebAssembly port of DuckDB that runs in the browser, allowing users to query, read, and write Iceberg tables in a serverless manner.
-
Breaking Silos: Netflix Introduces Upper Metamodel to Bring Consistency across Content Engineering
Netflix has introduced the Upper metamodel within its Unified Data Architecture (UDA) to standardize domain definitions and generate consistent data container representations. UDA links conceptual models to GraphQL, Avro, SQL, and Java artifacts, supporting projections, mappings, and knowledge graph-based discovery across content, advertising, and operational systems.