InfoQ Homepage AI, ML & Data Engineering Content on InfoQ
-
/articles/liquid-clustering-lakehouse-layout/en/smallimage/liquid-clustering-lakehouse-layout-thumbnail-1791465984987.jpg)
When Should You Use Liquid Clustering? Benchmarking Lakehouse Layout Strategies
Physical data layout is one of the most consequential decisions in lakehouse engineering. The author compares Liquid Clustering, ZOrdering, and partitioning across four workload types on Delta Lake, uncovering two undocumented failure modes that likely affect production systems today and offering a decision framework for choosing the right layout strategy.
-
/articles/apache-kafka-golang-session-ordered-pipeline/en/smallimage/thumbnail-1790945957360.jpg)
Building a Session-Ordered Kafka Pipeline in Go
The article describes a custom implementation that provides a session-level ordering on top of Apache Kafka partitions, supporting strict message ordering across 1000s of independent channels. The solution required application-level routing, consistent hashing, retries, and contiguous watermark commits. Engineers conducted operational hardening supported by extensive performance testing.
-
/articles/platform-engineering-playbook-production-llms/en/smallimage/thumbnail-1790763235683.jpg)
The Platform Engineering Playbook for Production LLMs
In this article, author discusses his experience with AI agent hallucinations in an inventory recommendation system and how this problem was solved by treating the LLM stack as a platform infrastructure concern instead of as an application one. He makes a case for a shared LLM platform with common services like prompt registry & versioning, schema enforcement and token cost attribution by request.
-
/articles/agent-harness-build-one/en/smallimage/The-Agent-Harness-What-It-Is-and-Two-Ways-to-Build-One-thumb-1789997776393.jpg)
The Agent Harness: What it is and Two Ways to Build One
This article explains the development and operational layers of an agent harness through two implementations of a finance assistant: AWS AgentCore Harness and LangChain with Envoy AI Gateway. It compares how each handles tools, memory, model access, cost control, and observability, and examines the trade-offs in operational ownership, portability, and engineering effort.
-
/articles/secure-scalable-facial-verification/en/smallimage/secure-scalable-facial-verification-thumbnail-1789468796366.jpg)
Architecting Secure and Scalable Facial Verification Systems
When three thousand employees verify at once, synchronous API calls collapse. This article presents a four-layer architecture for high-volume face verification: client-side filtering that cut cloud costs 30%, decoupled detection and verification enabling 10x scaling, risk-based dynamic thresholds, and zero-trust privacy with consent gates and automated data purging for GDPR and HIPAA.
-
/articles/next-dsl-author-language-model/en/smallimage/Your-Next-DSL-Author-Is-a-Language-Model-thumb-1789048983233.jpg)
Your Next DSL Author is a Language Model
In this article, the author introduces Typed Domain Grounding, an approach to reducing LLM hallucinations in domain-specific languages by embedding them in mainstream typed languages. Using kUML benchmarks and an infrastructure-as-code example, he explores how compiler validation and generate-compile-repair loops can make model-generated DSL output more reliable.
-
/articles/durable-workflows-postgres/en/smallimage/Implementing-Durable-Workflows-on-Postgres-Without-an-External-Orchestrator-thumb-1789129413946.jpg)
Implementing Durable Workflows on Postgres without an External Orchestrator
Postgres can serve as the durable state store and coordination layer for workflows, eliminating the need for an external orchestrator. SKIP LOCKED enables concurrent work processing, primary-key checkpoints enforce idempotency, and leases support crash recovery. Workflow sleeps and human approvals can also be persisted as database state and survive restarts.
-
/articles/when-spec-driven-development-pays-off/en/smallimage/When-Spec-Driven-Development-Pays-Off-thumb-1788790605567.jpg)
When Spec-Driven Development Pays off
AI coding assistants have become a core part of software development. AI-generated code has shown productivity gains, but it's also contributing to security weaknesses and familiar bug patterns. In this article, author Nitin Garg highlights the bottleneck has moved from code generation to code verification, and how to detect & mitigate it when the AI-generated behavior diverges from the intent.
-
/articles/beyond-offset-lag-kafka-apache-hudi/en/smallimage/beyond-offset-lag-kafka-apache-hudi-thumbnail-1787577734265.jpg)
Beyond Offset Lag: Computing Time in Queue for Apache Hudi Data Lake Pipelines at Petabyte Scale
In this article, author Srikanth Mamidala discusses the data lake architecture used for analytics, reporting, and machine learning and shows how to manage the consumer lag metrics when using Kafka and Apache Hudi.
-
/articles/agentic-fitness-functions-evolutionary-architecture/en/smallimage/thumbnail-1786428822427.jpg)
Agentic Fitness Functions: Extending Evolutionary Architecture Beyond Deterministic Rules
Deterministic rules safeguard hard metrics, but what about architectural intent? Discover how agentic fitness functions combine AI agents and versioned rubrics to evaluate complex, judgment-heavy concerns—such as boundary fidelity, semantic contract drift, and stale ADR assumptions. Elevate evolutionary architecture governance with continuous, calibrated feedback loops.
-
/articles/ai-workflow-pattern/en/smallimage/thumbnail-1785766770921.jpg)
Runtime-Agnostic AI Workflows: a Pattern for Production Durability and Fast Eval Iteration
AI workflows have two needs that trade off directly. Running reliably in production requires persisting and distributing every step so it survives crashes, deploys, and restarts. But that same machinery is what makes runs too heavy for the fast, throwaway loop you need to check an LLM's output quality. The properties that buy durability are the ones that kill iteration speed.
-
/articles/securing-mcp-production-gateway/en/smallimage/securing-mcp-production-gateway-thumbnail-1784884301110.jpg)
Securing MCP in Production: Defense-in-Depth beyond the Gateway
This article presents a defense-in-depth approach for securing Model Context Protocol (MCP) deployments in production. It outlines four architectural control layers: safe execution, management infrastructure, outbound trust, and semantic integrity, arguing that production security requires enforcement beyond the gateway at the earliest trustworthy control points.