InfoQ Homepage AI, ML & Data Engineering Content on InfoQ
-
/articles/vector-search-hybrid-retrieval-rag/en/smallimage/vector-search-hybrid-retrieval-rag-thumbnail-1779972811121.jpg)
Why Vector Search Alone Isn't Enough: Hybrid Retrieval for RAG
In this article, author Aaditya Chauhan discusses the limitations of RAG pipelines based purely on vector search and how an internal omni-search application using Reciprocal Rank Fusion (RRF) that combines BM25 and vector results, can enhance the search solution.
-
/articles/solving-ai-productivity-paradox-test-automation/en/smallimage/solving-ai-productivity-paradox-test-automation-thumbnail-1779953915743.jpg)
The AI Productivity Paradox in Test Automation: Moving beyond Structural Validation to Perception and Intent
The AI productivity paradox states that AI scales whatever abstraction it is built on. If that abstraction is structurally brittle, it scales structural brittleness. This article shows that to build a future of reliable, AI-driven test automation, we must stop scaling DOM-centric abstractions and build a new testing paradigm grounded in perception and intent.
-
/articles/architecting-cloud-native-kafka/en/smallimage/architecting-cloud-native-kafka-thumbnail-1779433382367.jpg)
Architecting Cloud-Native Kafka: from Tiered Storage towards a Diskless Future
This article explores Kafka's transition toward a cloud-native architecture, examining how tiered storage, FinOps telemetry, elastic consumer scaling, virtual clusters, and Share Groups reshape the operational and economic model of event streaming platforms. It also analyzes emerging diskless-storage proposals and their architectural trade-offs.
-
/articles/secure-mcp-server-aws/en/smallimage/secure-mcp-server-aws-thumbnail-1778765095736.jpg)
Building a Secure MCP Server on AWS for a Million-Company B2B Platform
We wanted to expose a B2B intelligence platform built on more than one million company profiles to an LLM client through an MCP server so a user can ask “find SaaS companies in Germany with 50-200 employees” and receive results through the LLM client. The engineering problem was: how do you make that workflow useful without creating an unsafe bridge between an LLM and production data?
-
/articles/time-series-storage-design/en/smallimage/Time-Series-Storage-Design-Choices-That-Shape-Cost-and-Performance-thumb-1778155792101.jpg)
Time-Series Storage: Design Choices That Shape Cost and Performance
Every time-series database makes a set of storage design decisions: how to lay out rows, when to compress, what to partition on. These decisions determine cost and query performance more than the choice of database itself. This article works through those fundamentals from first principles, using widely available tools like PostgreSQL and Apache Parquet to make each trade-off measurable.
-
/articles/local-first-ai-inference-cloud/en/smallimage/Local-First-AI-Inference-A-Cloud-Architecture-Pattern-for-Cost-Effective-Document-Processing-thumb-image-1778141518292.jpg)
Local-First AI Inference: a Cloud Architecture Pattern for Cost-Effective Document Processing
The Local-First AI Inference pattern routes 70–80% of documents to deterministic local extraction at zero API cost, reserving Azure OpenAI calls for edge cases and flagging low-confidence results for human review. Deployed on 4,700 engineering drawing PDFs, it cut API costs by 75% and processing time by 55%, while bounding errors through a human review tier.
-
/articles/micro-batch-streaming-lessons-learned/en/smallimage/micro-batch-streaming-lessons-learned-thumbnail-1777381781538.jpg)
From Batch to Micro-Batch Streaming: Lessons Learned the Hard Way in a Delta Index Pipeline
This article describes how a production delta-index pipeline migrated from scheduled batch to micro-batch Spark Structured Streaming. It covers why record-level streaming was rejected, how partition-based watermarks replaced fragile S3 completion markers, overlap-window correctness, and restart-as-design strategies for better predictability in object-store–based ingestion systems.
-
/articles/securing-autonomous-ai-agents-kubernetes/en/smallimage/securing-autonomous-ai-agents-kubernetes-thumbnail-1777378848477.jpg)
Securing Autonomous AI Agents on Kubernetes: Trust Boundaries, Secrets, and Observability for a New Category of Cloud Workload
Autonomous AI agents break Kubernetes security assumptions with dynamic dependencies, multi-domain credentials, and unpredictable resource use. This article covers production-tested patterns: Job-based isolation, Vault for scoped short-lived credentials, a four-phase trust model from shadow mode to autonomous operation, and observability for non-deterministic reasoning cycles.
-
/articles/ai-code-guardian/en/smallimage/ai-code-guardian-thumbnail-1776157217464.jpg)
CodeGuardian: a Model Context Protocol Server for AI-Assisted Code Quality Analysis and Security Scanning
CodeGuardian is an MCP server that extends AI coding assistants with comprehensive code quality and security analysis capabilities. By implementing eleven specialized tools, CodeGuardian enables developers to access enterprise-grade analysis directly through their AI assistant, eliminating context-switching and reducing friction in adopting secure coding practices.
-
/articles/mcp-java-architectural-strategy-llm-integrations/en/smallimage/mcp-java-architectural-strategy-llm-integrations-thumbnail-1776772947180.jpg)
MCP in the Java World: Bringing Architectural Strategy to LLM Integrations
Discover how the Model Context Protocol (MCP) Java SDK is establishing a new architectural discipline for enterprise LLM integrations. By defining explicit contracts and leveraging MCP servers as anti-corruption layers, it ensures governance, loose coupling, and security alignment with the JVM ecosystem and existing operational practices, moving integrations beyond fragility to resilience.
-
/articles/orchestrating-agentic-multimodal-ai-pipelines-apache-camel/en/smallimage/orchestrating-agentic-multimodal-ai-pipelines-apache-camel-thumbnail-1776763980414.jpg)
Orchestrating Agentic and Multimodal AI Pipelines with Apache Camel
In this article, author Vignesh Durai discusses how agentic and multimodal AI systems can be engineered using Apache Camel and LangChain4j technologies. The key components in the solution include LLM-based reasoning, retrieval-augmented generation (RAG), and image classification.
-
/articles/lakehouse-sql-identifier-rules/en/smallimage/lakehouse-sql-identifier-rules-thumbnail-1776241856705.jpg)
Lakehouse Tower of Babel: Handling Identifier Resolution Rules across Database Engines
Lakehouse architectures enable multiple engines to operate on shared data using open table formats such as Apache Iceberg. However, differences in SQL identifier resolution and catalog naming rules create interoperability failures. This article examines these behaviors and explains why enforcing consistent naming conventions and cross-engine validation is critical.