InfoQ Homepage Articles
-
/articles/spark-oom-kubernetes-misconfigurations/en/smallimage/spark-oom-kubernetes-misconfigurations-thumbnail-1780044756757.jpg)
Two Misconfigurations That Caused Spark OOM Failures on Kubernetes
After migrating Spark pipelines to Azure Kubernetes Service, two infrastructure settings interacted destructively: spark.kubernetes.local.dirs.tmpfs=true backed shuffle spill with RAM instead of disk, and a hard podAffinity rule forced all executors onto one node. Together, they caused repeated OOM kills invisible to standard diagnostics.
-
/articles/vector-search-hybrid-retrieval-rag/en/smallimage/vector-search-hybrid-retrieval-rag-thumbnail-1779972811121.jpg)
Why Vector Search Alone Isn't Enough: Hybrid Retrieval for RAG
In this article, author Aaditya Chauhan discusses the limitations of RAG pipelines based purely on vector search and how an internal omni-search application using Reciprocal Rank Fusion (RRF) that combines BM25 and vector results, can enhance the search solution.
-
/articles/solving-ai-productivity-paradox-test-automation/en/smallimage/solving-ai-productivity-paradox-test-automation-thumbnail-1779953915743.jpg)
The AI Productivity Paradox in Test Automation: Moving beyond Structural Validation to Perception and Intent
The AI productivity paradox states that AI scales whatever abstraction it is built on. If that abstraction is structurally brittle, it scales structural brittleness. This article shows that to build a future of reliable, AI-driven test automation, we must stop scaling DOM-centric abstractions and build a new testing paradigm grounded in perception and intent.
-
/articles/adaptive-hedged-requests-p99-latency/en/smallimage/thumbnail-1779785816730.jpg)
Stragglers, Not Failures: How Adaptive Hedged Requests Reduce p99 Latency by 74 Percent
In fan-out microservice architectures, slow-but-completing requests accumulate across services and drive p99 latency far higher than per-service metrics suggest. This article presents an adaptive hedging mechanism that uses DDSketch for real-time quantile estimation, windowed rotation to handle distribution drift, and a token-bucket budget to prevent load amplification.
-
/articles/architecting-cloud-native-kafka/en/smallimage/architecting-cloud-native-kafka-thumbnail-1779433382367.jpg)
Architecting Cloud-Native Kafka: from Tiered Storage towards a Diskless Future
This article explores Kafka's transition toward a cloud-native architecture, examining how tiered storage, FinOps telemetry, elastic consumer scaling, virtual clusters, and Share Groups reshape the operational and economic model of event streaming platforms. It also analyzes emerging diskless-storage proposals and their architectural trade-offs.
-
/articles/schema-proliferation-problem/en/smallimage/schema-proliferation-problem-thumbnail-1779270222602.jpg)
The Schema Proliferation Problem in Kafka and Flink Pipelines: How to Solve It
Schema proliferation builds slowly and gets expensive fast. One schema per event type feels right until there are ten tables, union queries spanning all of them, and a single field rename touching every schema. Discriminator-based schema consolidation collapses that to two tables, turning multi-table unions into a single query, while new variants are additive and don't break existing consumers.
-
/articles/capacity-planning-queue-recovery/en/smallimage/The-Mathematics-of-Backlogs-Capacity-Planning-for-Queue-Recovery-thumb-1778227922596.jpg)
The Mathematics of Backlogs: Capacity Planning for Queue Recovery
Backlogs in distributed systems are arithmetic problems, not mysteries. This article provides practical formulas for calculating backlog drain time, sizing consumer headroom, and setting auto-scaling triggers. It covers key failure modes — retry amplification, metastable states, and cascading pipeline bottlenecks — plus when to shed load instead of draining.
-
/articles/ebpf-for-security-observability/en/smallimage/ebpf-for-security-observability-thumbnail-1778674557176.jpg)
Kernel-Level Ground Truth: Why eBPF is Replacing User-Space Agents for Security Observability
eBPF is emerging as a preferred method for security observability over traditional user-space agents. By attaching probes directly to the Linux kernel's syscall interface, it provides consistent visibility even during container-level compromises. eBPF reduces security-related CPU consumption and limits data volume by performing filtering at the kernel level, enhancing operational efficiency.
-
/articles/secure-mcp-server-aws/en/smallimage/secure-mcp-server-aws-thumbnail-1778765095736.jpg)
Building a Secure MCP Server on AWS for a Million-Company B2B Platform
We wanted to expose a B2B intelligence platform built on more than one million company profiles to an LLM client through an MCP server so a user can ask “find SaaS companies in Germany with 50-200 employees” and receive results through the LLM client. The engineering problem was: how do you make that workflow useful without creating an unsafe bridge between an LLM and production data?
-
/articles/time-series-storage-design/en/smallimage/Time-Series-Storage-Design-Choices-That-Shape-Cost-and-Performance-thumb-1778155792101.jpg)
Time-Series Storage: Design Choices That Shape Cost and Performance
Every time-series database makes a set of storage design decisions: how to lay out rows, when to compress, what to partition on. These decisions determine cost and query performance more than the choice of database itself. This article works through those fundamentals from first principles, using widely available tools like PostgreSQL and Apache Parquet to make each trade-off measurable.
-
/articles/local-first-ai-inference-cloud/en/smallimage/Local-First-AI-Inference-A-Cloud-Architecture-Pattern-for-Cost-Effective-Document-Processing-thumb-image-1778141518292.jpg)
Local-First AI Inference: a Cloud Architecture Pattern for Cost-Effective Document Processing
The Local-First AI Inference pattern routes 70–80% of documents to deterministic local extraction at zero API cost, reserving Azure OpenAI calls for edge cases and flagging low-confidence results for human review. Deployed on 4,700 engineering drawing PDFs, it cut API costs by 75% and processing time by 55%, while bounding errors through a human review tier.
-
/articles/asp-net-core-side-car/en/smallimage/Implementing-the-Sidecar-Pattern-in-Microservices-based-ASP-NET-Core-Applications-thumb-1778079284259.jpg)
Implementing the Sidecar Pattern in Microservices-Based ASP.NET Core Applications
Today's applications require monitoring, logging, configuration, etc. Each of these concerns can be implemented as a component or a service. These cross-cutting concerns can be tightly integrated into the application. While this tight coupling ensures effective use of shared resources, an outage in any of these components can take your application down. Enter the sidecar design pattern.