InfoQ Homepage Infrastructure Content on InfoQ
-
/articles/agent-reinforcement-learning-apache-spark/en/smallimage/thumbnail-1769431334097.jpg)
Autonomous Big Data Optimization: Multi-Agent Reinforcement Learning to Achieve Self-Tuning Apache Spark
This article introduces a reinforcement learning (RL) approach grounded in Apache Spark that enables distributed computing systems to learn optimal configurations autonomously, much like an apprentice engineer who learns by doing. The author also implements a lightweight agent as a driver-side component that uses RL to choose configuration settings before a job runs.
-
/articles/durable-objects-handle-inflight-requests/en/smallimage/durable-objects-handle-inflight-requests-thumbnail-1769161248259.jpg)
One Cache to Rule Them All: Handling Responses and In-Flight Requests with Durable Objects
Traditional caching fails to stop "thundering herds" where multiple clients trigger the same work during a miss. This article proposes using Cloudflare Durable Objects to treat in-flight work and finished results as two states of one cache entry. By routing to a single owner, systems eliminate redundant tasks. This pattern replaces complex locks with simple promises, simplifying the system design.
-
/articles/reducing-false-positives-retrieval-augmented-generation/en/smallimage/reducing-false-positives-retrieval-augmented-generation-thumbnail-1762786576150.jpg)
Reducing False Positives in Retrieval-Augmented Generation (RAG) Semantic Caching: a Banking Case Study
In this article, author Elakkiya Daivam discusses why Retrieval Augmented Generation (RAG) and semantic caching techniques are powerful levers for reducing false positives in AI powered applications. She shares the insights from a production-grade evaluation with 1,000 query variations tested across seven bi-encoder models.
-
/articles/building-resilient-platforms-mission-critical-infrastructure/en/smallimage/building-resilient-platforms-mission-critical-infrastructure-thumbnail-1762169439707.jpg)
Building Resilient Platforms: Insights from over Twenty Years in Mission-Critical Infrastructure
Building resilient platforms requires understanding the art and science of creating infrastructure that others depend on for critical applications. This perspective applies to anyone who builds software consumed by others at scale. Whether developing infrastructure platforms, software development platforms, or messaging systems, principles address how to build software that others consume at scale
-
/articles/llms-evolution-ai-infrastructure/en/smallimage/llms-evolution-ai-infrastructure-thumbnail-1758096401785.jpg)
Disaggregation in Large Language Models: the Next Evolution in AI Infrastructure
Large Language Model (LLM) inference faces a fundamental challenge: the same hardware that excels at processing input prompts struggles with generating responses, and vice versa. Disaggregated serving architectures solve this by separating these distinct computational phases, delivering throughput improvements and better resource utilization while reducing costs.
-
/articles/ransomware-resilient-storage-cyber-defense/en/smallimage/ransomware-resilient-storage-cyber-defense-thumbnail-1755589602893.jpg)
Ransomware-Resilient Storage: the New Frontline Defense in a High-Stakes Cyber Battle
Cybersecurity has evolved, with ransomware now primarily targeting data storage and backups. To combat this, modern defense strategies focus on making storage systems more resilient. Key tactics include using immutable storage that prevents data from being altered or deleted, employing AI-powered detection, and implementing air-gapping to create isolated, tamper-proof recovery points.
-
/articles/zero-downtime-cloud-upgrades/en/smallimage/zero-downtime-cloud-upgrades-thumbnail-1754398638654.jpg)
Zero-Downtime Critical Cloud Infrastructure Upgrades at Scale
Engineers can avoid common pitfalls in large-scale infrastructure upgrades by studying others' experiences. The article provides lessons learned from big firms like eBay and Snowflake, offering solutions for legacy systems, performance validation, and rollback planning. It emphasizes systematic preparation and clear communication to handle challenges and ensure zero-downtime upgrades at scale.
-
/articles/one-network-service-policy-architecture/en/smallimage/one-network-service-policy-architecture-thumbnail-1754550916269.jpg)
One Network: Cloud-Agnostic Service and Policy-Oriented Network Architecture
Bringing together software infrastructure leads to faster development time and easy control of large, spread-out systems through clear rules. In this QCon SF 2024 presentation, Anna Berenberg shared learnings and achievements when building One Network, addressing complex infrastructure layers, open-source integration, and uniform policy enforcement for improved reliability and security.
-
/articles/ceph-rbd-open-source/en/smallimage/ceph-rbd-open-source-thumbnail-1751612399293.jpg)
Ceph RBD Turns 15: a Story of Open Source Creation
Fifteen years ago, Ceph RBD began as a community-driven idea that grew into essential infrastructure powering today's cloud platforms. This insider story from Yehuda Sadeh-Weinraub reveals how two developers started a distributed storage that now supports OpenStack and Kubernetes through transparent, collaborative development.
-
/articles/apache-kafka-stretch-cluster-failures/en/smallimage/apache-kafka-stretch-cluster-failures-thumbnail-1750152098922.jpg)
Analyzing Apache Kafka Stretch Clusters: WAN Disruptions, Failure Scenarios, and DR Strategies
Proficient in analyzing the dynamics of Apache Kafka Stretch Clusters, I assess WAN disruptions and devise effective Disaster Recovery (DR) strategies. With deep expertise, I ensure high availability and data integrity across multi-region deployments. My insights optimize operational resilience, safeguarding vital services against service level agreement violations.
-
/articles/scalable-resilient-event-systems/en/smallimage/scalable-resilient-event-systems-thumbnail-1748422904015.jpg)
Designing Resilient Event-Driven Systems at Scale
Learn how to design resilient event-driven systems that scale. Explore key patterns like shuffle sharding and decoupling queues to handle load spikes and failures. Understand common pitfalls like over-relying on retries and neglecting observability for robust, scalable architectures.
-
/articles/complex-applications-storage-constraints/en/smallimage/compex-applications-storage-constraints-thumbnail-1747126977374.jpg)
Binary Size Matters: the Challenges of Fitting Complex Applications in Storage-Constrained Devices
This article explores developing software for microcontrollers in C or C++, where constraints are the limited amount of volatile memory and the embedded hardware platform on which the software runs. It shows how to adopt languages like C++ while optimizing for binary size due to stringent hardware constraints, and trade off between runtime efficiency and binary size in architecture decisions.