InfoQ Homepage Algorithms Content on InfoQ
-
Uber Redesigns M3DB Sharding with Subclusters to Limit Failure Impact
Uber has redesigned shard placement in M3DB with fixed size subclusters to limit the impact of node failures, maintenance, and cluster scaling. The approach bounds shard dependencies, preserves replica isolation, and uses a greedy algorithm to select shard migrations while avoiding a separate rebalancing pass and unnecessary data movement.
-
From Memory-Hungry HNSW to Quantized SPANN: the Technical Evolution of Pinterest's Manas Platform
Pinterest Engineering has enhanced its Manas search platform to manage vast data, improving efficiency in search and discovery functions. By applying Scalar and Product Quantization, memory usage decreased significantly while maintaining high recall rates. The platform utilizes SSDs for optimized performance, and is transitioning to multi-vector models for refined relevance matching.
-
JioHotstar Explains the Distributed Engineering behind Personalized Ad Requests at Streaming Scale
JioHotstar explains the distributed architecture behind its real-time ad request workflow, covering ad decisioning, waterfall tiering, pacing algorithms, latency optimization, and service coordination required to select and deliver personalized advertisements during streaming playback at scale.
-
Ink & Switch Introduces Bijou64: Canonical Variable-Length Integer Encoding for Safe Parsing
Ink & Switch published bijou64, a variable-length integer encoding where every number has exactly one byte representation, closing the canonicality bug class behind attacks on PKCS#1, JWT libraries, and Bitcoin. The design also decodes two to ten times faster than LEB128. Community ports to Elixir, Go, Perl, and Java followed, while HN commenters debated SIMD performance and residual range checks.
-
DoorDash Uses Envoy and Valkey for a 1.5M RPS Proxy Cache with 99.99999% Availability
DoorDash has developed Entity Cache, a transparent proxy caching platform built on Envoy and Valkey to reduce redundant service-to-service requests across its microservices architecture. Operating within DoorDash’s service mesh, the platform serves over 1.5M requests per second with 99.99999% availability through caching, event-driven invalidation, failure handling, and performance optimizations.
-
Pinterest Uses Content Fingerprints for URL Deduplication across Millions of Domains
Pinterest introduced MIQPS, a URL normalization system that identifies which query parameters affect page identity using rendered content fingerprints. It reduces duplicate processing across millions of domains by replacing rule-based approaches with offline analysis, anomaly detection, and runtime parameter maps, improving ingestion efficiency and scalability in large-scale content pipelines.
-
Expired Oracle Patent Opens Fast Sorting Algorithm to Open Source Databases
A recent article reports that an Oracle patent on a fast sorting method has expired, allowing open source databases to use it freely. Mark Callaghan, the inventor behind the sorting algorithm, shows how this 20-year-old approach can speed up sorting similar data and could make database systems faster and more efficient.
-
Airbnb Adds Adaptive Traffic Control to Manage Key Value Store Spikes
Airbnb upgraded Mussel, its multi-tenant key-value store, replacing static per-client rate limits with an adaptive, resource-aware traffic control system. The redesign ensures resilience during traffic spikes, protects critical workflows, and maintains fair usage across thousands of tenants while scaling efficiently.
-
DevGreenOps: How to Design Sustainable Digital Services
DevGreenOps, also known as DevSusOps, is an extension of the DevOps approach, in which environmental sustainability considerations are integrated into every step of the DevOps cycle, Jochen Joswig said in his talk at OOP Conference. Applying transparency, minimalism, efficiency, and awareness helps us to design sustainable digital services.
-
How Developers Can Eliminate Software Waste and Reduce Climate Impact
High performance and sustainability correlate; making software go faster by improving the efficiency of algorithms can reduce energy requirements, Holly Cummins said at QCon London. She suggested switching systems off when not in use to reduce the environmental footprint. Developers can achieve more by doing less, improving productivity, she said.
-
Green Software Development - What Can You Do Now, and Where is the Industry Heading?
Making code more efficient often ends up saving carbon. Storing less information and compressing it can also lower your carbon footprint. There are open-source projects and standards and guides available that can be used to increase sustainability in software development. Measurement standardization is needed to compare the environmental impact of cloud suppliers.
-
Amazon Switched Compression from Gzip to Zstd for Own Service Data
A tweet from Adrian Cockcroft, former VP at Amazon, recently highlighted the benefits of switching from gzip to Zstandard compression at Amazon and triggered discussions in the community about the compression algorithm. Other large corporations, including Twitter and Honeycomb, shared interesting gains using zstd.
-
Microsoft's New Simulation Framework FLUTE Accelerates Federated Learning Algorithm Development
Microsoft Research has recently released Federated Learning Utilities and Tools for Experimentation (FLUTE), a new simulation framework to accelerate federated learning ML algorithm development. The main goal of federated learning is to train complex machine-learning models over massive amounts of data without the need to share that data in a centralized location.
-
Every Question Has an Answer: an Impossible Thing for Developers
We tend to assume that every question has an answer, which for instance isn’t true when we want to find out what the current time is. Developers should increase awareness of unexpected failure modes, advertise the possibility of failure, and use time-outs to recover from waiting for an answer that will never come.
-
Infinite Representations: an Impossible Thing for Developers
Developers can face impossible things in their daily work. It’s impossible to directly represent infinity or to hold infinite precision on a discrete physical computer. Storage and representations are bounded. Ignoring or being unaware of this impossibility can lead to bugs or systems behaving differently than expected. Kevlin Henney gave the keynote Six Impossible Things at QCon London 2022.