InfoQ Homepage Key-Value Store Content on InfoQ
-
Meta’s ZGateway Cuts ZippyDB Connections 19x While Handling 1B+ Operations per Second
Meta has introduced ZGateway, a stateless proxy for ZippyDB that centralizes connection management, traffic routing, caching, load balancing, and admission control. The gateway handles more than 1 billion operations per second and about 40% of ZippyDB traffic, while Meta’s model estimates a 19x reduction in persistent connections.
-
Home Made CobbleDB Replaces DynamoDB at Perplexity to Cut Query Latency 5x and Reduce Cloud Storage
Perplexity has migrated its search infrastructure from Amazon DynamoDB to CobbleDB, an internally developed key-value store in Rust. This change reduced latency and costs associated with handling large document batches. The new architecture supports high query volumes more efficiently, achieving improved latency and reduced storage expenses while managing significant production traffic.
-
Inside Netflix’s Graph Abstraction: Handling 650TB of Graph Data in Milliseconds Globally
Netflix engineers built Graph Abstraction, a high-throughput platform managing 650 TB of graph data with millisecond latency. Supporting services from Netflix Gaming’s social graphs to operational topology graphs, it maintains global availability via asynchronous replication. This article covers its architecture, caching, and traversal design for high-scale performance.
-
Airbnb Adds Adaptive Traffic Control to Manage Key Value Store Spikes
Airbnb upgraded Mussel, its multi-tenant key-value store, replacing static per-client rate limits with an adaptive, resource-aware traffic control system. The redesign ensures resilience during traffic spikes, protects critical workflows, and maintains fair usage across thousands of tenants while scaling efficiently.
-
Airbnb’s Mussel V2: Next-Gen Key Value Storage to Unify Streaming and Bulk Ingestion
Airbnb’s engineering team re-architected its internal key-value storage system, Mussel, to unify streaming and bulk ingestion while simplifying operations, achieving over 100,000 writes per second and sub-25ms read latencies on 100-terabyte tables, while leveraging Kubernetes, Kafka, and a NewSQL backend to improve scalability, reliability, and operational efficiency across its internal services.
-
Cloudflare Rearchitects Workers KV Following GCP Outage, Achieves 40x Performance Gain
Cloudflare has recently redesigned Workers KV with a hybrid storage architecture that automatically routes objects between distributed databases and object storage based on size characteristics, while operating dual storage backends. This change improved the p99 read latencies from 200ms to under 5ms for their global key-value store while handling hundreds of billions of key-value pairs.
-
How Cloudflare Migrated Quicksilver to Multi-Level Caching While Serving Billions of Requests
The engineering team at Cloudflare recently shared how they transitioned Quicksilver, their internal global key-value store, to a tiered caching architecture. They described their incremental journey from storing everything everywhere to adopting a distributed caching system, improving storage efficiency while preserving consistency guarantees and low-latency reads at the edge.
-
Redis Returns to Open Source under AGPL License: Is It Too Late?
Redis 8 has recently hit general availability, switching to the AGPLv3 license. A year after leaving its open source roots to challenge cloud service providers and following the birth of Valkey, Redis has rehired its creator and moved back to an open source license.
-
Valkey 8.0 Now Generally Available with Improved Memory Efficiency
The Linux Foundation has announced the general availability of Valkey 8.0, the open source in-memory storage solution developed as a successor to Redis. By introducing a dictionary per slot and embedding keys directly into dictionary entries, developers can achieve up to 20% more capacity, allowing for the storage of additional keys per node.
-
Netflix’s Pushy: Evolution of Scalable WebSocket Platform That Handles 100Ms Concurrent Connections
Netflix shared details on the evolution of Pushy, a WebSocket messaging platform that supports push notifications and inter-device communication across many different devices for the company’s products. Netflix’s engineers implemented many improvements across the Pushy ecosystem to ensure the platform's scalability and reliability and support new capabilities.
-
Async Ops and Scalable Keyspaces Make Threads Go Viral
Meta's Engineering team has published a post explaining how they built the infrastructure for Threads, their new online social media and networking service. The decision to launch was abrupt, with the infrastructure teams having just two days' notice. However, the teams were confident in Meta's infrastructure's maturity and past performance to effectively support the app's rapid growth.
-
AWS Introduces Amazon CloudFront KeyValueStore: a Low Latency Datastore for CloudFront Functions
AWS recently announced Amazon CloudFront KeyValueStore, a secure global low-latency key-value datastore that allows read access from within CloudFront Functions, enabling advanced customizable logic at the CloudFront edge locations.
-
JunoDB: PayPal Open Sources Key-Value Store Powering 350 Billion Daily Requests
PayPal recently open-sourced JunoDB, a distributed key-value store that uses RocksDB as the underlying storage engine. A highly available and security-focused database, JunoDB serves daily 350 billion requests at PayPal.
-
Amazon DynamoDB - Evolution of a Hyper-Scale Cloud Database Service: Akshat Vig at QCon SF 2022
Akshat Vig, a principal engineer in NoSQL databases at Amazon, spoke at QCon San Francisco about Amazon DynamoDB: Evolution of a Hyper-Scale Cloud Database Service. The talk was part of the "Architectures You've Always Wondered About" track.
-
How Pinterest Scaled up Its Ad-Serving Architecture
On the Pinterest Engineering Blog, Nishant Roy wrote about their strategy to overcome a scaling problem with their ad corpus. Their existing solution had hit its scaling limit, but further growth was necessary. Its redesign offloaded the ad index to a key-value store and optimised garbage collection in their Go applications to increase their ad corpus size by a factor of 60.