Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Uber's CacheFront: Powering 40M Reads per Second with Significantly Reduced Latency

Uber's CacheFront: Powering 40M Reads per Second with Significantly Reduced Latency

This item in japanese

Uber developed an innovative caching solution, CacheFront, for its in-house distributed database, Docstore. CacheFront enables over 40M reads per second from online storage and achieves substantial performance improvements, including a 75% reduction in P75 latency and over 67% reduction in P99.9 latency, demonstrating its effectiveness in enhancing system efficiency and scalability.

Cache vs storage engine latency comparison (Source)

Uber's engineers, Preetham Narayanareddy, Eli Pozniansky, Zurab Kutsia, Afshin Salek, and Piyush Patel, describe the challenges that required them to develop CacheFront:

Most of the microservices at Uber use databases backed by disk-based storage in order to persist data. However, every database faces challenges serving applications that require low-latency read access and high scalability.

This came to a boiling point when one use case required much higher read throughput than any of our existing users. Docstore could have accommodated their needs, as it is backed by NVMe SSDs, which provide low latency and high throughput. However, using Docstore in the above scenario would have been cost-prohibitive and would have required many scaling and operational challenges.

Some teams at Uber used Redis caching to speed up read access and overcome these limitations. However, each team has to provision and maintain their own Redis cache for their respective services. They also had to implement their invalidation logic for its use case. In a region failover, teams must either maintain caching replication to stay hot or suffer higher latencies while the cache is warming up in other regions. One of CacheFront's goals was to centrally implement and manage these features, allowing teams to focus on their core logic.

Since Uber wanted to enable and scale Redis caching separately from the storage engine while maintaining API compatibility with previous Docstore versions, they decided to integrate it into Docstore's query engine. The query engine is a stateless component acting as a frontend to a MySQL-based stateful storage engine.

CacheFront design (Source)

CacheFront uses a cache aside strategy to implement cached reads. The query engine gets a read request for one or more rows. If caching is enabled, it tries to get rows from Redis and stream responses to users. It retrieves the remaining rows (if any) from the storage engine and asynchronously populates Redis with the remaining rows while streaming them to users.

Uber's engineers leveraged Docstore's integrated Change Data Capture (CDC) engine to handle cache invalidation. Whenever the CDC identifies a DB update, the correlating row in Redis is either updated or invalidated. As a result, Uber can make the cache consistent within seconds of the database change, as opposed to minutes, by using standard Time-to-Live (TTL) mechanisms. Additionally, by using CDC, uncommitted transactions don't pollute the cache.

CacheFront read and write paths for invalidation (Source)

To measure the cache's consistency level versus the DB, Uber's engineers added a special mode that shadows read requests to the cache. When reading back, it compares the cached and database data and verifies that they are identical. Any mismatches are logged and emitted as metrics. Using CDC-based cache invalidation, they measured the cache is 99.99% consistent.

Uber's Docstore operates in an active-active deployment across two geographical regions to ensure high availability and fault tolerance. However, this setup presented a challenge for CacheFront, particularly in maintaining "warm" caches in both regions to prevent increased database loads due to cache misses during failovers. To address this, Uber's engineers tail the Redis write stream and replicate row keys, not values, to the remote region. In the remote region, the replication engine fetches the up-to-date value from storage on a cache miss.

Cache warming (Source)

Uber identified challenges in setting optimal timeouts for Redis operations, where a timeout that is too short could lead to premature request failures and unnecessary database load. At the same time, a too long timeout could adversely affect latencies. Uber implemented adaptive timeouts to address this, allowing automatic and dynamic adjustment of Redis operation timeouts based on performance data.

This approach ensures that the vast majority of requests (99.99%) are served quickly from the cache, with a mechanism in place to promptly cancel and redirect the few requests that exceed the dynamically adjusted timeout to the database, thus avoiding manual adjustments and optimizing both cache efficiency and database load management.

About the Author

Rate this Article