Dropbox recently explained how it improved storage efficiency in Magic Pocket, the company's internal immutable blob store for storing user files at scale, by redesigning compaction strategies to reclaim space from severely underfilled storage volumes. The system now periodically reorganizes valid data into new volumes, allowing old, partially used ones to be cleared and reused.
As described in the "Increasing Magic Pocket write throughput by removing our SSD cache disks" article, Dropbox introduced a new service last year that changed how data is distributed across the immutable blob store, reducing write amplification for background writes. However, the change also led to an unintended increase in data fragmentation.
Dropbox's backend stores files as small objects across different servers and treats stored data as immutable. While this design improves reliability, the old data cannot be removed immediately when files are updated or deleted. Facundo Agriel, staff software engineer at Dropbox, explains the challenge:
Because data is immutable, deletes do not immediately free up disk space. Old data stays on-disk inside storage volumes. Once a volume is closed, it is never reopened. The tradeoff is that deletes leave unused space behind, and that waste grows over time unless we actively reclaim it. Without reclamation, volumes gradually become partially filled, spreading live data across more disks than necessary. Fragmentation due to a lack of reclamation can significantly increase storage overhead.
Earlier this year, Dropbox discovered that a new "Live Coder" service was creating severely underfilled storage volumes, sometimes using less than 5% of their capacity. This spread data across many nearly empty volumes, increasing fragmentation and storage overhead and exposing limits in the existing compaction system.
Dropbox's original compaction strategy worked well when most storage volumes were nearly full, but became inefficient when many volumes were severely underfilled. To address this, Dropbox redesigned the system responsible for reclaiming space. The updated approach prioritizes the most inefficient volumes and manages cleanup work more carefully to avoid straining system resources. Agriel adds:
Compaction performs the physical reclamation. Because volumes cannot be modified once closed, we gather the live blobs from volumes, write them into new volumes, and retire the old ones. This is how deletes eventually translate into reusable space.
The new compaction strategy, called L2, is now used to reclaim space more quickly when many storage volumes are underfilled. Instead of slowly topping off already dense volumes as in the previous approach, L2 combines multiple sparse volumes into a single, nearly full one, allowing the system to reclaim space faster.
Source: Dropbox blog
Dropbox also introduced a third compaction strategy, L3, to handle extremely underfilled storage volumes that earlier methods could not reclaim efficiently, streaming remaining live data from these sparse volumes through the Live Coder service and gradually rewriting it into new erasure-coded volumes.
Compaction reclaims disk space in Dropbox's immutable storage system by rewriting live data into new volumes and retiring old ones, while erasure coding protects data from hardware failures by splitting it into fragments with parity pieces that allow reconstruction if parts are lost.
In a Hacker News thread, some users question the product's usability ("such fantastic engineering work is buried behind a product with so many annoyances") and its pricing, while others question the "unintended consequence" scenario in a large corporation. User nopurpose writes:
Me thinking big corps with huge infrastructure bills meticulously model changes like that using the production data they have (...) Turned out they are like me: deploy and see what breaks.
Agriel replied in the thread, noting that large-scale systems operate slowly and unevenly, making the effects of infrastructure changes difficult to detect.
As previously reported on InfoQ, Magic Pocket is Dropbox's proprietary, horizontally scalable, exabyte-scale blob storage system that replaced Amazon S3 as an object store, maintaining 99.99% availability and extremely high durability. Agriel presented Magic Pocket's implementation at QCon Plus 2023.