BT

Facilitating the spread of knowledge and innovation in professional software development

Contribute

Topics

Choose your language

InfoQ Homepage News Facebook Compression Algorithm Zstandard 1.5 Improves Performance

Facebook Compression Algorithm Zstandard 1.5 Improves Performance

Bookmarks

Facebook open sourced Zstandard almost six years ago with the aim of outperforming Zlib in both speed and efficiency. Zstandard 1.5 improves compression speed at intermediate compression levels, compression ratio at higher levels, and brings faster decompression speed.

Zstandard supports compression levels up to 22. Thanks to a new default match finder, Zstardard 1.5 reaches higher compression speed for levels between 5 and 12 and inputs larger than 256K. According to Facebook benchmarks, improvements range from +25% to +140% without significant loss in terms of compression ratio. Facebook claims even better results on heavily loaded machines under significant cache contention.

The biggest gains are achieved on files typically larger than 128KB. On files smaller than 16KB, by default we revert back to the legacy match finder which becomes the faster one. This default policy can be overriden manually: the new match finder can be forcibly enabled with the advanced parameter ZSTD_c_useRowMatchFinder, or through the CLI option --[no-]row-match-finder.

At the opposite end of the compression level spectrum, Zstandard uses block splitting by default to improve compression ratio, although not by such an impressive margin as in the previously described case.

The amount of benefit varies depending on the workload. Compressing archives comprised of heavily differing files will see more improvement than compression of single files that don’t vary much entropically (like text files/enwik).

In particular, Facebook benchmarks show that Zstandard 1.5 may get up to 0.71% better compression ratio at level 22, the maximum allowed.

Zstandard 1.5 also significantly improves decompression speed, with improvements up to 21% in the best case. Actual results, though, depend largely on compiler version, payload, and compression level, says Facebook.

In general, a majority of scenarios see benefits ranging from +1 to +9%. There are also a few outliers here and there, from -4% to +13%. The average gain across all these scenarios stands at ~+4%.

Besides improving compression performance, Zstandard 1.5 builds by default with multithreaded support, standardizes a few new APIs, and deprecates a number of older ones. You can find the full detail in the official release notes.

Zstandard is based on work by Facebook engineers Yann Collet and Chip Turner. In particular, it leverages previous work by Collet based on Asymmetric numeral system (ANS).

Zstandard is integrated both within the Linux kernel, where it is used for the btrfs and squashfs filesystems, and FreeBSD, where it is used to compress core dumps. Additionally, it is used in Arch Linux and Fedora. If you want to try out Zstandard, the open-source file archiver 7zip includes support for it, along with many other codecs.

We need your feedback

How might we improve InfoQ for you

Thank you for being an InfoQ reader.

Each year, we seek feedback from our readers to help us improve InfoQ. Would you mind spending 2 minutes to share your feedback in our short survey? Your feedback will directly help us continually evolve how we support you.

Take the Survey

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT

Is your profile up-to-date? Please take a moment to review and update.

Note: If updating/changing your email, a validation request will be sent

Company name:
Company role:
Company size:
Country/Zone:
State/Province/Region:
You will be sent an email to validate the new email address. This pop-up will close itself in a few moments.