When Microsoft engineer Andres Freund noticed SSH was taking longer than usual, he discovered a backdoor in xz utils, one of the underlying libraries for systemd, that had taken years to be put in place. The United States Cybersecurity & Infrastructure Security Agency (CISA) has assigned CVE-2024-3094 to the issue. The backdoor had found its way into testing releases of Linux distributions like Debian Sid, Fedora 41 and Fedora Rawhide, but was caught before propagating into more highly used stable releases. However, there’s evidence that the attackers were pressuring distro maintainers to speed up its deployment.
Evan Boehs provides a detailed timeline and analysis of the attack in "Everything I know about the XZ backdoor", which runs back to 2021 when a GitHub account, JiaT75, was created for "Jia Tan". Initial activity from that account was on the libarchive code, but in April 2022 "Jia Tan" moved on to XZ, creating a patch, and another persona "Jigar Kumar" started pressuring the project maintainer, Lasse Collin. Over time, "Jia Tan" took over a substantial part of the ongoing maintenance of XZ and used their position to insert the backdoor using a sophisticated attack against the build process where the code was hidden inside of tests. Earlier efforts at making the code and build process safer and more secure were also undermined, with "improved security" routinely used as the false reason for the changes. Security expert Bruce Schneier links to Thomas Roccia’s infographic in describing it as "a masterful piece of work", and goes on to say:
It affects the SSH remote login protocol, basically by adding a hidden piece of functionality that requires a specific key to enable. Someone with that key can use the backdoored SSH to upload and execute an arbitrary piece of code on the target machine. SSH runs as root, so that code could have done anything. Let your imagination run wild.
This isn’t something a hacker just whips up. This backdoor is the result of a years-long engineering effort. The ways the code evades detection in source form, how it lies dormant and undetectable until activated, and its immense power and flexibility give credence to the widely held assumption that a major nation-state is behind this.
Lasse Collin has provided his own account of events in an XZ Utils backdoor page. It’s somewhat evocative of XKCD 2347 ‘Dependency’, where the stability and security of an entire ecosystem is propped up by a lone maintainer. It’s also a painful illustration of why the "bus factor" is an important measure of the health of a dependency, which is why it’s included in measures like the Open Source Security Foundation (OpenSSF) Best Practices. Whoever the attackers are, they took time to identify the weakest link in the software supply chain and exploit the human frailties associated with that.
As the backdoor only got as far as test systems, it’s mostly being treated as a "near miss" incident that the industry can learn from. The OpenJS Foundation has published an alert in partnership with OpenSSF "Social Engineering Takeovers of Open Source Projects", where they identify similar attempts to subvert JavaScript projects. Industry veteran Tim Bray looks to the future in proposing "Open Source Quality Institutes" (OSQI) as a means to provide funding and governance for critical open source projects.
Software supply chain security has become a hot topic in recent years, and this attack only serves to highlight why it’s so important. If the back door code hadn’t revealed itself to a diligent engineer by being just a bit too slow, then over the course of months and years it would have left many systems open to the attackers. Though it’s not the only "pre auth" failure to crop up recently, with similar issues impacting Palo Alto (CVE-2024-3400) and Ivanti (CVE-2024-21887).