OpenSSL 3.0.4, released less than a month ago, introduced a bug that enabled a remote code execution vulnerability on machines computing 2048 bit RSA keys on X86_64 CPUs. A fix is now available in OpenSSL 3.0.5.
SSL/TLS servers or other servers using 2048 bit RSA private keys running on machines supporting AVX512IFMA instructions of the X86_64 architecture are affected by this issue.
As Guido Vranken explains, the OpenSSL 3.0.4 release included a fix for a bug affecting four code paths: RSAZ 1024, RSAZ 512, Dual 1024 RSAZ, and Default constant-time Montgomery modular exponentiation. This bug had no security implications, but its fix was applied incorrectly to the dual 1024 RSAZ path due to a wrong argument being passed into a function. This caused a heap buffer overflow, i.e., accessing memory outside of the expected bounds, which could be exploited to corrupt memory.
As a consequence of that, the RSA implementation of 2048 keys was broken and the heap overflow could be triggered by an attacker, for example, when doing a TLS handshake. An exploit could then disrupt services, steal or alter confidential information, and execute arbitrary code, although only DoS could be performed trivially.
I want to note that each of these except DoS are speculative and not trivial to perform, and are predicated upon meticulously controlling the regions that are overread and overwritten by way of influencing the state machine flow, though automatic exploit generators based on symbolic execution reportedly do exist.
According to Vranken, the vulnerability had a potential for being worse than Heartbleed, although a number of factors concurred to reduce its gravity, including the fact that many people have not upgraded to OpenSSL 3. Above all, it only impacted X86_64 CPUs with AVX512IFMA SIMD capabilities.
As mentioned, the fix for this vulnerability is included in OpenSSL 3.0.5, which can be downloaded from the OpenSSL site or GitHub. A workaround is also available for users of OpenSSL 3.0.4, which consists in disabling AVX512IFMA by setting OPENSSL_ia32cap=:~0x200000 in your execution environment.