The vulnerability is a buffer overflow in the rust-openssl crate, affecting applications using OpenSSL 1.1.1. The core issue lies in the Deriver::derive and PkeyCtxRef::derive functions. These functions are Rust wrappers around the native OpenSSL function EVP_PKEY_derive. The vulnerability arises because the Rust code incorrectly assumed that the EVP_PKEY_derive function would always respect the size of the output buffer provided. However, in OpenSSL 1.1.x, for certain key derivation algorithms like X25519 and X448, the function ignores the provided buffer length and writes the entire derived key. If the caller provides a buffer that is smaller than the full key, a heap or stack-based buffer overflow occurs. This can lead to a crash or potentially arbitrary code execution.
The patch mitigates this by introducing a size check. Before calling the native function, the patched code determines the required buffer size. If the caller's buffer is too small, a temporary buffer of the correct size is allocated. The key derivation is performed into this safe temporary buffer, and then the result is copied (and truncated) back to the original buffer. This fix is specifically enabled for OpenSSL versions where the buggy behavior is present (1.1.x) and is not needed for OpenSSL 3.x, which handles this correctly.