The vulnerability lies in several internal FFI (Foreign Function Interface) trampoline functions within the openssl::ssl::callbacks module. These functions are responsible for bridging the gap between user-provided safe Rust callbacks and the underlying C-based OpenSSL library for handling PSK (Pre-Shared Key) and DTLS cookie logic.
The root cause of the vulnerability is a failure to validate the length (usize) returned by these user callbacks. The trampolines would receive this length and pass it directly to OpenSSL. A user's callback, either maliciously or accidentally, could return a length greater than the actual size of the buffer provided for the PSK or cookie. When OpenSSL used this unchecked length, it would read past the end of the allocated buffer into adjacent memory. This out-of-bounds data could then be sent over the network to a peer, resulting in a memory leak (buffer over-read).
The patch addresses this by introducing a check in each of the four identified trampoline functions (raw_client_psk, raw_server_psk, raw_stateless_cookie_generate, raw_cookie_generate). The fix ensures that the length returned by the callback does not exceed the capacity of the buffer. If an invalid length is detected, the operation is now safely aborted instead of passing the dangerous value to OpenSSL.