The vulnerability is a resource exhaustion issue in Netty's SSL/TLS handling logic. The core of the issue lies in the SslClientHelloHandler.decode method, which is responsible for parsing the initial ClientHello message. This method reads the total handshake length from the TLS record and allocates a buffer of that size. A check exists to prevent excessively large allocations (handshakeLength > maxClientHelloLength && maxClientHelloLength != 0), but it could be bypassed.
The vulnerability was exploitable because the SniHandler and AbstractSniHandler classes, which are commonly used for SNI (Server Name Indication) support, were initializing their parent SslClientHelloHandler with a maxClientHelloLength of 0. This value effectively disabled the length validation check.
An attacker could exploit this by sending a specially crafted TLS ClientHello message fragmented into multiple records. The first fragment, as small as nine bytes, could declare a handshake length of up to 16MB (the maximum allowed by the 24-bit length field). The SslClientHelloHandler, with its disabled validation, would then immediately attempt to allocate a 16MB buffer, leading to an OutOfMemoryError and a denial of service.
The patch addresses this by introducing a sane default limit of 64KB for the maxClientHelloLength and ensuring that the SniHandler and AbstractSniHandler constructors use this safe default instead of 0, thereby enabling the crucial length validation.