The vulnerability, GHSA-5x3r-wrvg-rp6q, exists because Netty's HTTP/2 server implementation did not enforce a default limit on the number of concurrent streams a client could open. The DefaultHttp2Connection.DefaultEndpoint would initialize its maxActiveStreams to Integer.MAX_VALUE. An attacker could exploit this by repeatedly opening new streams on a single connection, leading to uncontrolled resource consumption (denial-of-service) on the server.
The patch, found in commit f5da73e6fcde976020b336dc7d44808f61b3e70c, rectifies this by proactively setting a limit on the remote endpoint during the connection handler's initialization. This is done in the AbstractHttp2ConnectionHandlerBuilder class, within the buildFromConnection and buildFromCodec methods. A new method, enforceMaxActiveStreams, is introduced and called from these builder methods to apply either the user-defined maxConcurrentStreams setting or a sane default (100) if none is provided.
During an exploit, the io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.createStream function would be the one appearing in a runtime profile, as it is invoked for every new stream request. The root cause, however, lies in the AbstractHttp2ConnectionHandlerBuilder methods (build, buildFromConnection, buildFromCodec) which failed to properly configure the stream limit in the first place.