The vulnerability, described as an HTTP/2 Reset Attack vector, stems from Netty's handling of the SETTINGS_MAX_HEADER_LIST_SIZE setting. According to the vulnerability description, Netty, when acting as a server, would honor this advisory setting from a client. A malicious client could set this to an extremely low value, causing the server to generate exceptions when attempting to write response headers, leading to a denial of service.
Analysis of the commits between the last vulnerable version (4.1.134.Final) and the first patched version (4.1.135.Final) revealed a specific commit addressing this issue: b831454889b20776be6a73daad329404c4682e94 with the message "HTTP/2: Treat clients MAX_HEADER_LIST_SIZE as advisory".
The changes in this commit are located in the io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder class, specifically within the remoteSettings method. Before the patch, this method would read the maxHeaderListSize from the client's settings and configure the header encoder with it. The patch introduces a conditional check to determine if the connection is a server connection. If it is, it nullifies the maxHeaderListSize setting from the client, effectively ignoring it and preventing the DoS attack. Therefore, the remoteSettings function is the vulnerable function as it's the entry point for processing the malicious setting.