The vulnerability, identified as GHSA-jfg9-48mv-9qgx, is an uncontrolled resource consumption flaw in Netty's MQTT codec. The root cause lies in the io.netty.handler.codec.mqtt.MqttDecoder class. When decoding an MQTT message, the decoder would parse the variable header, including a potentially very large "Properties" section, before it checked the overall message size against the configured maxBytesInMessage limit.
An attacker could exploit this by crafting an MQTT message with a small payload but an extremely large set of properties in the header. This would trigger two issues:
- Memory Exhaustion: The
decodeProperties method would be called to parse these properties, leading to the allocation of a large amount of memory to buffer them.
- CPU Exhaustion: Because
MqttDecoder extends ReplayingDecoder, if the malicious properties section was sent in chunks, the decoder would enter a loop, repeatedly re-parsing the buffered data until the complete section was received, causing high CPU utilization.
The patch addresses this in two key places:
- In the
decode method, the logic is changed to catch Signal exceptions thrown by the ReplayingDecoder. If a signal is caught (indicating more data is needed) and the amount of data already buffered exceeds maxBytesInMessage, the decoder now immediately throws a TooLongFrameException instead of continuing to buffer and parse.
- In the
decodeProperties method, a new check is added to verify that the buffer contains the complete properties block (as indicated by its length field) before attempting to parse it. If the data is incomplete, it triggers a replay signal efficiently, preventing the CPU-intensive loop of re-parsing the incomplete data.
Therefore, the key functions that would appear in a runtime profile during exploitation are io.netty.handler.codec.mqtt.MqttDecoder.decode and io.netty.handler.codec.mqtt.MqttDecoder.decodeProperties. These functions are directly involved in processing the malicious input and were modified in the security patch to enforce resource limits correctly.