The vulnerability lies in io.netty.handler.codec.redis.RedisArrayAggregator, which is susceptible to a Denial of Service attack due to memory exhaustion. An attacker can achieve this by sending a crafted Redis payload containing deeply nested arrays. The root cause is the absence of a depth limit for these nested arrays.
Analysis of the provided patch commits confirms this. Specifically, commit 728c98b8ec6597faaccebdc20bf757d6f4b6310b introduces a configurable limit, maxNestedArrayDepth, to control how many nested arrays can be processed. The core of the fix is in the decodeRedisArrayHeader method (which is called by the main decode method), where a check is added to ensure the current nesting depth does not exceed this new limit. If the limit is exceeded, a CodecException is thrown, preventing further processing and the associated memory allocation.
The primary vulnerable function is io.netty.handler.codec.redis.RedisArrayAggregator.decode, as it is the entry point in the channel pipeline for processing the malicious Redis messages. During an exploit, this function would be on the call stack as it attempts to decode the unending stream of nested array headers, leading to the allocation of AggregateState objects until memory is exhausted. The constructors were also modified to support this new security feature.