The vulnerability is a denial-of-service in Netty's native epoll transport, caused by the improper handling of TCP connections that are half-closed and then receive a RST packet. This leads to stale channels that are never cleaned up and can cause a 100% CPU busy-loop in the event loop thread.
The root cause of the vulnerability is a flaw in the management of file descriptors within the epoll set. When a connection is half-closed, the interest in read events might be removed. If a RST is received subsequently, an error event is generated. The vulnerable code failed to unregister the file descriptor from epoll when its interest set became empty. This meant that the event loop would be constantly notified of an error on the file descriptor but would not process it correctly, as it was short-circuited by checks for the half-closed state. This leads to a busy-loop on epoll_wait.
The fixing commit, 0ec3d97fab376e243d328ac95fbd288ba0f6e22d, directly addresses this issue by modifying the io.netty.channel.epoll.EpollIoHandler.submit function. This function is central to managing epoll registrations. The patch introduces logic to detect when the interest set for a file descriptor becomes empty (EpollIoOps.NONE). When this occurs, the function now explicitly calls Native.epollCtlDel to remove the file descriptor from the epoll instance. This prevents the file descriptor from getting into a state where it can cause an infinite loop.
Therefore, io.netty.channel.epoll.EpollIoHandler.submit is the key function where the vulnerability logic resided. During exploitation, this function would be called to update the channel's event registration, and its failure to correctly remove the file descriptor from epoll is what triggers the denial-of-service condition.