The vulnerability lies in the cometd-java-server-common package, specifically within the acknowledgement extension. A malicious client can cause the server's unacknowledged message queue to grow without bounds, leading to an OutOfMemoryError. This is achieved by the client not acknowledging received messages, or by repeatedly sending a low, fixed acknowledgement batch number.
The investigation of the provided patches, such as commit fd3621c901092cda90f2826c35eae3a6114df674, reveals that the core of the issue is in the AcknowledgedMessagesSessionExtension class. This class implements the ServerSession.QueueListener interface, and its queued(ServerSession sender, ServerMessage message) method is invoked whenever a message is added to the session's queue.
Prior to the patch, the queued method would add messages to its internal queue (_queue) without any validation of the queue's size. The patch introduces a size check within this method, adding a configurable maxQueueSize and a mechanism to notify listeners when this limit is exceeded. The absence of this check is the root cause of the vulnerability.
Therefore, the function org.cometd.server.ext.AcknowledgedMessagesSessionExtension.queued is the primary vulnerable function. During an attack, this function would be repeatedly called as messages are sent to the malicious client's session, and its execution would be a clear indicator in a runtime profile of the resource consumption leading to the server outage.