The vulnerability is an integer overflow in the rust-yamux library, which can be triggered by a remote, unauthenticated attacker to cause a denial of service. The root cause lies in the FlowController.increase_send_window_by function, located in yamux/src/connection/stream/flow_control.rs. In the vulnerable versions, this function used checked_add with .expect(), which would cause a panic if the addition of credit to the stream's send window resulted in an overflow.
The attack is initiated when a malicious WindowUpdate frame is sent by a peer. This frame is processed by the Active.process_frame function in yamux/src/connection.rs. This function extracts the credit value and, through a call to Shared.increase_send_window_by, triggers the vulnerable logic in FlowController.increase_send_window_by. The resulting unhandled panic crashes the connection's state machine.
The patch addresses this by changing the function signatures to return a Result<(), ConnectionError> instead of panicking. The checked_add is replaced with .ok_or(ConnectionError::InvalidWindowUpdate), and the calling function Active.process_frame is updated to handle this error gracefully by terminating the connection with a protocol error instead of crashing.