The vulnerability, identified as GHSA-w567-gjr2-hm5j, describes an uncontrolled resource consumption issue in the MessagePack-CSharp library, specifically within the Unity blit formatters. The advisory states that UnsafeBlitFormatterBase<T>.Deserialize reads a byteLength from the input payload and allocates an array of that size without first validating it against the actual data available in the stream. This allows a malicious actor to trigger a large memory allocation with a small payload, leading to a denial of service.
To pinpoint the vulnerable function, I analyzed the commits between the last vulnerable version (3.1.6) and the first patched version (3.1.7). The commit 0555f07cbf217773070a150c97cbbd2b3a37ac8d with the message "Validate Unity blit lengths for CWE-789" directly addresses this issue.
The patch modifies the Deserialize method in src/MessagePack.UnityClient/Assets/Scripts/MessagePack/Extension/UnsafeBlitFormatter.cs. Before the patch, the code read byteLength directly from the reader and immediately used it for allocation: var result = new T[byteLength / sizeof(T)];. The fix introduces a crucial validation step. It now reads the extension data into a bounded reader and checks if the provided byteLength is negative, doesn't align with the element size, or exceeds the actual number of remaining bytes in the extension payload. The allocation is only performed after these checks pass.
Therefore, the function MessagePack.Formatters.UnsafeBlitFormatterBase<T>.Deserialize is the exact location of the vulnerability. During exploitation, a runtime profiler would capture this function call as it processes the malicious message pack data, attempts the large allocation, and subsequently throws an exception.