The vulnerability lies in the fact that psd-tools allocated memory for images based on dimensions declared in the PSD file's header without validating these dimensions against the actual file content. An attacker could provide a very small PSD file that declares huge dimensions, causing the application to attempt to allocate an extremely large amount of memory, leading to a denial of service (DoS) through an out-of-memory (OOM) error.
The analysis of the patch commit a3d9a53ad51e667b5772a4f636ca6f2e16f4b271 reveals the exact locations of the vulnerability. The functions get_image_data (for .numpy() calls) and convert_image_data_to_pil (for .composite() calls) were modified to pass the number of channels to the check_pixel_size function, which itself was enhanced to calculate a potential memory allocation size and check it against a budget. This indicates that the original functions were not performing adequate checks.
Furthermore, the decompress function in psd_tools.compression was patched to prevent a fallback to allocating a large black image when decompression fails. In the vulnerable version, if the compressed data was invalid, the code would create a black image based on the header's dimensions, which is the core of the DoS vector. The patch introduces a check to ensure the allocated size is not disproportionately larger than the input data size.
Therefore, the vulnerable functions are the ones that either directly handle the memory allocation based on unchecked dimensions (get_image_data, convert_image_data_to_pil) or have a fallback mechanism that leads to excessive allocation (decompress). These functions would appear in a runtime profile when the vulnerability is triggered.