The vulnerability lies in the FitsGzipDecoder.decode function within the src/PIL/FitsImagePlugin.py file of the Pillow library. The function was responsible for decompressing GZIP-compressed data from FITS image files. The vulnerability existed because the code read the entire compressed data stream into memory before decompression, without any limits. This made it susceptible to a decompression bomb attack, where a small, heavily compressed file could expand to an enormous size, leading to excessive memory consumption and a denial-of-service (DoS) condition.
The patch addresses this issue by changing the way the data is read and decompressed. Instead of reading the entire file at once, the patched code uses gzip.open to handle the compressed data as a stream. It then reads only the necessary amount of data based on the image's dimensions (self.state.xsize * self.state.ysize * 4). This ensures that the amount of memory allocated is bounded by the expected size of the image, effectively neutralizing the decompression bomb threat. The identified vulnerable function, FitsGzipDecoder.decode, is the exact location where this flawed logic existed and was subsequently fixed.