The analysis began by examining the provided GitHub Security Advisory (GHSA-36wm-hprc-mcf5), which describes a heap buffer overflow in the ImageMagick IPL decoder. The advisory specified the patched versions, allowing for a targeted analysis of the code changes. By comparing the git tags for the last vulnerable version (7.1.2-22) and the first patched version (7.1.2-23), a specific commit was identified with the message directly referencing the GHSA ID: 3aa35741316909f9e384d13cee197334dc3296d7.
This commit modifies the file coders/ipl.c. Although the advisory refers to a vulnerability in the "decoder", the patch is located within the WriteIPLImage function, which is the IPL format encoder. The vulnerability is triggered when processing a sequence of images with varying dimensions. The function iterates through a linked list of images but failed to re-initialize the pixel handling structures for each image. If an image in the sequence was larger than its predecessor, the previously allocated buffer would be too small, leading to an out-of-bounds write.
The identified patch addresses this by inserting a call to SetQuantumDepth() at the beginning of the loop for each image. This ensures that the quantum_info structure is correctly configured for the current image's properties before its pixels are processed, thereby preventing the buffer overflow. The vulnerable function is therefore WriteIPLImage, which would be invoked when an application attempts to create a multi-frame IPL image from source images of different sizes.