The vulnerability is a classic stack-based buffer overflow due to uncontrolled recursion in the DestroyXMLTree function in ImageMagick's MagickCore/xml-tree.c. When parsing an XML file, ImageMagick builds a tree structure in memory. To free this memory, the DestroyXMLTree function is called, which recursively traverses the tree to destroy each node.
The vulnerability lies in the fact that prior to the patch, there was no limit on the recursion depth. An attacker could craft an XML file with a very deep nesting of tags. When ImageMagick attempts to destroy the in-memory representation of this XML, the DestroyXMLTree function calls itself for each level of nesting via the helper functions DestroyXMLTreeChild and DestroyXMLTreeOrdered. This leads to an excessive number of nested function calls, eventually exhausting the stack space allocated to the program, causing a crash and a Denial of Service (DoS).
The patch mitigates this by introducing a recursion depth counter. A new static function, DestroyXMLTree_, is created which accepts a depth parameter. The original DestroyXMLTree function now simply calls DestroyXMLTree_ with an initial depth of 0. Inside DestroyXMLTree_, the depth is checked against a maximum allowed recursion depth (MagickMaxRecursionDepth). If the depth exceeds this limit, a fatal exception is thrown, preventing the stack overflow. The recursive calls within DestroyXMLTreeChild and DestroyXMLTreeOrdered are updated to call the new DestroyXMLTree_ function, incrementing the depth counter with each call.