The vulnerability is an infinite loop denial of service in OpenMcdf when parsing a crafted Compound File Binary (CFB) file. The root cause is the lack of cycle detection when traversing the directory entry tree, which is structured as a red-black tree. An attacker can create a file where directory entries' sibling pointers form a cycle (e.g., entry A points to B, and B points back to A).
When OpenMcdf processes this file, two code paths are affected:
-
When Storage.EnumerateEntries() is called, it uses DirectoryTreeEnumerator. The MoveNext() method of the enumerator would traverse the cycle indefinitely, never returning false. This would cause any loop iterating over the entries (like a foreach) to hang, consuming CPU and memory.
-
When Storage.OpenStream() is called, it internally calls DirectoryTree.TryGetDirectoryEntry to locate the stream's directory entry. This function also traverses the directory tree and, without cycle detection, would loop forever when encountering a crafted cycle.
The provided patch for commit 24f445a557fc4f46461cf6d02d296cce16c293a0 shows the fix applied to DirectoryTreeEnumerator.MoveNext() by implementing Brent's cycle detection algorithm. This prevents the infinite loop when enumerating entries. While the patch for DirectoryTree.TryGetDirectoryEntry was not directly provided in the commit details, the vulnerability description explicitly identifies it as a vulnerable function. The fix for both is the introduction of cycle detection in the tree traversal logic.