The vulnerability is a heap buffer overflow in the SMILES parser of Open Babel, which can be triggered by a specially crafted SMILES string. The analysis of the patch commit reveals that the vulnerability is addressed by adding several crucial checks in two functions within the src/formats/smilesformat.cpp file.
The primary function, OpenBabel::OBSmilesParser::ParseSmiles, which is the entry point for parsing, received a new check to validate the size of the _hcount vector against the number of atoms. This prevents a scenario where a malformed SMILES string could cause an inconsistency between the allocated memory and the actual number of atoms, leading to an overflow.
The helper function OpenBabel::OBSmilesParser::ParseNextBond was also patched to validate that characters for ring closures are indeed digits before they are processed. This prevents out-of-bounds reads by functions that expect numeric input. Additionally, a check for the null terminator was added to prevent reading beyond the end of the input string.
Both functions are directly involved in processing the untrusted SMILES string, and the lack of these checks in the vulnerable versions makes them susceptible to a heap buffer overflow. Therefore, both functions would appear in a runtime profile during the exploitation of this vulnerability.