The vulnerability originates from an integer overflow in the C implementation of the ultrajson library, triggered when a large or negative indent value is passed to functions like ujson.dumps(). The analysis of the patch commit 486bd4553dc471a1de11613bc7347a6b318e37ea reveals the root cause and the affected functions.
The execution flow of the vulnerability starts in the objToJSON function in src/ujson/python/objToJSON.c, which is the C-level entry point for the Python API. This function accepted the indent parameter without validation. The value was then used in the encode function (src/ujson/lib/ultrajsonenc.c) to calculate the necessary buffer size for pretty-printing the JSON. The calculation enc->indent * (enc->level + 1) was susceptible to an integer overflow if indent was a large number. This overflow resulted in Buffer_Reserve allocating a much smaller buffer than needed. Subsequently, when Buffer_AppendIndentUnchecked was called to write the indentation, it would write past the allocated buffer's boundary, causing a buffer overflow and a segmentation fault. A large negative indent could cause an underflow, leading to an infinite loop in the buffer resizing logic.
The patch addresses this by adding validation in objToJSON to cap the indent value at 1000 and by changing the integer types used in the calculation within encode from int to the larger ptrdiff_t to prevent the overflow.