The vulnerability (CVE-2017-0641) is a denial of service in libvpx caused by specially crafted files, leading to a device hang or reboot. This typically implies resource exhaustion, most likely memory, due to processing frames with excessively large dimensions. The provided patch (commit 698796fc930baecf5c3fdebef17e73d5d9a58bcb) addresses this by introducing configuration macros DECODE_WIDTH_LIMIT and DECODE_HEIGHT_LIMIT (set to 4096) via the generate_config.sh script. These macros are then compiled into the library for various architectures.
The vulnerable functions are those within libvpx that are responsible for allocating memory based on frame dimensions, which, prior to this patch, would not have been constrained by these specific 4K limits. When processing a malicious file with overly large declared dimensions, these functions would attempt to allocate an unmanageable amount of memory.
vpx_img_alloc: This is a general utility function in libvpx for allocating memory for vpx_image_t structures. Its memory consumption is directly proportional to the image width and height. Without enforced limits on these dimensions (as introduced by the patch), it's a prime candidate for causing DoS.vp9_alloc_frame_buffers (and its VP8 equivalent vp8_alloc_frame_buffers): These codec-specific functions allocate sets of frame buffers needed for the decoding process. The number and size of these buffers are also derived from the frame dimensions. Lack of dimension capping would make them vulnerable to excessive allocation requests.The patch itself doesn't modify these C functions directly but provides the limiting mechanism (the macros). The vulnerability existed in these (and potentially other) C functions because they operated without such limits. The functions listed would appear in a runtime profile during exploitation as they attempt to handle the large memory requests triggered by the malicious file's dimensions.