The root cause of the vulnerability is improper input validation for hostnames, leading to a Server-Side Request Forgery (SSRF) bypass. The primary vulnerable function, _host_is_blocked in spider_tools.py, failed to resolve hostnames before checking them against a blocklist of IP addresses. It used socket.inet_aton, which does not handle domain names. This allowed an attacker to use a specially crafted hostname (e.g., a nip.io domain) that would fail the initial check but later resolve to an internal IP address like 127.0.0.1.
A similar flaw existed in the web_crawl function, which had its own vulnerable implementation of URL validation. The patch addresses these issues by replacing the flawed validation logic with socket.getaddrinfo to correctly resolve hostnames to all their associated IP addresses and then checking each IP against a list of restricted ranges (private, loopback, etc.). This ensures that even if a hostname resolves to multiple IPs, the check will fail if any of them are internal.