5.7

CVSS Score

3.1

Basic Information

CVE ID

CVE-2021-41125

GHSA ID

GHSA-jwqp-28gf-p498

EPSS Score

0.61032%

CWE

CWE-200

Published

10/6/2021

Updated

10/26/2024

KEV Status

Technology

Python

Concerned about an active attack path?

Talk to our security experts and see Miggo in action.

Miggo Vulnerability Database

→

CVE-2021-41125

CVE-2021-41125:
Scrapy HTTP authentication credentials potentially leaked to target websites

Impact

If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target.

This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXT_OBEY setting is set to True, or as requests reached through redirects.

Patches

Upgrade to Scrapy 2.5.1 and use the new http_auth_domain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials.

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead.

Workarounds

If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.

For more information

If you have any questions or comments about this advisory:

(GitHub Advisory)

5.7

CVSS Score

3.1

Basic Information

CVE ID

CVE-2021-41125

GHSA ID

GHSA-jwqp-28gf-p498

EPSS Score

0.61032%

CWE

CWE-200

Published

10/6/2021

Updated

10/26/2024

KEV Status

Technology

Python

Technical Details

CVSS Vector

CVSS:3.1/AV:N/AC:L/PR:L/UI:R/S:U/C:H/I:N/A:N

Package Name	Ecosystem	Vulnerable Versions	First Patched Version
Scrapy	pip	< 1.8.1	1.8.1
Scrapy	pip	>= 2.0.0, < 2.5.1	2.5.1

Vulnerability Intelligence
Miggo AI

Root Cause Analysis

The vulnerability stems from HttpAuthMiddleware's unconditional credential handling. The pre-patch code in httpauth.py's process_request method added Basic Auth headers to every request (if http_user/http_pass were set), regardless of target domain. The commit b01d69a introduced domain validation checks (urlparse_cached + url_is_from_any_domain) to restrict credential exposure. The vulnerable function is clearly the process_request implementation that lacked these domain checks, leading to credential leakage across domains.

Vulnerable functions

Only Mi**o us*rs **n s** t*is s**tion

WAF Protection Rules

WAF Rule

### Imp**t I* you us* [`*ttp*ut*Mi**l*w*r*`](*ttp://*o*.s*r*py.or*/*n/l*t*st/topi*s/*ownlo***r-mi**l*w*r*.*tml#mo*ul*-s*r*py.*ownlo***rmi**l*w*r*s.*ttp*ut*) (i.*. t** `*ttp_us*r` *n* `*ttp_p*ss` spi**r *ttri*ut*s) *or *TTP *ut**nti**tion, *ll r*qu*s

Reasoning

T** vuln*r**ility st*ms *rom *ttp*ut*Mi**l*w*r*'s un*on*ition*l *r***nti*l **n*lin*. T** pr*-p*t** *o** in *ttp*ut*.py's pro**ss_r*qu*st m*t*o* ***** **si* *ut* *****rs to *v*ry r*qu*st (i* *ttp_us*r/*ttp_p*ss w*r* s*t), r***r*l*ss o* t*r**t *om*in.

5.7

Basic Information

Concerned about an active attack path?

CVE-2021-41125: Scrapy HTTP authentication credentials potentially leaked to target websites

Impact

Patches

Workarounds

For more information

5.7

Basic Information

Technical Details

Vulnerability IntelligenceMiggo AI

Root Cause Analysis

Vulnerable functions

WAF Protection Rules

WAF Rule

Reasoning

CVE-2021-41125:
Scrapy HTTP authentication credentials potentially leaked to target websites

Vulnerability Intelligence
Miggo AI