Per-Request Proxies with Scrapy's HTTP Proxy Middleware
Use Scrapy request metadata and its proxy middleware before copying an old custom authentication middleware into a crawler.
Scrapy's HTTP proxy middleware reads a proxy from request metadata:
yield scrapy.Request(
'https://example.com/',
callback=self.parse,
meta={'proxy': 'http://203.0.113.10:8080'},
)Replace the documentation endpoint and keep the middleware enabled. The HTTP proxy list offers public candidates for this request URL. This selects the request's route; a proxy pool needs separate selection logic.
Authentication follows the selected endpoint
The built-in middleware handles credentials supplied through its supported proxy URL and the associated authorization header. Escape reserved URL characters and avoid logging that URL.
A separate middleware should not retain a Basic header for the previous proxy after the endpoint changes. Let one layer manage the route and its credentials together.
Selection order and defaults
If custom downloader middleware chooses the proxy, run it before the built-in proxy middleware processes the request. Check the installed release's ordering and API; obsolete scrapy.contrib imports belong to older releases.
The process environment can also supply scheme-specific defaults. Use the documented special proxy metadata behavior for a deliberately direct request: an absent key does not necessarily mean direct access.
Inspect the actual outcome. A 407 concerns proxy authentication, while a website rate limit or access denial can arrive after the route worked correctly.