A Proxy for One Fetch in Scrapy Shell

Use a Request’s proxy metadata inside Scrapy shell and inspect the response from that configured request.

Inside Scrapy shell, construct the request you want the downloader to make:

from scrapy import Request

fetch(Request(
    "https://www.example.com/",
    meta={"proxy": "http://proxy.example:8080"},
))
response.status

HttpProxyMiddleware reads the proxy value from that Request's metadata. It must be enabled in the active downloader middleware configuration. The response variable afterward belongs to this fetch. For this public URL, select an unauthenticated HTTP address from the HTTP proxy list if public intermediaries are permitted, then place its URL in that request’s proxy metadata.

Starting the shell from the Scrapy project lets it load that project's configuration. That selects middleware and other settings; it does not itself choose a particular proxy endpoint. The request metadata above supplies that choice directly.

For a proxy using Basic credentials, Scrapy documents user information in the proxy URL. Encode reserved characters in the account components before constructing it. There is no need to add a website Authorization header for that account.

Scrapy can also discover proxies from its supported environment variables. A request's explicit proxy metadata takes precedence. For an intentionally direct request, a proxy value of None disables that middleware's environment-based selection for the request.

Allakkat attuumassuteqartut