Read Scrapy's Proxy Endpoint and Connection IP
Use request metadata for the selected proxy, treat response.ip_address as handler-supplied peer information, and verify exit IP at the destination.
Use the Request attached to the completed response when inspecting its selected route. Keep three observations distinct:
| Field | Meaning |
|---|---|
response.request.meta.get('proxy') | This request's selected proxy URL, when present |
response.ip_address | Download-handler-supplied address, possibly None |
| A trusted destination's source report | The public source observed there |
Inside a callback, omit credential-bearing URL parts from routine diagnostics:
from urllib.parse import urlsplit
proxy = response.request.meta.get('proxy')
proxy_host = urlsplit(proxy).hostname if proxy else None
self.logger.info(
'target_host=%s proxy_host=%s peer_ip=%s',
urlsplit(response.url).hostname, proxy_host, response.ip_address,
)In Scrapy 2.13.3's HTTP/1.1 handler, the address comes from the response connection peer, not an exit-IP service. Some paths, including zero-length responses, can leave it None. Other handlers can populate it differently. None does not by itself prove direct routing.
For an exit check, query a trusted endpoint under the intended proxy policy and keep its request identity with the observation. Rotating middleware may give a later check another endpoint.
Redirects create another request too. Final metadata describes that stage; inspect redirect history and any middleware reselection instead of treating it as proof of the first URL's route.