Scrapy Can Show the Redirect Before Following It

Temporarily expose a proxied GET's initial redirect status and Location through request metadata, keeping redirect handling separate from HTTP-error filtering.

A proxied GET can reach a server and then follow redirects until the eventual failure obscures the first response. Expose that initial response for diagnosis:

request = scrapy.Request(
    'https://service.example/page',
    callback=inspect_redirect,
    meta={
        'proxy': 'http://proxy.example:8080',
        'dont_redirect': True,
        'handle_httpstatus_list': [301, 302, 303, 307, 308],
    },
)

Schedule this request through the spider's existing start or callback flow. For an allowed public-page probe, the HTTP proxy list supplies candidates for the request’s proxy URL. inspect_redirect should record the response status and Location header instead of blindly following it again.

dont_redirect stops automatic redirection for this request. handle_httpstatus_list lets the named statuses reach the callback instead of being rejected by HTTP-error filtering. The settings address different middleware responsibilities.

Check what the redirect selects: an intended canonical URL, a different scheme or hostname, a login page, or a repeating cycle. A 302 received after successful transport is a different observation from failing to open the proxy connection.

When normal redirect handling is restored, Scrapy exposes redirect_urls and redirect_reasons in response request metadata. Those can preserve the path leading to a final result without leaving redirects disabled for the whole crawl.

A terminal response is not automatically the page the application intended. Keep the destination and response content with the routing diagnosis.

Artiglyn mynjee