Retry a Scrapy GET Through Another Proxy

Changing response.request.meta changes a Python object. To perform another download, give Scrapy another Request to schedule.

For a GET that your callback explicitly decides to retry, Scrapy's public retry helper can keep the retry accounting with the new request:

from scrapy.downloadermiddlewares.retry import get_retry_request

def retry_through(self, response, proxy_url):
    retry = get_retry_request(
        response.request,
        spider=self,
        reason='choose_another_proxy',
    )
    if retry is None:
        return None
    retry.meta['proxy'] = proxy_url
    retry.headers.pop('Proxy-Authorization', None)
    return retry

This is a spider helper. Its caller must return or yield the resulting Request when one is available. None means the helper declined another retry under the applicable limit.

The retry helper copies the request, updates its retry count and supplies the scheduler behavior appropriate for a retry. Merely yielding the same URL as an ordinary new request can instead encounter duplicate filtering.

Use a proxy URL supported by the enabled HttpProxyMiddleware. For retries of allowed public GET requests, an unauthenticated HTTP candidate from the HTTP proxy list can supply proxy_url. Clearing a previously attached proxy-authorization field avoids carrying an old intermediary's header into this selection; let the middleware derive the new account from the new proxy URL when applicable.

Keep this callback-level choice separate from automatic RetryMiddleware handling of its configured failures. Decide which layer owns the retry so the operation does not consume two independent budgets. The example concerns replayable GET work, not a promise that repeating an arbitrary submitted transaction is safe.