dont_proxy Does Not Clear Scrapy’s Proxy Field

A request can opt out of a custom proxy middleware and still carry an active proxy URL in its metadata. Scrapy’s built-in HttpProxyMiddleware reads meta["proxy"]; it does not interpret a generic dont_proxy flag.

For a request intended to go direct, explicitly set:

request.meta["proxy"] = None

The explicit None prevents the built-in middleware from selecting a proxy from its environment settings for that request. Simply deleting the key permits that automatic selection again.

If another installed middleware uses dont_proxy, keep that flag for its documented purpose too. Arrange the middleware sequence so a later component does not assign a proxy URL again. Inspect the metadata after the last component that can select a route.

A 407 after setting a custom bypass flag is consistent with a remaining proxy route, but the status alone does not identify which middleware selected it. Follow the effective proxy field through that request’s processing rather than assuming every component shares the same opt-out vocabulary.