Two Settings Can Disable Scrapy’s Proxy Middleware

Check the merged downloader middleware list and HTTPPROXY_ENABLED before adding another copy of Scrapy’s built-in proxy middleware.

HttpProxyMiddleware belongs to Scrapy’s built-in downloader defaults. It need not appear in the project’s own DOWNLOADER_MIDDLEWARES dictionary to be available.

There are two separate controls to inspect:

  • HTTPPROXY_ENABLED controls whether the built-in component is enabled.
  • A None value for its class path in DOWNLOADER_MIDDLEWARES removes it from the merged middleware chain.

For example, this explicitly disables the default entry:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None,
}

Remove that override when the built-in component should return. Do not copy an old DOWNLOADER_MIDDLEWARES_BASE table into the project merely to restore one entry; Scrapy combines its defaults with project overrides.

An enabled middleware still needs a route, from supported environment values or the request’s proxy metadata. Its presence in a startup log is not itself a request assignment.

Very old Scrapy releases differed in when the component loaded without proxy environment values. Keep the installed version with the diagnosis, rather than treating a historical 1.3-era observation as the current enablement rule.

Artiglyn mynjee