Load Scrapy Proxy Settings into a CrawlerProcess Script

A spider can use the expected proxy under scrapy crawl and go directly when launched by a Python script. Check how that script constructs its settings before changing the proxy server.

For an existing Scrapy project and its registered spider:

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

settings = get_project_settings()
process = CrawlerProcess(settings=settings)
process.crawl('catalog')
process.start()

Replace catalog with the project's spider name. The project must be discoverable through its scrapy.cfg, or through a correctly set SCRAPY_SETTINGS_MODULE pointing to an importable settings module.

Constructing CrawlerProcess() alone does not import the project's settings file merely because the script also imports a spider class. A custom middleware enabled by DOWNLOADER_MIDDLEWARES can consequently be absent from that run.

Loading the settings supplies the configured components; it does not invent a proxy address. Scrapy's HTTP proxy middleware still needs its normal environment proxy or an explicit request.meta['proxy'] value. Check that HTTPPROXY_ENABLED has not disabled it. For a permitted public-page crawl, the HTTP proxy list provides candidates for an HTTP proxy value supplied through that metadata or environment.

Compare the launcher's loaded settings and active middleware with the command-line run. If they match, continue with the selected request's proxy metadata and the actual transport error.