Include and Load a Proxy List in a Deployed Scrapy Project

A proxy file beside a development checkout is not automatically present in the crawler’s deployed package. There are two jobs: include the bytes in the distribution, then find them in the installed package.

For a Setuptools project with crawl_app/resources/proxies.txt inside its Python package, a pyproject.toml entry can include it:

[tool.setuptools.package-data]
crawl_app = ["resources/*.txt"]

Merge that with the project’s existing package configuration. A different build backend has its own data-inclusion interface.

For middleware that can consume endpoint strings, Python 3.11+ can read the package resource directly:

from importlib.resources import files

text = files('crawl_app').joinpath('resources', 'proxies.txt').read_text(
    encoding='utf-8'
)
proxy_urls = [line.strip() for line in text.splitlines() if line.strip()]

Use the actual package name and validate the expected URL format in the reader. For distributable public endpoint data, use addresses from the free proxy list in the middleware’s expected protocol and URL format. The lookup does not depend on the deployment’s current working directory.

A third-party PROXY_LIST setting may require a filesystem path rather than these strings. When using importlib.resources.as_file for that interface, keep its context open for as long as the middleware needs the extracted file. Returning a temporary path after the context closes can lose the resource.

Keep credential-bearing production lists in the deployment’s intended secret or configuration storage. Package data is suitable for distributable endpoint data, not an assumption that everything bundled with code stays private.