Scrapy Proxy Concurrency Across Multiple Domains
Use the downloader's global concurrency cap when several target domains consume the same proxy allowance, then account for other crawlers sharing that account.
If every destination uses one proxy account, that account's simultaneous-request allowance can be consumed across all those destinations. A per-domain limit alone does not express that shared budget.
For a hypothetical account allowing at most 20 simultaneous requests, reserve some room and set the crawler's global downloader cap:
CONCURRENT_REQUESTS = 16This is an illustrative allocation, not a recommended value for every provider. Scrapy defines CONCURRENT_REQUESTS as the downloader's maximum simultaneous request count. Per-domain controls can further reduce concurrency; they do not replace the global ceiling.
Include other crawlers in the arithmetic. Two independent crawlers each allowed 16 requests can jointly exceed a shared allowance of 20. Divide the budget deliberately or enforce a shared limit where those clients converge.
Also establish what the provider counts. HTTP requests, open TCP connections, active tunnels and requests per second are different quantities. Persistent connections and multiplexing can make their counts diverge. A request-concurrency setting is therefore not a complete connection or rate limiter.
Keep target-site pacing as a separate constraint. Lower concurrency or a download delay may still be appropriate for an individual destination even when the proxy account has unused capacity.