A Blocking Proxy-Pool Lookup Can Stall Scrapy
Keep a synchronous proxy-pool API call out of Scrapy’s event-loop path and give endpoint discovery its own failure policy.
The crawler has plenty of concurrent requests configured. It still pauses whenever it asks the proxy pool for another address. Adding more proxies does not touch that pause.
A common middleware pattern calls requests.get(pool_url) inside process_request(). That is a synchronous network operation in a path expected to cooperate with Scrapy's asynchronous engine. While the call waits, it can hold up unrelated work on the same event-loop thread.
There are two requests in the job
One request obtains an endpoint from the pool service. The other uses that endpoint to contact the destination. Measure and report them separately. A slow pool API can make every selected proxy look slow before the application has even connected to it.
Use an asynchronous discovery integration supported by the installed Scrapy version, or obtain and refresh a bounded local endpoint set outside the per-request critical path. If blocking work must be offloaded, give that executor its own limits and timeout handling; an unlimited queue merely relocates the delay.
Do not issue another discovery call for every request automatically. A short-lived cache or a leased batch can reduce dependency on the pool service, provided its lifetime fits the service's actual endpoint policy.
Define what a failed lookup means
An empty or failed pool response should not silently remove the proxy metadata and allow a direct request. Choose an explicit outcome: wait within the task budget, fail, or use an approved cached endpoint.
Keep the pool API's own route clear too. If it is an internal service, sending its private hostname through the very proxy it helps select can create a circular dependency.
Once discovery works without blocking unrelated work, inspect the ordinary proxy transaction. Those are separate repairs, and their timing deserves separate fields in the log.