Scrapy AutoThrottle Through a Proxy: Slots and Latency
Keep target-site download slots distinct from proxy routes and interpret latency as the observed request path.
By default, Scrapy assigns download slots from the request URL’s domain. Giving a request an HTTP proxy does not change that URL into the proxy’s hostname. Requests to different sites can therefore retain different slots while sharing one intermediary.
AutoThrottle adjusts each slot’s delay using observed download latency. The standard per-domain concurrency limit and minimum download delay still apply.
Check grouping before changing the algorithm
A custom middleware can assign request.meta["download_slot"], overriding normal slot selection. Some proxy extensions use their own grouping. Inspect those settings if unrelated sites appear to share one throttle.
Conversely, assigning a different slot for every proxy can let multiple proxies contact the same target concurrently. That may defeat a site-wide limit the crawler was meant to respect. Choose the slot key according to the resource being protected.
The delay includes the path you used
Scrapy measures latency from sending the request to receiving HTTP headers. A request carried through a proxy can include the intermediary’s queueing and forwarding delay in that observation. AutoThrottle does not obtain a separate origin-only processing time.
Increasing concurrency cannot make that measurement isolate the website. Keep target politeness and proxy capacity as separate limits when both matter. A shared proxy may need an additional concurrency budget across target slots, while each target still keeps its own rate policy.