Does a Forward Proxy Enforce robots.txt for a Crawler?
Keep robots policy in the crawler and understand the limits of enforcing it at a forwarding proxy.
Giving a crawler an HTTP proxy changes how its requests reach the website. It does not make the proxy fetch robots.txt, choose the crawler's rule group or schedule its requests politely.
The Robots Exclusion Protocol describes rules for crawlers. It includes matching, retrieval failures, redirects and caching behavior. A proxy that simply relays the requested document has not implemented that policy on the crawler's behalf.
Keep the rules decision close to the crawler's request queue. That is where the intended identity, target URI and scheduling state are available before work is sent.
Why a proxy-side filter is harder than it sounds
For plain HTTP, a specially built filter may see the target path and User-Agent. It still needs a real rules implementation and a defined policy for fetching, caching and updating the applicable robots file. Converting a few Disallow lines into static URL blocks is not the complete protocol.
For HTTPS through ordinary CONNECT, the forward proxy sees the tunnel destination rather than each encrypted page path and inner User-Agent. It lacks information needed for path-level robots decisions unless another deliberate arrangement supplies it.
Changing the exit IP also does not turn a disallowed path into an allowed one. Coordinate pacing across workers that share the crawler's activity; rotating routes is not a substitute for that policy.
Finally, robots rules are not access authorization. A permitted path does not override authentication or the site's other access requirements, and a public robots file does not secure private content.