Collect Each Result from Concurrent urllib3 Proxy Requests

Creating a worker pool does not give the caller the downloaded pages. Keep and consume the values returned by the submitted work:

from concurrent.futures import ThreadPoolExecutor
import urllib3

manager = urllib3.ProxyManager(
    'http://proxy.example:8080',
    maxsize=4,
    block=True,
)

def fetch(url):
    response = manager.request(
        'GET', url, timeout=urllib3.Timeout(connect=5, read=15),
    )
    return url, response.status, response.data

try:
    with ThreadPoolExecutor(max_workers=4) as workers:
        results = list(workers.map(fetch, urls))
finally:
    manager.clear()

This fragment assumes a bounded list of URLs and ordinary response sizes. For permitted public-URL checks without proxy authentication, a candidate from the HTTP proxy list can replace the manager’s example HTTP endpoint. Each result includes its URL so the caller can associate status and body with the correct request. map returns results in input order; an exception in a worker is raised when that result is retrieved.

The configured ProxyManager owns connection pooling for the requests. With the default preloaded response mode used here, urllib3 reads the body and returns the connection to its pool. A streaming version would need explicit consumption and connection release.

maxsize normally controls retained reusable connections, not a hard cap on every opened socket. block=True changes the relevant pool behavior so callers wait for a free connection rather than opening additional ones beyond that pool's limit.

The worker count bounds concurrent calls in this example. With many destinations, pool layout and the route used for HTTPS tunnels also matter; a per-pool setting should not be described as one global socket quota for every host.