Assign a URL and Proxy to Each Python ThreadPoolExecutor Job

Assign a proxy before submitting a Python request job instead of letting workers race over shared mutable selection state.

A pool’s map passes one item to the worker at a time. Make that item a complete job, including the URL and selected proxy, so the worker does not infer its route from a changing global list.

For finite urls, supply a nonempty list of validated proxy URLs. Public HTTP candidates are available in the HTTP proxy list:

from concurrent.futures import ThreadPoolExecutor
from itertools import cycle
import requests

def fetch(job):
    url, proxy_url = job
    proxies = {"http": proxy_url, "https": proxy_url}
    with requests.get(url, proxies=proxies, timeout=(5, 15)) as response:
        response.raise_for_status()
        return url, response.text

if not proxy_urls:
    raise ValueError("No proxy endpoints were supplied")

jobs = zip(urls, cycle(proxy_urls))
with ThreadPoolExecutor(max_workers=5) as executor:
    for url, body in executor.map(fetch, jobs):
        handle_result(url, body)

urls, proxy_urls and handle_result are supplied by the application. The coordinator assigns endpoints in a repeating order; each worker receives one fixed pair. Iterating the results also surfaces worker exceptions.

Five workers limit active jobs, not active jobs per proxy. If the list has two endpoints, several jobs can use the same endpoint concurrently. Add a separate per-endpoint capacity rule when that is required. For one active job per endpoint URL, a synchronized queue lease can reserve and return each entry.

A retry should create a new bounded attempt for the same URL. Avoid recursive retry calls that remove entries from a shared list: another worker may be selecting from it, and an exhausted list still needs an explicit outcome.

Allakkat attuumassuteqartut