Share a Proxy Pool Between Python Threads with a Queue

When several threads share a small proxy pool, selecting a random entry can give two active operations the same proxy. A queue can hold the entries currently available to borrow:

from contextlib import contextmanager
from queue import Queue

available = Queue()
for endpoint in dict.fromkeys(proxy_urls):
    available.put(endpoint)

@contextmanager
def borrow_proxy():
    endpoint = available.get(timeout=5)
    try:
        yield endpoint
    finally:
        available.put(endpoint)

def process_url(url):
    with borrow_proxy() as endpoint:
        return fetch_one(url, endpoint)

Build the queue once, before starting the worker threads. proxy_urls contains complete endpoint URLs. Public candidates are available in the free proxy list; use the protocol and URL format supported by fetch_one. The existing synchronous fetch_one function must finish its network operation and close any response before returning. Otherwise the lease would end while the connection still uses that proxy.

Queue.get atomically removes one available entry. An exhausted pool waits rather than repeatedly scanning shared flags. If no entry arrives within five seconds, queue.Empty reaches the caller; choose the application’s retry, failure or cancellation policy for that result.

The finally block returns an acquired entry after success or an exception. It does not declare that a failed endpoint is healthy. Quarantine, replacement and cooldown policies need an explicit path rather than silently discarding every proxy that sees a temporary error.

Deduplicating identical URL strings prevents duplicate queue entries. Distinct URLs can still refer to the same physical proxy or exit; group those deliberately if exclusivity must apply to that shared resource. The queue coordinates threads in this process, not workers on other machines.