Your Proxy File Is a Pool, Not a Requests Routing Table

Convert an authenticated proxy-list entry into a selected endpoint without using proxy addresses as destination-routing keys.

Select one proxy from the file, then build the route map for the website request. A pool of candidate endpoints is not itself a Requests routing dictionary.

  1. Parse the entry according to a defined format. An informal IP:port:user:password line is ambiguous when IPv6 addresses or credentials contain colons. For an IPv4 host, numeric port and colon-free username, a limited split can leave the remaining text as the password. Use separate structured fields when those restrictions do not fit.
  2. Choose the endpoint for this operation. Retain its actual protocol and encode the credential components when constructing its URL. Keep that secret-bearing value out of routine logs.
  3. Map destination schemes to that selected URL. For one selected HTTP proxy, the map can be:
routes = {
    'http': selected_proxy_url,
    'https': selected_proxy_url,
}

The keys http and https select destination schemes. A key such as http://203.0.113.10 instead selects requests to that exact destination host; it does not label the proxy at that address. Using every proxy IP as a key can leave an unrelated website without the intended route.

Parse the pool once, select an endpoint, and create the mapping for the client call. Keeping those steps distinct lets you change the selection policy without changing what Requests' routing keys mean.

Artiglyn mynjee