Turn an HTTP Proxy URL into a Twisted Endpoint

A conventional proxy value such as http://proxy.example:8080 is a URL. Twisted’s clientFromString accepts an endpoint description, such as tcp:host=proxy.example:port=8080. The two strings describe related information using different grammars.

If the application accepts a conventional HTTP proxy URL, parse it before creating the endpoint:

import os
from urllib.parse import urlsplit
from twisted.internet.endpoints import HostnameEndpoint
from twisted.web.client import ProxyAgent

proxy = urlsplit(os.environ['http_proxy'])
if proxy.scheme != 'http' or not proxy.hostname:
    raise ValueError('Expected an HTTP proxy URL')
port = proxy.port if proxy.port is not None else 80
endpoint = HostnameEndpoint(reactor, proxy.hostname, port, timeout=5)
agent = ProxyAgent(endpoint, reactor=reactor)

The fragment assumes the application already has its reactor. Specify the real proxy port in the configuration when it is not the HTTP URL default. The HTTP proxy list offers endpoint candidates for an allowed public HTTP request with no proxy account. HostnameEndpoint also avoids embedding the hostname into another colon-delimited configuration string.

This converts the routing input only. Credentials embedded in a URL do not become an authorization header just because the host and port were parsed. Provide any required supported proxy authentication through the request’s authentication arrangement. For a plain HTTP destination and an intermediary offering Basic, ProxyAgent’s Basic request headers provide one such arrangement.

An HTTPS proxy URL requires a TLS-capable connection to the proxy. A SOCKS URL requires SOCKS negotiation. Neither is implemented by sending those schemes into a plain HTTP ProxyAgent endpoint conversion.