A Task-Local Requests Proxy in Google Cloud Composer

Create a task-local Requests Session with an explicit route rather than changing a shared worker process’s proxy environment.

For a Cloud Composer task that needs one outbound HTTP request through a proxy, configure the HTTP client inside the task’s callable:

def fetch_status():
    import requests

    with requests.Session() as session:
        session.trust_env = False
        with session.get(
            'https://api.example/health',
            proxies={'https': 'http://proxy.example:8080'},
            timeout=(5, 30),
        ) as response:
            response.raise_for_status()
            return response.status_code

Use this callable through the DAG’s existing task arrangement. The proxy must be reachable from the worker executing it. For permitted public-API calls without proxy authentication, the HTTP proxy list supplies candidates for the session’s HTTP proxy URL. Keep an authenticated URL or other account configuration in the deployment’s intended secret source.

The Session and response close as the function exits. trust_env=False disables Requests’ environment integration, preventing inherited proxy settings from replacing this client’s intended behavior, and also disables Requests’ environment-based CA bundle selection. Supply the intended CA bundle through this client’s verify option if needed.

Environment variables configured for a managed Airflow environment have wider scope and platform-defined restrictions. Passing the route to this Session keeps the decision attached to this task’s HTTP operation without changing os.environ for other code in the worker process.

Erthyglow kevrennek