Why ProxyMesh’s Exit Header May Be Missing in Scrapy

An absent X-ProxyMesh-IP in a Scrapy HTTPS callback does not prove direct routing. For tunneled HTTPS, ProxyMesh places routing headers on the CONNECT reply; the later website response is inside TLS and does not automatically include that earlier metadata.

Where the header is exposed in a callback, read it conditionally:

exit_ip = response.headers.get(b'X-ProxyMesh-IP')
if exit_ip is not None:
    self.logger.info('Reported exit IP: %s', exit_ip.decode('ascii'))

This is the provider-reported exit for that exchange, not the configured entry hostname.

If tunnel metadata is required, use a compatible handler integration exposing CONNECT headers. ProxyMesh maintains one for this purpose; check its supported Scrapy version and download handler. Spider parsing cannot reveal headers the transport never supplied.

Associate metadata with its connection. Reusing a tunnel does not create another CONNECT reply for every page. For an independent destination-observed source address, inspect that destination's incoming connection rather than treating a provider header as a universal IP detector.