Choose a Proxy for Each Splash Resource

Select a Splash proxy at render scope or in an on_request callback when individual resources need different routes.

A Scrapy request can contact Splash successfully while the browser inside Splash still needs a different outbound route. Those are two connections:

Scrapy → Splash API
Splash browser → forward proxy → website

For one proxy per render, pass the HTTP API's proxy argument. Splash 3.5 documents a proxy URL such as http://proxy.example:8080, or a configured proxy-profile name. Its URL format also supports SOCKS5 and optional credentials.

When individual page resources need different endpoints, use an execute script and register the selection before navigation:

function main(splash, args)
  splash:on_request(function(request)
    local host = args.default_proxy
    if request.url == args.special_resource then
      host = args.special_proxy
    end
    request:set_proxy{host = host, port = 8080, type = 'HTTP'}
  end)

  assert(splash:go(args.url))
  return splash:html()
end

Pass default_proxy and special_proxy as hostnames, and special_resource as the exact resource URL to route separately. Replace the single example port when the two listeners differ. Other resources receive the default route. The HTTP proxy list provides endpoint candidates for these host and port fields when public-resource routes are allowed and no proxy credentials are applied to the candidates.

The request method accepts optional username and password fields. With type HTTP, HTTPS destinations use CONNECT through that HTTP proxy; they do not require an invented HTTPS proxy type.

The callback runs before each request. Its request methods cannot change a request after it has been sent, and the callback cannot call asynchronous Splash methods such as splash:go or splash:wait.

Keep route selection stable when the page expects one session or source identity. Randomly assigning every image and API request to another exit can produce a rendered page whose requests no longer belong to one coherent session.

Related articles