Fetch a Page Through an HTTP Proxy in jsoup

Use jsoup’s request-level HTTP proxy API and create separate requests when several workers share session settings.

jsoup can fetch the page through an HTTP proxy and parse the response in the same call:

Document page = Jsoup.connect("https://www.example.com/")
    .proxy("proxy.example", 8080)
    .get();

The hostname and integer port identify the proxy. For permitted public-page requests, select an HTTP endpoint from your service or the HTTP proxy list. The URL passed to connect remains the page you want. You do not need to download the HTML into a string with a separate URLConnection just to add this route.

proxy(String, int) selects an HTTP proxy for that request. A port number such as 1080 does not change its protocol into SOCKS; the proxy endpoint must speak the type being configured. jsoup also exposes an overload accepting a java.net.Proxy, but support for a particular route still depends on the HTTP implementation in use.

Give each worker its own request

A request-level option makes parallel work easier than changing JVM system properties around each download. Keep the mutable request separate for each worker. Modern jsoup can share a session's settings and cookie store while creating a fresh request:

Connection session = Jsoup.newSession();
Document first = session.newRequest()
    .url("https://www.example.com/one")
    .proxy("proxy-a.example", 8080)
    .get();
Document second = session.newRequest()
    .url("https://www.example.com/two")
    .proxy("proxy-b.example", 8080)
    .get();

The calls above run sequentially for clarity. If work is dispatched concurrently, create the requests independently rather than changing the proxy on one in-flight Connection. A shared session also shares cookies; use different sessions when the browsing identities should remain separate.

These interfaces are present in jsoup 1.21.2. The no-argument newRequest() interface dates from 1.14.1, so an older application should check its dependency before adopting the session example.

Credentials are a further requirement for an authenticated proxy. A successful route selection does not supply an account, and receiving a proxy challenge is different from failing to connect to its port.

Tënë sô zo