Put Nutch’s Proxy in the Fetcher Configuration
Set Nutch’s proxy host and port in nutch-site.xml and select a protocol plug-in that implements the required proxy features.
Set the basic endpoint in the crawl's loaded nutch-site.xml:
<configuration>
<property>
<name>http.proxy.host</name>
<value>proxy.example</value>
</property>
<property>
<name>http.proxy.port</name>
<value>8080</value>
</property>
</configuration>For an HTTP fetcher route to permitted public sites without proxy authentication, replace the hostname and separate port with a candidate from the HTTP proxy list. Keep local overrides in the site file and compare the installed release's distributed defaults. A shell export alone cannot configure a plug-in that never reads it.
| Proxy property | Documented consumer |
|---|---|
http.proxy.host, http.proxy.port | The basic fetcher route, according to the selected implementation |
http.proxy.username, http.proxy.password, http.proxy.realm | protocol-httpclient, which configures credentials on its HTTP client |
http.proxy.type | An option identified for protocol-okhttp |
Ensure the required plug-in is installed and included in effective selection. A field appearing in a defaults file does not mean every HTTP implementation consumes it.
Inspect the implementation and configuration of the process actually fetching. Distributed workers and launch scripts need the intended inputs; a file edited in another installation does not change an already configured fetcher.