A SOCKS Socket Factory in Hadoop's Client Configuration

Load the SOCKS endpoint and factory class together, then account for Hadoop's protocol-specific factory overrides.

Hadoop needs both the SOCKS endpoint and a client socket factory that reads it. Merge them into the operation's loaded core-site.xml:

<property>
  <name>hadoop.socks.server</name>
  <value>proxy.example.net:1080</value>
</property>
<property>
  <name>hadoop.rpc.socket.factory.class.default</name>
  <value>org.apache.hadoop.net.SocksSocketFactory</value>
</property>

In Hadoop 3.4.2, SocksSocketFactory reads the host:port endpoint from its Configuration and creates Java sockets with that route. For a permitted public TCP test without proxy authentication, a compatible entry from the SOCKS proxy list can supply hadoop.socks.server; confirm access to all required protocol destinations and ports.

Before assuming the default is selected, inspect the protocol-specific key: NetUtils first checks hadoop.rpc.socket.factory.class. followed by the protocol class's simple name. It uses the default only when that lookup selects no factory.

A value from another loaded file can therefore keep the old factory active. Inspect the effective client Configuration rather than one XML file on a server.

This applies where Hadoop uses that TCP factory. It preserves the component's destination-address and DNS behavior; selecting the proxy is not an instruction to invent replacement DNS results.

متعلقہ مضامین