Hadoop DistCp: Configure the S3A Proxy on Worker Tasks

Configure the Hadoop S3A connector’s proxy and make the configuration available where DistCp workers perform their transfers.

For Hadoop 3.4.2's S3A connector through an HTTP proxy, merge these entries into the operation's Hadoop configuration, normally the applicable core-site.xml:

<property>
  <name>fs.s3a.proxy.host</name>
  <value>gateway.example</value>
</property>
<property>
  <name>fs.s3a.proxy.port</name>
  <value>8080</value>
</property>
<property>
  <name>fs.s3a.proxy.ssl.enabled</name>
  <value>false</value>
</property>

For an independently permitted test against a publicly reachable S3 endpoint with no proxy credentials configured, the HTTP proxy list supplies candidate values for the hostname and explicit port. fs.s3a.proxy.ssl.enabled=false selects a plain HTTP connection to this proxy; it does not disable TLS from S3A to S3. Use the value matching the actual proxy listener.

Check both consumers. The submitting client may list S3 successfully while DistCp map-task containers cannot reach the proxy or load the same filesystem configuration. Those tasks perform the transfers.

For a failed task, retain execution host, effective configuration and selected endpoint with the error. The submission shell's environment is not enough evidence for a worker.

These settings govern s3a://. Older Jets3t connector recipes configure another implementation and do not substitute for the S3A options used by this job.

Artiglyn mynjee