Hadoop DistCp: lungiselela i-proxy ye-S3A emisebenzini yabasebenzi

Lungiselela i-proxy ye-Hadoop S3A futhi wenze ukulungiselela kutholakale lapho abasebenzi be-DistCp benza khona ukudluliswa.

Ku-connector ye-S3A ye-Hadoop 3.4.2 nge-HTTP proxy, hlanganisa lezi zinto ekulungiseleleni kwe-Hadoop yomsebenzi, ngokuvamile i-core-site.xml efanele:

<property>
  <name>fs.s3a.proxy.host</name>
  <value>gateway.example</value>
</property>
<property>
  <name>fs.s3a.proxy.port</name>
  <value>8080</value>
</property>
<property>
  <name>fs.s3a.proxy.ssl.enabled</name>
  <value>false</value>
</property>

Ukuze kube nokuhlolwa okuvunyelwe ngokwahlukile ngokumelene nephedi ye-S3 efinyeleleka emphakathini ngaphandle kobufakazi be-proxy, uhlu lwama-proxy e-HTTP lunikeza amanani angaba khona egama lomsingathi kanye ne-port ecacile. fs.s3a.proxy.ssl.enabled=false likhetha ukuxhumana okuvamile kwe-HTTP kule proxy; akukucimi i-TLS kusuka ku-S3A kuya ku-S3. Sebenzisa inani elihambisana nesilaleli sangempela se-proxy.

Hlola abasebenzisi bobabili. Iklayenti elithumelayo lingase likwazi ukufaka uhlu lwe-S3 ngempumelelo kuyilapho amakhonteyina emisebenzi yemephu e-DistCp engakwazi ukufinyelela ku-proxy noma ukulayisha ukulungiselela okufanayo kwesistimu yamafayela. Yimisebenzi leyo eyenza ukudluliswa.

Emsebenzini ohlulekile, gcina umsingathi wokwenza, ukulungiselela okusebenzayo kanye nephedi ekhethiwe kanye nephutha. Imvelo ye-shell yokuthumela ayisibufakazi esanele kumsebenzi.

Lezi zilungiselelo zilawula s3a://. Amaresiphi amadala e-Jets3t connector alungiselela enye indlela yokusebenza futhi awathathi indawo yezinketho ze-S3A ezisetshenziswa yilo msebenzi.

Izihloko ezihlobene