Nikeza i-spark-shell package resolver iproxy yayo ngesikhathi sokuqalisa

spark-shell --driver-java-options "-Dhttp.proxyHost=proxy.example -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example -Dhttps.proxyPort=8080" --packages group:artifact:version

Ekugalisweni kwe-spark-shell ohlangothini lwe-client, lezi zinketho ze-JVM ze-driver zifika ngaphambi kokuxazululwa kwezincikile. Shintsha iphoyinti lokugcina lesibonelo kanye ne-group:artifact:version ngamavelu ahlosiwe usebenzisa izinketho ezisekelwa inguqulo ye-Spark efakiwe. Igama le-host ne-port ye-Java ziyizakhiwo ezihlukene. Okokulanda kwezincikile zomphakathi okuvunyelwe, uhlu lwama-proxy e-HTTP lunikeza abantu abazongenela lezo zinkambu.

Sebenzisa le ndawo lapho i---packages ihluleka ngaphambi kokuba i-shell itholakale. Ikhodi ye-application efakwa kamuva ayikwazi ukulungisa ukulanda kwangaphambilini. I-Spark ihlukanisa izinketho zokuqalisa kusuka kuzilungiselelo zeprogram; kumodi ye-client i-JVM ye-driver isivele ikhona lapho ikhodi ye-application isebenza.

Imodi yokusabalalisa nayo ibalulekile ku-spark.driver.extraJavaOptions: i-driver ingasebenza kumshini omunye, odinga indlela yawo eya ku-proxy. Isicelo se-API kamuva se-executor siyinqubo nendawo yenethiwekhi ehlukile; ukuhlela ama-executor akusho ngempela ukuhlela ukuxazululwa kwesikhathi sokuqalisa.

Lapho izinketho zokuqalisa zikhona kodwa ukulanda kwehluleka

Landela igama le-host yesitoreji, uhlobo lwesicelo kanye nephutha lokuqala. Imethadatha namafayela angavela kuma-host amaningi. I-URL yesitoreji ehambisana ne-Maven ayifakazeli ukuthi i-Spark ifunda i-settings.xml esetshenziswa umyalo we-Maven ophumelelayo.

Isilanda se-Java kufanele sisekele indlela ye-proxy nokuqinisekiswa okudingekayo. Izakhiwo ze-host/port azinikezi yonke imodi, futhi isakhiwo se-http.proxyUsername esiqanjiwe asibi nokusebenza nje ngokuthi i-Java yamukela igama. Gcina ndawonye isilungiselelo sangaphambi kokuqalisa nokulanda okuhlulekile okuqondile lapho uhlola lesi sigaba.