Give spark-shell's Package Resolver Its Proxy at Launch

spark-shell --driver-java-options "-Dhttp.proxyHost=proxy.example -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example -Dhttps.proxyPort=8080" --packages group:artifact:version

For a client-side spark-shell launch, these driver JVM options arrive before dependency resolution. Replace the illustrative endpoint and group:artifact:version with the intended values using options supported by the installed Spark version. The Java hostname and port are separate properties. For permitted public dependency downloads, the HTTP proxy list offers candidates for those fields.

Use this placement when --packages fails before the shell becomes available. Application code entered afterward cannot repair the earlier download. Spark distinguishes launch options from programmatic settings; in client mode the driver JVM already exists by the time application code runs.

Deployment mode also matters for spark.driver.extraJavaOptions: the driver can run on another machine, which needs its own path to the proxy. An executor's later API request is another process and network location; configuring executors does not necessarily configure startup resolution.

When launch options are present but the download fails

Follow the repository hostname, request type and first error. Metadata and artifacts can come from several hosts. A Maven-compatible repository URL does not establish that Spark reads the settings.xml used by a successful Maven command.

The Java downloader must support the proxy mechanism and required authentication. Host/port properties do not provide every scheme, and an invented http.proxyUsername property does not become effective simply because Java accepts the name. Keep the pre-start launch configuration and specific failed download together when diagnosing this stage.