Hello,
I created a Docker image following the guide, and uploaded it to a private Docker Hub registry account. When I try to send a script via spark-submit, I get an error with the following message. If I try a public image I don't get this problem, so I guess I need to enter my login information somewhere. Is it possible to do this? Or do I need to upload my images somewhere else?
Application report for application_1640081147608_3034 (state: FAILED)
21/12/22 14:22:32 INFO Client:
client token: N/A
diagnostics: Application application_1640081147608_3034 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1640081147608_3034_000001 exited
with exitCode: 7
Failing this attempt.Diagnostics: [2021-12-22 14:22:26.758]Exception from container-launch.
Container id: container_e5006_1640081147608_3034_01_000001
Exit code: 7
Exception message: Launch container failed
Shell error output: image: registry.hub.docker.com/flucio/gedap is not trusted.
Disable mount volume for untrusted image
image: registry.hub.docker.com/flucio/gedap is not trusted.
Disable mount volume for untrusted image
image: registry.hub.docker.com/flucio/gedap is not trusted.
Disable cap-add for untrusted image
Docker capability disabled for untrusted image
Unable to find image 'registry.hub.docker.com/flucio/gedap:latest' locally
/usr/bin/docker: Error response from daemon: pull access denied for registry.hub.docker.com/flucio/gedap, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See '/usr/bin/docker run --help'.
Shell output: main : command provided 4
main : run as user is luciof
main : requested yarn user is luciof
Creating script paths...
Creating local dirs...
Getting exit code file...
Changing effective user to root...
Wrote the exit code 7 to /data3/hadoop/yarn/local/nmPrivate/application_1640081147608_3034/container_e5006_1640081147608_3034_01_000001
/container_e5006_1640081147608_3034_01_000001.pid.exitcode
[2021-12-22 14:22:26.780]Container exited with a non-zero exit code 7.
[2021-12-22 14:22:26.782]Container exited with a non-zero exit code 7.
For more detailed output, check the application tracking page: https://epod-master2.vgt.vito.be:8090/cluster/app/application_1640081147608_3034 Then click on links to logs of each attempt.. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1640179342928
final status: FAILED
tracking URL: https://epod-master2.vgt.vito.be:8090/cluster/app/application_1640081147608_3034
user: luciof
21/12/22 14:22:32 INFO Client: Deleted staging directory hdfs://hacluster/user/luciof/.sparkStaging/application_1640081147608_3034
Exception in thread "main" org.apache.spark.SparkException: Application application_1640081147608_3034 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1269)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1627)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Comments
Dear Luciof, Due to…