Question about RDD pipe

Question about RDD pipe

Hi, im trying to run an external script on spark using rdd.pipe() and
although it runs successfully on standalone, it throws an error on cluster.
The error comes from the executors and it's : "Cannot run program
"path/to/program": error=2, No such file or directory".

Does the external script need to be available on all nodes in the cluster
when using rdd.pipe()?

What if i don't have permission to install anything on the nodes of the
cluster? Is there any other way to make the script available to the worker
nodes?

(The external script is loaded in HDFS and is passed to the driver class
through args)

Hi, im trying to run an external script on spark using rdd.pipe() and
although it runs successfully on standalone, it throws an error on cluster.
The error comes from the executors and it's : "Cannot run program
"path/to/program": error=2, No such file or directory".

Does the external script need to be available on all nodes in the cluster
when using rdd.pipe()?

What if i don't have permission to install anything on the nodes of the
cluster? Is there any other way to make the script available to the worker
nodes?

(The external script is loaded in HDFS and is passed to the driver class
through args)