Galaxy is designed to run jobs on your local system by default, but it can be configured to run jobs on a cluster. The front-end Galaxy application runs on a single server as usual, but tools are run on cluster nodes instead.

If you do not already have a DRM, Pulsar is available which does not require an existing cluster or a shared filesystem and can also run jobs on Windows hosts.

Installing and configuring your cluster hardware and management software is outside the scope of this document (and specific to each site). That said, a few pitfalls commonly encountered when trying to get the user Galaxy runs as (referred to in this documentation as galaxy_user) able to run jobs on the DRM are addressed here:

The host on which the Galaxy server processes run (referred to in this documentation as galaxy_server) should be configured in the DRM as a “submit host”.

galaxy_user must have a real shell configured in your name service (/etc/passwd, LDAP, etc.). System accounts may be configured with a disabled shell like /bin/false (Debian/Ubuntu) or /bin/nologin Fedora/RedHat.

If Galaxy is configured to submit jobs as real user (see below) then the above must be true for all users of Galaxy.

The Galaxy server and the worker nodes are running the same version of Python (worker nodes will run Python scripts calling the Galaxy code and its dependencies to set job output file metadata).

To continue, you should have a working DRM that galaxy_user can successfully submit jobs to.

Galaxy (with the exception of the Pulsar runner) currently requires a shared filesystem between the application server and the cluster nodes. There is some legacy code in the PBS runner that does file staging, but its operational status is unknown. The path to Galaxy must be exactly the same on both the nodes and the application server (although it is possible to use symlinks to partially subvert the absolute path requirement). This is because absolute paths are used to refer to datasets and tools when running the command on the cluster node. The shared filesystem and absolute pathnames are limitations that will eventually be removed as development time permits.

If your cluster nodes have Internet access (NAT is okay) and you want to run the data source tools (upload, ucsc, etc.) on the cluster (doing so is highly recommended), set new_file_path in galaxy.yml to a directory somewhere in your shared filesystem:

new_file_path:/clusterfs/galaxy/tmp

Additionally some of the runners including DRMAA may use the cluster_files_directory for sharing files with the cluster, which defaults to database/pbs. You may need to create this folder.

cluster_files_directory:database/pbs

You may also find that attribute caching in your filesystem causes problems with job completion since it interferes with Galaxy detecting the presence and correct sizes of output files. In NFS caching can be disabled with the -noac mount option on Linux (on the Galaxy server), but this may have a significant impact on performance since all attributes will have to be read from the file server upon every file access. You should try the retry_job_output_collection option in galaxy.yml first to see if this solves the problem.

This documentation covers configuration of the various runner plugins, not how to distribute jobs to the various plugins. Consult the job configuration file documentation for full details on the correct syntax, and for instructions on how to configure tools to actually use the runners explained below.

Galaxy interfaces with DRMAA via drmaa-python. The drmaa-python module is provided with Galaxy, but you must tell it where your DRM’s DRMAA library is located, via the $DRMAA_LIBRARY_PATH environment variable, for example:

TORQUE: The DRMAA runner can also be used (instead of the PBS runner) to submit jobs to TORQUE, however, problems have been reported when using the libdrmaa.so provided with TORQUE. Using this library will result in a segmentation fault when the drmaa runner attempts to write the job template, and any native job runner options will not be passed to the DRM. Instead, you should compile the pbs-drmaa library and use this as the value for $DRMAA_LIBRARY_PATH.

Slurm: You will need to install slurm-drmaa. In production on usegalaxy.org we observed pthread deadlocks in slurm-drmaa that would cause Galaxy job handlers to eventually stop processing jobs until the handler was restarted. Compiling slurm-drmaa using the compiler flags -g-O0 (keep debugging symbols, disable optimization) caused the deadlock to disappear.

Most options defined in the DRMAA interface are supported. Exceptions include remoteCommand, jobName, outputPath, and errorPath since these attributes are set by Galaxy. To pass parameters to your underlying DRM, use the nativeSpecification parameter. The format of this parameter is dependent upon the underlying DRM. However, for Grid Engine, it is the list of command line parameters that would be passed to qsub(1).

As of May 2014 there are still some outstanding bugs in pbs_python. Notably, error code translation is out of alignment. For example if you get error #15025 “Bad UID for Job Execution” pbs_python will report this error incorrectly as “Queue already exists”. You may consult the TORQUE source code for the proper error message that corresponds with a given error number.

Most options available to qsub(1b) and pbs_submit(3b) are supported. Exceptions include -o/Output_Path, -e/Error_Path, and -N/Job_Name since these PBS job attributes are set by Galaxy. Parameters can be specified by either their flag (as used with qsub) or long name (as used with pbs_submit).

Galaxy will submit jobs to HTCondor as the “galaxy” user (or whatever user the Galaxy server runs as). In order for vanilla job execution to work as expected, your cluster should be configured with a common UID_DOMAIN to allow Galaxy’s jobs to run as “galaxy” everywhere (instead of “nobody”).

If you need to add additional parameters to your condor submission, you can do so by supplying <param/>s:

Runs jobs via Galaxy Pulsar. Pulsar does not require an existing cluster or a shared filesystem and can also run jobs on Windows hosts. It also has the ability to interface with all of the DRMs supported by Galaxy. Pulsar provides a much looser coupling between Galaxy job execution and the Galaxy server host than is possible with Galaxy’s native job execution code.

The RemoteShell parameters translate to a command line of %<shell_rsh>[-l<shell_username>]<shell_hostname>"<remote_command_with_args>", where the inclusion of -l is dependent on whether shell_username is set. Alternate values for shell_rsh must be compatible with this syntax.

The SecureShell plugin works like the RemoteShell plugin and takes the same parameters, with the following differences:

The shell_rsh default value is ssh

The command line that will be executed is %<shell_rsh>-oStrictHostKeyChecking=yes-oConnectTimeout=60[-l<shell_username>]<shell_hostname>"<remote_command_with_args>"

The GlobusSecureShell plugin works like the SecureShell plugin and takes the same parameters, with the following difference:

The shell_rsh default value is gsi-ssh

The ParamikoShell option was added in 17.09 with this pull request https://github.com/galaxyproject/galaxy/pull/4358 from Marius van den Beek.

The Torque plugin uses qsub(1) and qstat(1) to interface with a Torque server on the command line.

job_<PBS_JOB_ATTR> </td>

<PBS_JOB_ATTR> refers to a qsub(1B) or pbs_submit(3B) argument/attribute
(e.g. <paramid="job_Resource_List">walltime=24:00:00,ncpus=4</param>).

Torque attributes can be defined in either their short (e.g. qsub(1B) argument as used on the command line or in a script as #PBS-<ARG>) or long (e.g. pbs_submit(3B) attribute as used in the C library) oforms. Some additional examples follow:

<paramid="job_-q">queue</param>: set the PBS destination (in this example, a queue), equivalent to <paramid="job_destination">queue</param>

<paramid="job_Priority">128</param>: set the job priority, equivalent to <paramid="job_-p">128</param>

Galaxy runs as a process on your server as whatever user starts the server - usually an account created for the purpose of running Galaxy. Jobs will be submitted to your cluster(s) as this user. In environments where users in Galaxy are guaranteed to be users on the underlying system (i.e. Galaxy is configured to use external authentication), it may be desirable to submit jobs to the cluster as the user logged in to Galaxy rather than Galaxy’s system user.

Since this is a complex problem, the current solution does have some caveats:

All of the datasets stored in Galaxy will have to be readable on the underlying filesystem by all Galaxy users. Said users need not have direct access to any systems which mount these filesystems, only the ability to run jobs on clusters that mount them. But I expect that in most environments, users will have the ability to submit jobs to these clusters or log in to these clusters outside of Galaxy, so this will be a security concern to evaluate for most environments.

Technical details - Since Galaxy maintains dataset sharing internally and all files are owned by the Galaxy user, when running jobs only under a single user, permissions can be set such that only the Galaxy user can read all datasets. Since the dataset may be shared by multiple users, it is not suitable to simply change ownership of inputs before a job runs (what if another user tried to use the same dataset as an input during this time?). This could possibly be solved if Galaxy had tight control over extended ACLs on the file, but since many different ACL schemes exist, Galaxy would need a module for each scheme to be supported.

The real user system works by changing ownership of the job’s working directory to the system prior to running the job, and back to the Galaxy user once the job has completed. It does this by executing a site-customizable script via sudo.

Two possibilities to determine the system user that corresponds to a galaxy user are implemented: i) the user whos name matches the Galaxy user’s email address (with the @domain stripped off) and ii) the user whos name is equal to the galaxy user name. Until release 17.05 only the former option is available. The latter option is suitable for Galaxy installations that user external authentification (e.g. LDAP) against a source that is also the source of the system users.

The script accepts a path and does nothing to ensure that this path is a Galaxy working directory per default (and not at all up to release 17.05). So anyone who has access to the Galaxy user could use this script and sudo to change the ownership of any file or directory. Furthermore, anyone with write access to the script could introduce arbitrary (harmful) code – so it might be a good idea to give write access only to trustworthy users, e.g., root.

You’ll need to ensure that all datasets are stored on the filesystem such that they are readable by all users that will use Galaxy: either made readable by a group, or world-readable. If using a group, set your umask(1) to 027 or for world-readable, use 022 Setting the umask assumes your underlying filesystem uses POSIX permissions, so if this is not the case, your environment changes may be different.

The directory specified in new_file_path in the Galaxy config should be world-writable, cluster-accessible (via the same absolute path) and have its sticky bit (+t) set. This directory should also be cleaned regularly using a script or program as is appropriate for your site, since temporary files created here may not always be cleaned up under certain conditions.

The outputs_to_working_directory option in the Galaxy config must be set to True. This ensures that a tool/job’s outputs are written to the temporary working directory, which (when using the real user system) is owned by the real user who submitted the job. If left set to the default (False), the tool will attempt to write directly to the directory specified in file_path (by default, galaxy-app/database/files/), which must be owned by the Galaxy user (and thus will not be writable by the real user).

For releases later than 17.05 you can configure the method how the system user is determined in config/galaxy.yml via the variable real_system_username. For determining the system user from the email adress stored in Galaxy set it to user_email, otherwise for determining the system user from the Galaxy user name set it to username.

Once these are set, you must set the drmaa_external_* and external_chown_script settings in the Galaxy config and configure sudo(8) to allow them to be run. A sudo config using the three scripts set in the sample galaxy.yml would be:

If your sudo config contains Defaultsrequiretty, this option must be disabled.

For Galaxy releases > 17.05, the sudo call has been moved to galaxy.yml and is thereby configurable by the Galaxy admin. This can be of interest because sudo removes PATH, LD_LIBRARY_PATH, etc. variables per default in some installations. In such cases the sudo calls in the three variables in galaxy.yml can be adapted, e.g., sudo-EPATH=...LD_LIBRARY_PATH=.../PATH/TO/GALAXY/scripts/drmaa_external_runner.py. In order to allow setting the variables this way adaptions to the sudo configuration might be necessary. For example, the path to the python inside the galaxy’s python virtualenv may have to be inserted before the script call to make sure the virtualenv is used for drmaa submissions of real user jobs.

Also for Galaxy releases > 17.05: In order to allow external_chown_script.py to chown only path below certain entry points the variable ALLOWED_PATHS in the python script can be adapted. It is sufficient to include the directorries job_working_directory and new_file_path as configured in galaxy.yml.

It is also a good idea to make sure that only trusted users, e.g. root, have write access to all three scripts.

Some maintenance and support of this code will be provided via the usual Support channels, but improvements and fixes would be greatly welcomed, as this is a complex feature which is not used by the Galaxy Development Team.

Galaxy tries to define special environment variables for each job that contain
the information on the number of available slots and the amount of available
memory:

GALAXY_SLOTS: number of available slots

GALAXY_MEMORY_MB: total amount of available memory in MB

GALAXY_MEMORY_MB_PER_SLOT: amount of memory that is available for each slot in MB

More precisely Galaxy inserts bash code in the job submit script that
tries to determine these values. This bash code is defined here:

lib/galaxy/jobs/runners/util/job_script/CLUSTER_SLOTS_STATEMENT.sh

lib/galaxy/jobs/runners/util/job_script/MEMORY_STATEMENT.sh

If this code is unable to determine the variables, then they will not be set.
Therefore in the tool XML files the variables should be used with a default,
e.g. \${GALAXY_SLOTS:-1} (see also https://planemo.readthedocs.io/en/latest/writing_advanced.html#cluster-usage).

In particular GALAXY_MEMORY_MB and GALAXY_MEMORY_MB_PER_SLOT are currently
defined only for a few cluster types. Contributions are very welcome, e.g. let
the Galaxy developers know how to modify that file to support your cluster.