Galaxy is a large Python application with a long list of Python module dependencies. As a result, the Galaxy
developers have made significant effort to provide these dependencies in as simple a method as possible while remaining
compatible with the Python packaging best practices. Thus, Galaxy’s runtime setup procedure makes use of virtualenv
for package environment isolation, pip for installation, and wheel to provide pre-built versions of dependencies.

Using a virtualenv in .venv under the Galaxy source tree is not required. More complicated Galaxy setups may
choose to use a virtualenv external to the Galaxy source tree, which can be done either by not using run.sh directly
(an example of this can be found under the Scaling and Load Balancing documentation) or using the --no-create-venv
option, explained in the Options section. It is also possible to force Galaxy to start without a virtualenv at all,
but you should not do this unless you know what you’re doing.

To manually create a virtualenv, you will first need to obtain virtualenv. There are a variety of ways to do this:

Normally, run.sh calls common_startup.sh, which creates the virtualenv and installs dependencies. You can call
this script yourself to set up Galaxy pip and the dependencies without creating a virtualenv using the
--no-create-venv option:

If your $PYTHONPATH is set, it may interfere with the dependency installation process. Without
--no-create-venv the $PYTHONPATH variable will be automatically unset, but we assume you know what you’re
doing and may want it left intact if you are using --no-create-venv. If you encounter problems, try unsetting
$PYTHONPATH as shown in the example above.

All Galaxy jobs run a metadata detection step on the job outputs upon completion of the tool. The metadata detection
step requires many of Galaxy’s dependencies. Because of this, it’s necessary to make sure the metadata detection step
runs in Galaxy’s virtualenv. If you run a relatively simple Galaxy deployment (e.g. run.sh) then this is assured for
you automatically. In more complicated setups (running under supervisor and/or the virtualenv used to start Galaxy is
not on a shared filesystem) it may be necessary to make sure the handlers know where the virtualenv (or a virtualenv
containing Galaxy’s dependencies) can be found.

If the virtualenv cannot be located, you will see job failures due to Python ImportError exceptions, like so:

If you encounter this error or your Galaxy server’s virtualenv isn’t available on the cluster you can create one manually using the instructions under Managing dependencies manually and activate it using the above-mentioned env tag in job_conf.xml.

If using Pulsar’s option to set metadata on the remote server, the same conditions as with Galaxy job handlers
apply. You should create a virtualenv on the remote resource, install Galaxy’s dependencies in to it, and set an
<env> tag pointing to the virtualenv’s activate as in the Galaxy job handlers section. Instructions on how to
create a virtualenv can be found under the Managing dependencies manually section.

These instruction apply to Galaxy release 19.01 or newer. Please consult the documentation for your version of Galaxy.

Conda and virtualenv are incompatible, unless an adapted virtualenv package from the conda-forge channel is used.
Galaxy can create a virtualenv using the adapted virtualenv package. Once a valid .venv environment exists it will be used.

Tip

If you would like to use a virtualenv created by Conda, the simplest method is:

Ensure .venv does not exist.

Place conda on your PATH if it isn’t.

Start galaxy using shrun.sh or execute shscripts/common_startup.sh.

A Conda environment named _galaxy_ will be created using python 2 and the appropriate virtualenv package will be installed into this environment.
Using this environment a .venv is initialized. This is a one-time setup, and all other activation and dependency
management happens exactly as if a system python was used for creating .venv.

run.sh should automatically set --virtualenv on uWSGI’s command line. However, you can override this using the
virtualenv option in the uwsgi section of galaxy.yml as described in the Managing dependencies manually
section.

Unpinned dependencies may be useful for development but should not be used in production. Please do not install
unpinned dependencies unless you know what you’re doing. While the Galaxy Committers
will do their best to keep dependencies updated, they cannot provide support for problems arising from unpinned
dependencies.

Galaxy’s dependencies can be installed either “pinned” (they will be installed at exact versions specified for your
Galaxy release) or “unpinned” (the latest versions of all dependencies will be installed unless there are known
incompatibilities with new versions). By default, the release branches of Galaxy use pinned versions for three reasons:

Using pinned versions insures that the prebuilt wheels will be installed, and no
compilation will be necessary.

Galaxy releases are tested with the pinned versions and this allows us to give as much assurance as possible that the
pinned versions will work with the given Galaxy release (especially as time progresses and newer dependency versions
are released while the Galaxy release receives fewer updates.

Pinning furthers Galaxy’s goal of reproducibility as differing dependency versions could result in non-reproducible
behavior.

If you would like to install unpinned versions of Galaxy’s dependencies, install dependencies using the unpinned
requirements file, and then instruct Galaxy to start without attempting to fetch wheels:

You may be able to save yourself some compiling by adding the argument --index-urlhttps://wheels.galaxyproject.org/simple/ to pipinstall, but it is possible to install all of Galaxy’s
dependencies directly from PyPI.

New packages can be added to Galaxy, or the versions of existing packages can be updated, using pipenv and Starforge, Galaxy’s Docker-based build system.

Note

Dependency pinning management is being migrated to pipenv. As of this release, pinning for packages used for Galaxy
development are managed by pipenv, but pinning for regular runtime packages are still managed with manual changes
to pinned-requirements.txt. See Pull Request #4891 for details.

The process is still under development and will be streamlined and automated over time. For the time being, please use
the following process to add new packages and have their wheels built:

Install Starforge (e.g. with pipinstallstarforge or pythonsetup.pyinstall from the source). You will
also need to have Docker installed on your system.

Obtain wheels.yml (this file will most likely be moved in to Galaxy in the future) and add/modify the wheel
definition.

Use starforgewheel--wheels-config=wheels.yml<wheel-name> to build the wheel. If the wheel includes C
extensions, you will probably want to also use the --no-qemu flag to prevent Starforge from attempting to build
on Mac OS X using QEMU/KVM.

If the wheel build is successful, submit a pull request to Starforge with your changes to wheels.yml.

A Galaxy Committers group member will need to trigger an automated build of the wheel
changes in your pull request. Galaxy’s Jenkins service will build these changes using Starforge.

If the pull request is merged, submit a pull request to Galaxy modifying the files in lib/galaxy/dependencies as
appropriate.

You may attempt to skip directly to step 4 and let the Starforge wheel PR builder build your wheels for you. This is
especially useful if you are simply updating an existing wheel’s version. However, if you are adding a new C extension
wheel that is not simple to build, you may need to go through many iterations of updating the PR and having a
Galaxy Committers group member triggering builds
before wheels are successfully built. You can avoid this cycle by performing
steps 1-3 locally.