All validations are currently stored in a single directory. This makes
it inconvenient to try and write new validations, update from a remote
repository or to add an entirely new (perhaps private) source.

We will store a default set of TripleO validations in a Swift container called
tripleo-validations. These will be shared across all plans and are not
expected to be updated by the deployer. This container should be created on
initial undercloud deployment.

We will provide a mechanism for deployers to add a custom set of validations
per deployment plan. These plan-specific validations will be stored in a
custom-validations subdirectory in the plan’s Swift container. Storing them
together with the plan makes sense as these validations can be specific to
particular deployment plan configuration, as well as makes the import/export
easier.

Since custom validation will be stored as part of the plan, no additional
workflows/actions to perform CRUD operations for them will be necessary; we can
simply use the existing plan create/update for this purpose.

The validation Mistral actions (e.g. list and run_validation)
will need to be updated to take into account this new structure of
validations. They will need to look for validations in the
tripleo-validations Swift container (for default validations) and the
plan’s custom-validations subdirectory (for custom validations), instead of
sourcing them from a directory on disk, as they are doing now.

If a validation with the same name is found both in default in custom
validations, we will always pick the one stored in custom validations.

Note

As a further iteration, we can implement validations as per-service
tasks in standalone service Ansible roles. They can then be consumed
by tripleo-heat-templates service templates.

In order to add their own validations, the deployer will need to
update the deployment plan by adding a custom-validations directory to it,
and making sure this directory contains the desired custom validations. The
plan update operation is already supported in the CLI and the UI.

Since the validation sources will now be Swift containers, downloading
validations will potentially be necessary on each run. We will have to keep an
eye on this an potentially introduce caching if this turns out to be a problem.