Migrating between Archive Storage Drivers

Overview

To cleanly migrate data from one archive driver to another, Anchore Engine includes some tooling that automates the process in the 'anchore-manager' tool packaged with the system.

The migration process is an offline process; Anchore Engine is not designed to handle an online migration.

For the migration process you will need:

The original config.yaml used by the services already, if services are split out or using different config.yaml for different services, you need the config.yaml used by the catalog services

An updated config.yaml (named dest-config.yaml in this example), with the archive driver section of the catalog service config set to the config you want to migrate *to*

The db connection string from config.yaml, this is needed by the anchore-manager script directly

Credentials and resources (bucket etc) for the destination of the migration

At a high-level the process is:

Shutdown all anchore engine services and components. The system should be fully offline, but the database must be online and available. For a docker-compose install, this is achieved by simply stopping the engine container, but not deleting it.

Prepare a new config.yaml that includes the new driver configuration for the destination of the migration (dest-config.yaml) in the same location as the existing config.yaml

Test the new dest-config.yaml to ensure correct configuration

Run the migration

Get coffee... this could take a while if you have a lot of analysis data

When complete, view the results

Ensure the dest-config.yaml is in place for all the components as config.yaml

Start anchore-engine

Migration Example Using Docker Compose Deployed Anchore Engine

The following is an example migration for an anchore-engine deployed via docker-compose on a single host with a local postgresql container--basically the example used in 'Installing Anchore Engine' documents. At the end of this section, we'll cover the caveats and things to watch for a multi-node install of anchore engine.

This process requires that you run the command in a location that has access to both the source archive driver configuration and the new archive driver configuration.

Step 1: Shutdown all anchore engine services

All services should be stopped, but the postgresql db must still be available and running.

[user@host aevolume]$ docker-compose stop anchore-engine

Step 2: Prepare a new config.yaml with the new archive configuration

Both the original and new configurations are needed, so create a copy and update the archive driver section to the configuration you want to migrate to

Step 3a: Test the current config.yaml if you are running the migration for a different location than one of the anchore engine containers

Same as above but using /config/config.yaml as the input to check (skipped in this instance since we're running the migration from the same container)

Step 4: Run the Migration

By default, the migration process will remove data from the source once it has confirmed it has been copied to the destination and the metadata has been updated in the anchore db. To skip the deletion on the source, use the '--nodelete' option. it is the safest option, but if you use it, you are responsible for removing the data later.

Things to Watch for in a Multi-Node Anchore Engine Installation

Ensure the place you're running the migration from has the same db access and network access to the archive locations

After migration:

Ensure that all components get the update config.yaml. Strictly speaking, only containers that run the catalog service need the update configuration, but its best to ensure that any config.yaml in the system which has a services.catalog definition also has the proper and up-to-date configuration to avoid confusion or accidental reverting of the config.