Introduction

Only specific version-to-version upgrade sequences are supported. Some upgrades require multiple steps. For more information, see the
supported upgrade sequences.

For Fusion 3.1 and later releases, a migrator is available for upgrading Fusion.

During the upgrade process, the migrator uses a properties file. After downloading and installing the migrator, the properties files is in the /opt/lucidworks/fusion/3.1.x/var/upgrade directory (on Unix or MacOS) or the C:\lucidworks\fusion\3.1.x\var\upgrade\ directory (on Windows). The file names reference the versions you are upgrading from and to. For example:

To upgrade 3.1.5 to 4.2.0, the migrator uses the 3.1.x-4.2.x.properties file.

Migration entails down time and multiple starts and stops of Fusion services. Plan accordingly, especially in terms of disabling external load balancers or monitors that might react adversely to the starts and stops.

Download the latest migrator immediately before upgrading. This helps ensure that the upgrade goes smoothly.

Important

The newer Fusion instance must be newly untarred and never started.

About the upgrade

This section describes how connectors, object migrations, and signals are migrated during an upgrade.

Connectors

In Fusion 3.1.0 and above, only a subset of
connectors
are included by default.

If a connector to be upgraded wasn’t available during the upgrade, then a message in /opt/lucidworks/fusion/3.1.x/var/upgrade/tmp/migrator.log (on Unix) or C:\lucidworks\fusion\3.1.x\var\upgrade\tmp\migrator.log (on Windows) indicates this.

Only datasources for connectors that are supported in the new Fusion version are upgraded. Datasources for custom connectors aren’t upgraded.

If no Internet connection is available

If no Internet connection is available during an upgrade, the migrator can’t automatically download the connectors it needs. Using the Fusion UI or API later to install the connectors also might not be an option.

In this case, download the connector zip files for all existing connectors and any connectors that you are adding from http://lucidworks.com/connectors/ and place them in apps/connectors/bootstrap-plugins for the new deployment (on all Fusion nodes). Do so at the time indicated in the procedures that follow.

Adding connectors during an upgrade

You can add connectors during an upgrade (that is, add connectors that aren’t in the old deployment).

Download the connector zip files from http://lucidworks.com/connectors/ and place them in apps/connectors/bootstrap-plugins for the new version (on all Fusion nodes).

Object migrations and transformations

The migrator upgrades all Fusion 3.1 object types:

Collections

Index pipelines

Query pipelines

Search cluster configurations

Schedules

Aggregations

Datasources

Dashboards

Parsing configurations

Object groups

Links

Tasks

Jobs

Spark objects

The migrator adds these Fusion Server object types:

Apps

Appkit apps

Index profiles

Query profiles

Blobs

Important

In Fusion Server 4.0 and later, most objects exist in the context of apps. When you migrate from Fusion Server 4.0.x to 4.1.y, the migrator upgrades app objects, all objects in or linked to objects in apps, and objects that are not linked to apps. You can explore the objects in Object Explorer.

Noteworthy transformations

While migrating objects, the migrator makes these transformations:

It adds the new log-shipper service to fusion.properties.

It removes obsolete log-cleanup tasks.

It converts the format of job history records.

Migration of index profiles and query profiles

In Fusion Server 4.0, index profiles and query profiles are objects. They have capabilities that exceed those of index profiles and query profiles in prior releases.

Prior to Fusion 4.0 – A single index profile could reference multiple index pipelines. A single query profile could reference multiple query pipelines.

In Fusion 4.0 and later – A single index profile can reference a single index pipeline. A single query profile can reference a single query pipeline.

During migration from 3.1.x to 4.0.y, the migrator upgrades index profiles and query profiles as follows:

Index profiles:

If a 3.1.x index profile contains a reference to the index pipeline default, then the migrated profile retains that single reference.

If a 3.1.x index profile doesn’t contain a reference to the index pipeline default, then the migrated profile references an index pipeline that has the same name as the index profile.

Query profiles:

If a 3.1.x query profile contains a reference to the query pipeline default, then the migrated profile retains that single reference.

If a 3.1.x query profile doesn’t contain a reference to the query pipeline default, then the migrated profile references a query pipeline that has the same name as the query profile.

After migration, you might need to adjust the pipeline references of index profiles and query profiles by hand, and/or create new index profiles and query profiles.

Signals and Spark jobs

Fusion Server 4.2.y indexes search log events into the <collection>_signals collections as response signals, instead of indexing them into the <collection>_logs collections.

Roughly 90 new fields used by App Insights are added to existing <collection>_signals collections.

You are encouraged to adopt the new signal fields, but you can continue using the old dynamic field names, such as user_id_s, until Fusion 5.0. If you adopt the new signals schema, then you must update any Spark jobs that rely on the old field names.

Fusion Server 4.2.y replaces the _signals_ingest index pipeline with a new version that works with the new signals schema.

If you made changes to the _signals_ingest pipeline, then you’ll need to manually add those changes to the new configuration after migration.

The migrator preserves legacy-style aggregation jobs. Optionally, you can manually convert these to SQL-based aggregation jobs.

Fusion Server 4.2.y adds a new session rollup job for each collection with signals enabled. The session rollup job creates session signals that contain aggregated information about user activity in a session.

Review known issues

Before upgrading, review the known issues to see whether any of them apply to the circumstances of your upgrade. Some known issues might require actions before upgrading.

Upgrade on Unix

Use this procedure to upgrade Fusion on a single Unix node or on multiple Unix nodes. If there is a single node, perform all steps on that node unless otherwise indicated.

Perform the steps in this procedure on the indicated nodes on which Fusion is running ("Fusion nodes"). To perform an upgrade, Fusion nodes must have at least these services running:

API service (api)

Fusion UI service (ui)

If Solr and/or ZooKeeper instances are also running on other nodes (without Fusion), you don’t need to do anything with the external Solr and/or ZooKeeper instances.

Important

For every step on multiple nodes, ensure that the step completes on all Fusion nodes before going to the next step. There is the notion of a "main node" during the migration process. This node will be used for certain centralized migration activities that do not need to be done on every node, such as downloading connectors that are then uploaded to blob storage that is shared by all, etc. Just pick one of your Fusion nodes to be the "main node"; there’s no special requirement as to which one you pick.

Ensure that your current version of Fusion has a valid license

Ensure that your current version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a valid license.properties file in the /opt/lucidworks/fusion/4.2.y/conf directory.

For more information, see
link:/fusion-server/4.2/deployment/licensing.html.

For example, if Fusion is currently installed in /opt/lucidworks/fusion/3.1.x, then change your working directory to /opt/lucidworks/ and extract the file there. Don’t run the new version of Fusion yet.

Ensure that the new version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a valid license.properties file in the /opt/lucidworks/fusion/4.2.y/conf directory.

(If there are customjarfiles) If your deployment has custom jar files, add them to the new Fusion deployment.

(If you are performing an upgrade without Internet access) Without Internet access, the migrator can’t download new versions of connectors automatically. Download the new versions of connector zip files for your current connectors from http://lucidworks.com/connectors/ and place them in apps/connectors/bootstrap-plugins for the new deployment.

(If you are addingnewconnectors) If you want your new deployment to use connectors that are not in the current deployment, you can add them now. Download the connector zip files from http://lucidworks.com/connectors/ and place them in apps/connectors/bootstrap-plugins for the new deployment.

Verify that there is sufficient disk space for a second copy of the Solr index directory, fusion/3.1.x/data/solr. If there isn’t sufficient disc space, free up space before proceeding.

(If upgrading from Fusion 3.1.0 through 3.1.3; On all Fusion nodes) Start Solr for the new Fusion version:

$FUSION_NEW/bin/solr start

(If upgrading from Fusion 3.1.0 through 3.1.3; Only on the main Fusion node) Run a script to remove all old plugins from the blob store. Replace solr-address and solr-port as appropriate (as shown in the example):

(On all Fusion nodes) Start all Fusion services for the new version of Fusion:

$FUSION_NEW/bin/fusion start

(Only on the main Fusion node) Import the second part of configuration data into the new version of Fusion:

java -jar $FUSION_OLD/var/upgrade/migrator.jar --fusion-import

This message indicates that the step finished successfully:

New Fusion object import (--fusion-import) finished successfully.

Tip

After migration, you can find details about the results in the fusion/3.1.x/var/upgrade/tmp directory. If the migration produces unexpected results, the files in this directory are helpful for troubleshooting.

Validate the new version of Fusion

To validate the new version of Fusion

(Only on the main Fusion node) Restart the new version of Fusion (all services defined in fusion.properties):

$FUSION_NEW/bin/fusion restart

Log into the Fusion UI (your admin password is the same as for the old installation), and confirm the release number of the new version of Fusion:

Clear your browser’s cache.

Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior.

In a browser, open the Fusion UI at http://localhost:8764/ (replace localhost with your server name or IP address if necessary).

Log in.

Navigate to Admin > About Fusion.

The About Fusion panel should display the newer Fusion release number.

Ensure that all connectors were installed automatically during the upgrade.
From the Fusion launcher, click the tile for a migrated app. Click System > Blobs.
If any connectors are missing from the list, click Add > Connector Plugin and install them manually.

Ensure that all customizations you made in the former version of Fusion are present in the new one.

When you are satisfied with the migration and you have backed up the fusion/3.1.x/ directory, you can rm -fr fusion/3.1.x/ to remove the older version of Fusion (on all Fusion nodes).

Add support for business rules to existing apps

Fusion AI 4.2 introduces functionality for using business rules, that is, manually created formulas for rewriting queries and responses.

You can add support for business rules to apps that were created in versions of Fusion AI prior to version 4.2. To do so, perform the steps in this section.

The add-rule-objects-xyz.zip file (where xyz is a version number) specifies the objects to add to an app. It is supplied in the Fusion migrator zip file at the top level. After installing the migrator, the location is $FUSION_OLD/var/upgrade/import-files.

Note

Adding support for business rules has a costs. Additional collections and objects are created. Only add support for business rules to apps in which you plan to use them.

Upgrade on Windows

Use this procedure to upgrade Fusion on a single Windows node or multiple Windows nodes. If there is a single node, perform all steps on that node unless otherwise indicated.

Perform the steps in this procedure on the indicated nodes on which Fusion is running ("Fusion nodes"). To perform an upgrade, Fusion nodes must have at least these services running:

API service (api)

Fusion UI service (ui)

If Solr and/or ZooKeeper instances are also running on other nodes (without Fusion), you don’t need to do anything with the external Solr and/or ZooKeeper instances.

Important

If you are upgrading Fusion on multiple nodes, then, for every step on multiple nodes, ensure that the step completes on all Fusion nodes before going to the next step. There is the notion of a "main node" during the migration process. This node will be used for certain centralized migration activities that do not need to be done on every node, such as downloading connectors that are then uploaded to blob storage that is shared by all, etc. Just pick one of your Fusion nodes to be the "main node"; there’s no special requirement as to which one you pick.

Ensure that your current version of Fusion has a valid license

Ensure that your current version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a valid license.properties file in the C:\lucidworks\fusion\3.1.x\conf directory.

For more information, see
link:/fusion-server/4.2/deployment/licensing.html.

Download and install the newer version of Fusion

Move the fusion-4.2.y.zip file to the directory that contains the fusion\ directory.

For example, if Fusion is installed in C:\lucidworks\fusion\3.1.x, then move the file to C:\lucidworks.

Unzip the fusion-4.2.y.zip file. Don’t run the new version of Fusion yet.

Ensure that the new version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a valid license.properties file in the C:\lucidworks\fusion\4.2.y\conf directory.

(If there are customjarfiles) If your deployment has custom jar files, add them to the new Fusion deployment.

(If you are performing an upgrade without Internet access) Without Internet access, the migrator can’t download new versions of connectors automatically. Download the new versions of connector zip files for your current connectors from http://lucidworks.com/connectors/ and place them in apps\connectors\bootstrap-plugins for the new deployment.

(If you are addingnewconnectors) If you want your new deployment to use connectors that are not in the current deployment, you can add them now. Download the connector zip files from http://lucidworks.com/connectors/ and place them in apps\connectors\bootstrap-plugins for the new deployment.

Verify that there is sufficient disk space for a second copy of the Solr index directory, fusion\3.1.x\data\solr. If there isn’t sufficient disc space, free up space before proceeding.

(If upgrading from Fusion 3.1.0 through 3.1.3; On all Fusion nodes) Start Solr for the new Fusion version:

%FUSION_NEW%\bin\solr.cmd start

(If upgrading from Fusion 3.1.0 through 3.1.3; Only on the main Fusion node) Run a script to remove all old plugins from the blob store. Replace solr-address and solr-port as appropriate (as shown in the example):

(On all Fusion nodes) Start all Fusion services for the new version of Fusion:

%FUSION_NEW%\bin\fusion.cmd start

(Only on the main Fusion node) Import the second part of configuration data into the new version of Fusion:

java -jar "%FUSION_OLD%\var\upgrade\migrator.jar" --fusion-import

This message indicates that the step finished successfully:

New Fusion object import (--fusion-import) finished successfully.

Tip

After migration, you can find details about the results in the fusion\3.1.x\var\upgrade\tmp directory. If the migration produces unexpected results, the files in this directory are helpful for troubleshooting.

Validate the new version of Fusion

To validate the new version of Fusion

(On all Fusion nodes) Restart all Fusion services for the new version of Fusion:

%FUSION_NEW%\bin\fusion.cmd restart

Log into the Fusion UI (your admin password is the same as for the old installation), and confirm the release number of the new version of Fusion:

Clear your browser’s cache.

Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior.

In a browser, open the Fusion UI at http://localhost:8764/ (replace localhost with your server name or IP address if necessary).

Log in.

Navigate to Admin > About Fusion.

The About Fusion panel should display the newer Fusion release number.

Ensure that all connectors were installed automatically during the upgrade.
From the Fusion launcher, click the tile for a migrated app. Click System > Blobs.
If any connectors are missing from the list, click Add > Connector Plugin and install them manually.

Ensure that all customizations you made in the former version of Fusion are present in the new one.

When you are satisfied with the migration and you have backed up the fusion\3.1.x directory, you can remove the older version of Fusion by removing that directory (on all Fusion nodes).

Add support for business rules to existing apps

Fusion AI 4.2 introduces functionality for using business rules, that is, manually created formulas for rewriting queries and responses.

You can add support for business rules to apps that were created in versions of Fusion AI prior to version 4.2. To do so, perform the steps in this section.

The add-rule-objects-xyz.zip file (where xyz is a version number) specifies the objects to add to an app. It is supplied in the Fusion migrator zip file at the top level. After installing the migrator, the location is %FUSION_OLD%\var\upgrade\import-files\.