Upgrade Planning

Upgrading AEM is an extensive task. Depending on the size of your deployment, planning needs to be undertaken in order to ensure the success of the upgrade. For more information, see Planning Your Upgrade.

Upgrading Steps for JAR Based Installations

Note:

Before starting the preparation steps, make sure you run the source instance first by executing it with the java -jar aem-quickstart.jar command. This is required in order to make sure that the quickstart.properties file is generated properly. If it is missing, the upgrade will not work.

Alternatively, you can check whether the file is present by looking under crx-quickstart/conf in the installation folder of the source instance.

Pre-Upgrade Maintenance Optimization

Because of the level of customization AEM allows, environments usually do not adhere to a uniform way of performing upgrades. This makes creating a standardized procedure for upgrades a difficult process.

In previous versions, it was also difficult for AEM upgrades that were stopped or that have failed to be safely resumed. This led to situations where restarting the full upgrade procedure was necessary or where defective upgrades were carried out without triggering any warnings.

In order to address these issues, Adobe has added several enhancements to the upgrade process, making it more resilient and user friendly. Pre-upgrade maintenance tasks that before had to be performed manually are being optimized and automated. Also, post-upgrade reports have been added so that the process can be fully scrutinized in the hope that any issues are found more easily.

Pre-upgrade maintenance tasks are currently spread over various interfaces which are partially or completely performed manually. The pre-upgrade maintenance optimization added to AEM 6.3 enables a unified way to trigger these tasks and be able to inspect their result on demand.

All tasks included in the pre-upgrade optimization step are compatible with all versions from AEM 6.0 onwards.

How to Set It Up

In AEM 6.3, the pre-upgrade maintenance optimization tasks come included in the quickstart. For older versions of AEM 6, they are made available through separate packages that you can download from the Package Manager.

How to Use It

The PreUpgradeTasksMBean OSGI component comes preconfigured with a list of pre-upgrade maintenance tasks that can be run all at once. You can configure the tasks by following the below procedure:

Go to the Web Console by browsing to http://serveraddress:serverport/system/console/configMgr

Search for "preupgradetasks", then click on the first matching component. The full name of the component is com.adobe.aem.upgrade.prechecks.mbean.impl.PreUpgradeTasksMBeanImpl

Modify the list of maintenance tasks that need to run as shown below:

The task list differs depending on the run mode that is being used to start the instance. Below is a description of the run mode each maintenance task is designed for.

Task

Run Mode

OSGi Configuration

TarIndexMergeTask

crx2

-

DataStoreGarbageCollectionTask

crx2

-

WorkflowPurgeTask

crx2/crx3

Adobe Granite Workflow Purge Configuration

ConsistencyCheckTask

crx2

-

RevisionCleanupTask

crx3

-

com.day.cq.audit.impl.AuditLogMaintenanceTask

crx3

Audit Log Purge Scheduler

GenerateBundlesListFileTask

crx2/crx3

-

Caution:

DataStoreGarbageCollectionTask is calling a Datastore Garbage Collection operation with the mark and sweep phase if used. For deployments that use a shared datastore make sure to either reconfigure it or properly or prepare the instance to avoid deletion of items referenced by another instance. This might require running the mark phase manually on all instances before triggering this pre-upgrade task.

The WorkflowPurgeTask and com.day.cq.audit.impl.AuditLogMaintenanceTask tasks require a configuration and will not work without one. If they fail, the missing configuration is most likely the reason.

Therefore, make sure to add OSGI configuration for these tasks or remove them altogether from the pre-upgrade optimization tasks list if you do not wish to run them.

Default Configuration of the Pre-Upgrade Health Checks

The PreUpgradeTasksMBeanImpl OSGI component comes pre-configured with a list of pre-upgrade health check tags to execute when the runAllPreUpgradeHealthChecks method is called:

system - the tag used by the granite maintenance health checks

pre-upgrade - this is a custom tag that could be added to all the health checks that you can set to run before an upgrade

The list is editable. You can use the plus (+) and minus (-) buttons besides the tags to add more custom tags, or remove the default ones.

Going to the JMX Console at http://serveraddress:serverport/system/console/jmx

Search for PreUpgradeTasks and click the result.

Select any method from the Operations section and select Invoke in the following window.

Below is a list of all the available methods that the PreUpgradeTasksMBeanImpl exposes:

Method Name

Type

Description

getAvailablePreUpgradeTasksNames()

INFO

Displays the list of available pre-upgrade maintenance tasks names.

getAvailablePreUpgradeHealthChecksTagNames()

INFO

Displays the list of pre-upgrade health checks tag names.

runAllPreUpgradeTasks()

ACTION

Runs all the pre-upgrade maintenance tasks in the list.

runPreUpgradeTask(preUpgradeTaskName)

ACTION

Runs the pre-upgrade maintenance task with the name given as the parameter.

isRunAllPreUpgradeTaskRunning()

ACTION_INFO

Checks if the runAllPreUpgradeTasks maintenance task is currently running.

getAnyPreUpgradeTaskRunning()

ACTION_INFO

Checks if any pre-upgrade maintenance task is currently running and returns an array containing the names of currently running tasks.

getPreUpgradeTaskLastRunTime(preUpgradeTaskName)

ACTION

Displays the exact running time of the pre-upgrade maintenance task with the name given as the parameter.

getPreUpgradeTaskLastRunState(preUpgradeTaskName)

ACTION

Displays the last running state of the pre-upgrade maintenance task with the name given as the parameter.

runAllPreUpgradeHealthChecks(shutDownOnSuccess)

ACTION

Runs all the pre-upgrade health checks and saves their status in a file named preUpgradeHCStatus.properties that is located in the sling home path. If the shutDownOnSuccess parameter is set to true, the AEM instance will be shut down, but only if all the pre-upgrade health checks have an OK status.

The properties file will be used as a precondition for any future upgrade and the upgrade process will be stopped if the pre-upgrade health check execution failed. If you want to ignore the result of the pre-upgrade health checks and launch the upgrade anyway, you can delete the file.

detectUsageOfUnavailableAPI(aemVersion)

ACTION

Lists all the imported packages that will no longer be satisfied when upgrading to the specified AEM version. The target AEM version must be given as parameter.

Note:

The MBean methods can be invoked via:

The JMX Console

Any external application that connects to JMX

cURL

Pre-Upgrade Compatibility Checks

The pre-upgrade compatibility checks allow checking existing AEM instances for their upgradability by detecting APIs in use that are not compatible with 6.3 and might potentially break after upgrade. This could serve as an assessment of the development effort that might be involved in moving to the next version.

In order to use this feature, you need to install the latest version of the pre-upgrade-tasks-content content package on the instance you are upgrading from.

You can download the packages for AEM 6 versions by accessing the links provided in this section.

The checks can be invoked using the above described PreUpgradeTasksMBean MBean.

The method to be invoked for the Pre Upgrade Compatibility check is detectUsageOfUnavailableAPI with the version you want to check against passed as a parameter, as shown here:

The list of imported packages and bundles that use incompatible APIs and will be unavailable after the upgrade will be returned in the response as shown below. The results will also be added to upgrade.log.

Other Preparation Steps

Even though the upgrade process has been designed to be as simple as possible with the automation of many pre-upgrade tasks in 6.3, any upgrade of a production environment is a significant task and still requires some preparation steps to be performed manually. More specifically, we recommend you perform the items in this checklist before proceeding with the update:

Create a full backup on the instance that is going to be upgraded. For info on how to do this, consult the Backup and Restore documentation.

Check the list of obsolete bundles that will be uninstalled during the upgrade. If your custom applications depend on any of these bundles, contact Adobe Support and ask for compatibility packages for the affected area.

Run a test suite that validates your instance, including any custom applications. After the instance and the test suite have been validated, you can reuse the test suite later on in the process to validate the upgrade.

Make sure that any hostname based mappings under /etc/map (or any other similar configurations) are compatible with the environment in which you will be testing the upgraded instance.

Preparation of the AEM Quickstart jar file

Download the new AEM jar file and use it to replace the old one in the crx-quistart folder.

Next, verify that your Java Virtual Machine (JVM) and its parameters are appropriate for this new quickstart jar. Depending on your requirements or the deployment type, you will need to run a different version of the JVM that you used for your old CQ or AEM version, or use different JVM parameters.

Content Repository Migration and Upgrade

Adobe provides a tool that can be used to migrate the repository to the new version of the Oak Segment Tar present in AEM 6.3. It is provided as part of the quickstart package.

This step is mandatory if you are upgrading from an older version of TarMK or want to switch the new Segment Tar from another type of persistence. For more information on what the benefits of the new Segment Tar are, see the Migrating to Oak Segment Tar FAQ.

The actual migration is performed using the standard AEM 6.3 quickstart jar file, executed with a new -x crx2oak option which executes the crx2oak tool in order to simplify the upgrade and make it more robust.

There are some prerequisites that need to be satisfied before running the migration:

Migration Prerequistes

Minimum Required Java version: The migration tool only works with Java versions 7 and up.

Upgraded Instance: If you are upgrading from a version older than 5.4, make sure you have performed an in-place upgrade to AEM 6.0 by following the procedure described above. For more information, see the 6.0 version of the Upgrade documentation.

Once these conditions are met, the exact procedure you need to follow is:

First, stop the instance if it is running.

Delete the old quickstart jar file and replace it with the new 6.3 one.

Unpack the new quickstart jar by running:

java -Xmx4096m -jar aem-quickstart.jar -unpack

Start the repository migration. The command you need to run depends on the migration path you are following. Below is a list of all migration combinations and their associated commands:

Caution:

If you are performing the upgrade on a Windows system where Java memory mapping is not handled correctly, please add the --disable-mmap paramater to the below commands.

For migrating instances using an Amazon S3 Data Store, please consult this article.

Check that your migration finished correctly by inspecting WARN and ERROR messages in the upgrade.log file located under crx-quickstart/logs in the AEM installation directory.

Note:

Packages in the install folder in the repository will be reinstalled after the upgrade.

Because of this, make sure to not make any changes to the instance that might overlap with packages in that folder before the upgrade.

Start AEM to bring up the instance for the inplace upgrade.

java -Xmx4096m -XX:MaxPermSize=2048m -jar aem-quickstart.jar

Note:

The upgrade process will automatically take over the correct runmodes (install options) according to your migration profile, while also retaining the older valid ones. Therefore, there is no need for chaining install options when starting the instance after an upgrade - custom runmodes still need to be passed like before.

Check that the inplace upgrade finished correctly by again inspecting WARN and ERROR messages in the upgrade.log file located under crx-quickstart/logs in the AEM installation directory.

Manually updating the helper JAR

The crx2oak helper JAR can be manually upgraded if needed, by manually replacing it with newer versions after unpacking the quickstart. Its location in the AEM installation folder is:

<aem-install>/crx-quickstart/opt/extensions/crx2oak.jar

The newest version of the CRX2Oak migration tool is available for download from the Adobe Repository at this location:

When run, the crx2oak-quickstart-extension.jar will display its version and the SHA-1 checksum, together with the same information for the crx2oak.jar. If you manually upgrade these JARs and want to report bugs related to repository migration please provide the versions and SHA-1 checksums of the files that you used to support.

Post Upgrade Check Automation

In the past, inspecting the post upgrade state of your instance required careful inspection of various logfiles, parts of the repository and the launchpad. Generating a post upgrade report can help detect defective upgrades before going live.

The main purpose of this feature is to reduce the need for manual interpretation or complex parsing logic across multiple endpoints required to qualify the success of an upgrade. The solution aims to provide unambiguous information for external automation systems to react on the success or the indentified failure of an update.

More specifically, it ensures that:

Upgrade failures detected by the upgrade framework can are centralized in a single upgrade report;

The upgrade report includes indicators about necessary manual intervention.

To accomodate this, changes have been made in the way logs are generated in the upgrade.log file.

Here is a sample report that show no errors during upgrade:

Here is a sample report that shows a bundle that was not installed during the upgrade process:

Upgrading Steps for Application Server Installations

This section describes the procedure that needs to be followed in order to update AEM for Application Server installations.

All the examples in this procedure use JBoss as the Application Server and imply that you have a working version of AEM already deployed. The procedure is meant to document upgrades performed from AEMversion 5.6 to 6.3

First, start JBoss. In most situations, you can do this by running the standalone.sh startup script, by running this command from the terminal:

You now need to change the run modes in the AEM 6.3 war file. In order to do that, first create a temporary folder that will be housing the AEM 6.3 war. The name of the folder in this example will be temp.

Once the war file has been copied over, extract its contents by running from inside the temp folder:

jar xvf aem-quickstart-6.3.0.war

Once the contents have been extracted, go to the WEB-INF folder and edit the web.xml file to change the run modes.

To find the location where they are set in the XML, look for the sling.run.modes string. Once you find it, change the run modes in the next line of code, which by default is set to author:

If you have made any InDesign Script customizations, they will not be preserved after upgrade.

Preparing Assets Customizations for Upgrade

Note:

This procedure is required only for upgrades from versions older than AEM 6.2.

Instances that have customized Assets deployments need to be prepared for the upgrade. This is needed to ensure that all customized content is compatible with the new 6.3 node structure.

You can prepare the Assets content by doing the following:

On the instance that needs to be upgraded, open CRXDE Lite by going to http://server:port/crx/de/index.jsp

Go to the following node:

/apps/dam/content

Rename the content node to content_backup. You can do this by right clicking the explorer pane in the left hand side of the window and choosing Rename.

Once the node has been renamed, create a new node named content under /apps/dam named content and set its node type to sling:Folder.

Move all the children nodes of content_backup to the newly created content node. You can do this by right clicking each children node in the explorer pane and selecting Move.

Delete the content_backup node.

Generating Asset IDs for Existing Assets

To generate asset IDs for existing assets, upgrade the assets when you upgrade your AEM instance to run AEM 6.3. This is required to enable the Assets Insights feature. For more details, see Adding Embed code.

To upgrade assets, configure the Associate Asset IDs package in the JMX console. Depending on the number of assets in the repository, migrateAllAssets may take a long time (roughly an hour for 125k assets on TarMK).

If you require asset IDs for a subset of your entire assets, use the migrateAssetsAtPath API.

For all other purposes, use the migrateAllAssets() API.

Upgrading Custom Search Forms

Custom Search Facets require some manual adjustments after the upgrade in order to function properly. For more details, see Upgrading Custom Search Forms.

Analyzing Issues With The Upgrade

This section contains some issue scenarios one might face along the upgrade procedure to AEM 6.3.

These scenarios should help to track down the root cause of issues related to ugprade and should help to identify project or product specific issues.

Repository Migration Failing

The data migration from CRX2 to Oak should be feasible for any scenario starting with Source Instances based on CQ 5.4. Make sure that you exactly follow the upgrade instructions in this document which include the preparation of the repository.xml, making sure no custom custom authenticator is started via JAAS and the instance has been checked for inconsistencies before starting the migration.

If the migration still fails you can figure out what is the root cause by inspecting the upgrade.log. If the issue is not yet known, please report it to Customer Support.

Packages and Bundles Fail to Update

In case packages fail to install during the upgrade, the bundles they contain will not be updated either.

This category of issues is usually caused by data store misconfiguration. They will also appear as ERROR and WARN messages in the error.log.

Since in most of these cases the default login may fail to work, you can use CRXDE directly in order to inspect and find the configuration problems.

Some AEM Bundles are not switching to the active state

In case of bundles not starting up you should check for any unsatisfied dependencies.

In case this problem is present but it is based on a failed package installation which led to bundles not being upgrade they will be deemed incompatible for the new version. For more info on how to troubleshoot this, see Packages and Bundles Fail to Update above.

It is also recommended to compare the bundle list of a fresh AEM 6.3 instance with the upgraded one to detect the bundles that where not upgraded. This will provide a closer scope of what to search for in the error.log.

Custom Bundles not switching to the active state

In case your custom bundles are not switching to the active state it is most likely that there is code that is not importing change API. This will most often lead to unsatisfied dependencies.

API that was removed should be marked as deprecated in one of the pervious releases. You might find instructions about a direct migration of your code in this deprecation notice. Adobe aims for semantic versioning where possible so the versions can indicate breaking changes.

It is also best to check if the change that has caused the problem was absolutely necesarry necessary and revert it if it is not. Also check if the version increase of the package export was increased more than necessary, following strict semantic versioning.

Malfunctioning Platform UI

In case of certain UI functionality that is not working properly after the upgrade, you should first check for custom overlays of the interface. Some structures might have changed and the overlay might need an update or is obsolete.

Next, check for any Javascript errors that can be tracked down to custom added extensions that are hooked to clientlibs. The same can apply for custom CSS that might be causing problems to the AEM layout.

Finally, check for misconfiguration that Javascript might not be able to deal with. This is usually the case with improperly deactivated extensions.

Malfunctioning custom components, templates or UI extensions

In most cases, the root causes for these issues are the same as for bundles that are not started or packages not being installed with the only difference that the issues start occurng when first using the components.

The way to deal with erroneous custom code is to first perfom smoke tests in order to identify the cause. Once you find it, look at the recommendations in this section of the article for ways of fixing them.

Analyzing the error.log and upgrade.log

In most situations the logs need to be consulted for errors in order to find the cause of a problem. However, in case of upgrades it is also necessary to monitor dependency issues as old bundles might not be upgraded properly.

The best way to do this is to strip down the error.log by removing of all messages that are expected to be unrelated to the issue you are facing. You can do this via tool like grep, by using:

grep -v UnrelatedErrorString

Some error messages might not be immediately explicative. In this case, looking at the context in which they occur can also help understand where the error was created. You can separate the error using:

grep -B for adding lines before the error;

or

grep -A for adding lines after.

In a few cases errors can also be found WARN messages as there can be valid cases leading to this state and the application is not always able to decide if this is an actual error. Make sure you consult these messages as well.

Twitter™ and Facebook posts are not covered under the terms of Creative Commons.