Note: This guide covers upgrades on Ambari 2.1. You should look through the table at the top of the Ever Better Upgrades post to see which HDP upgrade guide is right for you.

Upcoming releases of OneFS will support Ambari's ability to upgrade HDP across the cluster with rolling upgrades. In the meantime, current OneFS releases require jumping through a few hoops to upgrade HDP from 2.2 to 2.3. This guide will walk you through the process.

Background

There are three current OneFS versions affected: 7.2.0.3, 7.2.0.4, and 7.2.1.0. The upcoming release 7.2.1.1 will be similarly impacted. 7.2.0.3 and 7.2.0.4 would only allow upgrade from one release of HDP 2.2 to another maintenance release, eg 2.2.6 to 2.2.8. OneFS 7.2.1.0 introduced compatibility with Ambari 2.1.0 and HDP 2.3, which is the target of this post.

These steps walk through manual upgrade, based on the HDP upgrade guide. HDP 2.3.0 and 2.3.2 are both possibilities, but because of compatibility requirements of OneFS, Ambari 2.1 must be used. Also the steps can be applied to upgrades to maintenance releases such as 2.2.6 to 2.2.8, but they are specifically written for an upgrade from 2.2 to 2.3 and align with those HDP upgrade steps. The examples assume that postgresql is used for the Ambari database, and the steps were confirmed on Ambari Server installed on CentOS 6.6.

In the below steps, these variables appear:

$ONEFSHOST - This is the host name of your OneFS cluster, as registered in Ambari Server.

$STACK - This is the 2-number stack of HDP that you are upgrading to. This is "2.3" currently.

$HDPVERSION-PACKAGE - This is the 4-number release version and 4 digit package version of HDP. The two HDP releases of 2.3 that are available today are 2.3.0.0-2557 and 2.3.2.0-2950.

$VERSIONDISPLAYNAME - This is the text that appears in bold type in Ambari Server's version page. For example, "HDP-2.3.0.0".

$CLUSTERNAME - This is the name that you assigned to your cluster during installation, which appears on the top left of Ambari's web UI

Ensure Correct Pre-Install State

Because OneFS is not sending expected information to Ambari Server, in some cases no entry will be created during installation for it in the database table that tracks the state of HDP versions per host. In this sequence you will check if this is true and correct it if so.

Enter the Ambari database. From the command line of your Ambari Server machine, run the following command.

psql -U ambari

When prompted for database password input "bigdata"

Check to make sure that the OneFS host has a CURRENT entry in the host_version table.

Before the final step of "Installing a New Version" (2.2 step 6), the UI will be hung with with the status "Installing" and the progress bar will be full but blue. Clicking on "Installing" will show that all hosts are complete and green. Ambari Server is waiting to hear version information about successful installed on the OneFS host. It will look like this:

Stop Ambari Server. From the command line:

ambari-server stop

Set the OneFS host and the cluster to INSTALLED. Enter the database as described above. Then enter the following queries:

update host_version set state = 'INSTALLED' where state in ('INSTALLING', 'INSTALL_FAILED');
update cluster_version set state = 'INSTALLED' where state in ('INSTALLING', 'INSTALL_FAILED');

Exit psql as described above and then start Ambari Server with this command:

ambari-server start

Installation in the Ambari Server version page should complete at this point, transitioning to a "Perform Upgrade" button. Do not press it.

Upgrade HDP

In this section you will follow the "Preparing to Upgrade" section of the HDP upgrade guide, skipping a number of irrelevant steps for HDFS. The Ambari database will need to be updated as if OneFS has completed upgrade. The normal step to complete upgrade will not work, so that is replaced with further database edits and an ambari-server command line action.

Begin 2.3, "Preparing to Upgrade"

Complete through 2.3.5 step 1

Skip 2.3.5 steps 2 to 8.

Resume from 2.3.5 step 9.

Do not perform 2.4.1 on OneFS, but do it on other HDP hosts.

At 2.4.4 step 1, move OneFS components to the new version. First stop Ambari Server from the command line then enter the database as described above. Then enter the following queries:

update hostcomponentstate set upgrade_state = 'COMPLETE' where host_id in (select host_id FROM hosts where host_name = '$ONEFSHOST') ;
update hostcomponentstate set version = '$HDPVERSION-PACKAGE' where component_name in ('SECONDARY_NAMENODE', 'NAMENODE', 'DATANODE');

Exit psql and start Ambari Server again.

In 2.4.4, skip steps 1 to 7.

At 8(b), perform Start instead of Restart All.

8(c) is irrelevant. Skip steps 9 to 13.

Resume and complete service upgrade tasks.

Skip 2.4.20.

After 2.4.21 step 1, set all components to the current stack, move OneFS host and cluster state to UPGRADED.

update hostcomponentstate set current_stack_id = subquery.stack_id from (select stack_id from stack where stack_version = '$STACK') as subquery;
update hostcomponentstate set upgrade_state = 'NONE';
update host_version set state = 'UPGRADED' where state IN ('UPGRADING', 'INSTALLED', 'INSTALLING');
update cluster_version set state = 'UPGRADED' where state IN ('UPGRADING', 'INSTALLED', 'INSTALLING');