Overview

At the beginning of 2018 a number of vulnerabilities were discovered which allow malicious user space processes to read kernel memory and malicious code in VM guests to read hypervisor memory. These vulnerabilities affect most CPU manufacturers – Intel, AMD, ARM, MIPS, etc.

The vulnerabilities were nicknamed “Spectre” and “Meltdown” and are outlined in the following CVEs:

From a CloudStack point of view the main affected components are the system VM templates. This advisory outlines the fix provided for Meltdown only in CloudStack 4.9 (no fixes are available for Spectre). CloudStack 4.11 system VM templates were patched at release time and are therefore not affected.

Affected components summary:

CVE

ACS 4.9 System VMs

ACS 4.11 System VMs

Hypervisors

Spectre 1

No fix available - upgrade to 4.11

Fix in release

Affected - consult with vendor

Spectre 2

No fix available - upgrade to 4.11

Fix in release

Affected - consult with vendor

Meltdown

Fixed - new system VM template

Fix in release

Affected - consult with vendor

Effect On CloudStack

The impact on CloudStack environments is two-fold since the vulnerabilities affect both the compute hypervisor hosts and the CloudStack system VMs.

Hypervisors

As these are low level CPU call vulnerabilities all hypervisors are affected. Hypervisor vendors have been providing patches – and may continue to do so as further analysis is carried out and potential fixes are developed. The issue with the hypervisor patches is they will potentially impact performance, something which may affect hypervisor VM density figures and/or VM guest performance. ShapeBlue therefore advise users to carry out thorough testing to determine each CloudStack environment impact before rolling these out to production. ShapeBlue can not provide further information or advise on these patches and we recommend all our community users and customers to discuss with the respective hypervisor vendors.

CloudStack System VMs

The CloudStack system VMs are also affected by Spectre and Meltdown. However since these vulnerabilities require local user access someone with malicious intent would first have to gain local access to the system VMs. Since these are locked down and secured in the first place the risk to CloudStack environments is considered low as long as general CloudStack security best practices are followed.

The CloudStack LTS branches system VMs are based on 64-bit Debian releases:

CloudStack 4.11 utilises Debian 9 “Stretch” 64-bit system VMs

CloudStack 4.9 utilises Debian 7 “Wheezy” 64-bit system VMs

CloudStack 4.11 was released in February 2018 at which point the Spectre and Meltdown fixes were already provided, and these were therefore included in the system VM templates.

However – CloudStack 4.9 utilise Debian 7 “Wheezy” system VM templates – and “Wheezy” went support end-of-life on May 31st 2018 (https://wiki.debian.org/LTS). At this point the Debian community have only provided patches for Meltdown, and there are no indications Spectre fixes will be provided. As a result ShapeBlue have made the decision to provide new CloudStack 4.9 system VM templates with only the Meltdown patch included. Our overall recommendation if full patching of the vulnerabilities is required is to upgrade to CloudStack version 4.11.

CloudStack 4.9 system VM templates / patching procedure

Whilst system VMs may be patched in-situ they will require reboots for the patches to take effect, and the ShapeBlue recommendation is therefore to update the system VM templates to ensure the Meltdown patch is permanently applied. ShapeBlue have built new system VM templates for CloudStack 4.9 for XenServer, VMware and KVM hypervisors. These can be downloaded from http://packages.shapeblue.com/systemvmtemplate/4.6/meltdown/. The new system VM templates have gone through the full test cycle and no regressions have been found.

The procedure for updating the system VM templates is as follows:

For each hypervisor type in the CloudStack environment upload the new system VM template with the following information:

Name: use a descriptive name, e.g. systemvm-<hypervisor>-4.6-meltdown

Description: add template description

Zone: pick the correct zone(s)

Hypervisor: pick the correct hypervisor

Format: VHD (XenServer) / OVA (VMware) / QCOW2 (KVM)

OS Type: Debian GNU/Linux 7.0 (64-bit) (or the highest Debian release number available in the dropdown)

Extractable: no

Password Enabled: no

Public: no

Featured: no

Routing: yes

Update the global settings for “router.template.<hypervisor>” to the same as the name configured during the template upload.

Restart the management service on all management servers.

Destroy SSVMs and CPVM instances – CloudStack management will recreate these with the new template.

Restart all networks with the “cleanup” option, which will recreate all VRs with the new system VM template.

Further information

For ShapeBlue support customers, please contact the support team for further information.