Note: On May 21st, 2018 an additional vulnerability identified as variant 4, called Speculative Store Bypass, was disclosed as CVE-2018-3639. Mitigations for variant 4 are not discussed in this advisory.

We are aware of a new series of issues affecting all modern CPU models which can allow for applications to potentially access information and execute code that should not be allowed. Some reports of these problems began to surface in the media starting around January 1st, but the reports were incomplete. This class of vulnerabilities is reported to affect all operating systems and CPU types, including servers, home computers, and likely mobile and embedded devices. This situation is serious and affects everyone, so we are making it a top priority to gather and communicate information as soon as we have it, and we are committed to ensuring mitigations are in place as soon as possible.

The vulnerabilities have been labeled CVE-2017-5715, CVE-2017-5753, and CVE-2017-5754 in the Common Vulnerabilities and Exposures database. They have also been given the nicknames Meltdown and Spectre.

Due to mistakes in the responsible disclosure process, some information about the issues was unintentionally disclosed early. Consequently the reporters decided to move up the formal disclosure date to January 3rd from the following week. As a result of the shortened preparation window by major vendors, most vendors did not have mitigation patches available by the disclosure date.

Patching

The following information describes patching processes for various operating systems. Please find the section below matching your operating system to learn how to apply the relevant fixes for it.

Note that in addition to the fixes provided in your operating system, hardware changes may also be needed. At this time, very little information is available as to which hardware will be fixed. See the hardware fixes section for information about progress.

Important! Due to the nature of the issue, some of the patches may negatively impact the performance of your server. This happens because the vulnerable behavior is part of a performance optimization feature. Removing the optimization helps mitigate the problem but also eliminates the performance benefit of the optimization. If you experience significant performance issues with your applications after applying updates, please check with your application vendor for advice or contact our support team so we can work with you to look for ways to counter the performance loss.

Windows Server

Microsoft has produced updates for the following supported Windows server versions effective January 3rd:

Windows Server 2016: Build 14393.2007 (KB4056890)

Windows Server 2012 R2: KB4056898

Windows Server 2008 R2: KB4056897 or KB4056894

Windows Server 2008 and 2012 (non-R2 variants) do not currently have available updates. Windows Server 2003 is End of Life and will not be updated.

To protect computer systems running Windows desktop or server operating systems, open the Windows Update application for your version of Windows. Select the option "Check for Updates" and make sure that an update matching the appropriate KB number above is included in the list of available updates. Make sure the matching updates are selected, then choose to install the updates. After installing updates you must reboot your computer.

Due to the performance impact of Windows updates, Microsoft has not fully mitigated the vulnerabilities by default. Additional actions may be required depending upon the types of services running on your server. Please see the guidance article from Microsoft Support for further steps to protect your server.

In order to fully patch Spectre variant 2, you also need to apply microcode updates. These are available for some systems inside of Windows Server 2016, but the hotfix is not available via Windows update. Please see this article for information on how to obtain these hotfixes, please see this article.

CentOS

Package updates for CentOS 7 and CentOS 6 are available as of January 4th. CentOS 5 and older are End of Life and no fixes will be provided for these versions. You should keep applying newer kernel updates as they become available to ensure you have the latest versions of Meltdown and Spectre fixes.

Currently, two packages need to be updated. You can check the version numbers in this list to verify they are up to date. The highlighted numbers are the ones that will have changed from previous package versions. Effective in May 2018, Red Hat published updated microcode_ctl packages which include Intel processor microcode updates which should be applied to enhance mitigation of Spectre variant 2.

CentOS 7

kernel: 3.10.0-693.21.1.el7

IS NOTsafe to run on the Steadfast public cloud

ISsafe to run on other types of servers

kernel-plus: 3.10.0-693.21.1.el7.centos.plus

ISsafe to run on the Steadfast public cloud

ISsafe to run on other types of servers

microcode_ctl: 2.1-29.2.el7_5.x86_64

CentOS 6

kernel: 2.6.32-696.20.1.el6

IS safe to run on the Steadfast public cloud

IS safe to run on other types of servers

microcode_ctl: 1.17-25.6.el6_9.x86_64

To check the current versions of the packages run the command:

rpm -q kernel kernel-plus microcode_ctl

To apply all available updates to your server (recommended), run the following commands:

yum clean metadatayum update

If you would rather limit the update to the specific packages relevant to this issue, run these commands instead:

yum clean metadatayum update kernel\* microcode_ctl

If you are running CentOS 7 in a VM, make sure an update for the "kernel-plus" package is available and matches the safe version number indicated above. if it does not, please run the following command first, then repeat the two commands above:

Make sure that the version numbers of the packages to be installed are greater than or equal to the numbers listed above. After the update process completes, you must reboot your server by running the reboot command for the changes to take effect. After rebooting, run the command uname -r to verify that your running kernel version matches the one in the table above.

If you are running KVM-based virtual machines on your server, you should also update the libvirt and QEMU packages. The updated versions are as follows:

CentOS 7

libvirt: 3.2.0-14.el7_4.7

qemu: 1.5.3-141.el7_4.6

CentOS 6

libvirt: 0.10.2-62.el6_9.1

qemu: 0.12.1.2-2.503.el6_9.4

To update these packages, run the following commands, or simply run a full system update as noted above:

yum clean metadatayum update libvirt\* qemu\*

It isn't necessary to reboot the server after applying these particular updates, but you must restart your virtual machines for the new protections to be effective. Don't forget to apply the full set of updates inside your virtual machines as well.

If you need any assistance with any part of this process, please contact support and we'll be happy to assist you.

CloudLinux

CloudLinux 7 and 6 patches are now available for production use. CloudLinux servers are based upon CentOS, so to apply the updates, follow the same process as with CentOS indicated above, but note that the following kernel versions should be included rather than the CentOS versions:

CloudLinux 7: 3.10.0-714.10.2.lve1.4.79.el7

CloudLinux 6 Hybrid: 3.10.0-714.10.2.lve1.4.79.el6h

CloudLinux 6: 2.6.32-896.16.1.lve1.4.51

After applying updates, you must reboot for the changes to become effective.

Debian

Debian has released an update which only addresses CVE-2017-5754 (Meltdown). Debian releases prior to 7 (Wheezy) are End of Life and will not be fixed.

You can check the version numbers in this list to verify they are up to date. The highlighted numbers are the ones that will have changed from previous package versions.

Debian 9 (Stretch): 4.9.65-3+deb9u2

Debian 8 (Jessie): 3.16.51-3+deb8u1

Debian 7 (Wheezy): 3.2.96-3

To check the current versions of the packages run the command:

dpkg -s linux-image-`uname -r` | grep Version

To apply all available updates to your server (recommended), run the following commands:

apt-get updateapt-get upgrade

If you would rather limit the update to the specific packages relevant to this issue, run these commands instead:

apt-get updateapt-get install linux-image-`uname -r`

Make sure an updated package for linux-image is included among the updates. After the update process completes, you must reboot your server by running therebootcommand for the changes to take effect. After rebooting, run the command uname -v to verify that your running kernel version matches the one in the table above.

If the portion of the version number of your linux-image package before the - symbol is substantially different than the version above, you may first need to make sure your are using the latest kernel series for your version of Debian. For this case, first run these commands, filling in the number from the list below in place of the word "version", then reboot your server:

If you need any assistance with any part of this process, please contact support and we'll be happy to assist you.

KernelCare

KernelCare is currently providing patches for CentOS 6 and 7 kernels that fix part of Spectre (CVE-2017-5753) and fix Meltdown (CVE-2017-5754). If you have automatic updates enabled (the default) you should receive the fixes automatically. If you observe any stability or behavioral issues with severs that have been patched, you may want to apply security fixes using the normal process for your operating system noted above instead. You must reboot after applying package updates.

VMware

Updated VMware ESXi releases are available for currently supported products as of January 3rd:

The recommended procedure for updating VMware is using the vSphere Update Manager. If you need assistance with applying software updates for VMware products licensed through Steadfast, please contact our support team. Don't forget to apply the relevant security updates to your virtual machine operating systems as well.

Steadfast Public Cloud

The latest information regarding our Public Cloud platform is now located in a separate article. Please check here for further information.

Hardware Patches (BIOS and Microcode Updates)

Intel provided microcode updates for all CPUs that Steadfast currently supports. This does not include Pentium 4, Pentium D, Celeron, Core 2 Quad, Core 2 Duo, or any Xeon lines prior to the E55xx series, which are considered EOL by Steadfast. If you have a server with a CPU that is not supported, please contact our sales team to arrange an upgrade.

Steadfast uses exclusively Supermicro motherboards for servers. Supermicro has disclosed a list of products for which provide CPU code fixes. See the Supermicro advisory for status information.

Currently motherboards with code updates available include the X11, X10, and some X8 series. X9 motherboard BIOS updates have not yet been made available. Steadfast provides newly ordered servers running the latest available BIOS code, which will include the relevant microcode patches if they are available.

For customers running CentOS or Windows Server 2016, it is not necessary to apply the BIOS update, provided you take OS-level action instead. OS-provided microcode updates include the latest set of Intel code for CPUs that have already been fixed, and this code will be applied each time you reboot your server after the package has been installed. A more permanent BIOS fix is optional in this situation. See the above sections for CentOS and Windows for information on how to apply the current microcode updates to your servers. Microcode updates are not applied by other supported operating systems.

Due to the logistics and risks of updating BIOS code on all servers operating in the data centers and because most of our customers run versions of Windows or Linux with OS-level microcode patching support, we have decided not to proactively update BIOS versions on any hardware that is already provisioned. If you are running a system which cannot apply OS-level microcode updates or would like extra peace of mind from having a BIOS update that contains the microcode, please reach out to support to request a version check and update. Please note that BIOS updates require downtime and may be risky, so we advise avoiding the update unless it is necessary.