Red Hat has been made aware of multiple microarchitectural (hardware) implementation issues affecting many modern microprocessors, requiring updates to the Linux kernel, virtualization-related components, and/or in combination with a microcode update. An unprivileged attacker can use these flaws to bypass conventional memory security restrictions in order to gain read access to privileged memory that would otherwise be inaccessible. There are 3 known CVEs related to this issue in combination with Intel, AMD, and ARM architectures. Additional exploits for other architectures are also known to exist. These include IBM System Z, POWER8 (Big Endian and Little Endian), and POWER9 (Little Endian).

Background Information

An industry-wide issue was found with the manner in which many modern microprocessor designs have implemented speculative execution of instructions (a commonly used performance optimization). There are three primary variants of the issue which differ in the way the speculative execution can be exploited. All three rely upon the fact that modern high performance microprocessors implement both speculative execution, and utilize VIPT (Virtually Indexed, Physically Tagged) level 1 data caches that may become allocated with data in the kernel virtual address space during such speculation.

The first two variants abuse speculative execution to perform bounds-check bypass (CVE-2017-5753), or by utilizing branch target injection (CVE-2017-5715) to cause kernel code at an address under attacker control to execute speculatively. Collectively these are known as "Spectre". Both variants rely upon the presence of a precisely-defined instruction sequence in the privileged code, as well as the fact that memory accesses may cause allocation into the microprocessor’s level 1 data cache even for speculatively executed instructions that never actually commit (retire). As a result, an unprivileged attacker could use these two flaws to read privileged memory by conducting targeted cache side-channel attacks. These variants could be used not only to cross syscall boundary (variant 1 and variant 2) but also guest/host boundary (variant 2).

The third variant (CVE-2017-5754) relies on the fact that, on impacted microprocessors, during speculative execution of instruction permission faults, exception generation triggered by a faulting access is suppressed until the retirement of the whole instruction block. Researchers have called this exploit "Meltdown". Subsequent memory accesses may cause an allocation into the L1 data cache even when they reference otherwise inaccessible memory locations. As a result, an unprivileged local attacker could read privileged (kernel space) memory (including arbitrary physical memory locations on a host) by conducting targeted cache side-channel attacks.

Acknowledgements

Red Hat would like to thank Google Project Zero for reporting these flaws.

While Red Hat's Linux Containers are not directly impacted by kernel issues, their security relies upon the integrity of the host kernel environment. Red Hat recommends that you use the most recent versions of your container images. The Container Health Index, part of the Red Hat Container Catalog, can always be used to verify the security status of Red Hat containers. To protect the privacy of the containers in use, you will need to ensure that the Container host (such as Red Hat Enterprise Linux or Atomic Host) has been updated against these attacks. Red Hat has released an updated Atomic Host for this use case.

Attack Description and Impact

The attacks described in this article abuse the speculative execution capability of modern high-performance microprocessors. Modern microprocessors implement a design optimization known as “Out-of-Order” (OoO) execution, meaning the microprocessor will begin to execute independent instructions as soon as their data dependencies become available (so-called “data flow” model) rather than always executing instructions in the literal order provided by the programmer through their application binary. The illusion of a sequential execution is maintained within the processor by means of various internal reordering structures that buffer these intermediate execution states of the processor and present the in-order results. Out-of-Order (OoO) Execution was originally invented by Robert Tomasulo in 1967 for use in the early IBM mainframe systems. In the intervening decades, it has become a standard feature of nearly all microprocessors.

An extension of the Out-of-Order execution model adds highly sophisticated branch prediction algorithms that aim to predict whether a given path (branch) of software code will be executed. A branch can be thought of as changing the flow of instructions being executed by the processor in response to an “if” statement of the form “if this, then do A, otherwise do B”. The condition upon which a branch is taken or not taken often depends upon data that may not immediately be available (for example, because it requires a load from slower RAM into the internal microprocessor cache hierarchy). Since the branch condition may not be ready in a timely manner, the processor may begin to speculatively execute the most likely path, based on input from the branch predictor. Results from this execution are stored in such a manner that the entire path can be discarded if the speculated branch direction later turns out to be incorrect. Thus, speculative execution is normally completely invisible to the programmer, or to other users of the same system.

The attacks described in this article rely upon breaking open the black box that is the internal state of the microprocessor during speculative execution. In particular, the attacks rely upon a technique known as cache side-channel analysis. During speculative execution, a processor will not intermediately make results available in memory or registers visible to the programmer, to other processors, or to other running applications. Yet, in order to access memory within speculated code paths, it must bring the data in the processor cache. Side-channel analysis allows an attacker to observe speculated allocations (loads) into the system caches, even those coming from execution paths that ultimately have been discarded. A specially crafted program can then be designed to speculatively perform loads into the cache from privileged memory locations, and monitor the results which can be used to infer the content of that privileged memory.

One case that triggers CPU speculative execution is branches. An attacker can start by training the branch predictor that a particular branch in kernel code is heavily taken (or not taken). The next time the branch executes, the processor will start executing the code in the direction chosen by the attacker. If the attacker chooses a path that loads a value from memory, such load will be executed speculatively. Attacks against branch prediction can (in some affected microprocessor implementations) be extended across the kernel/hypervisor boundary, allowing unprivileged guest Operating Systems to exert influence over the execution of the hypervisor and, when combined with side-channel analysis, to extract sensitive hypervisor memory.

The effects of speculative execution however can be even more wide-ranging. Because the internal state of the microprocessor is not visible to the programmer, or to other users or applications running on the system, the processor may perform speculative data accesses even before checking whether they are permitted. Permission checks will occur in parallel and ultimately trigger an abort of the speculation prior to retiring the speculated instructions and making their execution results visible outside the processor. If the processor speculatively uses cached data from memory prior to completing the permission checks, then it becomes possible to observe that data by using it in subsequent memory accesses.

One example of such permission checks is page access checks from the memory management unit (MMU). Paging, also known as virtual memory, is a common feature of high performance microprocessors; it lets the operating system control the mapping of virtual addresses into the physical addresses of the system RAM, and also limit accesses to the virtual addresses through access control bits. For example, a page can be marked as “read-only” (so that writes cause a page fault exception) or as “kernel memory” (so that user-mode accesses cause a page fault exception).

Because the processor’s permission checks enforce that user applications cannot access kernel memory, it is standard practice in the industry for operating system kernels (including Linux) to map kernel virtual memory addresses into the same address space as user applications. Doing so creates a significant performance advantage since applications make frequent use of kernel-provided system calls, and switching address spaces during each system call would incur a significant performance overhead. Each switch would require flushing (invalidating) the content of many internal CPU structures, such as TLBs (Translation Lookaside Buffers) that cache virtual-to-physical memory translations and accelerate the use of virtual memory.

Sharing the page tables between the kernel and user applications, however, enables another kind of attack against speculative execution. In this case, the preparatory phase has the attacker trick the kernel into loading an “interesting” virtual address into the processor’s Level 1 (L1) data cache. The L1 data cache is commonly organized using a technique known as VIPT (Virtually Indexed, Physically Tagged), which lets the virtual-to-physical address translation and permission checks occur in parallel with access. In the presence of a shared virtual address space, kernel privileged virtual addresses might be accessed speculatively through the L1 cache by untrusted user code during speculative execution, and the loaded values can be used further down the speculatively-executed path. A second speculative memory access can thus fill the cache in a manner that depends on privileged memory contents, and the effects can be observed to derive those contents.

These microprocessor side-channel attacks may allow an untrusted user with access to a machine to extract sensitive information from privileged kernel or hypervisor memory, as well as from other applications or virtual machines running on the same system. Mitigation involves a number of discrete steps, some or all of which may be required depending upon the exact make and model of the microprocessor, each of which may be vulnerable to a differing extent to each variant of the attack:

Separating the kernel and user virtual address spaces: this is performed using a design change to the Operating System kernel known as KPTI (Kernel Page Table Isolation), sometimes referred to using the older name “KAISER”.

Disabling indirect branch prediction upon entry into the kernel or into the hypervisor: New capabilities have been added to many microprocessors across the industry through microcode, millicode, firmware, and other updates. These new capabilities are leveraged by updates to Red Hat Enterprise Linux which control their use.

Fencing speculative loads of certain memory locations: Such loads have to be annotated through small changes to the Linux kernel, which have been integrated into Red Hat updates.

These software solutions, in combination with microcode, millicode, and firmware updates can mitigate the attacks described in this article. However, the mitigation comes at some cost to system performance. Depending upon the specific system, make, and model of the microprocessors, as well as the characteristics of the workloads, the performance impact can be significant. Red Hat is taking a proactive position that favors security over performance, while allowing users the flexibility to assess their own environment and make appropriate tradeoffs through selectively enabling and disabling the various mitigations.

The Red Hat Performance Engineering team has created a Knowledgebase article reporting observed performance impact for a variety of representative workloads, and describing options users can take to disable parts of the security fixes to regain the desired level of performance if the customer is confident their computers are physically isolated. Additional performance data will be published as this incident develops.

The Red Hat Performance Engineering and Product Engineering teams have developed a Knowledgebase article detailing the tunables available to selectively enable or disable these new security features.

Detection & Diagnosis

Determine if your system is vulnerable

Red Hat has created a Labs app to assist in detection of exposure to these vulnerabilities. Subscribers can use the following link to access our Labs tool.

Determine if your system is vulnerable. Use the detection script to determine if your system is currently vulnerable to this flaw. To verify the legitimacy of the script, you can download the detached GPG signature as well. The current version of the script is 2.3.

For subscribers running Red Hat Virtualization products, a Knowledgebase article has been created to verify OEM-supplied microcode/firmware has been applied.

Take Action

Red Hat customers running affected versions of the Red Hat products are strongly recommended to update them as soon as errata are available. Customers are urged to apply the appropriate updates immediately. All impacted products should apply fixes to mitigate all 3 variants; CVE-2017-5753 (variant 1), CVE-2017-5715 (variant 2), and CVE-2017-5754 (variant 3).

Due to the nature of changes required, a kpatch for customers running Red Hat Enterprise Linux 7.2 or greater will NOT be available.

Variant 2 can be exploited both locally (within the same OS) and through the virtualization guest boundary. Fixes require CPU microcode/firmware to activate. Subscribers are advised to contact their hardware OEM to receive the appropriate microcode/firmware for their processor. An additional Knowledgebase article is available with more details.

Carefully read the text of the Security Errata. Not all architectures had mitigations released initially. The x86_64 mitigations do not include Red Hat Enterprise Linux 6 32bit kernels.

For customers using Red Hat Satellite 6, an additional Satellite 6 Knowledgebase article is also available. NOTE: Please use the table below to help quickly identify the specific RHSA ID's that you want to apply to your systems.

*An active ELS subscription is required for access to this patch. Please contact Red Hat sales or your specific sales representative for more information if your account does not have an active ELS subscription.

**An active EUS subscription is required for access to this patch. Please contact Red Hat sales or your specific sales representative for more information if your account does not have an active EUS subscription.

****An active TUS subscription is required for access to this patch in RHEL TUS.

***** Subscribers should contact their hardware OEMs to get the most up-to-date versions of CPU microcode/firmware.

Mitigation

There is no known mitigation, other than applying vendor software updates combined with hardware OEMs CPU microcode/fiirmware. All Red Hat customers should apply vendor solutions to patch their CPUs, and update the kernel as soon as patches are available.

Customers are advised to take a risk-based approach in mitigating this issue. Systems that require high degrees of security and trust should be addressed first, and should be isolated from untrusted systems until such time as treatments can be applied to those systems to reduce the risk of exploit.

Customers running Xen hypervisors should be aware of technical limitations of that software that cannot completely eliminate the variant 2 exploit, and cannot eliminate the variant 3 exploit on paravirtualized guests. Red Hat has prepared a Knowledgebase article detailing the Xen situation and options available to Xen hypervisor users.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

We will be pushing errata out as soon as they have passed our QA team's testing. The more modern versions were easier to backport patches from upstream, and as you progress backwards the fixes change from a backporting exercise into a complete rewrite. We expect all packages for RHEL7 to be available shortly, with RHEL6 following closely behind. Our layered products will be respinning now that the signed packages are available, so products like OpenStack or RHEL Atomic will be available within a few days. RHEL5 should be complete in the next week or so, but working with 10+ year old code we are very cautious about ensuring there are no unforeseen regressions the newer code could introduce. Our goal is to provide the necessary fixes and be as minimally invasive as possible to that older code to ensure that it continues to work consistently and stably as our subscribers expect it to.

the first fix supplied is for virtual guest containers.
Does this mean that bare metal installs are less affected by the bug?
Also the kernel fix available for RHEL 6 is version 6.9 and my customers still rely on RHEL 6.7. How can this be resolved bearing in mind we do not have extended support.

Hi,
In regards to the concern over 6.7 access, this is available to all customers with an active EUS contract. Please contact Red Hat Sales, if you wish to inquire for.
Bare metal and Virt systems are both impacted by this, in slightly different variations, where CVE-2017-5715 requires changes to multiple (kernel, microcode and various virtualization) RPMS to be applied to both guest and host systems.

Hi there Chris. Our Engineering teams have compiled the tunable options for this issue into a swanky new KCS for your reading pleasure: https://access.redhat.com/articles/3311301 (also cited in the References and Impact section of this article).

Could you refer to something to read more about this:
"Additional exploits for other architectures are also known to exist. These include IBM System Z, POWER8 (Big Endian and Little Endian), and POWER9 (Little Endian)."

Adding my vote for more clarity regarding Power. IBM is currently silent on this and we are trying to determine how exposed our Power systems are. Currently, Red Hat is the only source of any information my team can find regarding Power and these exploits.

Saad - RHEL4 has been out of support March 31, 2017 and will receive no further updates. You will need to move to a more modern platform to be able to get fixes for this issue. Technologically RHEL4 and the modern Linux kernel are drastically different and make it incredible difficult to duplicate fixes even if it was in support.

I see where new errata is available for the servers I am responsible for. However, getting downtime to reboot is rare, but not impossible. Do you foresee the current errata will fix the kernel side-channel attacks or do you expect there will be other kernel updates within a week or so to fix additional issues that may appear after these updates are applied? I know you don't have a crystal ball to see into the future, but as I stated, getting downtime is rare for me.

The current patches will eliminate the exposure to the vulnerability. It depends on your risk tolerance, patching policies, and change management processes if you decide to wait a bit longer. You might, as a business, decide that completely mitigating this risk does not justify the performance hit your systems might incur. No functional changes are slated for the near future. As we have more time and now the ability to collaborate with the larger Linux community we will be able to refine the fixes and improve performance in future updates. We're currently testing POWER-based patches, which should be available in a respin we're planning in a future release in the near future.

What happened to the microcode_ctl package lines that were in the "Resolve" chart above earlier today? Is this package being re-built? There's no longer a line with a reference to the proper RHSA to resolve this package in the chart above. The latest package version I see in the downloads area for RHEL6 is 1.17-25.2.el6_9 from Dec 15th in the changelog.

Subscribers are advised to contact their hardware OEM to receive the appropriate microcode/firmware for their processor.
Given the tight timeframe that incident was released, Red Hat was not provided a complete set of microcode to cover all affected CPUs. We will be providing both microcode_ctl and linux_firmware that covers the limited subset of chipsets we were able to test, but this will NOT address many CPUs that you may have in use in your server fleet. Again, contacting your hardware vendor will ensure you have the appropriate software to enable the protections for Variant 2 of this issue.

For clarification, since people keep interchanging the words microcode and firmware, and I'm not quite 100% clear on what the microcode_ctl package does vs. BIOS/EFI-level firmware code. Do you mean getting another microcode_ctl-specific package from the hardware vendor that installs at the OS/Linux level, or are you talking about hardware BIOS/EFI-level firmware that updates that are independent from the OS/Linux?

Or in other words -- if I update the kernel and update the regular BIOS/EFI-level firmware to an approved level, is that sufficient to resolve this, or does there still need to be some microcode_ctl-specific package update from the vendor, in addition to the BIOS/EFI-level firmware update?

As far as I can see under the 'Resolve' tab, 6.3 is not even in the list! Does that mean 6.3 is not affected by either of the malware and needs no precautionary steps/updates to be performed?
Or, is the update simply not supported and should upgrade the version (to i.e, 6.4 which supports this meltdown and spectre patches)?

Let's better say that all x86_64 CPUs are affected (Intel with both Spectre/Meltdown and AMD/ARM just with Spectre). So 6.3 is for sure impacted as well. In general, all kernel (and other affected packages) versions prior to errata versions from 2018-01-03 are affected because there is no fix for these vulnerabilities.

Hello Jaroslav, Thank you for the response. Indeed, you are right. I just finished my studies on the malware.
However, I'm still wondering how to fix this 6.3 virtual box as I'm unable to find any official updates for the same under the resolve tab!

As there is no active update stream for Red Hat Enterprise Linux 6.3 (or 6.0, 6.1, or 6.8) it is not listed. The Resolve tab only lists version with active update stream that will have updates released. If you're still on 6.3, you should upgrade to a newer version that is still supported and where you have active subscription for the relevant update stream.

I have two issues-
1 - I took over a RHEL 6.9 VM server running on Hypr-V that had been built before I started. I just noticed that microcode_ctl is not installed on the server. It has been running fine for the past 3-4 years without it. Should I be concerned?.

2 - I updated the latest package available including dracut, linux-firmware, microcode_ctl and kernel on 2 RHEL 7.4 servers. I rebooted after updating the linux-firmware and also after updating the microcode_ctl. no issues. after I updated the kernel - it went straight to the grub > prompt.
This was an issue/bug Red Hat Bugzilla – Bug 1398698
with the microcrode_ctl in 7.3 where it was intermittently work or not work. This was fixed in RHBA-2017:0028.
to get around the issue i would boot into rescue mode. I would verify that everything was ok which it was. then exit out and boot normally
this would always fix the issue.
Out of both servers 1 worked 1 did not. - these 2 servers were VM's
I have other physical RHEL7 servers that i test this out on.
Has anyone else had this issue happen yet. again?