Machine Check Error on Page Size Change - CVE-2018-12207

Executive Summary

Red Hat is aware of a microarchitectural (hardware) implementation issue that could allow an unprivileged local attacker to bypass conventional system security controls to cause system wide denial-of-service condition.

At this time, this specific flaw is only known to affect Intel-based processors. This flaw is hardware-based and requires kernel updates to remediate. This issue affects all releases of Red Hat Enterprise Linux 8 and earlier. 

Technical Details and Background 

The Machine Check Error on Page Size Change issue has been assigned CVE-2018-12207 with a severity impact of Important.

A flaw was found in computer hardware of the Intel microprocessors related to the Instruction Translation Lookaside Buffer (ITLB) that caches translations from guest (and host) virtual addresses into physical addresses, also referred to as an address-translation cache, dedicated for executable instructions. This provides a level of abstraction for applications to have a simplified view of system memory. The ITLB functionality is transparent to software and its use significantly improves performance.

As instructions are executed, their linear (or virtual) address is translated to the corresponding physical address. This mapping of virtual to physical address, for executable instructions, is cached in the ITLB.

A privileged software (operating system or Virtual Machine Monitor (VMM)) may change the page size (ex. 4KB, 2MB, 1GB) and/or other attributes in the paging structures. Generally, such paging structure changes should be followed by invalidation of the cached ITLB entries corresponding to the changed pages. However, this ITLB invalidation may be delayed. This delay offers a time window wherein an attacker could invoke instruction fetch, which will result in processor using an invalid cached address translation from ITLB to access an invalid physical address, which would result in a machine check error exception, leading the CPU to go into a hang state.

A privileged attacker inside a guest VM can control its guest page table entries and craft a malicious kernel module that is able to execute specific instructions to create inconsistent ITLB entries without properly flushing the cached ITLB entries, such that subsequent instruction fetch would find multiple incompatible ITLB entries. Such a condition will trigger an uncorrectable Machine Check Error that will bring the CPU into an unrecoverable hang state, resulting in system wide DoS scenario.

Updates to these caches require synchronization with the processor’s internal data structures requiring the use of specific processor instructions (INVLPG and CR3 modifications in microcode). An attacker can instruct a malicious guest to request a mapping for one location, switch the page size, and re-issue the request with the new page size without issuing the correct invalidation sequence required to invalidate internal CPU cache state.

Acknowlegements

The issue was found internally by Intel. Red Hat would like to thank Deepak Gupta.

Additional References

What is the IU

What is a page of memory

Red Hat Errata update policy​​​​​​​

Intel November 2019 Platform Update

Intel-SA-00210 

Missing microcode information? See: Microcode specific KCS

Missing TSX information? See: TSX specific KCS

Impact Products

The following Red Hat product versions are impacted:

Red Hat Enterprise Linux 5
Red Hat Enterprise Linux 6
Red Hat Enterprise Linux 7
Red Hat Enterprise Linux 8
Red Hat Atomic Host
Red Hat Enterprise MRG 
OpenShift Container Platform
Red Hat OpenShift Online
Red Hat Virtualization (RHV-H)
Red Hat OpenStack Platform (images shipping kernel)

While Red Hat Enterprise Linux based containers are not directly impacted by kernel issues, their security relies upon the integrity of the host kernel environment. Red Hat recommends that you use the most recent versions of your container images. The Container Health Index, part of the Red Hat Container Catalog, can always be used to verify the security status of Red Hat containers. To protect the privacy of the containers in use, you will need to ensure that the Container host (such as Red Hat Enterprise Linux or Atomic Host) has been updated against these attacks. Red Hat has released an updated Red Hat Enterprise Linux Atomic Host for this use case.

Take Action

Red Hat customers running affected versions of the Red Hat products are strongly recommended to update them as soon as erratas are available. Customers are urged to apply the appropriate updates immediately.

Updates for Affected Products

Impact

Performance Impact:

• On RHEL-7 and RHEL-8, we measured the impact to mostly be in the 0-4% range.  Latency sensitive applications with small text footprints, (e.g. network benchmarks), and applications with large text footprints, (e.g. database workloads) tended to fall into that range. 

• On RHEL-7 and RHEL-8, when the mitigation is active, KVM will also convert executable 4KB pages back to non-executable large pages in the background, to recover the performance for pages that no longer contain code. This recovery process may cause rare short pauses (~10-100’s of microseconds or more) in the guest; if the guest is running a latency sensitive or realtime workload, Red Hat recommends disabling large page recovery with the newly introduced kvm.nx_huge_page_recovery_ratio=0 module parameter.

• On RHEL-6, the performance impact may be higher because both text and data hugepages are broken down into 4K pages.

Host/bare metal system:

Unprivileged applications running on the bare metal system may not be able to modify (or control) paging structure entries. For example, unprivileged system software may not be able to update page size attribute in the paging structures and induce a machine check error exception, resulting in the DoS scenario mentioned above.

Hypervisor/Virtual guest environment:

The most important attack vector for this DoS scenario is a virtual guest inducing Machine Check Error (MCE) on the host (or hypervisor) system causing DoS to numerous guest VMs   running on the host. This is extremely severe for the cloud environments, wherein guests and code running inside guests can not be trusted.

Mitigations for this issue come in two forms

1. A hardware microcode update from Intel enables new Machine Specific Register (MSR) bit (6) in the IA32_ARCH_CAPABILITIES MSR.

IA32_ARCH_CAPABILITIES[PSCHANGE_MC_NO=6]

Presence and status (0/1) of this bit(6) indicates if the CPU is vulnerable to this DoS issue or not, and if the software algorithmic mitigations in the kernel/hypervisor are required or not.

2. The software updates to the Kernel/KVM module provide following mitigations to prevent this issue

• Make large (or huge) pages (> 4KB) non-executable. Ie. processor will not be able to execute an instruction residing in the large page (ie. 2MB, 1GB, etc.) without causing a trap into the host kernel/hypervisor.
• If a large page is non-executable and an instruction is executed on the page, KVM breaks the large page into 4KB pages and gives executable permission to 4KB pages.
• A new kvm module and command line boot parameter kvm.nx_huge_pages=force/off/auto is introduced in RHEL-7 and RHEL-8 to enable/disable this mitigation.
  • kvm.nx_huge_pages=off disable mitigations, make huge pages executable
    • The existing "mitigations=off" kernel command line flag will also disable this CVE at kernel boot time.
  • kvm.nx_huge_pages=auto     enable mitigations, only if CPU is affected
  • kvm.nx_huge_pages=force   enable mitigation
  • The /sys/module/kvm/parameters/nx_huge_pages can be used to enable or disable this CVE at runtime, for example
    • echo off > /sys/module/kvm/parameters/nx_huge_pages
    • echo auto > /sys/module/kvm/parameters/nx_huge_pages
    • echo force > /sys/module/kvm/parameters/nx_huge_pages
• A new kvm module and command line boot parameter kvm.no_huge_pages=0/1 is introduced in RHEL-6 to enable/disable this mitigation
  • kvm.no_huge_pages=0 disable mitigation
  • kvm.no_huge_pages=1 enable mitigation
  • The mitigation is disabled by default and cannot be toggled at runtime.

3. The Kernel/KVM update also introduces the sysfs interface below for users to query the system for the vulnerability and if mitigations are enabled or not.

$ cat /sys/devices/system/cpu/vulnerabilities/itlb_multihit
-Not affected
-KVM Mitigation: Split huge pages
-KVM Vulnerable: no mitigation enabled

Mitigation

The only known mitigation for Red Hat impacted products to this flaw is applying an update of the kernel package or kpatch.

Red Hat has created fixes in KVM virtualization technology that prevent guest operating systems from taking advantage of this flaw. Other virtualization vendors may implement specific fixes, please consult your virtualization providers support for an update or mitigation information.