Print | Rate this content

Advisory: (Revision) Linux - KVM Virtual Machine Does Not Start and Displays "Device is ineligible for IOMMU domain attach due to platform RMRR requirement" Message on HPE ProLiant Systems with an Intel Processor

SUPPORT COMMUNICATION - CUSTOMER ADVISORY

Document ID: c04781229

Version: 1

Advisory: (Revision) Linux - KVM Virtual Machine Does Not Start and Displays "Device is ineligible for IOMMU domain attach due to platform RMRR requirement" Message on HPE ProLiant Systems with an Intel Processor
NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.

Release Date: 2016-03-30

Last Updated: 2018-06-28

DESCRIPTION

Document Version
Release Date
Details
2
03/31/2016
Removed the following: HP ProLiant Gen9-series servers with HP NICs using QLogic NX2 Ethernet Controllers and HP ProLiant Gen9-series servers with HP NICs using specific Broadcom Ethernet Controllers because the pci pass-through feature is not supported.
1
10/09/2015
Original Document Release.

HPE ProLiant systems with an Intel processor use regions of system memory to communicate device status information for management purposes. The system ROM sets up Reserved Memory Region Reporting (RMRR) structures to specify these memory regions and the devices that need to access them to the operating system.

A conflict was discovered between the Linux kernel and the HPE ProLiant System ROM which impacts the device pass-through capability available through KVM due to the use of RMRR structures. This conflict may result in unexpected system behavior for the virtual machines (VMs) which utilize PCI device assignment, also referred to as PCI or device pass-through. The use of SRIOV is not affected.

To avoid this conflict, a change was implemented in Version 3.16 of the Linux kernel and has been included in Red Hat Enterprise Linux 7.1. The result of this change is that many of the devices in the system will be restricted from utilizing pass-through to a VM.

Devices may be identified via dmesg entries as follows:

IOMMU: Setting identity map for device 0000:03:00.2
[0xbdf7f000 - 0xbdf82fff]

In this case, PCI device 0000:03:00.2 is associated with an identity mapping for memory starting at 0xbdf7f000 through the use of an RMRR record.

When an attempt is made to power on a VM with the PCI device directly assigned, the kernel will report the following dmesg entry in addition with the description of the affected device:

Device is ineligible for IOMMU domain attach due
to platform RMRR requirement. Contact your platform
vendor.

When libvirt is used for VM management, libvirt will generate a message similar to the following when attempting to start a VM with such a device:

qemu-kvm: -device vfio- pci,host=03:00.2,
id=hostdev0,bus=pci.0,addr=0x7: vfio: failed
to set iommu for container: Operation not
permitted qemu-kvm: -device vfio-pci,host
=03:00.2,id=hostdev0,bus=pci.0,addr=0x7:
vfio: failed to setup container for group
22 qemu-kvm: -device vfio-pci,host=03:00.2,
id=hostdev0,bus=pci.0,addr=0x7: vfio: failed
to get group 22 qemu-kvm: -device vfio-pci,
host=03:00.2,id=hostdev0,bus=pci.0,addr=0x7:
Device initialization failed. qemu-kvm:
-device vfio-pci,host=03:00.2,id=hostdev0,
bus=pci.0,addr=0x7: Device 'vfio-pci' could
not be initialized

This is not HPE ProLiant system-specific.

SCOPE

Any HPE ProLiant system with an Intel processor running Red Hat Enterprise Linux Server 7.1 (or later) or running Linux with a kernel Version 3.16 (or later). For a list of ProLiant servers that support Red Hat Enterprise Server 7.1, refer to the support matrix available at:

http://h17007.www1.hpe.com/us/en/enterprise/servers/supportmatrix/redhat_linux.aspx

This does not affect SR-IOV functionality. HPE Moonshot servers are not affected.

RESOLUTION

For a subset of configurations, PCI device assignment can be enabled on systems affected by this change. The resolution requires a specific ROM update, configuration and in some cases a NIC firmware update. Configurations that can support PCI device assignment include:

  • HPE ProLiant Gen8 and HPE ProLiant Gen9-series servers with GPUs
  • HPE ProLiant Gen8-series servers with HPE PCI network adapters using Intel Ethernet controllers or third-party NICs
  • HPE ProLiant Gen9-series servers with HPE network adapters using Intel Ethernet controllers
  • HPE ProLiant Gen9-series servers with HPE NICs using Mellanox ConnectX3 Controllers
  • HPE ProLiant Gen9-series servers with third-party NICs

ProLiant Gen8 and Gen9-series servers with GPUs

ProLiant Gen8 and Gen9 servers support GPU device pass-through, if the GPU and driver support pass-through, with these firmware versions:

  • For the ProLiant WS460c Gen8 server blade, update the System ROM to a version dated May 1, 2013 (5/1/2013) or later.
  • For the ProLiant DL580 Gen8, update the System ROM to version P79_1.70_02_23_2015 (or later).
  • For ProLiant Gen8-series servers, update the System ROM to a version dated June 2013 (or later).
  • For ProLiant Gen9-series servers, update the system ROM to version 1.20_08-26-2014 (19 Sep 2014) or later.

NIC pass-through with HPE ProLiant Gen8-series servers with HPE PCI network adapters using Intel Ethernet controllers or third-party NICs

ProLiant Gen8-series servers, with the exception of the HP ProLiant DL580 Gen8 server, can be configured to support device pass-through for following set of NICs:

  • HP Ethernet 1GB 2-port 361T Adapter
  • HP Ethernet 1GB 4-port 366M Adapter
  • HP Ethernet 10GB 2-port 560SFP+ Adapter
  • HP Ethernet 10GB 2-port 560M Adapter
  • HP Ethernet 10GB 2-port 561T Adapter
  • HP Ethernet 10GB 1-port P560SFP+ Adapter

Third-party NICs may also be supported if the NIC and its driver support PCI pass-through. Contact the NIC vendor to confirm support.

When using one of the above NICs or a third-party NIC, the server must be configured as follows:

  1. Update the system to the latest firmware.
  2. Install the hp-health and hp-scripting-tools RPMs. The RPMs can be obtained from the HPE Support Center or the HPE Software Delivery Repository. For example:

    https://h20392.www2.hpe.com/portal/swdepot/index.do


    Depending on the requirements for health monitors, these packages may be removed after performing the BIOS configuration.
  3. Download this file, which contains the conrep configuration information for managing individual PCI slots:

    https://downloads.hpe.com/pub/softlib2/software1/pubsw-linux/p1472592088/v95853/conrep_rmrds.xml

  4. Create a file (called exclude.dat in these instructions) with the following line:

    Replacing the "X" in "SlotX" with the number of the PCI slot containing the NIC. The Linux "lspci" command can be used to identify the physical PCI slot. The information is also available from Device Inventory tab on the iLO System Information screen.
  5. Issue the following command to disable the conflict (enabling PCI device pass-through) for the slot:
    # conrep -l -x conrep_rmrds.xml -f exclude.dat

    Output similar to the following should be displayed, depending on the specific server:

    conrep 4.1.2.0 - HP Scripting Toolkit Configuration
    Replication Program Copyright (c) 2007-2014
    Hewlett-Packard Development Company, L.P.

    System Type: ProLiant BL460c Gen8 ROM Date :
    02/10/2014 ROM Family :I31 Processor
    Manufacturer : Intel

    XML System Configuration: conrep_rmrds.xml
    Hardware Configuration:exclude.dat Global
    Restriction: [3.40 ] OK

    Loading configuration data from exclude.dat

    Conrep Return Code: 0

    Step 5 must be repeated for each PCI slot to be enabled for PCI device pass-through.
  6. Confirm the BIOS settings by using the conrep command to report the current configuration: # conrep -s -x conrep_rmrds.xml -f verify.dat

    Output similar to the following should be displayed:

    conrep 4.1.2.0 - HP Scripting Toolkit Configuration
    Replication Program Copyright (c) 2007-2014
    Hewlett-Packard Development Company, L.P.
    System Type: ProLiant BL460c Gen8 ROM Date : 02/10/2014
    ROM Family : I31 Processor Manufacturer : Intel
    XML System Configuration: conrep_rmrds.xml Hardware
    Configuration: verify.dat Global Restriction: [3.40 ]
    OK Saving configuration data to verify.dat
    Conrep Return Code: 0Confirm that only the slots configured in the previous step are listed in verify.dat with "Endpoints_Excluded".
  7. If using one of the HP-branded NICs listed above, do NOT install the "Intel Active Health System Agent for HP ProLiant Network Adapters for Linux x86_64"contained in the hp-ocsbbd RPM. This agent expects to use the RMRR area to transfer sensor and health data. This agent should NOT be run when the RMRR area is disabled. If the RPM is installed, it must be removed.
  8. Reboot the server.

NIC device pass-through with HPE ProLiant Gen9-series servers with HPE NICs using Intel Ethernet Controllers

The following instructions apply to HPE ProLiant Gen9-series servers with the following NICs:

  • HP Ethernet 1GB 2-port 361i Adapter
  • HP Ethernet 1GB 2-port 361T Adapter
  • HP Ethernet 1GB 2-port 361FLB Adapter
  • HP Ethernet 1GB 4-port 366i Adapter
  • HP Ethernet 1GB 4-port 366M Adapter
  • HP Ethernet 10GB 2-port 560FLB Adapter
  • HP Ethernet 10GB 2-port 560SFP+ Adapter
  • HP Ethernet 1GB 4-port 366FLR Adapter
  • HP Ethernet 10GB 2-port 560FLR-SFP+ Adapter
  • HP Ethernet 10GB 2-port 560M Adapter
  • HP Ethernet 10GB 2-port 561FLR-T Adapter
  • HP Ethernet 10GB 2-port 561T Adapter
  • HP Ethernet 10GB 1-port P560SFP+ Adapter
  • HP Ethernet 1GB 2-port 363i Adapter
  • HP Ethernet 1GB 1-port 364i Adapter
  • HP Ethernet 1GB 2-port 367i Adapter
  • HP Ethernet 10GB 2-port 562i Adapter

The server must be configured as follows:

  1. Update the System ROM to Version 1.30_12-24-2014 (or later).
  2. Update the firmware for the HPE NIC to Version 1.1067.0 (or later). This firmware bundle is part of firmware RPM hp-firmware-nic-intel-1.8.15-1.1.x86_64.rpm.
  3. Reboot the server and Select F9 (System Utilities) during the Power-On Self-Test (POST).
  4. Select System Configuration.
  5. Select the appropriate network adapter.
  6. Select Device Level Configuration.
  7. Select "Enabled" for "Disable HP Shared Memory features" to disable the shared memory as follows:

  8. Save the configuration change and reboot the server.

Systems running the "Intel Active Health System Agent for HPE ProLiant Network Adapters for Linux x86_64"contained in the hp-ocsbbd RPM will see errors when starting the service in this configuration. The errors do not affect the operation or functionality of the server; however, the service is no longer necessary so the package should be removed.

NIC device pass-through with HPE ProLiant Gen9-series servers with HPE NICs using Mellanox ConnectX3 Controllers.

The following instructions apply to HPE ProLiant Gen9-series servers with the following NICs:

  • HP Ethernet 10G 2-port 546FLR-SFP+ Adapter
  • HP Ethernet 10G 2-port 546SFP+ Adapter
  • HP InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+QSFP Adapter
  • HP InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter
  • HP InfiniBand QDR/Ethernet 10Gb 2-port 544+FLR-QSFP Adapter
  • HP InfiniBand QDR/Ethernet 10Gb 2P 544M Adapter
  • HP InfiniBand FDR/Ethernet 10/40Gb 2P 544M Adapter
  • HP InfiniBand QDR/Ethernet 10Gb 2-port 544+M Adapter
  • HP InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+M Adapter

The server must be configured as follows:

  1. Update the System ROM to Version 1.30_12-24-2014 (or later).
  2. Update the firmware for the HP NIC to Version 2.33.5220-Flex-3.4.467(14 Apr 2015) or later.
  3. Reboot the server and Select F9 (System Utilities) during the Power-On Self-Test (POST)
  4. Select System Configuration.
  5. Select the appropriate network adapter.
  6. Select Device Level Configuration.
  7. Select "Disable" for "HP Shared Memory features" to disable the shared memory as follows:

  8. Save the configuration change and reboot the server.

NIC device pass-through for HPE ProLiant Gen9 servers with third-party NICs.

HPE ProLiant Gen9-series servers with firmware version 1.30_12-24-2014 (or later) support PCI device pass-through with third-party NICs, assuming that the NIC and driver support PCI device pass-through. Contact the NIC vendor for support.

Perform the following steps to obtain the appropriate System ROM/NIC firmware:

  1. Click on the following link:

    http://h20566.www2.hpe.com/portal/site/hpsc

  2. Select "Product Support > Download options > Drivers and Software."
  3. Enter a product name (e.g., DL380 Gen9) and click on Search.
  4. Select the appropriate operating system.
  5. Click on the appropriate category.
  6. Locate, download, and install the appropriate System ROM/NIC firmware.

RECEIVE PROACTIVE UPDATES : Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively via e-mail through HPE Subscriber's Choice. Sign up for Subscriber's Choice at the following URL: Proactive Updates Subscription Form.

NAVIGATION TIP : For hints on navigating HP.com to locate the latest drivers, patches, and other support software downloads for ProLiant servers and Options, refer to the Navigation Tips document .

SEARCH TIP : For hints on locating similar documents on HP.com, refer to the Search Tips document .

Hardware Platforms Affected: HPE ProLiant SL230s Gen8 Server, HPE ProLiant SL250s Gen8 Server, HPE ProLiant BL460c Gen8 Server Blade, HPE ProLiant DL360p Gen8 Server, HPE ProLiant DL380p Gen8 Server, HPE ProLiant ML350p Gen8 Server, HPE ProLiant DL160 Gen8 Server, HPE ProLiant BL420c Gen8 Server Blade, HPE ProLiant DL320e Gen8 Server, HPE ProLiant DL360e Gen8 Server, HPE ProLiant ML310e Gen8 Server, HPE ProLiant ML350e Gen8 Server, HPE ProLiant DL380e Gen8 Server, HPE ProLiant BL660c Gen8 Server Blade, HPE ProLiant DL560 Gen8 Server, HPE ProLiant DL320e Gen8 v2 Server, HPE ProLiant MicroServer Gen8, HPE ProLiant DL580 Gen8 Server, HPE ProLiant DL160 Gen9 Server, HPE ProLiant DL180 Gen9 Server, HPE ProLiant DL360 Gen9 Server, HPE ProLiant BL460c Gen9 Server Blade, HPE ProLiant DL380 Gen9 Server, HPE ProLiant ML350 Gen9 Server, HPE ProLiant XL230a Gen9 Server, HPE ProLiant XL250a Gen9 Server, HPE ProLiant DL120 Gen9 Server, HPE ProLiant ML150 Gen9 Server, HPE ProLiant DL60 Gen9 Server, HPE ProLiant DL80 Gen9 Server, HPE ProLiant ML110 Gen9 Server, HPE ProLiant BL660c Gen9 Server Blade, HPE ProLiant DL560 Gen9 Server, HPE ProLiant XL450 Gen9 Server
Operating Systems Affected: Not Applicable
Software Affected: Not Applicable
Support Communication Cross Reference ID: IA04781229
©Copyright 2018 Hewlett Packard Enterprise Development LP
Hewlett Packard Enterprise Development shall not be liable for technical or editorial errors or omissions contained herein. The information provided is provided "as is" without warranty of any kind. To the extent permitted by law, neither HPE nor its affiliates, subcontractors or suppliers will be liable for incidental, special or consequential damages including downtime cost; lost profits; damages relating to the procurement of substitute products or services; or damages for loss of data, or software restoration. The information in this document is subject to change without notice. Hewlett Packard Enterprise Development and the names of Hewlett Packard Enterprise Development products referenced herein are trademarks of Hewlett Packard Enterprise Development in the United States and other countries. Other product and company names mentioned herein may be trademarks of their respective owners.

Legal Disclaimer: Products sold prior to the November 1, 2015 separation of Hewlett-Packard Company into Hewlett Packard Enterprise Company and HP Inc. may have older product names and model numbers that differ from current models.

Provide feedback
Please rate the information on this page to help us improve our content. Thank you!
  1. Was the information on this page helpful?
  2. Very helpful
  3. Somewhat helpful
  4. Not helpful
Document title: Advisory: (Revision) Linux - KVM Virtual Machine Does Not Start and Displays "Device is ineligible for IOMMU domain attach due to platform RMRR requirement" Message on HPE ProLiant Systems with an Intel Processor
Document ID: emr_na-c04781229-7
How helpful was this document?
How can we improve this document?
Note: Only English language comments can be accepted at this time.
Please wait while we process your request.