Print | Rate this content

Advisory: (Revision) Linux - HPE Mellanox InfiniBand Online Firmware Upgrade Utility for Linux Does Not Update the Firmware on Certain Network Adapters When the HPE Server is Booted in “Secure Boot” Mode

SUPPORT COMMUNICATION - CUSTOMER ADVISORY

Document ID: a00041652en_us

Version: 2

Advisory: (Revision) Linux - HPE Mellanox InfiniBand Online Firmware Upgrade Utility for Linux Does Not Update the Firmware on Certain Network Adapters When the HPE Server is Booted in "Secure Boot" Mode
NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.

Release Date: 2018-02-13

Last Updated: 2018-10-23


DESCRIPTION

Document Version
Release Date
Details
2
10/23/2018
Updated Resolution section with information on functionality added to certain Linux smart component firmware versions.
1
02/13/2018
Original Document Release.

When "secure boot" mode is enabled on an HPE ProLiant server, the HPE Mellanox InfiniBand Online Firmware Upgrade Utility for Linux does not update the firmware on the InfiniBand and Ethernet adapters listed in the Scope section.

Below is an example of the message displayed when attempting to update the firmware on an HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter when the Linux firmware component is run manually from the command line:



> ./hpsetup
######################################################################
HPE Mellanox InfiniBand Online Firmware Upgrade Utility for Linux
Copyright (c) 2011 Hewlett-Packard Enterprise Development Company,L.P.
######################################################################

List of Network Adapters detected on the Server.................
[0] 0000:05:00.0 Intel Corporation
[1] 0000:81:00.0 Mellanox Technologies

If PSID or FW_Version is not found for some interfaces, please check /tmp/datadSmjj5
Interface 0000:05:00.0 is not Mellanox one.
NIC firmware update did not complete. Check log for errors.

When Smart Update Manager (SUM) is used to update the network adapter firmware on a server booted in secure boot mode, the SUM inventory process will display the message "Node up to date, No applicable component found" although the baseline includes an applicable firmware smart component.

SCOPE

Any HPE system when "secure boot" mode is enabled attempting to update the network adapter firmware using the following:

The following Linux firmware smart components are affected:

  • firmware-nic-mellanox-ethernet-only-1.0.6-1.1.x86_64.rpm (and older versions with ConnectX-4 Ethernet card support)
  • firmware-hca-mellanox-vpi-connectx4-1.0.2-1.1.x86_64.rpm (and older versions)
  • firmware-nic-mellanox-ib-cx4-cx5-1.0.0-1.1.x86_64.rpm


The following network adapters are affected:

  • HPE InfiniBand EDR 100Gb 1-port 841QSFP28 Adapter (HPE Part Number: 872725-B21)
  • HPE Apollo InfiniBand EDR 100Gb 2-port 840z Mezzanine FIO Adapter (HPE Part Number: 843400-B21)
  • HPE Ethernet 25Gb 2-port 640SFP28 Adapter (HPE Part Number: 817753-B21)
  • HPE Ethernet 25Gb 2-port 640FLR-SFP28 Adapter (HPE Part Number: 817749-B21)
  • HPE InfiniBand EDR/Ethernet 100Gb 1-port 840QSFP28 Adapter (HPE Part Number: 825110-B21)
  • HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter (HPE Part Number: 825111-B21)
  • HPE Apollo InfiniBand EDR 100Gb 2-port 841z Mezzanine Adapter (HPE Part Number: 872723-B21)
  • HPE InfiniBand FDR/Ethernet 40/50Gb 2-port 547FLR-QSFP Adapter (HPE Part Number: 879482-B21)
  • HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter (HPE Part Number: 872726-B21)
  • HPE Synergy 6410C 25/50Gb Ethernet Adapter (HPE Part Number: 868779-B21)
  • HPE Ethernet 100Gb 1-port 842QSFP28 Adapter (HPE Part Number: 874253-B21)

Note: Windows and VMware ESXi firmware smart components are not affected by this issue.

RESOLUTION

The Linux smart components for Mellanox network adapters use a user space firmware tool "mstflint". To access the network adapters firmware listed in the Scope section above, in secure boot mode, a kernel space tool (flint) along with signed kernel module (MST) is needed.

As a workaround, use HPE signed "mst" kernel module and "flint" tool from HPE MLNX-OFED Software Delivery Repository to manually update firmware on the network adapters listed in the Scope section above when the server has booted in secure boot mode. The HPE MLNX-OFED Software Delivery Repository is available at the following URL:

https://downloads.linux.hpe.com/SDR/project/mlnx_ofed/

A worked example for the firmware upgrade on an HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter (HPE Part Number: 825111-B21) is shown below for reference:

  1. To Verify SecureBoot status on the server:
    [root@localhost ~]# mokutil --sb-state
    SecureBoot enabled
  2. Subscribe to MLNX-OFED repository following MLNX-OFED SDR documentation and install the relevant RPMs by typing the following command:[root@localhost ~]# yum install mft kmod-kernel-mft-mlnx
    Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
    This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
    Resolving Dependencies
    --> Running transaction check
    ---> Package kmod-kernel-mft-mlnx.x86_64 0:4.8.0-1.rhel7u3 will be installed
    ---> Package mft.x86_64 0:4.8.0-26 will be installed
    --> Finished Dependency Resolution

    Dependencies Resolved

    ==============================================================================================================================================================
    Package Arch Version Repository Size
    ==============================================================================================================================================================
    Installing:
    kmod-kernel-mft-mlnx x86_64 4.8.0-1.rhel7u3 MLNX_OFED 12 k
    mft x86_64 4.8.0-26 MLNX_OFED 59 M

    Transaction Summary
    ==============================================================================================================================================================
    Install 2 Packages

    Total download size: 60 M
    Installed size: 132 M
    Is this ok [y/d/N]: y
    Downloading packages:
    (1/2): kmod-kernel-mft-mlnx-4.8.0-1.rhel7u3.x86_64.rpm | 12 kB 00:00:01
    (2/2): mft-4.8.0-26.x86_64.rpm | 59 MB 00:04:23
    --------------------------------------------------------------------------------------------------------------------------------------------------------------
    Total 231 kB/s | 60 MB 00:04:23
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
    Warning: RPMDB altered outside of yum.
    Installing : kmod-kernel-mft-mlnx-4.8.0-1.rhel7u3.x86_64 1/2
    Installing : mft-4.8.0-26.x86_64 2/2
    Verifying : mft-4.8.0-26.x86_64 1/2
    Verifying : kmod-kernel-mft-mlnx-4.8.0-1.rhel7u3.x86_64 2/2

    Installed:
    kmod-kernel-mft-mlnx.x86_64 0:4.8.0-1.rhel7u3 mft.x86_64 0:4.8.0-26

    Complete!

    Note: On a server installed with SUSE Linux Enterprise Server, use "zypper install mft kernel-mft-mlnx-kmp-default". The kernel mft RPM name for SLES (kernel-mft-mlnx-kmp-default) is different from that of RHEL (kmod-kernel-mft-mlnx).

  3. Start MST modules and identify device name by typing the following command:

    [root@localhost ~]# service mst start
    Starting MST (Mellanox Software Tools) driver set
    Loading MST PCI module - Success
    Loading MST PCI configuration module - Success
    Create devices
    Unloading MST PCI module (unused) - Success

    [root@localhost ~]# service mst status
    MST modules:
    ------------
    MST PCI module is not loaded
    MST PCI configuration module loaded

    MST devices:
    ------------
    /dev/mst/mt4115_pciconf0 - PCI configuration cycles access.
    domain:bus:dev.fn=0000:81:00.0 addr.reg=88 data.reg=92
    Chip revision is: 00
  4. Use the "flint" command to query the current firmware version and PSID of the device:

    [root@localhost ~]# flint -d /dev/mst/mt4115_pciconf0 q
    Image type: FS3
    FW Version: 12.21.1000
    FW Release Date: 29.10.2017
    Product Version: rel-12_21_1000
    Rom Info: type=UEFI version=14.14.22 cpu=AMD64
    type=PXE version=3.5.305 devid=4115 cpu=AMD64
    Description: UID GuidsNumber
    Base GUID: e0071bffff68d0bc 4
    Base MAC: 0000e0071b68d0bc 4
    Image VSD: N/A
    Device VSD: N/A
    PSID: HP_2190110032
    Security Attributes: N/A
  5. Download latest firmware binary for the adapter from HPE.com Support Center.
  6. Query the firmware binary to ensure that PSID matches with the device by typing the following command:

    [root@localhost ~]# flint -i fw-ConnectX4-rel-12_21_2010-825111-B21_Ax_Bx-UEFI-14.14.25-FlexBoot-3.5.305.bin q
    Image type: FS3
    FW Version: 12.21.2010
    FW Release Date: 27.11.2017
    Product Version: rel-12_21_2010
    Rom Info: type=UEFI version=14.14.25 cpu=AMD64
    type=PXE version=3.5.305 devid=4115 cpu=AMD64
    Description: UID GuidsNumber
    Base GUID: N/A 4
    Base MAC: N/A 4
    Image VSD: N/A
    Device VSD: N/A
    PSID: HP_2190110032
    Security Attributes: N/A
  7. After PSID is verified, run the following command to update the device firmware:
    [root@localhost ~]# flint -d /dev/mst/mt4115_pciconf0 -i fw-ConnectX4-rel-12_21_2010-825111-B21_Ax_Bx-UEFI-14.14.25-FlexBoot-3.5.305.bin burn

    Current FW version on flash: 12.21.1000
    New FW version: 12.21.2010

    Burning FW image without signatures - OK
    Restoring signature - OK
    -I- To load new FW run mlxfwreset or reboot machine.
  8. Reboot the server for firmware update to take effect.
  9. After the server is back online, query the device and verify firmware version by typing the following command:
    [root@localhost ~]# flint -d /dev/mst/mt4115_pciconf0 q
    Image type: FS3
    FW Version: 12.21.2010
    FW Release Date: 27.11.2017
    Product Version: rel-12_21_2010
    Rom Info: type=UEFI version=14.14.25 cpu=AMD64
    type=PXE version=3.5.305 devid=4115 cpu=AMD64
    Description: UID GuidsNumber
    Base GUID: e0071bffff68d0bc 4
    Base MAC: 0000e0071b68d0bc 4
    Image VSD: N/A
    Device VSD: N/A
    PSID: HP_2190110032
    Security Attributes: N/A

Alternatively, starting SPP release 2018.06.0, the Mellanox Linux smart components can be directly run after installing the prerequisite MFT RPMs from SDR to update Mellanox adapter firmware in secure boot mode. This functionality is added to the following Linux smart component firmware versions:

  • firmware-nic-mellanox-ethernet-only-1.0.8-2.1.x86_64.rpm and higher versions
  • firmware-hca-mellanox-vpi-connectx4-1.0.4-1.1.x86_64.rpm and higher versions
  • firmware-nic-mellanox-ib-cx4-cx5-1.0.2-1.1.x86_64.rpm and higher versions

A worked example is provided below:

  1. To Verify SecureBoot status on the server, type the following command:
    [root@localhost ~]# mokutil --sb-state
    SecureBoot enabled
  2. Subscribe to MLNX-OFED repository following MLNX-OFED SDR documentation and install the relevant RPMs by typing the following command:[root@localhost ~]# yum install mft kmod-kernel-mft-mlnx
    Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
    This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
    Resolving Dependencies
    --> Running transaction check
    ---> Package kmod-kernel-mft-mlnx.x86_64 0:4.10.0-1.rhel7u5 will be installed
    ---> Package mft.x86_64 0:4.10.0-104 will be installed
    --> Finished Dependency Resolution

    Dependencies Resolved

    ====================================================================
    Package Arch Version Repository Size ====================================================================
    Installing:
    kmod-kernel-mft-mlnx x86_64 4.10.0-1.rhel7u5 MLNX_OFED 27 k
    mft x86_64 4.10.0-104 MLNX_OFED 147 M

    Transaction Summary ====================================================================
    Install 2 Packages

    Total download size: 147 M
    Installed size: 147 M
    Is this ok [y/d/N]: y
    Downloading packages:
    (1/2): kmod-kernel-mft-mlnx-4.10.0-1.rhel7u5.x86_64.rpm |27 kB00:00:01 (2/2): mft-4.10.0-104.x86_64.rpm | 147 MB 00:04:23
    --------------------------------------------------------------------Total 231 kB/s | 147 MB 00:04:23
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
    Warning: RPMDB altered outside of yum.
    Installing : kmod-kernel-mft-mlnx-4.10.0-1.rhel7u5.x86_64 1/2 Installing : mft-4.10.0-104.x86_64 2/2
    Verifying : mft-4.10.0-104.x86_64 1/2
    Verifying : kmod-kernel-mft-mlnx-4.10.0-1.rhel7u5.x86_64 2/2

    Installed:
    kmod-kernel-mft-mlnx.x86_64 0:4.10.0-1.rhel7u5 mft.x86_64 0:4.10.0-104

    Complete!

    Note: On a server installed with SUSE Linux Enterprise Server, use "zypper install mft kernel-mft-mlnx-kmp-default". The kernel mft RPM name for SLES (kernel-mft-mlnx-kmp-default) is different from that of RHEL (kmod-kernel-mft-mlnx).
  3. Install the Linux smart component RPM for the Mellanox adapter and update firmware as shown below:
    [root@localhost ~]# rpm -ivh firmware-hca-mellanox-vpi-connectx4-1.0.4-1.1.x86_64.rpm
    Preparing... ################################# [100%]
    Updating / installing...
    1:firmware-hca-mellanox-vpi-connect
    ################################# [100%]
    [root@localhost ~]# cd /usr/lib/x86_64-linux-gnu/firmware-xxx-* [root@localhost ~]# ./setup ######################################################################
    HPE Mellanox InfiniBand Online Firmware Upgrade Utility for Linux Copyright (c) 2011 Hewlett-Packard Enterprise Development Company, L.P. ######################################################################
    MST modules:
    ------------
    MST PCI module loaded
    MST PCI configuration module loaded

    MST devices: ------------
    /dev/mst/mt4115_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:37:00.0 addr.reg=88 data.reg=92 Chip revision is: 00
    /dev/mst/mt4119_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:12:00.0 addr.reg=88 data.reg=92 Chip revision
    is: 00

    Starting MST (Mellanox Software Tools) driver set [warn] mst_pci is already loaded, skipping [warn] mst_pciconf is already loaded, skipping

    Create devices

    SecureBoot is enabled.

    List of Network Adapters detected on the Server.................
    [0] 0000:02:00.0 Broadcom Limited
    [1] 0000:12:00.0 Mellanox Technologies
    [2] 0000:37:00.0 Mellanox Technologies

    If PSID or FW_Version is not found for some interfaces, please check /tmp/dataEJMPz4

    Interface 0000:02:00.0 is not Mellanox one.

    Mellanox card info for 0000:12:00.0 = FW_VERSION 16.22.1402, BUS_INFO 0000:12:00.0, PSID HPE0000000009 0000:12:00.0
    DEVICE INFO---------->15b3 1017 1590 256
    pciIdString-------------->15B3-1017-1590-0256-HPE0000000009

    Mellanox card info for 0000:37:00.0 = FW_VERSION 12.21.1000, BUS_INFO 0000:37:00.0, PSID HP_2190110032 0000:37:00.0
    DEVICE INFO---------->15b3 1013 1590 c8
    pciIdString-------------->15B3-1013-1590-00C8-HP_2190110032

    Repository has New firmware version for interface 0000:12:00.0.................Please flash the newer version

    Current Firmware Version is 16.22.1402 on 0000:12:00.0
    Repository has these firmware versions............
    [1] Image Version 16.22.4030
    [2] Image Version 16.22.4030
    Would you like to flash the firmware?
    y/n/q (y):y

    Current FW version on flash: 16.22.1402
    New FW version: 16.22.4030

    Initializing image partition - OK
    Writing Boot image component - OK
    -I- To load new FW run mlxfwreset or reboot machine.
    Firmware Flashed: SUCCESS for interface 0000:12:00.0

    Repository has New firmware version for interface 0000:37:00.0.................Please flash the newer version

    Current Firmware Version is 12.21.1000 on 0000:37:00.0
    Repository has these firmware versions............
    [1] Image Version 12.22.4030
    [2] Image Version 12.22.4030
    Would you like to flash the firmware?
    y/n/q (y):y

    Current FW version on flash: 12.21.1000 New FW version: 12.22.4030

    Burning FW image without signatures - OK
    Restoring signature - OK
    -I- To load new FW run mlxfwreset or reboot machine.
    Firmware Flashed: SUCCESS for interface 0000:37:00.0

    Please Reboot node for new image to be loaded into silicon.
  4. Reboot the server for firmware update to take effect.
  5. After the server is back online, query the device and verify firmware version by typing the following command:
    [root@localhost ~]# flint -d /dev/mst/mt4115_pciconf0 q
    Image type: FS3
    FW Version: 12.22.4030
    FW Release Date: 2.4.2018
    Product Version: rel-12_22_4030
    Rom Info: type=UEFI version=14.15.20 cpu=AMD64
    type=PXE version=3.5.404 cpu=AMD64
    Description: UID GuidsNumber
    Base GUID: 98f2b3ffffcc8d54 4
    Base MAC: 98f2b3cc8d54 4
    Image VSD: N/A
    Device VSD: N/A PSID: HP_2190110032
    Security Attributes: N/A

    [root@localhost ~] # flint -d /dev/mst/mt4119_pciconf0 q
    Image type: FS4
    FW Version: 16.22.4030
    FW Release Date: 2.4.2018
    Product Version: 16.22.4030
    Rom Info: type=UEFI version=14.15.20 cpu=AMD64
    type=PXE version=3.5.404 cpu=AMD64
    Description: UID GuidsNumber
    Base GUID: 040973ffffc91e78 8
    Base MAC: 040973c91e78 8
    Image VSD: N/A
    Device VSD: N/A PSID: HPE0000000009
    Security Attributes: secure-fw

    Note: In secure boot mode:
  • For HPSUM online deployment of Mellanox firmware smart components, ensure the "mft" and "kmod-kernel-mft-mlnx" RPMs from the HPE MLNX-OFED Software Delivery Repository are installed on the target node.
  • HPSUM offline mode of deployment is not supported.




RECEIVE PROACTIVE UPDATES : Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively via e-mail through HPE Subscriber's Choice. Sign up for Subscriber's Choice at the following URL: Proactive Updates Subscription Form.

NAVIGATION TIP : For hints on navigating HPE.com to locate the latest drivers, patches, and other support software downloads for ProLiant servers and Options, refer to the Navigation Tips document .

SEARCH TIP : For hints on locating similar documents on HPE.com, refer to the Search Tips Document .

To search for additional advisories related to Linux, use the following search string:

+Advisory +ProLiant -"Software and Drivers" +Linux


Hardware Platforms Affected: HPE InfiniBand EDR/Ethernet 100Gb 1-port 840QSFP28 Adapter, HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter, HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter, HPE Ethernet 10/25Gb 2-port 640SFP28 Adapter, HPE Synergy 6410C 25/50Gb Ethernet Adapter, HPE InfiniBand EDR 100Gb 1-port 841QSFP28 Adapter, HPE Apollo InfiniBand EDR Adapter, HPE Apollo InfiniBand EDR 100Gb 2-port 841z Mezzanine Adapter, HPE InfiniBand FDR/Ethernet 40/50Gb 2-port 547FLR-QSFP Adapter, HPE Ethernet 100Gb 1-port 842QSFP28 Adapter
Operating Systems Affected: Not Applicable
Software Affected: Not Applicable
Support Communication Cross Reference ID: SIK2937
©Copyright 2018 Hewlett Packard Enterprise Development LP
Hewlett Packard Enterprise Development shall not be liable for technical or editorial errors or omissions contained herein. The information provided is provided "as is" without warranty of any kind. To the extent permitted by law, neither HPE nor its affiliates, subcontractors or suppliers will be liable for incidental, special or consequential damages including downtime cost; lost profits; damages relating to the procurement of substitute products or services; or damages for loss of data, or software restoration. The information in this document is subject to change without notice. Hewlett Packard Enterprise Development and the names of Hewlett Packard Enterprise Development products referenced herein are trademarks of Hewlett Packard Enterprise Development in the United States and other countries. Other product and company names mentioned herein may be trademarks of their respective owners.

Legal Disclaimer: Products sold prior to the November 1, 2015 separation of Hewlett-Packard Company into Hewlett Packard Enterprise Company and HP Inc. may have older product names and model numbers that differ from current models.

Provide feedback

Please rate the information on this page to help us improve our content. Thank you!
Document title: Advisory: (Revision) Linux - HPE Mellanox InfiniBand Online Firmware Upgrade Utility for Linux Does Not Update the Firmware on Certain Network Adapters When the HPE Server is Booted in “Secure Boot” Mode
Document ID: emr_na-a00041652en_us-4
How helpful was this document?
How can we improve this document?
Note: Only English language comments can be accepted at this time.
Please wait while we process your request.