Print | Rate this content

Advisory: HPE Network Adapters - Mellanox Ethernet + RoCE And OFED Driver Installation Conflicts With Broadcom NetXtreme-E Driver On HPE ProLiant Servers Running SUSE Linux Enterprise Server 12 SP3 (Or Later) And Configured With Certain Network Adapters

SUPPORT COMMUNICATION - CUSTOMER ADVISORY

Document ID: a00030192en_us

Version: 1

Advisory: HPE Network Adapters - Mellanox Ethernet + RoCE And OFED Driver Installation Conflicts With Broadcom NetXtreme-E Driver On HPE ProLiant Servers Running SUSE Linux Enterprise Server 12 SP3 (Or Later) And Configured With Certain Network Adapters
NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.

Release Date: 2017-11-06

Last Updated: 2017-11-06


DESCRIPTION

On any HPE ProLiant server running SUSE Linux Enterprise Server 12 SP3 (or later), and configured with any of the network adapters listed in the Scope section below, the installation of the Mellanox Ethernet + RoCE driver or MLNX-OFED driver may fail if the Broadcom NetXtreme-E driver is already installed on the node as follows:

When attempting to install MLNX-OFED v4.1-1.0.2.

> ./mlnxofedinstall
Detected sles12sp3 x86_64. Disabling installing 32bit rpms...
Logs dir: /tmp/MLNX_OFED_LINUX-4.1-1.0.2.0.21505.logs
Verifying KMP rpms compatibility with target kernel...
Warning: libgfortran3 rpm is required to run openmpi
libgfortran3 is available on SLES12 SDK DVD
This program will install the MLNX_OFED_LINUX package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with MLNX_OFED_LINUX, do not reinstall them.
Do you want to continue?[y/N]:y

Starting MLNX_OFED_LINUX-4.1-1.0.2.0 installation ...

Installing mlnx-ofa_kernel RPM
Preparing... ########################################
Updating / installing...
mlnx-ofa_kernel-4.1-OFED.4.1.1.0.2.1.g########################################
Installing mlnx-ofa_kernel-kmp-default 4.1 RPM
Failed to install mlnx-ofa_kernel 4.1 RPM
Collecting debug info...
See /tmp/MLNX_OFED_LINUX-4.1-1.0.2.0.21505.logs/mlnx-ofa_kernel-kmp-default-4.1.rpminstall.log

When attempting to install MLNX Ethernet + RoCE driver v4.1.

> rpm -Uvh mlnx-ofa_kernel-4.1-OFED.4.1.1.0.2.1.gc22af88.2.sles12sp3.x86_64.rpm mlnx-ofa_kernel-kmp-default-4.1_k4.4.73_5-OFED.4.1.1.0.2.1.gc22af88.sles12sp3.x86_64.rpm
warning: mlnx-ofa_kernel-4.1-OFED.4.1.1.0.2.1.gc22af88.2.sles12sp3.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 26c2b797: NOKEY
Preparing... ################################# [100%]
file /lib/modules/4.4.73-5-default/updates/drivers/infiniband/hw/bnxt_re/bnxt_re.ko from install of mlnx-ofa_kernel-kmp-default-4.1_k4.4.73_5-OFED.4.1.1.0.2.1.gc22af88.sles12sp3.x86_64 conflicts with file from package bnxt_en-kmp-default-1.8.23_k4.4.73_5-1.sles12sp3.x86_64

This issue occurs because Mellanox driver installation attempts to add a "dummy" NetXtreme RoCE Driver kernel module on SUSE Linux Enterprise Server 12 SP3 (or later) as follows:

filename: /lib/modules/4.4.73-5-default/updates/drivers/infiniband/hw/bnxt_re/bnxt_re.ko
version: 2.0.0
license: Dual BSD/GPL
description: bnxt_re dummy kernel module
author: Alaa Hleihel
srcversion: 8E54120CD1E13F234131F03
depends: mlx_compat
supported: external
vermagic: 4.4.73-5-default SMP mod_unload modversions
signer: Hewlett-Packard Company: HP UEFI Secure Boot 2013 DB key
sig_key: 1D:7C:F2:C2:B9:26:73:F6:9C:8E:E1:EC:70:63:96:7A:B9:B6:2B:EC
sig_hashalgo: sha256

From the Mellanox driver perspective, adding this dummy module is necessary to ensure compatibility with other kernel modules installed by MLNX-OFED; however, Broadcom NetXtreme RoCE driver installation already updates the "bnxt_re" kernel module to a newer version as follows:

# modinfo bnxt_re
filename: /lib/modules/4.4.73-5-default/updates/drivers/infiniband/hw/bnxt_re/bnxt_re.ko
version: 20.8.0.5
license: Dual BSD/GPL
description: Broadcom NetXtreme RoCE Driver
author: Eddie Wai <eddie.wai@broadcom.com>
srcversion: EB0249985EC34E2C6592D3D
depends: ib_core,configfs
supported: yes
vermagic: 4.4.73-5-default SMP mod_unload modversions
signer: Hewlett-Packard Company: HP UEFI Secure Boot 2013 DB key
sig_key: 1D:7C:F2:C2:B9:26:73:F6:9C:8E:E1:EC:70:63:96:7A:B9:B6:2B:EC
sig_hashalgo: sha256
parm: restrict_mrs: Restrict the no. of MRs 0 = 256K , 1 = 64K (uint)

When this occurs, the Mellanox driver RPM fails to add the "dummy" NetXtreme RoCE Driver kernel module, as it conflicts with Broadcom NetXtreme RoCE Driver kernel module.

SCOPE

Any HPE ProLiant server running SUSE Linux Enterprise Server 12 SP3 (or later), and configured with any of the following network adapters:

  • Mellanox network adapters:
    • HPE Ethernet 10/25Gb 2-port 640SFP28 Adapter
    • HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter
    • HPE InfiniBand EDR/Ethernet 100Gb 1-port 840QSFP28 Adapter
    • HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter
    • HPE Ethernet 10Gb 2-port 546SFP+ Adapter
    • HPE Ethernet 10Gb 2-port 546FLR-SFP+ Adapter
    • HPE InfiniBand QDR/Ethernet 10Gb 2-port 544+M Adapter
    • HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+M Adapter
    • HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+QSFP Adapter
    • HPE InfiniBand QDR/Ethernet 10Gb 2-port 544+FLR-QSFP Adapter
    • HPE InfiniBand QDR/Ethernet 10Gb 2-port 544+FLR-QSFP Adapter
    • HP InfiniBand QDR/EN 10Gb Dual Port 544M Adapter
    • HP InfiniBand FDR/EN 10/40Gb Dual Port 544M Adapter
    • HP InfiniBand FDR/Ethernet 10/40Gb 2-port 544QSFP Adapter
    • HP InfiniBand FDR/Ethernet 10/40Gb 2-port 544FLR-QSFP Adapter
    • HP InfiniBand QDR/Ethernet 10Gb 2-port 544FLR-QSFP Adapter
    • HP Infiniband QDR/Ethernet 10Gb 2P 544i Adapter
    • HPE InfiniBand FDR 2-port 545QSFP Adapter
    • HPE InfiniBand FDR 2-port 545FLR-QSFP Adapter
    • HPE InfiniBand FDR 2-port 545M Adapter
  • Broadcom network adapters:
    • HPE Ethernet 10/25Gb 2-port 631SFP28 Adapter
    • HPE Ethernet 10/25Gb 2-port 631FLR-SFP28 Adapter
    • HPE Ethernet 10Gb 2-port 535T Adapter
    • HPE Ethernet 10Gb 2-port 535FLR-T Adapter

RESOLUTION

This is a known behavior because the two drivers cannot co-exist on a node if the Operating System is SUSE Linux Enterprise Server 12 SP3 (or later).

To fix this issue, remove the Broadcom NetXtreme RoCE Driver before attempting to install the Mellanox Ethernet + RoCE driver or MLNX-OFED driver, by performing the following command in the command line console:

> rpm -e bnxt_en-kmp-default-1.8.23_k4.4.73_5-1.sles12sp3.x86_64

If both Mellanox and Broadcom adapters need to be used on the same node, the only option would be to rely on in-distro drivers for Mellanox adapters.




RECEIVE PROACTIVE UPDATES : Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively via e-mail through HPE Subscriber's Choice. Sign up for Subscriber's Choice at the following URL: Proactive Updates Subscription Form .

NAVIGATION TIP : For hints on navigating HPE.com to locate the latest drivers, patches, and other support software downloads for ProLiant servers and Options, refer to the Navigation Tips document .

SEARCH TIP : For hints on locating similar documents on HPE.com, refer to the Search Tips document .


Hardware Platforms Affected: HPE InfiniBand EDR/Ethernet 100Gb 1-port 840QSFP28 Adapter, HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter, HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter, HPE Ethernet 10/25Gb 2-port 640SFP28 Adapter, HPE Ethernet 10Gb 2-port 535FLR-T Adapter, HPE Ethernet 10Gb 2-port 535T Adapter, HPE Ethernet 10/25Gb 2-port 631FLR-SFP28 Adapter, HPE Ethernet 10/25Gb 2-port 631SFP28 Adapter, HPE InfiniBand FDR/Ethernet 10/40Gb 2-port 544QSFP Adapter, HPE InfiniBand FDR/Ethernet 10/40Gb 2-port 544FLR-QSFP Adapter, HPE InfiniBand QDR/Ethernet 10Gb 2-port 544FLR-QSFP Adapter, HPE InfiniBand QDR/EN 10Gb Dual Port 544M Adapter, HPE InfiniBand FDR/EN 10/40Gb Dual Port 544M Adapter, HP Infiniband QDR/Ethernet 10Gb 2P 544i Adapter, HPE InfiniBand FDR 2-port 545QSFP Adapter, HPE InfiniBand FDR 2-port 545FLR-QSFP Adapter, HPE InfiniBand FDR 2-port 545M Adapter, HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+QSFP Adapter, HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter, HPE InfiniBand QDR/Ethernet 10Gb 2-port 544+FLR-QSFP Adapter, HPE InfiniBand QDR/Ethernet 10Gb 2-port 544+M Adapter, HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+M Adapter, HPE Ethernet 10Gb 2-port 546SFP+ Adapter, HPE Ethernet 10Gb 2-port 546FLR-SFP+ Adapter
Operating Systems Affected: SUSE Linux Enterprise Server 12 (AMD64/EM64T)
Software Affected: Not Applicable
Support Communication Cross Reference ID: SIK2650
©Copyright 2018 Hewlett Packard Enterprise Company, L.P.
Hewlett Packard Enterprise Company shall not be liable for technical or editorial errors or omissions contained herein. The information provided is provided "as is" without warranty of any kind. To the extent permitted by law, neither HPE nor its affiliates, subcontractors or suppliers will be liable for incidental, special or consequential damages including downtime cost; lost profits; damages relating to the procurement of substitute products or services; or damages for loss of data, or software restoration. The information in this document is subject to change without notice. Hewlett Packard Enterprise Company and the names of Hewlett Packard Enterprise Company products referenced herein are trademarks of Hewlett Packard Enterprise Company in the United States and other countries. Other product and company names mentioned herein may be trademarks of their respective owners.

Provide feedback

Please rate the information on this page to help us improve our content. Thank you!