Print | Rate this content

Advisory: Linux - Both the Mellanox OFED or Mellanox Ethernet + RoCE driver and the Broadcom / QLogic RoCE Ethernet Drivers Will Not Load on an HPE ProLiant System

SUPPORT COMMUNICATION - CUSTOMER ADVISORY

Document ID: a00036618en_us

Version: 1

Advisory: Linux - Both the Mellanox OFED or Mellanox Ethernet + RoCE driver and the Broadcom / QLogic RoCE Ethernet Drivers Will Not Load on an HPE ProLiant System
NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.

Release Date: 2017-11-16

Last Updated: 2017-11-16


DESCRIPTION

Broadcom and QLogic Ethernet adapters with RoCE support do not load the RoCE module when the Mellanox Ethernet + RoCE or the MLNX_OFED driver is already installed. Broadcom and QLogic RoCE supported adapters depend on the native OS "ib_core" kernel module. Once the MLNX_OFED or the Mellanox Ethernet + RoCE driver is installed that updates the "ib_core" kernel module, it prevents other vendor's RoCE drivers to load, as shown in the following example:

Native "ib_core" kernel module:

[root@localhost ~]# modinfo ib_core
filename: /lib/modules/3.10.0-693.el7.x86_64/kernel/drivers/infiniband/core/ib_core.ko
license: Dual BSD/GPL
description: core kernel InfiniBand API
author: Roland Dreier
rhelversion: 7.4
srcversion: 2B35581D5A8815FA8684260
depends:
intree: Y
vermagic: 3.10.0-693.el7.x86_64 SMP mod_unload modversions
signer: Red Hat Enterprise Linux kernel signing key
sig_key: 4F:FD:D6:3C:93:7E:B4:A7:A1:14:BC:5E:89:1A:CB:DE:50:20:65:21
sig_hashalgo: sha256
parm: send_queue_size:Size of send queue in number of work requests (int)
parm: recv_queue_size:Size of receive queue in number of work requests (int)
parm: force_mr:Force usage of MRs for RDMA READ/WRITE operations (bool)

Mellanox Ethernet + RoCE - updated "ib_core" kernel module:

[root@localhost Mellanox-RoCE]# modinfo ib_core
filename: /lib/modules/3.10.0-693.el7.x86_64/extra/mlnx-ofa_kernel/drivers/infiniband/core/ib_core.ko
license: Dual BSD/GPL
description: core kernel InfiniBand API
author: Roland Dreier
rhelversion: 7.4
srcversion: 88498DC1AE00B29161E536C
depends: mlx_compat
vermagic: 3.10.0-693.el7.x86_64 SMP mod_unload modversions
signer: Hewlett-Packard Company: HP UEFI Secure Boot 2013 DB key
sig_key: 1D:7C:F2:C2:B9:26:73:F6:9C:8E:E1:EC:70:63:96:7A:B9:B6:2B:EC
sig_hashalgo: sha256
parm: send_queue_size:Size of send queue in number of work requests (int)
parm: recv_queue_size:Size of receive queue in number of work requests (int)
parm: roce_v1_noncompat_gid:Default GID auto configuration (Default: yes) (bool)
parm: force_mr:Force usage of MRs for RDMA READ/WRITE operations (bool)

Qlogic driver does not load after "ib_core" update is installed as follows:

[root@localhost Qlogic]# modinfo qedr
filename: /lib/modules/3.10.0-693.el7.x86_64/extra/qlgc-fastlinq/qedr.ko
version: 8.33.0.0
license: Dual BSD/GPL
author: QLogic Corporation
description: QLogic 40G/100G ROCE Driver
rhelversion: 7.4
srcversion: 91BE17AF97BBFDD84C3FCF2
depends: qede,qed,ib_core
vermagic: 3.10.0-693.el7.x86_64 SMP mod_unload modversions
signer: Hewlett-Packard Company: HP UEFI Secure Boot 2013 DB key
sig_key: 1D:7C:F2:C2:B9:26:73:F6:9C:8E:E1:EC:70:63:96:7A:B9:B6:2B:EC
sig_hashalgo: sha256
parm: debug: Default debug msglevel (uint)
parm: delayed_ack: iWARP: Delayed Ack: 0 - Disabled 1 - Enabled. Default: Disabled (uint)
parm: timestamp: iWARP: Timestamp: 0 - Disabled 1 - Enabled. Default: Enabled (uint)
parm: rcv_wnd_size: iWARP: Receive Window Size in K. Minimum is 64K. Default is set according to device configuration (uint)
parm: crc_needed: iWARP: CRC needed 0 - Disabled 1 - Enabled. Default:Enabled (uint)
parm: peer2peer: iWARP: Support peer2peer ULPs 0 - Disabled 1 - Enabled. Default:Enabled (uint)
parm: mpa_enhanced: iWARP: MPA Enhanced mode. Default:1 (uint)
parm: rtr_type: iWARP: RDMAP opcode to use for the RTR message: BITMAP 1: RDMA_SEND 2: RDMA_WRITE 4: RDMA_READ. Default: 7 (uint)
parm: insert_udp_src_port: Insert a non-zero UDP source port for RoCEv2 packets that is unique per QP. 0 - Disabled, 1 - Enabled. Default:Enabled) (uint)
parm: wq_multiplier: When creating a WQ the actual number of WQE created will be multiplied by this number (default is 3). (uint)

[root@localhost Qlogic]# modprobe qedr
modprobe: ERROR: could not insert 'qedr': Invalid argument

"/var/log/messages" output
Oct 25 16:40:28 localhost kernel: qed_init called
Oct 25 16:40:28 localhost kernel: QLogic FastLinQ 4xxxx Core Module qed 8.33.0.0
Oct 25 16:40:28 localhost kernel: qedr: disagrees about version of symbol ib_umem_release
Oct 25 16:40:28 localhost kernel: qedr: Unknown symbol ib_umem_release (err -22)
Oct 25 16:40:28 localhost kernel: qedr: disagrees about version of symbol ib_modify_qp_is_ok
Oct 25 16:40:28 localhost kernel: qedr: Unknown symbol ib_modify_qp_is_ok (err -22)

Broadcom driver does not load after ib_core updates from native OS version as follows:


[root@localhost Broadcom]# modinfo bnxt_re
filename: /lib/modules/3.10.0-693.el7.x86_64/extra/bnxt_en/drivers/infiniband/hw/bnxt_re/bnxt_re.ko
version: 20.8.0.6
license: Dual BSD/GPL
description: Broadcom NetXtreme RoCE Driver
author: Eddie Wai <eddie.wai@broadcom.com>
rhelversion: 7.4
srcversion: 4AA42989AAA09F07E785E15
depends: ib_core
vermagic: 3.10.0-693.el7.x86_64 SMP mod_unload modversions
signer: Hewlett-Packard Company: HP UEFI Secure Boot 2013 DB key
sig_key: 1D:7C:F2:C2:B9:26:73:F6:9C:8E:E1:EC:70:63:96:7A:B9:B6:2B:EC
sig_hashalgo: sha256
parm: restrict_mrs: Restrict the no. of MRs 0 = 256K , 1 = 64K (uint)

[root@localhost Broadcom]# modprobe bnxt_re
modprobe: ERROR: could not insert 'bnxt_re': Invalid argument

"/var/log/messages" output
Oct 25 16:44:19 localhost kernel: bnxt_re: disagrees about version of symbol ib_umem_release
Oct 25 16:44:19 localhost kernel: bnxt_re: Unknown symbol ib_umem_release (err -22)
Oct 25 16:44:19 localhost kernel: bnxt_re: disagrees about version of symbol ib_modify_qp_is_ok
Oct 25 16:44:19 localhost kernel: bnxt_re: Unknown symbol ib_modify_qp_is_ok (err -22)
Oct 25 16:44:19 localhost kernel: bnxt_re: disagrees about version of symbol ib_sg_to_pages

SCOPE

Any HPE ProLiant or HPE Synergy server running Linux and configured with both Mellanox IB/Ethernet and Qlogic OR Broadcom Ethernet RoCE supported adapters and installed with the following drivers:

  • Mellanox-OFED or Mellanox Ethernet+RoCE (mlnx-ofa_kernel)


AND

  • Broadcom NetXtreme-E RoCE driver -OR- QLogic FastLinQ RoCE driver.


The following Operating Systems are affected:

  • Red Hat Enterprise Linux (all supported versions).
  • SuSE Linux Enterprise Server (all supported versions).

The following adapters are affected:

Mellanox network adapters:

  • HPE Ethernet 25Gb 2-port 640SFP28 Adapter (HPE Part Number: 817753-B21)
  • HPE Ethernet 25Gb 2-port 640FLR-SFP28 Adapter (HPE Part Number: 817749-B21)
  • HPE InfiniBand EDR/Ethernet 100Gb 1-port 840QSFP28 Adapter (HPE Part Number: 825110-B21)
  • HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter (HPE Part Number: 825111-B21)
  • HP Ethernet 10Gb 2-port 546SFP+ Adapter (HP Part Number: 779793-B21)
  • HP Ethernet 10Gb 2-port 546FLR-SFP+ Adapter (HP Part Number: 779799-B21)
  • HP InfiniBand QDR/Ethernet 10Gb 2-port 544+M Adapter (HP Part Number: 764282-B21)
  • HP InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+M Adapter (HP Part Number: 764283-B21)
  • HP InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+QSFP Adapter (HP Part Number: 764284-B21)
  • HP InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter (HP Part Number: 764285-B21)
  • HP InfiniBand QDR/Ethernet 10Gb 2-port 544+FLR-QSFP Adapter (HP Part Number: 764286-B21)
  • HP InfiniBand QDR/EN 10Gb Dual Port 544M Adapter (HP Part Number: 644160-B21)
  • HP InfiniBand FDR/EN 10/40Gb Dual Port 544M Adapter (HP Part Number: 644161-B21/644161-B22)
  • HP InfiniBand FDR/Ethernet 10/40Gb 2-port 544QSFP Adapter (HP Part Number: 649281-B21)
  • HP InfiniBand FDR/Ethernet 10/40Gb 2-port 544FLR-QSFP Adapter (HP Part Number: 649282-B21)
  • HP InfiniBand QDR/Ethernet 10Gb 2-port 544FLR-QSFP Adapter (HP Part Number: 649283-B21)
  • HP Infiniband QDR/Ethernet 10Gb 2P 544i Adapter (HP ProLiant SL4545 G7 Server and HP ProLiant SL4540 Gen8 Server LOM)
  • HP Infiniband FDR 2-port 545QSFP Adapter (HP Part number: 702211-B21)
  • HP Infiniband FDR 2P 545FLR-QSFP Adapter (HP Part number: 702212-B21)
  • HP InfiniBand FDR 2-port 545M Adapter (HP Part number: 702213-B21)


Broadcom Ethernet adapters:

  • HPE Ethernet 10/25Gb 2-port 631SFP28 Adapter
  • HPE Ethernet 10/25Gb 2-port 631FLR-SFP28 Adapter
  • HPE Ethernet 10Gb 2-port 535T Adapter
  • HPE Ethernet 10Gb 2-port 535FLR-T Adapter

QLogic Ethernet adapters:

  • HPE Ethernet 10Gb 2-port 521T Adapter
  • HPE FlexFabric 10Gb 2-port 522FLR-T Converged Network Adapter
  • HPE Ethernet 10/25Gb 2-port 621SFP28 Adapter
  • HPE FlexFabric 10/25Gb 2-port 622FLR-SFP28 Converged Network Adapter
  • HPE Synergy 6810C 25/50Gb Ethernet Adapter
  • HPE Ethernet 4x25Gb 1-port 620QSFP28 Adapter

RESOLUTION

A QLogic OR Broadcom RoCE driver and a Mellanox OFED/Ethernet + RoCE driver cannot both be installed on the same HPE ProLiant or HPE Synergy server if both Mellanox and QLogic or Broadcom RoCE supported Ethernet adapters are to be used on the same node.

As a workaround, use the native OFED and inbox Mellanox RoCE driver directly from all supported versions of Red Hat Linux or SUSE Linux Enterprise Server.




RECEIVE PROACTIVE UPDATES : Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively via e-mail through HPE Subscriber's Choice. Sign up for Subscriber's Choice at the following URL: Proactive Updates Subscription Form.

NAVIGATION TIP : For hints on navigating HPE.com to locate the latest drivers, patches, and other support software downloads for ProLiant servers and Options, refer to the Navigation Tips document .

SEARCH TIP : For hints on locating similar documents on HPE.com, refer to the Search Tips document .

To search for additional advisories related to Linux, use the following search string:

+Advisory +ProLiant -"Software and Drivers" +Linux


Hardware Platforms Affected: HPE InfiniBand EDR/Ethernet 100Gb 1-port 840QSFP28 Adapter, HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter, HPE Ethernet 25Gb 1-port 620QSFP28 Adapter, HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter, HPE Ethernet 10/25Gb 2-port 640SFP28 Adapter, HPE Synergy 6810C 25/50Gb Ethernet Adapter, HPE Ethernet 10Gb 2-port 535FLR-T Adapter, HPE Ethernet 10Gb 2-port 535T Adapter, HPE Ethernet 10/25Gb 2-port 631FLR-SFP28 Adapter, HPE Ethernet 10/25Gb 2-port 631SFP28 Adapter, HPE Ethernet 10/25Gb 2-port 622FLR-SFP28 Converged Network Adapter, HPE Ethernet 10Gb 2-port 521T Adapter, HPE Ethernet 10/25Gb 2-port 621SFP28 Adapter, HPE Ethernet 10Gb 2-port 522FLR-T Converged Network Adapter, HPE Synergy 6410C 25/50Gb Ethernet Adapter, HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter, HPE InfiniBand FDR/Ethernet 10/40Gb 2-port 544QSFP Adapter, HPE InfiniBand FDR/Ethernet 10/40Gb 2-port 544FLR-QSFP Adapter, HPE InfiniBand QDR/Ethernet 10Gb 2-port 544FLR-QSFP Adapter, HPE InfiniBand FDR/EN 10/40Gb Dual Port 544M Adapter, HP Infiniband QDR/Ethernet 10Gb 2P 544i Adapter, HPE InfiniBand FDR 2-port 545QSFP Adapter, HPE InfiniBand FDR 2-port 545FLR-QSFP Adapter, HPE InfiniBand FDR 2-port 545M Adapter, HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+QSFP Adapter, HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter, HPE InfiniBand QDR/Ethernet 10Gb 2-port 544+FLR-QSFP Adapter, HPE InfiniBand QDR/Ethernet 10Gb 2-port 544+M Adapter, HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+M Adapter, HPE Ethernet 10Gb 2-port 546SFP+ Adapter, HPE Ethernet 10Gb 2-port 546FLR-SFP+ Adapter
Operating Systems Affected: Red Hat Enterprise Linux (Itanium), SUSE Linux Enterprise Server 11 (x86-64), Red Hat Enterprise Linux 7 (AMD64/EM64T), SUSE Linux Enterprise Server 12 (AMD64/EM64T)
Software Affected: Not Applicable
Support Communication Cross Reference ID: SIK2677
©Copyright 2018 Hewlett Packard Enterprise Development LP
Hewlett Packard Enterprise Development shall not be liable for technical or editorial errors or omissions contained herein. The information provided is provided "as is" without warranty of any kind. To the extent permitted by law, neither HPE nor its affiliates, subcontractors or suppliers will be liable for incidental, special or consequential damages including downtime cost; lost profits; damages relating to the procurement of substitute products or services; or damages for loss of data, or software restoration. The information in this document is subject to change without notice. Hewlett Packard Enterprise Development and the names of Hewlett Packard Enterprise Development products referenced herein are trademarks of Hewlett Packard Enterprise Development in the United States and other countries. Other product and company names mentioned herein may be trademarks of their respective owners.

Provide feedback

Please rate the information on this page to help us improve our content. Thank you!