Manually configuring ONIE (Cumulus NOS) HPE 6960 (Data) switches

Both the ONIE (Cumulus NOS) 6960 switches (6960-1 and 6960-2) are connected as a highly available active-active fabric.

Procedure
  1. Retrieve the following parameters of the ONIE (Cumulus NOS) 6960 switches from the Input JSON file:

    • Name of the switch

    • Initial temporary configuration IP address (initial_temp_config_ip) and VIM-PXE netmask

  2. Log in to the ONIE (Cumulus NOS) 6960 switch using management interface (eth0) with the default credentials.

    The default username is cumulus and the default password is CumulusLinux!.

  3. To modify the default password, run the following commands with suitable values: 
    sudo passwd cumulus
[sudo] password for cumulus: <Enter the new password>
Enter new UNIX password: <Enter the new password>
Retype new UNIX password: <Enter the new password again>
  4. To configure the MLAG, see the following examples:
    NOTE:

    The swp30, swp31, and swp32 interfaces are used in this example.

    PeerLink configuration for 6960-1 switch: 

    net add bond peerlink bond slaves swp30-32
net add interface peerlink.4094 clag peer-ip 169.254.1.2
net add interface peerlink.4094 clag priority 1000
net add interface peerlink.4094 clag sys-mac 44:38:39:FF:40:90
net add interface peerlink.4094 ip address 169.254.1.1/30
net add bond peerlink bridge vids 301-305,380-399

    PeerLink configuration for 6960-2 switch: 

    net add bond peerlink bond slaves swp30-32
net add interface peerlink.4094 clag peer-ip 169.254.1.1
net add interface peerlink.4094 clag priority 2000
net add interface peerlink.4094 clag sys-mac 44:38:39:FF:40:90
net add interface peerlink.4094 ip address 169.254.1.2/30
net add bond peerlink bridge vids 301-305,380-399
    NOTE:

    To prevent MAC address conflicts with other interfaces in the same bridged network, Cumulus Linux reserves a range of MAC addresses specifically to use with MLAG. This MAC addresses range is 44:38:39:ff:00:00 to 44:38:39:ff:ff:ff. When configuring MLAG and VRR, you must use this range of MAC addresses.

  5. To configure the uplink connectivity with the customer core switch for the Cumulus NOS 6960-1 and 6960-2 switches, see the following examples:

    Uplink configuration for 6960-1 switch: 

    net add bond uplink bond slaves swp1-10
net add bond uplink clag id 100
net add bond uplink alias LACP etherchannel to uplink for core
net add bond uplink bridge pvid 4032
net add bond uplink bridge vids 380-399,4032

    Uplink configuration for 6960-2 switch: 

    net add bond uplink bond slaves swp1-10
net add bond uplink clag id 100
net add bond uplink alias LACP etherchannel to uplink for core
net add bond uplink bridge pvid 4032
net add bond uplink bridge vids 380-399,4032
    NOTE:

    In this example, for uplink connectivity with the customer core switch, 10 ports from each switch are used (swp1 to swp10). The LACP bond is created to achieve redundancy. The transport VLAN is configured as pvid (4032 native VLAN) and tagged CAN (304), OAM (305), and provider VLAN (380-399) as a tagged VLAN.

    NOTE:

    Specify a unique clag-id for every dual-connected bond on each peer switch. The value must be in the range 1 to 65535, and must be the same on both peer switches for the bond to be considered dual-connected.

  6. Configure the Virtual Router Redundancy (VRR) IP interface configuration.

    The following table shows the sample values for the VRR IP interface configuration:

    Table 22: HPE 6960 VRR IP interface configuration
    VLAN name VLAN ID 6960-1 6960-2 Virtual IP MAC address
    VIM-PXE 301 10.131.1.2 10.131.1.3 10.131.1.1 44:38:39:FF:00:01
    BLS 302 10.131.2.2 10.131.2.3 10.131.2.1 44:38:39:FF:00:02
    CLM 303 10.131.3.2 10.131.3.3 10.131.3.1 44:38:39:FF:00:03
    CAN 304        
    OAM 305        
    Transport VLAN 4032 10.254.253.3 10.254.253.4 10.254.253.2 44:38:39:FF:00:09
    NOTE:

    The CAN and OAM IP interface must be configured at the customer core switch.

  7. To break the 10G switch port into four separate ports, run the following command:
    NOTE:

    Cumulus Linux permits the breakout of the 100G switch ports into 2x50G, 4x25G, 4x10G ports with breakout cables. Also, the 40G switch ports can be split into four separate 10G ports for use with breakout cables. 

    net add interface <switch port number> breakout 4x10G

    For example, 

    net add interface swp11 breakout 4x10G

    The swp11 port is split into the following:

    • Swp11s0

    • Swp11s1

    • Swp11s2

    • Swp11s3

    IMPORTANT:

    Repeat this step for ports swp12 and swp13.

  8. To configure the link speed on the four newly split ports, run the following command: 
    net add interface <switch port number range> link speed <value in ?>

    For example, 

    net add interface swp11s0-3 link speed 10000
    IMPORTANT:

    Repeat this step for ports swp12 and swp13.

  9. To break the 25G switch port into four separate ports, run the following command: 
    net add interface <switch port number> breakout 4x25G

    For example, 

    net add interface swp18 breakout 4x25G

    The swp18 port is split into the following:

    • Swp18s0

    • Swp18s1

    • Swp18s2

    • Swp18s3

    IMPORTANT:

    Repeat this step for port swp19.

  10. To configure the link speed on the four newly split ports, run the following command: 
    net add interface <switch port number range> link speed <value in ?>

    For example, 

    net add interface swp18s0-3 link speed 25000
    IMPORTANT:

    Repeat this step for port swp19.

  11. To configure the PortAdminEdge mode, run the following commands:
    NOTE:

    The PortAdminEdge feature or the PortFast feature enables or disables the initial edge state of a port in a bridge. All ports configured with PortAdminEdge bypass the listening and learning states to move immediately to the forwarding state. 

    net add interface <switch port number> stp bpduguard
net add interface <switch port number> stp portadminedge
net pending
net commit

    For example, 

    net add interface swp11s0 stp bpduguard
net add interface swp11s0 stp portadminedg
  12. To configure the STP with MLAG, see the following example: 
    net add bridge stp treeprio 4096
net commit
    NOTE:

    • The STP global configuration must be the same on both the switches.

    • The STP configuration for dual-connected ports must be the same on both the peer switches.

    • The STP priority must be the same on both the peer switches.

  13. To configure the MTU values, run the following commands:
    NOTE:

    When configuring the MTU for a bond, configure the MTU value directly under the bond interface. The configured value is inherited by the member links/slave interfaces. If you need a different MTU on the bond, set it on the bond interface. This configuration ensures that the slave interfaces pick it up. There is no need to specify MTU on the slave interfaces. 

    net add interface <switch port number range> mtu <value in ?>
net pending
net commit

    For example, 

    net add interface swp11s0-3 mtu 9216
  14. To configure the IP Interface configuration for VLAN on the Cumulus NOS 6960-1 switch, run the following commands:

    PXE VLAN IP interface: 

    net add vlan 301 ip address 10.131.1.2/24
net add vlan 301 ip address-virtual 44:38:39:FF:00:01 10.131.1.1/24
net add vlan 301 mtu 9216
net add vlan 301 vlan-id 301
net add vlan 301 vlan-raw-device bridge

    BLS Network VLAN IP interface: 

    net add vlan 302 ip address 10.131.2.2/24
net add vlan 302 ip address-virtual 44:38:39:FF:00:02 10.131.2.1/24
net add vlan 302 mtu 9216
net add vlan 302 vlan-id 302
net add vlan 302 vlan-raw-device bridge

    CLM Network VLAN IP interface: 

    net add vlan 303 ip address 10.131.3.2/24
net add vlan 303 ip address-virtual 44:38:39:FF:00:03 10.131.3.1/24
net add vlan 303 mtu 9216
net add vlan 303 vlan-id 303
net add vlan 303 vlan-raw-device bridge

    Transport Network VLAN IP interface (Customer Provided Network): 

    net add vlan 4032 ip address 10.254.253.3/29
net add vlan 4032 ip address-virtual 44:38:39:FF:00:09 10.254.253.2/29
net add vlan 4032 vlan-id 4032
net add vlan 4032 vlan-raw-device bridge
  15. To configure the IP Interface configuration for VLAN on the Cumulus NOS 6960-2 switch, run the following commands:

    PXE VLAN IP interface: 

    net add vlan 301 ip address 10.131.1.3/24
net add vlan 301 ip address-virtual 44:38:39:FF:00:01 10.131.1.1/24
net add vlan 301 mtu 9216
net add vlan 301 vlan-id 301
net add vlan 301 vlan-raw-device bridge

    BLS Network VLAN IP interface: 

    net add vlan 302 ip address 10.131.2.3/24
net add vlan 302 ip address-virtual 44:38:39:FF:00:02 10.131.2.1/24
net add vlan 302 mtu 9216
net add vlan 302 vlan-id 302
net add vlan 302 vlan-raw-device bridge

    CLM Network VLAN IP interface: 

    net add vlan 303 ip address 10.131.3.3/24
net add vlan 303 ip address-virtual 44:38:39:FF:00:03 10.131.3.1/24
net add vlan 303 mtu 9216
net add vlan 303 vlan-id 303
net add vlan 303 vlan-raw-device bridge

    Transport Network VLAN IP interface (Customer Provided Network): 

    net add vlan 4032 ip address 10.254.253.4/29
net add vlan 4032 ip address-virtual 44:38:39:FF:00:09 10.254.253.2/29
net add vlan 4032 vlan-id 4032
net add vlan 4032 vlan-raw-device bridge
  16. To configure the static routing on the Cumulus NOS 6960-1 and 6960-2 switches, run the following commands: 
    net add routing route 0.0.0.0/0 10.254.253.1
net add routing route 0.0.0.0/0 10.254.253.1
  17. To configure LACP bypass, run the following command:
    NOTE:

    • Cumulus Linux provides the LACP bypass feature that allows a bond configured in 802.3ad mode to become active and forward traffic even when there is no LACP partner.

    • LACP bypass must be configured for the VIM controller FLR port and the HCI node FLR port because these hosts will boot from the Undercloud. 

    net add bond vim1-flr bond lacp-bypass-allow
  18. To configure the LACP rate of negotiation, run the following command:
    NOTE:

    The LACP rate of negotiation sets the rate to ask the link partner to transmit the LACP control packets. The valid values are short (1 second; also called fast) or long (30 seconds; also called slow). The default value is fast. 

    net add bond vim1-flr bond lacp-rate slow