Sample Configuration for Mellanox SN2100 Switches
Border leaf
Switch automation script configures the following in the starter kit.
- Management traffic and data traffic are separated by VRF.
- Management traffic
Management traffic is part of mgmt-vrf.
OAM are part of management network.
interface are configured on leaf switch, whereas OAM is customer-provided NW which is configured on Customer Infra Switches.
- Data traffic
Data traffic is part of data-vrf.
Provider VLAN is part of data traffic, which carries the VNF traffic.
In this example, the provider network is configured. Management VRF and Data VRF have to be configured across all the VTEP. This section provides information about L3 and L2 VNI.There is a one-to-one mapping between a layer 3 VNI and a tenant (VRF).
The VRF to layer 3 VNI mapping has to be consistent across all VTEPs. The layer 3 VNI has to be provisioned by the operator.
Layer 3 VNI and layer 2 VNI cannot share the same number space; that is, you cannot have vlan10 and vxlan10 for example. Otherwise, the layer 2 VNI does not get created.
- In an MLAG configuration, the SVI used for the layer 3 VNI cannot be part of the bridge. This ensures that traffic tagged with that VLAN ID is not forwarded on the peer link or other trunks.NOTE:
VXLAN104001 is the L3 Routed VNI which is used to enable Symmetric routing. Configure this VNI on all the VTEPs.
clag vxlan-anycast-ip is used when MLAG is used with VXLAN. This way, other VTEPs sees the MLAG pair as a single entity.
VLANs and VRF id USED for sample configuration
| VRF | VLAN | |
| Management VRF | mgmt-vrf | ESXi_MGMT, vMotion, vSAN , API, VCENTER_HA, and OAM (301-307) |
| Data VRF | data-vrf | EXT, Overlay and Provider Vlan (308,309 350-354) |
Management Node and Resource & Edge Nodes are connected with VTEP-1 and VTEP-2.
| VTEP-1 | VTEP-2 |
|
|
All VTEPs are connected with Spine.
net add bgp autonomous-system 65010
net add bgp router-id XX.XX.XX.XX
net add bgp bestpath as-path multipath-relax
net add bgp bestpath compare-routerid
net add bgp neighbor FABRIC peer-group
net add bgp neighbor FABRIC remote-as external
net add bgp neighbor FABRIC capability extended-nexthop
net add bgp neighbor swp1 interface peer-group FABRIC
net add bgp neighbor swp2 interface peer-group FABRIC
net add bgp neighbor swp3 interface peer-group FABRIC
net add bgp neighbor swp4 interface peer-group FABRIC
net add bgp neighbor swp16 interface peer-group FABRIC
net add bgp ipv4 unicast redistribute connected
net add bgp ipv4 unicast redistribute static
net add bgp l2vpn evpn neighbor swp1 activate
net add bgp l2vpn evpn neighbor swp2 activate
net add bgp l2vpn evpn neighbor swp3 activate
net add bgp l2vpn evpn neighbor swp4 activate
net add bgp l2vpn evpn neighbor swp16 activate
net add bgp l2vpn evpn advertise-all-vniAs we need to advertise our Underlay routing into the EVPN domain, we will add a BGP instance on the border leaf connected to the WAN.
Configuring the VTEP connected to the WAN
net add bgp autonomous-system 65006
net add vrf vrf-mgmt vni 104001 prefix-routes-only
net add bgp router-id XX.XX.XX.XX
net add bgp bestpath as-path multipath-relax
net add bgp bestpath compare-routerid
net add bgp neighbor FABRIC peer-group
net add bgp neighbor FABRIC remote-as external
net add bgp neighbor FABRIC capability extended-nexthop
net add bgp neighbor swp16 interface peer-group FABRIC
net add bgp ipv4 unicast network XX.XX.XX.XX/32
net add bgp ipv4 unicast redistribute connected
net add bgp ipv4 unicast redistribute static
net add bgp ipv6 unicast neighbor FABRIC activate
net add bgp l2vpn evpn neighbor FABRIC activate
net add bgp l2vpn evpn advertise-all-vni
net add bgp l2vpn evpn advertise-svi-ip
net add bgp l2vpn evpn advertise ipv4 unicast
net add bgp vrf vrf-mgmt autonomous-system 65006
net add bgp vrf vrf-mgmt router-id XX.XX.XX.XX
net add bgp vrf vrf-mgmt ipv4 unicast network XX.XX.XX.XX/32
net add bgp vrf vrf-mgmt ipv4 unicast network XX.XX.XX.XX/29
net add bgp vrf vrf-mgmt ipv4 unicast redistribute connected
net add bgp vrf vrf-mgmt ipv4 unicast redistribute static
net add bgp vrf vrf-mgmt l2vpn evpn advertise ipv4 unicast
net add bgp vrf vrf-mgmt l2vpn evpn default-originate ipv4
net add bgp vrf vrf-mgmt l2vpn evpn default-originate ipv6
net add bgp vrf vrf-data autonomous-system 65006
net add bgp vrf vrf-data router-id XX.XX.XX.XX
net add bgp vrf vrf-data ipv4 unicast network XX.XX.XX.XX/32
net add bgp vrf vrf-data ipv4 unicast network XX.XX.XX.XX/29
net add bgp vrf vrf-data ipv4 unicast redistribute connected
net add bgp vrf vrf-data ipv4 unicast redistribute static
net add bgp vrf vrf-data l2vpn evpn advertise ipv4 unicast
net add bgp vrf vrf-data l2vpn evpn default-originate ipv4
net add bgp vrf vrf-data l2vpn evpn default-originate ipv6
net add interface swp1-16 breakout 1x
net add vxlan vx_API_306 vxlan id 100306
net add vxlan vx_EXmgmt_301 vxlan id 100301
net add vxlan vx_L3_4001 vxlan id 104001
net add vxlan vx_VCHA_305 vxlan id 100305
net add vxlan vx_VM_mgmt_304 vxlan id 100304
net add vxlan vx_ext_309 vxlan id 100309
net add vxlan vx_oam_307 vxlan id 100307
net add vxlan vx_ovlay_308 vxlan id 100308
net add vxlan vx_vMotion_302 vxlan id 100302
net add vxlan vx_vSAN_303 vxlan id 100303
net add vxlan vxlan350 vxlan id 100350
net add vxlan vxlan351 vxlan id 100351
net add vxlan vxlan352 vxlan id 100352
net add vxlan vxlan353 vxlan id 100353
net add vxlan vxlan354 vxlan id 100354
net add vxlan vxlan355 vxlan id 100355
net add bridge bridge ports swp1,vx_API_306,vx_EXmgmt_301,vx_L3_4001,vx_VCHA_305,vx_VM_mgmt_304,vx_ext_309,vx_oam_307,vx_ovlay_308,vx_vMotion_302,vx_vSAN_303,vxlan350,vxlan351,vxlan352,vxlan353,vxlan354,vxlan355
net add bridge bridge pvid 4030
net add bridge bridge vids 301-309,350-355,4001,4030
net add bridge bridge vlan-aware
net add interface swp1 link speed 40000
net add interface swp1,16 mtu 9216
net add loopback lo ip address XX.XX.XX.XX/32
net add vlan 301 ip address XX.XX.XX.XX/24
net add vlan 301 ip address-virtual 00:00:00:00:00:1a XX.XX.XX.XX/24
net add vlan 301 mtu 9216
net add vlan 301 vlan-id 301
net add vlan 301 vlan-raw-device bridge
net add vlan 301 vrf vrf-mgmt
net add vlan 302 ip address XX.XX.XX.XX/24
net add vlan 302 ip address-virtual 00:00:00:00:00:2a XX.XX.XX.XX/24
net add vlan 302 mtu 9216
net add vlan 302 vlan-id 302
net add vlan 302 vlan-raw-device bridge
net add vlan 302 vrf vrf-mgmt
net add vlan 303 ip address XX.XX.XX.XX/24
net add vlan 303 ip address-virtual 00:00:00:00:00:3a XX.XX.XX.XX/24
net add vlan 303 mtu 9216
net add vlan 303 vlan-id 303
net add vlan 303 vlan-raw-device bridge
net add vlan 303 vrf vrf-mgmt
net add vlan 304 ip address XX.XX.XX.XX/24
net add vlan 304 ip address-virtual 00:00:00:00:00:4a XX.XX.XX.XX/24
net add vlan 304 mtu 9216
net add vlan 304 vlan-id 304
net add vlan 304 vlan-raw-device bridge
net add vlan 304 vrf vrf-mgmt
net add vlan 305 ip address XX.XX.XX.XX/24
net add vlan 305 ip address-virtual 00:00:00:00:00:5a XX.XX.XX.XX/24
net add vlan 305 mtu 9216
net add vlan 305 vlan-id 305
net add vlan 305 vlan-raw-device bridge
net add vlan 305 vrf vrf-mgmt
net add vlan 306 ip address XX.XX.XX.XX/24
net add vlan 306 ip address-virtual 00:00:00:00:00:6a XX.XX.XX.XX/24
net add vlan 306 mtu 9216
net add vlan 306 vlan-id 306
net add vlan 306 vlan-raw-device bridge
net add vlan 306 vrf vrf-mgmt
net add vlan 307 ip address XX.XX.XX.XX/24
net add vlan 307 ip address-virtual 00:00:00:00:00:7a XX.XX.XX.XX/24
net add vlan 307 mtu 9216
net add vlan 307 vlan-id 307
net add vlan 307 vlan-raw-device bridge
net add vlan 307 vrf vrf-mgmt
net add vlan 308 ip address XX.XX.XX.XX/24
net add vlan 308 ip address-virtual 00:00:00:00:00:8a XX.XX.XX.XX/24
net add vlan 308 mtu 9216
net add vlan 308 vlan-id 308
net add vlan 308 vlan-raw-device bridge
net add vlan 308 vrf vrf-data
net add vlan 309 mtu 9216
net add vlan 309 vlan-id 309
net add vlan 309 vlan-raw-device bridge
net add vlan 309 vrf vrf-data
net add vlan 350 vlan-id 350
net add vlan 350 vlan-raw-device bridge
net add vlan 350 vrf vrf-data
net add vlan 351 vlan-id 351
net add vlan 351 vlan-raw-device bridge
net add vlan 351 vrf vrf-data
net add vlan 352 mtu 9216
net add vlan 352 vlan-id 352
net add vlan 352 vlan-raw-device bridge
net add vlan 352 vrf vrf-data
net add vlan 353 vlan-id 353
net add vlan 353 vlan-raw-device bridge
net add vlan 353 vrf vrf-data
net add vlan 354 vlan-id 354
net add vlan 354 vlan-raw-device bridge
net add vlan 354 vrf vrf-data
net add vlan 355 vlan-id 355
net add vlan 355 vlan-raw-device bridge
net add vlan 355 vrf vrf-data
net add vlan 4001 vlan-id 4001
net add vlan 4001 vlan-raw-device bridge
net add vlan 4001 vrf vrf-mgmt
net add vlan 4030 ip address XX.XX.XX.XX/29
net add vlan 4030 vlan-id 4030
net add vlan 4030 vlan-raw-device bridge
net add vlan 4030 vrf vrf-mgmt
net add vrf vrf-data,vrf-mgmt vrf-table auto
net add vxlan vx_API_306 bridge access 306
net add vxlan vx_API_306,vx_EXmgmt_301,vx_L3_4001,vx_oam_307,vx_VCHA_305,vx_VM_mgmt_304,vx_vMotion_302,vx_vSAN_303,vxlan350-355 bridge arp-nd-suppress on
net add vxlan vx_API_306,vx_EXmgmt_301,vx_L3_4001,vx_oam_307,vx_VCHA_305,vx_VM_mgmt_304,vx_vMotion_302,vx_vSAN_303,vxlan350-355 bridge learning off
net add vxlan vx_API_306,vx_EXmgmt_301,vx_L3_4001,vx_oam_307,vx_VCHA_305,vx_VM_mgmt_304,vx_vMotion_302,vx_vSAN_303,vxlan350-355 mtu 9216
net add vxlan vx_API_306,vx_EXmgmt_301,vx_ext_309,vx_L3_4001,vx_oam_307,vx_ovlay_308,vx_VCHA_305,vx_VM_mgmt_304,vx_vMotion_302,vx_vSAN_303,vxlan350-355 stp bpduguard
net add vxlan vx_API_306,vx_EXmgmt_301,vx_ext_309,vx_L3_4001,vx_oam_307,vx_ovlay_308,vx_VCHA_305,vx_VM_mgmt_304,vx_vMotion_302,vx_vSAN_303,vxlan350-355 stp portbpdufilter
net add vxlan vx_API_306,vx_EXmgmt_301,vx_ext_309,vx_L3_4001,vx_oam_307,vx_ovlay_308,vx_VCHA_305,vx_VM_mgmt_304,vx_vMotion_302,vx_vSAN_303,vxlan350-355 vxlan local-tunnelip 10.130.199.6
net add vxlan vx_EXmgmt_301 bridge access 301
net add vxlan vx_L3_4001 bridge access 4001
net add vxlan vx_VCHA_305 bridge access 305
net add vxlan vx_VM_mgmt_304 bridge access 304
net add vxlan vx_ext_309 bridge access 309
net add vxlan vx_oam_307 bridge access 307
net add vxlan vx_ovlay_308 bridge access 308
net add vxlan vx_vMotion_302 bridge access 302
net add vxlan vx_vSAN_303 bridge access 303
net add vxlan vxlan350 bridge access 350
net add vxlan vxlan351 bridge access 351
net add vxlan vxlan352 bridge access 352
net add vxlan vxlan353 bridge access 353
net add vxlan vxlan354 bridge access 354
net add vxlan vxlan355 bridge access 355
net add hostname QA-BL
# There are some configuration commands that are not yet supported by nclu.
# The following will append those commands to the appropriate files.
# ========================================================================
sudo sh -c "printf 'username cumulus nopassword\n' >> /etc/frr/frr.conf"
sudo sh -c "printf 'vrf vrf-data\n ip route 0.0.0.0/0 XX.XX.XX.XX\n' >> /etc/frr/frr.conf"
sudo sh -c "printf 'vrf vrf-data\n exit-vrf\n' >> /etc/frr/frr.conf"
sudo sh -c "printf 'vrf vrf-mgmt\n ip route 0.0.0.0/0 XX.XX.XX.XX\n' >> /etc/frr/frr.conf"
sudo sh -c "printf 'vrf vrf-mgmt\n exit-vrf\n' >> /etc/frr/frr.conf"
cumulus@BL:~$
NOTE:
Based on the customer's requirement, the customer has to add the range of VLAN and VxLAN.
EVPN show commands
| Commands | Description |
|
Displays the BGP peers participating in the layer 2 EVPN address-family and their states. |
|
Displays the configured VNIs on a network device participating in BGP EVPN. This command is only relevant on a VTEP. If symmetric routing is configured, this command displays the special layer 3 VNIs that are configured per tenant VRF. |
|
Displays the EVPN information for a specific VNI in detail. |
|
Displays all local and remote MAC addresses for a VNI. |
|
Displays all local and remote neighbors (ARP entries) for a VNI. This command is only relevant for a layer 2 VNI and the output shows both IPv4 and IPv6 neighbor entries. |
|
Displays all EVPN routes, both local and remote. The routes displayed here are based on RD as they are across VNIs and VRFs. |
|
Displays all routes received through the L3 VNI. |