Troubleshooting

This section lists the different failure scenarios associated with service tunnels along with possible failure cause and troubleshooting information.

Tunnel creation

ID Issue Possible cause Troubleshooting help
1

Failed error: SNMP_ERRORSTATUS_ COMMITFAILED

  • Cannot create a service tap tunnel because a mirror session is not available.

  • No more tunnels can be created. The maximum number is 4.

You must have a mirror session available to create a service tap tunnel.

2

Wrong value error: SNMP_ERRORSTATUS_WRONGVALUE

  • Tunnel DSCP values of -2 and -1 are not supported.

  • Truncate option is not supported in SI tunnels.

  • Choose a value other than -2 and -1 for your DSCP tunnel.

  • Do not use the truncate option for SI intercept tunnels.

3

User tries to create a service tunnel and gets the following SNMP error:

SNMP_ERRORSTATUS_

WRONGVALUE

Incorrect endpoint address

  • If local endpoint IP address of the tunnel interface is not a valid switch VLAN interface IP address, a tunnel cannot be created.


[NOTE: ]

NOTE: DHCP assigned local IP addresses or OOBM interface IP addresses are not considered to be valid local endpoint addresses.


  • If the remote endpoint IP address is a multicast address, a tunnel cannot be created.

  • If the to-be-created tunnel’s remote endpoint IP address is the same as one of switch’s interface IP addresses, a tunnel cannot be created.

To know what the error is, do an SNMP WALK on hpSwitchErrorMsgEntry MIB object. A message displays as to why the SET request failed.

For example, for an ‘invalid local IP’ case, the error message is as follows:

hpSwitchEntityErrorMsg.2.1.49.27 = Cannot create Service Tunnel because the specified source IP address is not configured on any interface.

To know the list of configured IP addresses on a device, run the following CLI command:

switch# show ip

All configured interface IP addresses get listed here. The tunnel’s local IP address should be one of the IP addresses listed here.

Relevant MIB objects:

ipAddressTable (RFC 4293 MIB)
4

User tries to create a service tunnel and gets the following SNMP error:

SNMP_ERRORSTATUS_

WRONGVALUE

Incorrect encapsulation parameters

  • If the GRE Key of the tunnel being created is 0, a tunnel cannot be created.

  • If the encapsulation method is set to non-GRE type, a tunnel cannot be created.

Do an SNMP walk on hpSwitchErrorMsgEntry MIB object. This will display a message as to why the SET request failed.

5

User tries to create a service tunnel and gets the following SNMP error:

SNMP_ERRORSTATUS_

INCONSISTENTVALUE

Exclusive features

The following switch features are mutually exclusive with service tunnels and if they are already enabled, service tunnels cannot be created.

  • Distributed Trunk

  • IPv4/IPv6 multicast routing

  • QinQ

  • Mesh

  • MAC Mirror

Do an SNMP walk on hpSwitchErrorMsgEntry MIB object. This will display a message as to why the SET request failed.

To know if any of the mutually exclusive features are already enabled, run the following CLI commands.

DT

switch# show trunks

Port| Name | Type | Group | Type

---- ------ ----------- ------- ------

1 | xyz | 100/1000T | Trk1 |dt-lacp

2 | xyz | 100/1000T | Trk1 |dt-trunk

Trunk Type dt-lacp or dt-trunk implies Distributed Trunking configuration.

Relevant MIB objects:

hpSwitchPortTable (hpSwitchConfig MIB)

IP MCAST

switch# show running-config

ip multicast-routing

Look for the above line which indicates that multicast routing is enabled on the device.

Relevant MIB objects:

ipMRouteEnable (RFC 2932 MIB)

QinQ

switch# show qinq

QinQ configuration information:

-------------------------------

Mode : svlan (or mixedvlan)

If the qinq mode is svlan or mixedvlan, it means qinq is enabled on the device.

Relevant MIB objects:

hpicfProviderBridgeType

(hpicfProviderBridge MIB)

Mesh

switch# show mesh

MAC Mirror

switch# show mac-mirror

6

User tries to create a service tunnel and gets the following SNMP error:

SNMP_ERRORSTATUS_

RESOURCEUNAVAILABLE

# of tunnels limit

There is a limit on the number of tunnels that can be configured on a device and once that limit has been reached, any further tunnel creation will be errored out.

Do an SNMP walk on hpSwitchErrorMsgEntry MIB object. This will display a message as to why the SET request failed.

Run the following CLI command:

switch# show interface tunnel type intercept

Max. Supported Tunnels : 16

Total Tunnels : 16

The ‘total tunnels’ field is the current number of tunnels configured on the box while ‘max tunnels’ is the limit.

Relevant MIB objects:

hpicfMaxIPv4ServiceTunnels

hpicfTotalIPv4ServiceTunnels

(hpicfServiceTunnels MIB)

7

User tries to create a service tunnel and gets the following SNMP error:

SNMP_ERRORSTATUS_

RESOURCEUNAVAILABLE

No V1 line card support

On 5400 chassis platforms, tunnels cannot be created when the chassis has been configured to run in v1-compat-mode.

Do an SNMP walk on hpSwitchErrorMsgEntry MIB object. This will display a user-friendly message as to why the SET request failed.

Run the following CLI command to check if the device is running in v1-compat mode.

switch# show system

Allow V1 Modules : Yes

This means that the device is in V1 compat mode.

Relevant MIB objects:

hpSwitchAllowV1Modules

(hpSwitchConfig MIB)

8

User tries to create a service tunnel and gets the following SNMP error:

SNMP_ERRORSTATUS_

RESOURCEUNAVAILABLE

No platform support

Tunnels cannot be created on 3500/6200/6600 platforms and SMB platforms.

Do an SNMP walk on hpSwitchErrorMsgEntry MIB object. This will display a message as to why the SET request failed.

9

User tries to create a service tunnel and gets the following SNMP error:

SNMP_ERRORSTATUS_

RESOURCEUNAVAILABLE

No hardware resource

Tunnel needs free TCAM entries on all slots to be available for creation.

Unavailability of these resources results in tunnel creation failure.

Do an SNMP walk on hpSwitchErrorMsgEntry MIB object. This displays a message as to why the SET request failed.

Run the following CLI command and check if TCAM resources are available.

switch# show qos resources

Look at the ‘Rules available’ and ‘Meters available’ column for all slots (or port groups).

For the first service tunnel to be created -

‘Rules available’ must be >= 26 and

‘Meters available’ must be >= 2.

For subsequent service tunnels to be created –

‘Rules available’ must be >= 6

Tunnel operational status

ID Issue Possible cause Troubleshooting help
1

Tunnel’s operstatus is reported as DOWN in the CLI or Event logs or via OpenFlow.

Network connectivity issue

  • The tunnel endpoint is on a directly connected subnet but down (not responding to ARP requests).

  • The tunnel endpoint is not on a directly connected subnet and there is no IP route on the switch to reach the tunnel endpoint IP address.

  • The tunnel endpoint is not on a directly connected subnet, there is an IP route on the switch to reach the endpoint IP address but the corresponding nexthop is down (not responding to ARP requests).

  • The uplink interface on the switch via which the endpoint or gateway is reachable is down.

To check for tunnel endpoint IP resolution status, run the following CLI command:

switch# show interface tunnel type intercept

Status

Interface State : Down

Down Reason : No route to destination

Destination Address Route :

Next Hop IP :

Next Hop Interface :

Next Hop IP Link Status :

Source Address :

Egress port :

Look for the ‘State’ field being ‘Down’ and also the ‘Down Reason’ and ‘Next Hop’ related fields below this field.

If there is no route to the destination, the Nexthop fields would be empty.

Check the IP route table, gateway availability and uplink interface status to troubleshoot further.

2

Tunnel’s operstatus is reported as DOWN in the CLI or Event logs or OpenFlow.

ASIC component failure

Loopback port going down can result in tunnels momentarily going down (software picks a new loopback port as soon as the current loopback port goes down).


[NOTE: ]

NOTE: A loopback port goes down only when a line card that houses the loopback port goes down. There is one loopback port on every linecard and so if one linecard goes down, another linecard’s loopback port will start handling tunnel packets.


During the transition, tunneled packets can be dropped.

On stackable platforms, each port group represents the equivalent of one linecard and has one loopback port. If a port group is down, a loopback port from another port group will handle tunneled packets.

Look for the following message in the event logs:

Service Tunnel decapsulation interface flap – Packets may have been dropped.

When the last line card goes down, the tunnel will be reported as down until a new line card comes up again.

Run the following CLI command to know if the tunnel is down due to resource unavailability.

switch# show interface tunnel type intercept

Status

Interface State  : Down

Down Reason : Out of resources

Look for the “Down Reason” field in the output that indicates there are no ASIC resources.

Encapsulation path - traffic steering to tunnel (via OpenFlow)

ID Issue Possible cause Troubleshooting help
1

FlowMod failure when trying to program an OpenFlow rule that is diverting packets to a tunnel interface.

Invalid tunnel ifindex

The tunnel ifindex in an OF FlowMod must match a configured tunnel interface’s ifIndex.

Otherwise, the FlowMod will fail.

Run the following CLI command and check if the ifindex in the OpenFlow Flow Mod rule is valid.

switch$ show interface

tunnel type intercept

Interface Index : 1000000001

Relevant MIB objects:

tunnelInetConfigIfIndex (rfc4087 MIB)

From an OF response perspective, the flow mod request will be rejected with error type="BAD_ACTION", code="BAD_OUT_PORT"

2

FlowMod failure when trying to program an OpenFlow rule that is diverting packets to a tunnel interface.

OpenFlow 1.0 instance

OpenFlow Forwarding to a tunnel interface is only supported on 1.3 instances.

If the instance is a 1.0, the FlowMod will fail.

NA

3

FlowMod failure when trying to program an OpenFlow rule that is diverting packets to a tunnel interface.

Tunnel ‘outport action’ limitations

There are some restrictions with using Tunnels as outports as listed here:

  • For Intercept tunnels only: Cannot add tunnel interface as one of the ports in a multiport output action.

  • For Intercept tunnels only: Cannot club a flow-rule’s output action of sending packets to a tunnel with “Normal” or “SendToController” actions.

  • Cannot club a flow-rule’s output action of sending packets to a tunnel with OpenFlow Strip-VLAN action as the inner encapsulated packets will always be VLAN tagged.

  • Tunnel interface cannot be part of a FLOOD action.

The flow mod request will be rejected with error type="BAD_ACTION", code="BAD_OUT_PORT".

4

FlowMod failure when trying to program an OpenFlow rule that is diverting packets to a tunnel interface.

Limitations with OpenFlow tables and tunnels

The outport action for tunnel interfaces is only supported on the Policy Engine (TCAM) tables and not on the other tables including the OpenFlow software tables.

The flow mod request is rejected with error type="BAD_REQUEST", code="BAD_TABLE_ID".

5

FlowMod failure when programming OpenFlow rules with in_port as tunnel IfIndex.

Match criteria limitations with tunnels

Tunnel Interfaces cannot be used as a match field (IN_PORT) in OpenFlow rules.

The flow mod request is rejected with error type="BAD_ACTION", code="BAD_ARGUMENT".

6

DNS/IP packets not sent out to the tunnel.

Higher precedence rule

Another higher precedence OpenFlow rule matched instead of the DNS/IP match rule.

Check if there are overlapping rules with higher precedence compared to tunnel rules.

If so, check the packet counts of the higher precedence overlapping rules to see if they are incrementing.

7

DNS/IP packets not sent out to the tunnel.

Packet encapsulation failure

If the tunnel is UP and packets are matching an OpenFlow rule that is directing traffic to the tunnel but packets are not being sent to the controller, it could be for one of the following reasons

  • MTU violation during encapsulation can cause packets to be dropped.

  • Uplink interface drops (link congestion).

  • Network congestion in the upstream network.

To know if there are MTU violations causing packets to not be tunneledRun the following CLI command:

switch$ show interface tunnel type intercept statistics

Service Tunnel Statistics

MTU Violation Drop : 10

Relevant MIB objects:

hpicfServiceTunnelStatsTxMTUViolationDrop

(hpicfServiceTunnels MIB)

If there are no MTU violations-related drops, run the following command to know if there are uplink interface TX drops.

To do that, first determine the uplink interface that this tunnel uses by running the “show interface tunnel type intercept” command and identify the ‘Egress port’.

switch$ show interface <port>

Look for TX drop counters on this interface.

8

DNS/IP packets not sent out to the tunnel.

Stale tunnels

When a tunnel is deleted but an OpenFlow rule that is diverting packets to that tunnel interface is still not removed, packets to the stale tunnels are dropped.

Run the following CLI command and check if the ifindex in the tunnel used is valid.

switch$ show interface tunnel type intercept

Interface Index : 1000000001

Relevant MIB objects:

tunnelInetConfigIfIndex (rfc4087 MIB)

Also in this case, the tunnel rule hit count increases as packets match the tunnel rule.

Decapsulation path

The Decapsulation path applies to Intercept tunnels only. It does not apply to Tap tunnels.

ID Issue Possible cause Troubleshooting help
1

Tunneled packets received from the remote endpoint not decapsulated at the switch.

GRE Key incompatibility

If the GRE KEY in the encapsulated packet does not match one of the configured tunnels’ GRE KEY values, encapsulated packets can be dropped.

To know if there are drops due to GRE KEY incompatibilities, run the following CLI command:

switch$ show interface tunnel type intercept statistics

Service Tunnel Statistics

Packets to Non-Existent Tunnels: 200

If this counter is incrementing, it means that there is GRE KEY incompatibility in traffic inbounds on one or more tunnels.

Relevant MIB objects

hpicfServiceTunnelStatsRxInvalidKey

(hpicfServiceTunnels MIB)

2

Encapsulated packets received from the remote endpoint not decapsulated at the switch.

IP fragmentation

If the GRE KEY matched that of a configured tunnel but the packet was a fragmented frame (‘More Fragments Bit’ set), the switch cannot reassemble these frames and therefore will drop them.

To know if there are drops due to fragmentation, run the following CLI command:

switch$ show interface tunnel type intercept statistics

Service Tunnel Statistics

Fragmented Packets Dropped (Rx): 100

If this counter is incrementing, it means that there is fragmented traffic inbound on one or more tunnels.

Relevant MIB objects

hpicfServiceTunnelStatsRxFragmentDrops

(hpicfServiceTunnels MIB)

3

Decapsulated packets not forwarded out by the switch.

Loopback interface oversubscription

Decapsulated packets (from all tunnels) hit the loopback port and are reinserted to the ASIC for a second pass processing.

This port has a 1G capacity and oversubscription on this port can cause packets to be dropped.

Look for the following message in the event logs:

Packets dropped due to oversubscription on tunnel decapsulation interface.

4

Decapsulated packets not forwarded out by the switch.

Source MAC address not learned

All decapsulated packets that hit the loopback port are reinserted to the ASIC for a second pass processing.

In this pass, the SRC MAC of the decapsulated frame must have already been learned on the packet’s inbound VLAN. If not, such traffic will be dropped.

If the VLAN in the decapsulated packet is not configured on the device, packets will be dropped as well.

To know if there are drops due to SRC MAC lookup failure, run the following CLI command:

HP-E8206zl# show interface tunnel type intercept statistics

Service Tunnel Statistics

Unknown Source MAC Packets Dropped (Rx): 200

If this counter is incrementing, it means there is traffic on the loopback interface that has failed the ‘SRC MAC on the inbound VLAN’ lookup.

Other tunnel-specific troubleshooting commands

ID Purpose CLI command or other
1

Tunnel interface TX/RX packet counters

(tunnel and heartbeat packets)

HP-E8206zl# show interface tunnel type intercept statistics 1000000001

Aggregate Statistics

Fragmented Packets Dropped (Rx)         : 0 

Packets to Non-Existent Tunnel : 0

Unknown Source MAC Packets Dropped (Rx) : 0

MTU Violation Drop : 0

Per Tunnel Statistics

Interface Index : 100663874

Name                                         : ServiceTunnel-01

Rx Packets                                   : 0

Tx Packets                                   : 0

Rx 5 Minute Weighted Average Rate (Pkts/sec) : 0

Tx 5 Minute Weighted Average Rate (Pkts/sec) : 0

Rx Heartbeat                                 : 0

Tx Heartbeat                                 : 0

Last Recv Heartbeat Timestamp : 01/01/90 00:00:00

2

Tunnel interface TX/RX packet rate

HP-E8206zl# HP-E8206zl# show interface tunnel type intercept statistics 1000000001

Aggregate Statistics

Fragmented Packets Dropped (Rx)         : 0 

Packets to Non-Existent Tunnel : 0

Unknown Source MAC Packets Dropped (Rx) : 0

MTU Violation Drop : 0

Per Tunnel Statistics

Interface Index : 100663874

Name                                         : ServiceTunnel-01

Rx Packets                                   : 0

Tx Packets                                   : 0

Rx 5 Minute Weighted Average Rate (Pkts/sec) : 0

Tx 5 Minute Weighted Average Rate (Pkts/sec) : 0

Rx Heartbeat                                 : 0

Tx Heartbeat                                 : 0

Last Recv Heartbeat Timestamp : 01/01/90 00:00:00

3

Debug logging for various tunnel-related events

HP-E8206zl# debug tunnel intercept
4

Show Tech command

HP-E8206zl# show tech tunnel intercept

Dumps all tunnel-related information including

  • Statistics

  • ASIC encapsulation table contents

  • ASIC decapsulation rules

  • loopback port information and so on

  • Tunnel endpoint resolution status

5

Clearing tunnel counters

HP-E8206zl# clear interfaces tunnel type intercept statistics

6

Clear/delete tunnels from CLI

HP-E8206zl# clear interfaces tunnel type intercept

7

Notifications for following events:

  • Tunnel Create

  • Tunnel Delete

  • Tunnel Up

  • Tunnel Down

Asynchronous notifications:

  • RMON Logs

  • SNMP Traps

  • OpenFlow PORT STATUS message

8

Capability to parse an encapsulated packet (with wireshark)

The switch encapsulates the GRE payload with protocol type set to ’Transparent Ethernet Bridging (0x6558)’. It is a standard encapsulation type that wireshark can natively understand and be able to dissect and display packet contents including payload information.