Troubleshooting
This section lists the different failure scenarios associated with service tunnels along with possible failure cause and troubleshooting information.
Tunnel creation
ID | Issue | Possible cause | Troubleshooting help | ||||||
---|---|---|---|---|---|---|---|---|---|
1 | Failed error: |
|
You must have a mirror session available to create a service tap tunnel. | ||||||
2 | Wrong value error: |
|
| ||||||
3 | User tries to create a service tunnel and gets the following SNMP error:
|
Incorrect endpoint address
|
To know what the error is, do
an SNMP WALK on For example, for an ‘invalid local IP’ case, the error message is as follows: hpSwitchEntityErrorMsg.2.1.49.27
= Cannot create Service Tunnel because the specified source IP address
is not configured on any interface. To know the list of configured IP addresses on a device, run the following CLI command: switch# show ip All configured interface IP addresses get listed here. The tunnel’s local IP address should be one of the IP addresses listed here. Relevant MIB objects: ipAddressTable (RFC 4293 MIB) | ||||||
4 | User tries to create a service tunnel and gets the following SNMP error:
|
Incorrect encapsulation parameters
|
Do an SNMP walk on | ||||||
5 | User tries to create a service tunnel and gets the following SNMP error:
|
Exclusive features The following switch features are mutually exclusive with service tunnels and if they are already enabled, service tunnels cannot be created.
|
Do an SNMP walk on hpSwitchErrorMsgEntry MIB object. This will display a message as to why the SET request failed. To know if any of the mutually exclusive features are already enabled, run the following CLI commands. DT
Trunk Type dt-lacp or dt-trunk implies Distributed Trunking configuration. Relevant MIB objects:
IP MCAST
Look for the above line which indicates that multicast routing is enabled on the device. Relevant MIB objects:
QinQ
If the qinq mode is svlan or mixedvlan, it means qinq is enabled on the device. Relevant MIB objects:
Mesh
MAC Mirror
| ||||||
6 | User tries to create a service tunnel and gets the following SNMP error:
|
# of tunnels limit There is a limit on the number of tunnels that can be configured on a device and once that limit has been reached, any further tunnel creation will be errored out. |
Do an SNMP walk on hpSwitchErrorMsgEntry MIB object. This will display a message as to why the SET request failed. Run the following CLI command:
The ‘total tunnels’ field is the current number of tunnels configured on the box while ‘max tunnels’ is the limit. Relevant MIB objects:
| ||||||
7 | User tries to create a service tunnel and gets the following SNMP error:
|
No V1 line card support On 5400 chassis platforms, tunnels cannot be created when the chassis has been configured to run in v1-compat-mode. |
Do an SNMP walk
on Run the following CLI command to check if the device is running in v1-compat mode.
This means that the device is in V1 compat mode. Relevant MIB objects:
| ||||||
8 | User tries to create a service tunnel and gets the following SNMP error:
|
No platform support Tunnels cannot be created on 3500/6200/6600 platforms and SMB platforms. |
Do
an SNMP walk on | ||||||
9 | User tries to create a service tunnel and gets the following SNMP error:
|
No hardware resource Tunnel needs free TCAM entries on all slots to be available for creation. Unavailability of these resources results in tunnel creation failure. |
Do an SNMP walk on hpSwitchErrorMsgEntry MIB object. This displays a message as to why the SET request failed. Run the following CLI command and check if TCAM resources are available.
Look at the ‘Rules available’ and ‘Meters available’ column for all slots (or port groups). For the first service tunnel to be created - ‘Rules available’ must be >= 26 and ‘Meters available’ must be >= 2. For subsequent service tunnels to be created – ‘Rules available’ must be >= 6 |
Tunnel operational status
ID | Issue | Possible cause | Troubleshooting help | ||||||
---|---|---|---|---|---|---|---|---|---|
1 | Tunnel’s operstatus is reported as DOWN in the CLI or Event logs or via OpenFlow. |
Network connectivity issue
|
To check for tunnel endpoint IP resolution status, run the following CLI command:
Look for the ‘State’ field being ‘Down’ and also the ‘Down Reason’ and ‘Next Hop’ related fields below this field. If there is no route to the destination, the Nexthop fields would be empty. Check the IP route table, gateway availability and uplink interface status to troubleshoot further. | ||||||
2 | Tunnel’s operstatus is reported as DOWN in the CLI or Event logs or OpenFlow. |
ASIC component failure Loopback port going down can result in tunnels momentarily going down (software picks a new loopback port as soon as the current loopback port goes down).
During the transition, tunneled packets can be dropped. On stackable platforms, each port group represents the equivalent of one linecard and has one loopback port. If a port group is down, a loopback port from another port group will handle tunneled packets. |
Look for the following message in the event logs:
When the last line card goes down, the tunnel will be reported as down until a new line card comes up again. Run the following CLI command to know if the tunnel is down due to resource unavailability.
Look for the “Down Reason” field in the output that indicates there are no ASIC resources. |
Encapsulation path - traffic steering to tunnel (via OpenFlow)
ID | Issue | Possible cause | Troubleshooting help |
---|---|---|---|
1 | FlowMod failure when trying to program an OpenFlow rule that is diverting packets to a tunnel interface. |
Invalid tunnel ifindex The tunnel ifindex in an OF FlowMod must match a configured tunnel interface’s ifIndex. Otherwise, the FlowMod will fail. |
Run the following CLI command and check if the ifindex in the OpenFlow Flow Mod rule is valid.
Relevant MIB objects:
From an OF response perspective, the flow mod request will be rejected with error type="BAD_ACTION", code="BAD_OUT_PORT"
|
2 | FlowMod failure when trying to program an OpenFlow rule that is diverting packets to a tunnel interface. |
OpenFlow 1.0 instance OpenFlow Forwarding to a tunnel interface is only supported on 1.3 instances. If the instance is a 1.0, the FlowMod will fail. |
NA
|
3 | FlowMod failure when trying to program an OpenFlow rule that is diverting packets to a tunnel interface. |
Tunnel ‘outport action’ limitations There are some restrictions with using Tunnels as outports as listed here:
|
The flow mod request will be rejected with error type="BAD_ACTION", code="BAD_OUT_PORT". |
4 | FlowMod failure when trying to program an OpenFlow rule that is diverting packets to a tunnel interface. |
Limitations with OpenFlow tables and tunnels The outport action for tunnel interfaces is only supported on the Policy Engine (TCAM) tables and not on the other tables including the OpenFlow software tables. |
The flow mod request is rejected with error type="BAD_REQUEST", code="BAD_TABLE_ID". |
5 | FlowMod failure when programming OpenFlow rules with in_port as tunnel IfIndex. |
Match criteria limitations with tunnels Tunnel Interfaces cannot be used as a match field (IN_PORT) in OpenFlow rules. |
The flow mod request is rejected with error type="BAD_ACTION", code="BAD_ARGUMENT". |
6 | DNS/IP packets not sent out to the tunnel. |
Higher precedence rule Another higher precedence OpenFlow rule matched instead of the DNS/IP match rule. |
Check if there are overlapping rules with higher precedence compared to tunnel rules. If so, check the packet counts of the higher precedence overlapping rules to see if they are incrementing. |
7 | DNS/IP packets not sent out to the tunnel. |
Packet encapsulation failure If the tunnel is UP and packets are matching an OpenFlow rule that is directing traffic to the tunnel but packets are not being sent to the controller, it could be for one of the following reasons
|
To know if there are MTU violations causing packets to not be tunneledRun the following CLI command:
Relevant MIB objects:
If there are no MTU violations-related drops, run the following command to know if there are uplink interface TX drops. To do that, first determine the uplink interface that this tunnel uses by running the “show interface tunnel type intercept” command and identify the ‘Egress port’.
Look for TX drop counters on this interface. |
8 | DNS/IP packets not sent out to the tunnel. |
Stale tunnels When a tunnel is deleted but an OpenFlow rule that is diverting packets to that tunnel interface is still not removed, packets to the stale tunnels are dropped.
|
Run the following CLI command and check if the ifindex in the tunnel used is valid.
Relevant MIB objects:
Also in this case, the tunnel rule hit count increases as packets match the tunnel rule. |
Decapsulation path
The Decapsulation path applies to Intercept tunnels only. It does not apply to Tap tunnels.
ID | Issue | Possible cause | Troubleshooting help |
---|---|---|---|
1 | Tunneled packets received from the remote endpoint not decapsulated at the switch. |
GRE Key incompatibility If the GRE KEY in the encapsulated packet does not match one of the configured tunnels’ GRE KEY values, encapsulated packets can be dropped. |
To know if there are drops due to GRE KEY incompatibilities, run the following CLI command:
If this counter is incrementing, it means that there is GRE KEY incompatibility in traffic inbounds on one or more tunnels. Relevant MIB objects
|
2 | Encapsulated packets received from the remote endpoint not decapsulated at the switch. |
IP fragmentation If the GRE KEY matched that of a configured tunnel but the packet was a fragmented frame (‘More Fragments Bit’ set), the switch cannot reassemble these frames and therefore will drop them. |
To know if there are drops due to fragmentation, run the following CLI command:
If this counter is incrementing, it means that there is fragmented traffic inbound on one or more tunnels. Relevant MIB objects
|
3 | Decapsulated packets not forwarded out by the switch. |
Loopback interface oversubscription Decapsulated packets (from all tunnels) hit the loopback port and are reinserted to the ASIC for a second pass processing. This port has a 1G capacity and oversubscription on this port can cause packets to be dropped. |
Look for the following message in the event logs:
|
4 | Decapsulated packets not forwarded out by the switch. |
Source MAC address not learned All decapsulated packets that hit the loopback port are reinserted to the ASIC for a second pass processing. In this pass, the SRC MAC of the decapsulated frame must have already been learned on the packet’s inbound VLAN. If not, such traffic will be dropped. If the VLAN in the decapsulated packet is not configured on the device, packets will be dropped as well. |
To know if there are drops due to SRC MAC lookup failure, run the following CLI command:
If this counter is incrementing, it means there is traffic on the loopback interface that has failed the ‘SRC MAC on the inbound VLAN’ lookup. |
Other tunnel-specific troubleshooting commands
ID | Purpose | CLI command or other |
---|---|---|
1 | Tunnel interface TX/RX packet counters (tunnel and heartbeat packets) |
|
2 | Tunnel interface TX/RX packet rate |
|
3 | Debug logging for various tunnel-related events |
HP-E8206zl# debug tunnel intercept |
4 | Show Tech command |
Dumps all tunnel-related information including
|
5 | Clearing tunnel counters |
|
6 | Clear/delete tunnels from CLI |
|
7 | Notifications for following events:
|
Asynchronous notifications:
|
8 | Capability to parse an encapsulated packet (with wireshark) |
The switch encapsulates the GRE payload with protocol type set to ’Transparent Ethernet Bridging (0x6558)’. It is a standard encapsulation type that wireshark can natively understand and be able to dissect and display packet contents including payload information. |