Troubleshooting is about three big things: predicting what can happen, determining the anomalies , and investigating why that anomalies happened. Many network admins break down network infrastructure problems by analyzing the Layer 3 path through the network, hop by hop, in both directions. This process helps them isolate the problem; once they determine which hop in the layer path fails, they can then look further into the details.
There are various tools that can help with network troubleshooting. Let’s review them and see which issues they can help you investigate.
Cisco Discovery Protocol (CDP)
The Cisco Discovery Protocol (CDP) discovers basic information about neighbor routers and switches without needing to know the passwords for those Cisco network devices. This is possible because Cisco routers and switches routinely send out CDP messages that announce information about themselves. Therefore, Cisco hardware that supports CDP can learn about other devices by listening for these messages. CDP discovers several useful details from neighboring Cisco devices:
- Device identifier: The host name
- Address list: Network and data link addresses
- Port identifier: The interface on the remote device that sent the CDP advertisement
- Capabilities list: Device type (for example, router or switch)
- Platform: The IOS version running on the device
To see this information, use the show cdp command:
show cdp neighbors
This command lists each neighboring device, one per line. Each line provides the most important topology information about the neighbor: its hostname (device ID), the local device’s interface, and its interface (under the Port heading). This interface command also lists the platform, identifying the specific model of the neighboring router or switch.
To get additional details, such as the full name of the model of switch and the IP address configured on the neighboring device, add the detail parameter as follows:
show cdp neighbors detail
Of course, being able to discover a lot of information about neighboring devices is a network security exposure. Cisco recommends disabling CDP on any IP interface that does not have a need for it. To toggle CDP off and on for an entire device, use the no cdp run and cdp run global commands. To toggle it on a specific interface, use the no cdp enable and cdp enable interface subcommands.
Show Version
You can use the Cisco IOS command show version in privileged exec mode to verify the Cisco IOS version and release number of the IOS software running on Cisco devices. It outputs the following information:
- Cisco IOS software version — The name and version number of the Cisco software
- Switch uptime — The length of time since the device was last booted
- Switch platform — Hardware platform information, including revision and amount of RAM
- Processor board ID — The device’s serial number
Ping
The basic purpose for ping is to check for reachability, round trip time (RTT) and packet loss. In order to troubleshoot a device for these properties, we need to use specify the IP address of the device — for example, ping 172.17.4.6. This command sends an Internet Control Message Protocol (ICMP) echo request and displays one of the following:
! — An ICMP echo reply packet was received within the timeout period (2 seconds, by default)
. — No reply was received within the timeout period.
You can ping from the particular interface by adding the source parameter with the interface name at the end of the command — for example, ping 172.17.4.6 source Ethernet 0/0.
Traceroute
Traceroute is a function that traces the path from one network to another, so it can help diagnose the source of many problems. Traceroute works by sending remote host a sequence of three UDP datagrams with a TTL of 1 in the IP header; this causes the datagram to time out when it hits the first router on the path, causing the router to respond with an “ICMP time exceeded” message. Then traceroute sends a set of three UDP datagrams with TTL 2, so they time out when they hit the second router, causing it to responds with timeout message. This process continues until the packet reaches the final destination and receives a “port unreachable” ICMP message.
Therefore, you can traceroute to test the path that packets chose to move to their destination.
You can also use an extended traceroute command to test connectivity from a specified source — for example, traceroute 10.10.60.6 source Loopback0.
Telnet
When you use Telnet to connect to a remote device, it uses the default port (23). You can use any port number from 1 to 65535 to test whether a remote device is listening to the specific port, for example, telnet 172.17.5.74 8080.
Show Interfaces Command and Interface Status Codes
Cisco switches use two different sets of interface status codes. Both sets of status codes can determine whether an interface is working.
- Show interfaces and show interfaces description — These commands list the line status and protocol status. These generally indicate whether Layer 1 is working (line status) and whether Layer 2 is working (protocol status). For LAN switch interfaces, both codes typically have the same value, either “up” or “down.”
- show interfaces status — Lists the interface status. This single status code corresponds to different combinations of the line status and protocol status, as shown in the table below. For example, the “connected” interface status corresponds to an up/up state for the other two statuses.
Here is the list of status codes and the problems they can indicate:
| Line Status | Protocol Status | Interface Status | Possible Root Cause |
| Administratively down | Down | Disabled | Interface is disabled due to a shutdown command. |
| Down | Down | Not connected | No physical connection, mismatched speed, device is powered off, error disabled. |
| Up | Down | Not connected | And interface is not expected on physical interfaces. |
| Down | Down error disabled | Error disabled | Port security has disabled the interface. |
| Up | Up | Connected | The interface is working. |
Cisco Shutdown Command
When you first configure an interface in configure terminal mode, you must administratively enable the interface before the router can use it to transmit or receive packets. Use the Cisco no shutdown command to allow the IOS software to use the interface.
Later, you might want to disable a specific interface to perform hardware maintenance on it or a segment of a network. You might also want to disable an interface if a problem exists on a specific network segment and you must isolate that segment from the rest of the network. The shutdown command administratively enables an interface. To restart the interface, use the no shutdown command.
Show IP Route
Most routing tables contain a combination of static routes and dynamic routes. However, before any static or dynamic routing can be used, the routing table must contain the directly connected networks that are used to access remote networks. To verify the static routes in the routing table, use the show ip route command, specifying the network address, subnet mask and IP address of next hop router or exit interface.
Common Issues on Cisco Devices
Interface Speed and Duplex Issues
Many UTP-based Ethernet interfaces support multiple speeds (full- or half-duplex) and IEEE standard auto-negotiation. These interfaces can be configured to use a specific speed using the speed {10 | 100 | 1000} interface subcommand, and to use a specific duplex using the duplex {half | full} interface subcommand. If both of these are configured for an interface, the switch or router disables the IEEE-standard auto-negotiation process on that interface.
Both the show interfaces and show interfaces status commands list the speed and duplex settings on an interface, but only the show interfaces status command indicates how the switch determined the speed and duplex settings; it lists all autonegotiated settings with a prefix of a-. For example, a-full means full-duplex as auto-negotiated, whereas full means full-duplex but as manually configured. While auto-negotiation works well, the default values allow for the possibility of a problem called a duplex mismatch, in which the devices considers the link to be up but one side would use half-duplex and the other side would use full-duplex.
The number of input errors and the number of CRC errors are just two of the counters in the output of the show interfaces command. The challenge is deciding which counters you need to see, which ones show that a problem is happening, and which ones are normal and of no concern. Here is the list of counters to help you to start understanding which ones point to problems and which ones are just counting normal events that are not problems:
- Runts: Frames that did not meet the minimum frame size requirement (64 bytes, including the 18- byte destination MAC, source MAC and type). Runts can be caused by collisions.
- Giants: Frames that exceed the maximum frame size requirement (1518 bytes, including the 18-byte destination MAC, source MAC and type).
- Input Errors: The total number of counters, including runts, giants, no buffer, CRC, frame, overrun and ignored counts.
- CRC: Received frames that did not pass the FCS math; these can be caused by collisions.
- Frame: Received frames that have an illegal format (for example, ending with a partial byte); they can be caused by collisions.
- Packets Output: Total number of packets (frames) forwarded out the interface.
- Output Errors: Total number of packets (frames) that the switch port tried to transmit but for which some problem occurred.
- Collisions: Counter of all collisions that occurred when the interface was transmitting a frame.
- Late Collisions: The subset of all collisions that happen after the 64th byte of the frame was transmitted. In a properly working Ethernet LAN, collisions should occur within the first 64 bytes; late collisions today often point to a duplex mismatch.
Predicting the Contents of the MAC Address Table
Switches learn MAC addresses and then use the entries in the MAC address table to make a forwarding/filtering decision for each frame. To know exactly how a particular switch will forward an Ethernet frame, you need to examine the MAC address table on a Cisco switch.
The show mac address-table exec command displays the contents of a switch’s MAC address table. This command lists all MAC addresses currently known by the switch. The output includes some static overhead MAC addresses used by the switch and any statically configured MAC addresses, such as those configured with the port security feature. The command also lists all dynamically learned MAC addresses. If you want to see only the dynamically learned MAC address table entries, simply use the show mac address-table dynamic EXEC command.
When predicting the MAC address table entries, you need to imagine a frame sent by a device to another device on the other side of the LAN and then determine which switch ports the frame would enter as it passes through the LAN.
Port Security and Filtering
When tracing the path that a frame takes through LAN switches, remember that different kinds of filters can discard frames, even when all the interfaces are up. For example, LAN switches can use filters called access control lists (ACLs) that filter based on the source and destination MAC address, discarding some frames. Additionally, routers can filter IP packets using IP ACLs. In some cases, you can easily tell that port security has taken action because it has shut down the interface.
However, in other cases, port security leaves the interface up, but simply discards the offending traffic. From a troubleshooting perspective, a port security configuration that leaves the interface up but still discards frames requires the network engineer to look closely at port security status, rather than just looking at interfaces and the MAC address table. Port security allows three violation modes (shutdown, protect and restrict), but only the default setting of shutdown causes the switch to err-disable the interface.
To find evidence that port security is up and running, you would need to run the show port-security interface command. Also, the MAC address table gives some hints that port security might be enabled. Because port security manages the MAC addresses, any MAC addresses associated with a port on which port security is enabled show up as static MAC addresses. As a result, the show mac address-table dynamic command does not list the MAC addresses of the interfaces on which port security is enabled. However, the show mac address-table and show mac address-table static commands do list these static MAC addresses.
Ensuring that the Right Access Interfaces Are in the Right VLANs
To ensure that each access interface has been assigned to the correct VLAN, engineers simply need to determine which switch interfaces are access interfaces instead of trunk interfaces, determine the assigned access VLANs on each interface, and compare the information to the documentation. If possible, start by using the show vlan and show vlan brief commands, because these show commands list all the known VLANs and the access interfaces assigned to each VLAN. Be aware, however, that these two commands do not list operational trunks. The output does list all other interfaces (those not currently trunking), no matter whether the interface is in a working or nonworking state.
If the show vlan and show interface switchport commands are not available, the show mac address-table command can also help identify the access VLAN. This command lists the MAC address table, with each entry including a MAC address, interface and VLAN ID. If an interface is assigned to the wrong VLAN, use the switchport access vlan vlan-id interface subcommand to assign the correct VLAN ID.
Access VLANs Not Being Defined
Switches do not forward frames for VLANs that are not configured or that are configured but disabled (shut down). The show vlan command always lists all VLANs known to the switch, but the show running-config command does not. Switches configured as VTP servers and clients do not list the vlan commands in the current running configuration or the startup-config file; on these switches, you must use the show vlan command. Switches configured to use VTP transparent mode, or that disable VTP, list the vlan configuration commands in the configuration files. (Use the show vtp status command to learn the current VTP mode of a switch.) After you determine that a VLAN does not exist, the problem might be that the VLAN simply needs to be defined.
Access VLANs being Disabled
Another step in troubleshooting is to verify that each VLAN is active. The show vlan command lists one of two states: active or act/lshut. The latter means that the VLAN is shut down. Shutting down a VLAN disables the VLAN on that switch only, so that the switch will not forward frames in that VLAN. Cisco IOS gives you two similar configuration methods with which to disable (shutdown) and enable (no shutdown) a VLAN.
Check the Allowed VLAN List on Both Ends of a Trunk
If the allowed VLAN lists on the ends of a trunk are mismatched, the trunk cannot pass traffic for that VLAN. The output of the show interfaces trunk command on each side will look completely normal; you can spot the problem only by comparing the allowed lists on both ends of the trunk.
Mismatched Trunking Operational States
If trunking is configured correctly, both switches forward frames for the same set of VLANs. If trunks are misconfigured, there can be a couple of different results. In some cases, both switches conclude that their interfaces do not trunk. In other cases, one switch believes that its interface is correctly trunking but the other switch does not.
The most common incorrect configuration — which results in both switches not trunking — uses the switchport mode dynamic auto command on both switches on the link. The word “auto” makes us think that the link would trunk automatically, actually both switches wait for the other device on the link to begin negotiations. To spot this incorrect configuration, use the show interfaces switchport command to check whether both switches have the administrative state “auto” and that both operate as “static access” ports.
Conclusion
Now you know the basic troubleshooting commands to investigate issues that network administrators face every day. You can also download the Cisco Commands Cheat Sheet for a quick reference list of troubleshooting commands and their descriptions at hand.
