Chapter 13
 | Standard Troubleshooting Procedures |
| TCP/IP Troubleshooting Utilities |
| Troubleshooting Process |
No matter how well implemented a network is, problems will always come up. The problem could be serious, such as losing TCP/IP communication between sites, or minor, such as an end user not having a TCP/IP address. The bottom line is that these problems require your immediate attention. A relatively minor problem may not seem urgent to you, but to the end user it might be critical, affecting his/her ability to effectively perform their job. By the same token, a network-wide crisis, such as the loss of TCP/IP communication between WAN nodes, obviously requires quick resolution. Fortunately, you have a methodical, systematic manner in which to approach all TCP/IP related problems.
Troubleshooting TCP/IP issues requires a good understanding of TCP/IP and awareness of tools and logical procedures designed to assist you. These troubleshooting tools and procedures will augment your network street-smarts. It is also important to grasp the big TCP/IP picture. Together, these elements help you successfully troubleshoot TCP/IP problems.
Network administrators and engineers have been troubleshooting TCP/IP problems for many years. There are some fundamental troubleshooting procedures that allow you to move up the learning curve to quickly identify and resolve your TCP/IP connectivity issues.
The first step is recognizing that a problem exists. End users will definitely let you know the moment they lose a service. Your WAN is not going to send you a message that the router is no longer passing TCP/IP traffic, but an end user who attempting to access a service across your WAN will let you know if it fails. TCP/IP connectivity issues fall into four different categories: network, subnet, host, and service. You will employ different tools depending on the clues you receive from attempting to duplicate problems or from asking the end user questions relating to the failure.
A network TCP/IP problem occurs when data is not being passed between networks. This can be the result of an incorrectly configured default gateway or a router that is failing or not configured to pass TCP/IP traffic to a needed destination. For example, Tom is an end user on your network. He receives instructions to visit a specific URL on your corporate Web site. When Tom types the URL in his Web browser, the page is not loaded. The configurations of Tom’s desktop are accurate; he can access any other corporate URL; but your corporate URL fails. After using the PING utility, you discover that your network router is not aware of the network where the requested URL address resides. The router configuration is amended and Tom gets access to the URL.
A subnet TCP/IP problem occurs when data is not being passed between hosts on the same subnet. This is usually the result of an incorrectly configured subnet mask on the affected host computer. For example, Sue is an end user on your network. She has a report due that will make or break the company’s earning potential for years to come. When Sue realizes that she can’t print the report, she immediately notifies you, believing that the printer is malfunctioning. Aware of such issues, you check her TCP/IP configuration and notice that she has the wrong subnet mask entered, which is preventing her from communicating with the network printer. The configuration is changed and Sue is now able to print.
A host TCP/IP problem occurs when a host computer/printer is configured incorrectly, preventing other hosts from communicating with it. This problem could be the result of an incorrect TCP/IP address, subnet mask, or DHCP configuration. For example, Tom changes the configuration of his workstation because he wants his IP address to be 1.1.1.1. Unfortunately, his network address is 200.200.200.0. After running IPCONFIG, you determine that with the IP address Tom specified, his work over the network would not get done. You give Tom the appropriate configuration and he is reconnected.
A service TCP/IP problem occurs when a service such as Telnet, FTP, or DNS is not functioning properly. The problem could be a result of the service not being installed or, in the case of DNS, not returning valid host to IP address information. For example, Sue has just learned the wonders of telneting from machine to machine. She successfully telnets to a UNIX machine on her network but when she attempts the same to the Windows NT server, the connection fails. Unfortunately, in this situation you have not installed the telnet service on your Windows NT server, and Sue cannot use an unimplemented service.
There are several TCP/IP utilities that can assist you in your investigation and troubleshooting of TCP/IP network issues.
The PING utility is a fast method to test TCP/IP connectivity and verify that a TCP/IP address is reachable. Ping can be used to test the TCP/IP connection on the local host or to a remote host. Ping sends an ICMP echo_request packet to a destination address that is expected to return an ICMP echo_response packet. The destination host, if reachable, returns an equal reply packet to the originating node. This response signals that you have reached the intended destination TCP/IP address and, in turn, have connectivity between your location and the pinged host. Several PING utilities offer summary information in addition to the echo_response. Summary information can include the number of packets transmitted, the number of packets received, the percentage of packet loss and minimum, and average and maximum packet round-trip time. Figure 13-1 shows an example of pinging the White House.
Figure 1: Using the PING Utility
As seen from another PING utility:
>ping www.whitehouse.gov
www.whitehouse.gov is alive.
Indeed www.whitehouse.gov sends a reply and the PING is successful. If you receive a message stating Bad IP address www.whitehouse.gov or PING: unknown host www.whitehouse.gov, your connection attempt failed. It is time to investigate the DNS and/or host file for existence and validity of the failed TCP/IP address to its domain-name.
Ping utilities offer several options, which allow for continuous pinging, varying packet size, and/or a specific number of echo requests to send.
Figure 13-2 shows an example of using command-line options to specify this Windows NT workstation to send five echo_requests [-n 5] of a 128 byte packet size [-l 128].
Figure 2: Using Command-Line Options with the PING Utility
Here are the ping results from another utility configured to send the same information:
> ping -s www.whitehouse.gov 128 5
PING www.whitehouse.gov: 128 data bytes
136 bytes from www1.whitehouse.gov (198.137.240.91): icmp_seq=0. time=411. ms
136 bytes from www1.whitehouse.gov (198.137.240.91): icmp_seq=1. time=292. ms
136 bytes from www1.whitehouse.gov (198.137.240.91): icmp_seq=2. time=165. ms
136 bytes from www1.whitehouse.gov (198.137.240.91): icmp_seq=3. time=157. ms
136 bytes from www1.whitehouse.gov (198.137.240.91): icmp_seq=4. time=89. ms
----www1.whitehouse.gov PING Statistics----
5 packets transmitted, 5 packets received, 0% packet loss
round-trip (ms) min/avg/max = 89/222/411
The PING utility comes in handy to quickly troubleshoot TCP/IP network connection issues.
Exercise 13-1 Troubleshooting Scenario Using PING
The following procedures are general examples of how PING can be implemented to determine TCP/IP connectivity failures on your network.
The first step is to send an echo_request to TCP/IP address 127.0.0.1. A fail response from the loopback address, 127.0.0.1, indicates that TCP/IP is not properly installed/configured on your computer. A reply from the loopback address confirms that TCP/IP is installed properly and that you should investigate further.
The second step identifies computers on your subnet. Ping the TCP/IP address of a neighboring computer or printer. A fail response is typically due to a typo or invalid subnet address in your TCP/IP configuration. If you receive a reply from the neighboring address, then you should plan to investigate deeper.
The third step checks for a response from the near-side interface of your router, necessary for internetwork communication. Ping your network default gateway. A fail response from the default gateway could mean that the router interface is down; start troubleshooting your router!
The fourth step checks for a response from the far-side interface of your router. Ping the far-side of the router. This ensures that your router has routed the echo_request to the correct interface. If the PING fails, troubleshoot the router again.
The fifth step verifies connection to a remote host and segment. Ping a well-known (reachable) remote host. It is possible you decided to skip pinging the router near and far interfaces, and went right for the glory of pinging the remote host after that successful local host reply. Without an echo_response, odds are that you could have an invalid default gateway setting in your machine’s TCP/IP settings.
The ARP utility is used to view and make changes to IP address to MAC address translation tables. ARP uses these tables to determine how your machine will resolve the IP address of transmitted packets to their MAC address. In order to view your computer’s ARP cache, type arp -a, as shown in Figure 13-3.
Figure 3: A Computer’s ARP Cache
The ARP utility can be used to add and delete entries in your machine’s arp cache. The command-line option [-d] deletes an arp cache entry and option [-s] adds a static entry to your arp cache, s shown in Figure 13-4.
Figure 4: Using arp -s to Add a Static ARP Cache Entry
The ARP utility is useful to determine the IP address to MAC address translation of transmitted packets on your computer.
Using arp -a will detail the IP address to MAC address information present in your machine’s arp cache. If the MAC address destination of the packet to transmit matches an entry in this cache, then your transmission will resolve the IP address. If the destination MAC address is not in the ARP cache, then ARP sends an ARP request packet to hosts on your subnet. Should a host reply, then the MAC to IP address is resolved., If not, another ARP request packet is sent to the subnet’s default gateway (router) to be routed to a remote network for host resolution.
The NETSTAT utility is useful for relating protocol statistics and current active connections utilizing TCP/IP. Local addresses are shown as IP address and port number of the connection. Foreign addresses are shown as IP address and port number of the connection. Host names may take the IP address place when HOSTS file entries are present. Asterisks (*) may also appear, denoting an unassigned port. To view your computer’s current connections, type netstat -a,. as shown in Figure 13-5.
Figure 5: Output from the NETSTAT Utility
Using command-line options, NETSTAT offers many ways to personalize the statistics you are given. The [-e] option relates Ethernet information, the [-s] option is used to specify which protocol’s statistics you are interested in observing, and the [-r] option relates the NETSTAT routing table.
Figure 13-6 shows an example of Ethernet statistics derived by using the command netstat -e.
Figure 6: Output of Ethernet Statistics Generated with netstat -e
NETSTAT is useful in troubleshooting protocol-related issues. A prime example is a situation where you suspect a problem with TCP connectivity. You use the netstat -a command to verify that the TCP status of a desired connection is not returning an ESTABLISHED status message. Other messages tend to indicate a problem with the connection or with the queue. To retrieve and verify routing information, use netstat -r..
The NBTSTAT utility is useful for relating protocol statistics and current active NBT connections describe NetBIOS over TCP/IP. Essentially, NBTSTAT describes the computer names cache of a computer. For example, my computer’s NetBIOS name is Ducati, my neighbor’s computer NetBIOS name is Honda. My computer is aware of Ducati as its computer name but uses an LMHOSTS file, WINS, or a broadcast to populate its NBT computer names cache. In order to view your computer’s NetBIOS cache type nbtstat -n, as shown in Figure 13-7.
Figure 7: An Example of a Computer’s NetBIOS Cache
NBTSTAT also has command-line options such as [-a remotename], which lists the remote computer’s name cache. Type nbtstat ? to view other NBTSTAT options.
Using NBTSTAT to troubleshoot is limited to NetBIOS-related issues. A common problem that can be resolved using NBTSTAT is one in which the local LMHOSTS file contains errors. This situation demands that you correct the LMHOSTS file. Once the file has been replaced, simply type nbtstat -R. This switch removes all the entries in the NBT name cache and reloads the information from the corrected LMHOSTS file. You might also use nbtstat to investigate an error message such as Event 4320, NetBt Error. This error tells you that another computer on your network has the same IP address. You can’t have more than one NetBIOS name per IP address; therefore, you must troubleshoot the issue using nbtstat -n. The result is the name of the machine with the same IP address.
The IPCONFIG utility is extremely useful, as it relates the entire TCP/IP configuration present in your machine. In addition to the configuration information the utility provides, it has the added capability of interfacing with DHCP services to renew or release a leased DHCP TCP/IP address. Using IPCONFIG will return the computer’s TCP/IP address, its subnet mask, and default gateway.
Figure 13-8 shows the results of typing ipconfig.
Figure 8: Sample IPCONFIG Output
IPCONFIG has several command-line options. The most widely used is the [/all] parameter. The ipconfig /all command will issue an abundance of information, including the computer’s host name, DNS servers, node type, physical address (MAC address), DHCP status, IP address, subnet mask, and default gateway. Computers that derive their IP addresses from a DHCP server can use additional command-line options of ipconfig [/renew [adapter]] and [/release [adapter]], which enable you to renew your DHCP IP address lease or release your DHCP lease on specific or all network adapters using DHCP.
Figure 13-9 shows an example of using command-line options to view all TCP/IP configuration information available on a Windows NT workstation.
Figure 9: Sample IPCONFIG Output Using the /all Parameter
IPCONFIG is one of the first tools to turn to in your troubleshooting efforts. Engineers use it regularly since it provides such detailed and useful information regarding a computer’s complete TCP/IP configuration. Typing ipconfig is a quick way to determine if an IP address, subnet mask, or default gateway is valid for your network. Imagine administrating two subnets. Tom is the only user having problems accessing an IP host on another subnet. The subnet mask of Tom’s network is 255.255.255.192 and the results of typing ipconfig on Tom’s machine are:
IP address : 204.241.155.70
Subnet mask : 255.255.255.192
Default gateway : 204.241.155.130
With a subnet mask of 255.255.255.192 on a Class C network, there can be two subnets with a block size of 64. Therefore, Tom’s IP address is not on the same subnet as the default gateway. You have determined that other users on Tom’s subnet have no problems connecting to the remote subnet and their IP addresses are within the same block with the default gateway. The conclusion is that Tom’s IP address is incorrect.
Exercise 13-2 Troubleshooting Scenario Using IPCONFIG
The TRACERT utility is commonly used to locate failures along a TCP/IP communications path. It is similar to PING in that it makes use of ICMP echo request packets to follow the route the packet takes to a specific destination. A benefit that TRACERT provides over PING is its implementation of Time-to-Live (TTL) values which effectively relate a route metric (hop count). Routers along the communications path of an ICMP echo request packet must reduce the packets TTL by at least one before forwarding it to the next router, and continuing in this fashion until the destination address is reached or the maximum TTL is reached. In order to determine the number of hops to the White House’s WWW server, you simply type tracert www.whitehouse.gov, as shown in Figure 13-10.
Figure 10: Using the TRACERT Utility
TRACERT offers several command-line options. A c[C]ommon parameter (s The [-w] option enables you to specify the time-out in milliseconds for each hop(,).To see the host name associated with each of the hops during a TRACERT, use the [-d] parameter. To specify the maximum number of hops, use [-h]. The [-j] option tells TRACERT to lose the source route along a host list.
Figure 13-11 shows an example of using the tracert -d command.
Figure 11: TRACERT Output Without Hostname Associations
TRACERT is helpful when attempting to troubleshoot the point at which an ICMP packet fails to be forwarded in a route. This information is useful for following the connection path of packets on your local network as well as an accepted method for testing Internet connections. A common situation in which to use TRACERT is when a user is accessing your network via a PPTP connection using their Internet Service Provider (ISP). For example, Sue is at home and wants to telecommute. She has dialed into her ISP and is attempting to establish a connection to you network but is unsuccessful. The first step in supporting Sue is instructing her to use TRACERT. If she receive the message "destination host unreachable," you know that TRACERT lost its route at that point. Sue may also find that TRACERT times out or enters a loop state where a problem exists.
Exercise 13-3 Troubleshooting Scenario Using TRACERT
The ROUTE utility is used to configure network routing tables. This is the tool used to add, delete, change, or print entries in your routing tables. Manually entering routes may be tedious but necessary in some cases to ensure connectivity. For example, to add and delete a route:
ROUTE ADD 200.200.210.0 MASK 255.255.255.0 200.200.210.1
ROUTE DELETE 200.200.210.0 MASK 255.255.255.0
Additional features help make using the ROUTE command more bearable. The command-line option [-f] in addition to a command cleans the routing table prior to making a change. This is a useful feature that allows you to make changes to your routing tables, while at the same time ensuring they will be refreshed and updated with your change. Imagine having to retype routes every time the router is recycled or restarted. Another parameter ensures that routes added will remain static and persistent. This is the option [-p], which is used with the ADD command.
The ROUTE utility is used mainly as a result of troubleshooting. The following scenario depicts a situation in which you would use the ROUTE add command as the last phase of troubleshooting. The IS department decides to update the company’s intranet. During their upgrade they plan to change the IP address associated with the home page URL. The weekend comes and goes, and IS has successfully updated the intranet. Tom shows up the next morning and is the first end user who decides to surf the corporate Web site. Tom enters the URL and the usual page does not appear. Tom notifies you that he is not able to access the intranet home page. You immediately remember that without an interior routing protocol such as RIP, you must manually update the routing table of your router using the ROUTE add utility. Once the routes are updated reflecting the new subnets, TCP/IP traffic can flow to the new subnets on which the intranet Web pages now reside.
NSLOOKUP is used to examine information from DNS servers. It is helpful to have an understanding of how DNS servers provide IP address to host name resolution services. NSLOOKUP consists of both interactive and noninteractive modes based on the original UNIX tool. Tasks requiring one bit of information should be derived in noninteractive mode. Tasks that require more information require the use of interactive mode.
Here is an example of using noninteractive mode:
NSLOOKUP WWW.WHITEHOUSE.GOV DNSSERVER1
and an example of using interactive mode:
NSLOOKUP -
The NSLOOKUP utility is used only for troubleshooting DNS issues. For example, nslookup would be useful if you find that IP address to host name translation is not working as expected. Using NSLOOKUP you can review information contained in the reverse lookup file or other DNS zone files of your DNS server.
Event Viewer and Network Monitor are tools analyze and identify TCP/IP connectivity issues.
Event Viewer is a valuable tool for front-line recognition of a problem that may be hardware, software, network, security, or system related. It has a user-friendly interface with icons that run blue in color if acceptable, yellow for a potential problem, and red for an event that should be researched immediately. A description of the event noted will be included with logged events.
Figure 13-12 shows the details of an Event Viewer.
Figure 12: Example of an Event Detail
Network Monitor is capable of a high degree of complexity in diagnosing network traffic. It is capable of using both capture and display filters to aid in your evaluation of network packets. Network Monitor utilizes the NDIS 4.0 driver, which enables a particular network interface card (NIC) to capture outbound or inbound packets of the associated computer. Therefore, NDIS 4.0 enables Network Monitor to consume much less of a processor’s resources than a NIC card set to operate in promiscuous mode. In essence, Network Monitor can function as a protocol analyzer or as NIC specific trap filter.
Event Viewer offers a quick and simple method for identifying a problem. Simply open the Event Viewer and you see current status information that is potentially relevant to am issue you are attempting to resolve.
Network Monitor enables you to collect protocol statistics. In order to view TCP/IP protocol statistics, you should enable a capture filter. This filter should be configured to capture TCP/IP statistics and log them to a file. This is a nice feature, allowing you to go back and review the files at your leisure. In addition, you can set up a display filter to diagnose the captured TCP/IP statistics based on properties such as source and destination address of the packets.
Exercise 13-4 Using Event Viewer to View TCP/IP Messages
Troubleshooting processes vary based on the administrator’s experience in dealing with and resolving TCP/IP connectivity issues. For example, you cannot just turn to page 69 in a manual and find out why your network just lost TCP/IP connectivity with your home office in Montana. A manual certainly will not provide instructions on dealing with your end users, all of whom have urgent deadlines, need connectivity with that office, and will soon be haunting your every move. Remember to approach your dilemma logically and methodically. It is often best to start locally and progress more toward the WAN in your network diagnosis. In addition, attempt to re-create the problem, draw the problem representation on paper, and verify the extent to which your network is being affected. Finally, narrow down the possible problem areas by successively testing different network areas using the troubleshooting utilities discussed here.
Let’s troubleshoot a scenario that could occur many times within any given day. A user calls and tells you they cannot log on to the Windows NT domain in a TCP/IP network.
It is best to start analyzing the problem at the bottom layer of the TCP/IP network architecture.
If the answer is yes, then IP between the Network layer and the Internet layer is working fine.
IP is resolving the IP address to hardware address. The lower layers are working great.
If the answer is no, then check the workstation’s IP configuration information. Use IPCONFIG to view the IP parameters. Look at the values of the IP address, subnet mask, and default gateway. Usually the problem will be in one of these settings.
If the answer is yes, then things are working fine between the Network all the way up through the Application layer.
If the answer is no, then the problem is within resolving the name to the IP address. Check the various methods that your network may be using to resolve names to IP addresses. An incorrect entry in any of these will prevent name resolution (WINS, DNS, LMHOSTS file, or HOSTS file).
— by D. Lynn White, MCT, MCSE
The troubleshooting section on the Microsoft exam are heavily rooted in your ability to evaluate results from the utilities discussed in this chapter and apply them to a TCP/IP connectivity issue. The most commonly used utilities that deal with most TCP/IP network problems are PING, IPCONFIG, and TRACERT. The other utilities are definitely important but are specialized for specific issues.
It is essential that you understand the concepts of IP addressing, subnetting, network classes, and basic network connectivity. Reviewing how each utility assists in determining the nature of a problem also plays a big role in how effectively you can tie the TCP/IP ideas together. You should know the results of an incorrectly configured subnet mask, default gateway, and IP address. You must also understand how to troubleshoot from your desktop to the remote networks in your system.
| There are some fundamental troubleshooting procedures that allow you to move up the learning curve to quickly identify and resolve your TCP/IP connectivity issues. |
| The first step in troubleshooting is recognizing that a problem exists. |
| A network TCP/IP problem occurs when data is not being passed between networks. |
| A subnet TCP/IP problem occurs when data is not being passed between hosts on the same subnet. |
| A host TCP/IP problem occurs when a host computer/printer is configured incorrectly, preventing other hosts from communicating with it. |
| A service TCP/IP problem occurs when a service such as Telnet, FTP, or DNS is not functioning properly. |
| There are several TCP/IP utilities that can assist you in your investigation and troubleshooting of TCP/IP network issues. |
| The PING utility is a fast method to test TCP/IP connectivity and verify that a TCP/IP address is reachable. |
| When attempting to troubleshoot a connectivity issue, ping in successive order: the loopback address, a local host, the default gateway, the far-side of the router, and finish by pinging the remote host. |
| The ARP utility is used to view and make changes to IP address to MAC address translation tables. |
| The NETSTAT utility is useful for relating protocol statistics and current active connections utilizing TCP/IP. Local addresses are shown as IP address and port number of the connection. |
| The NBTSTAT utility is useful for relating protocol statistics and current active NBT connections describe NetBIOS over TCP/IP. |
| The IPCONFIG utility is extremely useful, as it relates the entire TCP/IP configuration present in your machine. |
| Make sure you understand how to utilize ipconfig output to diagnose potential connectivity problems such as the example presented in previous section. |
| The TRACERT utility is commonly used to locate failures along a TCP/IP communications path. |
| The ROUTE utility is used to configure network routing tables. |
| NSLOOKUP is used to examine information from DNS servers. |
| Event Viewer and Network Monitor are tools analyze and identify TCP/IP connectivity issues. |
| Troubleshooting processes vary based on the administrator’s experience in dealing with and resolving TCP/IP connectivity issues. |
| Remember to approach your dilemma logically and methodically. It is often best to start locally and progress more toward the WAN in your network diagnosis. |
The Self Test questions will help you measure your understanding of the material presented in this chapter. Read all the choices carefully, as there may be more than one correct answer. Choose all correct answers for each question.
D. Ping. Ping is the only utility mentioned which sends an ICMP echo_request packet to a remote host.
2. Which utility is used in order to make changes to IP address to media access card address translation tables?
C. ARP. The ARP utility is used to make changes to a computer’s IP address to MAC address translation table. Arp -s adds an entry to the table.
B. ROUTE delete. Using route delete with the -f option, end users will be prevented from accessing the subnet which the router was previously aware of. Filternet and Prevent /securenet are not valid utilities.
7. Which utility allows for researching DNS files for specific IP address to host name assignments?
B. Ping. The PING utility is the first and most commonly used diagnostic utility