Networking

Ran into a good case of Network troubleshooting today. We use the private 192.168.0.0/16 address range at work. All of a sudden, we had users calling us from a particular floor stating that they cannot access network resources. Here’s how the team solved the issue

1) Physical connectivity – Is the cable plugged into the workstation? Is there a green light on the NIC?
2) DNS Lookup – Open a command prompt and do a look up on a server in the network. In this case, one of our file servers. Users say that they are getting a “server inaccessible” error. Hmm.. Alright, so we have physical connectivity, but don’t have DNS resolution. Since the rest of the user population didn’t have any DNS resolution issues, the team dug deeper
3) IP Address Details – Open a command prompt and type in “ipconfig”. This showed that the affected users were getting a “172.16.0.x” address. This range does not exist in our user network segment. So looks like we have users getting an unqualified network. Since all users are configured to get their IP addresses through DHCP, the culprit might be DHCP.
4) DHCP Server : The Issue – Open a command prompt and type in “ipconfig /all”. This showed the following

Aha.. looks like there is a rouge DHCP server in our network. The team checked the LAN room on the floor and found the culprit server. Looks like the server was installed in it’s default state where it acted as a DHCP server.

Lesson learnt : DHCP broadcasts sent by the client are addressed by servers in the same broadcast (Layer 2) domain before being routed to other DHCP servers (IP Helper Addresses).

We use 802.1x for our wireless security at work. The wireless controller uses Microsoft’s IAS as the RADIUS server. Recently during one of our maintenance windows, we installed a couple of critical patches and rebooted the IAS server. This was over the weekend and we didn’t check if wireless was working after the maintenance (one of the lessons learnt from his story :), put in automated monitoring so that you don’t have to worry about what services have come up or not after a maintenance window).

On Monday, our helpdesk gets swamped with calls of “wireless is not working”. We checked the controller and everything looked okay. Only error on the controller was that the RADIUS server was not responding. We checked the RADIUS server and the IAS service was running fine. But there were a ton of errors in the System event log with the following details

Access request for user XXXX\XXXXXX was discarded. Fully-Qualified-User-Name = XXX.XXXX.NET/XXX.XXX/TECHNOLOGY/DEVELOPMENT/XXX XXXXXX NAS-IP-Address = 192.168.128.10 NAS-Identifier = ACHIAS01IT Called-Station-Identifier = 00-0B-85-06-0C-A0:wacker Calling-Station-Identifier = 00-14-A4-28-4C-EE Client-Friendly-Name = ACHIAS01IT Client-IP-Address = 192.168.128.10 NAS-Port-Type = Wireless – IEEE 802.11 NAS-Port = 1 Proxy-Policy-Name = Use Windows authentication for all users Authentication-Provider = Windows Authentication-Server = Reason-Code = 23 Reason = Unexpected error. Possible error in server or client configuration. For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

“Unexpected error. Possible error in server or client configuration.“, now that is real informational :). We scratched our heads.. Thought it might be an issue with the controller (it was on an older firmware). Upgraded the firmware and rebooted the controller. Still no go. Same error. Finally frustrated, we opened up a case with MSFT. Even the eng. from MSFT was flabbergasted. The usual “Everything looks good, it should work!!”.

Finally we resolved the issue to an expired computer certificate for the IAS server. The certificate had expired a couple of weeks ago, but looks like the authentication was cached and when the server was rebooted, it caused the IAS service to error out. Renewing the cert caused the wireless clients to start authenticating immediately.

Am looking into what other services depend on a valid cert to work properly.

We use Microsoft PPTP as our VPN solution at work. Although PPTP is not robust as a IPSEC based vpn, it is a lot easier to deploy and maintain. The biggest advantage is that most of the Microsoft OS’s (Windows 2000, XP, 98) have PPTP clients built into the OS. With IPSec, one has to deploy client software to the workstations. Recently SSL VPN‘s have made strides in the remote connectivity market and are even easier to maintain.. but that is for another post.

We had a very interesting problem with remote users trying access hosts/URLs in our network recently. We have several nodes in the network, which we publish on external and internal DNS. So users on the internal network would access the node via the internal IP address and get access to privileged areas, while public users accessing the node would be restricted to only some areas of the site. Users connecting to the office network via VPN were being directed to the external address even though their VPN was configured to use the remote gateway as the default gateway. This in theory should direct all traffic to the internal network. We scratched our heads for a while 🙂 and during the troubleshooting session discovered that the DNS of the node was resolving to the external IP address. So the workstation was using the DNS server provided by the ethernet interface rather than the PPTP interface!!!.. How do we solve this?? Well Google, to the rescue and we find this obscure article on Mircrosoft’s website, which tells you how to change the bind order of the network interfaces so that the PPTP interface is used by default. Since this is a registry change, we have to figure out a way to push this out to the workstations. But that is a story for another post :)..

Cacti is a great tool to graph network utilization. I discovered it during my previous job to create some utilization graphs of satelite links. I highly admire the talent of the kid maintaining this software… Cacti can be used by any organization wanting to graph network utilization. It is also flexible enough to graph other stats (disk utilization, CPU utilization.. etc.)

I used Linux for all of my previous installs of Cacti. The whole install is very well documented. My team is still not very Linux savvy and wanted to try Cacti out in a Windows environment. There is adequate documentation for installing Cacti on Windows, but I ran into several issues when following this guide. If I was starting all over again, I would rather try this documentation. It is more up-to-date and detailed. The author missed mentioning that you have to change the “DocumentRoot” value in the Apache conf file.. But that is a minor issue.

Am still having issues with the scheduler tool in Windows to run the poller every 5 minutes. The scheduled job is only running when someone is logged into the server. As soon as you log off the server, the scheduler seems to be stopping. I will post an update as soon as I fix this.

My team uses the “Remote Assistance” functionality offered in Windows XP pretty extensively. One of the problems with the tool for the tech support personnel is that there is no easy shortcut to “offer” remote assistance. One has to launch remote assistance, search for “offer assistance” and then click on the link that shows up. Sounds easy, but if you are doing it 20 times a day, gets rather irritating :)..

Here is a trick to bypass the search.. Right click on your desktop and go to “New” –> “Shortcut”. Enter “hcp://CN=Microsoft%20Corporation,L=Redmond,S=Washington,C=US/Remote%20Assistance/Escalation/Unsolicited/Unsolicitedrcui.htm” (without qoutes “”) into the location of the item box and hit next. Choose a name for the shortcut and click on Finish.

Double click on the shortcut and voila.. instant access to offering remote assistance :).