Managing the unmanageable

Uphill battle

When I first started working at LES, sometime in the last century, the computers were networked together using some high-tech gizmos called “hubs“. These hubs would reach a maximum speed of 10Mbps on a good day, if there were only two devices connected and the solar flares were at a minimum.

Time marched on and we upgraded to 10/100Mbps hubs, then 10/100Mbps switches, and then, finally, in the last few years to unmanaged gigabit switches. One of the biggest problems with using unmanaged switches is that the network can be brought to a standstill using a simple patch cable, plugged into two network sockets. I’ve become pretty adept at recognizing the signs of a network switching loop (the lights on the switches are flickering like the last few seconds on the timer in Mission Impossible, the servers are inaccessible, the teachers are waiting outside my office with baseball bats). One of our network loop disasters hiccups even managed to anonymously make it to a site dedicated to technology-related problems.

Over the last month, though, I had lots of small problems that never quite reached the level of crashing the network. Our Fedora systems, connected to the server via NFS, would occasionally freeze for a few seconds, and then start working again. Our accountants, who are running Windows, complained that their connection to the server was being broken a couple of times each day, causing their accounting software to crash. And pinging any server would result in a loss of ten-fifteen packets every ten minutes or so.

I checked our switches for the flicker of death and came up dry. I tried dumping packets from a server on one side of the school to a server on the other side of the school and consistently reached 1Gbps. In desperation, I retipped the Cat6 cable connecting the switches that form the backbone of our network. All to no avail. I decided to wait until evening and then unplug the switches one at a time until I found the problem. The problem disappeared.

The next morning it was back. I had two options. Disconnect the switches one port at a time in the middle of the school day, while teachers, students and accountants are all trying to use the system. Or put in a request for some managed switches and see if they could help us figure out what the heck was going on. Hundreds of irritated users outside my door… or new kit. It was a hard call, but I went for the new kit.

We started with an eight-port MikroTik switch/router, and, after I tested it for a day, we quickly grabbed a couple more 24-port MikroTik switches (most of our backbone locations have nine or ten ports that need to be connected and MikroTik either does 8 or 24 ports).

After we got the three core locations outfitted with switches, I quickly got messages on the switches pointing to a potential network loop on a link to one of our unmanaged leaf switches in the computer room, which was connected to another unmanaged five-port switch that had apparently had a bad day and decided it would start forwarding packets back through itself.

I replaced the five port switch with a TP-Link five-port router running OpenWRT and, just like that, everything was back to normal.

I am never going back to unmanaged switches again. Having managed switches as our network’s backbone reduced the time to find the problem by a factor of 10 to 20, and, if we’d had managed switches all the way through the network from the beginning, we could have zeroed in directly on the bad switch rather than spending weeks trying to work out what the problem was.

So now we’re back to a nice quiet network where packet storms are but a distant nightmare. Knock on wood.

First Work: Myth of Sisyphus detail #1 by AbominableDante, used under a CC BY-NC-ND license


Comments

oxtan
Friday, May 1, 2015

running a work environment on home equipment is a nightmare. Those are one of the big red signs you have to tackle and get fixed or leave.

Another one is not provisioning funds for replacing hardware.

Jonathan Dieter
Monday, May 4, 2015

While I do agree with your assessment in theory, you do have to remember that we’re in a school in Lebanon where money isn’t as plentiful as it is in the West. The other side of the tradeoff, though, is that some downtime is acceptable. My boss is pretty understanding when we have problems because of the hardware we use.

Having said that, the school has been quite good at getting me the hardware I need when I ask for it. I just have to be careful about asking for what I need, not what I want. And, after this experience, managed switches have moved from the “want” column to the “need” column.

Cristian Ciupitu
Monday, May 4, 2015

Shouldn’t the STP protocol detect and prevent these loops? If I remember correctly even some cheap unmanaged switches had support for it.

Jonathan Dieter
Tuesday, May 5, 2015

Yes, but our unmanaged switches were cheap enough (and probably old enough) that they didn’t support STP.

 

Thank you

Your comment has been submitted and will be published once it has been approved.

To check its status, you may click here

OK

There was a problem

Your post has not been submitted. Please return to the page and try again.

OK

There was a problem

Your IP has been blocked from posting comments because it comes from an IP range that has attempted to post spam on this site. If you want to post a comment, you must email me the comment, including the URL that you want the comment posted on and your name.

OK