Reality Bites

The big device with all the cables plugged into it is a 48 port switch.  All computers on a network "talk" to the server through a switch like this. This switch connects 48 computers or other network devices (like printers or scanners) to the server. Actually it's more complicated than that, but that's enough for now.

At 5:50 on Tuesday morning my cell phone rang. It was "the guys" who remotely support and manage the network at the company where I work. I was lying in bed, awake, but my alarm had not yet rung.

"Hi Lynne, how are you today?"

I laughed, "If you guys are calling me at not-quite-six in the morning, my day has just gone to hell, and I'm not even out of bed yet!"

Turns out several users out in the factory had no network connection. Problem was, SOME of them did, and SOME of them didn't. There's nothing harder to troubleshoot than something that is intermittent.

The folks at the far end of the factory connect to the network through two smaller switches hung up in the rafters of the building. It was clear that one of them wasn't working.  Except when I rode up the lift truck to look at them (about twenty feet above the factory floor) they were lit up and seemed to be working.

I went back out to the front office and since it was barely 8:30 AM, I went to the front to turn on the lights and was stopped by the VP of Manufacturing. "I can't get on the network" he told me.

WELL! That meant my problem was not one of the small switches out in the factory, but hey, the users who complained were all out there when I arrived at 7 AM, so I had to start there.

As soon as I opened the door to the server room, I could see one switch that didn't have any lights blinking. Obviously that was the source of the trouble. I called "the guys" and while I was talking I unplugged the switch and plugged it back in, hoping to jump start it. Nope. So I unplugged it again and tried another UPS (uninterruptible power supply) because sometimes outlets die, and we had had a ton of rain the day before that leaked into the first aid room next door. The switch came to life, lighting up like a Christmas tree, and then running through a series of tests where the lights blinked on every port.

AND THEN I HEARD A BIG BANG AND SMELLED ELECTRICAL BURNING.  OH CRAP!

I yanked out the cord and made sure nothing was on fire.

By the time I reached the phone to tell "the guy" at the other end of the line that the switch was well and truly dead and we needed a replacement ASAP two of my colleagues were standing at the door asking if I was OK. Apparently I either squeaked or screamed. I was certainly shaking.

So that was how my day started. A little while later one of "the guys" showed up in person and we painstakingly identified connections that weren't working and moved those cables to other switches, thus bringing everything important back on line. We knew we'd miss some (like the 3-D printer, but we fixed that the next day) but by and large everybody was working, the UPS computer was up so we could ship product, the computer that connected the two CNC machines was working so that computer could transmit jobs to the CNC machines (thus keeping production going. Being "dead in the water" is a real risk, an EXTREMELY big deal, and should be avoided at all costs), and the guys in the stockroom could pull raw materials to fill shop orders, and receive orders into the system, bar code labels could be printed and the big laser printer out there could print.

Later as a colleague and I reviewed the quotes for a new switch, and as we debated getting one or two (since we'd had planned to replace the one that died and one other eventually, but not quite yet) he mentioned that the other big switch was almost 10 years old and the estimated life cycle of a switch is 10 years. "Then we should replace both of them" I said, "because if the other one goes, then everything in the front office goes down with it - sales, engineering, purchasing, accounting, HR and payroll."

He stopped dead in his tracks, his eyes widening.

"OK, so we replace both of them."

So that's the next trick.

In case you all wondered what I did in my day job.  Trust me, figuring out how to put a quilt together is a hell of a lot less stressful.