Thursday, April 19, 2018

Reality Bites

The big device with all the cables plugged into it is a 48 port switch.  All computers on a network "talk" to the server through a switch like this. This switch connects 48 computers or other network devices (like printers or scanners) to the server. Actually it's more complicated than that, but that's enough for now.

At 5:50 on Tuesday morning my cell phone rang. It was "the guys" who remotely support and manage the network at the company where I work. I was lying in bed, awake, but my alarm had not yet rung.

"Hi Lynne, how are you today?"

I laughed, "If you guys are calling me at not-quite-six in the morning, my day has just gone to hell, and I'm not even out of bed yet!"

Turns out several users out in the factory had no network connection. Problem was, SOME of them did, and SOME of them didn't. There's nothing harder to troubleshoot than something that is intermittent.

The folks at the far end of the factory connect to the network through two smaller switches hung up in the rafters of the building. It was clear that one of them wasn't working.  Except when I rode up the lift truck to look at them (about twenty feet above the factory floor) they were lit up and seemed to be working.

I went back out to the front office and since it was barely 8:30 AM, I went to the front to turn on the lights and was stopped by the VP of Manufacturing. "I can't get on the network" he told me.

WELL! That meant my problem was not one of the small switches out in the factory, but hey, the users who complained were all out there when I arrived at 7 AM, so I had to start there.

As soon as I opened the door to the server room, I could see one switch that didn't have any lights blinking. Obviously that was the source of the trouble. I called "the guys" and while I was talking I unplugged the switch and plugged it back in, hoping to jump start it. Nope. So I unplugged it again and tried another UPS (uninterruptible power supply) because sometimes outlets die, and we had had a ton of rain the day before that leaked into the first aid room next door. The switch came to life, lighting up like a Christmas tree, and then running through a series of tests where the lights blinked on every port.

AND THEN I HEARD A BIG BANG AND SMELLED ELECTRICAL BURNING.  OH CRAP!

I yanked out the cord and made sure nothing was on fire.

By the time I reached the phone to tell "the guy" at the other end of the line that the switch was well and truly dead and we needed a replacement ASAP two of my colleagues were standing at the door asking if I was OK. Apparently I either squeaked or screamed. I was certainly shaking.

So that was how my day started. A little while later one of "the guys" showed up in person and we painstakingly identified connections that weren't working and moved those cables to other switches, thus bringing everything important back on line. We knew we'd miss some (like the 3-D printer, but we fixed that the next day) but by and large everybody was working, the UPS computer was up so we could ship product, the computer that connected the two CNC machines was working so that computer could transmit jobs to the CNC machines (thus keeping production going. Being "dead in the water" is a real risk, an EXTREMELY big deal, and should be avoided at all costs), and the guys in the stockroom could pull raw materials to fill shop orders, and receive orders into the system, bar code labels could be printed and the big laser printer out there could print.

Later as a colleague and I reviewed the quotes for a new switch, and as we debated getting one or two (since we'd had planned to replace the one that died and one other eventually, but not quite yet) he mentioned that the other big switch was almost 10 years old and the estimated life cycle of a switch is 10 years. "Then we should replace both of them" I said, "because if the other one goes, then everything in the front office goes down with it - sales, engineering, purchasing, accounting, HR and payroll."

He stopped dead in his tracks, his eyes widening.

"OK, so we replace both of them."

So that's the next trick.

In case you all wondered what I did in my day job.  Trust me, figuring out how to put a quilt together is a hell of a lot less stressful.



10 comments:

Nancy J said...

And now we know the regular 9 to 5work,or in this case a whole lot earlier.So glad you all were safe. No wonder you so enjoy quilting!!!

Megan said...

A work phone call before 6am is never good news! And everything else can go down, but NOT PAYROLL!!! Everyone's got bills to pay. LOL

Megan
Sydney, Australia

Linda Swanekamp said...

Never had a quilt light up and go bang. Thankfully. Sounds like a wild ride.

Quiltdivajulie said...

Now I know why you sent silent .... they should be grateful you were there and able to help get it resolved! And yeah - I like my quilts/studio far more than I did my old day jobs -- no contest!

Just Ducky said...

Heaven forbid that payroll be off line. Heck yeah, replace them both.

Pat said...

I've had a few of "those" calls in my past and am very happy to be retired. Glad you found the problem and are taking appropriate measures to fix/prevent it. I wondered where you were yesterday as you are seldom AWOL, now I know. :)

Robby said...

First of all, big hurrah that you didn't go up in smoke! Second, at least you've gotten a chance to replace those expensive bits before one dies and then there is a sympathy death by it's companion. Hope you're back to worrying about quilting soon.

Judy P. said...

I would like to say that I don't know what you are talking about, but I was a computer programmer and was on call 7 days a week. I would get a call at 2am and the operator would start talking like I was sitting there waiting for the phone to ring.

Glen QuiltSwissy said...

Even though I am retired, and have sold the company and the rent houses, I still get those calls now about the Animal Rescue Shelter. The rain is coming in the roof and we are all going to die! The cats have no more litter because we forgot to order before the delivery time and we are all going to die! I totallly understand those phone calls!




cocoya said...

A work phone call before 6am is never good news! And everything else can go down, but NOT PAYROLL!!! Everyone's got bills to pay. LOL
ทางเข้า D2BET
Gclub