Just before I visited Maryland in June, one of our network switches (which connected 24 users to the network) started working only intermittently. I placed an order for a new one and scheduled the installation. (Do you want to wait until after you get back from vacation, they asked me. HELL NO! I replied.) Alas, hardware works until it doesn't and it died for good two days after I left. It turned out to drag down other things with it, with the result that it was a holy hell week at the company. Bless them, they never called me and I never knew until I got back how bad it had been. Of course, there was nothing I could have done about it anyway...
I had been distracted by the virtual servers running out of memory. The servers (we have three physical servers) are set up so that if one dies, another can take over the tasks. Unfortunately if the servers are running at more than 50+ % capacity, it means another server CAN'T take over because you can't max out the memory without crashing. So I had to make sure to replace the RAM on all three servers. RAM memory is cheap, but server RAM is not. If one server failed, and another couldn't fill the gap, then the company's computer network would be "dead in the water." So I was focused on that.
About the same time we were looking at the age of all our equipment and realized our Firewall (the barrier of protection between our network and The Bad Guys) was beyond End Of Life. So we were working on getting quotes and ordering that. None of this is easy. It's not like buying a computer and just setting it up. There were hours of configuration our techs had to do to set it up before they brought it over for installation yesterday.
IN THEORY... it was a straightforward swap. Install the new firewall into the rack, disconnect the old one, connect the new one and test. Internet activity worked great. External email, not so much. After much tinkering it looked OK at 7 PM and I went home.
About an hour later, a text from the CEO arrived. "Can't get mail on my phone." Sigh. So I tested and sure enough, that wasn't working. I called "the guys" and they worked on it. It took them till 11 PM to figure it out. It's fine now.
Yet another thing to keep me away from the sewing studio, darn it.
I have a favorite quote about being a Geek:
"In Theory, theory and practice are the same. In Practice, they are not."
Sure do hope there are no other surprises lurking...
13 comments:
They are lucky to have you there ... seriously!!
Great job. I had a job once where I had to get up anytime night or day to start a new run on a Cray if anything interrupted the code. It could have completed a run, had a reset because of the Cray software, had a reset because of the operators checks and about 10 more reasons. I was the only person allowed to go into the operations room to reset this code so I feel your pain. There were times that I would restart, drive 20 minutes to my house and get the call to come back - before cell phones. Computers are a great bonus, but things go wrong and in the case I was working computing capability wasted was just too expensive. Sounds like that is what you are juggling with the server backup issue. Good luck as always so many things can cause issues and most of the world thinks you just flip a switch and everything is fixed. At least your boss notified you last night, because he probably saved you a horrible day with half the world calling to tell you they couldn't access their mail.
Hopefully you have this all taken care of for many years and you won't have such a convergence of issues all at the same time again.
Hubby does what you do on a larger scale. He says "oh yeah".
You've found some kind patsy .. er .. volunteer to hold your work cell while you're in CA, right?
Wow, I had no idea this was your " Day Job" .Each one of us diversifies in different ways, and I am in total awe of you and the responsibility. Hope it all runs smoothly from now on.
A nice reminder from you. I seldom took a vacation...I was one of those statistics they seem to be writing about now. I would always have more work when I came back and it would take me a month to catch up. Now that I'm retired I look back and think how crazy I was. The reminder from you is that I am so happy I am retired, lol.
xx, Carol
I'm retired now but a long time ago (still in this galaxy) I did something similar although not as complicated just because it wasn't that complicated then ... but I still feel your pain.
The responsibility you accept on behalf of the company sounds heavy indeed, Lynne.
Megan
Sydney, Australia
I totally understand all this....being married to a network designer at a data center for a major airplane manufacturer I've heard it all -- multiple all your woes by about 100 and then add layers and layers of management and various groups who "own" specific pieces of the puzzle.....
Ahhhhh, the magic word...lurking.
When you least expect it ...
Computers are fun when they work.
Nancy and the kitties
And they say that computers make our lives easier. If only "they" knew, right? So sorry to hear about this. I hope things will work like they ought to from now on. Hopefully some sewing will soothe the frazzled nerves.
OMG whatever they pay you is not enough... I cannot even imagine the stress!
Post a Comment