Just before I visited Maryland in June, one of our network switches (which connected 24 users to the network) started working only intermittently. I placed an order for a new one and scheduled the installation. (Do you want to wait until after you get back from vacation, they asked me. HELL NO! I replied.) Alas, hardware works until it doesn't and it died for good two days after I left. It turned out to drag down other things with it, with the result that it was a holy hell week at the company. Bless them, they never called me and I never knew until I got back how bad it had been. Of course, there was nothing I could have done about it anyway...
I had been distracted by the virtual servers running out of memory. The servers (we have three physical servers) are set up so that if one dies, another can take over the tasks. Unfortunately if the servers are running at more than 50+ % capacity, it means another server CAN'T take over because you can't max out the memory without crashing. So I had to make sure to replace the RAM on all three servers. RAM memory is cheap, but server RAM is not. If one server failed, and another couldn't fill the gap, then the company's computer network would be "dead in the water." So I was focused on that.
About the same time we were looking at the age of all our equipment and realized our Firewall (the barrier of protection between our network and The Bad Guys) was beyond End Of Life. So we were working on getting quotes and ordering that. None of this is easy. It's not like buying a computer and just setting it up. There were hours of configuration our techs had to do to set it up before they brought it over for installation yesterday.
IN THEORY... it was a straightforward swap. Install the new firewall into the rack, disconnect the old one, connect the new one and test. Internet activity worked great. External email, not so much. After much tinkering it looked OK at 7 PM and I went home.
About an hour later, a text from the CEO arrived. "Can't get mail on my phone." Sigh. So I tested and sure enough, that wasn't working. I called "the guys" and they worked on it. It took them till 11 PM to figure it out. It's fine now.
Yet another thing to keep me away from the sewing studio, darn it.
I have a favorite quote about being a Geek:
"In Theory, theory and practice are the same. In Practice, they are not."
Sure do hope there are no other surprises lurking...