When/If you post a comment (and I hope you will!), be sure to put in a URL --- even if you just use mine (http://alan.batie.org/) --- if you don't, your email address will get put in as a link on your name, and you'll suddenly find yourself on Spam Central. Not my doing, I promise you! They scan websites for embedded email addresses. You've been warned --- I really don't wish spam on anyone (well, ok, maybe the spammers themselves)!
Finally, if you want to be notified when I post the this blog, signup on the lemmings mailing list.
I'm not superstitious. Really, I'm not. But this week has been one to try your faith in the rational.
Saturday, I had a great time visiting the M&LK Railroad with Marc. And I've been paying for it ever since. More than paid: I think the Karma Police owe me!
It started Sunday: upgrading the mail server at sawnet (converting from qmail to postfix, because spamcop decided that qmail was "bad" and blacklisting sawnet as a result). It was a disaster. OK, I don't know postfix as well as I thought I did. I'd planned on doing this upgrade on another system where I could have gotten it working and tested first, but with spamcop blacklisting us, we had to get something in place ASAP.
Oh, and my Windoze box died miserably Sunday too, but at least I'd already move to the Mac for my main desktop...
I did get it mostly working Monday morning, though it was horribly backlogged, and then at Peak, we had a DOS attack on a customer. Simple, have our network guy put in a block on our router to stop the bad traffic. He's doing that, pasting in the new rules, when right in the middle of the paste, the system he's doing it from loses a hard disk and crashes. It's mirrored, but apparently it's crap for raid hardware (like the rest of that system). The result is our main router left in a semi-functional state. I get the management system back up, while others get the router working again.
Then we find that it took out our dialup access servers. It had done the last time it crashed, a month and a half ago, and we never did figure out why, but this time, rebooting the switch they're connected to didn't fix it as it had before. Finally figured out that a subtle ordering of config statements was causing the problem for some weird reason. So, we get that back up.
Then, on Tuesday, we get complaints of network slowness, and eventually track that down to a switch that connects us to one of our major WAN circuits had a hardware failure.
I spend all night Tuesday clearing the backlog at sawnet and tuning the system, fixing some remaining problems.
Wednesday, things are starting to resume some normalcy as we start to rearrange things to mitigate future problems when the same things happen...
And then Thursday, one of the mail servers reboots and it takes a while to find out that the software on it is old and buggy and it doesn't come back up automatically. I reconfigured it so it should now (though it still needs tested).
Oh, and I discovered today that it looks like the user shell server has been hacked.
In other words, The Week From Hell.
Posted by abatie at March 11, 2005 07:33 PM | TrackBack