We finally relaunched BlackPlanet.com, and it was... well, it was an adventure. The site crashed and burned under load at first, and we spent several days tuning and tweaking until it was stable. It took over a week before all of the functionality was restored. (Some of the searches were just killing the DBs.) Whoops?
Our network is a mess - somehow, in the past two years, all of the effort we put into cleaning it up and making it work well just... went away. When we first "rebuilt" the network, we took the time to make sure it had everything it needed - for instance, trunking ports between the two core switches. I remember setting up at least 5 gig ports, so we'd have redundancy and plenty of bandwidth. One by one, they disappeared. I'm not sure where they went, but people yanked the cables out and unconfigured them. Wha...? Yeah, really. We fixed it.
Today I discovered another great side-effect - it turns out that our edge switches weren't configured correctly (or are just buggy, I don't honestly know right now which it is) so traffic is passing between the core switches not by going over the trunk, which makes sense, but by passing through the edge switches and back into the core, completely defeating the purpose of the setup and killing our performance on top of that. We fixed that, too (or at least mostly - more stuff to clean up tomorrow).
Every day it seems like there's another problem to find and clean up. It's good to find them, and there's some real satisfaction in solving problems, but I can't believe what a mess the place has gotten to be.
One of the positive things I've seen recently is the advent of the new LV (low voltage) Xeons - with dual cores on each cpu, they're as fast or faster (at least for our needs) than the old hyper-threaded Xeon chips, and they use less than half of the power. What does that mean? It means you can pack 'em in more densely, and you save on getting additional circuits at the data center. Sweet.
Anyways, recently, I've been working around 12 hours a day, 5 days a week, and often a few hours on both days of the weekend. I've been doing a lot of technical work, recently, which is a bit of a blast from the past - I've been heads down in contract negotiations and budgets and planning for the past year and have had very little time to focus on the tech stuff. Recently, though, I haven't had a choice - and in some ways I'm glad. I've been getting awfully sick of doing the business work, and it's nice to roll up my sleeves and go back to being the systems architect and admin for a while. I've been working on networks and machines and writing scripts and having more fun doing it all than I have for a while. It almost - but not quite - makes the ridiculous hours worthwhile.
Oh well. Off to bed. Now that my AC is finally working, I'm hoping to get my first good night of sleep in weeks. I've been walking around in a fog for way too long. Sleep, here I come!