Ok, where do I start? Well, on Monday, the site was working on a very sub-standard level. It was down for at least an hour when I first got home between class and Solid State, and came back up just before I left, but with no databases working. I figured they just needed more time to come back and hoped they would be back by the time I came home. Boy was I wrong, everything was gone. Site lasted a few more hours, databases were still down, and eventually late Monday night the whole thing went kaput. Whole thing means literally whole thing, site, databases, even my e-mail account. I knew this had to be something big, as when I went to our host's site, that was down as well. Since it was getting late, I went to bed ready to kick some ass on Tuesday.
Nothing changed by the time I got up, and their e-mail wasn't working either, so I couldn't do anything. Not long after my first work shift ended, an e-mail made it's way to me and all other customers from the host, explaining the problem. Apparentally they had slowly been shifting sites on their new systems, and that a hard drive had a fatal error on Monday night, forcing them to do a clean restart. They said that it seemed like they would be able to recover all data from just before the crash, noting that it wouldn't be until the next day when everything would be normal again, so I figured it would just be a waiting game until all was normal again. Nothing can be just that damn simple for me, can it?
Throughout the day yesterday, nothing. Then, at about 10:45PM EST, the site was up! I go around to see the damage, and see right when I scrolled down that the last update wasn't from last week, it was from June 18th. I fell into a slight state of insanity, laughing uncontrollably for almost 20 minutes I've been told. When I regained the ability to breathe again (and yes, I'm serious, this really happened, gave my housemate a good laugh seeing me like that), I went to find out what more damage there was and made this little list:
If you're reading this, you see that I've been able to restore the site itself to current day. Everything else will be worked on throughout free time I have today and the next few days, as not everything can be done from my end. Forums are down until further notice. There's nothing I can do about the databases right now, and I apologize for not being a good admin and not backing up those databases at least once a week. Chances are that we will have to start from scratch there, as much as I want to stay positive, realistically, with them not up in any state as of yet, I don't expect them to show back up. Hopefully I'm wrong, I'd love to be proved wrong in this way, and in all sense, SHOULD be wrong. I'll admit my own sloppiness with not keeping backups of the databases myself, but what the hell has out host been doing? Last backup in late June and complete loss of databases? Geez.
However, I can assure you that steps will be taken to avoid this happening in the future. I am already taking steps in undergoing a project to build my own server and run the site right here in my house. I hope this project turns out to be a successful one and that this kind of stuff will end. As for right now, be assured that I'm doing my best to restore everything here back to the state we were in before Monday, and that I will keep you all updated right here, so please don't e-mail me. Well, you can't e-mail me, it'll just get bounced back, but don't e-mail Bob or Tiff about this, trust me they're not going to know much more that what's typed in here except on how to get a hold of me.
October 2nd, 2003
Dan - I love crashes almost as much as crappy customer service
Affilates
