In a series of mishaps and mistakes, best characterized as the sysadmin equivalent of a perfect storm, we lost more than 3 weeks of our database.
It all started when we decided to move posts to a whole new system of forums. In that process we screwed up. But we had made a backup just before, so we decided to go back to that, dropping the screwed-up database. Then it turned out that the most recent backup we made was truncated bacause of a memory limit, so it was unuseable.
This dropped us back to an older backup, which in this case was Over three weeks old!!!
This means that we lost:
- All forum postings since October 4th
- All new member registrations since October 4th
- Records of donations since October 4th (this is recoverable...)
- All News items (recoverable too)
If you have any important postings still on disk, in browser cache, or anywhere else, we'd like you to repost them.
Re-working the forum structure will still hapen, sometime later today, and without any further dramatic losses.
I apologize for this event, and we'll work even harder to make sure something like this will never happen again.