Drive Crash
In the summer of 2011, it was finally decided that the existing OpenLDAP based server authentication system was unmanageable, and the decision was taken to migrate to a Windows domain on the fileserver, with all other servers authenticating against the Windows domain. This also required reformatting what was then the UserData drive to run on the new operating system.
Prior to this some time had been spent working out where the existing data could go so that nothing was lost, and using a few spare drives ordered for the Edit PCs a few days earlier, along with the archive, enough space had been found. Some simple tests revealed that the best way would be to install the spare drives in the existing file server and copy the files across, so the server was pulled out of place and the new drive was screwed in and powered.
When it came to connect a data lead to the drive, things went a little wrong, as one of the main storage drives was pulled instead of what was though to be a spare lead, due to the cramped conditions inside the machine. In theory the array is designed to recover from this gracefully and rebuild from redundant data on the other disks. Sadly on this occasion something went wrong, possibly due to the array having been expanded by an extra 2TB a few months earlier.
Many nerve-wracking hours, and a serious discussion between Michael Chislett, Michael Cullen, and Sam Nicholson about leaving to join URY later, the conclusion was drawn that in true television drama style, there was one last possibility to recover the data, a risky, little used command inside the filesystem repair utility fsck.
Unfortunately, unlike in fictional dramas, the gamble didn't pay off, so the three spent many minutes watching every piece of data on the drives erased, piece by piece.
Finally the rest of the society were told (after being plied with large quantities of alcohol to soften the blow), and it was time to pick up the pieces and find out what was missing. Many painful weeks of recapturing tapes and trawling laptops, forgotten USB sticks and emails later, only two major projects were missing, one of which 75% of the tapes still existed for. The other was the Opening Ceremony of that year's Roses, which was eventually discovered scattered across the tape stock in October of 2011.
As a result of this accident, YSTV now have a fairly [1] robust backup solution in place, meaning we can now tolerate the loss of any three of our main drives at the same time, without losing more than a day's work.