We weren't supposed to have survived this downtime
Posted on July 25, 2017 at 17:15
And yet here we are.
Sequence of events pieced together from what Jeremy has told me as he was fixing things:
- The main server ran KVM
- 64digits ran as a self-contained virtual machine
- 64digits was also stored on a logical volume...
- ...managed by Debian 6's LVM for some reason
- Our VM was on 8.7 Jessie
- This combo is apparently unstable under high I/O
- never noticed because we are all but dead
- Automated MySQL backups sometimes push us to the limits
- First freak out two weeks ago, system goes in read only mode
- dmesg shows nothing
- reboot VM and carry on
- second freak out
- reboot VM again
- error: unknown file system.
- dmesg returns I/O errors out the wazoo
- shit shit shit shit shit
- fsck does its thing
- inodes for important directories were wiped out and could not be restored, extent of file system damage is unknown
- what we know as 64digits is now nothing more than just a 320gb blob of zeroes and ones on the hard drive
- SHIT SHIT SHIT SHIT SHIT
- a grep of the blob shows the MySQL database to be more or less intact
- maybe all hope is not lost yet
- as I start researching ways to file carve SQL databases back into existence, Jeremy rolls up his sleeves and starts guessing where the inodes are and manually restores them
- he manages to guess correctly for /var
- he finds a SQL dump dated Sunday morning
- Glory to 64Digitzka!
- the database is more important than your files, but he keeps going
- user files restored!
- not enough room on drive to spin up a second VM
- not enough room on drive to create archives
- can't delete backups or blob of zeroes and ones until we are certain the restored files can be used to rebuild the website
- buying a new server is cheaper than uploading/downloading the tens of millions of individual files to/from cloud storage
- rsync everything to our new home
- 64digits is now running off of the bare metal this time
So yeah...don't forget to thank Jeremy for his hard work to save this site!