We weren't supposed to have survived this downtime

Posted by Aistarin on July 25, 2017, 4:15 p.m.

And yet here we are.

Sequence of events pieced together from what Jeremy has told me as he was fixing things:

- The main server ran KVM
- 64digits ran as a self-contained virtual machine
- 64digits was also stored on a logical volume…
- …managed by Debian 6's LVM for some reason
- Our VM was on 8.7 Jessie
- This combo is apparently unstable under high I/O
- never noticed because we are all but dead
- Automated MySQL backups sometimes push us to the limits
- First freak out two weeks ago, system goes in read only mode
- dmesg shows nothing
- reboot VM and carry on
- second freak out
- reboot VM again
- error: unknown file system.
- dmesg returns I/O errors out the wazoo
- shit shit shit shit shit
- fsck does its thing
- inodes for important directories were wiped out and could not be restored, extent of file system damage is unknown
- what we know as 64digits is now nothing more than just a 320gb blob of zeroes and ones on the hard drive
- a grep of the blob shows the MySQL database to be more or less intact
- maybe all hope is not lost yet
- as I start researching ways to file carve SQL databases back into existence, Jeremy rolls up his sleeves and starts guessing where the inodes are and manually restores them
- he manages to guess correctly for /var
- he finds a SQL dump dated Sunday morning
- Glory to 64Digitzka!
- the database is more important than your files, but he keeps going
- user files restored!
- not enough room on drive to spin up a second VM
- not enough room on drive to create archives
- can't delete backups or blob of zeroes and ones until we are certain the restored files can be used to rebuild the website
- buying a new server is cheaper than uploading/downloading the tens of millions of individual files to/from cloud storage
- rsync everything to our new home
- 64digits is now running off of the bare metal this time

So yeah…don't forget to thank Jeremy for his hard work to save this site!


Cpsgames 2 years, 8 months ago

You're in the know, right?

samuraikill 2 years, 8 months ago

Glad we're still here :)

Josea 2 years, 8 months ago

Hell yes

aeron 2 years, 8 months ago

If we bought Jeremy a beer for every time he's saved 64digits he'd die of alcohol poisoning. Thanks again to all who help keep us up and running :)

Fabio 2 years, 8 months ago

Jani_Nykanen 2 years, 8 months ago

We don't deserve so quick a death, that's it.

F1ak3r 2 years, 8 months ago

Well done all round!

twisterghost 2 years, 8 months ago

Thats fuckin amazing

Moikle 2 years, 8 months ago

Glory to 64Digitzka!

Cheers germy Jerms

Alert Games 2 years, 8 months ago

sounds awesome

Copyright 2019 64Digits.