Aistarin

Last Login:
August 22, 2017
Warn:

Rank:
Moderator




User Profile
Follow

Hits: 312,402
Joined February 07, 2007
Games (5)
Examples (4)
Favorite Users


We weren't supposed to have survived this downtime
Posted on July 25, 2017 at 17:15

And yet here we are.

Sequence of events pieced together from what Jeremy has told me as he was fixing things:

- The main server ran KVM
- 64digits ran as a self-contained virtual machine
- 64digits was also stored on a logical volume...
- ...managed by Debian 6's LVM for some reason
- Our VM was on 8.7 Jessie
- This combo is apparently unstable under high I/O
- never noticed because we are all but dead
- Automated MySQL backups sometimes push us to the limits
- First freak out two weeks ago, system goes in read only mode
- dmesg shows nothing
- reboot VM and carry on
- second freak out
- reboot VM again
- error: unknown file system.
- dmesg returns I/O errors out the wazoo
- shit shit shit shit shit
- fsck does its thing
- inodes for important directories were wiped out and could not be restored, extent of file system damage is unknown
- what we know as 64digits is now nothing more than just a 320gb blob of zeroes and ones on the hard drive
- SHIT SHIT SHIT SHIT SHIT
- a grep of the blob shows the MySQL database to be more or less intact
- maybe all hope is not lost yet
- as I start researching ways to file carve SQL databases back into existence, Jeremy rolls up his sleeves and starts guessing where the inodes are and manually restores them
- he manages to guess correctly for /var
- he finds a SQL dump dated Sunday morning
- Glory to 64Digitzka!
- the database is more important than your files, but he keeps going
- user files restored!
- not enough room on drive to spin up a second VM
- not enough room on drive to create archives
- can't delete backups or blob of zeroes and ones until we are certain the restored files can be used to rebuild the website
- buying a new server is cheaper than uploading/downloading the tens of millions of individual files to/from cloud storage
- rsync everything to our new home
- 64digits is now running off of the bare metal this time

So yeah...don't forget to thank Jeremy for his hard work to save this site!


"Holy, holy, holy are the 64digits;
The whole earth is full of our glory."

<3
Posted by hel July 25, 2017 17:50 - 29 days ago
| [#01]

You're in the know, right?
Posted by Cpsgames July 25, 2017 17:58 - 29 days ago
| [#02]

Glad we're still here :)
Posted by samuraikill July 25, 2017 19:03 - 29 days ago
| [#03]

Hell yes
Posted by Josea July 25, 2017 20:36 - 28 days ago
| [#04]

If we bought Jeremy a beer for every time he's saved 64digits he'd die of alcohol poisoning. Thanks again to all who help keep us up and running :)
Posted by aeron July 25, 2017 22:46 - 28 days ago
| [#05]

Posted by Fabio July 25, 2017 23:19 - 28 days ago
| [#06]

We don't deserve so quick a death, that's it.
Posted by Jani_Nykanen July 26, 2017 3:13 - 28 days ago
| [#07]

Well done all round!

Posted by F1ak3r July 26, 2017 3:35 - 28 days ago
| [#08]

Thats fuckin amazing
Posted by twisterghost July 26, 2017 14:02 - 28 days ago
| [#09]

Glory to 64Digitzka!

Cheers germy Jerms
Posted by Moikle July 26, 2017 18:07 - 28 days ago
| [#10]

sounds awesome
Posted by Alert Games July 30, 2017 11:17 - 24 days ago
| [#11]

For the record, this website will go down in our terms, nobody else's.
Posted by Aistarin July 31, 2017 3:13 - 23 days ago
| [#12]

rm -rf /
Posted by Jeremy July 31, 2017 3:15 - 23 days ago
| [#13]

Posted by Aistarin July 31, 2017 3:26 - 23 days ago
| [#14]

This place as far as I can tell must have been dead years ago but yet here I am typing 64digits.com and shit still shows up.

Kudos to you guys that make this possible. Don't think I don't appreciate it.
Posted by NeutralReiddHotel August 02, 2017 21:31 - 20 days ago
| [#15]

All hail Jeremy! Talk about getting your hands dirty with low-level, manually restoring inodes is one hell of a thing!
Posted by DesertFox August 07, 2017 12:14 - 16 days ago
| [#16]

I guess I missed all the excitement D=
Posted by KaBob799 August 21, 2017 18:33 - 2 days ago
| [#17]

Recent Activity
 
Active Users (0)