File system corruption

Normally when I power up my headless Ubuntu server, I wait 1 minute for it to boot and I'm in business. I have access to my files, SVN repository and backups. But not last week. I waited for 15 minutes... nothing happened.

Using SSH to connect to the box was not working. So I hooked up a monitor to the machine and found out it gave me errors that I figured out had to do with ext3 file system corruption. Ohhhh hell.

But because this machine is used as backup server and development environment, luckily, this issue is not the end of the world for me, because I still have 95% of the data that's on that server on other machines. It's just a time consuming job to get everything back and centralized again.

So the terminal greeted me with the "Enter root password or press Ctrl-D" prompt. I was clueless. I read a million different forum threads on using the fsck tool with a mix of command line options. I tried one fsck command that looked safe to execute, tried another, but after it ran the system was still in the same state.

Hrmmms... the next thing I tried was pull out the disc from the Ubuntu server and mount it into a Windows box. I installed the Ext2 FS driver, rebooted and *big smile*... the drive showed up and I was able to recover the data that was unique to the crashed system.

After getting the disc back in the machine and doing a full format of the drivers in the server everything looked fine. I reinstalled the machine with Debian 5.0.4 instead of Ubuntu and only got Samba going again for now, because that was most urgent. After that, I spent a few evenings backing up my data again.

So I now feel a little less secure with that backup server. But only because I still have no idea what's the best way to recover from an ext3 file system corruption. I know I succeeded using a Windows environment to recover parts of it, but still, I should be able to do it using that "Enter root password or press Ctrl-D" prompt.

Note: Trying to be prepare better if it ever happens again, I today Googled some more about the subject and I stumbled on a free Windows tool called R-Linux , that some other guy used to recover his data from an ext3 partition. But still no new on a pure Linux based solution.