noi - Holy F*cking sh*t I did it!!! (Restored LVM Drive)

Noi, I did it, I did it!!! Yippie!!!!!!!!!!

Thank you so much my friend, my friend, my friend!

Ohmster's LVM Disaster

Scenario:

I had a Fedora Core 3 system for a long time, long enough to be able to
install Fedora Core 5 and that is what I wanted to do. But, I had a lot
of work put into my old disto. 3 FQDNs on an apache server, vsftpd setup
just the way I want it, all of my stuff for my family in the public_html
directories, scripts, and system tweaks that I had learned for years. I
for sure did not want to lose this stuff so I decided to do a clean
install of Fedora Core 5 on a brand new 200Gb IDE hard drive and pull the
existing 200Gb hard drive out for safekeeping. Cool, I install Fedora
Core 5, all went well, but for the fact that FC5 won't config my ISA
Soundblaster AWE64 card.

I then install the old FC3 drive as /dev/hdb. Great. Time to mount that
sucker. Not happening, it appears that the drive is an LVM drive, and to
make it worse, it has the same name as the LVM drive that Fedora Core 5
has created for the system. What to do? I fiddle around, ask questions in
Usenet and out of desperation, decide to experiment with webmin on it. I
decide to rename the drive from VolGroup00 to VolGroup01. It did not
appear to be successful but oh well. At least I could not tell if it was
successful. I then, like an idiot, decide to create a new physical volume
on the drive, not at all seeing the warning, ALL DATA WILL BE LOST. I
commit to the change, see the warning, try to stop it. Oh oh...

I shut down the system, reconnect the drive as /dev/hda again and boot to
the drive as I have always been able to do. Kernel panic, no /
filesystem. Oh crap, the warning was not kidding. I cry to Usenet,
alt.os.linux again, everyone tries to help but little by little, they all
peter out, but for noi. He hangs in there with me. Giving me ideas, URLs
(Thank you so much for the URLS noi!), and mostly, inspiration, and
confidence. A week goes by and no luck. I finally stumble on the answer,
thanks to noi's URLs.

The Fix:

A good URL that got me started was this one:
http://www.linuxjournal.com/article/8874

It showed me how to grab a backup file from the toasted drive by doing
this:
dd if=/dev/md2 bs=512 count=255 skip=1 of=/tmp/md2-raw-start

Modified for my system as such:
dd if=/dev/hdb bs=512 count=255 skip=1 of=/tmp/md2-raw-start

I then follow the instructions and try to create a backup file from it as
described in the above URL and also this URL, (Big help):
http://blog.herlo.org/2006/09/15/lvm-from-failure-to-recovery/

Now my original backup file was not worth a darn because the 2nd URL
showed me that I did indeed need the original logical volume IDs as well
as the physical volume IDs. I recreated this backup by pulling up two
term windows, one with a backup of /etc/lvm/backup/VolGroup00 in a vim
editor, and the other one of my crap from the dd output file. I just
copied and pasted all the relevant numbers, IDs, etc., into the existing
backup file of VolGroup00's file.

Now since both drives did contain the same name, VolGroup00, the existing
backup was useless. I did end up with a backup of VolGroup01 but it was
after the disaster (Remember, I did rename it with webmin.) and so it was
useless.

So now I saved the very much edited backup of /etc/lvm/backup/VolGroup00
as /root/recoverybox/test/test-no-lv2. I named it that because what I was
able to salvage from the dd output file was enough to recreate the
physical volume and the first logical volume, the 2nd logical volume was
lost in binary gibberish. This might be okay because the first logical
volume was the big one and that is the one I wanted, with all my stuff in
it. Here is what it looks like:
---------------------------------------------------------------------
# Generated by LVM2: Sun Oct 1 16:33:20 2006

contents = "Text Format Volume Group"
version = 1

description = "Created *after* executing 'vgscan'"

creation_host = "ohmster.com" # Linux ohmster.com 2.6.17-1.2187_FC5 #1
Mon Sep 11 01:17:06 EDT 2006 i686
creation_time = 1159734800 # Sun Oct 1 16:33:20 2006

VolGroup01 {
id = "3agBFX-D3N0-Bp3c-I7se-JVLY-ZIIa-fqMcIF"
seqno = 2
status = ["RESIZEABLE", "READ", "WRITE"]
extent_size = 65536 # 32 Megabytes
max_lv = 0
max_pv = 0

physical_volumes {

pv0 {
id = "v40FVM-IWMU-T26b-I9eD-UprZ-WNqu-a9MTGq"
device = "/dev/hdb2" # Hint only

status = ["ALLOCATABLE"]
pe_start = 384
pe_count = 6074 # 186.188 Gigabytes
}
}

logical_volumes {

LogVol00 {
id = "aaOmGX-y3gI-zucm-xEkQ-IxnV-w7Od-Xauy6E"
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 1

segment1 {
start_extent = 0
extent_count = 6011 # 184.219
Gigabytes

type = "striped"
stripe_count = 1 # linear

stripes = [
"pv0", 0
]
}
}
}
}
---------------------------------------------------------------------
By the way, all this lvscan, pvscan, and all the mucking around with the
drive removed what was still in the first sector of the disk and all I
had now was binary giberish. Thank God that I did not delete my original
dd output file in the /tmp directory!

Okay, tried to restore it with the command:
vgcfgrestore -f /root/recoverybox/test/test-no-lv2 VolGroup01

No such luck, said it could not find the physical drive as named in the
recovery file. Bummer. I would have to create the physical drive, and use
the physical volume ID number of what the original drive used to contain,
never mind that one already existed. This is what I did:

pvcreate -u 3agBFX-D3N0-Bp3c-I7se-JVLY-ZIIa-fqMcIF /dev/hdb2

That would not work because it would overwrite an existing physical
group, I would have to force it with the -ff flag, here goes.

pvcreate -ff -u 3agBFX-D3N0-Bp3c-I7se-JVLY-ZIIa-fqMcIF /dev/hdb2

Hmmm, that seemed to go all too well, no output is good output. Does the
Volume Group exsist?

[root@ohmster recoverybox]# lvscan
ACTIVE '/dev/VolGroup01/LogVol00' [187.84 GB] inherit
ACTIVE '/dev/VolGroup00/LogVol00' [184.22 GB] inherit
ACTIVE '/dev/VolGroup00/LogVol01' [1.94 GB] inherit
[root@ohmster recoverybox]#

Indeed it does! Will it mount? Maybe, but I am sure I will have to fsck
the hell out of it, lose a lot of data in the process too, no doubt. I
try to mount it.

mount /dev/VolGroup00/LogVol01 /mnt/old_sys

"/dev/VolGroup00/LogVol01 is not a block device" WTF?!

I use the mount command to see just how these kind of drives are mounted
on my system...

[root@ohmster recoverybox]# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)

Ohh, I see, let's try again.

mount dev/mapper/VolGroup00-LogVol00 /mnt/old_sys

Huh, no output, the command worked! But what is left of all my stuff?

I go see.

[root@ohmster recoverybox]# cd /mnt/old_sys
[root@ohmster old_sys]# ls -la
total 304
drwxr-xr-x 23 root root 4096 Oct 1 13:11 .
drwxr-xr-x 3 root root 4096 Sep 30 20:57 ..
-rw------- 1 root root 154 Dec 10 2005 .bash_history
drwxr-xr-x 2 root root 4096 Jun 29 04:42 bin
drwxr-xr-x 2 root root 4096 May 7 2005 boot
drwxr-xr-x 4 root root 4096 May 7 2005 dev
-rw-r--r-- 1 root root 644 Oct 1 13:05 env.reboot
drwxr-xr-x 104 root root 12288 Oct 1 13:11 etc
-rw-r--r-- 1 root root 27599 May 7 2005 .fonts.cache-1
-rw-r--r-- 1 root root 0 Oct 1 13:11 halt
drwxr-xr-x 8 root root 4096 Dec 8 2005 home
drwxr-xr-x 2 root root 4096 Aug 12 2004 initrd
drwxr-xr-x 10 root root 4096 Jun 29 04:14 lib
drwx------ 2 root root 16384 May 7 2005 lost+found
drwxr-xr-x 6 root root 4096 Jan 29 2006 media
drwxr-xr-x 2 root root 4096 Mar 23 2005 misc
drwxr-xr-x 2 root root 4096 Oct 8 2005 mnt
drwxr-xr-x 2 root root 4096 Aug 12 2004 opt
drwxr-xr-x 2 root root 4096 May 7 2005 proc
drwxr-x--- 38 root root 4096 Sep 27 19:45 root
drwxr-xr-x 2 root root 12288 Jun 29 04:43 sbin
drwxr-xr-x 2 root root 4096 May 7 2005 selinux
drwxr-xr-x 2 root root 4096 Aug 12 2004 srv
drwxr-xr-x 2 root root 4096 May 7 2005 sys
drwxrwxrwt 12 root root 61440 Oct 1 13:07 tmp
drwxr-xr-x 18 root root 4096 Oct 19 2005 usr
drwxr-xr-x 25 root root 4096 Jun 23 19:12 var
[root@ohmster old_sys]#

OMFG, it worked!!! I go and browse my stuff, it is all there, ALL of it!

"Joy to the world, all the boys and girls. Joy to the fishes in the deep
blue see. Joy to you and me!"

I quickly copy my stuff home where it belongs with the cp - a command.
All is well again.

The Moral:

Don't give up. If you don't know what you are doing right now, leave it
alone. Keep whatever data files you have found or created. Don't discard
them. Make backups! (Oh I had better do that right now!). Go back and try
again as you learn more about this project.

Very Helpful URLS:
http://www.linuxjournal.com/article/8874
http://blog.herlo.org/2006/09/15/lvm-from-failure-to-recovery/
http://codeworks.gnomedia.com/archives/2005/general/lvm_recovery/

Thanks:
noi

--
~Ohmster
theohmster at comcast dot net
Put "messageforohmster" in message body
to pass my spam filter.
.

Relevant Pages

  • Re: unable to access sda2 on boot
    ... I'm assuming that means its fstab i'll have to mess with? ... I'm running Fedora Core 2.6.20-1.2925.fc6 ... That's drives, not ports, it doesn't count unused ports. ... The resume message means the system couldn't find the drive partition to ...
    (Fedora)
  • Driver/program that supports NEC CD changer?
    ... and have put it into my Fedora Core 2 machine, ... should be able to read multiple-disc drives, ... I mainly want to play music CD's with this drive, ... music properly from the first slot, much less do anything with the rest. ...
    (comp.os.linux.hardware)
  • Re: USB Flash drive ??
    ... the drives in your machine). ... I had read that Fedora core should auto detect and place an icon on ... but this I believe is a kudzu function - If you boot your system with the ... Stuart Sears RHCE, RHCX ...
    (RedHat)
  • FC2 post-install hangs on dual boot system
    ... Setup Agent stalls and will not finish dual boot installation after reboot. ... Trying to dual boot Windows XP Pro and Fedora Core 2 on a Dell Dimension XPS ... two SATA drives, one CD-ROM, one DVD drive, ... Disabled SATA RAID in BIOS, enabled SATA for both hard drives, ...
    (linux.redhat.install)
  • Re: SBS 2003 raid1 C: drive failed - need guidance quickly
    ... drive was in a RAID1 and when one of the drives ... backup file isn't complete, I am starting off in fairly lousy shape, ... I am assuming that I just need to do a clean install of SBS 2003. ... the domain, exchange, sharepoint - exchange is ...
    (microsoft.public.windows.server.sbs)