Recovering RAID array from a Thecus NAS

I had to recover a RAID-5 array from a client’s Thecus NAS which had stopping reading the disks or even starting up properly.

Fortunately this NAS, a Thecus N4100PRO, is basically a Linux server and uses the standard software RAID and LVM formats.

To start with I pulled the hard drive out of my desktop, plugged in the four drives from the NAS and booted up off of a Parted Magic CD.

After confirming that the hard drives were indeed healthy, I started with the following command:

  mdadm --examine --scan --verbose /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2

which gave the following output:

  ARRAY /dev/md1 level=raid5 num-devices=4 UUID=0bb2ce09:500bcf2f:945dc254:7108bfd3
  devices=/dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2

Copy this into /etc/mdadm.conf and save.

Now we try to mount it…

[email protected]:~# mdadm -Av /dev/md1
mdadm: looking for devices for /dev/md1
mdadm: /dev/sr0 is not one of /dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2
mdadm: /dev/sde1 is not one of /dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2
mdadm: /dev/sde is not one of /dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2
mdadm: /dev/sdd1 is not one of /dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2
mdadm: /dev/sdd is not one of /dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2
mdadm: /dev/sdc1 is not one of /dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2
mdadm: /dev/sdc is not one of /dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2
mdadm: /dev/sdb1 is not one of /dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2
mdadm: /dev/sdb is not one of /dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2
mdadm: /dev/sda1 is not one of /dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2
mdadm: /dev/sda is not one of /dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2
mdadm: /dev/loop1 is not one of /dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2
mdadm: /dev/loop0 is not one of /dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2
mdadm: /dev/sdd2 is identified as a member of /dev/md1, slot 3.
mdadm: /dev/sdc2 is identified as a member of /dev/md1, slot 2.
mdadm: /dev/sdb2 is identified as a member of /dev/md1, slot 1.
mdadm: /dev/sda2 is identified as a member of /dev/md1, slot 0.
mdadm: added /dev/sdb2 to /dev/md1 as 1
mdadm: added /dev/sdc2 to /dev/md1 as 2
mdadm: added /dev/sdd2 to /dev/md1 as 3
mdadm: added /dev/sda2 to /dev/md1 as 0
mdadm: /dev/md1 has been started with 4 drives.

Great! We have our RAID array mounted.

[email protected]:/mnt# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md1 : active raid5 sda2[0] sdd2[3] sdc2[2] sdb2[1]
2924400000 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>

Now we need to mount the file system on the RAID.

  [email protected]:/media/md1# mount /dev/md1 /media/md1
  mount: unknown filesystem type 'LVM2_member'

xxxxx

Using the LVM tools, we need to find the LV (Logical Volume) and VG (Volume Group) names. First we do a disk scan:

root@PartedMagic:/media/md1# lvmdiskscan
 /dev/ram0    16.00 MiB
 /dev/loop0   35.21 MiB
 /dev/ram1    16.00 MiB
 /dev/loop1   137.79 MiB
 /dev/md1     2.72 TiB LVM physical volume /dev/ram2    16.00 MiB
 /dev/ram3    16.00 MiB
 /dev/ram4    16.00 MiB
 /dev/ram5    16.00 MiB
 /dev/ram6    16.00 MiB
 /dev/ram7    16.00 MiB
 /dev/ram8    16.00 MiB
 /dev/ram9    16.00 MiB
 /dev/ram10   16.00 MiB
 /dev/ram11   16.00 MiB
 /dev/ram12   16.00 MiB
 /dev/ram13   16.00 MiB
 /dev/ram14   16.00 MiB
 /dev/ram15   16.00 MiB
 /dev/sde1    232.88 GiB
 0 disks
 19 partitions
 0 LVM physical volume whole disks
 1 LVM physical volume

Now we run lvdisplay:

  [email protected]:/media/md1# lvdisplay
  --- Logical volume ---
  LV Name                /dev/vg0/syslv
  VG Name                vg0
  LV UUID                rGlvdG-JYaD-rjDd-nrOH-zRw1-DIxz-AGKiy8
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                1.00 GiB
  Current LE             512
  Segments               1
  Allocation             inherit
  Read ahead sectors     16384

  --- Logical volume ---
  LV Name                /dev/vg0/lv0
  VG Name                vg0
  LV UUID                MGZbUD-m7vl-g7Dt-JRi5-XPEy-U2AN-f7WoJ5
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                1.29 TiB
  Current LE             676352
  Segments               1
  Allocation             inherit
  Read ahead sectors     16384

And now a vgdisplay just to make sure it’s the correct drive:

  [email protected]:/media/md1# vgdisplay
  --- Volume group ---
  VG Name               vg0
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               2.72 TiB
  PE Size               2.00 MiB
  Total PE              1427929
  Alloc PE / Size       676864 / 1.29 TiB
  Free  PE / Size       751065 / 1.43 TiB
  VG UUID               QfPXRZ-nT9Z-o4rU-OMUS-bvuq-PFbF-H5XMQu

So, now I tried to mount using the file system using the volume group name”

[email protected]:/media/md1# mount /dev/vg0/lv0 /mnt/thecus
mount: special device /dev/vg0/lv0 does not exist

Running an lvscan shows that the volumes are actually still inactive.

The following commands should fix that:

[email protected]:/media/md1# modprobe dm-mod
[email protected]:/media/md1# vgchange -ay
2 logical volume(s) in volume group "vg0" now active

Another run of lvscan shows that the LV’s are now active. Now, I should be able to mount the file system:

[email protected]:/media/md1# mount /dev/vg0/lv0 /mnt/thecus
[email protected]:/media/md1# cd /mnt/thecus/
[email protected]:/mnt/thecus# ls -al
total 92
drwxr-xr-x 17 root root  4096 Oct 13  2011 ./
drwxr-xr-x  5 root root    60 May 14  2012 ../
drwxrwxrwx  2   99  102  4096 Oct 13  2011 _Module_Folder_/
drwxrwxrwx  2   99  102  4096 Oct 13  2011 _NAS_Module_Source_/
d--x--x--x  2 root root  4096 Oct 13  2011 _NAS_NFS_Exports_/
drwxrwxrwx  3   99  102  4096 Oct 27  2011 _NAS_Picture_/
drwxr-xr-x  4 root root  4096 Aug 27  2009 dlnamedia/
drwxr-xr-x  2 root root  4096 Mar  8 11:34 ftproot/
drwx------  2 root root 16384 Aug 27  2009 lost+found/
drwxr-xr-x  6 root root  4096 Oct 13  2011 module/
drwxrwxrwx  2   99  102  4096 Apr  8  2011 naswebsite/
drwx------  5   99  102  4096 Oct 13  2011 nsync/
drwxrwxrwx  5   99  102  4096 Oct  9  2011 share/
drwxr-xr-x  2 root root  4096 Aug 29  2009 target_usb/
drwxr-xr-x  3 root root  4096 Oct 13  2011 tmp/
drwxrwxrwx  2   99  102  4096 Aug 27  2009 usbcopy/
drwxrwxrwx  3   99  102  4096 Aug 29  2009 usbhdd/

Success! We can see his files and copy them off to another hard drive. There’ll be another happy client tomorrow.

—-

Sources:

  • http://serverfault.com/questions/32709/how-do-i-move-a-linux-software-raid-to-a-new-machine
  • http://superuser.com/questions/117824/how-to-get-an-inactive-raid-device-working-again
  • http://pissedoffadmins.com/?p=481