We recently suffered a strange issue with one of our NAS devices - a QNAP 8-bay NAS.
The device itself has two RAID-5 volumes of four disks each.
After a power outage that went beyond the time of UPS' ability to keep things running, the unit shut down.
To our dismay, when it powered back up, only one of the two RAID volumes was available. The second was marked "Unmounted".
So, we open a terminal and SSH into the NAS in order to run a few diagnostic commands and see what's going on.
In this situation, a knowledge of *nix will go a long way to helping you inderstand what the particular problems you are facing entail. The QNAP NAS boxes run a custom linux implementation based on Ubuntu (linux kernel 2.6).
mdadm - this command is used to manage and monitor software RAID devices in linux.
[~] # mdadm --detail /dev/md0 /dev/md0: Version : 01.00.03 Creation Time : Sun May 21 16:12:53 2017 Raid Level : raid5 Array Size : 11716348608 (11173.58 GiB 11997.54 GB) Used Dev Size : 3905449536 (3724.53 GiB 3999.18 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Fri May 26 18:55:48 2017 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Name : 0 UUID : xxxxxxxx:xxxxxxxx:xxxxxxxx:513ee963 Events : 21 Number Major Minor RaidDevice State 0 8 3 0 active sync /dev/sda3 1 8 19 1 active sync /dev/sdb3 2 8 35 2 active sync /dev/sdc3 3 8 51 3 active sync /dev/sdd3
[~] # mdadm --detail /dev/md1 /dev/md1: Version : 01.00.03 Creation Time : Sun May 21 16:15:40 2017 Raid Level : raid5 Array Size : 11716348608 (11173.58 GiB 11997.54 GB) Used Dev Size : 3905449536 (3724.53 GiB 3999.18 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 1 Persistence : Superblock is persistent Update Time : Fri May 26 18:55:48 2017 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Name : 1 UUID : xxxxxxxx:xxxxxxxx:xxxxxxxx:514de4de Events : 21 Number Major Minor RaidDevice State 0 8 67 0 active sync /dev/sde3 1 8 83 1 active sync /dev/sdf3 2 8 99 2 active sync /dev/sdg3 3 8 115 3 active sync /dev/sdh3
Both volumes seem to still exist, and the RAID disk info is intact...
df -h (a command to check disk space/usage), we can see the
/dev/md0 volume is ok, but the `/dev/md1' volume is gone:
[~] # df -h Filesystem Size Used Available Use% Mounted on /dev/ramdisk 151.1M 134.6M 16.5M 89% / tmpfs 64.0M 372.0k 63.6M 1% /tmp tmpfs 494.4M 4.0k 494.4M 0% /dev/shm tmpfs 16.0M 0 16.0M 0% /share /dev/sda4 371.0M 248.8M 122.2M 67% /mnt/ext /dev/md9 509.5M 121.5M 388.0M 24% /mnt/HDA_ROOT df: /share/external/UHCI Host Controller: Protocol error tmpfs 1.0M 0 1.0M 0% /mnt/rf/nd /dev/md0 10.9T 7.1T 3.8T 65% /share/MD0_DATA
Trying to manually mount failed, but it turns out there's a simple solution... Run a manual file system check and repair first:
[~] # e2fsck_64 -fp -C 0 /dev/md1
Depending on the size of your volume this could potentially take quite some time - about 4 hours in the case of our 12TB RAID-5 volume. So, be patient... some parts of the process don't update you on the status.
Now, mount the device:
[~] # mount -t ext4 /dev/md1 /share/MD1_DATA
Of course, this may not resolve your specific issues, but if this fails you may have bigger problems, like a corrupt swap partition or something that may take some more work to reconstruct.
After researching some other peoples solutions, there are some other steps you may be able to take that could help with your raid volumes...
First, check if the RAID is online:
Then run a file system check:
e2fsck_64 -f /dev/mapper/cachedev1
This repairs the inconsistent file system. Do not cancel this.
Then check again:
e2fsck_64 -fp -C 0 /dev/mapper/cachedev1
Then you can try and re-mount the file system.
mount -t ext4 /dev/mapper/cachedev1 /share/CACHEDEV1_DATA
After this you may need to re-start your NAS - the file system should work again.
For further reading, the QNAP forums have a wealth of information shared by some very knowledgable users.
Check out the following: