Recover data from RAID1 with one disk in rebuilding state

Question

I am attempting to recover data on a MDADM RAID1 array with one HDD that suffered a mechanical failure and another that is stuck in a rebuilding state.

I have two 2TB HDDs that were installed on a machine running under a mdadm RAID1 array named /dev/md1.

Sda1 and Sdb1 were both part of the array. Sda had a mechanical failure and it's been replaced with a new HDD with the same capacity, on which a new partition of the same size was created.

Upon attempting to add sda1 to the array I've received the following error:

sudo mdadm /dev/md1 --manage --add /dev/sda1
mdadm: cannot load array metadata from /dev/md1

What I've tried

I've been following this guide (https://ahelpme.com/linux/recovering-md-array-and-mdadm-cannot-get-array-info-for-dev-md0/) to attempt to 'activate' the array.

These are the steps of the guide:

1. Remove ALL current configuration by issuing multiple stop commands with mdadm, no inactive raids or any raids should be reported in "/proc/mdstat".
2. Rename mdadm configuration files in /etc/mdadm/mdadm.conf
3. Rescan for MD devices with mdadm. The mdadm will load the configuration from your disks.
4. Add the missing partitions to your software raid devices.

I've executed the first two steps.

On running step 3, the raid was discovered but, unlike the guide, it seems sdb1 is in rebuilding state and the array cannot be started because of it.

I'm reluctant to try anything involving --force since I'm unsure as to the exact state of the data and the fact that, unfortunately, the entirety of the data is very precious.

My questions

• How can I recover the data?
• How can I know what files on the disk that is 'Rebuilding' are 'incomplete' or corrupted?
• Why, when running sudo mdadm --misc --detail /dev/md1 it says raid0 but when running sudo mdadm -E /dev/sdb1 it says raid1?

Any and all help is highly appreciated.

System info

:/$ sudo mdadm --assemble --scan --verbose
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/loop10
mdadm: no recogniseable superblock on /dev/loop9
mdadm: no recogniseable superblock on /dev/loop8
mdadm: no recogniseable superblock on /dev/sdc2
mdadm: Cannot assemble mbr metadata on /dev/sdc1
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: no recogniseable superblock on /dev/sda1
mdadm: Cannot assemble mbr metadata on /dev/sda
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb
mdadm: No super block found on /dev/loop7 (Expected magic a92b4efc, got 118a6b61)
mdadm: no RAID superblock on /dev/loop7
mdadm: No super block found on /dev/loop6 (Expected magic a92b4efc, got e7e108a6)
mdadm: no RAID superblock on /dev/loop6
mdadm: No super block found on /dev/loop5 (Expected magic a92b4efc, got 3a23b8f9)
mdadm: no RAID superblock on /dev/loop5
mdadm: No super block found on /dev/loop4 (Expected magic a92b4efc, got 3a23b8f9)
mdadm: no RAID superblock on /dev/loop4
mdadm: No super block found on /dev/loop3 (Expected magic a92b4efc, got e7e108a6)
mdadm: no RAID superblock on /dev/loop3
mdadm: No super block found on /dev/loop2 (Expected magic a92b4efc, got a6eff301)
mdadm: no RAID superblock on /dev/loop2
mdadm: No super block found on /dev/loop1 (Expected magic a92b4efc, got e06997af)
mdadm: no RAID superblock on /dev/loop1
mdadm: /dev/sdb1 is identified as a member of /dev/md/1, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md/1
mdadm: added /dev/sdb1 to /dev/md/1 as 1 <---------
mdadm: /dev/md/1 assembled from 0 drives and 1 rebuilding - not enough to start the array. <--------- 
mdadm: looking for devices for further assembly
mdadm: No arrays found in config file or automatically

:/$ lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
NAME SIZE FSTYPE TYPE MOUNTPOINT
loop1 43.1M squashfs loop /snap/certbot/1670
loop2 76.2M squashfs loop /snap/lxd/22340
loop3 55.5M squashfs loop /snap/core18/2253
loop4 61.9M squashfs loop /snap/core20/1270
loop5 61.9M squashfs loop /snap/core20/1328
loop6 55.5M squashfs loop /snap/core18/2284
loop7 43.3M squashfs loop /snap/snapd/14295
loop8 43.4M squashfs loop /snap/snapd/14549
loop9 76.3M squashfs loop /snap/lxd/22358
loop10 43.1M squashfs loop /snap/certbot/1788
sda 1.8T disk
`-sda1 1.8T ext4 part
sdb 1.8T disk
`-sdb1 1.8T linux_raid_member part
sdc 465.8G disk
|-sdc1 517.7M vfat part /boot/efi
`-sdc2 400G ext4 part /

:/$ sudo mdadm --misc --detail /dev/md1
/dev/md1:
 Version : 1.2
 Raid Level : raid0 <---------
 Total Devices : 1
 Persistence : Superblock is persistent
 State : inactive
 Working Devices : 1
 Name : xxxx:1 (local to host xxxx)
 UUID : e6193d43:0b7be2b3:fd27645d:4cb0f619
 Events : 14150
 Number Major Minor RaidDevice
 - 8 17 - /dev/sdb1

:/$ sudo mdadm -E /dev/sdb1
/dev/sdb1:
 Magic : a92b4efc
 Version : 1.2
 Feature Map : 0x83
 Array UUID : e6193d43:0b7be2b3:fd27645d:4cb0f619
 Name : xxxx:1 (local to host xxxx)
 Creation Time : Tue May 11 14:46:10 2021
 Raid Level : raid1 <---------
 Raid Devices : 2
 Avail Dev Size : 3905983761 (1862.52 GiB 1999.86 GB)
 Array Size : 1952991880 (1862.52 GiB 1999.86 GB)
 Data Offset : 264192 sectors
 Super Offset : 8 sectors
Recovery Offset : 12630656 sectors
 Unused Space : before=264112 sectors, after=0 sectors
 State : clean
 Device UUID : 58da91c5:df6f5be4:91860671:53d40e22
Internal Bitmap : 8 sectors from superblock
 Update Time : Mon Aug 16 23:01:12 2021
 Bad Block Log : 512 entries available at offset 16 sectors
 Checksum : b1e8f237 - correct
 Events : 14150
 Device Role : Active device 1
 Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : inactive sdb1[3]
 1952991880 blocks super 1.2
unused devices: <none>

Answer 1

I just got the same error message:

# cat /proc/mdstat 
md0 : active raid1 nvme1n1p1[1]
 1046528 blocks super 1.2 [2/1] [_U]
# mdadm /dev/md0 --add /dev/nvme0n1p1
mdadm: cannot load array metadata from /dev/md0
# dmesg
[1732198.131943] attempt to access beyond end of device
 nvme1n1p1: rw=0, want=2097151, limit=2095104
# mdadm -E /dev/nvme1n1p1 
/dev/nvme1n1p1:
 Unused Space : before=4016 sectors, after=-2047 sectors DEVICE TOO SMALL
 State : clean TRUNCATED DEVICE

The problem lies in the currently active partition. After I extended it to the correct size with fdisk, everything worked as expected.

• Community: As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.