2

We have a problem with a Degraded array on software raid. After a recent yum update (centOS7 linux 7, 1708) on our linux cluster a message was sent to root indicating that a DegradedArray event had been detected on md device /dev/md/swap|root|boot. After running some commands and asking some colleagues it was concluded that sdb was no longer associated with sdc, perhaps as a result of the upgrade. The partition in sdb is not the same as that in sdc. By running the command mdadm – E ... is seems that the association would have to be: root md125 sdc3 sdb5; swap md126 sdc1 sdb2; and boot md127 sdc2 sdb3. We attempted to swap the hard disks but that did not help. We are aware of the response given to a similar question in (https://www.linuxquestions.org/questions/linux-hardware-18/degraded-array-on-software-raid-429857/). Would that response be appropriate even if the sdb and sdc partitions are different? Below is the output of: MESSAGE_1 - Message to root MESSAGE_2 – output of mdadm --detail /dev/md125 ; mdadm --detail /dev/md126 ; mdadm --detail /dev/md127 MESSAGE_3 – output of fdisk -l and lsscsi.

Thanks in advance for your time and effort.

MESSAGE_1

From root@jtainer_lab_01.mdanderson.edu Tue Oct 17 17:17:07 2017
Return-Path: <root@jtainer_lab_01.mdanderson.edu>
X-Original-To: root
Delivered-To: root@jtainer_lab_01.mdanderson.edu
Received: by jtainer_lab_01.mdanderson.edu (Postfix, from userid 0)
id 7979A5272BF; 2017年10月17日 17:17:07 -0500 (CDT)
From: mdadm monitoring <root@jtainer_lab_01.mdanderson.edu>
To: root@jtainer_lab_01.mdanderson.edu
Subject: DegradedArray event on /dev/md/swap:jtainer_lab_01.mdanderson.edu
Message-Id: <20171017221707.7979A5272BF@jtainer_lab_01.mdanderson.edu>
Date: 2017年10月17日 17:17:07 -0500 (CDT)
This is an automatically generated mail message from mdadm
running on jtainer_lab_01.mdanderson.edu
A DegradedArray event had been detected on md device /dev/md/swap.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid1]
md125 : active raid1 sdc3[1]
142701440 blocks super 1.2 [2/1] [_U]
bitmap: 2/2 pages [8KB], 65536KB chunk
md126 : active raid1 sdc1[1]
32767872 blocks super 1.2 [2/1] [_U]
md127 : active raid1 sdc2[1]
204736 blocks super 1.0 [2/1] [_U]
unused devices: <none>
From root@jtainer_lab_01.mdanderson.edu Tue Oct 17 17:17:07 2017
Return-Path: <root@jtainer_lab_01.mdanderson.edu>
X-Original-To: root
Delivered-To: root@jtainer_lab_01.mdanderson.edu
Received: by jtainer_lab_01.mdanderson.edu (Postfix, from userid 0)
id 8E07B52BD73; 2017年10月17日 17:17:07 -0500 (CDT)
From: mdadm monitoring <root@jtainer_lab_01.mdanderson.edu>
To: root@jtainer_lab_01.mdanderson.edu
Subject: DegradedArray event on /dev/md/boot:jtainer_lab_01.mdanderson.edu
Message-Id: <20171017221707.8E07B52BD73@jtainer_lab_01.mdanderson.edu>
Date: 2017年10月17日 17:17:07 -0500 (CDT)
This is an automatically generated mail message from mdadm
running on jtainer_lab_01.mdanderson.edu
A DegradedArray event had been detected on md device /dev/md/boot.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid1]
md125 : active raid1 sdc3[1]
142701440 blocks super 1.2 [2/1] [_U]
bitmap: 2/2 pages [8KB], 65536KB chunk
md126 : active raid1 sdc1[1]
32767872 blocks super 1.2 [2/1] [_U]
md127 : active raid1 sdc2[1]
204736 blocks super 1.0 [2/1] [_U]
unused devices: <none>
From root@jtainer_lab_01.mdanderson.edu Tue Oct 17 17:17:08 2017
Return-Path: <root@jtainer_lab_01.mdanderson.edu>
X-Original-To: root
Delivered-To: root@jtainer_lab_01.mdanderson.edu
Received: by jtainer_lab_01.mdanderson.edu (Postfix, from userid 0)
id 830EB52BD72; 2017年10月17日 17:17:07 -0500 (CDT)
From: mdadm monitoring <root@jtainer_lab_01.mdanderson.edu>
To: root@jtainer_lab_01.mdanderson.edu
Subject: DegradedArray event on /dev/md/root:jtainer_lab_01.mdanderson.edu
Message-Id: <20171017221707.830EB52BD72@jtainer_lab_01.mdanderson.edu>
Date: 2017年10月17日 17:17:07 -0500 (CDT)
This is an automatically generated mail message from mdadm
running on jtainer_lab_01.mdanderson.edu
A DegradedArray event had been detected on md device /dev/md/root.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid1]
md125 : active raid1 sdc3[1]
142701440 blocks super 1.2 [2/1] [_U]
bitmap: 2/2 pages [8KB], 65536KB chunk
md126 : active raid1 sdc1[1]
32767872 blocks super 1.2 [2/1] [_U]
md127 : active raid1 sdc2[1]
204736 blocks super 1.0 [2/1] [_U]
unused devices: <none>

MESSAGE_2 ("mdadm --detail /dev/md125"; "mdadm --detail /dev/md126"; "mdadm --detail /dev/md127")

/dev/md125:
Version : 1.2
Creation Time : Tue Jan 10 06:49:38 2017
Raid Level : raid1
Array Size : 142701440 (136.09 GiB 146.13 GB)
Used Dev Size : 142701440 (136.09 GiB 146.13 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Oct 18 12:44:11 2017
State : clean, degraded 
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Consistency Policy : unknown
 Name : localhost:root
 UUID : efa7c05e:02986ab9:5e588700:f0c62cb2
 Events : 4631277
 Number Major Minor RaidDevice State
 - 0 0 0 removed
 1 8 35 1 active sync /dev/sdc3
/dev/md126:
Version : 1.2
Creation Time : Tue Jan 10 06:49:33 2017
Raid Level : raid1
Array Size : 32767872 (31.25 GiB 33.55 GB)
Used Dev Size : 32767872 (31.25 GiB 33.55 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Update Time : Sun Oct 15 01:01:03 2017
State : clean, degraded 
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Consistency Policy : unknown
 Name : localhost:swap
 UUID : 3774f886:17152e89:79d96cdb:ce258389
 Events : 99
 Number Major Minor RaidDevice State
 - 0 0 0 removed
 1 8 33 1 active sync /dev/sdc1
/dev/md127:
Version : 1.0
Creation Time : Tue Jan 10 06:50:40 2017
Raid Level : raid1
Array Size : 204736 (199.94 MiB 209.65 MB)
Used Dev Size : 204736 (199.94 MiB 209.65 MB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Update Time : Wed Oct 18 12:29:03 2017
State : clean, degraded 
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Consistency Policy : unknown
Name : localhost:boot
UUID : ae97ea74:2a96c6db:c2dd6443:be74b4e8
Events : 732
Number Major Minor RaidDevice State
 - 0 0 0 removed
 1 8 34 1 active sync /dev/sdc2

MESSAGE_3 (output of "fdisk -l" and "lsscsi")

root@jtainer_lab_01:~$ fdisk -l
WARNING: fdisk GPT support is currently new, and therefore in an experimental
phase. Use at your own discretion.
Disk /dev/sda: 28001.6 GB, 28001576157184 bytes, 54690578432 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: gpt
Disk identifier: 59269F14-5508-4597-97D4-B5E68F5B4826
# Start End Size Type Name
1 2048 54690578398 25.5T Linux filesyste 
Disk /dev/sdb: 180.0 GB, 180045766656 bytes, 351651888 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000ac099
Device Boot Start End Blocks Id System
/dev/sdb1 2048 6143 2048 83 Linux
/dev/sdb2 6144 65574911 32784384 fd Linux raid autodetect
/dev/sdb3 * 65574912 65984511 204800 fd Linux raid autodetect
/dev/sdb4 65984512 351651887 142833688 5 Extended
/dev/sdb5 65986560 351651839 142832640 fd Linux raid autodetect
Disk /dev/sdc: 180.0 GB, 180045766656 bytes, 351651888 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0002bda2
Device Boot Start End Blocks Id System
/dev/sdc1 2048 65570815 32784384 fd Linux raid autodetect
/dev/sdc2 * 65570816 65980415 204800 fd Linux raid autodetect
/dev/sdc3 65980416 351647743 142833664 fd Linux raid autodetect
Disk /dev/md127: 209 MB, 209649664 bytes, 409472 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/md126: 33.6 GB, 33554300928 bytes, 65535744 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/md125: 146.1 GB, 146126274560 bytes, 285402880 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@jtainer_lab_01:~$ lsscsi
[0:2:0:0] disk LSI MR9271-8i 3.46 /dev/sda 
[5:0:0:0] disk ATA INTEL SSDSC2BF18 LWDi /dev/sdb 
[6:0:0:0] disk ATA INTEL SSDSC2BF18 LWDi /dev/sdc
Hauke Laging
94k21 gold badges132 silver badges185 bronze badges
asked Nov 3, 2017 at 15:02

1 Answer 1

1

All you have to do is add the partitions on sdb to the respective array, e.g.

mdadm --re-add /dev/md125 /dev/sdb5

After that you should see the sync progress in the output of cat /proc/mdstat. It makes sense to delay the restore / sync of an array until the sync of the previous one has fin ished.

answered Nov 4, 2017 at 11:01
2
  • Could you please explain to me why mdadm --re-add /dev/md125 /dev/sdc3 and not mdadm --re-add /dev/md125 /dev/sdb5 (the missing part)? Thanks. Commented Nov 6, 2017 at 15:39
  • @abacolla Sorry, that was a copy/paste error. At least it would not have caused any damage. Commented Nov 6, 2017 at 19:56

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.