Guide on removing a Linux
mdadm RAID1 array while preserving existing partition data, avoiding the need to reinstall or copy files around.
If you ever end up with a similar need, here is how I did it. It goes without saying that you should back up data before doing anything, otherwise you accept the risk of losing it all. Hic sunt dracones 🐉
A dedicated SoYouStart server with two disks, for hosting virtual machines via VMware ESXi.
Unfortunately VMware ESXi does not support software RAID: as a workaround, it is possible to attach two VMDK (virtual disk) to each VM, with each VMDK stored on a separate datastore (hardware disk), and setup software RAID1 directly from the VM via
This annoying limitation actually was one of the main motivators for migrating from VMware ESXi to Proxmox VE, which supports software RAID1 over ZFS. With RAID handled at the host level, virtual machines do not need to manage software RAID themselves anymore, and only one disk needs to be kept for each VM.
We are going to remove devices from the array, then trick
mdadm into being cool with a RAID1 array consisting of only one device.
First, inspect partitions and disks to identify where is what and what needs to be done:
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 100G 0 disk ├─sda1 8:1 0 953M 0 part [SWAP] └─sda2 8:2 0 99.1G 0 part └─md0 9:0 0 99.1G 0 raid1 / sdb 8:16 0 100G 0 disk ├─sdb1 8:17 0 953M 0 part /var/tmp └─sdb2 8:18 0 99.1G 0 part └─md0 9:0 0 99.1G 0 raid1 / # cat /proc/mdstat md0 : active raid1 sda2 sdb2 103872512 blocks super 1.2 [2/2] [UU] # mdadm --detail /dev/md0 State : active
For example, here:
md0is spread over
sdadisk and keep
Once everything is settled:
# mdadm /dev/md0 --fail /dev/sda2 mdadm: set /dev/sda2 faulty in /dev/md0 # mdadm /dev/md0 --remove /dev/sda2 mdadm: hot removed /dev/sda2 from /dev/md0 # mdadm --zero-superblock /dev/sda2
In our case, we first dropped the
sda1 swap partition (not shown above: disabled with
swapoff and an
/etc/fstab update), and then removed
sda2 from the array.
If we stopped at this stage
mdadm would complain the array is degraded because it is missing a device:
# cat /proc/mdstat md0 : active raid1 sdb2 103872512 blocks super 1.2 [2/1] [_U] # mdadm --detail /dev/md0 State : clean, degraded
Fortunately, we can trick the array into being cool with one device so that it does not complain it is degraded:
# mdadm --grow /dev/md0 --force --raid-devices=1 raid_disks for /dev/md0 set to 1 # cat /proc/mdstat md0 : active raid1 sdb2 103872512 blocks super 1.2 [1/1] [U] # mdadm --detail /dev/md0 State : clean
If the removed partition / disk was also the boot partition / disk, make sure to update
/etc/fstab as necessary:
# vi /etc/fstab # grub-install /dev/sdb # update-initramfs -u
In our case:
md0is the boot partition but the bootloader was only installed on
sda(check presence of
dd if=/dev/sda bs=512 count=1 2> /dev/null | strings), so we had to reinstall
grubon the remaining
sdbdisk to ensure it can be booted from.
initramfsreferences the swap partition in its
RESUMEvariable, so we had to remove it from
initramfssince the swap partition was removed.
The machine can now be shutdown, and the unused disk detached from the hardware. Update the VM disk boot order if necessary and boot: voilà!
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 100G 0 disk ├─sda1 8:1 0 953M 0 part /var/tmp └─sda2 8:2 0 99.1G 0 part └─md0 9:0 0 99.1G 0 raid1 /
The solution above works completely fine.
However, if we’d like to go the extra mile and remove
mdadm altogether, then we also need to fiddle with the remaining device: the rough idea is to rewrite the
mdadm partition to strip out the RAID superblock while keeping the rest intact.
This sort of funky partition business is typically done from rescue mode, but in this article we’re going to do it directly from the live system for maximum thrill.
# blkid /dev/sda2: UUID="<sda2_uuid>" TYPE="linux_raid_member" /dev/md0: UUID="<md0_uuid>" TYPE="ext4"
Take note of both the
linux_raid_member RAID partition (here
sda2) and the underlying partition itself (here
linux_raid_memberpartition and take note of the version and data offset:
# mdadm --examine /dev/sda2 Version : 1.2 Data Offset : 16384 sectors
From the man page:
The different sub-versions store the superblock at different locations on the device, either at the end (for 1.0), at the start (for 1.1) or 4K from the start (for 1.2). “1” is equivalent to “1.2” (the commonly preferred 1.x format). “default” is equivalent to “1.2”.
Since we have version 1.2, everything before the data offset in the partition is the RAID superblock, everything after is the underlying partition.
fdiskon the disk:
# fdisk -u /dev/sda
fdiskprompt, print the partitions using
Command (m for help): p Device Boot Start End Sectors Size Id Type /dev/sda1 * 2048 1953791 1951744 953M 83 Linux /dev/sda2 1953792 209715199 207761408 99.1G fd Linux raid autodetect
dand its partition number.
pfor primary and reuse the same partition number.
16384 + 1953792 = 1970176using the example above).
w. This should position the new
sda2partition right in place of the underlying partition (formerly
Spoiler: if the destroyed partition was the boot partition, then the jump scare moment is right now 😱
This is our case: we hit
grub rescue because we destroyed the boot partition.
To recover from that, we are going to manually point
grub rescue to our brand new partition.
grub rescueprompt, display existing boot values with
grub rescue> set prefix=(mduuid/<sda2_uuid>)/boot/grub root=mduuid/<sda2_uuid>
As we can see, the issue is that
grub points to the partition we just destroyed (
Take note of the
prefix filepath (here
lsto display the existing partitions, named
grub rescue> ls (hd0) (hd0,msdos2) (hd0,msdos1)
lson each partition until we find the one containing the
grub rescue> ls (hd0,msdos1)/boot/grub error: file '/boot/grub' not found grub rescue> ls (hd0,msdos2)/boot/grub ./ ../ unicode.pf2 i386-pc/ locale/ fonts/ grubenv grub.cfg
grub rescue> set prefix=(hd0,msdos2)/boot/grub grub rescue> set root=(hd0,msdos2)
normalmodule and start it:
grub rescue> insmod normal grub rescue> normal
This should boot the system as usual.
sda2has properly replaced the former
# blkid /dev/sda2: UUID="<md0_uuid>" TYPE="ext4"
mdadmwas properly removed:
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 100G 0 disk ├─sda1 8:1 0 953M 0 part /var/tmp └─sda2 8:2 0 99.1G 0 part / # cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] unused devices: <none> # mdadm --detail /dev/md0 mdadm: cannot open /dev/md0: No such file or directory # mdadm --examine /dev/sda2 mdadm: No md superblock detected on /dev/sda2.
grubso as to point it to the right UUID and avoid hitting
# grub-install /dev/sda