Monitor and repair software RAID

Our Linux installs use mdadm software RAID and, by default, mirror two identical disks (RAID 1). This guide covers health checks, alerts, disk replacement and layouts for storage servers. You need root access. The examples use NVMe device names such as nvme0n1; SATA and SAS disks appear as sda, sdb and so on.

The default layout

Both disks get the same partition table, and each pair of partitions forms a RAID 1 array, for example md0 for /boot and md1 for /. The exact set depends on the operating system and boot mode. To see yours:

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT

Check array health

cat /proc/mdstat
sudo mdadm --detail /dev/md1

md1 : active raid1 nvme1n1p3[1] nvme0n1p3[0]
      998848512 blocks super 1.2 [2/2] [UU]

[UU] means both members are active; [U_] means the array is degraded and running on one disk. In the mdadm --detail output, check the State line and look for members marked faulty or removed.

Set up alerts

A degraded mirror keeps working, so without alerts you may only notice when the second disk fails. Tell mdadm where to report in /etc/mdadm/mdadm.conf (Debian, Ubuntu) or /etc/mdadm.conf (AlmaLinux, Rocky Linux):

MAILADDR root
PROGRAM /usr/local/sbin/raid-alert

Alerts to root need a working mail transfer agent, such as Postfix delivering locally or relaying through your mail provider. The optional PROGRAM line runs a script of yours with the event and device names as arguments, for example to post to a chat webhook. Restart the monitor and send a test alert for every array:

sudo systemctl restart mdmonitor
sudo mdadm --monitor --scan --oneshot --test

Replace a failed disk

1. Identify the failed disk and note the serial numbers of both disks:

   cat /proc/mdstat
   lsblk -o NAME,SIZE,MODEL,SERIAL
   sudo smartctl -i /dev/nvme1n1

2. Mark the failed disk’s partitions as faulty and remove them from every array. Skip this step if the disk has vanished and mdadm already lists it as removed.

   sudo mdadm /dev/md0 --fail /dev/nvme1n1p2 --remove /dev/nvme1n1p2
   sudo mdadm /dev/md1 --fail /dev/nvme1n1p3 --remove /dev/nvme1n1p3

3. Open a support ticket with the serial number of the disk to replace, the serial number of the disk to keep, and whether we may shut the server down for the swap. Replacement starts within one hour of the fault being confirmed; see the SLA.
4. After the swap, find the new, empty disk with lsblk. Copy the partition table of the healthy disk onto it, then give the copy unique GUIDs:

   sudo sgdisk --replicate=/dev/nvme1n1 /dev/nvme0n1
   sudo sgdisk --randomize-guids /dev/nvme1n1

5. Add the new partitions to the arrays and watch the rebuild:

   sudo mdadm /dev/md0 --add /dev/nvme1n1p2
   sudo mdadm /dev/md1 --add /dev/nvme1n1p3
   watch -n 10 cat /proc/mdstat

6. Make the new disk bootable. On BIOS systems, run sudo grub-install /dev/nvme1n1, or grub2-install on AlmaLinux and Rocky Linux. On UEFI systems where the EFI partition isn’t part of an array, copy it from the healthy disk with sudo dd if=/dev/nvme0n1p1 of=/dev/nvme1n1p1 bs=4M.

In sgdisk --replicate=TARGET SOURCE, the disk after the equals sign is overwritten. Swap the two and you erase the partition table of your only healthy disk. Take the same care with if= (source) and of= (target) in dd.

By default, a rebuild runs at up to about 200 MB/s, the limit in /proc/sys/dev/raid/speed_limit_max. That is roughly 1.4 hours per terabyte, and more than a day for a 22 TB disk. On an NVMe server, you can raise the limit with sudo sysctl -w dev.raid.speed_limit_max=2000000.

Layouts for storage servers

Storage servers combine 4, 12 or 24 hard disks of 22 TB with two NVMe drives. Keep the operating system on the NVMe mirror and build the data array from the hard disks. Because a 22 TB rebuild takes more than a day, use double parity (RAID 6 or ZFS RAIDZ2) or mirrors, not RAID 5.

Figures are before filesystem overhead, and tools such as df report TiB. RAID 10 gives the best random I/O, while RAID 6 and RAIDZ2 survive any two failures and, from 12 disks up, give more space. More, narrower groups rebuild faster and perform better, at some cost in capacity. Keep ZFS pools below about 80% full.

With ZFS, spare NVMe partitions can also speed up the pool; plan them when you install the operating system:

• A cache device (L2ARC) extends the read cache beyond RAM. It needs no redundancy: if it fails, the pool carries on.
• A special vdev keeps metadata, and optionally small files, on flash, which speeds up directory scans and small-file workloads. Losing it loses the whole pool. With two NVMe drives it can only be a two-way mirror, which survives one failure while RAIDZ2 survives two, so zpool refuses the mismatch unless you add -f. Use it only if you accept that trade-off.

Check disk wear and SMART data

sudo smartctl -a /dev/nvme0
sudo smartctl -H -A /dev/sda

For NVMe drives, watch Percentage Used, the share of rated write endurance consumed (100% doesn’t mean the drive has failed), along with Available Spare, Media and Data Integrity Errors and Critical Warning, which should be 0x00. On hard disks, rising Reallocated_Sector_Ct, Current_Pending_Sector or Offline_Uncorrectable values mean the disk should be replaced before it fails. The smartd daemon from the smartmontools package checks these values continuously and can email you; on AlmaLinux and Rocky Linux, enable it with sudo systemctl enable --now smartd.