Homelab incident / NVMe firmware

The reboot that ate a mirror: recovering two SN200 drives from "Post Crash" mode without wiping them

gvfx ·

I typed reboot on my Proxmox host. Both halves of a ZFS mirror came back as bricks. The vendor's fix wipes the drive. This is how the fault works, and how we cleared it with every byte intact.

The hardware

ServerDell PowerEdge R640 (VxRail E560F chassis), Proxmox VE
Drives2 x WD/HGST Ultrastar SN200 3.2 TB U.2 NVMe (HUSMR7632BDP301), firmware KNGND110
PoolZFS mirror, about 1 TB of VM disks on it
NoteThe SN200 is not on Dell's qualified drive list for this server. Dell ships no firmware for it.

What happened

After a routine warm reboot, both SN200s came up in a firmware diagnostic state the logs call "Post Crash startup". No namespace, no data access, and the Linux NVMe driver stuck in a reset loop because the drive kept reporting a persistent internal error. Both mirror members were hit at the same time, so the pool was simply gone.

The machine had run for over a year without this. The reason turned out to be that it had never been warm-rebooted in that time. Every earlier restart was a power-off or a power loss.

Root cause

Three things have to line up:

  1. 01A warm reboot. Linux sends each NVMe drive an orderly shutdown, but the drive keeps its power through the restart.
  2. 02During POST, the iDRAC inventories the drives over the SMBus sideband using NVMe-MI, and sends a tunnelled Admin Identify (MI message type 2, opcode 6).
  3. 03An SN200 in the shut-down state never answers. The iDRAC aborts and retries.
  4. 04On the third unanswered attempt, the firmware asserts and writes a crash dump to its internal SPI flash. Three separate crash dumps show the same pattern, with the last attempt 0.86 s, 0.54 s and 0.21 s before the crash.
  5. 05On every later power-on the firmware finds that dump and refuses normal operation. It stays that way until the dump is cleared.

A cold power cycle never triggers it, because the drive comes up fresh and answers the iDRAC normally. BIOS and kernel versions were ruled out: the BIOS update in the logs landed three hours after the first crash.

Why the official fix is not a fix

WD's tooling has a "clear crash dump" command. We pulled the KNGND110 firmware apart to see what it actually does, and confirmed each step on the sacrificial drive:

The trick: make the wipe order impossible to write

The SPI journal turned out to be the weak point, in our favour:

So we fill the journal with harmless shutdown/enable cycles until there are zero free slots, and then send the clear. The crash dump is erased and marked. The firmware then tries to record the re-init order, finds no room, and halts. No wipe order exists. After a power cycle the drive finds no crash dump, finds its last valid "clean shutdown" record, and starts normally.

The one way this loses data: a single free journal slot left when the clear is sent. The whole procedure is built around proving that number is zero first.

Procedure

  1. Detach without a shutdown. Drop the drive's PCIe link on its root port (Link Disable, then enable) instead of letting the host driver shut it down, and pass it through to a rescue VM on kernel 5.10. That kernel does not reset-loop on the drive's error event, where current kernels do.
  2. Read-only capture with nvme wdc cap-diag. Require: crash dump present, power-fail section erased, newest journal record "CLEAN shutdown / valid".
  3. Fill. Unbind and rebind the driver inside the VM, about 2.8 s per cycle. Take a capture after every batch and check that N cycles used exactly N slots. Do the last 12 slots one at a time.
  4. Guard. The final capture must show 0 free slots with the clean record still valid. Only then send the clear.
  5. Get out of the way. 6.3 s after sending, drop the slot's PCIe link and hard-stop the VM.
  6. Physically reseat the drive and re-enable the link.

Results

We proved it on one drive (bay 7) and sacrificed its copy of the mirror to do so. We reproduced a real iDRAC-induced crash on it, wrote a 10 GiB test pattern, and ran the full procedure.

fill model, run 1     463 cycles, exact slot accounting, wrap 479 -> 0
fill model, run 2     462 cycles, exact slot accounting
bay 7 after reseat    "SYS: Normal startup", pattern verified, 0 bad chunks

Then the real thing, on the drive holding the data (bay 6):

17:02  pre-check   crash dump present, clean record valid, 436 free slots
17:23  fill done   0 free, clean record valid -> clear sent, link dropped, VM stopped
17:25  reseat      normal mode, 2.9 TB namespace, zfs_member detected
       import -o readonly=on   all datasets present, "No known data errors"
17:28  read-write import, zpool replace onto bay 7
17:53  resilver    1013G in 25 min, 0 errors, mirror ONLINE

Every byte came back.

What went wrong along the way

Testing took the host down twice. A halted SN200 that still has its PCIe link up can raise fatal PCIe errors on this platform, then a CPU machine check, then a hard reset. The first time, a halted drive was still attached to a running VM. The second time, we tried to detach it by switching off its memory decoding. After that we dropped the slot's link around every expected halt, and the problem never came back. One behaviour is still unexplained: on the first run, the drive halted on its own about 20 seconds after its journal filled.

Prevention

Donereboot.target and kexec.target are masked, so a warm reboot simply fails. Restarts are poweroff and power on, or an iDRAC cold cycle.
RuleNever leave an SN200 shut down while it still has power: no unbinding the driver, and no passing it to a VM that is then stopped.
UntestedA patched nvme driver that skips the shutdown on restart, reboot=pci, or turning off the iDRAC's "Collect System Inventory on Restart". The iDRAC also polls drives in the background, so that last one may not be enough.
Real fixReplace the drives with ones Dell has qualified for the server. Nothing else removes the firmware flaw.

One case we did not test: whether an unclean reset, such as a kernel panic or watchdog, can trigger the same fault on healthy drives.

If this happens to you