The reboot that ate a mirror: recovering two SN200 drives from "Post Crash" mode without wiping them
I typed reboot on my Proxmox host. Both halves of a ZFS mirror came back as bricks. The vendor's fix wipes the drive. This is how the fault works, and how we cleared it with every byte intact.
The hardware
| Server | Dell PowerEdge R640 (VxRail E560F chassis), Proxmox VE |
| Drives | 2 x WD/HGST Ultrastar SN200 3.2 TB U.2 NVMe (HUSMR7632BDP301), firmware KNGND110 |
| Pool | ZFS mirror, about 1 TB of VM disks on it |
| Note | The SN200 is not on Dell's qualified drive list for this server. Dell ships no firmware for it. |
What happened
After a routine warm reboot, both SN200s came up in a firmware diagnostic state the logs call "Post Crash startup". No namespace, no data access, and the Linux NVMe driver stuck in a reset loop because the drive kept reporting a persistent internal error. Both mirror members were hit at the same time, so the pool was simply gone.
The machine had run for over a year without this. The reason turned out to be that it had never been warm-rebooted in that time. Every earlier restart was a power-off or a power loss.
Root cause
Three things have to line up:
- 01A warm reboot. Linux sends each NVMe drive an orderly shutdown, but the drive keeps its power through the restart.
- 02During POST, the iDRAC inventories the drives over the SMBus sideband using NVMe-MI, and sends a tunnelled Admin Identify (MI message type 2, opcode 6).
- 03An SN200 in the shut-down state never answers. The iDRAC aborts and retries.
- 04On the third unanswered attempt, the firmware asserts and writes a crash dump to its internal SPI flash. Three separate crash dumps show the same pattern, with the last attempt 0.86 s, 0.54 s and 0.21 s before the crash.
- 05On every later power-on the firmware finds that dump and refuses normal operation. It stays that way until the dump is cleared.
A cold power cycle never triggers it, because the drive comes up fresh and answers the iDRAC normally. BIOS and kernel versions were ruled out: the BIOS update in the logs landed three hours after the first crash.
Why the official fix is not a fix
WD's tooling has a "clear crash dump" command. We pulled the KNGND110 firmware apart to see what it actually does, and confirmed each step on the sacrificial drive:
- At boot the firmware enters Post Crash unless SPI section 19 (crash dump) and section 20 (power-fail dump) both begin with the state word
0x7F7F7F7F, meaning "erased". Only the firmware's own erase writes that word. - The clear (admin opcode
0xFF,cdw12=0x0503) erases section 19 in about 5 seconds and writes the marker. Then, because the drive is in Post Crash mode, it also records a "drive re-init" order in an internal SPI journal. The next firmware start carries it out and wipes the drive. - No host command skips that step. Interrupting the erase does not help, since a half-erased section still counts as a crash. The firmware image is RSA-signed, so patching it is out.
The trick: make the wipe order impossible to write
The SPI journal turned out to be the weak point, in our favour:
- It has exactly 480 record slots.
- If the firmware tries to write a journal record when no slot is free, it halts before writing anything.
- In Post Crash mode, every orderly NVMe shutdown appends exactly one record, and nothing is ever freed.
So we fill the journal with harmless shutdown/enable cycles until there are zero free slots, and then send the clear. The crash dump is erased and marked. The firmware then tries to record the re-init order, finds no room, and halts. No wipe order exists. After a power cycle the drive finds no crash dump, finds its last valid "clean shutdown" record, and starts normally.
Procedure
- Detach without a shutdown. Drop the drive's PCIe link on its root port (Link Disable, then enable) instead of letting the host driver shut it down, and pass it through to a rescue VM on kernel 5.10. That kernel does not reset-loop on the drive's error event, where current kernels do.
- Read-only capture with
nvme wdc cap-diag. Require: crash dump present, power-fail section erased, newest journal record "CLEAN shutdown / valid". - Fill. Unbind and rebind the driver inside the VM, about 2.8 s per cycle. Take a capture after every batch and check that N cycles used exactly N slots. Do the last 12 slots one at a time.
- Guard. The final capture must show 0 free slots with the clean record still valid. Only then send the clear.
- Get out of the way. 6.3 s after sending, drop the slot's PCIe link and hard-stop the VM.
- Physically reseat the drive and re-enable the link.
Results
We proved it on one drive (bay 7) and sacrificed its copy of the mirror to do so. We reproduced a real iDRAC-induced crash on it, wrote a 10 GiB test pattern, and ran the full procedure.
fill model, run 1 463 cycles, exact slot accounting, wrap 479 -> 0
fill model, run 2 462 cycles, exact slot accounting
bay 7 after reseat "SYS: Normal startup", pattern verified, 0 bad chunks
Then the real thing, on the drive holding the data (bay 6):
17:02 pre-check crash dump present, clean record valid, 436 free slots
17:23 fill done 0 free, clean record valid -> clear sent, link dropped, VM stopped
17:25 reseat normal mode, 2.9 TB namespace, zfs_member detected
import -o readonly=on all datasets present, "No known data errors"
17:28 read-write import, zpool replace onto bay 7
17:53 resilver 1013G in 25 min, 0 errors, mirror ONLINE
Every byte came back.
What went wrong along the way
Testing took the host down twice. A halted SN200 that still has its PCIe link up can raise fatal PCIe errors on this platform, then a CPU machine check, then a hard reset. The first time, a halted drive was still attached to a running VM. The second time, we tried to detach it by switching off its memory decoding. After that we dropped the slot's link around every expected halt, and the problem never came back. One behaviour is still unexplained: on the first run, the drive halted on its own about 20 seconds after its journal filled.
Prevention
| Done | reboot.target and kexec.target are masked, so a warm reboot simply fails. Restarts are poweroff and power on, or an iDRAC cold cycle. |
| Rule | Never leave an SN200 shut down while it still has power: no unbinding the driver, and no passing it to a VM that is then stopped. |
| Untested | A patched nvme driver that skips the shutdown on restart, reboot=pci, or turning off the iDRAC's "Collect System Inventory on Restart". The iDRAC also polls drives in the background, so that last one may not be enough. |
| Real fix | Replace the drives with ones Dell has qualified for the server. Nothing else removes the firmware flaw. |
One case we did not test: whether an unclean reset, such as a kernel panic or watchdog, can trigger the same fault on healthy drives.
If this happens to you
- Do not run the vendor crash-dump clear on a drive with data you need. It schedules a wipe.
- Do not keep power-cycling or hammering the drive. Each orderly shutdown in Post Crash mode uses up a journal slot, which is fine, but you want to know the count before you start.
- Use an older kernel in a VM or rescue environment if a current one reset-loops on the drive.
- If the method fails, a chip-level fallback exists: read the SPI flash with a CH341A programmer and fix the section markers offline. We prepared the tooling but only tested it on a synthetic image.