How bad RAM ruined my week

“Backup failed.”
Chilling words to see in your email, but no worries. Maybe it was just the NAS timing out or something. I can restart the job. 25% into the backup job, it fails. Input/Output error, and the VM hangs on a locked state. No problem, I can just unlock it manually from CLI.

Log into the proxmox host, run the qm command, VM unlocks. 15 minutes later, VM locks up entirely with no indication. I tried restarting it, and BAM, VM can’t restart anymore.

For an entire week, I was chasing ghosts in the system thinking that perhaps it was a bad CPU I bought online, especially since dmesg in the VMs was showing kernel panics and dumps….only to have that theory blown apart by not seeing anything in the logs on the host.

Not a problem, maybe one of the NVMEs or SSDs is failing. I can just restore from a backup. All restores fail almost immediately, complaining about a bad chunk or checksum failing. That’s when I realized something was extremely off.

At first, I thought it was my ZFS pools failing, scrubs, and statuses were reporting permanent errors on a few VM disks. However, on a pool that was mirrored, they were corrupting at the same time. So it was either both of my drives failing, somehow, at the same time, or something else. Checking S.M.A.R.T. revealed nothing, and the wearout and health seemed green.

I read online that ZFS relied on RAM extensively, so after a lot of complaining (and prodding from ChatGPT to help me debug what the hell was going on), I booted up into a memtest environment. Running a test returned over one thousand failures in less than a minute. Whelp, found my problem.

Let this be a caution tale to those who come after to check your equipment methodically, and to test your backups often. While it seemed that my data was not corrupted (replacing the bad stick cleared ZFS pool errors almost immediately after a scrub), I think I got off easy. But the odds of me backing up corrupted data was not zero.