A dead microSD pushed this Pi server rebuild

A failed microSD card turned a long-running Raspberry Pi 4B home server into a recovery project — and a chance to redesign it properly. The trouble started when the server stopped responding: torrent lists would not load, an SMB mount failed, and SSH was unreachable. On reboot, both the current and previous NixOS generations hung after loading the kernel.

A filesystem check initially seemed promising, but fsck.ext4 kept finding new errors every time the card was reinserted. The conclusion was simple: the card was likely dead, with a recent building-wide power outage delivering the final blow.

The loss was limited. Most important data lived on external hard drives, while the system configuration was largely preserved in NixOS. What disappeared from the card were .torrent files and caches for Navidrome, Jellyfin, and slskd, all recoverable. Immich data was already backed up on an external drive.

How the new setup cuts microSD wear

The rebuild focused on three goals: reduce writes to the microSD, add redundancy to external storage, and back up anything important.

To avoid wearing out the new card, the setup now uses:

Recommended reading

PsiQuantum bets on light for a 100-cabinet quantum computer

• zram for swap instead of disk-based swap
• tmpfs for /tmp
• volatile journald logging in memory
• noatime on the root filesystem to avoid write-on-access behavior

The root filesystem still lives on the microSD, but far less data is written to it. The replacement card is a 64GB model marketed as high endurance, replacing the previous 128GB card.

A btrfs pool built from old drives

For bulk storage, the server now uses two old 2.5-inch HDDs — one 500GB drive and one 320GB drive — combined into a btrfs pool with raid1 for both data and metadata. The pool is named ポンコツ, roughly “piece of junk.”

The author chose btrfs for its flexible storage pools and subvolumes. Each service gets its own subvolume, managed declaratively through a custom autosubvol module that creates mount units and ensures subvolumes exist before dependent services start. A separate subvolume was also created for /var/cache.

Permissions were cleaned up too. A shared media group gives services including Transmission and Jellyfin access to the same files, with Transmission configured so new downloads inherit the correct group ownership.

Backups rely on restic, integrated through a small Nix module bridging sops-nix and the restic nixpkgs module. For now, remote storage via S3 and replication stands in for a more elaborate local snapshot system.

The storage pool has already grown. It now includes a 1TB disk alongside the 500GB and 320GB drives, still using raid1 for data and metadata, with plans to add another old 500GB disk later. A separate 10TB ext4 drive full of Linux ISOs remains outside the pool.

The final piece was deployment. Instead of installing NixOS manually on the Pi, the author builds a bootable sdImage directly from the system configuration and writes it to the card with dd. After wiring the rebuilt setup into a newly purchased UPS, the server was back online in time to stream music from Navidrome for the vacation drive.