I have a load-bearing raspberry pi on my network - it runs a DNS server, zigbee2mqtt, unifi controller, and a restic rest server. This raspberry pi, as is tradition, boots from a microSD card. As we all know, microSD cards suck a little bit and die pretty often; I’ve personally had this happen not all that long ago.
I’d like to keep a reasonably up-to-date hot spare ready, so when it does give up the ghost I can just swap them out and move on with my life. I can think of a few ways to accomplish this, but I’m not really sure what’s the best:
- The simplest is probably cron + dd, but I’m worried about filesystem corruption from imaging a running system and could this also wear out the spare card?
- recreate partition structure, create an fstab with new UUIDs, rsync everything else. Backups are incremental and we won’t get filesystem corruption, but we still aren’t taking a point-in-time backup which means data files could be inconsistent with each other. (honestly unlikely with the services I’m running.)
- Migrate to BTRFS or ZFS, send/receive snapshots. This would be annoying to set up because I’d need to switch the rpi’s filesystem, but once done I think this might be the best option? We get incremental updates, point-in-time backups, and even rollback on the original card if I want it.
I’m thinking out loud a little bit here, but do y’all have any thoughts? I think I’m leaning towards ZFS or BTRFS.
Why are you using a MicroSD at all? Get a used minipc with two sata ssds or a single board computer with eMMC or mpcie support
Get a used minipc
Why, when the Pi is working fine? Just get a USB to SATA cable. This one works great with the Pi: https://a.co/d/8Jv2Erj
Yeah, I definitely won’t be buying a new pi anytime soon for exactly that reason but I’ve had this one for awhile and would prefer to do something useful with it.
BTW in my anecdata I’ve yet to have a failure on any of my SanDisk Extreme Pro SD cards. I have 4 in active use on different Pi 4s. The oldest one is in use since 2018-ish. It used to be a NAS till 2022 so roughly 3-4 years of 24/7 use with a bog-standard OS. It’s now running OpenWrt which does fewer writes.
Why not just buy a good SD card? My dashcam has been recording ~16/5 for the past 3 years onto a Sandisk extreme dashcam SD card and it’s still going strong with no issues. If it can survive the extreme heat and cold of being in a car I’m sure it will survive in a Pi just fine.
All of my SD cards that have failed have been bargain brand cards. None of my high quality ones have failed on me, I lose them before they go bad.
I have a decent Samsung card in there now, it might survive. Can’t remember what brand the one that failed was, but I don’t tend to buy crappy ones
Which cards are you using? Just because it’s samsung doesn’t mean it’s good.
My dashcam uses mostly this “SanDisk 256GB High Endurance Video” SD card, and my backup is a 512gb Samsung Pro Plus (not rated for dashcam use). For anything that I want reliability I use one of these SanDisk cards, that Samsung, or a SanDisk extreme that I bought the other day. My “I don’t really care about this, but I’d like it to not fail” cards are Samsung Evo Select drives (or something green and Samsung). Only my “I really don’t give a shit about these” drives are those $3 Microcenter cards.
Why not just connect an ssd via USB and save yourself the hassle and torment?
That wouldn’t solve the problem though would it? It might make it less likely to fail but there’s still significant downtime if there’s no hot spare for this USB drive.
I couldn’t count the number of failed sd cards I’ve seen across all my fingers and toes.
I’ve seen like 4 ssds in my entire life fail. Plus you could just do mdraid 1 / btrfs across 2 of them if you want
Any failures of SanDisk Extreme Pro / Samsung Evo Plus?
I buy the redundancy argument. I’d still use ZFS for that if possible though. 😂 All my machines use mdraid 1 for their system drives but now that I know enough about ZFS, I’d likely use it on root next time around.
I’ve had very bad luck with raspberry Pi’s and SDCards. They just don’t seem to last very long. I swapped to usb storage and things got somewhat better. I just had a usb drive die after 3 to 4 years of use. When I was still using SD it seemed like multiple times a year. Heat. Power loss, you can only punch holes in silicon so many times before it wears out. Whatever the reason.
My approach for this is configuration backup not the entire os. I think this approach is better for when it’s time to upgrade the os or migrate to a new system.
For my basic Pi running WireGuard and DNS, I keep an archive of documentation on steps to reconfigure the system after a total loss. Static configs are backed up once, and If there are critical configuration items that change then I back those up weekly. I’ve got two systems (media related servers, not Pi’s) that I keep ansible playbooks to configure 90% of the system from scratch so it’s as hands off as it can be.
Yeah, I’m getting a pretty strong consensus here that an SSD is the way to go. I’ve also had at least one SD card die on me, and because I didn’t have backups it was pretty inconvenient. Had to recreate my homeassistant setup from scratch.
I get the config only backup, but when I have a mondohuge nas available and we’re dealing with like less than 100 gigs, why not just take a full disk image?
Ditch the SD card all together and get a cheap SSD to use as the boot drive instead.
