I have a load-bearing raspberry pi on my network - it runs a DNS server, zigbee2mqtt, unifi controller, and a restic rest server. This raspberry pi, as is tradition, boots from a microSD card. As we all know, microSD cards suck a little bit and die pretty often; I’ve personally had this happen not all that long ago.
I’d like to keep a reasonably up-to-date hot spare ready, so when it does give up the ghost I can just swap them out and move on with my life. I can think of a few ways to accomplish this, but I’m not really sure what’s the best:
- The simplest is probably cron + dd, but I’m worried about filesystem corruption from imaging a running system and could this also wear out the spare card?
- recreate partition structure, create an fstab with new UUIDs, rsync everything else. Backups are incremental and we won’t get filesystem corruption, but we still aren’t taking a point-in-time backup which means data files could be inconsistent with each other. (honestly unlikely with the services I’m running.)
- Migrate to BTRFS or ZFS, send/receive snapshots. This would be annoying to set up because I’d need to switch the rpi’s filesystem, but once done I think this might be the best option? We get incremental updates, point-in-time backups, and even rollback on the original card if I want it.
I’m thinking out loud a little bit here, but do y’all have any thoughts? I think I’m leaning towards ZFS or BTRFS.
I’ve always used dd + sshfs to backup the entire sd card daily at midnight to an ssh server; retaining 2 weeks of backups.
Should the card die, I’ve just gotta write the last backup to a new card and pop it in. If that one’s not good, I’ve got 13 others I can try.
I’ve only had to use it once and that went smoothly. I’ve tested half a dozen backups though and no issues there either.
Yeah “dd if=/dev/mmcblk0 of=$HOSTNAME.$(date +%Y.%m.%d).img” and while its running. (!!! Make sure the output is NOT going to the sd card you are backing up…)
I deliberately chose a time when it’s not very active to perform the backup. Never had an issue, going on 6 years now.
Another option is to use Image File Utilities on the Pi to create an image backup. You can use cron + a bash script to create incremental backups using the tool (e.g., take a ‘fresh’ backup each month, with daily incremental backups in between). I mount a network ‘backup’ drive (a local NAS, but you could use anything) to save the image to so I can actually access it. Then, just use balena etcher to flash the backup iso in the event of a failure.
- Here’s Image File Utilities: https://forums.raspberrypi.com/viewtopic.php?t=332000
- And a backup script + cron entry: https://forums.raspberrypi.com/viewtopic.php?p=2199452&hilit=cron#p2199452
Happy to help! There’s plenty of other options too (e.g., SSD) as folks mentioned, but this works well out of the box with no additional hardware. SD has been absolutely fine for my use (Pi-Hole), while still requiring maximum uptime so the family doesn’t riot if the internet is out.
I also use those, but (shame on me) I’ve never tested the backups. Have you ever restored such an image backup?
Get a USB to SATA cable. This one works great with the Pi: https://a.co/d/8Jv2Erj
Instead of having a warm spare, a better solution is to attach two drives and use them in a RAID1 config. Unfortunately, I don’t think the Pi supports RAID1.
Unfortunately, I don’t think the Pi supports RAID1.
I haven’t ran any Pi with hard drives, but I don’t see any reason why it wouldn’t work with software raid on linux.
The issue is that I don’t think its standard bootloader supports booting from RAID. I guess you could use a MicroSD for booting then have everything else on the RAID1. There’s unofficial ways to boot using GRUB, which should work with RAID1 too.
Grub supports software raid just fine. The main issue is that you need to modify grub configuration to add bootloader on both drives, but even if you don’t it’s pretty simple to recreate needed files for second drive when the primary one dies.
Why not just buy a good SD card? My dashcam has been recording ~16/5 for the past 3 years onto a Sandisk extreme dashcam SD card and it’s still going strong with no issues. If it can survive the extreme heat and cold of being in a car I’m sure it will survive in a Pi just fine.
All of my SD cards that have failed have been bargain brand cards. None of my high quality ones have failed on me, I lose them before they go bad.
I have a decent Samsung card in there now, it might survive. Can’t remember what brand the one that failed was, but I don’t tend to buy crappy ones
Which cards are you using? Just because it’s samsung doesn’t mean it’s good.
My dashcam uses mostly this “SanDisk 256GB High Endurance Video” SD card, and my backup is a 512gb Samsung Pro Plus (not rated for dashcam use). For anything that I want reliability I use one of these SanDisk cards, that Samsung, or a SanDisk extreme that I bought the other day. My “I don’t really care about this, but I’d like it to not fail” cards are Samsung Evo Select drives (or something green and Samsung). Only my “I really don’t give a shit about these” drives are those $3 Microcenter cards.
Why are you using a MicroSD at all? Get a used minipc with two sata ssds or a single board computer with eMMC or mpcie support
Yeah, I definitely won’t be buying a new pi anytime soon for exactly that reason but I’ve had this one for awhile and would prefer to do something useful with it.
BTW in my anecdata I’ve yet to have a failure on any of my SanDisk Extreme Pro SD cards. I have 4 in active use on different Pi 4s. The oldest one is in use since 2018-ish. It used to be a NAS till 2022 so roughly 3-4 years of 24/7 use with a bog-standard OS. It’s now running OpenWrt which does fewer writes.
Get a used minipc
Why, when the Pi is working fine? Just get a USB to SATA cable. This one works great with the Pi: https://a.co/d/8Jv2Erj
