I have a load-bearing raspberry pi on my network - it runs a DNS server, zigbee2mqtt, unifi controller, and a restic rest server. This raspberry pi, as is tradition, boots from a microSD card. As we all know, microSD cards suck a little bit and die pretty often; I’ve personally had this happen not all that long ago.
I’d like to keep a reasonably up-to-date hot spare ready, so when it does give up the ghost I can just swap them out and move on with my life. I can think of a few ways to accomplish this, but I’m not really sure what’s the best:
- The simplest is probably cron + dd, but I’m worried about filesystem corruption from imaging a running system and could this also wear out the spare card?
- recreate partition structure, create an fstab with new UUIDs, rsync everything else. Backups are incremental and we won’t get filesystem corruption, but we still aren’t taking a point-in-time backup which means data files could be inconsistent with each other. (honestly unlikely with the services I’m running.)
- Migrate to BTRFS or ZFS, send/receive snapshots. This would be annoying to set up because I’d need to switch the rpi’s filesystem, but once done I think this might be the best option? We get incremental updates, point-in-time backups, and even rollback on the original card if I want it.
I’m thinking out loud a little bit here, but do y’all have any thoughts? I think I’m leaning towards ZFS or BTRFS.
Ditch the SD card all together and get a cheap SSD to use as the boot drive instead.
I can’t remember the steps (they were simple though) but when my Home Assistant raspi SD card died, I bought a 128gb SSD from AliExpress and a usb-sata cable.
I then did something to the pi that meant it can boot from the SSD, and flashed the SSD using Balenetcher or RUFUS or whatever (same program I was using to flash my SD cards basically).
Then it was just a case of plugging in and turning it on.
Runs exactly the same as with an SD card with less dying because SD cards aren’t meant for a lot of read/write but SSDs do.
I’ve had very bad luck with raspberry Pi’s and SDCards. They just don’t seem to last very long. I swapped to usb storage and things got somewhat better. I just had a usb drive die after 3 to 4 years of use. When I was still using SD it seemed like multiple times a year. Heat. Power loss, you can only punch holes in silicon so many times before it wears out. Whatever the reason.
My approach for this is configuration backup not the entire os. I think this approach is better for when it’s time to upgrade the os or migrate to a new system.
For my basic Pi running WireGuard and DNS, I keep an archive of documentation on steps to reconfigure the system after a total loss. Static configs are backed up once, and If there are critical configuration items that change then I back those up weekly. I’ve got two systems (media related servers, not Pi’s) that I keep ansible playbooks to configure 90% of the system from scratch so it’s as hands off as it can be.
Yeah, I’m getting a pretty strong consensus here that an SSD is the way to go. I’ve also had at least one SD card die on me, and because I didn’t have backups it was pretty inconvenient. Had to recreate my homeassistant setup from scratch.
I get the config only backup, but when I have a mondohuge nas available and we’re dealing with like less than 100 gigs, why not just take a full disk image?
Another option is to use Image File Utilities on the Pi to create an image backup. You can use cron + a bash script to create incremental backups using the tool (e.g., take a ‘fresh’ backup each month, with daily incremental backups in between). I mount a network ‘backup’ drive (a local NAS, but you could use anything) to save the image to so I can actually access it. Then, just use balena etcher to flash the backup iso in the event of a failure.
- Here’s Image File Utilities: https://forums.raspberrypi.com/viewtopic.php?t=332000
- And a backup script + cron entry: https://forums.raspberrypi.com/viewtopic.php?p=2199452&hilit=cron#p2199452
I also use those, but (shame on me) I’ve never tested the backups. Have you ever restored such an image backup?
Happy to help! There’s plenty of other options too (e.g., SSD) as folks mentioned, but this works well out of the box with no additional hardware. SD has been absolutely fine for my use (Pi-Hole), while still requiring maximum uptime so the family doesn’t riot if the internet is out.
I’ve always used dd + sshfs to backup the entire sd card daily at midnight to an ssh server; retaining 2 weeks of backups.
Should the card die, I’ve just gotta write the last backup to a new card and pop it in. If that one’s not good, I’ve got 13 others I can try.
I’ve only had to use it once and that went smoothly. I’ve tested half a dozen backups though and no issues there either.
Yeah “dd if=/dev/mmcblk0 of=$HOSTNAME.$(date +%Y.%m.%d).img” and while its running. (!!! Make sure the output is NOT going to the sd card you are backing up…)
I deliberately chose a time when it’s not very active to perform the backup. Never had an issue, going on 6 years now.
