I saw this post today on Reddit and was curious to see if views are similar here as they are there.
- What are the best benefits of self-hosting?
- What do you wish you would have known as a beginner starting out?
- What resources do you know of to help a non-computer-scientist/engineer get started in self-hosting?
The big thing for #2 would be to seperate out what you actually need vs what people keep recommending.
General guidance is useful, but there’s a lot of ‘You need ZFS!’ and ‘You should use K8s!’ and ‘Use X software!’
My life got immensely easier when I figured out I did not need any features ZFS brought to the table, and I did not need any of the features K8s brought to the table, and that less is absolutely more. I ended up doing MergerFS with a proper offsite backup method because, well, it’s shockingly low-complexity.
And I ended up doing Docker with a bunch of compose files and bind mounts, because it’s shockingly low-complexity. And it’s just running on Debian, instead of some OS that has a couple of layers of additional software to make things “easier” because, again, it’s low-complexity.
I can re-deploy the entire stack on new hardware in about ~10 minutes (I’ve tested this a few times just to make sure my backup scripts work), and there’s basically zero vendor tie-in or dependencies that you’d have to get working first since it’s just a pile of tarballs and packages from the distro’s package manager on, well, ANY distro.
IMO a homelab for learning and a server that you’re self-hosting services on really aren’t the same thing and maybe shouldn’t be treated that way, if you can swing it.
I’d rather my password manager or jellyfin or my peertube instance or whatever not be relying on a tech stack I don’t entirely understand and might not be able to easily fix if it breaks.
I guess a lot of it is new to doing this vs greybeard split, since the longer I’ve done sysadmin work the less I care about the cool new thing and have a preference for the old, stable, documented, bugfixed, supported, and with a clear roadmap software.
I should probably get a job doing sysadmin work for a bank, lmao.
I have made that migration myself going from a Raspberry PI 4 to a n100 based NAS. It was 10 minutes for the software stack as you said This not taking into account media migration which was done on the background over a few hours on WiFi (I had everything on an external hard drive at the time).
That last part is the only thing I would change about my self hosting solution. Yes, the NAS has a nice form factor, is power efficient and has so far been very optimal for my needs (no lag like rpi4), however I have seen they don’t really sell motherboard or parts to repair them. They want you to replace it with another one. Reason 2 on the same is vendor lock in. Depending on the options you select when creating the storage groups/pools (whatever they are called), you could be stuck needing to get something from the same vendor to read your data if the device stops working but the disks are salvageable. Reason 3 is they’ve had security incidents so a lot of the “features” I would not recommend using ever to avoid exposing your data to ransomware over the internet. I don’t trust their competitors either. I know how commercial software is made with the smallest amount of care for security best practices.
Yeah, I just use plain boring desktop hardware. (Oh no! I’m experiencing data corruption due to the lack of ECC!) It’s cheap, it’s available, it’s trivial to upgrade and expand, and there’s very few little gotchas in there: you get pretty much exactly what it looks like you get.
Also nice is that that you can have a Ship of Theseus NAS by upgrading what needs upgrading as you go along and aren’t tied into entire platform swaps unless it makes sense - my last big rebuild was 3 years ago, but this is basically a 10 year old NAS at this point.
btrfs with its send/receive (incremental fs-level backups) is already stable enough for mostly everything (just has some issues with raid 5/6), and is much more performant than zfs. And it is also in the linux kernel tree (quite hugely useful). Of course, if more zfs-like functionality is what you look for.
“Already stable enough”
- no it isn’t.
- if fucking should be, it’s been around 15 years!
My only experience with btrfs was when trying out Opensuse Tumbleweed. Within a couple days my home partition was busted, next time it was another partition. No idea if the problems could be fixed as these were fairly new installations to give Opensuse a try and I couldn’t be bothered to fix a system that’s troubling me from the very beginning.
Between all the options that just work ™, btrfs is the one I’ve learned to stay away from.
EDIT: that was four or five years ago
Honestly it’s not; BTRFS has been in my ‘that’s neat, but it’s still got a non-zero chance of deciding to light everything on fire because it’s bored’ list for, uh, a decade now?
The NAS build is old enough to more or less predate BTRFS being usable (closing in on a decade since I did the initial OS install, jeez) and none of the features matter for what I’m storing: if every drive in my NAS died today, I’d be very annoyed for a couple of hours during the rebuild, and would lose terrabytes of linux ISOs that I can just download again, if I wanted to use Jellyfin to install them a 2nd time. (Any data I care about is pulled offsite at least once a day, so I’ve got pretty comprehensive backups minus the ISOs.)
I know EXT4 and mergerfs and snapraid are not cool, or have shiny features, but I’ve also had zero problems with them over the last decade, even between Ubuntu upgrades (16.04, 18.04, 20.04, 22.04) and hardware platform upgrades (6600k, 8700k, 10950k) and the entire replacement of all the system drives (hdd -> ssd -> nvme) and the expansion of and replacement of dead HDDs, of varying sizes (4tb drives to 8tb drives to 16tb drives to some 20tb drives).
It all just… worked, and at no point was I concerned about the filesystem not working if I replaced or upgraded or changed something, which is not something ZFS or BTRFS would have guaranteed during that same time window.
IMHO 99% of the time btrfs features are used as a band-aid for things that would be much better done otherwise. Generally by using a stable distro and a decent backup solution (like Debian + Borg). And you get to use a truly stable, proven, boring fs ike ext4 or xfs.
Stable yes, but no protection from bitrot, and the journal of ext4 is the band aid, instead of a cow fs like zfs or btrfs.
Can you elaborate on how your backup script re-deploys on new hardware? Sounds very nice to have.
elaborate
It’s a really simple script.
Everything is deployed with a docker compose, and all the docker volume data are bind mounts and, for example, a Jellyfin install would have everything in /stacks/jellyfin.
The backup script makes a tarball of each service individually (and stops the stack if there’s anything in there doing database things or anything else that might end up being inconsistent by just archiving the filesystem), and uploads them to a S3 storage provider AND burns them to a BluRay.
The recovery script does the opposite: it downloads and unarchives the data.
As long as you’re on Linux and have Docker, it should just magically work.
- you do not need kubernetes
- you do not need anything to be „high availability”, that just adds a ton of complexity for no benefit. Nobody will die or go broke if your homelab is down for a few days.
- tailscale is awesome
- docker-compose is awesome
- irreplaceable data gets one offsite backup, one local backup, and ideally one normally offline backup (in case you get ransomwared)
- yubikeys are cool and surprisingly easy to use
- don’t offer your services to other people until you are sure you can support it, your backups are squared away, and you are happy with how things are set up.
To piggy back on your “You don’t need k8s or high availability”,
If you want to optimize your setup in a way that’s actually beneficial on the small, self hosted scale, then what you should aim for is reproducibility. Docker compose, Ansible, NixOS, whatever your pleasure. The ability to quickly take your entire environment from one box and move it to another, either because you’re switching cloud providers or got a nicer hardware box from a garage sale.
When Linode was acquired by Akamai and subsequently renamed, I moved all my cloud containers to Vultr by rsyncing the folder structure to the new VM over SSH, then running the compose file on the new server. The entire migration short of changing DNS records took like 5 minutes of hands-on time.
I’ve been in love with the concept of ansible since I discovered it almost a decade ago, but I still hate how verbose it is, and how cumbersome the yaml based DSL is. You can have a role that basically does the job of 3 lines of bash and it’ll need 3 yaml files in 4 directories.
About 3 years ago I wrote a big ansible playbook that would fully configure my home server, desktop and laptop from a minimal arch install. Then I used said playbook for my laptop and server.
I just got a new laptop and went to look at the playbook but realised it probably needs to be updated in a few places. I got feelings of dread thinking about reading all that yaml and updating it.
So instead I’m just gonna rewrite everything in simple python with a few helper functions. The few roles I rewrote are already so much cleaner and shorter. Should be way faster and more user friendly and maintainable.
I’ll keep ansible for actual deployments.
I have a k3s cluster for fun and I can admit that k8s is way too complicated.
I don’t want to dig hours through documentation to find what I’m looking for. The docs sometimes feel like they were written for software devs and you should figure part of the solution yourself.
I have a ExternalName service that keeps fucking up my cluster everytime it restarts, bringing down my ingresses, because for some reason it doesn’t work and I have no idea where to look at to figure out why it doesn’t work - I just end up killing the service and reapplying the yaml file and it works.
I had to diagnose why my SSL certificates would get stuck in “issuing” in cert-manager, had to dig through 4 or 5 different resources until I got to an actual, descriptive error message telling me that I configured my ClusterIssuer wrongly.
I wanted a k3s cluster to learn but every time I have issues with it I realize it’s a terrible idea.
I wish I had podman + compose but it does seem like a docker-compose is more complicated. Also, I wish I could do ansible but I have no idea where to start (nor how it works).
EDIT: oh yeah I also lost IPv6 support because k3s by default doesn’t enable v6 and I was planning on using Hetzner CCM to have a 2 node cluster until I realized Hetzner Networks don’t support v6.
I just moved everything from vultr to self host because of their latest changes.
EDIT: As I suspected, the changes that u/mesamunefire is referencing are the ones that taken out of context awhile back and incorrectly assumed to apply to user VPS’ and the data on them, which is not the case. Those terms only apply to information posted publicly to their website, like the community forums.
What changes would those be
Not needing Kubernetes is a broad statement. It allows for better management of storage and literally gives you a configurable reverse-proxy configured with YAML if you know what you’re doing.
Well I guess podman works fine for the first few months. Interestingly I still use build-ah heavily for building my custom images
Heavy disagree on the storage statement from what I’ve used and seen but it works for lots of people so not going to detract. NFS is always a pain but longhorn seems to have advantages
NFS is a pain, no question about it. I used to use longhorn but these days since I’m doing a single node k3s I’m just doing hostpath. It’s that PVCs make intuitive sense to me, but I guess podman will likely work just fine for such cases other than canary deployments and OOTB service-meshes
I wish I knew not to trust closed source self-hosted applications, such as Plex. Would have saved a lot of time and money.
Plex is a great example here. I’ve been Hetzner customer for many many years, and bought a lifetime license to Plex. Only to receive few months later a notification from Plex that I am no longer allowed to self-host Plex for myself(and only myself) at Hetzner and that they will block all access to my self-hosted Plex instance. I tried to ask for leniency or a refund, but that was wasted effort as well.
In short, I was caught on a crossfire when for-profit company tried to please hollywood by attempting to reduce piracy, so they could get new VC funding.
…
I am now a happy Jellyfin user and warmly recommend all Plex users to try it, the Jellyfin community is awesome!
(Use your favourite search engine to look up “Hetzner Plex ban” for more details)
It is much easier to buy one “hefty” physical machine and run ProxMox with virtual machines for servers than it is to run multiple Raspberry Pis. After living that life for years, I’m a ProxMox shill now. Backups are important (read the other comments), and ProxMox makes backup/restore easy. Because eventually you will fuck a server up beyond repair, you will lose data, and you will feel terrible about it. Learn from my mistakes.
My reason for self hosting is being in control of my shit, and not the cloud provider.
I run jellyfin, soulseek, freshRSS, audiobookshelf and nextcloud. All of that on a pi 4 with an SSD attached and then accessible via wireguard. Also that sad is accessible as nfs share.
As I had already known Linux very well before I’ve started my own cloud, I didn’t really had to learn much.
The biggest resource I could recommend is that GitHub repository where a huge amount of awesomely selfhosted solutions are linked.
