Network design. I started my homelab / selfhost journey about a year ago. Network design was the topic that scared me most. To challenge myself, and to learn about it, I bought myself a decent firewall box with 4 x 2.5G NICs. I installed OPNsense on it, following various guides. I setup my 3 LAN ports as a network bridge to connect my PC, NAS and server. I set the filtering to be applied between these different NICs, as to learn more about the behavior of the different services. If I want to access anything on my server from my PC, there needs to be a rule allowing it. All other trafic is blocked. This setup works great so far an I’m really happy with it.
Here is where I ran into problems. I installed Proxmox on my server and am in the process of migrating all my services from my NAS over there. I thought that all trafic from a VM in Proxmox would go this route: first VM --> OPNsense --> other VM. Then, I could apply the appropriate firewall rules. This however, doesnt seem to be the case. From what I’ve learned, VMs in Proxmox can communicate freely with each other by default. I don’t want this.
From my research, I found different ideas and opposing solutions. This is where I could use some guidance.
- Use VLANs to segregate the VMs from each other. Each VLAN gets a different subnet.
- Use the Proxmox firewall to prevent communication between VMs. I’d rather avoid this, so I don’t have to apply firewall rules twice. I could also install another OPNsense VM and use that, but same thing.
- Give up on filtering traffic between my PC, NAS and server. I trust all those devices, so it wouldn’t be the end of the world. I just wanted the most secure setup I could do with my current knowledge.
Is there any way to just force the VM traffic through my OPNsense firewall? I thought this would be easy, but couldn’t find anything or just very confusing ideas.
I also have a second question. I followed TechnoTim to setup Treafik and use my local DNS and wildcard certificates. Now, I can reach my services using service.local.example.com, which I think is neat. However, in order to do this, it was suggested to use one docker network called proxy. Each service would be assigned this network and Traefik uses lables to setup the routes. ’
Would’t this allow all those services to communciate freely? Normally, each container has it’s own network and docker uses iptables to isolate them from each other.
Is this still the way to go? I’m a bit overwhelmed by all those options.
Is my setup overkill? I’d love to hear what you guys think! Thank you so much!
I would probably put my WLAN on a different subnet/VLAN then the rest of the network, maybe even split it into trusted and guest networks if your AP’s support broadcasting multiple SSID’s. I don’t see anything about NAT/PAT so I assume this isn’t going to touch the Internet and just for home brew fuckery.
Both public and local services. I have limited hardware for now, so I’m still using my ISP router as my WLAN AP. Not the best solution, I know, but it works and I can seperate my Home-WLAN from my Guest-WLAN easily.
I want to use an AP at some point in the future, but I’d also need a managed switch as well as the AP itself. Unfortunately, thats not in my budget for now.
I’m not knowledgeable on communication between VMs and how to best restrict communication there, but I have tried to make my docker networks more secure.
I went a bit overkill for my reverse proxy and all the docker networks it’s connected to. For each service I want to expose through my reverse proxy, I manage a network specifically for that service in my caddy docker compose file. I then refer to that external network in my servjce’s docker compose file, so that caddy can access it. For example, caddy is on caddy_net-grafana and on caddy_net-homepage. Grafana and homepage are on those networks respectively. So with this setup, caddy can talk to Grafana and homepage, but Grafana and homepage cannot talk to each other.
It wasn’t too bad to setup. I made my own conventions for keeping it manageable and it works for me. I did run into the problem where I had to increase the default subnet pool, as after you create like 30 or 31 networks there aren’t any subnets left to give out to new docker networks.
This sounds promising. If I understand correctly, you have a ton of networks declared in your proxy, each for one service. So if I have Traefik as my proxy, I’d create traefik-nextcloud, traefik-jellyfin, traefik-portainer as my networks, make them externally available and assign each service their respective network. Did I get that right?
By making a bridge in the opensense interfaces you have created a layer2 network. This means that all the devices connected on that network are broadcasting their Mac addresses and are added to the ARP table on the opensense. Since they all are on the same physical network and the same subnet, none of the traffic will ever hit the layer 3 rules on your opensense.
If you want opensense to handle the rules of the traffic you will need to put the devices on different subnets and separate clans. Create a gateway address for every vlan on the opensense and point your devices to the opensense as their gateway.
Ah, I did not know that. So I guess I will create several VLANs with different subnets. This works as I intended it, trafic coming from one VM has to go through OPNsense.
Now I just have to figure out, if I’m being to paranoid. Should I simply group several devices together (eg, 10=Servers, 20=PC, 30=IoT; this is what I see mostly being used) or should I sacrifice usability for a more fine grained segeration (each server gets its own VLAN). Seems overkill, now that I think about it.
VLANs all the way. I have several VLANs, including:
- Virtual Servers
- Bare metal
- Trusted devices
- IoT devices
- Guest network etc.
EDIT: An alternative would be to replace or supplement Proxmox with Docker/Podman on the bare metal of the server. The container networking would be isolated by default. If you can replace your VM needs with containers, that may get you what you want.
