Necessary disclaimer that I haven't used Proxmox before and don't know enough yet to know how big the learning curve will be.
I am starting to encounter more and more issues with Truenas Scale that make me want to have more control over my homelab while keeping things modular and easily maintainable. After a little bit of deliberation, I decided that migrating to Proxmox would likely allow me to more easily do the things I want and also becore a great opportunity to incorporate high availability into my setup.
Here is what I want to acheive with that new setup:
- Have multiple VMs running my PFsense install, Pihole, VM for docker, HAOS, sandboxing...
- Have high availability set up for every component I deem "necessary" and automatically fire it up on a secondary node once it's primary node goes offline.
- Keep full backups of everything without simply "syncing" my main and backup pools to prevent data loss
Here is the current hardware I intend to use with the new cluster:
- Router:
Dell R220, E3-1286v3, 4GB RAM
Would mostly only run PFsense/Pihole in normal operation
- Server 1:
Dell R520, dual E5-2450, 64GB RAM
Hosts the primary storage pool
Would run the docker VM in normal operation
- Server 2:
Custom built, Ryzen 5 3400G, 32GB RAM
Hosts the backup storage pool
Would run nice-to-have VMs, sandboxes...
I realize that such a hardware mismatch could cause complications as to high availability, and I am open to replacing one of the nodes with something else if that will measurably improve reliability, simplicity, performance, etc. I would however like to avoid buying too many new drives as I already had to buy two 16TB drives earlier this year wich left a pretty big mark on the homelab budget.
I wanted to have all three nodes have an "apps" drive that would be synced across all three of them (from what I understand, this should be running in a CEPH cluster), all the VMs and containers would operate from that pool from their host machines.
The main "Mass" storage pool would still be hosted on the R520, with the other backup pool serving as a replication target. My concern is, in the event of a node failure for this specific node, could there be a way for the cluster to keep operating from the backup and store a diff somewhere that could eventually be sent to the main storage node once it becomes available again? So far, everything has been on a single dataset of that main ZFS pool, could I split things into multiple datasets and have a portion of the pool be replicated across the other nodes in real time and place the important data there? I want to avoind placing all my media on the apps pool.
My goal with high availability is to be able to fully shut down one machine for cleaning, maintenance, upgrades and whatnot without impacting the operation of my network. I want to be able to click shut down on any node and my router, jellyfin and everything to keep working (I often decide its time to do some maintenance when specifically streaming with jellyfin so that'd be a nice touch). Having the services stay up and running in the event of a failure with minimal downtime would just be icing on the cake.
Am I seeing too big? Would such a goal require a real cluster of 3 similarly spec'ed machines and buying waaay more drives?
Thanks everyone!