state of the cluster: 1/23

The homelab has come a long way in the past couple of months. It’s still a heap of e-waste, but now it’s a BIG heap of e-waste!

It’s been so fun to learn all of this stuff. It’s been so fun, in fact, that I want it to be my job.

How it started:

It began with a single thinkpad. Just the x270, with a baremetal install of ubuntu server. That little 12″ thinkpad was hosting a litany of services, including: this wordpress site, my nextcloud server, the jellyfin media server we use for multiple TVs, our network file shares, a bunch of docker apps, a pihole with unbound dns server, a wireguard vpn tunnel, a lychee photo gallery, and more. It was a lot of stuff for a little thinkpad!

There were a couple of glaring problems with this setup which you might already be imagining:

1. Using it for live services while also using it as a learning/testing tool
2. No backups!!

The disaster:

You saw it coming, right? I should have, too!

I was still awake at 5am, goofing around with a docker container. I should have swept myself off my feet and carried myself to bed. Never make configuration changes to a live server with all your crap on it at 5am! That’s the first thing they teach you.

I was editing the yaml file, trying to add a path to my external storage (with about 4tb of my stuff on it). My sleepy eyes saw the word “path” and typed the path on that line.

I had accidentally written the /path/to/my/storage in the path to the docker container itself. After hitting ctrl+x and restarting docker, POOF– all my stuff was gone. The whole drive overwritten by a docker container file.

I was too mad to learn the ins and outs of ext4 data recovery, inodes, or whatever else I could have done to recover it. I just decided to start over and do it smarter.

Proxmox has entered the chat

I decided the best way to prevent that from happening again would be to virtualize everything. And so began my new hobby, obsession, and hopefully career!

Virtualizing all of your self-hosted services on a hypervisor comes with a number of huge advantages. To list a few:

1. Gives you the ability to test new things, learn new skills, etc safely–without endangering your running services and data
2. Makes backups and snapshots a breeze, so you can test a change and roll it back instantly if you don’t like it. Or recover instantly from a huge disaster
3. Lets you move services around to other computers (while they’re running!), so you can make hardware changes, install updates, or whatever you need to do without downtime
4. It’s crazy fun

All major hypervisors these days support clustering, which is an important feature for point 3 above. Having another server node running that you can seamlessly migrate containers or virtual machines to and from is a huge benefit.

I chose Proxmox for a couple reasons specifically:

1. Great support for consumer hardware (like my pile of e-waste) rather than enterprise-grade stuff
2. It’s based on Linux KVM. If you know linux, you know Proxmox!

The setup as of January:

The x270 is still there! It has been joined by a few friends now–two more laptops and my ancient desktop. The 4 server nodes now are:

1. thinkpad x270 – i5 7200u, 8gb, 256gb ssd, 5tb usb storage
2. thinkpad x250 – i5 5200u, 8gb, 256gb ssd
3. inspiron 15 – i3 3227u, 8gb, 256gb ssd, 320gb usb storage, tp-link usb gigabit nic
4. ancient desktop – i7 2600k, 32gb, 128gb ssd, 3x 2tb hdd in raidz1, a 750gb hdd, and a radeon hd6350 (won’t boot without a gpu installed!)
5. asus ax58u
6. cisco 2800 series… for looks
7. server turtle

In Proxmox, it looks like this:

The 2600k does most of the tuff stuff, since it has the most memory. It runs a few ZFS pools, one of them in raidz1. 3x 2tb yields nearly 4tb of usable space at pretty decent performance. Not bad for drives that were sitting in a closet collecting dust for nearly a decade. These pools contain the fileserver and other containers that don’t need to be super fast. They run ztsd compression and with a 16gb ARC cache, seem to be generally awesome for my uses so far.

The inspiron does mostly light stuff. You know, low weight high reps. It runs the primary dns server and the wireguard vpn tunnel that my cellphone is always connected to. I use Pi-hole and Pi-vpn for these tasks, they’re awesome.

The x250 gets used and abused for testing new stuff. I usually spin up new virtual machines and containers there first, before moving them somewhere else. Right now it’s home to my Windows Server 2022 virtual machine, where I’m learning Active Directory stuff–coming soon to a resume near you.

What about the x270???

I thought you’d never ask!

Remember those backups we were talkin’ about?

That’s its whole job! The x270 also runs a zfs pool, on its 5tb usb drive. That pool is the primary datastore for my Proxmox Backup Server. PBS can be run all sorts of ways, including on a host. But the coolest way is to run it in standalone mode like this:

Twice per day, this little thinkpad zips through all the other nodes and snapshots every container and virtual machine. It only transfers back the data that has changed since the last snapshot. It then compresses the poop out of these snapshots and saves them for as long as I want (2 weeks).

Even with such frequent snapshots, the data usage is really super minimal:

I’ve already used this to save my butt several times since I implemented it last month. It can even restore individual files from inside the whole snapshots of virtual drives!!

Conclusion

So there you have it–my setup is significantly more idiot-proof than it was a few months ago. It’s still vulnerable to fires or tornadoes or floods or cats or dogs or wives, but it’s protected from me, which honestly was its biggest natural threat.

And it cost me nothing! All of this crap was laying around the house in various closets and drawers. Now it’s a super cool server cluster and a learning platform for me to launch a new chapter of my life. Neat!

Now go build your own homelab!