summer server upgrade wrap-up

It’s finally upgrade time!

>(click here to skip to the new stuff)<

I originally built this homelab cluster in December of 2022 for a few reasons: for fun, for learning, and to help me pivot my career away from manufacturing and into IT.

I saved up a bunch of money, quit my factory job, and spent some months finishing up community college and learning homelab stuff full-time. During that time, I also blasted out 162 job applications.

I wanted to be as frugal as possible, but I still needed lots of hardware to build a server cluster and learn this stuff. Hence, me using a ton of e-waste, old thinkpads, and miscellaneous parts from my random gaming rigs over the years. Here’s how the cluster has evolved over the past several months!

saying goodbye to The old stuff:

Just a quick recap — This is a cluster of 4 machines: a thinkpad x270, a thinkpad x250, a dell inspiron (ivybridge i3), and the ATX board has a sandybridge 2600k. The giant pile of harddrives in the middle of the photo there is a ZFS raidz array with 4 drives, plus another couple external USB drives for proxmox backup server. There are some ssd’s hiding in there for boot drives as well.

The ax3000 up top served as both the router and switch for all this mess, with two wifi 6E asus ET8’s serving as access points elsewhere in the home.

The dell laptop is over 10 years old and belonged to my mother in law, my wife, and then to me — being used for college each time. Now it’s a server node!

The 2600k was from a gaming computer I built in 2011. It’s almost a teenager.

It maxed out at 32gb ram and had a sata controller that ran like an oldsmobile diesel. It also wouldn’t boot without a GPU, cannot passthrough devices (“K” series cpus of that era do not have VT-d), and since that radeon 6350 doesn’t have any transcoding features — it was just sitting there drawing power to allow the thing to boot.

I’ll miss it 🙁

On the storage side, I’m upgrading a dump truck load of mechanical drives to a few SSDs. I was using 3x 2TB 5400rpm wd green drives in a raidz array for my primary storage, a 750gb seagate drive for scratch area, and couple external drives as well.

Check out the manufacturing date on those old drives:

2009 and 2011 baby! Still kicking. (except the 2009 drive failed recently. RIP)

They are being replaced by a 4TB samsung evo and a 1TB crucial bx500. Snagged the samsung on amazon and stole the crucial out of my PS4 that I never play anymore. Interesting playstation engravings on the screw holding the drive into the PS4:

So what did I buy?????

I needed three things from this upgrade:

1. VT-d / pcie passthrough + more ram — the 2600k was not capable of this
2. Faster networking — backup/restore times were awful
3. More stuff to learn — managed switch features, ceph networked storage, HA, etc

So I scoured eBay for a few months. Now that I have landed an IT job and I’m earning again, I started buying stuff.

I managed to find these two SICK deals.

First, a QNAP qsw-m408-2c switch. This is a managed switch with 2x 10-gigabit SFP ports, 2x 10-gigabit SFP/ethernet combo ports, 8x 1-gigabit ports, and a management port.

This switch was on eBay for $179, mislabled as a much cheaper model unmanaged switch with way less features. Bank error in my favor.

Second, a Thinkstation P330 SFF with i7 8700, 32gb ddr4, and a Supermicro X540 10-gigabit nic. The seller had used this as a Proxmox host as well. It was meant to be! I snagged this for about $275.

The Upgrade Process:

First, I added some things to the Thinkstation.

I was able to squeeze 3 sata SSD’s (almost 6TB) into the hinged front panel:

Then I maxed out the ram to 64gb and threw in an additional 2.5 gigabit nic underneath the supermicro 10-gig:

This 2.5 gigabit nic was only about 40 bucks on ebay, and it came with a freakin’ minidisc! and a mini screwdriver. and a minicat.

I’ll probably end up using that bottom slot for a low-pro gpu in the future.

Then I installed Proxmox 8 on it:

While Proxmox 8 was installing on the Thinkstation, I installed the switch and got rid of the ax3000. It has served me well.

My switch is missing the cool looking circular power adapter but I won’t complain for the price.

yay:

Then I moved the server over to the cabinet and tidied it up a little:

My assistants were getting distracted at this point:

It took well over 12 hours to restore all the VMs and containers to the node once it was in place. My proxmox backup server is still running on the thinkpad x270 with a usb external drive.

Once everything was up and running, I created a virtual machine for OPNsense and set it up as my primary router. I used one of the 10-gig ports as the LAN interface, and one of the 2.5-gig ports as the WAN interface:

I also called and upgraded my internet service to the 1 gigabit down / 35 meg up package. They sent me a new modem with a 2.5-gig port. Combined with the opnsense setup, it’s pretty dang snappy!

All in all, the upgrade was super fun and pretty effective. Nextcloud is way faster, jellyfin now has hardware transcoding, and I am able to run a 32gb ARC for zfs while still having 32gb remaining for other stuff.

I think the proxmox backup server is next on the upgrade list! Can’t wait to keep chasing bottlenecks and throwing dollars at them.