Athena
Tuesday, Jan 4, 2022 · 2300 words · approx 11 mins to readBack in October 2020 I started the search for a replacement for the NAS I’d put together to run my home network. At the time the disks had just passed the 5 year mark in terms of power-on hours, and so they were ripe for replacement at the very least. Despite having a trio of them in a configuration where one could fail and the array would survive, their age meant that I was starting to worry that one drive failing would cause a cascading failure in the other two, as they were put under stress as the array rebuilt after any failed drive was repaired.
The machine is the central hub of my entire LAN, carrying out a wide range of tasks on behalf of the other devices on the network, on top of the traditional functionality of networked storage. In reality, the networked storage part of the machine is probably the least useful to me and the least leaned upon. My data storage needs are low and the extra data I need to storage grows very slowly year-on-year.
Some of that is down to the advent of cloud storage for my photos, which would make up the vast bulk of the personal data that I couldn’t stand to lose. Some of it is down to music and video streaming turning to that model, so I stopped buying and ripping Blu-Rays at any serious rate years ago. Honestly speaking, the rest of it is down to me just being very unadventurous and conservative in what I do with my computers. I primarily deal in text, be that writing or code, and so the amount of data I add to my personal archives that I need to store at home is almost nothing.
Still, I couldn’t really trust that modest increase in data storage requirements to 5 year old spinning rust disks, despite their seeming reliability, and so the hunt was on. 8 months later in June of this year and I finally pulled the trigger. Cost was a factor in it taking so long to do something after the search started, since the replacement machine needs to have a decent amount of CPU and memory capacity due to the VMs and jails I need to run, and I wanted a 10G Ethernet port to do high-speed networking to my Mac Pro. Those ports still don’t come that cheap.
That 10G Ethernet desire also put pressure on the storage subsystem, since you can’t get data off of disk storage and over the wire at that kind of line rate unless it’s from either a huge spinning rust array (at least 10 disks in some kind of perilous RAID), or SSDs. SSDs aren’t cheap enough to use for bulk storage, but they are cheap enough to use as read and write caches in front of bulk storage, in order to much more easily drive higher-than-1G line rates and therefore make investment in a 10G port worth it.
On top of that I had a healthy dollop of COVID-driven depression which manifest itself in a complete inability to make any kind of decision about almost anything, never mind an expensive NAS purchase. I’m sure you can relate. Thankfully, being depressed also sometimes pulls the be-incredibly-reckless-with-money lever in my brain (see the aforementioned Mac Pro, and recently purchased MacBook Air), which overrode the crippling inability to make a decision, and so I was off to the races!
So what got ordered on hardware side for NAS 2021, compared to NAS 2016? I had no real idea what to do on the CPU side, but I knew that in the 5 years that have passed I’ve ended up doing quite a lot with the 2-core CPU in the old one, so I set at least a 2x increase in overall performance as the benchmark there. I flirted with the idea of building an embedded 8-core EPYC 3251-based system for a while, centred around a Supermicro M11SDV-8C-LN4F, but it was so difficult to get hold of that board in the UK that I gave up considering it eventually, plus it meant taking care of the rest of the build myself.
One of the great benefits of something like the HPE Gen8 Microserver that I was replacing is that someone else has taken care of integrating the core logic into a decent chassis, cabling it all up and optimising for the available space. On a device like the Microserver you also get the advantage of one particular extra, and very desirable, feature: completely remote management. The pull of something like that again was too strong in the end, so while my paymasters won’t like to hear that I didn’t pull the trigger on the EPYC, I eventually went with HPE’s latest Gen10 Plus Microserver again.
The Gen10 Plus comes with a range of CPU options, and there are a bunch of potential CPUs that you can fit yourself that HPE don’t sell, too. ServeTheHome have an excellent guide to the possible CPU upgrades for the Gen10 Plus, along with advice on what else can you an add to the system via PCI Express. More on that in a bit. I ended up selecting one of the configurations with an Intel Xeon E-2224, a 4-core processor with no HyperThreading and ECC support. The lower-end 2-core, 4-thread Pentium G5420 variant also supports ECC, but it wouldn’t have been much of an upgrade over the CPU in the Gen8, despite the E-2224 being an 8th generation “Coffee Lake” processor variant at higher clocks.
I have 16 GiB of RAM in the Gen8, which ended up not being a huge limitation in the lifetime of that machine, however the Gen10 Plus with a Xeon preinstalled is only available with a single 16 GiB stick, and running any modern processor with just a single memory channel is never advisable if you care about performance. So I bought a Crucial/Micron MTA18ASF2G72AZ-2G6E2 16 GiB ECC UDIMM to fill the 2nd slot in the system and give me 32 GiB total.
Lastly I needed to figure out storage and connectivity. The outgoing Gen8 had a trio of 3TB WD Reds in a ZFS raidz1, plus a 480GB SATA SSD for less important storage. I ended up running a good chunk of my network’s services on the SSD in the end because it was silent and didn’t cause seek activity on the WD Reds, which aren’t the quietest disks in the world. I wanted to replicate that mostly silent running as much as possible, but I couldn’t convince myself to go all SSD on the storage. In hindsight I regret that decision immensely.
What I ended up getting was a quartet of Seagate IronWolf 5900rpm NAS drives, 4TB each. They fit into the 4 3.5” drives bays on the Gen10 Plus and provide the bulk storage. However they’re incredibly noisy when they seek, and with the machine right with me in my office the noise is very noticeable. I’d always planned to mitigate that with an SSD in front of them as a read cache, but with the 4 bays filled and no M.2 storage, I had to get a little bit creative when it came to adding SSDs to the system.
Thankfully, the Gen10 Plus does have a PCI Express 3.0 x16 slot. The amount of bandwidth provided by that slot, which is directly connected to the Xeon CPU, is more than enough for a couple of 4-lane NVMe SSDs and a 10Gbps network connection. Thankfully, someone makes a low-profile device that provides those three devices over a single PCIe x8 3.0 device that fits into the Gen10 Plus’ PCIe slot. Enter theSynology E10M20-T1, which I populated with a pair of Samsung 970 Evo Plus 1TB NVMe M.2 SSDs.
Designed to provide NVMe-based disk caches and a 10G Ethernet port to Synology’s NAS systems, the device is built around a standard PCIe switch and a 10G Ethernet system from Marvel Aquantia. FreeBSD 12+ has serviceable drivers for that network ASIC (although beware, they have quirks if you’re using that OS like I am), so the E10M20-T1 allows me to bring some M.2 NVMe to the party and front that with a very high-speed Ethernet solution that allows that storage to be accessed with high performance and not be limited by the Gen10 Plus’ on board 1G Ethernet ports. The lack of 10G Ethernet and M.2 slots for SSD drives is the one glaring flaw in the Gen10 Plus spec sheet, something that HP should fix with their own E10M20-T1-like add-in board.
Why their own, if Synology make something? Because while the E10M20-T1 fits into the Gen10 Plus, it doesn’t without some slight modification. Due to the design of the interior of the Gen10 Plus chassis, part of the internal chassis fouls on the top edge of the end of the E10M20’s PCB, so a piece of that needs to be cut off. I wish I took some good pictures of the modification so that I could guide others through it, but the upshot is that I took a Dremel to that part of the PCB and took a 1cm x 2cm ish notch out of the end. Cutting the PCB there is fine since there are no components, but it does expose some of the copper used for the PCB layers, which you need to mask off with some electrical tape. It’s a bit of a severe thing to do in order to add 10G and NVMe SSDs to the Gen10 Plus, especially given the cost of the card, but there really is no other good way that’s all internal.
The Gen10 Plus doesn’t have any high-speed USB expansion which might be usable for a 10G Ethernet NIC, since the USB 3.1 Gen 1 ports top out at 5Gbps each. On top of that, nobody else seems to make a device like the E10M20-T1 that has 2 M.2 slots and a 10G Ethernet ports, in a low-profile form factor. I donate one of the 970 Evo Plus drives to the storage subsystem in the form of an ARC cache, to help reduce access on the IronWolf drives, and the other is just mounted as a single ZFS storage volume with no redundancy, so I regularly snapshot that filesystem and send it to the raidz1 array on the IronWolf drives for backup.
That’s about it! I run FreeBSD 13 on the machine at the time of writing, and administer it myself, rather than use a pre-packaged NAS-friendly OS like TrueNAS. I used FreeNAS (now TrueNAS!) on the older Gen8, and it was fine, but the FreeNAS folks are a little too happy to break backwards compatibility as they develop their OS, hiding older features or deprecating them entirely in new OS releases, despite those features being ones you might rely on in your system.
That meant I had this weird split-brain way of administering the old FreeNAS-based system that I disliked in the end. I’m happy to maintain a simple OS configuration myself, and spin up VMs, manage the storage and do other maintenance tasks myself, but TrueNAS is what I’d recommend to anyone else.
The system’s been up and running for a good while now and has been rock solid, bar some issues getting the 10G port on the E10M20 to come up properly during the boot process. The FreeBSD drivers for the Aqtion NIC aren’t the most mature and polished, so it’s not really feasible to have the 10G port as the only one active in the system, but as long as I have one of the Intel-based 1G ports enabled as the one the NAS can boot and come up with, and bring the 10G port online separately after the system has booted FreeBSD, that works fine.
As well as providing network storage for my LAN, as mentioned I run all kinds of other stuff on it. My DNS and DHCP setups for the LAN, the controller for my UniFi setup, a NextCloud instance, a UISP instance to manage my router, a couple of utility VMs for various backup automations, a system that grabs and verifies the remote backups that my server at Hetzner makes. It’s the hub of my LAN and without it the entire LAN just stops working, so it’s the most important computer in the house in a lot of ways.
Oh, before I forget as I wrap up: because HPE are horrible, they make you buy the iLO remote management port separately. It’s about £35, but it’s essential on a server like the Gen10 Plus. The system needs a password to boot because of disk encryption, and there’s no way I can realistically drag it to a monitor every time I boot it, so I need a way to enter that password completely remotely. The iLO lets me do that, so if you’re buying a Gen10 Plus for yourself, make sure you get the iLO Enablement Kit, which is what HPE call it, for yours.
So that’s NAS 2021 (even though I’m finishing writing this in very early 2022!). Hopefully it takes me through to NAS 2026, given the cost. Total investment is quite a bit more than what I paid for NAS 2021 (almost double in fact), but it’s a solid upgrade in all major areas. 2x the CPU performance, 2x the RAM, and easily 2x the basic storage performance due to the NVMe. I just wish the IronWolf drives were silent. If I chance upon a cheap deal for 4TB SATA SSDs, I might just move to all SSD storage on the thing in the future. If you’re building something similar, think about that yourself if you can afford it.