2.5 Gigabit Ethernet with Ubiquiti Unifi US-16-XG and Ubuntu Linux 18.04

A while ago, I got a Ubiquiti Unifi US-16-XG Switch, which is a 10 Gigabit Ethernet switch with SFP+ and some RJ45 ports. I use Intel X520 PCI Express cards with SFP+ transceivers or DAC cables. That's all good if I have a PC that actually has a free PCIe x4 slot, or an x1 slot that is open at the end to fit a larger card (and take the speed hit due to the lower speed of an x1 port).

But in my case, I had a mainboard that only had a PCI Express 2.0 x1 slot that was closed, and I needed more than a Gigabit connectivity. One option is to use a card with 2 Gigabit ports and use bonding/trunking/link aggregation, which is fully supported and both my Switch and Linux support LACP/IEEE 802.3ad. But the problem is that even though that would give me a 2 GBit/s total network connection, a single TCP connection cannot exceed 1 GBit/s. Bonding is a way to do load balancing across multiple ports to increase the total bandwidth, but e.g, transferring a file over SMB/CIFS is still limited by the speed of a single port, and that's 1 GBit/s.

Now, there is a relatively new Ethernet standard, previously called NBASE-T, then standardized in IEEE 802.3bz and resulting in 2.5GBASE-T and 5GBASE-T speeds - that's 2.5 GBit/s and 5 GBit/s. That turned out to be the winner, because while no one makes a 10 GBit/s NIC in x1 size (which wouldn't make sense anyway since 10 GBit/s needs at least PCIe 3.0 x2), there are 2.5 GBit/s cards for PCIe 2.0 x1.

The Hardware

In my specific case, I bought a TRENDnet TEG-25GECTX for about $50, which is based on the Realtek RTL8125 chipset. There are other cards like that on the market, some for about $30, and they all seem to be the same - but the above is the one that I got, and thus the only one I can talk about with authority.

This solves the PC side of things, but now I need to solve the Switch-side: The US-16-XG can indeed do 2.5 GBit/s - but only on the SFP+ ports, not on the RJ45 ones. Connecting the card to one of the RJ45 Switch ports indeed only negotiated 1 GBit/s - bummer!

So it has to be connected to the SFP+ port, and that means getting a Transceiver. Now, Transceivers exists for 1 GBit/s and 10 GBit/s, not for 2.5 or 5 GBit/s. And as you'd expect, a 1 GBit/s transceiver isn't going to work for 2.5 GBit/s. So I had to buy a 10GBase-T transceiver, which cost another $40 - those things aren't cheap, sadly. I bought a 6COM SFP-10G-T, which was listed as "For Ubiquiti" on Amazon (ASIN B075WR8FHT).

The Switch

With that, I could connect the card to an SFP+ port on my US-16-XG - and got a link error. That is to be expected since the Transceiver is 10 GBit/s, the Switch is 10 GBit/s, and the card can't do 10 GBit/s. The final piece of the puzzle was to configure the Switch to 2.5 GBit FDX instead of autonegotiation.

With that out of the way, everything just started working, and I'm indeed seeing 2.5 GBit/s performance from host-to-host. I'm making a note here: HUGE SUCCESS.

Note that 2.5 GBit/s shows up as light gray instead of White in the Unifi Controller UI.

The Operating System

I am using Ubuntu Server 18.04 LTS, which doesn't have a driver for the Realtek 8125 built-in, and thus the card doesn't show up (this is true as of 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 x86_64, which is the latest as of this post).

I had to install and update using the built-in Gigabit NIC, then download the driver from Realtek (2.5G Ethernet LINUX driver r8125 for kernel up to 4.15). Someone seems to have mirrored it to GitHub, but I haven't tried that and downloaded directly from Realtek.

I had to apt-get install build-essential to get build tools installed, then a simple ./autorun.sh in the extracted driver directory compiled and installed it. After a reboot, the card was found and connected perfectly fine.

lsmod or lspci -nnk shows the driver module as r8125, and it's located in /lib/modules/4.15.0-88-generic/kernel/drivers/net/ethernet/realtek/r8125.ko to match my kernel version. I assume I'll have to recompile it after kernel updates, but as none was released yet, I don't know for sure.

05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. Device [10ec:8125]
        Subsystem: Realtek Semiconductor Co., Ltd. Device [10ec:0123]
        Kernel driver in use: r8125
        Kernel modules: r8125

I'm using netplan, and to make sure I can connect either NIC (but don't stall the bootup process if one isn't connected), I've marked both as optional. I've also match them based on their driver as in my case, both the onboard and 2.5 GBit/s NIC use different drivers, so this works fine and I don't have to worry about device names changing.

Here's my /etc/netplan/01-netcfg.yaml:

network:
  version: 2
  renderer: networkd
  ethernets:
    id0:
      match:
        driver: r8125
      dhcp4: yes
      dhcp6: yes
      optional: yes
    id1:
      match:
        driver: r8169
      dhcp4: yes
      dhcp6: yes
      optional: yes

The Result - and alternatives

The numbers don't lie, and these numbers say that I'm getting ~275 MB/s (out of a theoretical maximum of 312.5 MB/s) from one host to the PC - so yeah, full on success here.

Is it worth spending $80 on such a solution? I'd recommend going with a 10 GBit NIC whenever possible, even if you have to stick it into an x1 slot (assuming the slot is open, so that it physically fits) - it's about the same price, and is much more futureproof.

And even if you're not on 10 GBit/s and want to go faster than 1 GBit/s, you can get some really well priced Microtik switches, like the $130 CRS305-1G-4S+IN. And you don't need expensive Fiber transceivers, because for short connections (up to 7 Meters) there are DAC cables, which are copper cables with SFP+ connections.

There is financial benefit from staying with a copper-based solution (and you might already have all the cabling in place), and 2.5 GBit is cheaper than 10 GBit for sure. Besides, 250 MB/s is plenty fast for most home networks. But that said, I'd still go for 10 GBit/s, which can be done over regular Ethernet cables assuming they are high enough quality.

The 2.5 GBit/s solution helped me here for one machine that didn't have a slot suitable for a 10 GBit/s card, but that's pretty much the only reason I'd go with that speed.