NAT performance on EdgeRouter X and Lite with EdgeOS, OpenBSD, and OpenWRT

Sep 27, 2018 02:33 · 832 words · 4 minutes read OpenBSD EdgeOS OpenWRT EdgeRouter Lite EdgeRouter X

TL;DR

This one has tables so just scan to them.

a time-save gone wrong

After posting to /r/openbsd a couple of times about the EdgeRouter Lite that I had been messing with as my jump box, a few people asked about network performance. I wasn’t too worried about it as my jump box was not being used to route traffic or run NATing. Just a shell relay and looking glass that I could trust.

Then I started redoing my whole network with wireguard. Four sites with one behind NAT, a roaming travel router, and two roaming laptops. There’s a whole post about wireguard next. That shit is wicked. Anyway, putting that all together required manually adding the wireguard.deb to the EdgeRouters. And when I upgraded their firmwares the config and .deb would be wiped out again. So I thought about tossing OpenWRT on a EdgeRouter X and testing out wireguard on that. Being pretty familiar with OpenWRT buildroot and building linux distros in general, I figured having my own bespoke firmware image that could be easily flashed and didn’t loose VPN config would save me time.

After building the firmware of course.

So the tradeoff basically amounted to a push I guess.

And then, this guy, on a thread long dead, asks about performance again. And since I was already flashing and fucking around with everything, I just had to get back to him for all of those fake internet points…

alternatives

I figured I would throw the EdgeOS 1.10.6 against OpenWRT and OpenBSD 6.3 (where applicable) and even toss in the EdgeOS 2.0 alpha 3 into the mix. OpenWRT would be 18.06.1 for the ERL because I got a bit lazy after starting testing on the ERX, which was running whatever was current in git when I built my images. Which, honestly isn’t that far off from the 18.06.1 release.

a simple test

iperf client (mbp) -> Juniper EX2200C -> Test Router -> iperf server (Debian vm)

The test router had the simplest rule sets applied possible. Basically just NAT out to the Debian test machine. You can see the OpenBSD pf config here.

iperf3 was used with tests 30 seconds long. I ran three tests per configuration, did some quick math. Seriously, there wasn’t a hell of a lot of rigor used here. If I find time, I might do a scatter plot. But the results kinda speak for themselves.

And it’s a pretty simple synthetic test being run here, but what do you expect for free?

EdgeRouter Lite

OS EdgeOS 1.10.6 EdgeOS 1.10.6 EdgeOS 2a3 EdgeOS 2a3 OpenBSD 6.2 OpenWRT 18.06.1 OpenWRT 18.06.1
Acceleration Hardware No Hardware No No Software No
Peak Mbits/sec 940 94.9 940 50.1 189 661 235
Lowest Mbits/sec 904 88.3 814 33.4 179 640 219
Average Mbits/sec 936 90.7 927 39.3 184 657 230

There were no surprises here. Honestly. There’s no open access to the Cavium hardware acceleration, so OpenBSD and OpenWRT can not push the performance that EdgeOS does. But OpenWRT does make a hell of a go at it with it’s software acceleration enabled.

It’s worth mentioning that if you are sitting on a 150Mbit connection, any of these OSes will be able to saturate that pipe. But if you’re on something faster, OpenBSD can not compete at NAT throughput on this hardware. That’s not a knock against it necessarily. End the OS holy wars.

It’s just a question of what else do you need your router to do? Yes, EdgeOS has access to Debian packages. I just prefer to have a bastion with a trusted environment where I’m happy and know where all my tools are. I’m not going to use EdgeOS as a jump box, and I’m not going to route traffic through OpenBSD on an ERL.

EdgeRouter X

OS EdgeOS 1.10.6 EdgeOS 1.10.6 EdgeOS 2a3 EdgeOS 2a3 OpenWRT r8053-9926f7cf29 OpenWRT r8053-9926f7cf29
Acceleration Hardware No Hardware No Hardware No
Peak Mbits/sec 561 265 195 151 869 465
Lowest Mbits/sec 420 249 177 113 820 296
Average Mbits/sec 475 260 188 144 846 429

On the EdgeRouter X the story was way different. My jaw literally dropped. Not only did OpenWRT put out close to twice the performance of EdgeOS on it’s own hardware, OpenWRT on the ERX outperformed OpenWRT on the ERL. The people that committed the MT7621 SQM code into OpenWRT, I want to buy you all pints. Not only does the OpenWRT give you access to a more modern kernel, customizability and buildroot, but it’s faster than the best firmware that Ubiquiti is shipping.

And you don’t have to reinstall and reconfigure wireguard with every firmware upgrade you choose to do. Which was kinda why I started going down this path in the first place. I can finally justify all that time now. OpenWRT on ERX is worth it, not just for the wireguard, but for the performance.

Which I probably won’t ever benefit from because I don’t have WAN gig links anywhere on the network. So… yeah.