NAT performance on EdgeRouter X and Lite with EdgeOS, OpenBSD, and OpenWRT

TL;DR

This one has tables so just scan to them.

a time-save gone wrong

This entry has been updated after performance numbers for ER-X EdgeOS were called out for being wrong. Upon inspection, they were. I had transposed some columns. So I re-ran some test. Wil - 2018/12/27

After posting to /r/openbsd a couple of times about the EdgeRouter Lite that I had been messing with as my jump box, a few people asked about network performance. I wasn’t too worried about it as my jump box was not being used to route traffic or run NATing. Just a shell relay and looking glass that I could trust.

Then I started redoing my whole network with wireguard. Four sites with one behind NAT, a roaming travel router, and two roaming laptops. There’s a whole post about wireguard next. That shit is wicked. Anyway, putting that all together required manually adding the wireguard.deb to the EdgeRouters. And when I upgraded their firmwares the config and .deb would be wiped out again. So I thought about tossing OpenWRT on a EdgeRouter X and testing out wireguard on that. Being pretty familiar with OpenWRT buildroot and building linux distros in general, I figured having my own bespoke firmware image that could be easily flashed and didn’t loose VPN config would save me time.

After building the firmware of course.

So the tradeoff basically amounted to a push I guess.

And then, this guy, on a thread long dead, asks about performance again. And since I was already flashing and fucking around with everything, I just had to get back to him for all of those fake internet points…

alternatives

I figured I would throw the EdgeOS 1.10.6 against OpenWRT and OpenBSD 6.3 (where applicable) and even toss in the EdgeOS 2.0 alpha 3 into the mix. OpenWRT would be 18.06.1 for the ERL because I got a bit lazy after starting testing on the ERX, which was running whatever was current in git when I built my images. Which, honestly isn’t that far off from the 18.06.1 release.

a simple test

iperf client (mbp) -> Juniper EX2200C -> Test Router -> iperf server (Debian vm)

The test router had the simplest rule sets applied possible. Basically just NAT out to the Debian test machine. You can see the OpenBSD pf config here.

iperf3 was used with tests 30 seconds long. I ran three tests per configuration, did some quick math. Seriously, there wasn’t a hell of a lot of rigor used here. If I find time, I might do a scatter plot. But the results kinda speak for themselves.

And it’s a pretty simple synthetic test being run here, but what do you expect for free?

EdgeRouter Lite

There were no surprises here. Honestly. There’s no open access to the Cavium hardware acceleration, so OpenBSD and OpenWRT can not push the performance that EdgeOS does. But OpenWRT does make a hell of a go at it with it’s software acceleration enabled.

It’s worth mentioning that if you are sitting on a 150Mbit connection, any of these OSes will be able to saturate that pipe. But if you’re on something faster, OpenBSD can not compete at NAT throughput on this hardware. That’s not a knock against it necessarily. End the OS holy wars.

It’s just a question of what else do you need your router to do? Yes, EdgeOS has access to Debian packages. I just prefer to have a bastion with a trusted environment where I’m happy and know where all my tools are. I’m not going to use EdgeOS as a jump box, and I’m not going to route traffic through OpenBSD on an ERL.

EdgeRouter X

On the EdgeRouter X the story was way different. My jaw literally dropped. Not only did OpenWRT put out close to twice the performance of EdgeOS on it’s own hardware, OpenWRT on the ERX outperformed OpenWRT on the ERL.

Going back over my notes, and then doing some more testing, things are more equal but still somewhat surprising considering the ERL results. OpenWRT pulls ahead with hardware acceleration but doesn’t necessarily dominate. It’ll be interesting to see how the EdgeOS 2 release looks once everything is finished baking.

The people that committed the MT7621 Hardware Acceleration code into OpenWRT, I want to buy you all pints. Not only does the OpenWRT give you access to a more modern kernel, customizability and buildroot, but it’s faster than the best firmware that Ubiquiti is shipping. just as solid as the commercial product put out by Ubiquiti.

And you don’t have to reinstall and reconfigure wireguard with every firmware upgrade you choose to do. Which was kinda why I started going down this path in the first place. I can finally justify all that time now. OpenWRT on ERX is worth it, not just for the wireguard, but for the performance.

Which I probably won’t ever benefit from because I don’t have WAN gig links anywhere on the network. So… yeah.