WireGuard vs. IPSec vs. OpenVPN: VPN performance on Ubiquiti EdgeRouter Lite and X.
Wil Knoll · 2018 Dec 26, 01:40 · 802 words · 4 minutes read
TL;DR
If you’re looking for a modern performant VPN with few configuration hassles then invest your time on WireGuard. Performance on ERX with OpenWRT is very compelling for cheap hardware.
wireguard. man.
It’s simple.
WireGuard is a connection-less VPN. It ships with a suite of protocols and primitives defined, so none of that has to be negotiated and avoids downgrade attacks. Like SSH, each endpoint generates a key pair. Any packet sent to the endpoint is encrypted with the public key, decrypted with the private. All two parties need to know to set up a VPN is the public keys of the other endpoint, what interior address they will be using, and only one of the parties needs to know the exterior IP of the other.
You can build the configs in seconds. The tunnels come up in a heartbeat. The tunnels are as fast if not faster than IPSec, hardware being equal. It’s a modern take on how VPNs should be built, implemented in less than 4k lines of code, and I’m ripping out OpenVPN and IPSec everywhere for this.
And since that was all happening just after I had set up a lab for other performance testing on a bunch of Ubiquiti gear, I figured we might as well knock out some WireGuard tests.
it’s not a dining table, it’s a workbench
The lab at home only needed a few tweaks to support doing end to end testing. Two old MacBook Pro’s had their gig ports wired into a pair of EdgeRouter Lites or Xes, and a port on each of them plugged into that same Juniper EX2200 from last time. Each EdgeRouter NATed out to the shared segment in the EX2200. Network was tested out for packet loss and found to be running clean.
MBP 1 -> EdgeRouter 2 -> Juniper EX2200 -> EdgeRouter 2 -> MBP 2
iperf3 with 30 second tests were run 3 times. Min/Max/Average calculated from those results.
I followed Ubiquiti’s documentation for OpenVPN and IPSec Site to Site VPNs, so if you want to know where the dials were set it’s all right there. As for WireGuard, thankfully you don’t need to do any of that. Barebone configs.
EdgeOS used was 1.10.7. WireGuard for EdgeOS was 0.0.20181007. OpenWRT was whatever was used on the last post… r8053-9926f7cf29. I sat on the results for a few months because other things were more pressing.
EdgeRouter Lite
| Firmware | EdgeOS | EdgeOS | EdgeOS | EdgeOS | OpenWRT |
|---|---|---|---|---|---|
| VPN | IPSec | IPSec | OpenVPN | WireGuard | WireGuard |
| Hardware Accel | No | Yes | No | No | No |
| Max Mbit/sec | 36.1 | 66.1 | 17.8 | 66 | 177 |
| Mean Mbit/sec | 34.98 | 61.97 | 11.12 | 62.03 | 136.29 |
| Min Mbit/sec | 29.9 | 57.2 | 8.05 | 57.7 | 126 |
It’s no contest. If your use case requires you to have the highest throughput you can with a couple of ERLs, and you own those boxes, it’s worth it to get OpenWRT with WireGuard on those guys. This must have something to do with the age of the kernel thats running in EdgeOS. It’ll be interesting to see what happens with the EdgeOS 2 firmware inbound.
EdgeRouter X
| Firmware | EdgeOS | EdgeOS | EdgeOS | EdgeOS | OpenWRT |
|---|---|---|---|---|---|
| VPN | IPSec | IPSec | OpenVPN | WireGuard | WireGuard |
| Hardware Accel | No | Yes | No | No | No |
| Max Mbit/sec | 40 | 151 | 29.9 | 158 | 279 |
| Mean Mbit/sec | 39 | 144.48 | 24.95 | 152.68 | 160.23 |
| Min Mbit/sec | 35.3 | 118 | 19.6 | 119 | 145 |
We could basically copy/past the results from the ERL testing here. The combination of OpenWRT and WireGuard really delivers bang for the buck on this hardware.
But the big head turner is the ER-X appears to outperform the ERLite-3 in these tests. When not hardware accelerated, it looks like it’s the extra CPU torque, 880 MHz dual-core on the ER-X, vs the 500 MHz dual-core on the ERLite-3 that makes the difference.
I don’t know if WireGuard is multi-threaded on either architecture or firmware, but it does appear to be CPU bound here and I wished I had the foresight to take some performance metrics from onboard the devices while doing this testing.
And if your WAN link is only 100 Meg, this isn’t going to matter. Your best bet to saturate it with VPN traffic is WireGuard. If you have lots of firewalling and routing happening in the LAN you’ll probably still want the ERLite-3 for packet routing throughput however.
so worth it
I don’t know. Maybe I’m fixated on this for no good reason. I just wasn’t satisfied knowing there was this gorgeous code that could run so much better on this hardware, it just required some extra work up front. But again, all of my endpoints have 15Meg up so until that changes…
The iOS client was released just before Christmas. I couldn’t have wished for a better gift for the network nerd in me.
Cheers to Jason A. Donenfeld et al. for all the hard work on this.