Benchmarking ZeroTier vs. OpenVPN and Linux IPSec
It’s been a while since we published any performance numbers, so today we decided to benchmark the pre-release of ZeroTier 1.2.4 against IPSec and OpenVPN.
Our benchmark setup consisted of two single-core Linux (CentOS 7) virtual machines running on VMWare Workstation on the same Core i7 at 2.8ghz. Benchmarking on the same physical host means that we’re only measuring the CPU-constrained impact of each tested virtual network stack. Since there is no actual physical network there are no other factors. By assigning each virtual machine a single core we ensure that they do not compete with one another. (The host CPU has four physical cores.)
Testing was performed using iperf3 in TCP mode transferring a gigabyte of random data. Random payload prevents data compression from impacting transfer speed, though the sender’s attempt at compression (if enabled) still contributes to CPU overhead.
| Software | Encryption / Compression | Speed |
|---|---|---|
| Nothing (VMWare bridge) | — | 4760 mbps |
| IPSec / Linux 3.10.0 / libreswan 3.15 | AES-128-CBC / None | 497 mbps |
| ZeroTier 1.2.3 (pre-1.2.4) | Salsa20 / LZ4 (default) | 484 mbps |
| OpenVPN 2.4.1 | AES-256-CBC / None | 309 mbps |
| OpenVPN 2.4.1 | AES-256-CBC / LZO | 290 mbps |
| OpenVPN 2.4.1 | Blowfish-CBC / None | 234 mbps |
| OpenVPN 2.4.1 | Blowfish-CBC / LZO | 221 mbps |
We didn’t expect to beat OpenVPN by such a margin, and we expected IPSec to be at least 10% faster. IPSec’s main encapsulation path lives in the kernel, avoiding two kernel/user mode context switches and at least two rounds of memory copying. It also makes use of CPU AES-NI instructions for encryption. Despite these factors ZeroTier clocked nearly identical transfer speeds. We repeated the test several times and with slightly different iperf3 modes and flags and got the same or similar results.
These results tell us ZeroTier’s encryption and encapsulation path must be faster than IPSec by enough of a margin to compensate for the cost of kernel/user mode context switching and additional memory copying. Either that or the two are equivalent and we’re over-estimating kernel/user mode costs. IPSec turns out to be a little under 3% faster, so maybe that’s the overhead of not living in the kernel.
This also means ZeroTier would likely beat IPSec by 5-15% if we ported it to the kernel. We have no plans to do so in the immediate future, but if our users start demanding higher performance we have at least one path forward.
Needless to say we are very happy with these numbers! Our performance is almost identical to IPSec, which is the standard for “enterprise” network tunnels.