For a start, lets note that this article deals solely with measuring OpenVPN performance, as to be honest this is the only VPN protocol we really care about, as the others are rubbish (well SSTP is ok, but it is limited to Windows computers only, is poorly supported by VPN providers, and offers nothing that OpenVPN can’t do anyway). Please see here for more details.
When testing VPN connection speeds, most people turn to the web’s most prominent internet speed testing website, speedtest.net. Unfortunately, although speedtest.net is a very good indicator of naked internet speed, it can yield some very odd results when measuring VPN speeds, often indicating speeds much faster than not just the naked internet results, but than the cap put on a service by the ISP .
The reason for this is that the Flash based speedtest.net tool does not account for LZO compression, which is built into the OpenVPN protocol. We will discuss this in more detail below, but what it does show is that we need a more accurate way to measure data transfer speeds over OpenVPN.
Factors that influence connection speeds
ISP speed is only between you and ISP server
When ISPs offer a set broadband speed, this only applies to the connection between your computer and the ISP’s nearest server, after which data transfer between ISPs and countries is a free-for-all that no ISP has any control over.
This is not a fact that ISPs like to boast about, but is a necessary consequence of how the internet works. More than anything else, this factor can lead to highly inconsistent results, but there is little that can be done about it other than performing many tests, and averaging the findings.
OpenVPN has built into it the optional ability to use the LZO lossless compression library. Much like the better known .zip format, this compresses the size of some files types, which can indeed increase data throughput, but does not count against your ‘real’ bandwidth usage. Files that are already compressed, such as .zip, .rar, and .mp3, and .jpg files, do not benefit much from additional compression. As we noted earlier, speedtest.net is easily confused by the use of LZO compression.
TCP and UDP
OpenVPN can be run over either the TCP (Transmission Control Protocol) or UDP (User Datagram Protocol) transports. TCP uses error correction to verify that a data packet has arrived before resending the packet (if no confirmation is received), or sending the next packet (if confirmation is received). UDP on the other hand doesn’t care, and keeps on sending packets regardless of whether they have arrived or not.
UDP is consequently faster (but less reliable) than TCP. See here for more details.
Also known as traffic shaping or bandwidth limiting, most ISP do this to some extent (and in the US the whole notion of net neutrality, where all internet traffic is treated equally, has been effectively trashed – at least for now).
BitTorrent traffic is regularly throttled, but subtler shaping due to peak usage times, and other factors is common. If you want see how much throttling your ISP is responsible for, check out M-Labs transparency measurement tool.
Our UK ISP is Virgin…
Programs that hog bandwidth
When trying to perform an accurate speed test, it is important to check that you are not running background programs and processes that are hogging your bandwidth and slowing down your connection.
This is particularly a factor with uploads, as while most ISPs are fairly generous with download limits, upload bandwidth is typically much more limited. As a rule of the thumb, using more than 1/4 of your upload speed will noticeably slow down your download speeds.
BitTorrent clients are particularly notable culprits for hogging upload bandwidth, but you should also keep an eye on online storage services such as Dropbox and Google Drive (and pause them during bandwidth speed testing).
VPN always comes with a performance hit thanks to the fact that data often has to travel further between your computer and the internet, as it has to go via a VPN server. This problem is made worse the greater the distance you are from the VPN server, so you should always test using the VPN server geographically closest to you.
In addition to this, the encryption and decryption process takes up processing power on your computer, which may in practical terms affect speed (although it should not affect these tests). This is not a major problem on most modern desktop/laptop computers, but may affect low-end smart phones and the like. It is also worth remembering that the more encryption used, the more processing power is required.
More importantly, handling VPN traffic tasks a VPN server, and insufficient processing power at the VPN server end of things can significantly slow down connection speeds.
In theory, when connected to a nearby VPN server using OpenVPN, we should typically expect to see around a 10 percent loss of speed compared to not using a VPN (this at least is a commonly quoted figure, and is bourn out by our testing results (see this article). These results however, have mainly been obtained using speedtest.net, so as this article discusses, may not be as accurate as we would like)
VPN speed testing methods
Tests were performed using a 20Mb/s UK broadband connection, and for VPN using TorGuard. Our methodology was to run three tests for each test method without VPN, and then three with VPN , in quick succession so that factors beyond our control (the speed of the internet) were as consistent as possible.
Ideally, we would like to have tested the different methods as simultaneously as possible, but real-world considerations mean that our results were obtained over a couple of days. The idea was to do some measurements, crunch some figures, and see what conclusions we could draw from the results.
Here the first (lower) three results are without VPN, followed by results when connected to TorGuard. On average download speeds were:
- no VPN = 20.1Mb/s
- with VPN = 22.9Mb/s,
- VPN = 12.1% faster.
On average upload speeds were:
- no VPN = 1.3Mb/s
- with VPN =1.1Mb/s
- no VPN = 15.4% faster.
Here we can see clearly see the problem with speedtest.net, as the download results are notably higher than both the no VPN results, and our ISP imposed speed cap, which is simply not possible were the results accurate. LZO compression is almost certainly to blame for this.
Testmy.net uses the more robust HTML5 for its speed tests, and claims to compensate for compression. Unfortunately, the closest test server to our UK location is in the Netherlands, so some slowing down due to lag is to be expected.
With UK VPN
As the Testmy.net test server was in the Netherlands, we also thought we would see how using an NL VPN server affected the results.
Average download speeds were:
- No VPN = 16.4Mb/s
- with UK VPN – 16.4Mb/s
- with NL VPN = 13.7Mb/s
- UK VPN = same as no VPN, while NL VPN = 16.5% slower than no VPN
Average upload speeds were:
- No VPN = 1.1Mb/s
- with UK VPN = 1.09Mb/s
- with NL VPN = 1.06Mb/s
- UK VPN = 1% slower than no VPN, while NL VPN = 4% slower than no VPN
It has to be said the results are rather inconsistent, although the fact that the test server was so far away from us may well have contributed towards both these issues. On average, the NL VPN result was actually 0.01Mb/s faster than not using a VPN, but this is a minor discrepancy, easily explained by the chaotically variable nature of the internet, and by the inconsistency of our results.
What is clear is that we achieved better speeds when connected to our closest VPN server, rather than one close to the test server.
Another HTML5 web-based speed test, Speedof.me automatically selects the test server based on stability, rather than distance. It chose London for these tests, which is the closest anyway, and should give us the most accurate idea of real internet speeds.
Average download speeds were:
- No VPN = 18Mb/s
- With VPN = 8Mbs/s
- VPN = 65.6% slower than no VPN
Average upload speeds were:
- No VPN = 1.2Mb/s
- With VPN = 1.9Mb/s
- VPN = 37% faster than no VPN
Other than TorGuard having a terrible day for download speeds, the overall results show some of the best consistency we have seen so far. An exception is the highly anomalous 3.25Mb/s upload result, which approaches three times the speed of every other upload test so far, and gives us pause when considering the overall accuracy of the speedof.me tests.
Unlike the previous tests, ttbMeter from the UK’s thinkbroadband service (‘an independent broadband news and information site helping you to get the most out of your broadband) is a downloadable tool with a ton of configuration options and tests available. Unfortunately, many of the tools did not seem to work, so we opted for a straight download test using a 100Mb file.
Average download speeds were:
- No VPN = 16.2
- With VPN = 17.4
- VPN = 7% faster than no VPN
As a straight timing test for a 100MB file, these results are highly consistent. Again, the very slight discrepancy in speeds being faster with VPN than without can be explained by variable factors. On a practical level, however, we found tbbMeter frustrating to use, as it often appeared not to work, and only displayed the results briefly as a notification bar bubble.
When it comes to testing internet speeds there is nothing wrong with going back to basics. This series of tests simply involved downloading a test file, and using a stopwatch to time how long the download took.
When performing such tests it is important to use compressed files to compensate for LZO compression. We u
Without VPN using a 100MB test zip file, courtesy of thinkbroadband (100MB = 800Mb)
- Result 1: 800Mb / 50.92 seconds = 15.7Mb/s
- Result 2: 800Mb / 71.32 seconds = 11.2Mb/s
- Result 3: 800Mb / 76.35 seconds = 10.5Mb/s
- Average = 12.5Mb/s
- Result 1: 800Mb / 80.04 seconds = 10Mb/s
- Result 2: 800Mb / 78.33 seconds = 10.2Mb/s
- Result 3: 800Mb / 91.16 seconds = 8.8Mb/s
- Average = 9.7Mb/s
So no VPN is 22.4% faster than with VPN.
We also uploaded a 5MB (40Mb) file to Dropbox:
- Upload without VPN: 1.08Mb/s (36.9 seconds), 1.09Mb/s (36.55 seconds), 1.08Mbs (37.09 seconds). Average = 1.08Mb/s
- Upload with VPN: 0.96Mb/s (41.85 seconds), 1.09Mb/s (36.55 seconds), 0.77Mb/s (51.66 seconds. Average = 0.95
So no VPN is 12.2% faster than with VPN.
Although it lacks a certain precision (hitting the stopwatch and download start buttons at exactly the same time is tricky, and uploading to Dropbox is not the most clinical method), this test does give a very ‘real world’ measurement of internet connection speeds, and how VPN affects them. Interestingly, it generally gives us much slower download times than the automated tests.
So taking all these figures together (except the speedtest.net one which is definitely out) , we find that:
- Average download speed without VPN = 15.8Mb/s
- Average download speed with VPN = 11.3Mb/s
- Average upload speed without VPN = 1.1Mb/s
- Average upload speed with VPN = 1.3Mb/s
- On average downloads are 28.5% slower using VPN
- On average uploads are 7% faster using VPN (clearly an anomalous result)
The biggest take-away from these tests is how inconsistent the results are, not just between different test methods, but when performing identical tests back-to-back with each other. It therefore has to be said that any published speed test results (including those on the BestVPN website) should be taken with a big pinch of salt.
This is not to say they cannot be useful for providing a general guide to VPN performance, but only after performing a large number of tests over a period of time can any very accurate picture be gathered. Other things we observed during the tests include:
- TorGuard’s results were rather lackluster (other providers should hopefully yield better results)
- For best speed results you should connect to a server near you (even when connecting to a test server further away)
- The manual approach probably gives the best results, but is too impractical for daily use
Because of its relative consistency and ease of use, BestVPN will use Testmy.net from now on, but please be aware of the inherent limitations of such tests.