NEWS

Why we’re changing our VPN Review speed tests

Accurately testing the speed of an Internet connection is notoriously difficult. Lots of things can get in the way of a good test. For starters, there’s the very nature of the Internet as a series of interconnected computer systems to consider. When your connection to a speed test site has to take a route that goes through many different networks, anything which affects any of those networks will affect your speed test (though, the reality for most of us is that our Internet traffic takes place over backbones that are pretty reliable). Your Internet Service Provider itself might not be that reliable, and cause speeds to fluctuate from one moment to the next. Or your neighbours may choose the moment of your test to start downloading the entirety of Game of Thrones in 4K HD.

At BestVPN.com, we’ve obviously been aware of this issue for a while, and have always said the speed tests in our VPN reviews should be taken with a grain of salt. While that’s still true, we’re changing how we do our speed tests, and how we present the results.

Previously we’ve tended to run a few tests for each VPN server on TestMy.net and choose one of them as more or less representative of the speeds readers can expect while connected to that server. Then, we’ve taken a screenshot of the result and posted in the article. Typically we’ve run tests on several servers but only posted results for a few, as posting lots of pictures of numbers isn’t that pretty or interesting.

 

Our old way of posting speed test results. Boring, boring, boring.
Our old way of posting speed test results. Boring, boring, boring.

 

Instead, what we’re doing now is aggregating the data from multiple tests of multiple servers into graph form. This means running multiple tests at different times and on different days, so that any unusual circumstances will be less able to affect our impression of a VPN, and readers can get a better idea of the overall performance they should be able to expect.

Because we do this for our control connection too – that is, the tests conducted with no VPN server involved– VPN speed tests will also be less affected by any wobbles in our own base connection; it’ll be really easy to see if a VPN is really playing up, or if actually our tester’s ISP was just having a bad week.

The Graphs

The new graphs look pretty and show a lot of information at once, but they may require some explanation. Let’s walk through them using an imaginary set of results for a made-up VPN with servers in London, Amsterdam, Washington DC and Dallas, TX.

Each column on the graph represents a set of speed tests from one VPN server to one test server. Control tests – that is, tests not connected to a VPN server – are in green, and VPN server tests are in blue. In the example below, the first set of four columns are tests conducted to the UK speed test server, and the last three are conducted to the US speed test server. You’ll see both regional tests include their own green column, i.e., their own control test. The control test is always the baseline to compare the other tests to as it shows what speeds are like to that region with no VPN involved. Here’s an example of some download tests.

 

Our new speed graphs. Exciting!
Our new speed graphs. Exciting!

 

The top of each column shows the highest speed we achieved during our testing. Obviously higher columns are therefore better. In the example below, look at the first green column on the left. It’s a no VPN (i.e., control) speed test to a UK test server. You can see that the highest speed to the UK test server was nearly 35Mbps.

download_example_annotated

The bottom of each column, as you might expect, shows the lowest speed we had during testing. The higher the bottom is, the higher the lowest speed was, and thus the better the VPN’s performance. In the example, the lowest speed was around 28Mbps. Finally, somewhere between the high and low is a line where the column changes colour. This line indicates the average speed over all of our tests of that server. In the example, the average speed happens to be pretty much exactly halfway between the highest and lowest at around 31Mbps, though this is not always the case.

The difference between the lowest and highest speeds is clearly shown by the overall size of the coloured column. A small column is actually very good, because it means speeds were consistent and didn’t have much variation. A very long column is not good, as it means that speeds were all over the place during testing and reliability might therefore suffer. You can see that in the example, our test connection is just a little bit wobbly, but not too bad: the variance is only around 7Mbps.

The precise figures on the graphs don’t matter so much as how they compare to each other. A good VPN will have a column on the graph that looks very similar to the green control column. This makes it a little easier to compare VPNs which may have been tested on differing speeds or qualities of test connection.

Analysis

To analyse our sample picture then, you can see we did six sets of tests overall. The first three were to a UK speed test server, testing first our control (no VPN) connection in green, then a VPN server in London, and then one in Amsterdam. Overall, the Amsterdam server actually performed slightly better than the London server, even though the speed test server was itself located in the UK.

The last three were tests to a US speed test server. Again our control (no VPN) connection comes first, in green. Then we did tests connected to a VPN server in Washington DC, and a VPN server in Dallas, Texas. The Washington VPN server has a high spike, but averages out quite a bit below our control connection, while the Dallas VPN server is quite a bit worse still.

Now here’s the upload tests.

upload_example

Not many surprises here – they’re very similar to the download tests. You can see, though, that the quality of our test connection for uploads is a bit variable – there’s much more difference between the high and low marks on the graph. What’s more, the London server performs significantly poorer for uploading than the Amsterdam server, even though it was about the same for downloads.

Overall, the graphs show that this would be a pretty good VPN: speeds are fairly consistent, they’re often not much lower than our basic no VPN connection, and there’s not that much difference from server to server.

Conclusion

Our new speed testing procedure will not only be more thorough – more tests on more servers – it’ll also present the results in a way which makes it much easier to see at a glance what’s going on, and somewhat more viable to compare different VPNs and VPN servers. What’s more, because we’re testing more servers at more times, the results will be fairer and more accurate as well as easier to see. We hope you enjoy the new speed tests!


Charles Tosh I'm a freelance writer and English teacher who loves travel, technology and mushy movies.

Related Coverage

More

3 responses to “Why we’re changing our VPN Review speed tests

  1. EXCELLENT. That gives much more information about what can be expected from any given VPN. I have noticed in the past, that my own speed tests were variable—sometimes significantly—and often, didn’t seem to match your results very closely. I look forward to your new results. As the Australians say, “Good on ya”.

    Noe

Leave a Reply

Your email address will not be published. Required fields are marked *