GUIDE

Custom Router Firmware: In-Depth Review

We recently did a ‘Custom Router Firmware Overview‘ but weren’t able to go beyond the surface as we didn’t have a router that would be able to run all the types of firmware. We have now been lucky enough to get our hands on the popular Asus RT-N66U courtesy of our friends Sabai Technology.

We have decided to not include OpenWRT as it is above most people’s scope and instead we have decided to include the stock firmware as a base for the comparison.

Index

  1. Quick Glance
  2. Installation
    1. Firmware
    2. Extras
  3. Pricing
  4. Features
    1. Aesthetics
    2. Navigation
  5. Speeds
    1. WiFi Speeds
    2. Speed optimisation
  6. VPN
    1. PPTP Setup
    2. OpenVPN Setup
    3. VPN Speeds
    4. Additional VPN Information
  7. Support
  8. Conclusion

1. Quick Glance

The main reason that people install custom firmware on their routers is that it provides extra services that aren’t available on the stock configuration or because they wish to optimise one specific area of operation. Here is a very quick glance at the types of firmware we’re going to be looking at. If you are interested only in a basic comparison then you can also view the article mentioned at the beginning of this post.

Stock Firmware This is the stock software that routers come with and vary by make and model. The Asus N66U has a lot of advanced features meaning that custom firmware don’t provide as much of an add on as  with lower end routers.
DD-WRT The most common custom firmware by far due to it’s extensive hardware support and extensive community it has. The firmware allows many add-ons that you can utilise and gives a lot of flexibility for whatever you wish to use it for. There are also a number of different sized builds that again allow for greater flexibility and hardware support. Pre-flashed routers can be purchased from FlashRouters or some VPN providers also sell them.

You can also read more about DD-WRT at our Definitive DD-WRT Guide

Tomato Tomato has many of the features that are present in DD-WRT but it is limited in hardware support (you are only able to install it on Broadcom routers). Tomato has the advantage of not requiring a reboot after most changes – which is a common problem with other firmwares – as well as generally being better for performance tracking and VPN setup. Though Tomato has a number of different branches we will be using TomatoUSB/Shibby as it is (in our opinion) the most rounded.
Sabai OS Sabai OS is an advanced Tomato build developed by Sabai Technology and is aimed at optimising VPN set-up and speeds. They also have some exclusive features such as Gateways which allows you to determine which device connected to your router uses the VPN and which doesn’t. Though open source it has it’s dedicated support team though this comes with an extra cost.

Unfortunately, Sabai doesn’t support the new AC standard yet, but this shouldn’t cause a problem for most people.

2. Installation

Under installation we will consider two different areas; how easy it is to install the firmware and how easy it is to install extras to the firmware so that you may use your router for even more.

2.1 Firmware

Installation will depend router to router but the process is generally the same and requires you to first clear the NVRAM memory and then use the upgrade option (using the firmware’s GUI) to load your desired new/upgrade firmware. With the N66U the processes is similar however since it is a fairly advanced model it has a special reset menu making the process independent of the firmware you are installing and thereby making each as easy as the other. We would however recommend Google-ing the process for your router before you begin as it can vary to some extent. The time required to carry out the firmware installation is 15 minutes but we recommend setting aside at least an hour as you will need to find the firmware and at least set the basic settings after installation.

As the table below shows the firmwares can vary greatly in size. We must note that both DD-WRT and Tomato come in sizes as small as 2MB so it can be installed on older/ less powerful routers. The firmware size can be important as the more space it takes up the less space you have to install additional code and add ons to your router. Fully using up the space on your router can also limit its speed. Therefore it’s good to see that all the custom firmwares are considerably smaller compared to the stock firmware even though they have more functionality.

 Firmware Where to find Size (for N66U)
Stock Manufacturer Website  25.1 MB
DD-WRT DD-WRT Supported Routers  7.46MB (mega)
Tomato (Shibby) Tomato by Shibby Router List  11.7MB (AIO – All in One)
Sabai OS Sabai GitHub or Sabai Website  6.45MB

2.2 Extras

With all par the stock firmware it is possible to use PuTTy (or similar) to install add-ons to your router. We gave this a test by installing Transmission (a torrent client) on all three of the firmwares. It was easy to do on all of them however for Sabai you have to change the router IP address to 192.168.1.1 for it to work (or the transmission config files but this is slightly harder). The guides we used were here for DD-WRT and here for Tomato/Sabai.

If you plan on playing around a lot with your router then it is highly recommended that you learn some basic Linux commands as this will help you understand what is going on exactly.

3. Pricing

Theoretically all the firmwares are free however it is possible to buy pre-flashed routers in case you don’t have a router you wish to use. The prices displayed here are specific to the Asus RT-N66U but they fair similarly for different routers.

Stock Firmware This is the cheapest as there are no extra steps to carry out. You can either buy directly from the manufacturer or search the internet for the cheapest prices. Currently $179.99 on Amazon.
DD-WRT You can install it yourself for buy pre flashed through FlashRouters for $249.95
Tomato You can install it yourself for buy pre flashed through FlashRouters for $249.95
Sabai OS You can install it yourself or buy through Sabai Technology for $289.99 with which you get great advanced support and hardware guarantee for a year (otherwise $75 and we recommend purchasing it if you install yourself).

Admittedly both FlashRouters and Sabai are slightly expensive with comparison to the stock but with Sabai this price includes extensive support and guarantees which is a great justification for the price.

4. Features

Each firmware has a different look, feel and navigation and we will have a look at these below.

4.1 Aesthetics

Though not really a deciding factor the aesthethics of a firmware help give an idea of what you can expect from the software. In the table below you can see screenshots of the homepage of each firmware. We can make a few deductions from these images. The Asus firmware with its clear diagrams and straight forward menu options clearly indicate that it is aimed at the everyday user. The same can be said for Sabai however once in their advanced settings you get an interface very similar to DD-WRT/ Tomato that, though it can be a bit mind boggling, clearly shows how much more the router can be utilised.

 Asus Homepage  DD-WRT Homepage
Asus Firmware Homepage DDWRT Firmware Homepage
 Tomato Homapage  Sabai Technology Homepage
Tomato Firmware Homepage Sabai Firmware Homepage

4.2 Navigation

The above images also show the root menus of the navigation tree and will help you achieve your desired set-ups. These menu options also help give some indicate to how much can be achieved with your router. As mentioned the Asus stock firmware is aimed at the everyday person so it is very straight forward.

We prefer the DD-WRT navigation the most since it seems to make the most sense and it is easy to find any settings/ commands that you might not know where it is located. We also like the fact that it is horizontal and the tab system makes it easier to see through the options.

The Sabai and Tomato are essentially the same once you open the Advanced options in Sabai par VPN which is prioritised for it.  Though they have a slightly odd layout it isn’t overly complicated and just needs some getting used to.

5. Speeds

We have a 30.0/3.0Mbps internet connection all of which we are able to utilise, independent of firmware, when our PC is wired to the router which in turn is connected to the modem. To test the WiFi Speeds we imagined and average household and placed the router approximately 10 meters from the computer on the other side of a thick concrete wall. We maintained all antenna locations the same for the tests and instead used software tweaks to see how much we could improve the connection speeds.

5.1 WiFi Speeds

As we can see from the below speeds all of the firmwares could retain most of the speed with the stock firmware marginally lagging behind and Tomato having the best performance. Also as expected the speeds on the 5GHz channel were faster but of course this has a lower range.

Stock DD-WRT Tomato Sabai
2.4GHz  29.5/3.0  29.6/3.0  30.0/3.0  29.6/3.0
5GHz  29.9/3.0  30.0/3.0  30.0/3.0  30.0/3.0

5.2 Speed Optimisation

Though the results were pretty great already we tried to see if we could squeeze a little bit more out of the system. We did this in the ways described in the below table as well as the improvements we were able to achieve. It’s important that you use some kind of WiFi Analyzer (there are plenty out there we recommend this for Android and the firmwares have some built in too).
Two important notes for improving performance independent of the firmware you are using:
1. Always keep 5GHz and 2.4GHz channels seperate by using different SSIDs
2. Use the least congested channels and use dedicated n or a/c channel setting if all your devices support this

We concentrated on getting the best internet throughput however if you are also concerned about the internal throughput if you are using a NAS then you can use a tool like LAN Speed Test (Lite). Some of settings we have used can make your router unstable and lower it’s lifespan so only make changes if you know what you are doing.

Stock Firmware We weren’t able to do much and hence only made the above two changes as well as trying different Channel Bandwidths. We weren’t able to get any noticeable improvement from it.
DD-WRT Apart from changing the channels we were able to overclock the CPU to 632MHz as well as increasing the TX Power to 150. After this we were able to get both wireless speeds up to 30.0/3.0.
Tomato We were able to overclock the Tomato slightly more at 663Mhz and we also increased the TX Power to 150. As it was already up to 30.0/3.0 we didn’t notice any changes (you could use internal testing as mentioned to find out) but we did notice it helped improve our VPN speeds.
Sabai OS  We made the same changes as to the Tomato and were able to achieve 30.0/3.0 on both channels.

6. VPN

As ever, here at BestVPN the most important factor for us is the use of a VPN. Since the router we are using is very advanced the stock firmware does support both OpenVPN and PPTP clients but for most routers this is not the case and that’s where a large advantage of custom firmwares come in.

We do not recommend using PPTP as it is full of security flaws and therefore we only considered OpenVPN in our testing.

6.1 OpenVPN Setup

We used VyprVPN for our tests because it provides some of the fastest speeds on the market but if you haven’t got a VPN account yet then we recommend you look around to find out what’s best for you.

An advantage of VyprVPN is that it has a very simple .ovpn file which means it can be easily set up on all platforms however most companies have complex files and weren’t sure how the stock firmware would handle these – we know for a fact that they work fine on the other platforms.

Stock Firmware Setting up the connection was fairly easy and required the upload of the .ovpn file and entering the username and password. However we had some minor issues with it and we were required to manually set the DNS server we wished to use.Asus OpenVPN Setup
DD-WRT On both Tomato and DD-WRT there are two main different setup methods and which one you use really depends on which one your provider prefers. It is possible to enter all the details into the commands shells and run it that way or you can set up using the client which is well configured.OpenVPN Client Setup Method)DDWRT OpenVPN Setup 2Command Shell Setup Method)DDWRT OpenVPN Setup 1
Tomato The Tomato setup is very optimised and allows for more flexibility than the DD-WRT one. It also has a much better status monitoring than DD-WRT.Client Configuration)Tomato OpenVPN Setup 1Status Monitoring)Tomato OpenVPN Setup 2
Sabai OS Sabai have a so called 3-click set up mechanism. You upload the .ovpn file, insert your name and password and you’re ready to go. It is as simple as the stock setup and though you don’t get the same amount of monitoring as with the Tomato by Shibby it does display the successful connection as well as it’s location.Step 1 – File import)Sabai OpenVPN Setup 1Step 2 – Making any necessary changes)Sabai OpenVPN Setup 2

6.2 VPN Speeds

We first tested the connection using the Windows software and connected to the router via Ethernet to obtain a baseline number: we achieved speeds of 28.3/2.7. Unsurprisingly the low CPU speeds and power means that only a margin of our available internet speed is actually being utilised.

From the results we can see that unsurprisingly the stock firmware is the slowest. This is followed in third place by DD-WRT. Interestingly for both of these firmwares the VPN speeds on the 5GHz channel were slower than on the 2.4GHz and we’re not really sure why – if you do, let us know.

Since Sabai is an advanced Tomato build it’s unsurprising that their speeds were very close with Sabai being a slightly bit faster.

Stock DD-WRT Tomato Sabai
Wired  11.7/2.7  10.8/2.9  12.9/2.9  13.4/2.9
2.4GHz  8.5/2.7  10.2/2.9  10.5/2.8  10.5/2.9
5GHz  6.1/2.7  9.5/2.8  11.0/2.9  11.6/2.9

6.3 VPN Comments & Discussion

Sabai Technology claim to have a VPN set-up and use optimised system. This shows through the results as not only do you have an easy set-up process but also show to have the best speeds. Further to this is the special feature of Sabai called ‘Gateways’ which is definitely one of it’s best features and we couldn’t find anything similar for the others. Gateways allows you to decide which devices, that are connected to the router, are using the VPN and which aren’t.

As highlighted by the community the setup, speeds and monitoring of VPN connections is not only easier on Tomato but it allows you to set-up 2 clients so you can easily change between different servers/providers that you like using. The VPN client on the Stock is clearly a bit of a gimmick  and the DD-WRT lies in the middle ground.

7. Support

Stock Firmware There are many mixed opinions of the Asus support however it is mainly viewed as negative. Unfortunately while trying to find out about optimising its speeds or solving the issues we had with the OpenVPN setup we had very negative/ useless responses from them and we couldn’t find any good solid community behind it. (Merlin, a slightly improved stock firmware, however does poise a little better on this front).
DD-WRT DD-WRT undoubtedly has the biggest community support. Though it may take some Googling and searching the Wiki Forums you are bound to find and answer/solution to your problem fairly quickly. 99% of issues we had with DD-WRT we’ve been able to solve or at least find out why it’s not working.
Tomato The community for Tomato is not as great as for DD-WRT but there is still plenty of help out there. Also since both of these firmwares are Linux builds in some cases you can find solutions on the DD-WRT forums.
Sabai OS A great feature of Sabai, apart from the aforementioned ‘Gateways’, is it’s tip-top tech support. Though a bit lacking when it comes to the very advanced set-ups they can solve most problems through remote, ticket or even phone support.You can also use the Tomato forums as they will also provide you with plenty of help for achieving your aims. As mentioned in pricing if you want remote or phone support this costs $75/year.

8. Conclusion

Using a custom firmware is definitely worth it especially if you have a lower end router as it allows you to get a lot more performance and settings out of it. Stock firmwares aren’t bad but they are usually they are very clogged up it’s close to impossible to achieve any complex set-ups.

Personally we have always been a big fans of DD-WRT due to it’s ease of use and amazing community support . However, after trying Tomato for a longer period of time I can definitely see the appeal of it as you can save a lot of time by not having to do endless reboots and the improved monitoring and VPN performance you can achieve.  Choosing between Tomato and DD-WRT is really a personal choice and depends on what areas of operation you wish to optimise as well as what hardware you have available. However, if your just using a customer firmware to slightly boost performance and enable some extra features then we would recommend DD-WRT.

Though Sabai comes at some cost in order to get the best out of it, it is definitely worth it especially if you have little experience with using and setting up custom router firmware.


Peter Selmeczy I am an engineer by trade and tech geek by night, who's passionate about sharing his knowledge with the people. Find me on Google+.

Related Coverage


5 responses to “Custom Router Firmware: In-Depth Review

  1. Very good analysis.Thnks a lot. more info needed on Sabai other solutions and set up. Missing is what is VPN servers theoretical speed.

    1. Hi Ashokalgude,

      The speed of a Sabai VPN server is not a function of the firmeware, per se. It is dependent on the router’s hardware (processor speed and memory) and on the raw speed of your internet connection.

    1. Hi Pat
      We probably should have phrased it slightly differently, what I meant is that it’s fairly expensive as you could buy a router and set Tomato up yourself but they have made improvements which are hard to do on your own so it is worth it. Especially if you get their VPN Accelerator as you definitely can’t achieve that on your own.
      Peter

  2. Dear Peter,

    Please note for your Thailand readers that the Asus RT-N66U Tomato router with VPN integration is also available from http://www.freedomrouters.com As of today, the cost is lower than the other companies you mention, delivery times are fast and no worries about shipping through customs.

    Kind regards,
    Nat

Leave a Reply

Your email address will not be published. Required fields are marked *