Making Tor Browser Updates Stable and Reliable with Fastly

Tor Browser is well-known for its tracking protection and fingerprinting resistance. We have spent a lot of time and energy as well to make the browser bit-by-bit reproducible in order to defend against compromises of our build machines. It is worth mentioning that this includes our update files, too, which are generated during the build process. But having reproducibly built update files, which are properly signed, is only one half of the setup we need: without being able to make them available in a timely and reliable manner our users are not as well-protected as they should be.

Back in 2014 when we introduced the browser updater with Tor Browser 4.0, we hosted the update files ourselves. But it soon became obvious that it would be a challenge to scale our infrastructure to keep up with the user demand and ever growing update sizes (just compare the 50 MB we had on Linux for full updates in 2014 to the 85 MB we have today). Gladly, Fastly stepped up in 2016 to guarantee smooth updates for Tor Browser users. After some experiments, updates were provided over Fastly's infrastructure from June 2016 on. On average roughly 2.1TB/day has been transferred since then, with spikes of over 14TB/day. Thus, a big thank you, Fastly, for hosting our Tor Browser update files for the past 18 months!

We don't know exactly how many Tor Browser users those transferred 2.1TB/day represent, but we started to collect download metrics a while back in order to at least observe trends in user update behavior. If we look at the past 18 months (see image above), we can see that the amount of daily update pings, which check whether an update is available, continually rose up to 2,000,000 in February 2017 and has stayed more or less constant at that level (apart from the outlier between February 2017 and April 2017 which gets investigated in bug 22346). Update requests on the other hand range from 600,000 to 1,200,000 on the day a new Tor Browser stable version gets released (with a sharp drop and a long tail afterwards) which shows the importance of a reliable and robust update infrastructure.

We'd like to emphasize that we've only been using Fastly for Tor Browser updates, which are fetched anonymously over Tor. One reason to focus just on updates is because updates produce the highest bandwidth spikes, which is where Fastly is helpful most to us. Moreover, Fastly does not store customer request logs, so they do not store logs for downloads of Tor Browser updates. This commitment is important as it shows a clear stance towards user privacy which is especially valuable in the update case: an attacker can't show up after the fact and learn update requests and behavior of particular users. This is a good first line of defense which is worth having in a non-update context as well. But Tor Browser takes this a step further by updating its users solely over Tor. Even if there were logs available or someone were able to compromise Fastly's servers, it would not be possible to target a particular Tor Browser user with a malicious update or learn about their update behavior as all users show up as coming from the Tor network on Fastly's systems. This is not only relevant for Tor Browser but is in general an important feature as outlined in an earlier blog post.

Thanks to Fastly we have found a way to solve the stability and reliability issues in our update infrastructure while still protecting our users. However, bandwidth load and update sizes will continue to rise, not only because we will hopefully attract even more users in the future, but also because we deliver Tor Browser to more and more platforms (see for instance the recently started support for 64bit Windows systems). This will likely introduce new scalability challenges given our privacy and security constraints and the need for reliable updates. But we are optimistic to solve them with our friends at Fastly should they arise.