At the Tor Project, we're always on the lookout for opportunities to contribute back to the communities around the platforms and tools we depend on to keep the lights on. Puppet and Debian are two such projects, so we're happy to announce that the upcoming Debian stable release, codename bookworm, will deliver an up-to-date suite of Puppet software thanks to the efforts of the Tor Project!

A year ago, TPA (AKA Tor Project sysadmin Team) started planning an upgrade of our fleet of nearly 100 Debian machines to the latest stable release, bullseye. One item of concern was that not only were the Puppet packages in Debian bullseye already nearly end-of-life (version 5.5), but the PuppetDB package was also now missing entirely from the distribution. At this point it seemed the only feasible option would be to migrate our entire Puppet infrastructure to the vendor-supplied packages.

Advantages of distribution packages

So why not switch over to the upstream Puppetlabs packages and call it a day? Essentially, because deploying software directly from vendors is not a decision we take lightly, and because Puppet is such a core component of our infrastructure, this called for careful consideration.

There are several reasons to prefer software shipped through distribution packages.

Improved security and reliability

Reports of increasing numbers of software supply-chain attacks in recent years have shed light on the difficult problem of shipping software securely. Many popular solutions, like npm or PyPI suffer from vulnerabilities which are not easily fixed, such as typosquatting and developper sabotage, at least not without breaking what makes them popular.

Puppetlabs doesn't rely on third-party package managers, and instead maintains its own APT repository to ship packages to end-users. Although this is an improvement, configuring an additional APT repository on a Debian-based system effectively grants whoever controls the repository the ability to deploy software and execute code with maximum privileges. This is a significant potential attack surface.

We understand the challenge of running an APT repository, and we trust in the ability of the Debian project to maintain their own repositories reliably and securely.

Even assuming the delivery channel is secure, concerns still remain about the process used to build the binary packages. Not only does Puppetlabs employ a somewhat exotic in-house build system, EZBake, but we have no means to evaluate the integrity of the build process itself.

On the other hand, Debian package are built from source (as opposed to redistributing upstream binaries) on purpose-built infrastructure, and logs generated during this process are made available for examination.

The Debian project, in collaboration with reproducible-builds.org even provides automated reproducibility tests for packages and encourages maintainers to fix issues hampering the ability for anyone to compile a given package and obtain a byte-for-byte identical binary. Both the puppet-agent and puppetserver package we uploaded to Debian are currently reproducible.

User-centric policies

Another concern is that the Puppetlabs packages ship with analytics, enabled by default. Although an opt-out configuration parameter exists, features like this are necessarily in conflict with users' right to privacy.

This is one area where Debian also shines: the project's social contract guarantees the interests of users are always placed first and foremost. This guides Debian the established practice of adressing privacy leaks in package resources and runtime code.

As such, Debian's puppetserver package doesn't implement telemetry, and the update check that "phones home" at regular intervals is disabled by default.

Greater system integration

The Puppetlabs packages deploys its runtime libraries and code under the /opt/puppetlabs directory. In addition to the Puppet applications, this includes copies of the Ruby, JRuby and Clojure interpreters, as well as dozens of third-party Clojure and Java libraries.

This duplication of code is a burden that is ultimately carried by system administrators because they're the ones responsible for ensuring their infrastructure is running code that is up to date and free from security issues. Debian has a solid track record on this front, and we're confident that security issues in, for example, the Ruby intrepreter, are adressed swiftly.

In addition, security updates to the Debian stable distribution are crafted in a way that minimizes as much as possible the upgrade footprint. Puppetlabs offers no such guarantees for their packages, so it's not unlikely that at some point, addressing a security issue might require a disruptive upgrade.

A sustained effort

Work on packaging new PuppetDB and Puppet Server Clojure dependencies in Debian had already progressed, thanks to funding from the Wikimedia Foundation. So it felt completing the work was within reach.

Thus, over the past 6 months, in addition to our regular TPA duties we have been working to get Puppet in Debian ready for bookworm, the upcoming Debian 12 stable release. We started with the Puppet agent package, before moving on to PuppetDB and finally Puppet Server, upgrading or adding dozens of dependency packages along the way including many Clojure and Ruby libraries, JRuby and other Puppet components such as Facter, Hiera and Trocla.

In collaboration with the Debian Puppet Team, we worked to triage and fix bugs reported in the Debian bug tracker, as well as implementing new or improved build and integration tests for Debian CI. We also collaborated with Puppetlabs to merge several patches upstream, including one patch to improve the reproducibility of Puppet agent builds.

Looking ahead

With this work now behind us, we can look forward to bringing our Puppet infrastructure into a more modern age. Dealing with legacy tools and technical debt is a constant challenge at the Tor Project, as it is no doubt in many other organizations, so we're very happy to share news of this accomplishment. Also planned is migrating our various tor and onion deployments to the community tor Puppet module.

We also know that many privacy-minded individuals and outfits (Tails, for example) also rely on Puppet being packaged in Debian, so we're thrilled that this work will directly benefit them.

What about Puppet over Tor?

For those of you who were hoping to learn about using Puppet over Tor onion services, you'll enjoy this tutorial from the good folks at the Immerda.ch tech collective. This isn't something we implement as the list of project-administered machines is public, but should be useful for anyone wishing to add a privacy layer to their Puppet server-node communications, especially if those are transported over the Internet.