One of the most important parts of the support infrastructure for any DICE platform are our mirrors of various upstream package repositories. Along with the obvious advantage of saving bandwidth by not having ~1000 machines reliant on fetching updates directly from the upstream site this gives us a lot more control over what we have installed on our machines.

One of the big issues we always experience when using upstream software repositories is that typically they remove old versions of packages as soon as security updates are released. This makes sense from the point of view of the distributor because they don’t want to be distributing software which is known to be insecure. However, we cannot just automatically apply all updates as soon as they become available, statistically there is just too much inherent risk in that process failing when lots of machines are involved. Furthermore, we occasionally need to pin back the versions of software for compatibility or reproducibility reasons. We prefer to have the option to choose if and when security updates should be applied based on an understanding of the risks involved. Debian (and thus Ubuntu) do have support for pinning back old versions of packages so that the apt tool will not upgrade them automatically but that still relies on the pinned version being available in a repository. How would we install a new machine such that it is using an older version of a package if that version is no longer available anywhere?

This might seem like a rare requirement but when you take into account the release management process we follow to ensure the stability of our DICE platform it becomes a necessity. All of our machines follow one of three releases:

develop

A rolling release which gets updates applied as soon as they are available.

testing

A weekly snapshot of develop which goes through various quality checks to ensure it can be reliably installed and upgraded.

stable

A copy of the weekly testing release which is only updated once all tests have been passed (and any problems resolved).

How would we install a machine using our stable release if recent (untested) security updates had caused the removal of older (tested) updates?

In the prototype phase of our Ubuntu project, for simplicity, we just used the upstream repositories which was fine with a bit of occasional manual hackery but this is now beginning to cause pain. In particular, our package manager – apteryx – will apply updates and then next time around attempt to downgrade a package to the required version then next time upgrade it, etc… We need to disable the automatic updates so that we can pin the versions. I previously investigated if we could use the reprepro tool to do the mirroring but this turned out to be fundamentally limited in that it only handles one version for each package name/architecture. That investigation led to aptly which is a much more sophisticated tool which we have chosen to use. In subsequent blog articles I will cover the recent work I’ve done to configure aptly to do all the repository mirroring we need.

Upstream package repository mirroring / LCFG Ubuntu Project by blogadmin is licensed under a Creative Commons Attribution CC BY 3.0