Any views expressed within media held on this service are those of the contributors, should not be taken as approved or endorsed by the University, and do not necessarily reflect the views of the University in respect of any particular issue.

LCFG Ubuntu Project

LCFG Ubuntu Project

Progress reports on the LCFG Ubuntu Project

Upstream package repository mirroring

One of the most important parts of the support infrastructure for any DICE platform are our mirrors of various upstream package repositories. Along with the obvious advantage of saving bandwidth by not having ~1000 machines reliant on fetching updates directly from the upstream site this gives us a lot more control over what we have installed on our machines.

One of the big issues we always experience when using upstream software repositories is that typically they remove old versions of packages as soon as security updates are released. This makes sense from the point of view of the distributor because they don’t want to be distributing software which is known to be insecure. However, we cannot just automatically apply all updates as soon as they become available, statistically there is just too much inherent risk in that process failing when lots of machines are involved. Furthermore, we occasionally need to pin back the versions of software for compatibility or reproducibility reasons. We prefer to have the option to choose if and when security updates should be applied based on an understanding of the risks involved. Debian (and thus Ubuntu) do have support for pinning back old versions of packages so that the apt tool will not upgrade them automatically but that still relies on the pinned version being available in a repository. How would we install a new machine such that it is using an older version of a package if that version is no longer available anywhere?

This might seem like a rare requirement but when you take into account the release management process we follow to ensure the stability of our DICE platform it becomes a necessity. All of our machines follow one of three releases:

A rolling release which gets updates applied as soon as they are available.
A weekly snapshot of develop which goes through various quality checks to ensure it can be reliably installed and upgraded.
A copy of the weekly testing release which is only updated once all tests have been passed (and any problems resolved).

How would we install a machine using our stable release if recent (untested) security updates had caused the removal of older (tested) updates?

In the prototype phase of our Ubuntu project, for simplicity, we just used the upstream repositories which was fine with a bit of occasional manual hackery but this is now beginning to cause pain. In particular, our package manager – apteryx – will apply updates and then next time around attempt to downgrade a package to the required version then next time upgrade it, etc… We need to disable the automatic updates so that we can pin the versions. I previously investigated if we could use the reprepro tool to do the mirroring but this turned out to be fundamentally limited in that it only handles one version for each package name/architecture. That investigation led to aptly which is a much more sophisticated tool which we have chosen to use. In subsequent blog articles I will cover the recent work I’ve done to configure aptly to do all the repository mirroring we need.

Leave a reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>


Report this page

To report inappropriate content on this page, please use the form below. Upon receiving your report, we will be in touch as per the Take Down Policy of the service.

Please note that personal data collected through this form is used and stored for the purposes of processing this report and communication with you.

If you are unable to report a concern about content via this form please contact the Service Owner.

Please enter an email address you wish to be contacted on. Please describe the unacceptable content in sufficient detail to allow us to locate it, and why you consider it to be unacceptable.
By submitting this report, you accept that it is accurate and that fraudulent or nuisance complaints may result in action by the University.