Let’s Encrypt Issue

As many of you will be aware by now, yesterday one of the Let’s Encrypt certificates expired and there was a certain amount of fallout…

Frustratingly, the necessary updates for the ca-certificates packages on both SL7 and Ubuntu Focal were released too recently to be in our latest LCFG stable release. Furthermore, on SL7 it was only issued as a “fastbugs” update so we didn’t notice until it was already too late.

I am planning to issue an additional stable release – 2021092701c – today which will have the necessary package update specifications. In the meantime you can add something like this to your headers:

#ifdef LINUX_EL7
!profile.packages mEXTRA(+ca-certificates=2021.2.50-72.el7_9/noarch)
#endif

#ifdef LINUX_UBUNTU
!profile.packages mEXTRA(+ca-certificates=20210119~20.04.2/noarch)
#endif

Due to the ancient version of openssl on SL7, if your LCFG clients on SL7 are using HTTPS to communicate with the LCFG server and that server uses Let’s Encrypt certificates you will have a problem that can only be fixed with manual intervention. The new profile will not be fetched until you update the ca-certificates package. For Informatics SL7 machines we ended up logging into each via ssh as root, using wget to fetch the rpm package, and then installing using the rpm command.

The LCFG client on Ubuntu is not affected so will pick up the new profile. However, for some reason which we don’t yet understand, if apt is configured to use HTTPS to fetch packages and the package server uses Let’s Encrypt certificates then it will not be able to communicate with the package server. Again, this leaves you needing a manual intervention. On Ubuntu we added a cron job to our LCFG headers like this:

!cron.additions         mADD(cacertfix)

cron.add_cacertfix     AUTOMINS 18 * * * wget -N -P /var/tmp https://deb.pkgs.inf.ed.ac.uk/ubuntu/pool/main/c/ca-certificates/ca-certificates_20210119~20.04.2_all.deb && dpkg -i /var/tmp/ca-certificates_20210119~20.04.2_all.deb

Of course, if you use that strategy you will need to adjust the timings appropriately. It seems this is safe enough to leave in place for a few days to allow for machines that aren’t running at the time of the first pass.

It’s tempting to manually edit the /etc/ca-certificates.conf file but note that once you’ve done that it will not be updated by future updates to the ca-certificates package. Both rpm and dpkg will preserve any modified config files. You would be left either needing to revert to the package version of the config file at some later point or having to actively manage any further changes that are required.

The nuisance here is that none of this completely helps with machines that are currently turned off. There will need to be an ongoing effort to identify broken machines and apply the manual intervention.

If any of this is unclear or you need any further advice I recommend you join the LCFG slack channel where we have been discussing the problem. Annoyingly, due to a dns issue, slack has been having it’s own problems today…