We’ve just added a new software package called Singularity to all hosts running our Ubuntu Linux Platform, including our Slurm Compute Cluster.

Singularity is a really cool piece of software and it has become very popular in the last few years, so we thought it would be worthy of a wee blog post to introduce it to folks in our School who maybe haven’t met it yet.

What is Singularity?

Singularity essentially makes it easier to run Linux software on different Linux systems.

It does this by putting your software – along with all of the other bits & pieces of software it needs – into a wee “box” (called a container) that can be easily run on other Linux systems. This can help work around a common problem with Linux software, whereby the other pieces of software that your own software/code requires either isn’t – or can’t easily – be installed, or isn’t the version that you require.

The folks who make Singularity have written a nice introduction to it:

Introduction to Singularity

How might Singularity help you?

Singularity can feel a bit abstract at first. So here are some examples of the types of problems it can solve. Some of these may hopefully resonate with your own work, especially if you’re trying to run Linux codes on multiple different systems.

• Some teams who produce scientific codes already make their codes available as Singularity containers that you can download from online repositories. It’s then super easy to run these on any Linux system that has the Singularity software, regardless of the “flavour” (e.g. Ubuntu vs. Scientific Linux) or version of the Linux system. These containers include cryptographic signatures, which help you make sure you’re not being tricked into running something dodgy.
• If you’ve been developing your own scientific code on your own Linux computer or laptop, you could box this into a Singularity container and then run it very easily on other computation facilities and clusters that support Singularity.
• If you’ve been using our School’s legacy Scientific Linux 7 Platform over the last few years, then you’ve maybe got some codes that currently work on SL7 and can’t easily be made to run on our current Ubuntu Linux Platform. (We know of a few examples of this in Nuclear Physics & Particle Physics Experiment.) As we’re going to be dropping support for SL7, this has the potential to become a problem! However Singularity might well solve this problem: it could be possible to put your code into a Singularity container that would run absolutely fine on Ubuntu… and potentially on other platforms.
• You may have developed some code that uses version 1.0 of another code library (for example, a high performance numerical libraries like OpenBLAS or Intel MKL) to do some of the work, and you’ve run your code to generate some results. You might later want or need to regenerate those results, but find that the code library you had been using is now at version 3.0 and gives slightly different results than it used to – or maybe even now works completely differently. One way to resolve this potential problem is to create a Singularity container for your code containing exactly the code versions that you need, thus giving you very precise control over your code and its dependencies, which can help make the results of your codes reproducible.

As well as now being installed on our Ubuntu Linux platform, Singularity is already installed on the University compute service (Eddie) and the UK National Supercomputing service (ARCHER2). Thus you could hopefully run the same Singularity container on the School’s Ubuntu Linux computation facilities and Eddie and ARCHER2.

A recent case study

Here’s a more concrete example of how we recently used Singularity within our School.

One of our MPhys Project students needed to run some newly developed medical imaging code on School computing facilities.

The small team who created this new code helpfully provided some instructions for getting the code working on an Ubuntu PC. As we also run Ubuntu locally, this was a good start! However their instructions required the user to have full (so-called root) access to the system, and the instructions wanted to install various codes, files and other assorted fluff in a few different parts of the system. Unfortunately we can’t give users the necessary freedom required to do this on our own facilities, so we had to find another way to make the software available for the student.

In these types of scenarios, we often first look at creating a “proper” Ubuntu package for the software. However due to the spread out nature of this code, and the fact that it was only going to be required by one user, we weren’t convinced this was the right approach to take here. Additionally, there was a chance that the student might end up needing to run the same software on the University Compute Cluster (Eddie), as the code can take advantage of GPU facilities, which are in relatively short supply locally.

So Singularity seemed like it might be a much better fit, and I was indeed able to use the provided build instructions to put the software into a Singularity Container for the student to try out. After a bit of testing and feedback, the student now has a working container that they can run via Singularity on their own laptop and our own Ubuntu-based Linux facilities, and it would be pretty easy to switch over to using Eddie later if necessary, as the Eddie team have installed Singularity there as well.

How do I find out more?

We’d recommend that you first speak to us if you want to use Singularity, as you’ll probably need a bit of support and we’d be really keen to learn how you get on with it.

The Singularity folks have some good documentation, available at:

Singularity User Guide

Note that Singularity is already installed on all of our Ubuntu hosts, so you shouldn’t try to install it yourself.

Also note that while you can run a container on our Ubuntu hosts, you can’t build a container there as it requires you to have administrator access, and we can’t give you that level of access. Thus you’d need to build a container on another system first. (Or you can ask us to help you build a container for you.) When I built the container described in the case study above, I actually built it in a VirtualBox Ubuntu virtual machine on my Mac laptop. This gave me the flexibility to explore this… and occasionally make a total mess of things… in my own little sandpit, so that I didn’t have to worry about disrupting my other work.

Please do speak to us if you’d like to explore Singularity:

• You can email the School Helpdesk: sopa-helpdesk@ed.ac.uk
• Alternatively, you can post in the SoPA Research Computing space in Teams.

New software: Singularity / Linux & Scientific Computing - Physics & Astronomy by blogadmin is licensed under a Creative Commons Attribution CC BY 3.0