Significant Issues with the School’s OpenAFS File System
We are currently experiencing significant issues with the School’s OpenAFS file system. This contains our user’s DICE home directories along with the majority of the School’s group and research data. The majority of our administrative data is now held on the University’s DataStore service and so is unaffected.
This issue is not restricted to Informatics but is affecting OpenAFS file systems world wide. The OpenAFS community, which includes organisations such as CERN and Carnegie Mellon University, has been working on this problem and it is believed that the cause has been identified and a solution will soon be available.
The general consensus is that there is no risk of permanent data loss due to this issue but users should be aware that they may at any time, without warning, temporarily lose access to any data they may have stored in our OpenAFS file system. They may therefore wish to copy any data they may need access to in the next few days to a SECURE location outwith the School’s file system. Ideally, data should be moved to another University service such as DataStore or DataSync (see http://computing.help.inf.ed.ac.uk/central-data-storage for more details of these services) but users may find it more convenient to copy their data to their local machine. Users should only do this if their local machine is secure (i.e. in a safe location and with an encrypted disk) and if they can make local arrangements to back up any changes made to their data.
We apologise for the inconvenience this may cause you and will keep you informed of progress in addressing this issue. If you need help with moving your data or have any questions, please contact Computing Support using the form at http://www.inf.ed.ac.uk/systems/support/form/ in the usual way.
Update 11am 15th January 2021
- The cause of the issue has been identified and a patch produced by Jeffrey Altman of Auristor. Many thanks to Jeffrey for his quick action.
- We now believe that the School OpenAFS file system will remain available, albeit with possible short interruptions as file servers are updated, PROVIDED THAT YOU DO NOT REBOOT YOUR MACHINE. Rebooting your machine will cause you to lose access to your data.
- We are currently planning deployment of the patch to desktop and lab machines. We will issue a further update later this afternoon with details of how this will happen.
Update 5pm 15th January 2021
- The servers providing the OpenAFS infrastructure have been patched and restarted.
- All desktop and lab machines will be patched and rebooted automatically at some point early next week.
- Until this patching takes place, machines should still be able to access the file system. If you do encounter issues, we recommend that you reboot the machine immediately. This will install the patch and should fix the issues.
As ever, if you are having problems, or have questions about any of this, please contact Computing Support using the contact form at http://www.inf.ed.ac.uk/systems/support/form/.
From one of our helpful users, regarding updating self-managed Ubuntu:
… resolving things on self-managed ubuntu 20.04 was pretty easy
once I figured out where to find the updated version. … this
is what worked for me (at least insofar as I can now cat text files on
AFS when I couldn’t before):
sudo add-apt-repository ppa:openafs/stable
sudo apt-get update
sudo apt-get install openafs_1.8.6-5