Flood in the server room (not for real!)
Periodically, we test our preparedness for disasters by holding a mock disaster exercise.
On 27th February, the computing staff were told that part of the roof of the Forum server room had collapsed due to flood damage and that this had destroyed a number of the server racks. Obviously in such a scenario, it would be unlikely that computing staff would be allowed access to the server room, so they had to work out from records which kit had been damaged.
Each computing unit was asked to produce a report on what services would have been affected and the state of the backups for those services. They were also asked to test reinstall one service just from those backups. The reports are available here.
In summary, the only data that would have been lost was data on a small number of servers, owned by a research group, that we had not been asked to backup. The owner of these servers has been approached and given the opportunity to rectify, if required. Where Institute servers are being used to mirror each other, we plan to ensure that they are physically in different buildings. The exercise highlighted a number of relatively minor procedural problems, mainly related to record keeping, which will be corrected.
Great exercise!