Forum server room UPS upgrade programme
Earlier this year, we mentioned that we intended to replace the Uninterruptible Power Supply (UPS) system which supplies power to all of the server rooms located in the Informatics Forum – see our blog post Forum server room UPS from January 24, 2019.
Since then, we’ve been doing lots of preparation, and we are now about to commence the actual replacement programme. The new UPS we’ve chosen will be able to sustain a power load of 200kW – more-or-less twice the load which the existing UPS can supply – and it will also be far more resilient than the existing system. We currently expect the new UPS will be fully in operation by Wednesday July 24, 2019 but, between then and now, there is a great deal of electrical infrastructure work to be completed, and some of that work will cause unavoidable disruption.
Some key events (and dates/times) for your attention are as follows:
-
Isolation of the building-wide Forum UPS: Tuesday 28th May, 2019; 7:00am
Explanation: As well as the UPS which supplies our server rooms, the Informatics Forum also has a completely separate UPS system which currently supplies all offices, and all IT closets. As part of the current programme, that UPS system will be decommissioned and permanently removed. Arranging that will require a brief (we expect no more than five minutes) power cut to all Forum offices and all IT closets at 7:00am on Tuesday 28th May, 2019.
-
Load-shedding from the Forum server rooms: Thursday 20th June, 2019 – Monday 22nd July, 2019
Explanation: During the replacement of the server room UPS, we will need to operate our server rooms for the above four week period using only one-half of the existing UPS system. In order to make that feasible, we will need to reduce the combined power load currently being used by all of the servers located in the Informatics Forum by about 20%.
-
Shutdown of Forum server rooms: Saturday 20th July, 2019; all day
Explanation: In order to bring the new server room UPS fully into service, we will need a total shutdown of all Forum server rooms on the Saturday 20th July, 2019.
We’ll be in touch with owners of self-managed servers regarding items 2 and 3 closer to the date. Meanwhile, if you have any questions about this work, please submit a support ticket in the usual way.
What is the expected amount of downtime that the UPS will save over its lifetime, based on historical grid power outages and the amount of time the UPS can expect to power the servers without the grid?
How does that compare to a month-long brownout and day-long outage?
It’s difficult for me to give a precise answer to those questions. Generally, in the UK, 95% of all mains outages last for less than five minutes, with anything longer likely to last for many hours. We do see occasional mains outages here, but we haven’t keep a detailed record of when those have occurred.
Of course, apart from protecting against complete outages, the UPS also provides a clean power supply which protects our equipment against spikes, surges, sags and brownouts.
It is incorrect to refer to the forthcoming period as a ‘month-long brownout.’ The requirement for load-shedding has arisen because we have been running the existing UPS system at well over 100% of its rated capacity. If we had previously constrained the introduction of new computing equipment here so that the UPS was running within its design capacity, the current UPS replacement work could have been arranged in way that was completely transparent to users: no load-shedding would have been necessary, and the only noticeable effect would have been a single necessary outage (to alter the supplies to the final distribution boards in the chain) in the final stages of the work. So, as usual, it’s all about trying to strike reasonable balances.
An alternative arrangement which would have avoided the need for any temporary load-shedding would have been to have run our systems for the next month with no UPS backup at all. We did explore that idea, but School management eventually decided against the risks which that would involve.
In an ideal world, I think we would have two distinct supplies to our server rooms: one backed by a central UPS system, and one directly supplied from the mains. That would give us options for the location of very power-hungry equipment. However, as things stand, every circuit in our server rooms is currently supplied by the central UPS system which is the subject of this blog article. Changing that arrangement – while possible – would involve a lot of disruptive work.