Essential maintenance on these mission critical systems is, as it says, ‘essential’, however on occasions easier said than done. Allowing maintenance to be undertaken with zero impact to business as usual can be a daunting prospect for a clients’ property management or critical engineering team to face.
With the aspirational targets set by the Tier resilience and redundancy ratings applied to many facilities this is now becoming easier to achieve. Maintenance of mechanical and electrical infrastructure is essential to ensure the resilience levels are sustained at the highest levels, with minimum impact to mission critical systems and operations. However, for businesses who do not have these redundancy levels set in place on their critical systems, careful planning management and organisation is required for maintenance to be achieved.
For many of Faithful+Gould’s financial sector clients, business systems such as in-house data centres, are in use 24/7 365 days per year, so the question arises, when and how can the essential maintenance work take place? The solutions lie in the clear planning and management of such events.
The planning process
What kind of power down is required?
Power downs tend to mean different things to different people, with some referencing the physical electrical power supplies, incorporating a “Black Building” scenario whilst others will refer to the power of down of servers and the IT infrastructure. Additionally, there may be various levels of power downs, e.g. partial where there is essentially “reduced resilience” for a period of time and full where there may be zero requirement for any of the supporting infrastructure to be available.
Where a full power down for maintenance works is required, for example, to allow the replacement or upgrade of critical plant or equipment, then it is highly likely that a full data centre shutdown is required. This involves agreeing a pre-arranged date with the IT team approved by all stakeholders internally and externally, to methodically power down all the IT infrastructure in readiness for the equipment to be isolated to allow maintenance to take place. In contrast, if the planned maintenance works could be undertaken with impact to part of the IT infrastructure such as a single comms room, it is feasible that only a partial power down would be required.
The planning process begins with the client identifying the need for the systems to be powered down. This can include general maintenance, upgrade works, reconfiguration works and or to undertake an Integrated Systems Test or Black Building test to demonstrate that the systems function as intended.
Once all stakeholders have agreed that a power down is required and sufficient benefits can be achieved from such an event, the detailed programming can commence. This is where the role of the Faithful+Gould Project Manager starts, ideally at least 90 days prior to the power down.
Certain clients have very specific windows of opportunity in which construction activity can occur before returning the building back to business as usual. This window may be limited to, for example, between 1pm on the Saturday to 6am Sunday to avoid clashes with trading timeframes. Given these tight parameters, meticulous precision programming is required for the whole period, so as much preparation time as possible must be allowed to ensure that these 17 hours are maximised. Given the costs associated with such events, and to minimise the disruption, Power Downs will seldom occur more than once a year, as a result there is an inclination to maximise the opportunity and undertake as much work as feasibly possible. It is not uncommon for up to 20 various workstreams and 100+ contractors on site.
The costs associated with supporting a full power down event taking in to account IT and incumbent suppliers can be anywhere from £200k to £300k per event, excluding any main contractor costs. As well as the maintenance and project contractors undertaking the work, costs are involved with property management teams, technical operations teams, IT, catering and standby support to name a few, not to mention the costs related to the loss of business incurred during the power down period. It is crucial however to consider that these costs should be offset against the risk (and potential costs) of not undertaking the Power Down, decreasing the likelihood of unplanned equipment failure due to lack of maintenance and system upgrades.
Faithful+Gould event management
Mission critical systems are unique to client’s business, therefore the Faithful+Gould Project Management team provide a bespoke service to clients’ requirements to include, but not limited to the following:
The provision of experienced project managers with engineering backgrounds supporting all aspects of the power down, from 90 days ahead of the power down through the power down window itself
Production of overall programme and hour-by-hour working schedule for the power down window
Chairing of risk workshops and production of risk registers
Identification of Go / No Go milestones
Engagement with stakeholders and approvers
Liaison with IT including management of handovers
Focal point of contact during the power down window
Creation and management of escalation plans and update protocols
Close out report and following up on any residual actions / discoveries