If your weekend was spent in darkness, you weren’t alone. A fire at an electrical substation in the Orange County caused a massive power outage and left an upwards of 28,000 residents and travelers in Irvine and Newport Beach without power. Additionally, the outage also led to temporary grounding of all flights at John Wayne Airport. In today’s day and age of modern technological advances, an outage might seem strangely unexpected, but the repercussions are far greater. Mishaps like these are not only a threat to human safety but the business impact (and IT issues emergencies that situations like these create) of a sudden shutdown is massive. Imagine an entire airport getting shut down and the domino effect on flight delays, passenger frustration, business, and more.
What exactly happened?
On Friday, August 2, 2019, Orange County Fire officials were informed of a vault fire near Jamboree & Michelson. About 100 occupants of an apartment building on Palatine were temporarily evacuated due to smoke. The fire seems to have been the cause of the power outage that affected over 28,000 residents and left John Wayne airport without power for over 12 hours. The airport was forced to cancel all outbound flights and divert all inbound flights until Saturday morning. Crews worked through the weekend and were successfully able to restore power in Irvine by Sunday morning. It is however believed that some people in Orange County may continue to experience outages as the Southern California electricity company works to switch back to the normal source of power throughout the coming days.
What is needed?
When disaster strikes, it strikes without warning. The series of earthquakes that struck California a month ago in July and the fire in Irvine over the weekend are classic examples of this. Crises like these can severely disrupt business operations and can be catastrophic for your organization. If you haven’t done so already, now is the time to revisit your disaster recovery and business continuity strategies. You need to be prepared for unexpected incidents and ensure that disasters have minimal impact on your business. Here are some things to consider:
- Identify the critical path infrastructure – what specific IT and other infrastructures are needed to keep the workforce progressing on daily business needs?
- Decide if you need business continuity or disaster recovery (BCDR), or both? Continuity solutions mitigate the cost of lost time (stalled business operations). Disaster recovery ensures data is recoverable. You may decide losing access to email data for 24 hours, while disruptive, does not require a solution that can provide continuity within minutes.
- Define a what a disaster means for you – it’s not always a fire. What triggers a disaster recovery event? What critical infrastructure failure would trigger a disaster recovery process?
- Build and regularly test your disaster recovery plan. The plan should cover, who would be contacted in event of a disaster, who is responsible for recovery operations, what systems would be brought up (and in what order), as well as defining the failover sites and where data is going. If you don’t have a plan or don’t regularly test your plan, at a minimum, ensure the plan covers the critical path infrastructure and is capable of ensuring continuity.
- Carry out periodic redundancy and modernization of your processes so that legacy, vulnerable systems do not cause any disruption.
- Make enterprise security your number one priority by maintaining consistent policies, periodically training employees on security threats, having systems in place that can proactively spot vulnerabilities, and improving the security posture of your organization.
- Limit access to critical data and have strong access control measures in place, implementing security methods like multi-factor authentication for improved security.
- Opt for Disaster Recovery as a Service to enable a full recovery in the cloud, in the event of a disaster; this way, you can not only protect your data, but also facilitate a more rapid data and application recovery by leveraging the cloud’s on-demand computing capacity.
What you can do
The influence of a disaster like this weekend’s fire can be extensive. In addition to leaving thousands of residents without power, two major issues of the substation fire are the impact to John Wayne grounding flights and inability to operate for extended hours and the potential impact on unprepared businesses. Given that operating business as-is is extremely critical in the event of a disaster, it is important you take the right steps:
- Make sure you have multiple layers in your BCDR plan in case of extended periods of shutdown. Your BCDR plan should include gracefully shutting down and securing systems before imminent failure. This could range from disk-level encryption to a well-secured facility with limited ingress points because during extended emergency situations, you do need to plan for potential looting, property destruction, etc.
- While it is good to back up your data, make sure it is backed up in an offsite location that is outside your core IT environment.
- If you’re investing in disaster recovery solutions like redundant systems and diesel generators, remember that those methods will eventually run out of fuel. Look to invest in fault-tolerant systems and understand the gaps in capabilities to have an all-encompassing, robust plan in place at all times.
- Loop in a Managed Services Partner like Synoptek to take the continuity burden off your head; a partner would not only ensure end-to-end disaster recovery, but also help develop and strengthen your business continuity plans, guaranteeing the ability to continuously meet the needs of your customers – today and tomorrow.
Is your organization prepared? Contact us and find out.