Grid: Locked — Managing the Risks of Hacking the Electric Grid

This post first appeared on Risk Management Magazine. Read the original article.

On Dec. 17, 2016, hackers successfully targeted an electric transmission substation outside of Kiev, Ukraine, leaving part of the city without power for about an hour. Widely thought to be attributable to the Russian government, the incident was the second attack to cause a power outage in Ukraine in as many years. The incident was far from catastrophic—indeed, the attack the year before cut off power to more people and for a longer period. But cybersecurity researchers now believe the 2016 attack was merely a dry run, testing out the most advanced malware ever deployed to target a grid—an adaptable, scalable tool purpose-built to disrupt critical infrastructure.

Supporting the theory that it was more a proof of concept, this attack did not even make use of all the functionality and modules built into the malware, dubbed “CrashOverride” by industrial control systems cybersecurity firm Dragos or “Industroyer” by Slovakian anti-virus firm ESET, the two companies that identified and analyzed it. The malware has two backdoors (a backup in case the first is discovered), a port-scanner that automatically maps out the network to identify target equipment, a wiper to cover its tracks after an attack, and the ability to record and report network logs so that attackers can better learn how the control systems function over time.

Most concerning, CrashOverride does not exploit a system vulnerability. Rather, it takes the knowledge gleaned from previous attacks and abuses the system’s functionality, sending messages directly to grid equipment to switch the flow of power on and off. “There’s no defense against the attack itself,” explained Robert Lee, CEO of Dragos. “There’s defense against [hackers] putting it in place, stopping the attack before it occurs. But once it is in, it just works because that’s how the electric grid works. As a result, there’s no technical limit to scalability, and the limitation of getting it placed is just a limitation of the humans.”

CrashOverride has a swappable component design, making the threat less about the specific malware and more its role as a framework that can be customized for different targets, either in other regions or, potentially, other industries. The precise protocols targeted in the current iteration of this malware are used in electric power control systems outside of the United States—it would work in all of Europe and most of the Middle East and Asia. The modifications required to make it work on grids in North America, Australia and New Zealand would take “less than a day,” according to Lee, who noted the issue is purely one of desired target. “We have not seen, in any way, an adversary with the intent to adapt it to work on the American power grid, but if they wanted to, it would be a very trivial thing to accomplish technically,” he added.

Based on the research from Dragos and ESET, the Department of Homeland Security’s United States Computer Emergency Readiness Team (US-CERT) issued an official alert, reporting that the tactics, techniques and procedures (TTP) used in CrashOverride could be modified to target U.S.-based entities, and could also be adapted for industrial control system (ICS) settings beyond electric power. Thus, the department urged, “all critical infrastructure organizations should be evaluating their systems to susceptibilities in the TTPs outlined.”

Lee is most concerned, however, by the aggressive escalation this malware represents. Over the past five years, hacks of critical infrastructure have evolved from identifying sites to stealing information to shutting down a nation’s electric grid. Even in the past two years, attackers demonstrated consistent advancement: The 2015 attack on Ukraine’s grid left about 230,000 people in the dark, but that feat required a team of about 20 people penetrating utilities’ networks and manually switching off power to electrical substations. The 2016 attack was fully automated, programmed to work directly on grid equipment to control the flow of power without requiring human intervention or management. As a result, blackout attacks could be performed more quickly, with less effort and fewer people.

From Defensible to Defended

Experts assure that there is no need for panic, however. Hackers are not going to cut power to a whole country in the near future. ESET and Dragos estimate that a reasonably sized adversary operations team could impact 12 to 15 sites in a coordinated attack, resulting in hours or even a couple of days of interruption. These stations could be selected to cut off power in a specific region, cause limited interruption in multiple cities, or target a specific industry. But at the end of the day, CrashOverride is not capable of bringing down an entire nation’s grid.

Indeed, even thinking of it as the power grid is a bit misleading—there are many grids that make up power infrastructure, and that complexity builds in some robustness. Lee was quick to say the electric grid is in a relatively good position in terms of overall defensibility. While that does not necessarily mean it is fully secure, the existing resilience helps ensure that most scenarios of long, widespread blackouts are extremely improbable.

Experts have varied opinions on the threat of hacking critical infrastructure, but much of the disagreement focuses less on the possibility than on timeline, scale of attack, and magnitude of damage or disruption.

Targeting civilian infrastructure marks a notable escalation in the aggressiveness of nation-state actors. What’s more, Lee pointed out that the incident marks a break with the historical worldwide understanding that public infrastructure was, to some extent, off limits, at least without drawing international ramifications like sanctions. Indeed, electric grid compromise scenarios have mainly been confined to active conflict zones and limited in scope. However, the increases in technical sophistication, the automation of what has historically required considerable manual effort, and the rate at which attacks and technical advancement have occurred all mean the risks posed by such systemic cyberattacks should be on the radar for utilities, businesses and individuals.

“Our adversaries are getting much more aggressive, so that defensible position we have now is a nice upper hand, but we need to take advantage and do something about it,” he said. “There’s a way to move from defensible to defended.”

Preparing for the Inevitable

In a survey of top information security professionals gathered for this year’s Black Hat conference, 60% of respondents believed that a successful cyberattack on U.S. critical infrastructure will occur in the next two years. Only 26% are confident that U.S. government and defense forces are currently equipped and trained to respond appropriately.

Hacking critical infrastructure requires skill and resources that are primarily limited to nation-state attackers, otherwise known as advanced persistent threats. Battling such highly sophisticated and determined actors makes the risk utilities face fundamentally different from the threats to other sectors. Few retailers, for example, will ever draw the full force of a well-funded nation-state. From possible damage to the expensive equipment in industrial environments to business interruption for the many enterprises that rely on an electric entity, there are many operational risks, but the headline risk can be far more dangerous and is hard to truly control.

Soubaghya Parija, the New York Power Authority’s chief risk officer, considers an attack that causes outage inevitable, and reports the NYPA is currently preparing for such an incident to happen at any time. “The risk scenarios are highly probable,” he said. “We’ll be directly impacted. That’s not a question.”

Operating from the assumption that an attack cannot be prevented, the power authority is focused on how to best respond when one eventually strikes. “Our goal is to contain disruption of the most critical systems and processes to the minimum period possible,” Parija said. “We’ve defined the maximum tolerable time that the system could go out and still function, so then it is a matter of identifying the most critical systems and processes and setting controls around them.”

Managing the risk of being hacked thus comes down to preparedness and response, not prevention. The closest any entity in the utility space can get to a vulnerability management program, he believes, is refining the best business continuity, crisis management, and communications plans to address both customers and industry or government partners.

Monitoring the network for abnormal traffic can help raise a red flag when CrashOverride is sending messages to switch breakers, but network visibility or threat intelligence are not enough to ensure the security of critical infrastructure. Some of the impediments to mitigate the risk of hacking the electric grid are systemic.

Systemic Challenges, Strategic Investments

Generic IT best practices are insufficient to secure ICS—these systems really require specialized education in what security means in that environment. But as in many other sectors, utilities face the challenges of a talent gap and an aging workforce. With retirements, turnover, and increasing reliance on short-term contractors, fewer employees have the institutional knowledge that comes with experience in these unique industrial settings. According to Parija, a significant number of NYPA workers have been there less than five years.

To strengthen existing talent, Parija said the power authority is dedicating significant attention to the risks posed by careless insiders, educating staff on best practices by, for example, conducting phishing tests and issuing reminders not to leave passwords on desktop notes. He recognizes, however, that these only mitigate risks from within.

“The problem that we and other utilities face is that, yes, we can put the foundational protections and all that in place, but we will never technologically match a state-sponsored cyberterrorist outfit because that is all they do,” he said. “All day long, teams of people, they are not after money, they are focused on creating headline news, trying to disrupt our critical industry.”

A cyberrisk profile like the NYPA’s requires different approaches to management—and different definitions of success. “Typically, the way we do risk management is to understand the risks and put mitigation strategies in place against that risk. With cyber, the threat scenarios are constantly changing, and it is very difficult to put something in place for or manage the unknown,” Parija said. “With so many unknowns, we cannot prepare for these risks appropriately. Instead, what we are doing is spending more resources on discovery and resilience as opposed to reducing the threat of the vulnerability.”

Lee agrees that the right focuses for any electrical entity are how early, how fast, and how confidently you can detect an intrusion and how well your plans actually mitigate the risk and reduce the recovery time required in practice. In a presentation on CrashOverride at this year’s Black Hat conference, he urged utilities to study the incident not to guard against the malware itself, but as a specific scenario for robust crisis-response planning. Ideally, entities should be running a tabletop exercise to examine what CrashOverride—and any other incidents in future—would mean in their facility, then developing a detailed playbook of emergency procedures to ensure a faster, more effective, more reliable response and restoration of operations. Key questions Lee suggested include: Do you have the right coordination between the teams that need to be involved? Do you have the right buy-in from leadership to be able to respond effectively? Do you know how you are going to respond from both a technical and an operational perspective? How will employees in the field who are tasked with restoring power work with the security team identifying the problem to determine if an event is the result of an attack or just a random outage?

Training and education must also be a greater focus—indeed, Lee believes that education is the only real long-term solution. However, he said, this cannot be basic IT training or phishing tests. As the management of cyberrisk is significantly different in ICS settings than at the enterprise level, utilities should consider investing in their workforce by taking what he called a “trade-school approach,” teaching employees about the threats and mitigation strategies specific to industrial environments. In doing so, this approach should also improve cooperation and communication between security and operations teams.

The Cavalry Is Not Coming

As in other industries, there have been strides in developing threat-sharing initiatives among utility companies and relevant government bodies. Parija noted the NYPA’s chief information security officer has the highest level of security clearance and engages regularly in threat briefings at the state and federal levels, and he believes that sharing intelligence among sister agencies and industry partners has been beneficial.

When Dragos and ESET publicly released detailed reports on CrashOverride/Industroyer, government entities like US-CERT issued industry alerts, amplifying the warnings of risk to critical infrastructure sectors and disseminating information about indicators of compromise and recommendations to mitigate risks to ICS. Currently, these entities play a significant role in spreading the security industry’s research and recommendations for industries like utilities. Existing federal, state and local government grants are also available to help fund investments in better security, and Lee suggested the government could use regulation and tax credits in the future to incentivize further improvements.

These government efforts are far different from active threat intelligence and emergency response services, however, and some in the power industry may have unrealistic expectations about what aid will come, even with their critical infrastructure status. As a result, some utilities may not be monitoring sufficiently for current vulnerabilities, identifying emerging threats, or planning appropriately for the challenges they will face when a cybersecurity crisis strikes.

Thus, Lee cautions against relying fully on government intelligence-sharing or emergency intervention. “Politically, threat-sharing sounds really good, but in truth, all that data has been horrible,” he said. “The government doesn’t have the data, it isn’t responding to ICS incident response cases out in the field, and it does not have the resources to do so.”

Rather, it is incumbent upon utilities and private industry to assess and prepare for these threats on their own. This can and should take the form of detailed tabletop exercises, business continuity planning, assessing whether internal resources are sufficient to boost enterprise security, and if supplemental work is needed from private firms, making those improvements as soon as possible.

“Whatever the answer is, there can’t just be some bullet in a playbook saying ‘call the government,’ because the cavalry’s not coming,” Lee warned.

Cybersecurity Is Everyone’s Business

A broader concern for utilities is that the costs that would come with best securing their operations were never factored into utility pricing models. Now, when it comes to financing robust risk mitigation, most simply do not have the money.

“Local power companies were never charging customers based on the fact that they’re going to have to defend themselves against Russian, Iranian and Chinese nation-state attacks,” Lee said. “The big energy companies are making those investments regardless, but the real impact is going to be at the level of local municipalities and co-ops. Since those are the ones communities rely on most, we should realize that times are changing and we might need to see increases in rate prices.”

Parija noted this challenge as well, suggesting it put something of a cap on the amount of money and resources an electric entity can really dedicate to cyberrisk. “At the end of the day, this is not our core business—our core business is to produce power and supply and distribute power,” he said.

While Lee agrees, he does not believe that can be the final word. “Ultimately, there needs to be a change in understanding that cybersecurity is part of everyone’s business now, especially in utilities, and they need to figure out ways to make that effective,” he said. “It might be partnership with the government in terms of tax credits, it might be rate increases to offset the cost, but regardless of the solution to funding, cybersecurity is core to everyone now.”

That change in perspective is not just confined to enterprises in the utility sector. “This idea that there is non-critical infrastructure is kind of silly,” Lee said. “It’s all interconnected, so in a way, these threats to large swathes of our infrastructure are impactful to everybody, period. The trickle-down effect is increased risk to everyone, and we need to appreciate the fact that investments in infrastructure benefit everybody.”

Ultimately, the impact of hacking the electric grid is not about the traditional measures of a “successful” attack, namely actual disruption or damage. The United States is not going to go dark from coast to coast, but it does not have to for hackers to have a massive impact and undermine public confidence in the security of core institutions.

“A couple things go into this that get problematic and potentially alarming,” Lee explained. “1. It doesn’t need to be an attack on critical infrastructure to be critical to local communities. People could take advantage of local events or take advantage of hitting multiple industries at the same time, and be very impactful. 2. There’s a very large psychological impact. Even if it was only six hours of outages, if somebody hit D.C., New York, San Francisco, Miami and Houston at the same time, that would cause a difference in the way people think, the way they vote—it’s going to have political impact. It has huge psychological implications for the United States or any country.”