On December 13-15, 2017, a group of 27 individuals met at the Pocantico Conference Center in Pocantico Hills, New York, courtesy of the Rockefeller Brothers Fund, for an intensive conversation about electric grid security challenges and opportunities.

The group consisted of leaders and subject matter experts from industry, research laboratories, philanthropy, and non-profit organizations. Sharon Burke of New America, a national non-profit think tank and civic enterprise, co-chaired the meeting with Miranda Ballentine of the Rocky Mountain Institute, a non-profit energy research and advisory organization, with support from Mark Dyson of the Rocky Mountain Institute, Wilson Rickerson and Michael Wu of Converge Strategies, and Braxton Bridgers of New America.

The following document includes a section of findings based on the meetings, followed by summaries of the plenary discussions and keynote speeches. Sharon Burke compiled the findings and summaries based on meeting notes and inputs from Mark Dyson, Wilson Rickerson, Michael Wu, and Braxton Bridgers, as well as observations from Miranda Ballentine. The findings reflect Burke's judgments and the ideas of the other organizers, but not necessarily the event participants. The plenary discussions, keynotes, and breakout working meetings were "not for attribution," so these summaries are a composite and may not be attributed in part or in whole to any individual who attended.

Although the attendees at the Pocantico retreat did not explicitly arrive at findings, the following key themes and observations were gleaned from their discussions.

There is an urgent need for a common threat picture.

The threat environment for the North American grid system is changing rapidly, which presents significant risks to public health, safety, and prosperity. Yet the utility industry lacks a shared understanding of the threats and hazards for the modern American grid, particularly when it comes to cyber. In addition to technological change, there is a significant amount of misinformation on grid threats, including deliberately “weaponized” misinformation. And while government entities, such as the Departments of Energy and Homeland Security, and industry organizations, such as NERC, the Edison Electric Institute, and utilities, play an important part in sharing threat information, there is neither a uniform assessment of the threat ecosystem nor uniform access to threat information, intelligence, or analysis by both government and industry. A handful of executives at large investor-owned utilities, for example, may have sustained access to classified government threat information and analysis, while small, rural cooperatives may not have access at all, even though they may experience a similar or greater risk exposure (i.e., they may have fewer resources to mitigate or prevent an attack). Also, there are common shortfalls in understanding supply chain vulnerabilities and interdependencies across sectors (e.g., natural gas, water, and communications). Moreover, there is a need for a common understanding in the public domain, one that can be shared with regulators, local governments, and ratepayers.

• There is a strong need for concerted, systematic government-produced warnings, indicators, and assessments, with extensive outreach to the utility sector. At the same time, the sector could also benefit from a companion, independent assessment process that can be broadly shared within the industry and even with ratepayers.

The Federal government should clarify roles and improve threat information sharing.

A number of Federal and regional agencies and organizations have responsibilities for electric grid security, particularly when it comes to cyber threats, meaning there are overlaps and gaps. The Federal government should reaffirm a lead mission owner for grid security, or at least for cyber security, and designate other agencies as supporting partners.

• As specified in statute (the FAST Act and Federal Power Act), the U.S. Department of Energy is the Sector Specific Agency for grid security. And while the Department of Energy has considerable subject matter expertise and the trust of the industry in many ways, the agency’s mastery of the grid security mission, particularly when it comes to cyber, remains unproven. The Department will require adequate staffing, technology support, innovative policies, effective programs, good leadership, and support from other agencies (including CYBERCOM) if it is to succeed and gain the confidence and ethusiasm of private sector partners when it comes to grid security. In particular, the Department of Energy should focus on collecting and sharing credible, thorough, and actionable information and threat assessments – and in blue sky times, not just during emergencies.

Innovation is not just about technology.

There is a range of promising technologies for improving grid security, from microgrids to smart, autonomous control systems to redundant transformers. In many cases, however, the innovation context, such as the policies, regulations, legislation, standards, workforce capability, and best practices lag the technology itself.

• Government agencies and utilities separately and in cooperation should look for ways to promote a culture of agility, rather than constantly layering on the next coat of new policies and regulations.
• Any new technologies added to the grid should incorporate security intrinsically to the greatest extent possible. Retrofits are generally more expensive and less effective.

State level policy, regulation, and legislation are important.

There isn’t one grid, of course, but rather a system of interlocking pieces largely overseen and operated at state and regional levels. This decentralized system can be a weakness, but it is also a strength and certainly a fact of life.

• The groups that convened this event or other related groups (e.g., Protect our Power) should consider follow-on discussions focused on state and regional policy, regulation, and legislation to promote grid security.
• “N-Groups” such as the National Governors Association, National Association of Regulatory Utility Commissioners, National Association of State Energy Officials, and the National Emergency Management Association are powerful convening bodies to bring together the appropriate state and regional officials for national dialogue to share best practices and create a degree of uniformity in how each state maintains their regulatory and policy environment to protect the grid, promote reliability through sound investment, and prepare to respond to grid emergencies more effectively.

The Department of Defense presents opportunities.

The Department of Defense is at once a major consumer of electricity and of threat information, with the country's lead responsibility for national security. It is also a customer with both significant resilience needs and uneven resilience practices.

• The Department of Defense has the potential to be a proving ground for grid security as a large consumer with a public mission, an urgent need for improvement, and unusual authorities (such as 30-year Power Purchasing Agreements). Conversely, NERC, utilities, trade associations, and regional transmission organizations should be more proactive about engaging the Department of Defense, as a major customer and potential target for grid attacks.

Define resilience.

Resilience is one of those qualities that is often discussed but rarely understood, at least in tangible terms. It is important to have a more actionable definition when referring to the electric grid, however, including how "resilience" might differ from "reliability," which the industry understands very well, indeed.

A Field of Silos

In the opening session, participants identified the key grid security concerns they hoped to discuss at the retreat. A consistent theme was the fragmentation of policy, planning, technology implementation, and even knowledge, with participants calling for proactive prioritization, harmonization, and integration across the electricity enterprise, from rural co-ops to large investor-owned utilities to government officials and researchers. That continuum of industry players has robust and mature partnerships when it comes to dealing with natural events, but dealing with direct national security threats is new, and those relationships still need to develop. Indeed, most participants said there is a lack of clarity about threats to the grid and inadequate preparedness, particularly when it comes to state-sponsored cyber attacks. “We need a better understanding of our own threat landscape and vulnerabilities,” one participant noted, “there’s a lot of hype.” There was a sense of missed opportunities “to mitigate fear” in public opinion, born of bad information, or to incorporate security intrinsically in new technologies, including renewable energy. A number of attendees pointed to a “compliance mentality” as a barrier to progress. One participant warned that without taking a holistic approach that accounts for interdependencies, we are only building “fake resilience,” which left others asking for a better, actionable definition of resilience. Money was a leitmotif throughout this session, with wide agreement that however we define resilience, it will take investment to achieve it — but who will pay? Everyone expected important lessons out of Puerto Rico.

Not So New Directions?

“Critical infrastructure is an afterthought until it doesn’t work,” one expert said, which sums up this session’s discussion about the (bipartisan) tendency toward reactionary policy, politics, and legislation when it comes to the electric grid. Indeed, one participant pointed out that the U.S. Congress actually tried to outlaw blackouts after the 2003 power outage. In that regard, no one at the meeting expected the Trump Administration or 115th Congress to be radical departures: there is no sign at this time of a coherent national electricity policy. Not everyone at the meeting thought the lack of an overarching policy was a bad thing, however; some cautioned against overly prescriptive or panacean policies that lag the marketplace, the threats, and the technology. It’s hard to legislate adaptability.

At the same time, the group saw an important role for government in grid security, which everyone agreed is a non-partisan concern. As one expert noted, squirrels are a utility responsibility, but nation-state threats are a federal government responsibility. Some participants said that current grid security-related regulations are inadequate or accepted with dangerous complacency. Others cited the difficulty of private sector information-sharing on threats and vulnerabilities. But no one was certain who in the U.S. Government is the “mission owner” for grid security, particularly for cyber-related threats and multi-sectoral vulnerabilities (i.e., natural gas, communications). Solutions offered to the political and policy shortfalls ranged from prioritization of critical nodes and threats, to a better planning regime, to faith in decentralization. “One should not be so quick to think that 50 states should be homogenized,” one expert warned.

The Threat Spectrum: Squirrels, Storms, and Nation States

*Please note that although we are sharing a summary of these remarks, the speaker was off the record and should not be cited by name.

The keynote speaker, a former government leader, described a “threat spectrum,” which ranges from routine natural hazards, such as severe storms, to sophisticated, deliberate attacks. This spectrum is not static: events that used to be unlikely and manageable are becoming more likely and unmanageable, from the heavy precipitation in Houston to wildfires in California to the cyber attack in Ukraine.

Russian interference in U.S. elections should be a wake up call: with a clear doctrine of early escalation in order to deter adversaries, the Russians will and can go after infrastructure, nor are they the only nation gaining such capabilities. At the same time, we have to balance that growing threat against the considerable misinformation or disinformation about threats to the grid. How do we know what’s true?

The complexity and interdependencies of critical infrastructure are part of the challenge, which means that any one element of the grid system is only as secure and resilient as its weakest link. Communications, for example, were a weak link in the 2013 Metcalf attack, in Ukraine, and in Puerto Rico. The supply chain itself can be a weak link.

The speaker cautioned that whatever the benefits of digitization, there are also risks in “engineering out the human element.” In Ukraine, the operators really knew their system and were able to troubleshoot and mitigate the damage manually. In contrast, there’s a coming scarcity of experts in the United States, with a wave of retirements. Given this picture, the speaker wondered why security and resilience are “bolted on” as an afterthought, especially given increased interest in and momentum for smart technologies. The stakes are high, because while utilities and the public sector are good at dealing with hurricanes, we’re not ready for truly high consequence events with effects across sectors. It’s time to use a public-private partnership to create a framework for agility.

New Threats, Hazards & Risks for the U.S. Electric Grid

The session started with a summary of remarks from the previous evening, largely that the electric grid is a complex, highly interdependent ecosystem in a state of change and that the country is not ready for a major event.

Participants agreed about the difficulty of charting the new threat landscape, ranging from lack of hard science, to knowledge gaps, to interest groups with a specific focus that catalyze Congressional action. There should be a standard way of assessing threats, potential system impacts, and factual, data-driven assessments of response strategies. One expert shared the draft Electricity Subsector Coordinating Council (ESCC) threat matrix, which took about a year to develop and will ideally incorporate cascading effects and multiple sectors. Participants also thought there should be a gap analysis. Another expert quickly countered that the biggest threats are still squirrels and trees, and that misinformation or even “weaponized” information remains a significant concern. One participant underscored the need for a common operating picture and threat identification before contingencies occur, suggesting that some brand or seal of approval, such as a “Cyberstar” akin to an “Energy Star” is worth considering. There was discussion about GridEx as a diagnostic tool for gaps, with wide agreement that this public-private exercise is very helpful, and a few observations that it could be improved (e.g., the event is generally in November, using regional, real-time weather data, which is not necessarily the most stressing case).

Several participants noted that top industry executives are committed to being more proactive, often with access to classified information, which helps the industry understand how to target investment in research and risk mitigation. That may not extend to all parts of the industry, however, and there were concerns that Investor Owned Utilities (IOUs) may have better threat information than municipal or cooperative utilities.

There was both discussion and disagreement about the role of the Department of Defense in grid security. Many participants expressed confidence in DoD; others cautioned that DoD should not be in charge of restoration or threat information sharing. Defense missions may actually compete with restoration priorities for transportation nodes and other resources during a crisis.

New Technologies, from Smart Grids to Generation

Experts from four national laboratories presented on innovations to improve grid security, followed by a group discussion. Resilience was a theme throughout the session, though one participant noted that the term needs a more tangible, actionable definition. Keeping up with determined adversaries is difficult, if not impossible, so inherent resilience is important, but not common in the legacy system. A utility executive noted that nuclear generating stations tend to be an exception, with extensive systems-level resilience built in.

One key area of research is in better diagnostics. With cyber threats, for example, when a threat or anomaly is detected, the current practice is generally to slow traffic flows. That can actually exacerbate the consequences, such as denial of service. Approaches to improved diagnostics include hardware-in-the-loop, rapid prototyping, and modeling virtual “playgrounds” or “walled gardens” to simulate power systems, power electronics, various perspectives on threat, and interdependent systems, without actually “flipping the switch.” One researcher highlighted the promise of machine-to-machine automated response, when microseconds matter.

There was agreement about the importance of collaboration when it comes to innovation. The Grid Modernization Lab Consortium (GMLC), for example, streamlines and pools the resources of 13 national laboratories, looking at areas such as reducing the time and cost to integrate new technologies, “plug and play” technologies, standards that will improve recovery times after an outage, and security resilience metrics. Utility executives also hailed the ESCC and NERC’s Electricity Information Sharing and Analysis Center as key private sector cooperation mechanisms and proposed community reliability assessments.

Throughout this session and the entire event, there was a great deal of discussion about decentralization in the grid system, and microgrids, in particular. “Whether or not we decide microgrids are the solution, that is where the momentum is,” one expert commented, with others noting that technology companies and the U.S. Air Force, in particular, are moving to microgrids. Shifting from centralized systems to decentralized systems could improve grid security in some ways and threaten it in others (e.g., a decentralized grid may provide an adversary with more discrete targets for a cyber attack, but disperse the effects of any one attack; it may eliminate a single point of failure of a centralized system and increase reliability, or it may introduce multiple points of failure with more reliability problems for smaller groups of customers). There was recognition that there needs to be more R&D focused on networking assets to provide resilience across the system, especially with distributed energy resources. Puerto Rico is a tragedy, but also an opportunity for this kind of innovation.

Another key theme for this discussion was the importance of a systems approach. The system complexity of electricity is increasing every day, with steady shifts in variability and supply, which often requires, “coordinating flexibility to offset vulnerability” according to one expert. Participants acknowledged that security is not necessarily a technology challenge: there’s a need to innovate and integrate policy, economics, and regulations as well. At the same time, one executive cautioned that it’s very important to understand those needs from “an operational perspective.” The current tendency is to focus on components, not the system, but one cannot make meaningful change at the device level. Unfortunately, the industry currently lacks the capability or capacity for comprehensive system analysis, which would better identify gaps and coordination of investment decisions.

In the discussion of the role of consumers in driving or accepting innovation, there was much discussion about the Department of Defense, as a particularly big customer and big target. This is also a customer with poor visibility of its own needs, costs, and status of reliability (i.e., all bases are required to have backup generators, but any one base may not know how many they have or may not keep them well maintained), and that’s a barrier to innovation. The FAST (Fixing America’s Surface Transportation) Act required understanding of the electricity vulnerabilities of key military installations and other national security infrastructure, a goal as yet unrealized. One executive noted that DoD’s challenges are not pervasive and the agency is a special case; all utilities work directly with customers and know this type of information.

From Hacks to Hurricanes: the View from the Private Sector

*Please note that although we are sharing a summary of these remarks, the speaker was off the record and should not be cited by name.

The speaker, the CEO and President of a large public power organization, offered a complementary perspective (there were no other public power representatives in the meeting).

He was extensively involved in the response to Superstorm Sandy, which offered many lessons, such as the importance of prepositioned equipment and good data analytics. Given that there was an extended power outage, they had to deal with considerable political pressure to get the lights back on, as well. All of those lessons are now being applied in Puerto Rico. At the request of his Governor, the CEO sent a score of the company’s experts to the island in the immediate aftermath of the storm, collaborating with the U.S. Army Corps of Engineers on the Initial Damage Assessment. Although Hurricane Maria made landfall on September 20th, the request for mutual aid didn’t surface until October 31st, and the executive actually drafted the request himself. Puerto Rico undoubtedly needed the help: in addition to the age of equipment and the utility’s debt, there were no rights of way, no access, and no vegetation management. And it would be hard to overstate the damage — he thought there might have been micro tornadoes associated with the storm, based on what he personally saw.

The speaker talked about the plan for the recovery of Puerto Rico’s grid, a $17 billion request. 50 percent of the plan is non-traditional, including microgrids and renewables. Indeed, he thought that solar panels with storage always would have been the best way to provide power for remote areas of the island. The rest of the funding would be for hardening measures.

As for grid security more generally, the speaker said it becomes a matter of risk management. He talked about ground-based sensors that alert an operations center if someone is approaching equipment and called his strategy a “beehive defense — crunchy outside and sectional inside.” He did stress that cyber “is the big one,” and that cyber-security should be built in whenever possible, as “it’s so much harder to bolt on later.” He also is very concerned about supply chain vulnerabilities. “But it’s the insider threat that keeps me up at night,” he asserted. This is both the case for deliberate acts and for workforce attrition. 35 percent of his personnel are eligible for retirement in the next two years, for example.

As for new technologies, the speaker highlighted automated distribution and wearables, including Google glasses, which give his engineers an “augmented reality.” He noted that drones have been very helpful in Puerto Rico, and that they are experimenting with submersible equipment (including submersible transformers) that can withstand flooding or other exposure to water. He considered his company lucky because so much of their baseload generation is hydropower, which is especially resilient and critical for black starts (he noted that hydro did not go down in the cascading regional blackout of 2003).

Finally, the speaker offered that a missing piece of grid resilience is coordination across sectors. Telecommunications companies, in particular, are less regulated than the electricity sector and technology companies are not regulated at all, and it’s hard to bring them to the table.

Scott Aaronson

Edison Electric Institute

Executive Director, Security and Business Continuity

Miranda Ballentine

Rocky Mountain Institute

Managing Director – Business Renewables Center

Roger Ballentine

Green Strategies, Inc.

President

Braxton Bridgers

New America

Millennial Public Policy Fellow

Sharon Burke

New America

Senior Advisor & Director

Caitlin Durkovich

Toffler Associates

Director

Mark Dyson

Rocky Mountain Institute

Principal

Sean Griffin

U.S. Department of Energy

Program Manager

Michele Guido

Southern Company

Business Assurance Principal

Bryan Hannegan

Holy Cross Energy

President & CEO

Tom Hassenboehler

The COEFFICIENT Group

Partner

Chantal Hendrzak

PJM

Manager – Operations Development

Dan Herway

Idaho National Laboratory

Director, DoD Energy Programs

Kelli Joseph

NRG Energy

Director, Market & Regulatory Affairs

Nick Judson

MIT-Lincoln Laboratory

Assistant Leader, Energy Systems Group

Erik Limpacher

MIT-Lincoln Laboratory

Group Leader

Arthur Lord

U.S. Department of Defense

Team Leader for Mission Assurance

Ron Melton

Pacific Northwest National Laboratory

Senior Researcher, Project Manager & Team Lead

Chris Murray

WGL Energy

Senior Business Development Manager

Steve Naumann

Exelon

Vice President, Transmission & NERC Policy

Sam Passmore

Mott Foundation

Program Director – Environment Program

Wilson Rickerson

Converge Strategies

Principal

David Roop

Dominion Virginia Power

Director – Electric Transmission Operations & Reliability

Juan Torres

National Renewable Energy Laboratory

Associate Lab Director

Michael Wu

Converge Strategies

Principal