Cities and other local governments are the core service providers for citizens and businesses. Ensuring the security of municipal systems is essential to ensuring basic safety, quality of life, and economic prosperity. Increasing digitization means some city services are now managed and/or delivered using technology. In the past, cities have established relationships with public and private sector partners to prepare for and respond to catastrophic events such as natural disasters or terrorist attacks, both of which can threaten the viability of normal operations and the security of the community.

At this juncture, however, efforts to build similar partnerships to respond to cyberattacks are still early stage in most jurisdictions, leaving cities around the country significantly less than well protected. This paper highlights ways in which cities are currently working with their federal and state partners, private sector companies, and nonprofit agencies and foundations to improve their cybersecurity and resiliency efforts.

There is great work being done now, but there are additional opportunities and policy changes that would increase the propensity, efficiency, and effectiveness of such cyber-partnerships going forward.

Recommendations for Municipal Leaders

Integrate and prioritize regular exercises and supporting activities into resiliency planning. This is a recommendation for city officials, their governments, and supporting organizations, but extends also to the federal and state officials working with local governments. This type of planning and activity is essential; it must be built in and prioritized in the support offered to municipalities.

Think regionally. Many cities lack the resources to handle cybersecurity challenges on their own. Even larger cities will be able to benefit from working with other governments and their related institutions in their region to pool resources.

Reform governance of cybersecurity issues. Cybersecurity should be a priority for city officials who should institutionalize that reality. There may be different models for doing so, depending on the type of city government and budgetary reality, but leadership should strive to decrease conflict of interest issues and increase high level visibility as much as possible.

Recommendations for Federal and State Policymakers:

Related to the recommendation above for local governments, federal and state policymakers should codify, exercise, and institutionalize federal resources with authorities to support SLTT organizations. Although there has been ongoing work in this area, more needs to be done, and many of the lessons learned need to be reflected in documentation, new or existing.

Provide better guidance around and address shortcomings in federal funding for cybersecurity. Existing funding mechanisms challenge local governments because of red tape, confusion, and competition with other homeland security-related threats. Federal efforts should be reformed to allow for and direct funding specifically to efforts for cybersecurity resilience and response.

Structure and prioritize federal and state outreach efforts to local governments. This activity needs to be strategic in nature and utilize the federal system of government; the federal government cannot do everything, but needs to link the outreach, service provision, and ongoing relationships between federal agencies and with sub-federal parts of government. State programs aiding municipalities need to be properly resourced, both in terms of outreach and execution.

In March of 2018, several agencies in Atlanta were forced to convert to paper as ransomware encrypted their computer systems. Ultimately, this malicious cyber event will cost the city over $17 million¹ to recover data, upgrade, and improve its systems and processes.

During the course of the incident and in its aftermath, Atlanta reached out to its public and private partners for assistance. Working together with technology experts employed by the city, they have helped get systems back online, protected others from susceptibility to similar failures, and made improvements that will increase security going forward.

Fortunately, the services impacted by the ransomware attack in Atlanta, while integral to a high functioning local government, were not critical in terms of life and death to its citizenry. There have been other cases, however, such as an infection in the Baltimore 911 system that occurred the same week as the Atlanta incident, that might have affected citizens’ basic safety. City governments and their affiliated agencies serve to ensure that all residents have clean water, emergency services, education, trash collection, and many other public works functions. If these municipal service systems go down, the security and, potentially, the very survival of residents would be in jeopardy.

Cities and other local governments are the core service providers for citizens and businesses. Ensuring the security of municipal systems is essential to ensuring basic safety, quality of life, and economic prosperity. Increasing digitization means some city services are now managed and/or delivered using technology. In the past, cities have established relationships with public and private-sector partners to prepare for and respond to catastrophic events such as natural disasters or terrorist attacks, both of which can threaten the viability of normal operations and the security of the community.

Ensuring the security of municipal systems is essential to ensuring basic safety, quality of life, and economic prosperity.

At this juncture, however, efforts to build similar partnerships to respond to cyberattacks are still early stage in most jurisdictions, leaving cities around the country significantly less than well protected. This paper highlights ways in which cities are currently working with their federal and state partners, private sector companies, and nonprofit agencies and foundations to improve their cybersecurity and resiliency efforts.

We will also seek to provide recommendations to cities and their partners on additional opportunities and policy changes that would increase the propensity, efficiency, and effectiveness of such cyber-partnerships going forward. While we focus on the American system of government, such lessons can be situationally applied to municipalities and provinces around the world.

Cities, whether in the United States or elsewhere, are incredibly diverse. This report is designed to address the needs of a range of municipalities; further, we will call out certain initiatives that may be more appropriate for cities of one size or another. The case studies will necessarily focus on larger cities, since it is those cities that so-far have had the most established and mature programs.

Still, cities with smaller populations, towns, or counties, can utilize the lessons learned in this paper and aim towards a more secure digital operation. All cities have the responsibility to provide reliable services to their residents, so the core mission remains the same. Add to that the idea that cities have a responsibility for the safety of their citizens²—online or not—and the cybersecurity mandate for cities grows greatly. Local governments, large or small, need assistance from their partners to fulfill that mission. In doing so, cities can also provide value in return, raising the overall security of the ecosystem in a way that should not be overlooked.

As former New York City mayor and civic issue philanthropist Michael Bloomberg has stated,

“Virtually all of society’s problems are problems that both originate in the cities and are being solved there…There are some things that can only be decided on the federal or state level—starting wars for example—but generally speaking, both the problems and the solutions are located in the cities… So finding solutions to these problems in one city will enable us to test them in other cities that experience similar problems.”³

Bloomberg, and other civic officials often expound that cities are the perfect laboratories for developing and refining solutions to critical and universal civic problems. Cybersecurity is and will remain an issue that impacts every citizen and business; cities and other local governments will be the key players in addressing the challenge of securing the digital world and ensuring the continuity of critical services.

Cities, towns, villages, and boroughs are all incorporated places in the United States. The definition of each, and even the existence of each, varies by state.⁴ Connecticut has no minimum population requirement to incorporate a city; conversely, the state of Ohio requires 25,000 citizens for the incorporation of a new city.

As noted in the introduction, this report generalizes the concept of a city to include a broad range of sizes. Very large cities, such as those with a population of over 1,000,000 (there are 398 in the world and ten in the United States⁵), have access to a wider range of talent, funds, and partners (such as businesses operating in the metro area), but also have responsibility for securing more data, residents, and businesses operating in their jurisdiction.

While there are federal grants available for all cities, it is important to note that only certain cities are eligible for federal assistance under the Urban Area Security Initiative (UASI) in the United States (though local communities receive 80% of the State Homeland Security Program (SHSP) grants given to the states). Urban areas’ eligibility is “determined through an analysis of relative risk of terrorism faced by the 100 most populous Metropolitan Statistical Areas in the United States.”⁶ There are other federal grants available for all cities; however, UASI grants are available only for eligible urban areas, with ineligible cities left to search out other funding sources. Federal funding and grant options will be explored in greater depth later in this paper.

As stated above, each city has unique characteristics, but in general, cities face similar problems. These challenges are not all confined to local government. Cybersecurity is at its core an architecture, management, and governance issue that is shared by other enterprises and institutions.⁷ However, because of the diverse and critical nature of municipal systems and services, budgetary and fiscal challenges, and the responsibility to provide quality and secure services to their residents, many cities continue to struggle with designing and implementing reliable means of forestalling and remediating a cyber attack.

This paper will review and analyze partnerships that have been demonstrated to assist cities in tackling this problem more efficiently and effectively. Subsequently, we will make a set of specific recommendations to policymakers in cities and at the federal and state level to further address this challenge.

Pressure to Digitize

As consumer society has steamrolled into the electronic age, there has been (and continues to be) pressure for government to digitize its services. This push towards digitization, however, came before the advent of affordable or widespread shared platform technology, pressing cities to develop and host their own applications and systems on proprietary infrastructure, often using hardware and software cobbled together over years or even decades.

Cloud-based systems are no guarantee for security, but if properly configured, they can help decrease risk. Transition to the cloud requires proper expertise in strategy, design, and implementation, and careful personnel management and training. To make the change, urban information security experts must recognize the advantages the cloud-based systems can offer, make the case for the expense of transitioning, and secure funding for all the elements needed for successful implementation.

As a result of these barriers to entry, most cities are still battling legacy systems, some of which have been patched together and/or so heavily customized that they make transition extremely costly and complicated. Some of these systems power critical infrastructure and safety services, further complicating any maintenance or transition period.⁸

Personnel and Authorities

As all organizations, private and public, large and small, are competing for the relatively small pool of skilled and available information security personnel, cities and other local governments struggle to provide competitive salaries and attract qualified talent. Although public sector institutions can attract talent with offers for training opportunities or other perks, they often struggle to retain that talent in the long term.⁹

Additionally, many municipalities lack a Chief Information Security Officer (CISO) position or have only recently created such a position. Even if a city did not have a CISO in the past, work was being done to protect the city’s information systems; however, it does make it more likely that there were fewer resources (personnel, money, and time) invested in that effort and that those activities were pursued in silos. Among the ten largest cities in the United States, the average age of the CISO position is five years, with the most recent having been created in 2018.¹⁰

Building a constituency in an administration and getting buy-in for security takes time and persuasive power, as does the creation and improvement of incident response protocols and procedures. Each city may take a different approach to addressing the need for information security, but having a centralized and empowered executive focused on that direct issue is essential for success.

Budget

There is no comprehensive or targeted data on cybersecurity challenges for cities as there is for states,¹¹ nor is there public information about funds requested by local governments to federal agencies. Anecdotal evidence as presented in interviews conducted for this paper is that budget remains a serious issue. The discrepancy between funds needed and available dollars for IT modernization, configuration, monitoring, and migration is one factor, but so is the ability to plan both short and long term, and adapt to changing technologies and the ever-evolving threat environment in cities where mayoral administrations and city council governance change frequently.¹²

The concepts of security, resiliency, and recovery will be used throughout this paper. It is worth defining these terms at the outset for the purposes of clarity and conformity.

Presidential Policy Directive 21 Critical Infrastructure Security and Resilience defines both security and resiliency as follows:

1. Security—reducing risk by physical means or defensive cyber measures.
2. Resiliency—the ability to prepare for and adapt to changing conditions and withstand and recover rapidly from disruptions. Resilience includes the ability to withstand and recover from deliberate attacks, accidents, or naturally occurring threats or incidents.

Recovery, as referenced above, is the final element of resilience—what happens after an event has occurred to restore services to an acceptable state.

Potential partners for cites fall into five categories—federal, state, nonprofit, private sector, and peer. Cities can and should take full advantage of the benefits that can come through engaging with organizations in each category, and will inevitably forge unique reciprocal relationships with each as well. This section will highlight a select number of ongoing partnerships, using examples from cities that are moving ahead of the curve in this direction.

The federal-city suite of partnerships is most robust in the areas where the federal government has teams deployed locally; for services and relationships that are controlled centrally in Washington D.C. or its environs, affiliations are more ad-hoc. There are thousands of cities in the United States and many more small local municipalities; coordinating and cultivating relationships with each of these local governments from Washington is unrealistic. With locally-driven federal cybersecurity efforts still evolving, there have been some real success stories, but work still remains in how to best assist cities and local governments, facilitate the delivery of services, and make them more secure and resilient.

Incident Response

The most immediate assistance the federal government can provide to a city that has experienced a cyber incident is with direct response services. Per Presidential Policy Directive—United States Cyber Incident Coordination (PPD 41)¹³, these services are grouped into three categories:

• Asset Response: furnishing technical assistance to affected entities to help them recover from the incident. This effort is led by the Department of Homeland Security (DHS) through the National Cybersecurity and Communications Integration Center (NCCIC).
• Threat Response: investigating the crime associated with a cyber incident. This effort is led through the Department of Justice (DOJ), through the Federal Bureau of Investigation (FBI) and the National Cyber Investigative Joint Task Force (NCIJTF).
• Intelligence Support: creating situational awareness about cyber threats. This effort is led by the Office of the Director of National Intelligence (ODNI), through the Cyber Threat Intelligence Integration Center (CTIIC).

PPD 41 sets down these roles for significant cyber incidents only, but in practice, this division of labor has extended down to any instance in which a city, state, or other entity requests federal assistance with a cyber incident.

In 2014, the Federal government released guidance to State, Local, Tribal, and Territorial (SLTT) organizations for how to properly report a cyber incident. The instruction breaks down the various types of cyber incidents and encourages SLTT incident managers to contact the most relevant federal entity.¹⁴ This approach can be confusing, however, as it may be initially unclear as to which type of incident has occurred.

The federal government currently responds to incidents from a variety of intake points. Still, DHS has promoted the concept of the NCCIC as a “911 for cybersecurity.”¹⁵ On the face of it, this idea has merit. Having one contact point for the federal government makes it easy for SLTT governments to reach out if they need assistance. Nonetheless, in practice, having such a policy would be hard to enforce. FBI agents are in the field and maintain regular contact with their SLTT partners, as do Federal Emergency Management Agency (FEMA) personnel in each of the 10 FEMA regions. It is no surprise, therefore, that many incidents are reported through the FBI or FEMA, a situation that is unlikely to change without significant active and persistent presence from DHS personnel.

Federal partners provide valuable assistance to cities, free of charge, but they are unlikely to be the only service provider during an incident. The role of federal responders, particularly from DHS, can vary greatly depending on the incident and the needs of the city's incident response team. In some cases, they provide hands-on remediation assistance, but in others cases provide only remote assistance or advice. Trust issues sometimes also limit local governments’ willingness to call in the federal government because of political concerns or misunderstandings around disclosure policies (Freedom of Information Act).

Within DHS, the role of FEMA in cyber incident response is still under evaluation. There is a report being produced by RAND¹⁶ in cooperation with FEMA that will undoubtedly provide some insights, but the practical manifestation of those decisions is still being worked through.

Resiliency

The closest and most robust relationship between the federal government and cities around the country exist in the fusion centers.¹⁷ In these facilities, city, state, and federal law enforcement officers sit side by side. The operational, bidirectional sharing facilitated by co-location is demonstrably useful, but there still remains a gap with regard to sharing insights and deterring future threats.¹⁸

In general, information sharing programs are confronted with the same barrier to efficacy: getting relevant and timely information declassified and ready for tear-line dissemination. Facilitating more clearances among security professionals in SLTT organizations will ameliorate the situation, but intelligence must be actionable to be useful, and information that must remain within a Sensitive Compartmented Information Facility (SCIF) hinders its effectiveness. Furthermore, information that must remain within a SCIF is generally not useful on unclassified networks, which essentially all municipal networks are.

The new National Risk Management Center (NRMC) at DHS is still being developed after its establishment in the fall of 2018. So far, it has announced that its first project will be a hard look at positioning, navigation, and timing (PNT) technology, used in GPS. It has since kicked off several sprints around finance, electric, and telecommunications.¹⁹ The NRMC could create traction on and for SLTT organizations if it marshals the resources of the federal government to take a truly strategic look at threats facing the various sectors or if it could provide a model for SLTT governments to assess their risks. Those who are engaged in developing its next steps should not forget the equally vital role played by SLTT governments and oversight of critical infrastructure.

In addition to information sharing, DHS and Department of Defense (DoD) run both national-level and local exercises. NCCIC and FEMA have teams dedicated to this service, and provides it to local government agencies free of charge. Exercises are essential for the maturation of any cybersecurity program and its integration in a city's greater incident response program. While these services are offered by the private sector for a fee, having access to them gratis is a huge benefit to cities, which are often strapped for cash.

FEMA is developing a cybersecurity preparedness guide,²⁰ but it has yet to be released.

Funding

FEMA is also a key partner for cities when looking for funding for cybersecurity-related projects. The 2018 Homeland Security Grant Program (HSGP) Notice of Funding Opportunity (NOFO) mandated that two of its three programs must include investment for cybersecurity.

Unfortunately, the specific amounts cities have requested, received or spent to date from this program are not publicly available. Anecdotally, we have found that some cities have used grants from the HSGP to fund cybersecurity projects. San Diego, for example, used Operation Stonegarden²¹ to fund its intrusion protection system. Meanwhile, other cities (and states) have struggled to understand which cybersecurity related activities might be eligible for HSGP funds. This issue is further addressed in the recommendations section below.

As state cyber programs mature, many states are reaching down to their local governments to build partnerships and offer services. While these overtures are positive, they can be complicated by existing political tensions between state and city governments and by resource constraints (states face challenges to staff and fund programs to manage their own cybersecurity, let alone offer services to local governments).²² Even in the face of these challenges, some compelling programs have been established that demonstrate how cities and states can work together on this problem set, and offer insight into how such state and local partnerships may evolve in the future.

These programs generally fall into two categories: resilience and response. Resilience activities can include assessments, exercises, and consulting services. Response services can include forensics or recovery services.

One of the most critical issues facing local governments is accurately assessing which elements they are missing and the current state of their cybersecurity programs. This resiliency service is offered by DHS, as described above, but that department does not and will not ever have the capability and staffing to handle all of the demand from local governments across the country. Some states have stepped up to offer similar services with a focus on their own local governments.

One of the most critical issues facing local governments is accurately assessing which elements they are missing and the current state of their cybersecurity programs.

In 2018, Michigan piloted a “CISO as a Service” program, offering assessments and advice to nine local governments.²³ The state is currently working to find a sustainable funding model to avoid annual appropriations challenges, but the success of the program yielded enough for the state’s information technology (IT) leadership to support it going forward.

The state’s IT office used a chargeback model to pay for the assessments at a cost lower than standard for private sector services. This kind of cost-savings is important to any state program’s success, as is the neutrality of the service providers; most of the counties requesting help from the state in this scenario were resource challenged.²⁴

This trend is not unique to Michigan. Georgia CIO Calvin Rhodes stated that his state’s IT and security offices are more likely to hear from small governments than large ones, in that larger cities tend to have their own staff. Still, the state offers consultation to all municipalities upon request, and pre-existing IT contracts that local governments can utilize for general purpose or incident response needs.²⁵ During the Atlanta incident, the city did not reach out to the state for assistance, but relied on its private sector and federal partners. Without comment from the Atlanta government, it is impossible to know if this is because they were unaware that such contracts were available, if they preferred their own vendors (existing or emergency), if they were unable to take advantage of them for some reason, or for another cause.

Other states, such as West Virginia, have also structured their contracts so that local governments can take advantage of them. In these circumstances, states have often failed to devote sufficient resources to outreach to ensure that local governments are aware of the existence of these programs, and that the contracts can be accessed by and for those jurisdictions.²⁶

New York and Virginia are taking a more proactive approach to providing assistance to their local governments. Through its Department of Homeland Security and Emergency Services, New York has begun providing vulnerability assessments evaluated against the NIST framework. Although those services cannot be linked to remediation efforts, they have started to deliver useful resources for county and local governments ahead of the wait time for similar DHS offerings.²⁷

Virginia is working through its emergency management agency to build relationships and raise awareness in local governments. They have held several meetings across the state in 2018 aimed at gaining a better understanding of the challenges local governments are facing. Virginia follows up with bringing local agencies’ representatives into the fusion center so that those connections can be strengthened and institutionalized.

Virginia also activated its National Guard to State Active Duty (SAD) status to perform vulnerability assessments and penetration tests on local government networks. In 2019, the Virginia government plans to hold additional exercises involving local governments in the exercise of the state emergency plan. Another initiative seeks to engage information security professionals across the state in the grants process through regional working groups, as the state’s funding for this type of activity is heavily reliant on homeland security grants.²⁸

Moving forward, the type of engagement that states such as Virginia have made will be vitally important to the success of state and local cybersecurity, especially given the emphasis on integrating site-specific IT and cybersecurity needs to the DHS FEMA grant process. Also encouraging are efforts from states such as Indiana, which has involved its local governments in the state’s Executive Council on Cybersecurity. This council’s mandate is to establish a strategic framework for Indiana’s future cybersecurity initiatives.²⁹ With more engagement and involvement from local governments, states can improve their policies, offerings, awareness, and long-term strategies to better serve to secure their cities, and thereby their citizens.

The Multi-State Information Sharing and Analysis Center (MS-ISAC)

The Multi-State Information Sharing and Analysis Center (MS-ISAC), a division of the Center for Internet Security under cooperative agreement with DHS, is a nonprofit entity formed to be the “focal point for cyber threat prevention, protection, response and recovery for the nation’s state, local, territory and tribal (SLTT) governments.”³⁰ Information sharing is central to the MS-ISAC’s mission; in addition, it provides services to SLTT governments to assist with incident response.

MS-ISAC is perhaps most known for its network intrusion detection system (NIDS) sensors known as “Albert”—modeled after the “Einstein” sensors used to protect federal government networks. MS-ISAC can become involved in an incident response effort proactively, by noticing something through the Albert network, or reactively, if an SLTT organization reaches out to them for assistance.³¹

MS-ISAC’s incident response team is designed to work remotely, but in rare circumstances deploys on-site with a customer. The incident response (IR) team concentrates on remediation, as does the DHS NCCIC’s Hunt and Incident Response Team (HIRT), and sometimes operate alongside that team or others from the FBI, Secret Service, or private sector.³² There is no set threshold for the MS-ISAC team to accept a case, but the decision regarding on-site or deployed status is up to that organization’s COO (currently Tom Duffy) and the affected entity. Although the team is able to deploy quickly, there is sometimes a delay if the team has not been previously approved by the entity’s insurance company.³³ As described above in the section on private sector and below in the recommendations section, it is essential to involve all stakeholders in incident response preparations. Doing so resolves red tape and administrative issues before an incident occurs, instead of dealing with them while in crisis mode to the detriment of efficient incident response.

Information Sharing and Analysis Organizations (ISAOs)

Although most cities belong to the MS-ISAC as the designated ISAC for state, local, tribal and territorial governments, many have also seen benefit from joining regional Information Sharing and Analysis Organizations (ISAO), which function similarly to the MS-ISAC, but are designed for non-federally designated critical infrastructure industries, or which are organized around another concept, such as a geographic area. The primary advantages of using a geographic base are that the membership can be cross-sector, helping to break through separate silos of information exchange, and that a smaller base of member operations can facilitate closer relationships.

Regional ISAOs have used these close relationships—built on a foundation of trust—to facilitate and mature information sharing programs that involve both the private and public sectors. The Arizona Cyber Threat Response Alliance boasts members from many regional cities. Its information sharing program anonymizes information so that its members can share back confident in their own protection.³⁴ Similarly, Los Angeles’ LA Cyber Lab has started receiving indicators of compromise from private sector companies that build a rich set of information for the city and the other members of the lab, and can be fed up to the federal government through NCCIC ingest.³⁵ L.A. is also the recipient of a 2018 DHS grant to turn the LA Cyber Lab into an official regional ISAO, institutionalizing its role in information sharing.³⁶

Universities

There are numerous examples of public sector institutions forming effective partnerships with universities. Partnering with universities has several benefits, including access to talent, technological insights, and resources, and the chance to match up real-world priorities with ongoing research.

Los Angeles is currently building several partnerships with its universities, including information sharing, data science and visualization, and workforce development. Most programs are still in their infancy, but have great potential to harness the technology capabilities and resources at the L.A.-area schools. For example, California State University, Long Beach is embarking on a project to build threat models and dashboards for L.A.’s Integrated Security Operations Center (ISOC).³⁷

The above-mentioned ISOC, part of the LA Cyber Lab, also processes feeds from universities that are partnered with the city and the other LA Cyber Lab members. L.A. is also looking to expand its internship programs to involve college and university students, increasing the skilled and trained workforce available to the city, while providing them with real-world experience.

Partnering with universities has several benefits, including access to talent, technological insights, and resources, and the chance to match up real-world priorities with ongoing research.

The state of Vermont has launched a new innovative program that warrants interest from cyber professionals. Although initiated by a state, this model could be replicated by cities with large academic institutions. Vermont has partnered with Norwich University to staff their level one security operations center (SOC) personnel.³⁸ Although the state employs cybersecurity specialists for level two and three activities, they have struggled to adequately staff a 24/7 level one SOC due to resource constraints. Filling the SOC level one slots with university students makes that endeavor more affordable and provides students with on-the-job training, which is often a prerequisite for employment in a cybersecurity position.

This type of hands-on training is incredibly valuable. The City of Pittsburgh’s department of computer services collaborated with a group of students at Carnegie Mellon to do a comprehensive security evaluation of the Pittsburgh city municipal computer network. After an incident in which an intruder into the Pittsburgh network was able to insert some “choice obscenities” into real estate tax bills, the city CIO reached out the university to gauge their interest in conducting a security audit.

This kind of relationship was highly beneficial to the city, which did not have the budget or personnel to conduct the audit using internal resources or external consultants. The five graduate students and their faculty advisor, under NDA, conducted external and internal penetration tests, used social engineering tactics, and analyzed policies. This long-term engagement led to both near- and long-term fixes and security upgrades.

This initiative spawned multiple follow-on collaborative projects. Its success relied on the experience and qualifications of the leadership both on the university and city sides of the project, and also their ability to recognize where synergies could be found between the two organizations. ³⁹

Partnerships with the private sector can come in many shapes and sizes, and are traditionally actuated through a client-vendor relationship, Today, these partnerships can expand beyond the typical client arrangement of fee for services to provide force augmentation and infrastructure. Cities contract with private sector companies for a variety of services, including force augmentation, management and technical consulting services, cloud services (Infrastructure as a Service, Platform as a Service, Software as a Service), managed security services, forensics and recovery assistance, and resiliency training.

Beyond typical pre-vetted public-private local contracts, it is possible for cities to work out specific partnership agreements with selected vendors or groups of vendors. San Diego, for example, has partnered with local startups to get a free or reduced cost demo of security tools for the first year of service, in return for feedback and enterprise deployment qualifications. Unfortunately, limited time and resources hinder the number of these trial partnerships, but they have proven to be of great benefit for both the city, which gets to try out new technologies with little long-term commitment, and for the companies, which gain entrée to the enterprise cybersecurity community, and sometimes can thereby secure a long-term contract.⁴¹

The San Diego government also maintains relationships with local companies with which it does not have any kind of contractual relationship. The San Diego CISO sits on the board of the local CISO Roundtable, which is akin to an informal ISAO. This very close-knit community shares threat information under Chatham House Rules,⁴² and has been so successful that communities in Hawaii and Denver are following its model.

At this writing, there are few, if any, institutionalized city-to-city cybersecurity partnerships. However, there is an informal network that works through cybersecurity professionals in each city, who speak and interact regularly and often tour each other’s cities and facilities.

In 2018, the Conference of Mayors facilitated the first information sharing session in its yearly meeting, but there is little organized and ongoing work on cybersecurity by the Conference of Mayors or other city-related organizations. These organizations are driven by the interests of their members; attention from city leaders (and their constituencies) is the only way that such a focus will gain momentum.

There is lots of room for growth in this area. Not only through city-focused organizations, but also through bilateral and multilateral relationships. A shared services model may be considered as an attractive, affordable way for cities to offer secure, reliable, and effective services to their citizens, but would require innovative thinking and management from city leaders across the country to preserve privacy, work out payment and budget allocations, and ensure the continuity of constituent services.

There is at least one example of such a program, which was piloted in Mission Viejo and other municipalities in Orange County, Calif. in 2016. These cities agreed to pool funds in order to purchase cybersecurity services.⁴³ This kind of relationship and others will be further delineated in the recommendations section.

As described above, the federal, state and local partnerships, both public and private, are relatively recent, with most conceptualized and formed within the past five years. These nascent partnerships are raising the overall security of the ecosystem, but they are only a start. Relationships take time to mature and to build trust, and must be ingrained through institutionalized and repeated interactions to stand the test of time and workforce turnover.

Practice, Practice, Practice

No amount of repetition would be excessive to hammer home the point that exercises are key to maximizing efficiency and effectiveness of incident response capability and resources. Whether self-moderated or in partnership with industry, state, or federal resources, cities should exercise different types of cyber incident scenarios regularly and include different stakeholders. While exercises are often seen merely as a mechanism to assess or evaluate existing training or capabilities, it is important to recognize that they’re increasingly viewed more broadly as a way to teach, learn, and develop organizational experience, and a mechanism to expose growing or new workforces to challenges they’re likely to face eventually.

Exercising an incident is of vital importance for all related organizations, but for cities, where the cybersecurity governance models are still maturing, personnel turnover is often frequent, and partner activities are constantly evolving, it is essential to do so frequently. This activity strengthens resiliency across multiple fronts. First, it increases awareness of the threat and possible impact of a cyber attack to the stakeholders who control key resources and may need to take a management position in the case of a serious incident. Second, it introduces the key players to each other and ensures that the rolodex of key personnel and institutions is built up before an incident, not during or after. And third, it can uncover issues in procedure, policy, law, personnel, and technology that may hamper the response to an actual incident.⁴⁴

No amount of repetition would be excessive to hammer home the point that exercises are key to maximizing efficiency and effectiveness of incident response capability and resources.

This mindset should be expanded further to a more strategic level. The John S. McCain National Defense Authorization Act (NDAA) for Fiscal Year 2019 Section 1649 points to the need to create a system of campaign planning at the local level to assess readiness and plan for various scenarios.⁴⁵ Section 1649 is a pilot program only, and limited to defense critical infrastructure, but an expansion of such an initiative aimed at local governments and the critical infrastructure in those areas could help create the kind of federal system of cybersecurity resiliency planning that is needed.

One of the key legal issues referenced above is contract negotiation. Agreements with forensics and incident response firms are best handled ahead of time if at all possible, whether or not these contracts are set on retainer.⁴⁶ Because of the nature of government contracting, retainer contracts can be difficult to implement as they involve an allocation of committed funds prior to any specific case, but are beneficial in that they come with guaranteed quick response times. If there is no incident, the money can be used for additional services. Even without a retainer contract, negotiating terms in advance can decrease rates, grease the approval process with insurance companies, and ensure that the contractors are familiar with the city’s systems and environment before an incident occurs.⁴⁷

This process should be no different with government/nonprofit IR teams. In fact, arranging advance memorandums of understanding (MOU) between DHS’ NCCIC and city governments would help decrease response time and foster a relationship between the two organizations. This process should be part of an organized outreach program, as described below.

Think Differently, Think Regionally

Cross-city and cross-state partnerships have a huge upside that has been barely acknowledged nor widely put into practice to date. Some National Guard Cyber Protection Teams (CPTs) are created through partnerships between states: guardsmen from Michigan, Indiana, and Ohio form one CPT; members from New York and New Jersey, another.⁴⁸ Not only do these units get to pull talent from multiple areas, but they have the ability to serve critical infrastructure that may span across multiple states more easily. Regional ISAOs, such as those described earlier, knit cities and their partners together to share information.

Taking a regional approach in new and innovative ways could provide services at scale and decrease the barrier to entry for local governments’ formation of a sophisticated cybersecurity program. Some examples of regional-based strategies could include:

• Signing MOUs between cities, between cities and states, or between states to provide incident response services.
• Expanding the use of regional ISAOs to facilitate the dissemination of information to SLTT organizations and their partners.
• Hiring a joint CISO who could oversee information security at multiple local governments.
• Creating a joint security operations center that could monitor the systems in multiple jurisdictions. Vermont and Maine have already started down this path, but we have yet to see it manifest for other groups of cities or local governments.
• Joining together to create a shared services platform used by multiple cities to provide similar services.
• Leveraging regional human resources ranging from educational institutions that develop such human capital, as well as regional talent pools in neighboring or nearby cities.

Governance Reform

There is an inherent conflict of interest with a CISO reporting directly to a CIO, yet that configuration, traditional but less and less common in the private sector, is the norm in most SLTT governments (and, in fact, at the federal level as well). A 2014 PwC survey illustrates this point using key incident response statistics. The survey found that organizations where the CISO reported directly to the CIO experienced 14 percent more downtime due to security issues and experienced over 40 percent financial loss from cyber incidents.⁴⁹ While this finding was sourced from private companies’ data, these lessons are applicable to any institution.

While some CIOs are particularly security minded and can adequately manage the tradeoffs between modernization, speed of deployment, and security, the speed at which personnel changes within government makes a less institutionalized approach (where the CISO must depend on the individual CIO’s proclivity towards security) even more suspect. Additionally, placing the CISO under the CIO necessarily infers that security is an IT risk—not a whole company or agency risk. Indeed, a 2018 report by the Financial Services-Information Sharing and Analysis Center (FS-ISAC) recommended that CISOs report directly to the CEO for purposes of executive visibility.⁵⁰

Such reorganization within public sector institutions may not be as easily executed as a private sector corporate reshuffle; cities and states may in some cases have to work with what they have. The state of New Jersey and New York City have both seen quantifiable benefits ensue from having an independent CISO position that reports outside of the IT line.⁵¹ In other cities, such as San Diego, the CISO still reports to the CIO, but also provides direct reporting to the mayor and other city executives multiple times per year.⁵²

For smaller cities, the ability to employ a qualified CISO may be a luxury, no matter where he or she sits within the government. Smaller cities should, as mentioned above, consider using new tactics to share CISOs with other cities or surrounding areas or contract for a CISO-as-a-Service to get part time assistance. Whatever the construct, security must remain a priority for all SLTT entities, with high level visibility.

Additionally, if not already done so, cities should consider moving towards a shared services/consolidated IT management system. Consolidating IT systems under one agency and through shared infrastructure can improve security in several ways:⁵³

• Personnel improvements: centralizing security and risk management functions allows the participating government agencies to share cost of security rather than duplicating it, meaning that the city could hire more specialists that could cover multiple agencies.
• Budgetary efficiency: shared purchasing across government functions reduces cost by consolidating vendors and allowing for bulk purchase agreements at scale. It also encourages greater capital investment in security and infrastructure when allocated as a whole government entity rather than through individual agency investments.
• Repeatable practices: a shared security model fosters a set and repeatable risk management process across government agencies.

Codify, Exercise, and Institutionalize Federal Resources with Authorities to Support SLTT Organizations

To make incident response more efficient and effective, whether for large or small incidents, the United States should prioritize deconflicting efforts, authorities, and responsibilities across the various agencies. The existing incident reporting guidance lists several points of contact that depend on the nature of the incident, which may or may not be known until well after the event. Furthermore, in many cases, verbal guidance provided to SLTT representatives from various federal agencies on how to report an incident has been conflicting.

Although there is high level guidance through PPD 41, as described above, concept of operations (CONOP)⁵⁴-level planning and exercising needs to be done and codified into plans, policies, and procedures.

Federal policymakers, lawyers, and lawmakers need to further define the authorities and allocation of resources to and between various federal agencies.⁵⁵ There are notable efforts currently underway at the National Guard Bureau and U.S. Coast Guard Headquarters to better delineate these roles—positive prospects that will add to the toolkit for federal and SLTT incident responders. Mapping out the capabilities of the National Guard CPTs in various states, such as the team in Missouri with deep forensics qualifications or the teams in Washington with experience on industrial control systems, would support a deeper level of planning and further cooperation between the various states which could form mutually beneficial peer partnerships based on complementary skill sets.

Reform Federal Funding for Cybersecurity

The current programmatic framework for providing federal funds for cybersecurity assistance to SLTT governments is challenged by red tape, confusion, and competition with other homeland security-related threats. This program should be reformed to allow for and direct funding towards efforts in both cybersecurity resilience and response.

The majority of funds provided to SLTT entities for cybersecurity emanates from FEMA’s Homeland Security Grant Program (HSGP), which itself has three programs:

• State Homeland Security Program (SHSP)
• Urban Area Security Initiative (UASI)
• Operation Stonegarden (OPSG)

The 2018 Notice of Funding Opportunity (NOFO) was notable in that, although these funds have been eligible for use for cybersecurity purposes for some time, FEMA mandated that both SHSP and UASI recipients were required to include an investment justification focused on cybersecurity projects.⁵⁶ FEMA also included state and urban area chief information officers and chief information security officers in the list of mandated representatives for the senior advisory committee (SAC) that builds the grant proposals for each eligible area.

These efforts will hopefully address the gap between the grant money spent by states and urban areas on cybersecurity and their perceived gaps, the areas that municipalities themselves identify as an area of need. Although the HSGP purports to be focused on national preparedness writ-large, and while it funds disaster planning efforts, it ties cybersecurity needs directly to counterterrorism threats:⁵⁷

“Recipients must limit the use of SHSP and UASI funds for projects that support the security and functioning of critical infrastructure and core capabilities as they relate to terrorism preparedness, and may simultaneously support enhanced preparedness for other hazards unrelated to acts of terrorism.”

By having the bulk of the available federal funds for cybersecurity come out of the HSGP, the federal government is, in essence, forcing the cybersecurity mission to “compete” with the counterterrorism mission in a process in which all spending must have a counterterrorism nexus. That is, not only is cyber forced to compete with terrorism as a problem set, it’s forced to do so in a process and program that is explicitly designed to focus on the latter, rather than the former.

By having the bulk of the available federal funds for cybersecurity come out of the HSGP, the federal government is, in essence, forcing the cybersecurity mission to “compete” with the counterterrorism mission.

Given the relative threat from terrorist groups vs. nation states and criminal groups in cyberspace, this seems to be a strange focus for grant money. Reportedly, FEMA does not require a direct linkage between threat and proposal⁵⁸ (the idea that a terrorist might possibly be behind the keyboard for any attack is enough); that then begs the question as to why the language is included at all, especially given that it might discourage proposals for arenas in which state and local governments should be taking a more active role, such as workforce development, that are not tied to attack prevention or preparedness.

In the short term, FEMA should give clearer guidance and examples about which kinds of initiatives are eligible for HSGP grant money. A more transparent process would help states and local governments formulate proposals likely to receive money, and could also feed into a gap analysis assessment for the federal government between the initiatives they would like to see SLTTs take on, and the funds they are currently providing.

In the long term, FEMA and DHS should consider distinct grants for cybersecurity that can be used to target the specific priorities, threats, and needs. This is not the first time this idea has been proposed; the idea was floated in 2017 through a bill co-sponsored by Reps. Derek Kilmer (D-Wash.) and Barbara Comstock (R-Va.) along with Sens. Mark Warner (D-Va.) and Cory Gardner (R-Colo.) proposed the idea in the State Cyber Resiliency Act. The need has not changed, though the threat to SLTT governments has become more public since 2017.⁵⁹

The programs and resources available for each state and local government are different; too-prescriptive guidance can create a backlog of under allocated funds and missed opportunities. Conversely, targeted grant money and associated analysis year-over-year could help feed both national and local priorities and enable the federal government to work with and through the states, instead of around them. Such a grant program would also give states a clear roadmap for federal funds, and enable the cybersecurity experts to guide the grant process, rather than being a minor part of a larger grant proposal that competes with other more easily quantifiable asks (such as fire trucks).

The topic of reforming federal funding for cybersecurity deserves a full-length paper of its own. This is a complex issue with many moving parts and myriad stakeholders. We hope that this paper will inform that debate as the discussion moves forward in and outside of the federal government.

Structure and Prioritize Federal and State Outreach Efforts

The structure and management of federal outreach efforts for SLTT governments is another subject that deserves further discussion. While we have touched on the topic herein, our focus has been city-oriented, and as a result, omitted issues relating to counties, tribes, and territories that also merit consideration. A key point bears mentioning: there should be a strategic approach to interacting with these organizations, from outreach, to service provision, to forming ongoing relationships. DHS has a prominent role in these activities, but outreach must be spread across several federal government agencies serving these communities, and will remain a shared responsibility.

State programs have been maturing over the last several years, and as described above, many are starting to think about working with local governments in a variety of ways. These programs could provide needed services, whether direct or in an advisory capacity, to local governments, but they need to be properly resourced both in terms of outreach (to make sure local governments are aware of them in the first place) and execution.

Ahern, Colin, Deputy CISO, City of New York, and Ben Woolsey, Manager, Mandiant. Interview by author. September 20, 2018.

Bennett, Darren, CISO, City of San Diego. Telephone interview by author. April 20, 2018.

Butler, Bob, Senior Vice President of Critical Infrastructure Protection Operations, AECOM, and John Esquivel, Senior Director, National Preparedness & Infrastructure Protection. Telephone interview by author. August 17, 2018.

Cappa, Franco, Cybersecurity Advisor, US Department of Homeland Security, and Benjamin Gilbert, Cybersecurity Advisor, US Department of Homeland Security. Telephone interview by author. August 28, 2018.

Clark, Alaina, Deputy Assistant Secretary, Office of Intergovernmental Affairs, US Department of Homeland Security. Telephone interview by author. September 07, 2018.

Bristow, Mark, Director, NCCIC Hunt and Incident Response Team (HIRT) at U.S. Department of Homeland Security. Interview by author. August 07, 2018.

Crummey, Chris, Executive Director, IBM Security. Telephone interview by author. September 28, 2018.

Derusha, Christopher, Deputy CISO, State of Michigan. Telephone interview by author. September 06, 2018.

Dieumegard, Eric, New York National Guard. Telephone interview by author. October 05, 2018.

Duffy, Thomas, Vice President of Operations, CIS, and Brian Calkin, Vice President of Operations, MS-ISAC. Telephone interview by author. July 26, 2018.

Finn, Jacob, Policy Director for Cybersecurity, Los Angeles Mayor Eric Garcetti, Office of Public Safety, Neeraj Bhatnagar, Director of Policy and Programs, Mayor's Office of Public Safety, City of Los Angeles, and Reuben Wilson, General Counsel for Public Safety at Los Angeles Mayor’s Office. Telephone interview by author. September 27, 2018.

Hayslip, Gary, CISO, Webroot. Telephone interview by author. August 16, 2018.

Janak, Isaac, Cyber Security Program Manager, Commonwealth of Virginia. Telephone interview by author. September 07, 2018.

King, Heather, COO, Cybersecurity Threat Alliance. Interview by author. August 20, 2018.

Kirtley, Tony, Incident Commander, Secureworks. Telephone interview by author. August 23, 2018.

Lauland, Andrew, Senior Policy Analyst, RAND Corporation. Telephone interview by author. July 27, 2018.

Lawlor, Joseph, Former Managing Director of Proactive Services, K2 Intelligence. Telephone interview by author. August 24, 2018.

Mitchell, Erica, LTC, US Army, Research Scientist, Army Cyber Institute, and Judy Esquivel, CW3, US Army, Research Scientist, Army Cyber Institute. Interview by author. August 21, 2018.

Mosley, Chetrice, Cybersecurity Program Director at Indiana Office of Technology & Indiana Department of Homeland Security. Interview by author. January 17, 2018.

Nichols, Steve, CTO, Georgia. Telephone interview by author. August 23, 2018.

Rhodes, Calvin, CIO, Georgia. Telephone interview by author. August 23, 2018.

Rice, Lisa, LCDR, US Coast Guard. Telephone interview by author. September 24, 2018.

Rohner, Boyden, Deputy Director for Operations, National Cybersecurity and Communications Integration Center (NCCIC), US Department of Homeland Security. Interview by author. August 07, 2018.

Schweitzer, Robert, Senior Advisor, Resilience, FEMA. Telephone interview by author. September 27, 2018.

Spence, Joshua, CISO, West Virginia. Telephone interview by author. September 11, 2018.

Stocchetti, Valecia, CERT Manager, MS-ISAC. Telephone interview by author. July 26, 2018.

Susmann, Phil, President, Norwich University Applied Research Institutes (08 31). Telephone interview by author. August 31, 2018.

Tama, Jason, CAPT, US Coast Guard, Sector Commander and Captain of the Port, Sector New York, Sarah Brennan, LCDR, US Coast Guard, and Emily Miletello, LT, US Coast Guard. Interview by author. August 22, 2018.

Additional members of the public and private sector were interviewed for this project but declined to be cited in the paper. Their insights are greatly appreciated.