Partners for Cities

Potential partners for cites fall into five categories—federal, state, nonprofit, private sector, and peer. Cities can and should take full advantage of the benefits that can come through engaging with organizations in each category, and will inevitably forge unique reciprocal relationships with each as well. This section will highlight a select number of ongoing partnerships, using examples from cities that are moving ahead of the curve in this direction.

The federal-city suite of partnerships is most robust in the areas where the federal government has teams deployed locally; for services and relationships that are controlled centrally in Washington D.C. or its environs, affiliations are more ad-hoc. There are thousands of cities in the United States and many more small local municipalities; coordinating and cultivating relationships with each of these local governments from Washington is unrealistic. With locally-driven federal cybersecurity efforts still evolving, there have been some real success stories, but work still remains in how to best assist cities and local governments, facilitate the delivery of services, and make them more secure and resilient.

Incident Response

The most immediate assistance the federal government can provide to a city that has experienced a cyber incident is with direct response services. Per Presidential Policy Directive—United States Cyber Incident Coordination (PPD 41)¹, these services are grouped into three categories:

• Asset Response: furnishing technical assistance to affected entities to help them recover from the incident. This effort is led by the Department of Homeland Security (DHS) through the National Cybersecurity and Communications Integration Center (NCCIC).
• Threat Response: investigating the crime associated with a cyber incident. This effort is led through the Department of Justice (DOJ), through the Federal Bureau of Investigation (FBI) and the National Cyber Investigative Joint Task Force (NCIJTF).
• Intelligence Support: creating situational awareness about cyber threats. This effort is led by the Office of the Director of National Intelligence (ODNI), through the Cyber Threat Intelligence Integration Center (CTIIC).

PPD 41 sets down these roles for significant cyber incidents only, but in practice, this division of labor has extended down to any instance in which a city, state, or other entity requests federal assistance with a cyber incident.

In 2014, the Federal government released guidance to State, Local, Tribal, and Territorial (SLTT) organizations for how to properly report a cyber incident. The instruction breaks down the various types of cyber incidents and encourages SLTT incident managers to contact the most relevant federal entity.² This approach can be confusing, however, as it may be initially unclear as to which type of incident has occurred.

The federal government currently responds to incidents from a variety of intake points. Still, DHS has promoted the concept of the NCCIC as a “911 for cybersecurity.”³ On the face of it, this idea has merit. Having one contact point for the federal government makes it easy for SLTT governments to reach out if they need assistance. Nonetheless, in practice, having such a policy would be hard to enforce. FBI agents are in the field and maintain regular contact with their SLTT partners, as do Federal Emergency Management Agency (FEMA) personnel in each of the 10 FEMA regions. It is no surprise, therefore, that many incidents are reported through the FBI or FEMA, a situation that is unlikely to change without significant active and persistent presence from DHS personnel.

Federal partners provide valuable assistance to cities, free of charge, but they are unlikely to be the only service provider during an incident. The role of federal responders, particularly from DHS, can vary greatly depending on the incident and the needs of the city's incident response team. In some cases, they provide hands-on remediation assistance, but in others cases provide only remote assistance or advice. Trust issues sometimes also limit local governments’ willingness to call in the federal government because of political concerns or misunderstandings around disclosure policies (Freedom of Information Act).

Within DHS, the role of FEMA in cyber incident response is still under evaluation. There is a report being produced by RAND⁴ in cooperation with FEMA that will undoubtedly provide some insights, but the practical manifestation of those decisions is still being worked through.

Resiliency

The closest and most robust relationship between the federal government and cities around the country exist in the fusion centers.⁵ In these facilities, city, state, and federal law enforcement officers sit side by side. The operational, bidirectional sharing facilitated by co-location is demonstrably useful, but there still remains a gap with regard to sharing insights and deterring future threats.⁶

In general, information sharing programs are confronted with the same barrier to efficacy: getting relevant and timely information declassified and ready for tear-line dissemination. Facilitating more clearances among security professionals in SLTT organizations will ameliorate the situation, but intelligence must be actionable to be useful, and information that must remain within a Sensitive Compartmented Information Facility (SCIF) hinders its effectiveness. Furthermore, information that must remain within a SCIF is generally not useful on unclassified networks, which essentially all municipal networks are.

The new National Risk Management Center (NRMC) at DHS is still being developed after its establishment in the fall of 2018. So far, it has announced that its first project will be a hard look at positioning, navigation, and timing (PNT) technology, used in GPS. It has since kicked off several sprints around finance, electric, and telecommunications.⁷ The NRMC could create traction on and for SLTT organizations if it marshals the resources of the federal government to take a truly strategic look at threats facing the various sectors or if it could provide a model for SLTT governments to assess their risks. Those who are engaged in developing its next steps should not forget the equally vital role played by SLTT governments and oversight of critical infrastructure.

In addition to information sharing, DHS and Department of Defense (DoD) run both national-level and local exercises. NCCIC and FEMA have teams dedicated to this service, and provides it to local government agencies free of charge. Exercises are essential for the maturation of any cybersecurity program and its integration in a city's greater incident response program. While these services are offered by the private sector for a fee, having access to them gratis is a huge benefit to cities, which are often strapped for cash.

FEMA is developing a cybersecurity preparedness guide,⁸ but it has yet to be released.

Funding

FEMA is also a key partner for cities when looking for funding for cybersecurity-related projects. The 2018 Homeland Security Grant Program (HSGP) Notice of Funding Opportunity (NOFO) mandated that two of its three programs must include investment for cybersecurity.

Unfortunately, the specific amounts cities have requested, received or spent to date from this program are not publicly available. Anecdotally, we have found that some cities have used grants from the HSGP to fund cybersecurity projects. San Diego, for example, used Operation Stonegarden⁹ to fund its intrusion protection system. Meanwhile, other cities (and states) have struggled to understand which cybersecurity related activities might be eligible for HSGP funds. This issue is further addressed in the recommendations section below.

As state cyber programs mature, many states are reaching down to their local governments to build partnerships and offer services. While these overtures are positive, they can be complicated by existing political tensions between state and city governments and by resource constraints (states face challenges to staff and fund programs to manage their own cybersecurity, let alone offer services to local governments).¹⁰ Even in the face of these challenges, some compelling programs have been established that demonstrate how cities and states can work together on this problem set, and offer insight into how such state and local partnerships may evolve in the future.

These programs generally fall into two categories: resilience and response. Resilience activities can include assessments, exercises, and consulting services. Response services can include forensics or recovery services.

One of the most critical issues facing local governments is accurately assessing which elements they are missing and the current state of their cybersecurity programs. This resiliency service is offered by DHS, as described above, but that department does not and will not ever have the capability and staffing to handle all of the demand from local governments across the country. Some states have stepped up to offer similar services with a focus on their own local governments.

One of the most critical issues facing local governments is accurately assessing which elements they are missing and the current state of their cybersecurity programs.

In 2018, Michigan piloted a “CISO as a Service” program, offering assessments and advice to nine local governments.¹¹ The state is currently working to find a sustainable funding model to avoid annual appropriations challenges, but the success of the program yielded enough for the state’s information technology (IT) leadership to support it going forward.

The state’s IT office used a chargeback model to pay for the assessments at a cost lower than standard for private sector services. This kind of cost-savings is important to any state program’s success, as is the neutrality of the service providers; most of the counties requesting help from the state in this scenario were resource challenged.¹²

This trend is not unique to Michigan. Georgia CIO Calvin Rhodes stated that his state’s IT and security offices are more likely to hear from small governments than large ones, in that larger cities tend to have their own staff. Still, the state offers consultation to all municipalities upon request, and pre-existing IT contracts that local governments can utilize for general purpose or incident response needs.¹³ During the Atlanta incident, the city did not reach out to the state for assistance, but relied on its private sector and federal partners. Without comment from the Atlanta government, it is impossible to know if this is because they were unaware that such contracts were available, if they preferred their own vendors (existing or emergency), if they were unable to take advantage of them for some reason, or for another cause.

Other states, such as West Virginia, have also structured their contracts so that local governments can take advantage of them. In these circumstances, states have often failed to devote sufficient resources to outreach to ensure that local governments are aware of the existence of these programs, and that the contracts can be accessed by and for those jurisdictions.¹⁴

New York and Virginia are taking a more proactive approach to providing assistance to their local governments. Through its Department of Homeland Security and Emergency Services, New York has begun providing vulnerability assessments evaluated against the NIST framework. Although those services cannot be linked to remediation efforts, they have started to deliver useful resources for county and local governments ahead of the wait time for similar DHS offerings.¹⁵

Virginia is working through its emergency management agency to build relationships and raise awareness in local governments. They have held several meetings across the state in 2018 aimed at gaining a better understanding of the challenges local governments are facing. Virginia follows up with bringing local agencies’ representatives into the fusion center so that those connections can be strengthened and institutionalized.

Virginia also activated its National Guard to State Active Duty (SAD) status to perform vulnerability assessments and penetration tests on local government networks. In 2019, the Virginia government plans to hold additional exercises involving local governments in the exercise of the state emergency plan. Another initiative seeks to engage information security professionals across the state in the grants process through regional working groups, as the state’s funding for this type of activity is heavily reliant on homeland security grants.¹⁶

Moving forward, the type of engagement that states such as Virginia have made will be vitally important to the success of state and local cybersecurity, especially given the emphasis on integrating site-specific IT and cybersecurity needs to the DHS FEMA grant process. Also encouraging are efforts from states such as Indiana, which has involved its local governments in the state’s Executive Council on Cybersecurity. This council’s mandate is to establish a strategic framework for Indiana’s future cybersecurity initiatives.¹⁷ With more engagement and involvement from local governments, states can improve their policies, offerings, awareness, and long-term strategies to better serve to secure their cities, and thereby their citizens.

The Multi-State Information Sharing and Analysis Center (MS-ISAC)

The Multi-State Information Sharing and Analysis Center (MS-ISAC), a division of the Center for Internet Security under cooperative agreement with DHS, is a nonprofit entity formed to be the “focal point for cyber threat prevention, protection, response and recovery for the nation’s state, local, territory and tribal (SLTT) governments.”¹⁸ Information sharing is central to the MS-ISAC’s mission; in addition, it provides services to SLTT governments to assist with incident response.

MS-ISAC is perhaps most known for its network intrusion detection system (NIDS) sensors known as “Albert”—modeled after the “Einstein” sensors used to protect federal government networks. MS-ISAC can become involved in an incident response effort proactively, by noticing something through the Albert network, or reactively, if an SLTT organization reaches out to them for assistance.¹⁹

MS-ISAC’s incident response team is designed to work remotely, but in rare circumstances deploys on-site with a customer. The incident response (IR) team concentrates on remediation, as does the DHS NCCIC’s Hunt and Incident Response Team (HIRT), and sometimes operate alongside that team or others from the FBI, Secret Service, or private sector.²⁰ There is no set threshold for the MS-ISAC team to accept a case, but the decision regarding on-site or deployed status is up to that organization’s COO (currently Tom Duffy) and the affected entity. Although the team is able to deploy quickly, there is sometimes a delay if the team has not been previously approved by the entity’s insurance company.²¹ As described above in the section on private sector and below in the recommendations section, it is essential to involve all stakeholders in incident response preparations. Doing so resolves red tape and administrative issues before an incident occurs, instead of dealing with them while in crisis mode to the detriment of efficient incident response.

Information Sharing and Analysis Organizations (ISAOs)

Although most cities belong to the MS-ISAC as the designated ISAC for state, local, tribal and territorial governments, many have also seen benefit from joining regional Information Sharing and Analysis Organizations (ISAO), which function similarly to the MS-ISAC, but are designed for non-federally designated critical infrastructure industries, or which are organized around another concept, such as a geographic area. The primary advantages of using a geographic base are that the membership can be cross-sector, helping to break through separate silos of information exchange, and that a smaller base of member operations can facilitate closer relationships.

Regional ISAOs have used these close relationships—built on a foundation of trust—to facilitate and mature information sharing programs that involve both the private and public sectors. The Arizona Cyber Threat Response Alliance boasts members from many regional cities. Its information sharing program anonymizes information so that its members can share back confident in their own protection.²² Similarly, Los Angeles’ LA Cyber Lab has started receiving indicators of compromise from private sector companies that build a rich set of information for the city and the other members of the lab, and can be fed up to the federal government through NCCIC ingest.²³ L.A. is also the recipient of a 2018 DHS grant to turn the LA Cyber Lab into an official regional ISAO, institutionalizing its role in information sharing.²⁴

Universities

There are numerous examples of public sector institutions forming effective partnerships with universities. Partnering with universities has several benefits, including access to talent, technological insights, and resources, and the chance to match up real-world priorities with ongoing research.

Los Angeles is currently building several partnerships with its universities, including information sharing, data science and visualization, and workforce development. Most programs are still in their infancy, but have great potential to harness the technology capabilities and resources at the L.A.-area schools. For example, California State University, Long Beach is embarking on a project to build threat models and dashboards for L.A.’s Integrated Security Operations Center (ISOC).²⁵

The above-mentioned ISOC, part of the LA Cyber Lab, also processes feeds from universities that are partnered with the city and the other LA Cyber Lab members. L.A. is also looking to expand its internship programs to involve college and university students, increasing the skilled and trained workforce available to the city, while providing them with real-world experience.

Partnering with universities has several benefits, including access to talent, technological insights, and resources, and the chance to match up real-world priorities with ongoing research.

The state of Vermont has launched a new innovative program that warrants interest from cyber professionals. Although initiated by a state, this model could be replicated by cities with large academic institutions. Vermont has partnered with Norwich University to staff their level one security operations center (SOC) personnel.²⁶ Although the state employs cybersecurity specialists for level two and three activities, they have struggled to adequately staff a 24/7 level one SOC due to resource constraints. Filling the SOC level one slots with university students makes that endeavor more affordable and provides students with on-the-job training, which is often a prerequisite for employment in a cybersecurity position.

This type of hands-on training is incredibly valuable. The City of Pittsburgh’s department of computer services collaborated with a group of students at Carnegie Mellon to do a comprehensive security evaluation of the Pittsburgh city municipal computer network. After an incident in which an intruder into the Pittsburgh network was able to insert some “choice obscenities” into real estate tax bills, the city CIO reached out the university to gauge their interest in conducting a security audit.

This kind of relationship was highly beneficial to the city, which did not have the budget or personnel to conduct the audit using internal resources or external consultants. The five graduate students and their faculty advisor, under NDA, conducted external and internal penetration tests, used social engineering tactics, and analyzed policies. This long-term engagement led to both near- and long-term fixes and security upgrades.

This initiative spawned multiple follow-on collaborative projects. Its success relied on the experience and qualifications of the leadership both on the university and city sides of the project, and also their ability to recognize where synergies could be found between the two organizations. ²⁷

Partnerships with the private sector can come in many shapes and sizes, and are traditionally actuated through a client-vendor relationship, Today, these partnerships can expand beyond the typical client arrangement of fee for services to provide force augmentation and infrastructure. Cities contract with private sector companies for a variety of services, including force augmentation, management and technical consulting services, cloud services (Infrastructure as a Service, Platform as a Service, Software as a Service), managed security services, forensics and recovery assistance, and resiliency training.

Beyond typical pre-vetted public-private local contracts, it is possible for cities to work out specific partnership agreements with selected vendors or groups of vendors. San Diego, for example, has partnered with local startups to get a free or reduced cost demo of security tools for the first year of service, in return for feedback and enterprise deployment qualifications. Unfortunately, limited time and resources hinder the number of these trial partnerships, but they have proven to be of great benefit for both the city, which gets to try out new technologies with little long-term commitment, and for the companies, which gain entrée to the enterprise cybersecurity community, and sometimes can thereby secure a long-term contract.²⁹

The San Diego government also maintains relationships with local companies with which it does not have any kind of contractual relationship. The San Diego CISO sits on the board of the local CISO Roundtable, which is akin to an informal ISAO. This very close-knit community shares threat information under Chatham House Rules,³⁰ and has been so successful that communities in Hawaii and Denver are following its model.

At this writing, there are few, if any, institutionalized city-to-city cybersecurity partnerships. However, there is an informal network that works through cybersecurity professionals in each city, who speak and interact regularly and often tour each other’s cities and facilities.

In 2018, the Conference of Mayors facilitated the first information sharing session in its yearly meeting, but there is little organized and ongoing work on cybersecurity by the Conference of Mayors or other city-related organizations. These organizations are driven by the interests of their members; attention from city leaders (and their constituencies) is the only way that such a focus will gain momentum.

There is lots of room for growth in this area. Not only through city-focused organizations, but also through bilateral and multilateral relationships. A shared services model may be considered as an attractive, affordable way for cities to offer secure, reliable, and effective services to their citizens, but would require innovative thinking and management from city leaders across the country to preserve privacy, work out payment and budget allocations, and ensure the continuity of constituent services.

There is at least one example of such a program, which was piloted in Mission Viejo and other municipalities in Orange County, Calif. in 2016. These cities agreed to pool funds in order to purchase cybersecurity services.³¹ This kind of relationship and others will be further delineated in the recommendations section.