Taking Down Terrorism: Strategies for Evaluating the Moderation and Removal of Extremist Content and Accounts

by Spandana Singh

The rise of the internet and the expansion of information and communication technologies have fundamentally transformed how individuals around the world can communicate, coordinate, and share information. However, the spread of new technologies has also yielded new challenges and threats. Over the past decade, for example, social media and other online platforms have increasingly been used by extremist groups to facilitate the spread of their ideologies and propaganda, engage with and radicalize individuals across the globe, and coordinate acts of terror. Use of these platforms has enabled these groups to connect, organize, and act with greater ease, speed, and breadth than their predecessors.¹

As technology has become more integral to the operations of extremist groups around the world, major internet companies, such as Facebook, Twitter, and Google, as well as smaller companies, have come under public pressure to prevent these groups from reaping the benefits offered by their platforms.² This has resulted in technology companies becoming integrated into the field of countering violent extremism (CVE), which was typically led by government agencies in partnership with civil society. As the use of online platforms by extremist groups has grown, several of these major technology companies have also become valuable actors in the space.

Despite the emergence of a distinct set of CVE activities, the field lacks empirical research, as well as substantive data and metrics, to guide its work. This is especially true for technology company-led CVE approaches. As a result, it has been challenging to identify best practices, and it is often unclear which CVE approaches are most effective and why. This is of significant concern, as without this knowledge, companies could be misallocating valuable resources and instituting programs that have unintended and detrimental consequences (e.g. further marginalizing and harming at-risk communities and individuals rather than disrupting their engagement with online extremism).

Use of these platforms has enabled extremist groups to connect, organize, and act with greater ease, speed, and breadth than their predecessors.

One common approach to CVE that technology companies frequently employ, and that is often scrutinized, is the moderation and removal of extremist content and accounts. Following major terrorist attacks, such as the September 2017 attack on a London Underground train at Parsons Green,³ it was found that several terrorist recruits had engaged with extremist content and individuals on platforms such as Facebook and Twitter and, subsequently, were radicalized and employed to coordinate acts of terror. As public pressure has mounted on these companies to remove extremist content and accounts, a number of researchers and civil society advocates have questioned whether these moderation and removal efforts are impactful, and whether they are the best use of a company’s resources and time.

This report explores how to make content moderation and removal efforts regarding terror-related content more effective and how to make evaluations of these efforts by companies and researchers more strategic. In addition, this report provides a set of recommendations on best practices and quantitative and qualitative data points that companies should collect and disclose, which can facilitate the impact evaluation of terror-related content moderation and removal efforts.

Because this is a growing field of inquiry, this report will focus on efforts that have targeted the Islamic State of Iraq and Syria (ISIS), an extremist organization well-known for their tech savviness and internet use and commonly targeted by content and account moderation and removal efforts. In addition, although content and account moderation are two different types of interventions, research on the effectiveness of moderation and removal practices typically evaluates these approaches together, given that when you moderate or takedown an account, you also moderate or remove its associated content. For the purposes of consistency and comparison, this report will do so as well. Furthermore, given the public nature and importance that social media platforms such as Facebook, Twitter, and YouTube have when it comes to ISIS’ online operations, they are typically the most-studied platforms and will therefore be largely featured in this report.

What Is Content Moderation?

Content moderation can be understood as the practice of “monitoring and vetting user-generated content (UGC) for social media platforms of all types in order to ensure that the content complies with legal and regulatory exigencies, site/community guidelines, user agreements, and that it falls within norms of taste and acceptability for that site and its cultural context.”⁴ On user generated content-based platforms, individual pieces of content, such as a Tweet or Facebook post, that violate a company’s content or community standards or local legal frameworks, can be moderated through practices such as temporary suspension pending further review or geo-blocking. A piece of content that has been found to violate a company’s content or community standards can also be removed from the platform permanently. In addition, a user who has violated content or community standards or local legal frameworks by posting prohibited content can also have their account moderated via temporary suspension or deletion.

In the past, companies typically relied on external parties, such as governments, non-governmental organizations (NGOs), and users themselves to flag content that violates platforms’ terms of service. YouTube, for example, has established a Trusted Flaggers program, which enables select government agencies and NGOs from around the world that have a track record of accurate content flagging to flag problematic content in bulk, receive reviews for their flags, and to engage directly with the company on its policy enforcement processes.⁵

However, as companies have come under increased pressure to moderate harmful content, they have worked collaboratively to develop tools and technologies that enable them to detect such content on their own. One example of this is PhotoDNA, a technology developed by Microsoft, originally for the purposes of identifying child pornography online.⁶ PhotoDNA works by converting existing child porn images online into a grayscale format. It then overlays the images onto a grid and assigns each square a numerical value. The designation of a numerical value converts the square into a hash, or a digital signature, which remains tied to the image and can be used to identify other recreations of the image online.⁷ The software has enabled companies to identify and remove child pornography materials at a much faster rate and has been particularly effective in identifying content despite color alterations and resizing of images.⁸ In 2016, the Counter Extremism Project developed eGLYPH, an adaption of the PhotoDNA technology that could be applied to extremist content, including still images, video, and audio files.⁹ This technology has been widely adopted by internet platforms of all sizes.

As companies have come under increased pressure to moderate harmful content, they have worked collaboratively to develop tools and technologies that enable them to detect such content on their own.

As content moderation efforts have increased, so have concerns about censorship and the stifling of online freedom of speech, often resulting from stringent platform content policies as well as increases in government requests for content moderation and removal.¹⁰ One way to address these concerns is through greater transparency regarding companies’ content moderation and removal practices. Such transparency has already become a norm in the context of addressing public concerns about companies’ participation in government surveillance programs. Following the Snowden revelations in 2013, many technology companies began publishing transparency reports in order to document the scope and extent of government and legal requests for users’ data.¹¹ As this has gradually become a widely adopted practice by internet platforms, transparency reporting has expanded to include data on government and legal requests for content moderation and removal as well as data on how companies moderate and remove content based on their own terms of service and content policies. However, the practice of reporting such data is still relatively new, and, as a result, only a few companies, such as Google (regarding YouTube), Facebook, Twitter and Automattic do it, with little standardization and several variations in terms of the granularity of data being reported. This lack of consistent metrics and reporting standards poses a number of challenges for those attempting to conduct cross-company comparisons and for those attempting to understand the impact of content moderation and removal efforts on a particular category of content such as extremist content.

The CVE Landscape Today

Social media platforms have increasingly become recognized as integral tools to ISIS’ recruitment, engagement, and coordination efforts, as these platforms enable a direct and continuous conversation between the group and its potential recruits. In addition, information shared on these platforms has been found to be perceived by individuals at-risk of becoming extremists as more reliable than news media, as the content is not framed by the perceived biases of these media outlets.¹²

As findings on the increased use and reach of the Islamic State’s online network have developed, technology companies have faced increased pressure from governments and institutions around the world to ramp up efforts to combat the group’s operations. On June 30, 2017, the German parliament passed a law, known as the Network Enforcement Act (“NetzDG”), which requires social media companies and large third-party content hosts (over 2 million registered users) to remove “obviously illegal” speech, including hate speech and terror content, within 24 hours of it being reported, or be subject to fines of up to €50 million.¹³ The law went into full effect at the beginning of 2018 and, within its first few days of enforcement, proved to be problematic. Fearing fines, companies began broadly censoring user speech, which quickly raised concerns regarding freedom of expression online.¹⁴

As content moderation efforts have increased, so have concerns about censorship and the stifling of online freedom of speech

This sort of governmental pressure has been seen in other countries. In 2017, internet platforms in the United Kingdom faced threats of taxation if they were unable to remove extremist content at a faster rate.¹⁵ In addition, in countries such as India and Indonesia, platforms and applications held responsible for facilitating radicalization and coordination have been banned, and a handful remain blocked today.¹⁶

Companies have faced similar pressures from the public and media. In the New Netwar, a report produced by British think tank Policy Exchange in 2017, a poll indicated that 65 percent of those surveyed believed that major internet companies were not doing enough to combat online radicalization, and over 70 percent of them believed these companies should be more proactive in locating and deleting extremist content, as this was considered their responsibility.¹⁷

In response to these pressures from governments and the public, particularly in the European Union, Facebook, Microsoft, Twitter, and YouTube formed the Global Internet Forum to Counter Terrorism (GIFCT) in July 2017.¹⁸ The group aims to share best practices and resources with one another as well as with smaller technology companies facing similar challenges with extremist groups online.

Although the presence of extremist content and individuals online is a problem to address, mandating firms to engage in CVE activities, the effectiveness of which is unproven, is problematic. It prevents companies from engaging in strategic and critical thinking on how to best institute their CVE efforts and approaches and risks resulting in broad censorship by companies to avoid government sanctions. In addition, the lack of meaningful data, clear definitions, and concrete metrics for success, complicate the ability of companies and researchers to evaluate content moderation and removal efforts. It also raises concerns that larger platforms are sharing unproven “best practices” with smaller companies through initiatives such as the GIFCT and creates the risk of steering these platforms away from approaches that could actually have an impact.

As researchers have attempted to assess the effectiveness of content moderation and removal efforts on countering violent extremism and preventing radicalization, the resulting literature has shown mixed results.

In 2017, the British Home Office funded a research project produced by the VOX-Pol Network of Excellence that aimed to assess the impact of the disruption of terror content on Twitter as well as the subsequent effectiveness of the removal of these pieces of content. The researchers monitored and analyzed 722 pro-ISIS accounts, which had collectively posted over 57,000 tweets between February 1 and April 7, 2017. The study found that 65 percent of these pro-ISIS accounts were suspended by Twitter within two and a half months of their creation, and 25 percent of them were immediately taken down after they were created. The researchers concluded that, as a result of these disruption efforts, ISIS’ ability to create and maintain strong and influential communities and relationships on the Twitter platform were significantly diminished, and therefore Twitter’s moderation and removal efforts were successful in countering the group’s operations.¹⁹

The lack of meaningful data, clear definitions, and concrete metrics for success, complicate the ability of companies and researchers to evaluate content moderation and removal efforts. 

Another study, conducted by J.M. Berger and Heather Perez at George Washington University’s Program on Extremism, monitored English-speaking ISIS supporters on Twitter. Although the authors note that this group consists of fewer than 1,000 users on the platform at any time, it is one of the most widely studied communities within the ISIS online network, as a large number of researchers primarily conduct English-language research. The study found that users who had to continuously create new accounts following account suspensions or deletions suffered drastic reductions in their follower counts and faced challenges when trying to regain the same level of engagement and support on the platform. On average, an ISIS user on the platform had between 300 and 400 followers. But, during waves of suspensions, this number decreased significantly, thus suggesting moderation and removal efforts were successful in reducing the presence of the group on the platform. In addition, as accounts were suspended and removed, so was the content they produced, and, as a result, the amount of extremist material on the platform also declined.²⁰

Many researchers, however, contend that moderation practices are ineffective when it comes to countering the Islamic State’s activities online and that the large press coverage and attention the issue of moderating extremist content receives have resulted in companies and governments exaggerating the decline of the group online as well as the effectiveness of moderation practices.²¹ As outlined by Dr. Ali Fisher, a former research fellow at the Center on Public Diplomacy at the University of Southern California, despite sustained suspension and moderation regimes, the Islamic State has continued to generate and share a large amount of extremist content and engage with and radicalize users online. Fisher argues that by aggregating follower numbers, companies and governments have been able to overstate the number of accounts within the group’s network. This has subsequently enabled them to exaggerate the decline in the number of accounts in the network and suggest that the group’s online presence is being successfully disrupted by moderation and removal efforts.²² In addition, Fisher has challenged the findings of Berger and Perez’s 2016 study, which claimed that moderation and removal efforts were successful in decreasing ISIS follower numbers despite the fact that “suspensions held the size and reach of the overall network flat, while devastating the reach of specific users who had been repeatedly targeted.”²³ According to Fisher, if the reach of the overall network remained flat, then even though some accounts suffered significant follower losses, the accounts that remained operational and strong were able to compensate and thus maintain the network’s prior reach levels.²⁴

Other reports, such as Policy Exchange’s the New Netwar report have made similar claims regarding the ineffectiveness of moderation and removal efforts. The report stated that the Islamic State has maintained a consistent online presence and output of content since 2014, producing approximately 100 new pieces of content every week despite attempts to stifle the group’s operations both online and offline.²⁵

Strategic Approaches to CVE Content Moderation and Removal

As companies aim to implement content moderation and removal efforts targeted at terror-related content, and as researchers strive to evaluate the impact of these efforts, a number of factors and variables should be considered in order to enhance the strategicness of these efforts. This will enable companies to have a greater impact, and will permit researchers to develop more nuanced and useful evaluations.

Differentiating Account Roles: Breaking Down the “Swarmcast” in the Islamic State Online Network

One key factor to consider when evaluating the effectiveness of content moderation and removal efforts on terror-related content and accounts is that not all accounts within the ISIS online network play the same role, and, as a result, not all accounts have an equal impact.

In a study examining American ISIS sympathizers on Twitter conducted by Lorenzo Vidino and Seamus Hughes at George Washington University’s Program on Extremism, it was found that users could be categorized into three segments—nodes, amplifiers and shout-out accounts—based on the roles and responsibilities they have within the network. Nodes were considered the principal accounts within these networks and were tasked with creating and sharing primary content, including news articles, official Islamic State tweets and content, and memes. Due to their status as first-line content creators and sharers, these accounts typically amassed a large number of followers. Amplifiers, on the other hand, did not exhibit content creation behavior but rather boosted the reach of node-created content via retweeting and favoriting. The study noted that it was often challenging to identify whether these accounts were operated by actual users or whether they were programmed bot accounts, but, regardless, they played an integral role within the American ISIS-sympathizer network on Twitter. Finally, shout-out accounts supported the reintegration of users who had had their accounts suspended or deleted by advertising newly created accounts within the sympathizer network. Despite the fact that these accounts did not generate or share a significant amount of content, they were found to have had the largest number of followers as they were essential to maintaining the resilience of these Twitter communities.²⁶

Despite sustained suspension and moderation regimes, the Islamic State has continued to generate and share a large amount of extremist content and engage with and radicalize users online.

In Policy Exchange’s the Net Netwar report, similar roles were identified in the Islamic State’s larger digital network, including on platforms such as Telegram. The report found that there was a small group of channels in the ISIS Telegram network that posted original content and a larger number of channels whose primary role was to collect and share this content further. These differential roles were integral to the creation and maintenance of a resilient online network, as they ensured that if primary content creators’ channels were removed, the content they had produced would live on via amplifying channels, and that if an amplifying channel was removed, interested users could still engage with the organization through primary content creators’ channels.²⁷

The importance of having different roles within the larger Islamic State online network can also be understood as integral to maintaining the group’s “Swarmcast.” A Swarmcast is an interconnected network that is constantly reconfiguring itself and is defined by speed, agility, and resilience.²⁸ Given that the Islamic State operates on a number of platforms, including social media platforms, file-hosting websites, forums, and encrypted messaging applications,²⁹ it is important to ISIS that users are able to access content³⁰ within the larger network and that they do not find themselves isolated due to their choice of platform. The Swarmcast is integral for this function. For example, following the November 2015 ISIS attack in Paris, the Islamic State posted a video titled No Respite and translated it into multiple languages. The video was downloaded and viewed approximately 400,000 times on United States-based Archive.org alone despite continued claims by companies such as Facebook and Twitter that the Islamic State’s English-language networks for content distribution had been weakened.³¹ The Swarmcast is also particularly important for the ISIS strategy because following a number of mass suspension and content moderation waves, ISIS users have begun assigning random names to videos, images, and posts, thus ensuring this content cannot be located through simple internet searches. The Swarmcast, therefore, performs an essential role for the organization, as it directs users to content on a number of platforms and ensures they remain engaged.³²

Because the Swarmcast plays such an important role in directing and maintaining the group’s online networks, moderation and removal efforts should focus on disrupting its system-wide structures and operations such as its resilience and speed.³³ One way of doing this is to focus moderation and removal efforts on accounts that play particular roles, as they are integral to maintaining the Swarmcast’s structure. In addition, the fact that different accounts and users play different roles within the ISIS network suggests that the impact of disruption efforts on the overall network is dependent on the type of account disrupted. By disrupting primary content creator channels and accounts, platforms would be able to stifle the entry of new content and information to the larger network. By disrupting the amplifying accounts and channels, platforms would be able to curb the rapid spread of terror propaganda and decrease the reach of the organization. Finally, by disrupting shout-out accounts on platforms such as Twitter, companies would be able to isolate users whose accounts have been previously disrupted and prevent them from reintegrating themselves into the ISIS network, thus decreasing the size and engagement of the group.

As suggested by the New Netwar report, this approach of distinguishing account types is likely to be more successful in disrupting the overall ISIS digital network, as it no longer results in the targeting of general members of the group, for whom suspensions are a mere inconvenience rather than a significant barrier.³⁴ In addition, this method would enable companies to direct their resources toward an approach that yields greater impact and would prove especially beneficial for resource-strapped companies. Researchers who integrate considerations of the differential roles various accounts play, as well as of the Swarmcast’s various operations and characteristics, into their evaluation of account moderation and removal efforts may also be able to achieve a more nuanced understanding and evaluation of these efforts.

By disrupting the amplifying accounts and channels, platforms would be able to curb the rapid spread of terror propaganda and decrease the reach of the organization

However, research on the various roles and responsibilities users have within the Swarmcast is still in its early stages. Further analysis should be conducted to identify other potential roles users may play, characteristics that are most associated with these accounts, and how users interface with one another within the Swarmcast in order to generate resilience and success in the ISIS online network. For example, this could enable the identification of coordinators or recruiters on the platform and could significantly improve the strategy and impact of moderation and removal efforts going forward.

Multi-Platform Moderation Strategies

Although a great deal of attention is focused on the role social media platforms such as Facebook and Twitter play in enabling the operations of the Islamic State, the group has a multiplatform strategy for disseminating content and facilitating recruitment and engagement, and numerous different platforms are often used in tandem.

For example, the 2017 VOX-Pol study found that 12.5 percent of the 57,574 tweets collected from pro-ISIS accounts contained links to 39 other platforms, including Justpaste.it, Archive.org, Sendvid.com, YouTube, Google Drive, and even ISIS’ own server.³⁵ A recent article by researchers from the University of Wollongong in Australia similarly found that Justpaste.it, Sendvid.com, and Dump.to collectively contributed approximately 20 percent of the content and information disseminated by ISIS on Twitter.³⁶ On the other hand, Twitter is recognized as hosting 40 percent of the identifiable traffic to jihadist content on the broader internet, while Facebook is recognized as hosting 12 percent of this traffic.³⁷

According to the VOX-Pol study, there is recognition by some technology companies of ISIS’ multiplatform strategy. Sixty-five percent of the pro-ISIS Twitter accounts that were surveyed in the study and that linked to content on other platforms were suspended within 17 hours of being created, therefore demonstrating that the company’s moderation strategy could have prioritized actioning this type of content.³⁸ The study also found that although the Islamic State utilizes a number of platforms to engage with potential recruits and spread information, Facebook and Twitter are by far the most widely used platforms.³⁹ The study concluded that removing these accounts stifles access to content, even if it is hosted elsewhere online. However, the removal of these accounts on Twitter and Facebook does not subsequently guarantee the removal of this content from the platforms on which they are being hosted. In addition, given the strength and resilience of the Swarmcast, it is likely that users who are within the network will be able to access this content via links posted on other platforms.

However, as demonstrated by the operations of the Swarmcast, the removal of extremist content from one platform does not guarantee the removal of said content from the broader internet. The removal of content from one platform may, therefore, only be a metric of success for that individual company. In many cases, users who are repeatedly disrupted on sites such as Facebook and Twitter will migrate to smaller platforms with fewer resources to address extremist moderation issues or to private, encrypted applications such as Telegram, Signal, and Threema, where it is much harder to detect and monitor the groups’ operations.⁴⁰

The removal of content from one platform may, therefore, only be a metric of success for that individual company. 

Future content moderation and removal efforts and the subsequent measurements of the success of these efforts should consider the flows, exchanges, and relationships between users and content on different platforms in order to improve targeting of moderation and removal efforts. This may in turn generate higher-impact approaches to disrupting the Swarmcast’s operations.⁴¹ In addition, companies should come together to ideate and implement multiplatform content moderation and removal approaches and strategies. The GIFCT could be an avenue for facilitating and evaluating such collaborations.

Definitions and Metrics for Success

Defining clear metrics for success is a challenge for researchers, governments, and companies alike, but essential for understanding what constitutes success in CVE efforts.

For example, as outlined in the VOX-Pol study, 65 percent of the monitored pro-ISIS accounts were suspended within 70 days, and over 25 percent of them were taken down within five days of creation.⁴² The VOX-Pol study concluded that moderation and removal efforts were successful, based on its findings that most of the extremist content monitored was removed over a period of two-and-a-half months. However, in the view of government representatives in the United Kingdom and Germany, for example, in order for moderation to be successful, it must take place within hours rather than days.⁴³ Although there has been contention regarding the actual timeline on which companies should be removing and moderating extremist content in order for it to be truly impactful, it is widely agreed that online-produced content is only impactful if it is viewed by users. Therefore, removing it before it is viewed and measuring the rates of content removal along this timeline is a potential metric for measuring the impact of takedowns. However, we currently lack a concrete set of metrics that define a timeline with which extremist content and accounts should be removed and that is widely adopted and accepted by companies, flaggers, governments, and other relevant stakeholders. As a result, any judgments and conclusions made on the CVE takedown efforts are subjective and prevent cross-company and comprehensive analysis.

Because the Islamic State operates using a multiplatform strategy, successful moderation and removal efforts are relative and based on one platform’s experiences and operations. These metrics do not account for the ripple effects in the broader internet landscape that result from that company’s efforts. These ripple effects include both extremist content migrating to platforms that undergo less moderation or no moderation as well as the Islamic State adapting its own content strategies in response to a large platform’s moderation and removal efforts. Accordingly, companies need to find a way to define clear metrics for assessing lasting success both on individual platforms and the broader internet landscape. Further, because extremist groups will continuously adapt their content strategies to changing platform conditions, metrics need to be dynamic and continuously in development. This is an area where the GIFCT can play a significant role, as the company-led collaborative body aims to share best practices and resources for combatting CVE.

The GIFCT could also serve as a valuable source of data reporting on multiplatform content moderation efforts. One year after the Forum was established, the founding members—Facebook, Microsoft, Twitter, and YouTube—released a joint statement outlining the progress they had collectively made. The group reported that they had built a shared database of over 40,000 digital hashes, which enabled the companies to identify and remove content, such as videos and images, that violated their content policies regarding terror groups.⁴⁴ However, the Forum failed to report on the impact this database has had, if any. For example, it did not provide additional data on the amount of content that had been removed by its member companies as a result of the hashes in the database.

Because the Islamic State operates using a multiplatform strategy, successful moderation and removal efforts are relative and based on one platform’s experiences and operations.

Currently, a small number of companies provide quantitative and qualitative information regarding their CVE content moderation and removal efforts. However, companies often lack concrete metrics to guide this reporting, and, where companies have adopted metrics, they are not consistently applied by all relevant companies, therefore making cross-company comparisons challenging. In recent months some companies have expanded their reporting regarding their content moderation practices, and these new reports provide some helpful models for metrics. However, more companies need to similarly expand their reporting, and further work is needed to create a set of best practices that is adopted across providers and that permits cross-company comparisons.

In July 2017, Automattic, Inc., a web development corporation best known for operating the free blogging platform WordPress.com, began publishing data on its content moderation and removal efforts related to terror content. The company added a new section to its transparency report that highlights notices of terror content on Automattic’s services that it received from government Internet Referral Units (IRUs). The report includes the total number of notices the company received, the total number of notices that resulted in suspended sites, and the total percentage of notices that resulted in suspended sites, and also provides monthly breakdowns for each of these categories.⁴⁵

In April 2018, Twitter released a corporate blog post that spotlighted its CVE moderation and removal efforts in conjunction with the release of its July-December 2017 transparency report. In the blog post, the company reported that between August 2015 and December 2017, it suspended over 1.2 million accounts for violations related to the promotion of terrorism. Of the 274,460 accounts suspended in Q4 2017, 93 percent were independently flagged by Twitter’s internal tools, and 74 percent of these accounts were removed before they were able to produce their first tweet.⁴⁶ In its transparency report, Twitter noted that it received 154 requests from governments around the world to remove content that violates the platform’s terms of service guidelines on the promotion of terrorism. These requests covered 597 accounts and resulted in 98 percent of accounts referred being actioned.⁴⁷

In late April 2018, Google released more data and insights regarding YouTube’s terror content moderation and removal efforts through its first Community Guidelines Enforcement Report, which covered the October-December 2017 period. The report highlighted that out of the 1,598,308 pieces of YouTube content flagged by human agents, including users, NGOs, trusted flaggers, and governments, 1.6 percent was related to the promotion of terrorism.⁴⁸

Following the release of YouTube’s Community Guidelines Enforcement Report, Facebook released its first Community Standards Enforcement Report in May 2018, which covers the period of October 2017-March 2018. The report highlighted that in Q4 2017, the company took action on 1.1 million pieces of terror-related content for violating its community standards. This number rose in Q1 2018 to 1.9 million pieces of content. In addition, in Q4 2017, 96.9 percent of terror-related content was flagged by Facebook’s internal tools before users reported it, and this number rose in Q1 2018 to 99.5 percent. Based on a November 2017 corporate blog post, 83 percent of the terror-related content flagged by internal tools at that time was removed within an hour of it being uploaded.⁴⁹

Of the 274,460 accounts suspended in Q4 2017, 93 percent were independently flagged by Twitter’s internal tools, and 74 percent of these accounts were removed before they were able to produce their first tweet.

These reports demonstrate a broad range of potential data points that companies can collect and disclose to provide further insight into their CVE moderation and removal efforts. However, they also demonstrate that the absence of well-defined metrics and inconsistent reporting undermine meaningful comparisons.

For example, Twitter’s transparency report shared data regarding the number of government notices it received flagging terror-related content that violated Twitter’s terms of service. In a separate blog post it shared the number of terror-related accounts it was able to identify independently. Twitter, however, did not report on legal requests (requests based on a country’s legal frameworks) to remove terror-related content, an increasingly important data point given that countries around the world have been gravitating toward passing legislation requiring companies to remove such content. Although Twitter has made strides in disclosing government requests pertaining to terror-related content, it only does so in the context of terms of service enforcement, and not on a broader scale. Similarly, Automattic reports on the number and results of governmental IRU notices for terror-related content but does not provide insight into the number of government, legal, and user requests related to terror content that it receives nor the subsequent results of these requests. Because the motivations behind the flags from these different parties often vary, distinguishing who submitted these notices in transparency reports is valuable and important and would enable readers to understand the full scope and nature of requests to take down terror-related content. In addition, Automattic’s reporting does not provide any information on the timeline of these takedowns nor on the number of independently identified cases of extremist content.

YouTube’s Community Guidelines Enforcement Report indicates the percentage of terror-related content that was flagged by human users, but does not disclose how much of this content was subsequently removed. Further, the report indicates that the vast majority of content that was removed (80 percent of over 8 million items) was flagged by automated tools.⁵⁰ Yet, the company does not provide a breakdown of these flags based on content category, such as extremist content, as it does for content flagged by humans. This makes it difficult to assess and understand the impact of its moderation and removal efforts on terror content on its platform and makes it difficult to understand the prevalence of terror content on its platform as well. Similarly, the report indicated that 75.9 percent of automatically flagged content was removed before it was viewed, a metric that could have been helpful in understanding the impact of its moderation and removal efforts had it provided a breakdown by content category.

In terms of defining concrete metrics, Facebook’s Community Standards Enforcement Report is the strongest thus far, as it clearly outlines three key metrics which it plans to report on:

• The prevalence of community standard violations on Facebook. This metric is based on the estimated percentage of views that each category of violating content received;⁵¹
• How much content Facebook took action on; and
• How much violating content Facebook was able to independently identify using internal tools before users flagged it.

The company has committed to developing a fourth metric, which will highlight how quickly it takes action on violations. To account for changes in the content landscape, it has also pledged to refine and develop these metrics continuously.⁵² The clear definition of these metrics and why they are important to understanding the moderation of different types of content on the platform is one of the report’s strongest aspects. However, Facebook has yet to provide data on the prevalence metric for terror-related content. The company stated:

Compared to some other violation types such as graphic violence, the number of views of terrorist propaganda content related to ISIS, al-Qaeda, and their affiliates on Facebook is extremely low. That’s because there is relatively little of it and because we remove the majority before people see it. Therefore, the sampling methodology we use to calculate prevalence can’t reliably estimate how much of this content is viewed on Facebook. We’re exploring other methods for estimating this metric.⁵³

By providing an explanation of why this metric is not available for terror-related content, the company gives us greater insight into how metrics are generated and what its limitations are. It also demonstrates the need for greater focus on this type of content in order to ensure proper evaluation is possible, both at the individual platform level as well as across platforms.

In order for best practices and impact in moderation and removal efforts to be identified, companies should collaboratively adopt a set of reporting metrics and standards that they consistently report on. Some granular metrics will undoubtedly vary between platforms, as not all platforms engage with the same types of content, but a uniform set of metrics that can be applied is still helpful and necessary for cross-company comparisons and impact evaluation. ⁵⁴

Along with more consistent metrics for measurement, companies also need to adopt clearer definitions of extremism and specify these in their reporting. In its Community Standards Enforcement Report, Facebook does this by specifying that the extremist content the report covered pertains to ISIS, al-Qaeda, and their affiliates. Similarly, in a November 2017 corporate blog post on Facebook’s content moderation progress, Facebook highlighted that it was able to remove 99 percent of extremist content produced by the Islamic State and al-Qaeda.⁵⁵ Other platforms have failed to specify which particular extremist groups their reporting focuses on, often because companies are hesitant to make these judgment calls; deciding which groups to include amounts to deciding which groups are permitted to speak online and which are not. To remedy this, companies often rely on the delineations of governments from around the world. But, this method is not completely reliably either as, for example, some governments label rival political groups as terrorists.⁵⁶ As a starting point, companies could adopt the U.S. State Department’s designations of Foreign Terrorist Organizations (FTOs).

Companies are hesitant to make these judgment calls; deciding which groups to include amounts to deciding which groups are permitted to speak online and which are not. 

Without clear delineations by companies of what groups their reporting and approaches focus on, it will be difficult to understand who their content moderation and removal efforts are impacting and how. This is an area that needs greater clarity, as research has indicated that there are profound differences in how companies approach different extremist groups and that not all platforms target the same groups equally.

For example, a study conducted by the Middle East Media Research Institute (MEMRI) identified variances in how YouTube approached the moderation of content produced by different extremist groups. In this particular study, MEMRI flagged and monitored videos that celebrated martyrs in jihadi groups on YouTube for a period of two years. They found that videos associated with the Islamic State were removed at a far higher rate than videos associated with other jihadi groups. For example, MEMRI flagged 100 videos that depicted al-Qaeda leader Osama Bin Laden celebrating the 9/11 terrorist attacks. Over the two-year period, 58 of them remained online. Similarly, the group flagged 127 videos featuring al-Qaeda cleric Anwar al-Awlaki and 125 videos of al-Qaeda leader Ayman al-Zawahiri, of which 111 and 57, respectively remained online.⁵⁷

In addition, in the Vox-POL study, in addition to Islamic State content, the researchers also surveyed 62,156 tweets produced by 451 other jihadist groups. They found that these groups were able to produce six times more content and had 13 times more followers than Islamic State accounts on the Twitter platform.⁵⁸ Furthermore, whereas 25 percent of pro-Islamic State accounts were suspended within five days of being created, less than 1 percent of these other jihadist accounts were removed within the same timeline.⁵⁹

These findings, in conjunction with the findings of the MEMRI study, suggest that content produced by less prominent extremist groups is less targeted by moderation efforts. Neither YouTube nor Twitter, however, specify which groups the terror-related content data they disclose pertain to. This prevents us from drawing conclusions on the impact of the company’s moderation and removal efforts on a particular group and creates the impression that the data disclosed is relevant and applicable to all of these groups despite the presence of nuances in how they are targeted. If the companies provided greater insight into the groups that fall under their definition of extremist groups, this would enable improved and more granular analysis of their moderation and removal efforts, and would also allow for researchers to identify groups that the company needs to allocate more resources towards targeting going forward.

Recommendations

Now that Google, regarding YouTube, and Facebook, have both released somewhat comprehensive transparency reports covering content moderation and removal based on their terms of service, other major platforms such as Twitter are likely to follow suit. As these companies expand their transparency reporting practices, they should strive to standardize the data points they share, where possible, and should aim to provide a more holistic overview of their content moderation and removal efforts by disclosing more meaningful data points related to the amount of terror content flagged and removed, the types of terror content flagged and removed, and how terror content was flagged and removed.

Policy and Research Recommendations

In order to better understand the role of the Islamic State online and subsequently guide content and account moderation and removal efforts aiming to disrupt the group, further research needs to be conducted on:

• How ISIS users utilize platforms within the broader ISIS network, such as Telegram, Ask.fm, Tumblr, and Justpaste.it, and how users of these platforms interface with and influence one another within the Swarmcast;
• The varying roles and responsibilities different ISIS users play on various networks and platforms within the Swarmcast;
• Additional factors that could influence and result in declines of ISIS activity and content production online such as technical security regulations and requirements that ISIS leadership mandates users must follow (e.g. banning usage of certain apps), the death of primary online participants (especially in the English-language supporter network), and the adoption of new communications strategies and platforms;
• The unintended consequences of taking down content (e.g. further isolation of at-risk or radicalized individuals, surges in extremist activity on other platforms, etc.);⁶⁰ and
• How non-English ISIS users utilize platforms, such as Facebook and Twitter.

Extremism and radicalization are incredibly complex concepts and processes of which the online experience is only one aspect. Content moderation and removal efforts should not and cannot be considered the single line of defense against radicalization and the expansion of terror groups. As a result, greater research is required on factors that influence radicalization online, such as race, age, social class, education, family background, socio-political contexts and cultural cleavages.⁶¹ In addition, greater research and investment is needed for other online CVE approaches, such as the development of counter-narratives to discredit terrorist propaganda as well as for offline, on-the-ground CVE approaches.

Content moderation and removal efforts will be most effective in decreasing terror content and presence in the broader internet landscape if companies collaborate with one another to counter violent extremism. Companies should continue to collaborate with one another, such as through the GIFCT, but should additionally forge relationships with other platforms in order to expand their shared knowledge base of how extremist content spreads across platforms, and, if possible, coordinate and implement multiplatform moderation strategies.

As part of this work, companies working on CVE should collectively establish clear definitions of the terror groups whose content they are addressing as well as comprehensive metrics for success and data reporting that will enable companies to efficiently allocate their resources and will permit researchers to monitor, assess, and compare moderation and removal efforts.

Transparency Reporting and Data Disclosure Recommendations

Currently, companies who make data disclosures regarding their content and account moderation and removal efforts pertaining to terror-related content disclose a wide variety of data with little standardization and little meaningful granularity. Because of this, evaluation of these content moderation and removal efforts is challenging. In order for individual company and collaborative industry moderation and removal efforts to be comprehensively understood, evaluated, and compared, companies should aim to regularly and uniformly report on the data points listed below.⁶² These data points are based on the frameworks for best practices in transparency reporting put forth by the Transparency Reporting Toolkit, established by New America’s Open Technology Institute and the Berkman Klein Center for Internet & Society⁶³ as well as the Santa Clara Principles on Transparency and Accountability in Content Moderation,⁶⁴ which were released in May 2018 by a group of organizations, advocates, and academic experts. These suggested data points should serve as a starting point for the minimum amount of data companies should seek to disclose regarding their efforts to moderate and remove terror-related content. This does not mean that new metrics or data points are not welcome or helpful. As extremist groups continuously adapt their online content strategies, new metrics will undoubtedly be needed to measure and evaluate subsequent approaches.

Individual companies should aim to report on the following data points. Unless otherwise indicated, all quantitative data points can and should be reported in numerical form, percentage form, or, if possible, in both, and should be reported for every reporting period.

The GIFCT should also strive to convene and facilitate similar reporting on joint company moderation and removal efforts. Below are suggested data points that the GIFCT should disclose. Once again, these data points serve as a starting point for the minimum amount of data the GIFCT should seek to disclose regarding its members’ efforts to moderate and remove terror-related content. This does not mean that new metrics or data points are not welcome or helpful. Unless otherwise indicated, all quantitative data points can and should be reported in numerical form, percentage form, or if possible, in both, and should be reported for every reporting period.

As more research is conducted on the role of extremist groups operating online as well as on the impact of moderation and removal efforts on extremist groups, reporting should expand to include the amount of different types of accounts (e.g. nodes, amplifiers, shout-out accounts, etc.) that have been disrupted as well as the amount of content that linked to other platforms that was disrupted.

Looking to the Future

Although the Islamic State is today considered to be the most prominent and tech-savvy Islamic extremist group, numerous scholars have predicted the demise of the organization, as despite its robust online presence, the organization has tied its success to physical assets. However, despite its offline decline, the group has built up a large compendium of digital resources that can still serve as sources of ideological inspiration for at-risk individuals, including lone-wolf actors around the world, and foster the coordination and implementation of acts of terror by others.⁶⁵

In addition, this “Ghost Caliphate” can serve as a valuable resource for budding extremist groups such as the Tahrir-al-Sham (also known as the Levant Liberation Committee) that have thus far demonstrated tendencies to root their ideological operations in the digital sphere.⁶⁶ This suggests that online counter extremism work is going to remain important in the future. Companies, governments, civil society, and researchers will have to work together to establish clearer definitions and metrics for success within this space, identify and implement impact evaluation methodologies, advocate for increased multiplatform moderation approaches, and disclose more meaningful and valuable data related to company content moderation and removal efforts. These approaches will enable companies to better strategize their efforts and enable researchers and companies to better evaluate and understand the impact of their programs. This will further benefit companies as it will more intelligently guide their allocation of resources and personnel and aid in delivering stronger and more impactful programs and results going forward.

Spandana Singh is a 2017-18 Millennial Fellow with the Open Technology Institute at New America. She thanks Liz Woolery, Sharon Bradford Franklin, Joshua Geltzer, Melody Frierson, and Reid Cramer for reviewing and supporting the development of her report.