Russian Interference, Radicalization, and Dishonest Ads: What Makes Them So Powerful?

Russian interference in recent U.S. elections and online radicalization by proponents of violent extremism are just two recent, large-scale examples of the content problems that we reference above.

Following the 2016 election, it was revealed that Russian government actors had attempted to influence U.S. election results by promoting false online content (posts and ads alike) and using online messaging to stir tensions between different voter factions. These influence efforts, alongside robust disinformation campaigns run by domestic actors,¹ were able to flourish and reach millions of voters (and perhaps influence their choices) in an online environment where content-shaping and ad-targeting algorithms play the role of human editors. Indeed, it was not the mere existence of misleading content that interfered with people’s understanding of what was true about each candidate and their positions—it was the reach of these messages, enabled by algorithms that selectively targeted the voters whom they were most likely to influence, in the platforms’ estimation.²

The revelations set in motion a frenzy of content restriction efforts by major tech companies and fact-checking initiatives by companies and independent groups alike,³ alongside Congressional investigations into the issue. Policymakers soon demanded that companies rein in online messages from abroad meant to skew a voter’s perspective on a candidate or issue.

How would companies achieve this? With more algorithms, they said.⁴ Alongside changes in policies regarding disinformation, companies soon adjusted their content moderation algorithms to better identify and weed out harmful election-related posts and ads. But the scale of the problem all but forced them to stick to technical solutions, with the same limitations as those that caused these messages to flourish in the first place.

Content moderation algorithms are notoriously difficult to implement effectively, and often create new problems. Academic studies published in 2019 found that algorithms trained to identify hate speech for removal were more likely to flag social media content created by African Americans, including posts using slang to discuss contentious events and personal experiences related to racism in America.⁵ While companies are free to set their own rules and take down any content that breaks those rules, these kinds of removals are in tension with U.S. free speech values, and have elicited the public blowback to match.

Content restriction measures have also inflicted collateral damage on unsuspecting users outside the United States with no discernible connection to disinformation campaigns. As companies raced to reduce foreign interference on their platforms, social network analyst Lawrence Alexander identified several users on Twitter and Reddit whose accounts were suspended simply because they happened to share some of the key characteristics of disinformation purveyors.

One user had even tried to notify Twitter of a pro-Kremlin campaign, but ended up being banned himself. “[This] quick-fix approach to bot-hunting seemed to have dragged a number of innocent victims into its nets,” wrote Alexander, in a research piece for Global Voices. For one user who describes himself in his Twitter profile as an artist and creator of online comic books, “it appears that the key ‘suspicious’ thing about their account was their location—Russia.”⁶

The lack of corporate transparency regarding the full scope of disinformation and malicious behaviors on social media platforms makes it difficult to assess how effective these efforts actually are.

For four years, RDR has tracked whether companies publish key information about how they enforce their content rules. In 2015, none of the companies we evaluated published any data about the content they removed for violating their platform rules.⁷ Four years later, we found that Facebook, Google, Microsoft, and Twitter published at least some information about their rules enforcement,⁸ including in transparency reports.⁹ But this information still doesn’t demonstrate how effective their content moderation mechanisms have actually been in enforcing their rules, or how often acceptable content gets caught up in their nets.

Companies have also deployed automated systems to review election-related ads, in an effort to better enforce their policies. But these efforts too have proven problematic. Various entities, ranging from media outlets¹⁰ to LGBTQ-rights groups¹¹ to Bush’s Baked Beans,¹² have reported having their ads rejected for violating election-ad policies, despite the fact that their ads had nothing to do with elections.¹³ Yet companies’ disclosures about how they police ads are even less transparent than those pertaining to user-generated content, and there’s no way to know how effective these policing mechanisms have been in enforcing the actual rules as intended. ¹⁴

The issue of online radicalization is another area of concern for U.S. policymakers, in which both types of algorithms have been in play. Videos and social media channels enticing young people to join violent extremist groups or to commit violent acts became remarkably easy to stumble upon online, due in part to what seems to be these groups’ savvy understanding of how to make algorithms that amplify and spread content work to their advantage.¹⁵ Online extremism and radicalization are very real problems that the internet has exacerbated. But efforts to address this problem have led to unjustified censorship.

Widespread concern that international terrorist organizations were recruiting new members online has led to the creation of various voluntary initiatives, including the Global Internet Forum to Counter Terrorism (GIFCT), which helps companies jointly assess content that has been identified as promoting or celebrating terrorism.

Scale matters—the societal impact of a single message or video rises exponentially when a powerful algorithm is driving its distribution.

The GIFCT has built an industry-wide database of digital fingerprints or “hashes” for such content. Companies use these hashes to filter offending content, often prior to upload. As a result, any errors made in labeling content as terrorist in this database¹⁶ are replicated on all participating platforms, leading to the censorship of photos and videos containing speech that should be protected under international human rights law.

Thousands of videos and photos from the Syrian civil war have disappeared in the course of these efforts—videos that one day could be used as evidence against perpetrators of violence. No one knows for sure whether these videos were removed because they matched a hash in the GIFCT database, because they were flagged by a content moderation algorithm or human reviewer, or some other reason. The point is that this evidence is often impossible to replace. But little has been done to change the way this type of content is handled, despite its enormous potential evidentiary value.¹⁷

In an ideal world, violent extremist messages would not reach anyone. But the public safety risks that these carry rise dramatically when such messages reach tens of thousands, or even millions, of people.

The same logic applies to disinformation targeted at voters. Scale matters—the societal impact of a single message or video rises exponentially when a powerful algorithm is driving its distribution. Yet the solutions for these problems that we have seen companies, governments, and other stakeholders put forth focus primarily on eliminating content itself, rather than altering the algorithmic engines that drive its distribution.