Case Study: Tumblr

Tumblr is a microblogging and social media website currently owned by Verizon Media. The platform ranks 78th for global internet engagement,¹ and as of April 2017, it had 738 million unique visitors worldwide,² with 2019 statistics citing that the number of blog accounts on the platform had grown to 463.5 million.³ The company utilizes a centralized, hybrid approach to content moderation, although it is unclear how many human moderators the company employs and to what extent the platform uses automated tools to moderate content. Although Tumblr is not one of the largest or most widely used internet platforms, it is an interesting one to consider when assessing how algorithmic decision-making is deployed for content moderation purposes because the platform recently amended its Community Guidelines to ban adult content and nudity. Before this policy change, the platform was considered a haven for graphic forms of expression on the internet. However, as of December 2018, Tumblr’s rules were updated to state that pornography and adult content were no longer permitted on the platform.⁴ In its announcement to users, Tumblr outlined that any content that violated this new policy would be flagged using a “mix of machine-learning classification and human moderation.” The use of automated tools to flag potentially violating content in this case makes sense, as the platform had millions of posts and blogs featuring adult content, and the scope and scale of removing such content could not be achieved by human moderators alone.⁵ However, identifying and removing such content also requires context. For example, although an algorithm could be trained to identify all images with female breasts, they are unlikely to be able to distinguish whether these images are graphic in nature or whether they are discussing or depicting mastectomies, gender confirmation surgeries, or breastfeeding. As a result, it is vital that a human moderator always remains in the loop during the content moderation process.

Prior to this announcement, Tumblr only filtered out adult content through its “Safe Mode” feature that allows users to select which content they will see. Shortly after Tumblr was acquired by Verizon Media in June 2017, it introduced this opt-in feature in order to let users filter “sensitive” content from their own dashboard and search results. However, the feature had flaws. Users quickly found that it filtered out non-adult content, including LGBTQ+ posts. It is unclear whether the company is deploying the same artificial intelligence technology used for Safe Mode to implement this new platform-wide ban on adult content, but WIRED reported that the company would be using modified proprietary technology. The company also announced it would be hiring more human moderators.⁶

Tumblr’s use of algorithmic decision-making to institute this new platform-wide ban on adult content created a range of issues. Following the introduction of the policy, users took to Twitter to document the numerous cases of erroneous takedowns of content which included the removal of everything from chocolate ghosts, to Joe Biden,⁷ to a cartoon scorpion with the hashtag #TooSexyForTumblr.⁸

One of the reasons for these flaws could be that Tumblr’s definition of adult content spans a range of content formats, including photos, videos, and GIFs which depict human genitals or female-presenting nipples. They also include photos, videos, GIFS, and illustrations that depict sex acts. In order to effectively identify and remove this content, Tumblr’s algorithms therefore need to be able to make complex determinations to prevent overbroad takedowns of user content. These include ensuring that any representation of human genitals or sexual body parts are real-life imagery and are not part of artwork such as paintings or sculptures. In addition, these include understanding the context behind certain forms of nudity, such as the previously discussed examples regarding depictions of breasts. This is not a problem unique to Tumblr. In some cases, more training data can help automated tools understand context and nuances to some extent. For example, if you are trying to teach a model the difference between non-sexual depictions of breasts, such as breastfeeding, and graphic depictions of breasts, you could provide it with more data to learn from. As a result, the model would likely logically decipher that most images of breastfeeding contain infants or children. However, based on this assumption, a user posting graphic depictions of breasts could avoid detection by featuring an infant anywhere in their image. In comparison to other categories of objectionable content, such as extremism and disinformation, however, adult content has clearer definitions and is easier to moderate and train models on, as adult content has stronger and more consistent visual elements.⁹

In addition, the wide range of content that was erroneously removed by Tumblr after the ban serves as an example of the impact of dataset and creator bias, as well as the concerning lack of accuracy and reliability in the platform’s models.¹⁰ For example, WIRED researchers ran several of the Tumblr posts that were erroneously removed through Matroid’s NSFW natural imagery classifier. The classifier correctly identified each one as not adult content (although it did indicate there was a 21 percent chance the chocolate ghosts could be adult content). This demonstrates a weakness in Tumblr’s own classifiers and raises questions around how the platform is training its models and with what datasets.¹¹ Machine learning models need to be trained on a vast amount of data in order to operate effectively and improve. Most of the models used in content moderation are supervised machine learning models, meaning that the content has been annotated to indicate whether or not it falls into a particular category. Platforms that have prohibited, and as a result moderated, nudity and adult content for a long period of time have a vast amount of annotated data in this domain that they can use to train their models on.

However, because Tumblr did not previously prohibit nudity and adult content on its platform, it likely did not have the same robust datasets that competing platforms have.¹² Researchers such as Tarleton Gillespie have speculated that the company began reviewing and annotating images before introducing the amendment to their Community Guidelines in order to obtain the training data they needed. However, the backlash from the rollout has indicated that this data was not sufficient.¹³ Although users are offered the opportunity to appeal adult content takedown decisions on Tumblr to a human moderator, this demonstrates a clear limitation and weakness in its adoption and implementation of automated tools for content moderation. Going forward, the company will need to develop more comprehensive datasets to train its classifiers on to ensure moderation of user speech in this category is more accurate and reliable.¹⁴

In addition, in order to provide greater accountability around its content takedown decisions, the platform should expand its appeals process so that users have the power to contest takedown decisions as a whole. Tumblr does not clearly disclose the types of takedowns for which appeals are available. Furthermore, Tumblr needs to begin providing adequate notice to users if their content or accounts are removed or suspended. These notices should offer meaningful explanations to users as to why their content or accounts were impacted. This should, at a minimum, include a URL, content excerpt, or other summary that enables the user to understand what specific content violated Tumblr’s Community Guidelines, which of Tumblr’s Community Guidelines the user violated, how the content was detected and removed, and how the user can appeal this decision.

“Further, with the use of black box algorithms, even the developers may not know how automated tools are making decisions.”

In addition, Tumblr’s content moderation system may have identified patterns between objects in images that its developers did not teach it, and as a result could be removing content erroneously. This demonstrates how algorithms can exacerbate hidden biases in the training data. Further, with the use of black box algorithms, even the developers may not know how automated tools are making decisions.¹⁵ Greater transparency around the data that models are trained on could help provide further insight into these processes. Without greater transparency into how these automated content moderation processes are impacting user speech, it is impossible to properly understand how accurate these algorithmic decision-making tools are, how much user speech they are impacting, and what percentage of the overall content being removed is accurately being removed. As with most public rollouts, the negative aspects and mistakes take center-stage and the positive attributes are left out. Greater transparency from Tumblr on this could shed further light on the successes and challenges associated with introducing new automated tools, and with leveraging existing automated tools to support the enforcement of new policies.

One way Tumblr could do this is by disclosing more data on their content takedowns in its transparency report. Currently, the transparency report only discloses data around copyright- and trademark-related content removals.¹⁶ A more comprehensive report should cover the total number of posts and accounts flagged and removed. In addition, it should include a breakdown of the posts and accounts flagged and removed organized by which of Tumblr’s Community Guidelines were violated, the format of the content at issue, and how they were flagged. These recommendations follow the Santa Clara Principles on Transparency and Accountability in Content Moderation, which outline minimum standards tech platforms must meet in order to provide adequate transparency and accountability around their efforts to take down user-generated content or suspend accounts that violate their rules. A more comprehensive transparency report should also highlight the number of appeals the platform received and how much content was restored on the platform based on the platform’s proactive efforts as well as user appeals.