The Benefits and Challenges of Transparency

Transparency reporting varies in scope and methods across industries and platforms. Building on the background provided above, the panel discussed and explained a number of similarities in the methods, and both the perceived and proven benefits of transparency reporting across industries. In addition, the panelists also highlighted that there were many similar risks and challenges associated with transparency reporting (and transparency efforts in general) across their sectors. The following section of this report breaks down the different methods, risks, benefits, and challenges discussed during the panel.

As outlined during the panel, transparency is defined differently across industries and even companies. As a result, the methods, goals, and outcomes of transparency efforts often vary.

Google’s David Lieber, for example, shared that the practice of issuing transparency reports on government requests for user data has served as a valuable mechanism for holding both governments and companies accountable for their practices related to surveillance and access to user data. In addition, Lieber highlighted the important role such transparency reports have played in informing related public policy discussions. OTI’s Spandana Singh echoed these sentiments, describing how transparency reporting on removals of extremist content and accounts by various platforms has enabled these platforms to demonstrate the extent of their efforts to combat extremist propaganda and influence online. In addition, these qualitative and quantitative data points in transparency reports have added context to ongoing public debates around the role platforms should be playing in online anti-extremism efforts. They have also served as a response to concerns from regulators, governments, and watchdog groups (including OTI) that platforms are not doing enough to secure their services from extremist groups.

For John Hopkins Medicine’s Allen Kachalia and Uber’s Rana Kortam, transparency reporting has proven to be a mechanism for sparking industry-wide improvements and changes in technology and healthcare.

As previously outlined, CMS requires some hospitals to disclose data on process and outcome measures. Some hospitals have also begun publishing such data independently. Kachalia noted that although patients do not necessarily seek out or review this data, publishing it publicly creates an incentive for healthcare providers to improve their services. In this way, transparency reporting efforts can drive industry-wide improvements in terms of standards for quality and care.

Kortam highlighted similar goals in the technology industry. According to Kortam, Uber sought to use its safety transparency report to demonstrate to users and the public how the online Uber platform interfaces with offline issues, such as road safety and sexual assault and misconduct. She noted that going forward she hopes other platforms will follow suit and release such reports, and that these companies could then collectively tackle these industry-wide issues.

Singh also highlighted the important role that transparency reporting can play in helping companies build, or rebuild, trusting relationships with their users and policymakers. For example, in 2018, both Motel 6 and 7-Eleven were found to have shared data on their guests and customers with law enforcement agencies. This resulted in a significant amount of backlash and a dip in consumer trust.¹ Voluntary transparency efforts by these companies could help remedy and rebuild these relationships going forward, and also highlight the scope and scale of such data-sharing efforts.

Transparency reporting is a mechanism for providing long-term transparency to users and the public about how a company operates and what impact it has on its stakeholders. However, Kachalia, Singh, and Lieber also explained the need for companies and institutions to provide short-term and real-time transparency to impacted users.

Although it is important for healthcare providers to report on any mistakes or malpractice incidents publicly, Kachalia noted that when a doctor or hospital makes a mistake, it is vital that they also notify the impacted patients and their families immediately. This often enables those impacted to mitigate the harm or to obtain closure. In addition, Kachalia also stated that as hospitals have begun providing greater real-time notice and transparency to impacted individuals, the number of claims for liability have actually dropped.² Both real-time notice and long-term transparency are integral for strengthening and rebuilding trusting relationships between providers and users.

Similarly, Singh noted that although internet platforms should publish overarching transparency reports that describe the scope and scale of their content takedown efforts, they should also be engaging in real-time transparency and mitigation efforts. In practice, she noted, this means that when a user’s content or account is suspended or removed by a platform, they should receive adequate notice explaining why this decision has been made. This notice should also be accompanied with the ability to appeal this decision in a timely manner.

Lieber also discussed the importance of providing notice to users who have been the subject of user data requests. However, he explained that in the case of government and law enforcement requests for user data, providing such notice may not always be possible, as at times, platforms may be prohibited from doing so by specific legal requirements or the scope of the legal order.

As calls for transparency are becoming more prominent across an array of industries, companies are facing increased pressure to issue transparency reports outlining the scope and scale of their operations and impact. However, many organizations and institutions have expressed concerns around the risks associated with publishing data on sensitive subjects.

For example, Kortam discussed the concerns the company had about becoming the first tech platform to release statistics and information related to safety incidents. “Safety should not be proprietary,” Kortam stated, and additionally, publishing such data should not suggest that one service is more or less safe than another. Rather, she hoped that publishing this data would encourage other companies to issue similar reports, and subsequently spark collaborative efforts to improve their services and hold each other to a higher standard of accountability when tackling safety issues.

Kachalia echoed these sentiments, stating that in terms of patient safety and quality in the healthcare industry, “if we punish everybody for putting those numbers out there, we’re never going to get better.” Rather, providers should publish these numbers so that patients and industry professionals can better understand risks and then begin working together to improve outcomes and scenarios.

As described in this report, there are currently a number of entities that are required by law to publish transparency reports. However, there are also a range of organizations and institutions that issue transparency reports on a voluntary basis. The panelists did not discuss which approach would be more valuable or preferable. However, Singh and Kortam both noted that voluntary reporting efforts are a useful mechanism for companies to provide greater transparency and accountability around their operations and impacts, and develop trusting and open relationships with their users and policymakers.

Today, many governments around the world are considering mandating that internet platforms issue transparency reports on areas such as content removals, so this will be an issue area to watch going forward.

One of the primary challenges related to transparency reporting that all of the panelists highlighted was how difficult it is to ensure standardization in reporting across an industry.

For example, Singh noted that one of the greatest challenges associated with evaluating how internet platforms are moderating and removing content is the fact that their transparency reports vary in terms of the metrics and data points they disclose. As Singh explained, this variance in reporting occurs for a number of reasons. First, internet platforms often have varying policies on content removal, sometimes even among their own products. As a result, reporting on exactly the same metrics is often challenging. Further, there is an ongoing debate between platforms regarding what meaningful transparency around content takedowns looks like. As a result, different platforms will report different metrics and data points based on their own understanding of meaningful transparency. While this provides researchers, policymakers, and the public with a range of valuable metrics, it makes comparing company efforts and outcomes difficult. Kachalia described similar concerns around standardization in the healthcare industry, stating that most hospitals have varying determinations of what metrics and data points should be disclosed, and how they should be measured. These variations have hindered effective comparisons of institutions. In addition, Lieber highlighted that legal obligations often complicate such reporting further, as some platforms may be impacted by regulations more than others, and this can impact their reporting.

Standardization is also an issue that Uber grappled with when crafting their safety transparency report. As Kortam discussed, when Uber began working on creating their safety report, they realized that there was no existing way to consistently classify and report on the types of incidents of sexual violence and sexual misconduct that their users were experiencing. In order to remedy this, Uber partnered with the National Sexual Violence Resource Center and the Urban Institute to develop a taxonomy for categorizing and counting incidents of sexual violence and sexual assault consistently. The company then applied this taxonomy to past safety reports they had received from drivers and users in the United States. In the spirit of encouraging other companies to report and to issue similarly standardized reports, Uber made this taxonomy open source, so other platforms can adopt and apply it.

Organizations such as OTI have released recommendations for how companies can improve and standardize their transparency reports. As noted in the background section above, OTI has published two transparency reporting toolkits, one on government requests for user data and another on content takedowns. These toolkits survey how internet platforms and telecommunications companies have been reporting on these issues thus far, and offer best practices for ensuring that future reporting is standardized, granular, and meaningful. In addition, in 2018 a coalition of organizations, advocates, and academics including OTI, released the Santa Clara Principles on Transparency and Accountability in Content Moderation. The principles outline minimum standards related to transparency reporting, providing adequate notice, and providing a robust appeals process that internet platforms should meet when seeking to provide transparency and accountability around their content takedown efforts.³ Thus far, several platforms have incorporated these recommendations into their reporting.

However, Kachalia also stressed the need for both flexibility and guidance when shaping transparency reporting. He warned that offering entities a predetermined “checklist of metrics” could prevent them from innovating and sharing information using other meaningful metrics. As a result, companies should work to craft their own meaningful standards for reporting as well.

Another major hurdle related to transparency reporting that panelists discussed was that there are significant challenges associated with creating meaningful and useful transparency reports that are also comprehensible for a general audience.

According to Kachalia, although there is a push for hospitals and healthcare providers to publicly share outcome data, this data is irrelevant unless providers can ensure that patients and the general public understand what this information means. Similarly, Singh described how transparency reports issued by technology companies are typically packaged so that they are of the most use to researchers, journalists, and policy-focused professionals, rather than users. Panelists noted that platforms therefore also need to think through how they can package these data points and insights in a manner that is valuable for users.

In addition, Kortam and Singh noted that across industries and issue areas, there is a push for transparency reporting efforts to be coupled with information around what companies are doing to subsequently improve their products and services. Kortam, for example, described how while developing the safety transparency report, many external stakeholders (such as civil society groups and organizations focused on sexual violence and sexual misconduct issues) pushed Uber to use the report as a way to also explain what the company was doing to prevent future safety incidents. Similarly, Singh echoed these points and stressed the need for companies to engage with such external stakeholders and experts in order to develop a more nuanced understanding of what a meaningful and useful transparency report looks like, both in terms of metrics and data reported, and in terms of the supplementary information provided.

Finally, the panelists also described a range of procedural and logistical challenges that often arise when a company or organization seeks to issue a transparency report or engage in a similar transparency effort.

As Lieber highlighted, the practice of issuing regular transparency reports is a long-term commitment that requires a significant amount of engineering and human resources. In addition, Kachalia explained that in the healthcare industry, most electronic systems have been designed to be transactional, rather than to foster data collection and extraction and performance evaluation. As a result, getting data out of these systems, both for mandatory reporting such as the CMS required reporting and voluntary reporting, has been challenging. Kortam shared experiences of similar challenges at Uber, prompting the creation of the taxonomy that the platform now uses. She also described the immense number of resources it took to retroactively apply this taxonomy to reports the company had received in the past.