Privacy Considerations and User Controls

Joe Westby, technology and human rights researcher at Amnesty International, noted during one of our events on algorithmic content shaping that there has been a longstanding push for comprehensive privacy legislation in the United States.¹ Despite this, the United States has still not enacted comprehensive federal privacy legislation, and the U.S. Congress has not yet even begun debating proposals. As a result, algorithmic content-shaping systems that rely on the collection of vast amounts of personal and behavioral data have become ubiquitous in today’s digital world. These data collection practices are part of what has been termed the “surveillance capitalism” economy, where companies monetize data on user behaviors and interests.² This can lead to further data collection practices because the companies—which are profiting financially from collecting and harnessing user data—may be incentivized to create vast datasets that can be used to both further train algorithmic systems and increase revenue.³Unless individuals are prepared to forego using these services, which are often integral to daily life,⁴ they may feel they have little choice but to accept that companies will continue to collect and monetize their personal data. Additionally, these expansive data collection practices are often not rights-respecting, and there is little transparency and accountability around how user data is collected, used, shared, and retained.⁵

For example, internet platforms collect vast amounts of data on users in order to compile their datasets and train their algorithms. However, users have little insight into this process and how their data is being used, and have little control over whether their data is used for these purposes. In addition, these algorithmic systems construct profiles about each user with little transparency or accountability. There is also little visibility into what data is being used as inputs into these systems, how these systems are processing this information, and what outputs they are generating. In this way, these systems assign an identity to users, and users have little agency to control or change the assumptions that are made about them. This is particularly concerning given that researchers have found that algorithmic systems—such as recommendation engines and ad targeting and delivery systems—can yield discriminatory, biased, and harmful results that disproportionately impact communities of color and other marginalized groups.

As companies have begun thinking through how to prevent further harmful and discriminatory outcomes and increase fairness in their systems, algorithmic audits have emerged as a method for assessing potential negative impacts. Researchers considering how these audits should be structured have raised questions related to what kind of data could be used to audit and better test existing algorithmic systems, and whether this would require collection of (and access to) sensitive information. Currently, many technology companies do not collect explicit data about race. However, their algorithms can infer race through certain data points. For example, an algorithm could assume an individual’s race based on whether they live in a neighborhood or ZIP code that has historically been associated with a specific racial group. Similarly, an algorithm could infer an individual’s race based on their interest in particular affinity groups or products, such as hair care products designed for Black women. These inferences are not always accurate, however, and thus using this data for testing purposes would not be as valuable. This has raised questions around whether companies should begin collecting or purchasing sensitive demographic data, such as race, in order to enable fairness testing, as well as what limits must be placed on this data if it is obtained. This raises additional questions around whether the benefits associated with having such data outweigh the potential harms that could result.

In addition to collecting a troubling amount of user data, internet platforms also offer users only a limited set of controls with which they can understand and determine how their experiences are being personalized.⁶ These tools are continuously evolving, and feedback from civil society, researchers, and other stakeholders is vital to driving this development process forward. As companies introduce these controls, it is important that they make the controls accessible and digestible, and do not require users to search through multiple pages or drop-down menus in order to acquire a certain piece of information or change a specific setting.⁷ Companies should also enable data portability, which would give users the ability to extract an archive of the data that they have shared with an internet platform (or that the internet platform has collected about them) in a format that is structured, machine-readable, and that allows transfer of this data to a different service.⁸ By enabling users to transfer their data from one service to another, and to use the data for their own purposes, data portability provides users with more agency and control over which companies have access to their sensitive data and how it is used.⁹ The cross-industry Data Transfer Project, which aims to create an open-source, service-to-service data portability platform that enables users to move their data between online service providers is a good starting point for these efforts.¹⁰

(Disclosure: New America receives funding from Apple, Facebook, Google, and Microsoft. View our full list of donors at www.newamerica.org/our-funding.)