Recommendations for Preventing Abusive Algorithmic Tools

Addressing the potentially harmful outcomes of AI will require a critical examination of these tools at each stage of their development and implementation. Algorithmic tools disproportionately harm people of color—especially Black and Brown communities—women, immigrants, religious minorities, members of the LGBTQ+ community, low-income individuals, and other marginalized communities. In addition to discriminatory impacts, the expansive collection of personal data necessary to power the algorithmic tools used in criminal justice, education, and employment settings harms individual privacy. Therefore, it is essential to center civil rights in the privacy debate and advocate for policies that would prevent discriminatory data practices.¹ Transparency requirements, impact assessments, and regular audits are all necessary and can be mandated in comprehensive privacy legislation. Whether or not there are legal requirements, institutions need to evaluate the consequences of algorithmic tools and determine if it is possible to deploy these systems equitably.

Transparency is a necessary, but not sufficient, condition for reforming discriminatory algorithmic systems. According to Gillmor, “the more that we can understand these systems the better we can defend ourselves against them.” Some systems use less complex algorithms that are more easily explained. Albert III noted that many risk assessments are more accurately described as calculators than as AI, and simply project the generalized assumptions about recidivism of demographic groups onto an individual belonging to that group. For example, the age of a defendant is a heavily weighted demographic in risk assessments. The system used in Virginia assigns a defendant more risk points for being 23 years old (as opposed to being older) than for having five or more prior incarcerations.² But, Albert III explained, if criminal defendants do not know what factors are used to calculate a risk assessment, they will not be able to defend themselves adequately. Therefore if a pretrial risk assessment is used, it “must be transparent, independently validated, and open to challenge by an accused person’s counsel.”³ For risk assessments, the current lack of transparency is often a legal issue rather than a technical issue.

When algorithms are more complex, such as systems based on machine learning, transparency can be a technical challenge. For black box systems, even the developers who created the system may be unable to explain precisely how the system operates and why it produced a particular output. Yet, when such systems are used to make critical decisions about a person’s life—including criminal justice, education, and employment determinations—it is not acceptable for the algorithms to be opaque and sealed from scrutiny. It is particularly important for these types of algorithms to undergo intensive audits.

Before an institution decides to implement an AI tool, it should conduct an impact assessment to evaluate the potential risks. Palmer recommends that colleges require a predictive analytics vendor to agree to conduct a disparate impact analysis before entering a contract. To assist education institutions, she wrote a guide that colleges can use before partnering with a vendor.⁴ The same principles apply to other institutions that hire vendors to create algorithmic models to predict outcomes for individuals; they should provide transparency, privacy, security, equity, and regular evaluations.

After an AI system is deployed, it should also undergo regular audits to detect flaws or harmful consequences. Since software requires continuous updates to fix bugs and make improvements, audits need to be ongoing to stay up to date with the current version of the software. Moreover, when a system introduces new training data, audits must be updated to include an evaluation of that new data. Although audits can be costly, due process does not become less important because it is resource intensive. If an institution finds that regular audits will be too costly, they should reconsider using AI systems. Organizations and entities who perform audits need to consider the possibility that a system should be temporarily or permanently suspended if a discriminatory impact or other harmful results are identified. For example, at the urging of Facebook employees, in July 2020 the company launched an investigation into whether its machine learning algorithms discriminate against minority ethnic groups.⁵ This internal review should be a helpful step in light of a final audit report, conducted by the company at the urging of civil society groups, which outlined a variety of discriminatory practices and made a series of concrete recommendations to address the harms the company perpetuates.⁶ But in order for the investigation to be meaningful, it must include the possibility that Facebook will need to suspend a profitable business practice.

Some AI systems could be used both to perpetuate historical discrimination or to attempt to confront it. Criminal justice authorities, institutions of higher education, and employers should evaluate their reasons for using an algorithmic tool and how they are framing the questions they want the tool to answer. People are prone to misinterpreting probabilities,⁷ and conclusions based upon faulty assumptions about the meaning of an algorithmic model’s predictions can have detrimental consequences. Palmer recommends that universities train advisors how to interpret and convey predictive data to avoid misuse of the algorithm’s outputs. For example, instead of a college advisor using a predictive model to discourage a student from pursuing a certain major, that advisor could use the same output to direct the student toward appropriate resources. In the criminal justice context, the way a risk assessment is presented to a judge or jury is essential to preserving the presumption of innocence. If risk assessments are used at all, they should only identify groups of people to be released immediately because automated recommendations for detention assume the guilt of the defendant.⁸

At the most fundamental level, institutions need to consider reframing the questions they ask these tools to answer. Algorithmic systems that reflect biases within an institution can be used to reform that institution rather than to make discriminatory decisions about individuals. As Gillmor noted, when a hiring system categorizes women as less likely to succeed at a company than men, that finding should be used to address a corporate culture of sexism rather than for hiring. Amazon correctly decided not to use its system in hiring after detecting that it was reflecting such biases. The next step would be for the company to examine why its system found that women were less likely to succeed. Palmer explained that risk assessments that identify Black students as less likely to graduate are reflective of systemic patterns in higher education, and universities should examine how they are failing to support those students. However, in some instances, it is better for institutions to stop using an algorithmic system rather than attempt to use it to address bias. In Albert III’s opinion, due to the amount of structural inequity in the criminal justice system, there is currently no way that risk assessment tools can be used equitably.

Legislation that requires transparency measures, impact assessments, and audits would help prevent abusive uses of algorithmic tools and mitigate discriminatory harms. Rep. Clarke’s bill requires entities using automated decision systems to conduct impact assessments and mandates additional safeguards for “high-risk information systems” such as those that use data about sensitive characteristics including race, gender, biometrics, and criminal arrests.⁹ The assessments must analyze characteristics that are central to traditional privacy legislation, including: data minimization practices, the retention period for personal information, and the ability of consumers to access and object to or correct the results of the automated decision system. Congress could pass these requirements as standalone legislation, or incorporate them into privacy legislation.

Another legislative tool to address the concerns raised is comprehensive privacy legislation. Comprehensive privacy legislation should require transparency, impact assessments, and regular audits to prevent algorithmic tools from being used in ways that disparately impact disadvantaged communities. Currently, U.S. law generally relies on “notice and consent” to protect consumer privacy, but this framework does not give individuals real choices about how their data are used and is insufficient to protect user privacy.¹⁰ There is a growing consensus among stakeholders to abandon this model in favor of a new approach that places restrictions on how data can be used and gives users enforceable rights over their personal information. In 2018, OTI, as part of a group of 34 civil rights, consumer, and privacy organizations, released public interest principles for privacy legislation, including the principle that “[d]ata practices must protect civil rights, prevent unlawful discrimination, and advance equal opportunity.”¹¹

Since then, members of Congress have introduced bills that include specific requirements for algorithmic tools. U.S. Senate Committee on Commerce, Science, and Transportation Ranking Member Maria Cantwell (D-Wash.) introduced the Consumer Online Privacy Rights Act that would require entities that advertise or make eligibility determinations for housing, education, employment, or credit opportunities to conduct “algorithmic decision-making impact assessments.”¹² The assessments must describe and evaluate the design of an algorithmic tool and the training data used and determine whether the decision-making system produces discriminatory results. The most recent comprehensive privacy bill, Sen. Sherrod Brown (D-Ohio)’s Data Accountability and Transparency Act of 2020, would go a step further and require “automated decision system risk assessments” to be made publicly available.¹³ The growing recognition of privacy as a civil right in Congress is a promising signal that the United States can address the risks posed by the increasing use of algorithmic tools.