Lack of Transparency and Perceived Objectivity Perpetuate Biases in Algorithmic Tools

As Gillmor noted, machine learning and other AI systems are primarily used as cost-shifting measures. He argues, they “are designed to make it possible to do things at a scale that your traditional mechanisms would not be able to do.” Companies that adopt AI technologies report having an increase in revenue, as well as reduced costs.¹ These technologies can also benefit society by predicting natural disasters or improving medical diagnostics, for example.² Algorithmic tools will be more appropriate for uses that do not rely on sensitive personal information and do not make consequential decisions about individuals. However, there is often a lack of transparency and understanding of how these models work, making it difficult to judge their accuracy and social fairness. As more companies rely on and invest in big data and AI systems, it is important to ensure that these systems do not result in disproportionately negative outcomes, particularly for historically disadvantaged groups.

If AI is not designed and monitored properly, the technology can have discriminatory consequences. For example, while these systems can help improve medical diagnostics, a study on a widely used health care algorithm showed that they can also systematically discriminate against Black people.³ The study found that the system failed to identify Black patients at risk for medical needs at the same rate as white patients, which resulted in Black patients being less likely to receive preventative care to improve their health. These results are particularly alarming during the COVID-19 pandemic, because Black and Brown communities are already disproportionately affected by the pandemic⁴ and AI systems may be used to help determine how to prioritize medical care if resources are scarce.⁵

As Albert III noted, decision-makers feel comfortable relying on algorithmic systems because there is often a perception that automated systems are less biased than human decision-makers because they rely solely on data and statistics. When companies and organizations actually test AI systems, however, they often find unintended, biased results.

There are a variety of reasons why algorithms can lead to biased results. One of the most common reasons is the underlying training data, which reflect biases and discriminiation that exist in the physical world. Discrimination continues to persist in the American education and criminal justice systems, as well as among employers in all sectors of the economy. Because many institutions were built on a foundation of racism and inequality, the data collected by those institutions will reflect those biases because predictive algorithms are designed based on correlations found in data of past experiences. So, as noted above, if historical data show that in the past only white men have held certain jobs, or that minorities who lack access to educational resources will not perform as well academically, these historical discriminatory patterns will be perpetuated by the algorithms. Further, discrimination can occur even when the underlying dataset does not explicitly contain sensitive categories such as race or gender, but only includes highly correlated variables that can serve as proxies for these characteristics.

For example, in 2018 Amazon built out and tested an AI recruiting tool to help find and review job applicants.⁶ Over time, Amazon found that the tool had a preference for male candidates over female candidates for technical positions. The tool was not built to consider gender for potential applicants, but it was built to identify candidates based on patterns in previous resumes submitted to Amazon—historically, applications for technical positions had predominantly come from men. Amazon abandoned the tool before it was used to actually recruit potential candidates, but the unintended outcome shows the importance of testing and auditing AI systems for discriminatory impacts before and after deployment.

It can be particularly difficult to hold companies accountable for discrimination caused by their AI tools, because those affected often lack knowledge of how the tools work and the ability to obtain that information. The final rule released by the Department of Housing and Urban Development (HUD) in September 2020 will exacerbate this problem for those trying to hold housing providers responsible for discrimination, because it places so much evidentiary burden on the plaintiff. HUD’s disparate impact rule, both in its proposed form (in 2019) and its final form (2020), show that HUD fails to recognize AI tools’ likelihood of leading to discriminatory housing decisions. OTI filed comments as part of a coalition of 23 civil rights and consumer advocacy organizations and individual experts,⁷ and separately,⁸ to oppose the HUD proposal, explaining that it reflected a complete lack of understanding of how algorithmic models work. Although HUD removed the controversial algorithmic defense, the final rule essentially has the same effect by significantly heightening the burden on plaintiffs and creating a new defense that permits disparate impact caused by an algorithm if the defendant establishes the algorithm provided some benefit to the protected class. In cases involving AI tools, plaintiffs must be more reliant on a showing of disparate impact, but HUD’s rule will make it extraordinarily difficult for their claims to prevail. The new requirement that plaintiffs establish a “robust causal link between the policy or practice and the adverse effect on members of a protected class” make it particularly difficult for plaintiffs to bring a claim when the discriminatory impact is caused by AI, because they will likely not have access to enough information about internal, technical practices to satisfy this requirement. That information is always difficult for plaintiffs to obtain, as companies will claim “trade secret” protections, making algorithms extraordinarily difficult to challenge in court generally—but could be an impassable hurdle at the pleading stage (as the new rule requires).⁹

Similarly, a ProPublica study showed that the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS), a criminal risk assessment tool, is twice as likely to classify Black defendants who do not recidivate as high risk compared to similar white defendants.¹⁰ Based on these findings, ProPublica found that COMPAS is unreliable and racially biased against Black defendants. Northpointe, the developer of COMPAS, asserted that the system is not racially biased because it predicts overall recidivism equally well for Black and white defendants. The discrepancy appears to stem from the fact that there have historically been greater levels of policing in Black communities, and consequently disproportionately higher arrest rates.¹¹ However, it is difficult to confirm these findings because the COMPAS algorithm is proprietary data that Northpointe has not released.¹² In State v. Loomis, the defendant argued that the use of COMPAS violated his due process rights because the tool considers gender in its analysis, although he could not confirm the specific method in which it did so because the algorithm was proprietary.¹³ The Wisconsin Supreme Court upheld the use of COMPAS, but required that COMPAS reports be accompanied by disclaimers of accuracy. Without accurate knowledge of how the COMPAS’s algorithm works, however, it will be difficult for any future defendants who suspect they were unfairly discriminated against to oppose their risk assessment score. Because of the evidence of discrimination in risk assessment tools, OTI and over 100 organizations have called on jurisdictions across the United States to abandon their use in pretrial decisionmaking.¹⁴

Further, AI systems are often not equipped to quickly adapt to unprecedented times. As Palmer noted, the pandemic’s effects on higher education have caused student data to differ from expectations before the pandemic, which can decrease the accuracy of a tool’s predictions dramatically. For example, universities often use algorithms to identify students who may be at risk of dropping out or failing. One of the metrics these tools consider is how often a student is engaging with an online learning management system. However, during the pandemic, many classes and other university services have gone remote, so all students use online-learning management systems to a higher degree. Students who were previously flagged as being at-risk are not anymore because their online engagement went up, although, according to Palmer, “all students are more at risk in this situation.” Because of the pandemic, these systems will no longer be as effective at identifying students who need more help, and universities will need to consider additional methods for identifying and helping students. There must be proper assessment systems in place to ensure that unprecedented shifts in data do not render algorithm decision-making ineffective.

AI can be extremely valuable to society and help organizations accomplish tasks that would otherwise not be possible. However, without proper testing and assessment tools in place, it can also lead to unintentional discriminatory outcomes. Before AI and algorithmic systems are used for autonomous decision-making, there should be more processes in place for auditing and assessing these tools to ensure their outcomes are not biased.