How Algorithmic Bias Undermines Disparate Impact Cases

Artificial intelligence—machines that can perform more complex actions than explicitly instructed to—can be incredibly powerful. When wielded correctly, these algorithms can diagnose early-stage cancer with remarkable accuracy or predict the risk of a natural disaster in vulnerable ecosystems. At worst, however, this technology can magnify algorithmic bias, or unfair prejudice in a computer’s decision-making that derives from systematic errors or imperfect data, thus intensifying discrimination or inequity.¹ Unintentional discrimination caused by AI can be quite harmful, yet much more difficult to settle legal disputes over.

To better understand how algorithmic bias appears in daily life, we can look to AI algorithms that are used to screen resumes for job applicants. While these algorithms were initially introduced to minimize human biases in hiring processes, they instead amplified those very prejudices. One algorithm tested (and ultimately scrapped) by Amazon sifted through former resumes of successful job applicants to train its AI how to identify the best candidates in a pile of new resumes. Yet, somewhat unsurprisingly, Amazon’s AI strongly favored male candidates because Silicon Valley has been mostly male-dominated in the past, even sometimes disadvantaging candidates who attended women’s colleges.² In a more troubling setting, the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) algorithm analyzes criminal defendants on two dozen factors, including “criminal personality” and “substance abuse,” to rank their likelihood to reoffend, ultimately using that data in those defendants’ sentencing decisions.³ A ProPublica study later revealed that COMPAS significantly overestimates Black recidivism rates while underestimating white recidivism rates, consigning defendants of color to overly punitive sentences.⁴

From a legal perspective, algorithmic bias is yet another manifestation of discrimination that pervades society’s most critical junctures, from the workplace to the courtroom. When it comes to discrimination, the letter of the law is clear — the Civil Rights Act of 1964 outlawed discrimination “on account of race, color, religion, sex, or national origin” with the full constitutional backing of the 14th Amendment’s Equal Protection Clause, guaranteeing that no state shall “deny to any person within its jurisdiction the equal protection of the laws.”⁵ And yet, despite this seemingly unequivocal stand for equality, unintentional discrimination is not immediately in violation of the Constitution.

Indeed, possible litigation over algorithmic bias appears to be a matter of “disparate impact” for cases with an adverse effect on a group of people with a shared identity. The judicial response to disparate impact cases is articulated in Washington v. Davis (1976), where the Supreme Court found that a law’s disparate impact alone does not violate the Equal Protection Clause, so long as there is no discriminatory intent. At the time, the Court dealt with a D.C. police department’s verbal test as a hiring procedure, which was failed more often by Black applicants than white applicants.⁶ As the Court ruled, however, such an aptitude exam could reasonably measure their readiness for the job and was not intentionally discriminatory. In the context of disparate impact, this finding makes sense given that other race-neutral classifications exist and are constitutional, even if they might be unfair otherwise, such as standardized testing including the SAT and ACT.

Without evidence of design criteria that are deliberately discriminatory, algorithms that result in disproportionate outcomes still appear to pass the Washington standard. This, however, does not sit comfortably with the examples of blatant inequality demonstrated in Amazon’s AI and COMPAS — institutional negligence of the consequences of these algorithms should not be set free under equal protection law.

To see how the legality of disparate impact can be a slippery slope, look to McCleskey v. Kemp (1987), which upheld the use of the death penalty despite comprehensive studies of racial bias in the criminal justice system. Citing Washington, the Court found that despite empirical statistics of systemic racism in death sentences, this evidence does not amount to discriminatory intent against Warren McCleskey himself.⁷ Through McCleskey, the Court enabled the criminal justice system to preserve antiquated punishments that turn a blind eye to the racially disparate impacts that they impose. When Black defendants are four times more likely to receive the death penalty than white defendants for similar crimes, the ignorance of this structural racism signifies a real failure of the disparate impact doctrine.⁸

This gray area between Washington and McCleskey is where algorithmic bias largely operates. It is generally true that AI designers do not intend to discriminate through their algorithms, which are often created for exactly the opposite purpose. Yet, in spite of these efforts, algorithms may still end up wildly biased because it is challenging to perfectly parametrize what the AI can use to make decisions and, perhaps more importantly, what they cannot. Personal characteristics such as place of residence and level of education broadly act as a proxy for race, further narrowing what algorithms could factor in. Moreover, just as legal courts may struggle to decode complex AI programs, Silicon Valley may be too myopic to internalize the discriminatory repercussions of their algorithms. How, then, should equal protection law adjudicate algorithmic bias cases?

Within the scope of legal solutions, a few possibilities arise. Courts could (somewhat ironically) utilize a more quantitative test to discern which impacts are too disparate. Such standards have already been proposed by federal agencies. In 1978, the Equal Employment Opportunity Commission (EEOC) proposed the so-called “80 percent rule”: as a rule of thumb, a company’s hiring procedure must hire minority groups at a rate that is at least 80 percent of the rate of non-protected groups.⁹ If the courts adopted a threshold akin to the 80 percent rule, it would certainly flag some instances of algorithmic bias by measuring the degree of discrepancy in outcomes. However, while this line could alleviate some of the fuzziness in equal protection standards, the 80 percent rule is too reminiscent of racial quotas that were declared unconstitutional in Regents v. Bakke (1978) and reaffirmed in later affirmative action cases like Gratz v. Bollinger (2003).¹⁰ By setting such an arbitrary and rigid benchmark, the 80 percent rule is an oversimplification of disparate impact law.

Rather, an AI algorithm should have to prove both a significant improvement in accuracy as well as a significant reduction in bias when compared to human classifications. In cases of future legal disputes over algorithmic bias, courts should hold AI to a higher level of judicial scrutiny and require that algorithms applied to human problems are “narrowly tailored.” Under this stricter framework, the use of AI with significant disparate impacts would be unconstitutional unless such an algorithm is better than all other plausible methods. Most importantly, courts cannot allow companies and governments to resort to algorithms simply because they are efficient, minimizing job applicants or individual defendants to data points out of an unwillingness to view them as full human beings.

References

¹ Baer, Tobias. Understand, Manage, and Prevent Algorithmic Bias: A Guide for Business Users and Data Scientists. Apress, 2019.

² Goodman, Rachel. “Why Amazon's Automated Hiring Tool Discriminated Against Women.” American Civil Liberties Union. American Civil Liberties Union, October 15, 2018. https://www.aclu.org/blog/womens-rights/womens-rights-workplace/why-amazons-automated-hiring-tool-discriminated-against.

³ DocumentCloud. Accessed April 7, 2022. https://www.documentcloud.org/documents/2840784-Practitioner-s-Guide-to-COMPAS-Core.

⁴ Julia Angwin, Jeff Larson. “Machine Bias.” ProPublica, May 23, 2016. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.

⁵ “Title VII of the Civil Rights Act of 1964.” U.S. Equal Employment Opportunity Commission. Accessed April 7, 2022. https://www.eeoc.gov/statutes/title-vii-civil-rights-act-1964.

⁶ Washington v. Davis, 426 U.S. 229 (1976)

⁷ McCleskey v. Kemp, 481 U.S. 279 (1987)

⁸ “Race and the Death Penalty - Prison Policy Initiative.” Accessed April 7, 2022. https://www.prisonpolicy.org/scans/aclu_dp_factsheet4.pdf.

⁹ “Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines on Employee Selection Procedures.” Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines on Employee Selection Procedures | U.S. Equal Employment Opportunity Commission. Accessed April 7, 2022. https://www.eeoc.gov/laws/guidance/questions-and-answers-clarify-and-provide-common-interpretation-uniform-guidelines.

¹⁰ Regents of Univ. of California v. Bakke, 438 U.S. 265 (1978); Gratz v. Bollinger, 539 U.S. 244 (2003).