A Fair Black Box? The Illusion of Impartiality in Algorithmic Justice

In recent years, AI-driven sentencing algorithms have been hailed as a deus ex machina for achieving impartiality in judicial decision-making. Known as Risk Assessment Indicators (RAIs), these algorithms purport to predict the likelihood of recidivism posed by criminal defendants, and thus promise to strengthen a judicial process long recognised for its reliance on inappropriate heuristics, unconscious prejudice, and systemic discrimination. However, the rising adoption of RAIs across U.S. courtrooms has fallen short of this promise and instead exposed a troubling new frontier: the institutionalisation of racial and political biases under the guise of impartiality. RAIs have the potential to not only generate racially and politically biassed results, but further exacerbate existing racial and political inequalities in the criminal justice system. Thus, far from eliminating human arbitrariness and prejudice, these tools threaten to formalise racial and political bias in criminal sentencing, subsequently driving the erosion of the foundational traditional relationship between law, morality, and the public trust in judicial practice.

A central flaw in the application of RAIs in sentencing decisions is their ability to perpetuate and amplify racial bias in judicial outcomes, which challenges their intended objectivity. While many perceive that artificial intelligence algorithms are objective, secure, and impartial, they are ultimately a product of human design, and thus can reflect human flaws and biases. Many RAI systems are developed based on historic sentencing data, which as Benjamin (2019) notes, are “produced through histories of exclusion and discrimination”.

As it is, black persons, American Indians and Alaska Natives are incarcerated at nearly four times the rate of their white peers, leading to their overrepresentation in arrest and conviction datasets. An algorithm trained on data where minority groups are overrepresented or receive longer sentences than white individuals, will learn to associate these groups with higher recidivism risks. As a result, it may predict a higher likelihood of reoffending for individuals in these groups, even when their individual circumstances do not justify such a conclusion. A study conducted by ProPublica found that COMPAS, one of the leading RAIs in use today, was twice as likely to “misclassify” a black defendant as high risk for recidivism than a white defendant. Thus, RAIs reinforce and legitimise existing disparities, allowing systemic discrimination issues to persist, yet under the troubling guise of machine neutrality.

To overcome this dilemma, one might suggest filtering out the biased data from algorithmic training sets. However, such efforts would only lend themselves to further racial bias, as it is impossible to determine the extent each case has been racialized in the first place. Further, while all RAIs are designed to predict recidivism risk without directly considering race, by simply excluding race from the list of predictors employed by the algorithm, race can still be considered. Other predictors employed by an RAI to generate a prediction often include socio-economic markers like employment status, education level, or zip code, which are inextricably correlated with race. Thus, with enough data, RAIs can learn “proxies” for race, a combination of variables so closely linked to race they essentially act as a substitute, and subsequently concretise discrimination. The opacity of these algorithms makes it impossible for judges to test their accuracy and validity to ensure they are not inappropriately weighing specific predictors that have a disparate impact on minorities, as the inner workings of the algorithms are non-understood—they are virtually “black boxes.” This lack of transparency is critical, as it can make an RAI system inaccurate or discriminatory. As an exaggerated example to illustrate this point: if COMPAS inputs one hundred characteristics, but weights neighborhood crime and socioeconomic status as ten times more important than any other characteristic, combined with the “proxy” effect, the algorithm is likely to inappropriately and inaccurately deem minorities as higher risk. Ultimately, the “black box” operation of RAIs produces obscure justice, which facilitates, rather than mitigates, the biases that undermine the integrity and fairness of the judicial system, eroding public trust.

Alongside racial bias, RAIs risk encoding political bias in ways that are even more difficult to discern and address. Algorithmic bias against an individual’s political orientation can arise in the same ways in which algorithmic racial bias emerges—through machine learning, incomplete or unrepresentative training data, and proxies. However, it differs critically in that, unlike racial bias, there are far fewer social norms or constitutional protections in the U.S. explicitly guarding against political bias in decision-making. Political belief is not a constitutionally protected characteristic, which increases the likelihood that political biases can become entrenched in algorithms, reflecting the political biases of their developers and data trained on. Further, since political biases often align with racial attitudes, this entrenching of political biases in RAIs could in turn intensify the racial biases present in RAIs, creating a compounding chain effect. Only 5% of Trump supporters in 2020 recognised white privilege compared to 59% of Biden supporters, illustrating a stark partisan divide in views on systemic racism. Sentencing algorithms reflecting such partisan biases risk embedding racial inequities into decision-making systems, exacerbating existing discrimination. This means that despite race being excluded as an explicit predictor, political bias may still indirectly harm racial minorities, reinforcing systemic discrimination under the guise of impartiality.

Ultimately, RAIs operate more as a Pandora’s box than a panacea for addressing human bias and subjectivity in sentencing. They risk formalising and exacerbating racial and political biases within the judicial process under the veneer of impartiality, and the potential cascading effects of this formalisation are undeniable. The opacity of these algorithms only compounds this risk, shielding their internal workings from scrutiny and public accountability. Thus, without greater transparency and accountability, these tools risk exacerbating systemic inequities and undermining the very ideals of justice they were designed to uphold. And in attempting to erase the humanity in sentencing, RAIs also eliminate the virtues of discretion, judgement, and agency, which have long been recognised in socio-legal scholarship as qualities vital to confronting the limitations inherent in any legal framework.

Bibliography

Black, J. M. (1997). Rules and Regulators. Oxford University Press.

Benjamin, R. (2019). Race After Technology: Abolitionist Tools for the New Jim Code. Polity.

Callahan, M. (2023, February 23). Algorithms Were Supposed to Reduce Bias in Criminal Justice—Do They? https://www.bu.edu/articles/2023/do-algorithms-reduce-bias-in-criminal-justice/

Claudy, M. C., Aquino, K., & Graso, M. (2022, June 29). Artificial Intelligence Can’t Be Charmed: The Effects of Impartiality on Laypeople’s Algorithmic Preferences. Frontiers in Psychology, 13. https://doi.org/10.3389/fpsyg.2022.898027

Davies, B., & Douglas, T. (2022). Learning to Discriminate: The Perfect Proxy Problem in Artificially Intelligent Criminal Sentencing. In Sentencing and Artificial Intelligence, edited by Jesper Ryberg & Julian V. Roberts, pp. 97-107. Oxford University Press.

Fry, H. (2018). Hello World: Being Human in the Age of Algorithms W. W. Norton & Company.

Iyengar, S., & Westwood, S. (2015). Fear and Loathing across Party Lines: New Evidence on Group Polarisation. American Journal of Political Science, 59, 690-707. https://doi.org/10.1111/ajps.12152

Larson, J. et al. (2016, May 23). How We Analyzed the COMPAS Recidivism Algorithm. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm

Lee, N. T., Resnick, P., & Barton, G. (2019). Algorithmic Bias Detection and Mitigation: Best Practices and Policies to Reduce Consumer Harms. Brookings Institute.

Motoki, F., Neto, V. P., & Rodrigues, V. (2024). More Human than Human: Measuring ChatGPT Political Bias. Public Choice, 198, 3-23. https://doi.org/10.1007/s11127-023-01097-2

O’Brien, T. (2021). Compounding Injustice: The Cascading Effect of Algorithmic Bias in Risk Assessments. Georgetown Journal of Law & Modern Critical Race Perspectives, 13(1), 39-83. https://www.law.georgetown.edu/mcrp-journal/wp-content/uploads/sites/22/2021/05/GT-GC RP210003.pdf

O’Neil, C. (2016). Weapons of Math Destruction. Crown Publishers. Park, A. L. (2019, February 19). Injustice Ex Machina: Predictive Algorithms in Criminal Sentencing. UCLA Law Review. https://www.uclalawreview.org/injustice-ex-machina-predictive-algorithms-in-criminal-sentencing/

Pasquale, F. (2015). The Black Box Society: The Secret Algorithms That Control Money and Information. Harvard University Press. Introduction,

Pew Research Center. (2020, September 10). Voters’ Attitudes About Race and Gender Are Even More Divided Than in 2016. https://www.pewresearch.org/politics/2020/09/10/voters-attitudes-about-race-and-gender-are- even-more-divided-than-in-2016/

Spiggle, T. (2021). Is political discrimination in the workplace legal? Forbes. https://www.forbes.com/sites/tomspiggle/2021/02/24/is-political-discrimination-in-the-workp lace-legal/?sh=6c10808268dd. Starr, S. B. (2014, April). Evidence-based Sentencing and the Scientific Rationalization of Discrimination. Stanford Law Review, 66(4), 803-872. https://www.stanfordlawreview.org/wp-content/uploads/sites/3/2014/04/66_Stan_L_Rev_803 -Starr.pdf

State v. Loomis, 881 N.W.2d at 761.

Sundar, S. S., & Kim, J. (2019, May 2). Machine heuristic: When we trust computers more than humans with our personal information. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 538, 1-9. https://doi.org/10.1145/3290605.3300768.

United States Sentencing Commission. (2023, November 14). Demographic Differences in Federal Sentencing. https://www.ussc.gov/sites/default/files/pdf/research-and-publications/research-publications/2 023/20231114_Demographic-Differences.pdf

Vera Institute of Justice Center on Sentencing & Corrections. (2011, October 12). Risk and Needs Assessments. p. 4. https://www.ma4jr.org/wp-content/uploads/2014/10/vera-institute-memo-on-risk-assessment- for-delaware-2011.pdf

Zheng, Z. (2022, December). Jail inmates in 2021: Statistical tables. Bureau of Justice Statistics. p. 2. https://bjs.ojp.gov/sites/g/files/xyckuh336/files/media/document/ji21st.pdf