 |
| Paul Heaton |
The center created the tool by using Google's pre-trained large language model RoBERTa and further training the model on data from three studies where participants were asked to identify a suspect from a lineup, express how confident they are with their identification in words and give their confidence a number value.
Paul Heaton, law professor at Penn Carey Law and academic director at the Quattrone Center, recently told Law360 Pulse that the center is focused on preventing errors in the criminal justice system and wanted to create a tool that could reduce errors in the process of witness identification.
"One of the things that we know from sources like the National Registry of Exonerations is that eyewitness misidentifications are a big source of errors," he said.
Since technology company OpenAI debuted its generative AI tool ChatGPT in November 2022, many legal tech companies have released tools using the same technology for different legal tasks including contract management, legal research and e-discovery.
The center's tool works by allowing users to input a witness statement, then it analyzes and classifies the statement as either highly confident, somewhat confident or not confident and gives a probability for each classification. Users can also input multiple witness statements at once.
When testing the tool, researchers found that it correctly classifies statements as high, medium or low confidence more than 70% of the time, according to a paper co-authored by Heaton. And the tool's accuracy increases to more than 80% when classifying statements as either highly confident or not highly confident. The researchers achieved similar results when using Meta AI's model OPT, or Open Pre-trained Transformer.
The paper noted that research participants sometimes used the same words to describe different confidence levels, preventing the tool from achieving 100% accuracy. For example, one participant with 80% confidence described their confidence as "pretty confident" and another participant with 60% confidence described their confidence as "pretty confident."
When the statement "I am confident that is the person" is put into the tool, the tool classifies the statement as 92% highly confident, 8% somewhat confident and 0% not confident.
Heaton said that the tool can be used by researchers and law enforcement to assess eyewitness confidence level based on the words that witnesses use to describe how confident they are in their identification.
Heaton noted that when law enforcement officers ask eyewitnesses to rate how confident they are in their identification on a scale of zero to 100, the number they give correlates with the accuracy of their identification.
For instance, eyewitnesses who rate their confidence on the high end have more accurate identification than eyewitnesses who rate their confidence on the low end, according to Heaton.
However, law enforcement officers often don't ask eyewitnesses to pick a number for their confidence level and trying to assign a number to a witness identification after the fact can be tricky, Heaton said.
"The technology could also be used by attorneys or law enforcement to be able to have an objective way to assess an identification," he said.
Heaton added that law enforcement have to consider other factors when investigating a crime besides an eyewitness's confidence like evidence in favor and against a suspect, viewing conditions and whether proper procedures were used during a lineup.
Heaton said the most challenging part of creating this tool was compiling enough quality data to use for training the AI model. Heaton and his research team solved this problem by collecting their own data and using data from other eyewitness identification studies.
"With AI, the output results are only as good as the training data that you input, and so collecting a large enough data set that had a variety of different eyewitness viewing characteristics and different scenarios was important for training the tool," he said.
Heaton added that researchers found a new metric using the tool called confidence entropy, which measures the vagueness of a witness's statement. With more study, researchers could find more new metrics for determining witness confidence, he said.
Or AI models could possibly be trained to predict the accuracy of a witness's identification without looking at confidence statements, he said.
Other ways that AI could be used to improve the criminal justice system is by using facial recognition software to create better suspect lineups and using large language models to review and summarize wrongful conviction applications received by innocence organizations, according to Heaton.
"There's a range of potentially interesting applications, but of course also a lot of these are untested, and we're going to need some research and experimentation to figure out which are the applications that are going to actually be helpful in improving the system," Heaton said.
--Editing by Nicole Bleier.
For a reprint of this article, please contact reprints@law360.com.
