Legal tech company Onit's AI Center of Excellence, based in New Zealand, recently completed a research report called "Better Call GPT, Comparing Large Language Models Against Lawyers." The report compared the time spent, cost and accuracy of lawyers reviewing contracts against LLMs, which are learning algorithms used by AI tools.
The LLMs completed contract reviews in mere seconds, surpassing the minutes that it took humans to do the same tasks. In terms of spending, the study found that the LLMs operated at a 99% reduction in legal costs to complete the reviews.
Researchers conducted this experiment to pit machines against human attorneys between August 2023 and December 2023, which is when the report was written. The report will be submitted this week to arxiv.org, which Whitehouse said is Cornell University's publishing mechanism.
Ten senior lawyers from two Asia-Pacific-based law firms participated in the research, as well as several junior attorneys from those law firms and attorneys from third-party legal process outsourcers, or LPOs. Researchers declined to reveal the names of the firms.
Lawyers were given 10 procurement contracts sourced from real-world legal agreements and anonymized to preserve confidentiality.
Senior lawyers took an average of 43.46 minutes per document to complete the review process. Junior lawyers took 56.17 minutes to complete the same review and LPOs averaged 201 minutes.
By contrast, most of the LLMs spent one to five minutes per document.
There was also a stark difference when it came to cost.
The average cost per document for senior lawyers was $75.92. Junior lawyers averaged $74.26 and LPOs cost $36.85 per document.
LLMs operated at a fraction of the cost, with most averaging $0.25 or less per document.
The cost per contract for lawyers was calculated based on the average time spent on each document and hourly rates from industry benchmark reports. The cost for LLMs was calculated based on the tokens used, which are the units of text that must be purchased for the LLMs to work.
However, the accuracy of the LLMs against humans is more complicated.
Using senior lawyers as a benchmark, researchers compared the ability of humans and LLMs to accurately review documents. In terms of determining legal issues in the documents, one LLM nearly tied the LPO reviewers, with junior lawyers coming in third behind them. When it came to locating more legal issues, LPOs had a clear lead, but multiple LLMs were more accurate than junior lawyers.
Whitehouse said the consequences of this accuracy shortfall will be minimal as long as legal professionals use LLMs to augment human review instead of entirely depending on them.
However, Whitehouse also said that legal teams working with LPOs may want to consider working with a legal tech provider that uses LLMs.
The report concluded that LLMs will force LPOs to change their business models from providing manual legal process services to perhaps managing LLM platforms. Junior attorneys may also face some disruption while routine tasks are gradually transitioned to LLMs. Meanwhile, legal departments may realize the efficiency gains and cost savings associated with this new technology.
--Graphics by Ben Jay. Editing by Rich Mills.
Correction: A previous version of this article incorrectly identified the publishing website for the study and Nick Whitehouse's job title. These errors have been corrected.
For a reprint of this article, please contact reprints@law360.com.
