By Anonymous.
Lawyers make mistakes. Thatâs why they take out professional liability insurance. This raises the question in the wake of the Stanford University study into the accuracy of Thomson Reutersâ and LexisNexisâ generative AI tools: what are we comparing these systems against when it comes to hallucinations and inaccuracy?
Or put it another way: yes, generative AI systems create errors, but are lawyers without such technology more accurate?
The regulation around this topic makes it clear that legal perfection is not expected. For example, in England & Wales the Solicitors Regulation Authority (SRA) requires firms to hold professional indemnity insurance. And itâs this ongoing need for insurance coverage that is a key reason why the SRA will step in when a law firm collapses. In short, regulators expect lawyers to make errors.
So when weâre comparing the accuracy rates of generative AI or another overhyped and under-explained technology with what lawyers do, then weâre comparing two inherently flawed processes. Humans are imperfect and so is generative AI. The question then is: how flawed are they and what level is acceptable?
What is not properly quantified at present is how many mistakes âaâ lawyer makes, or how many mistakes a legal âteamâ providing advice make. Clearly itâs not zero. So how good are human lawyers?
Can we get some hints from looking at how law firms are charged for insurance?
Going through the renewal process for professional indemnity cover (in England & Wales) youâre evaluated on your âgross fee incomeâ and this is then adjusted by the kind of work done and, obviously, your claims history. This rounds out at about 1.5% of turnover for larger firms in England & Wales. Even assuming actuarially fair insurance, this does not imply a 98.5% accuracy rate for legal work overall.
A lot of the mistakes of the sort that are being spotted in the Stanford study are unlikely to contribute to this figure as the structure of the work mitigates them: supervisors, managing associates and partners will catch their juniorsâ errors; opposing counsel will query language that doesnât make sense to them; and in the most embarrassing case, clients will spot things that seem off.
Also, many mistakes are never spotted. Or, if they are, they get smoothed over. Looking through old work bibles usually leads to comments like âIâm not sure I would have drafted it that wayâ and then a resolution not to use those documents in the future.
I.e. Insurance failure rates and Professional Indemnity Insurance claims may dramatically understate the frequency of mistakes.
This means that although regulators expect lawyers to make significant errors, many of those errors are 1) picked up internally, 2) highlighted by opposing counsel, 3) picked up by the client, or 4) manage to slip through the net and no-one notices….until someone eventually does. Â
This in turn raises the question of what is âgood enoughâ for a client, given that they must be accepting work product that is often imperfect?
Itâs clear that generative AI tools are not âgood enoughâ on their own. Itâs also not clear what âgood enoughâ actually looks like. We cannot write off these tools until we know what âgood enoughâ is for the tools, and for humans.
—
[ Note: Anonymous works in the field of legal technology, but asked not to be named due to their role. Although it can be said that they are not at any of the organisations mentioned in the Stanford study. ]
If you would like to respond to the points raised here in a following think piece, please let Artificial Lawyer know.
From this siteâs perspective this piece raises plenty of questions, such as:
- Errors will creep into legal work, but how much supervisory effort to correct those mistakes is acceptable? As mentioned in last weekâs AL piece on this subject, if juniors using generativeAI (or with no technology) are making a very high level of mistakes then a law firm would grind to a halt, as senior lawyers could not be expected to double-check everything, continually, especially the base layer of key facts, for every piece of work. I.e. there would be no point in having those junior lawyers in the first place, if that were the situation. So, what is an acceptable level of supervision and at what level does it break the law firm model?
- Lawyers are assumed to have âpassed the testâ once they qualify, and certainly once they progress through level after level of associate life in a firm. But, beyond that there are no accuracy records. Everything is taken on trust. And maybe thatâs a sensible system, as you canât expect lawyers to sit the Bar exam, or something similar, every year to prove their professional accuracy. Moreover, one could say that firms sort this out themselves, letting go junior lawyers who are not good with details. Although, we have no scientific evidence this is so. There is no public data on this. Instead we assume that âgood lawyersâ tend not to make mistakes and those who are senior make very few â and we live with that.
- The challenge with genAI, as with a lot of legal tech, is that is makes us question and think again about how human lawyers do what they do, and what âgoodâ looks like. If we are to judge AI, as the author notes, then it seems fair to judge human accuracy too.
- One other point is how these things blend together. A lawyer may make a mistake on their own, then add a mistake from a genAI system, which is then missed by someone more senior, and so this is fed into the final work product. Likewise, they may be on the ball, spot the genAI error on their own, and everything is fine. Or, in a more positive light, the genAI system correctly finds an answer that the junior lawyer would not have found on their own, this information gets ferried upwards and into the final work product and everyone is happy, including the client. All three scenarios are possible. The question is how frequent is each one?
- To conclude, accuracy and AI will remain a central issue. Why? Because if we want genAI to become all it can be in the legal field then we have to figure out the above questions and then we will be on solid ground for truly changing how the legal world works.
Discover more from Artificial Lawyer
Subscribe to get the latest posts sent to your email.