Kirkland Ellis, one of the world's richest law firms, claims to be investing $500m into building their own law AI platform. They're joining a rush of startups, VC, and law firms that are all building affordable legal services using AI. The dream is to replace expensive lawyers with cheap software. Or, if AI can't do the job on its own: to turn $200 lawyers into $1000 lawyers.
Why can't $200 lawyers be $1000 lawyers? Why can't $100 bankers be $1000 bankers? All of these platforms are being built with an understanding that amazing quality work can be produced at a fraction of the cost. Because LLMs are so amazing at coding, math, and meaningless writing, it's widely assumed that this is possible. Having tried for years to make this a reality with my own practice, I have my doubts, and I've written them up here for anyone else who's wondering why this is hard.
If You're So Smart Why Aren't You Rich?
I've programmed many things with LLM APIs, and I've tried to automate, scale, and build new services. My failures aren't because of a lack of skill in law or programming but rather because of the conceptual difficulty of making this work. And it's more or less the same difficulty law firms have in transforming $200 an hour lawyers into $1000 an hour. (Or, in the US, $200 lawyers into $2500 an hour lawyers.) This problem has plagued many organizations. IBM famously spent enormous amounts of money trying to develop a training program to make poor quality programmers into amazing ones, because obviously if it could figure out how to do that it could cheap people around the world and sell them at top dollar to American customers - it proved impossible. The problem is one I face, IBM faces, and Kirkland Ellis faces.
Limits Of LLMs Today
My only real success has been my contract pre-review service, which is now used by most of my clients: Savant1. It's much better than stock LLMs because it's scaffolding is my approach to commercial law, which shapes the LLMs results into something focused and useful. It's a replacement for tedious work, and not a replacement for me as a lawyer. And what it does for clients is gives them a way to ask questions about contracts that they find difficult to come up with. It's genuinely useful, saves me time, and helps clients. But it's not a replacement for legal advice, and it's different than what a lawyer does. LLMs can do more, but not reliably, and this is about the point where things start breaking down. The value of a service that does 50% of the work of a good lawyer, but with lots of it being wrong, and all of it being a bit murky - that's not useful.
Why Expert Legal Work Is Expensive
Expert lawyers, the kind that command high rates, have knowledge that LLMs don't. LLMs actually don't have great insights into a lot of what high-end professionals do because LLMs are mostly trained on Reddit, EDGAR material contracts, and textbooks (I'm exaggerating - but many important types of legal output are simply not found in their training databases.) But even if LLMs had all those materials, it still wouldn't be enough. Junior lawyers have access to this stuff in law firms and it's far from enough to make them into expert lawyers.
Expert legal work is expensive not because of lawyer regulation, law firm brand names, or the sheer volume of modern regulatory law (although these are all factors.) Fundamentally, expert lawyers are expensive because they have expert-level understanding of their clients, comparable clients, and the current best way of achieving their goal. The actual resolution of problems and creation of new opportunities using law requires someone to have a full world model, excellent understanding of people, and intense domain knowledge. Great M&A lawyers know what's come up in many previous deals. A great personal injury lawyer knows the law, juror's sympathies, and the judges. Experts know how to ask just the right number of questions, so their clients can get great outcomes. Where an LLM might say to sue a supplier, the expert knows how to craft a letter that'll create cooperation, rather than a lawsuit that'll destroy the business relationship.
Experts know how to operate within the current constraints, which are people, political, financial, legal, public opinion, etc. They have a firm grasp on the particular games that play out in their domain, in a way that non-experts, and LLMs, don't. It's actually shockingly difficult to make decisions based on imperfect facts, quickly, in a certain human environment. Experts know that world better than their clients do.
Why Can't It Be LLMs?
Kirkland Ellis obviously thinks that they'll be able to provide the materials that make a cheap lawyer into an expensive one, and they'll be able to do that with respect to an LLM, rather than a human training program. But if they can't do it with their junior lawyers, why would it work with LLMs? Kirkland Ellis doesn't have a secret training program that churns out spectacular counsel. They use the same methods as every other firm, which are primarily just experience, rather than training materials. Despite countless efforts for centuries, no one has come up with an educational program that achieves this goal, which is why in all countries, and all levels of firms, it's experience that builds the amazing attorney.
Law firms can improve their training programs. Usually they're bad at that, compared to other large companies. But ultimately they know that it won't work. There's always going to be a relatively small pool of highly skilled people who have the focus to concentrate on some narrow domain and do it day in and day out for decades.
LLMs lack a few key pieces: a world model, local understanding, specialty domain knowledge, and knowledge of the real law. The real law is the reality that is hidden by statute books and regulatory guidelines. In every field there's a secret reality that's known only to people who are in it. For example: most litigation deadlines in Ontario can be ignored, despite what the rules say, except some deadlines, which are incredibly important. Nowhere in the rules in Ontario does it say that you can actually not bother responding, but litigators know what they can actually do.
LLMs have a weak concept of what actually happens if rules are broken. They cheerfully report that all rules must be followed, while in real life that's not the case. Some rule-breaking has dramatic and expensive consequences, whereas other rules don't matter. Some risks never materialize, and some are constant problems for businesspeople. But sine many problems don't materialize as lawsuits or textbook entries, the LLM doesn't understand what the real consequence is. Cross-border enforcemene of contractual obligations is a great example of this, because LLMs struggle to correctly assess the odds of legal action, since they lack a good understanding of the true costs of doing this. The materials just aren't available to train on.
Kirkland Ellis's Data Problem
Obviously Kirkland Ellis actually does have many materials about consequences. They would know, if they could organize it, through emails, memos, and litigation dockets, exactly what happens when certain things don't go right. But they don't have any data about all the things the lawyers did that prevented other problems! The consequences can't be seen because they never happened, because the client hired Kirkland Ellis. The Kirkland Ellis data sources only include things that have happened under the watch of highly paid attorneys who did their jobs. Or, the unusual examples where they were sued for bad work. In any event, it's not at all a sample of what happens in life if you do or don't follow the rules. It's an incredibly biased sample, and it can only be a biased sample because Kirkland Ellis is a very good law firm!
The actual facts that go into settlements and lawsuits are often murky, and are difficult to organize into materials to train LLMs. The people who worked on the file are gone, and if they're not, they're not going to even be able to articulate how exactly the disaster happened. Typically a large corporation caused the problem a while before Kirkland Ellis got involved.
There are many subtle data problems.
The Problem Of Correlation In Law
When I was an articling student, I was known as the math/tech person, since I had a programmer background, went to Waterloo, and had summered at BlackBerry. A person responsible for recruiting came to ask me about the idea of studying their students and lawyers, to see what traits on CVs led to success later as partners. One example was whether or not a particular sport was associated with success, like if students did rugby then they'd be great lawyers, but not if they did hockey. I told them that they only thing they'd learn from this exercise is how they currently do hiring, because without changing the method of hiring they're only measuring their own process. They'd have to hire random students and see if they do just as well as the ones they have. They said they couldn't change how hiring is done. I said they'd never know if it's good or not. It's not a valid test. And a lot of law is like this.
It's very hard to separate correlation from causation in law because the system is acting as it is. It's entirely possible that students from less privileged backgrounds are being overlooked by major firms, or it's possible that privileged people make for better lawyers at big firms (since the customers, GCs, are almost always privileged people themselves, whether from birth or by occupation as GCs of major corporations.) Who knows? Well, probably the firms do. Whether or not anyone consciously knows the answer, it's quite likely that the firm structure is currently quite close to the optimal structure, because there's a natural financial incentive for that to happen. Star lawyers naturally gravitate to large firms, where they find a good platform for their work. Plenty of amazing lawyers don't like that structure, or operate great boutiques, or start their own large firms, but in general, the star lawyers are well-recognized. There's not too many hidden gems in law. There aren't too many opportunities to change what's being done and generate a fortune. Which is of course the appeal of LLMs to Kirkland Ellis and every other law firm.
What Executives Hope Will Happen
Executives hope that LLMs will replace $1000 lawyers with code that uses $10/1m tokens as the pricing, rather than fancy offices and expensive lunch budgets. Or, the less optimistic ones are aiming for a cheap paralegal or junior lawyer to suddenly do the high value work that they're incapable of doing. But what parts do the LLM provide?
Can LLMs really provide the missing pieces that separate the stars from the masses of just ok lawyers? Because if any law firm can pull of this trick, they'll get enormously rich. But there's startups doing the same thing. And there's well-founded legal research companies, like WestLaw, trying too. And there's massive VC funds backing new players. And of course, the LLM model developers themselves, like Google and Anthropic. All of these players are trying to make what law firms hope to make. Do the law firms really have the missing piece?
Google Has ~1500 Legal Professionals
Google runs a law department that's bigger than most law firms in the world. In 2023 their GC reported that they have over 1400 lawyers and professionals. In 2021 they reported having "more than 900 legal team members" in a chapter for a legal informatics handbook. These are huge numbers. And they make use of outside counsel in every country. In short: they have a full view of what's happening, and they see the legal matters from start to finish, since they know the corporate origin and then the outcomes of the outside matters. Everything they send to Kirkland Ellis is a matter they already know about, and in ways that Kirkland Ellis doesn't. So who has the secret sauce?
If Google can't make an LLM that replaces its own lawyers, with access to the world's best AI teams, full data on decades of law, and a gigantic in-house law firm, then what hope do others have? How will they do that and what pieces do they have that Google doesn't? This is rarely addressed in breathless pieces about the automation of law.
My Observations
Law is often not too hard when you have all the facts, but most people can't gather them. Lawyers are great at directing what facts to gather, and this alone makes it hard for clients to skip the lawyer step. Since everyone wants to jump ahead to the advice part, they skip the fact part, and the answers can't be as good.
Law is often highly dependent on the parties involved. Google's strategy with a small plaintiff in Brazil will be very different from dealing with TSMC in Taiwan. Clients don't believe this, and they think the law is neutral, and LLMs deliver accordingly.
Law is often messy. Every law student learns this when they go from the classroom to real life. LLMs are not good at messy things.
Law changes, is debatable, and sometimes random. LLMs aren't great at that, and they generally hew to the understanding that law is not like this. But the real trick of law is that it depends! For example, many rules are ostensibly serious and important but aren't followed, or aren't followed for the sympathetic plaintiff/defendant. More importantly, almost nothing goes to court, so the real life of law is that problems are dealt with outside of court, which means that they happen outside of the legal rules that people think applies. The reality of law is that it's very complicated, and it's really more like hundreds of different sub-professions, for the thousands of situation archetypes that happen in society.
Who's Got The Missing Pieces?
There's much more that can be said about the subtleties of law, but in short: no one has an LLM that does what people want. And the reason they don't have it yet is not that Kirkland Ellis hasn't spent their $500m yet. Large spend will not create good results if the theory is not there. And what's the theory? That Kirkland Ellis knows better than Google or Microsoft? Maybe. But in my experience, large corporates have excellent systems for management that usually have the metadata that law firms don't have, because they operate in a different way than law firms do. What's important within a law firm is not necessarily the same as what's within a large corporate, and this is exactly the space that Kirkland Ellis operates in.
A lot can be done with good prompting (like Savant1!) but right now, there's a lot of pieces missing to get from the current state-of-the-art to a replacement for $1000 lawyers.