Multi-agent contract review
Ask an AI whether a contract protects you, and “I didn't find it” looks exactly like “it isn't there.” One of those is fine. The other can cost you a lawsuit. This project makes a second AI try to prove the first one wrong — and measures how often that saves you.
Part one
Here is a question a lawyer asks about every contract that crosses their desk.
“If this deal goes wrong, is there a limit on how much we could be sued for?”
The AI answers: “No limit on damages found.”
It read the contract. It was 80% confident. It sounds authoritative.
liability_worker · status=MISSING · confidence=0.80 · 34 blocks examined
FOUND — PAGE 4, SECTION 18, HEADED “MISCELLANEOUS”
“Save in respect of fraud or death or personal injury caused by negligence, the total aggregate liability of each Party arising under or in connection with this Agreement shall not exceed the charges paid under the relevant Statement of Work in the twelve (12) months preceding the event giving rise to the claim.”
The limit was there all along. It sits four pages away from anything labelled “liability”, under a heading that means nothing, and its wording never uses the words limit, cap, or damages. The AI did not lie — it just did not look hard enough. And from its answer alone, you could not tell the difference.
That is the failure this project exists to catch.
Part two
No step requires you to trust the AI's word for anything.
Ingestion · no AI involved
The contract is split into paragraphs, and every paragraph keeps its page number and section number attached. That way, any answer later on can point back to the exact spot it came from. This step is ordinary code, not AI, so it behaves the same way every single time.
Worker agents · Claude
The firm writes a checklist — “must limit liability”, “must say which country's law applies”, and so on. A separate AI handles each item. Crucially, to claim an item is covered, it has to quote the contract word for word. If the quote doesn't actually appear in the document, the claim is thrown away.
Adversarial verification · Gemini
Whenever the first AI says “this is missing”, a second AI — built by a different company — searches the whole contract again, using different words. Its job is not to agree. Its job is to find the clause. If it finds one real example, the first AI's answer is overturned.
Part three
Press play. Each step is one thing the system actually did.
Press play to walk through this case one step at a time.
Part four
Six contracts were written with the answers known in advance, then run through the system — 48 checks in total. A “false alarm” means the system told you something was missing when it was really there.
The second AI overturned 21 findings. Every one of those 21 was correct — it never once wrongly cleared a problem that was genuinely there.
| Configuration | Precision | Recall | F1 | False pos. | False neg. |
|---|---|---|---|---|---|
| Workers only | 0.222 | 0.667 | 0.333 | 21 | 3 |
| + verification | 0.462 | 0.667 | 0.546 | 7 | 3 |
Positive class is an absence claim. Run with deterministic stand-in agents rather than live models, so these figures characterise the pipeline's recovery behaviour — not model accuracy. The system has never been benchmarked against Claude and Gemini, and no such claim is made.
Part five
A review tool that only shows you its wins is not a review tool.
If the first AI says “yes, this is covered”, nobody challenges it. So a contract with a section headed “Limitation of Liability” whose text actually says “nothing shall limit either party's liability” sails straight through. The heading reassures; the body does the opposite.
All three remaining mistakes in the test set are exactly this shape. The fix — challenging “found it” answers the same way — is the next thing to build. Until then the number stays published rather than hidden.
Part six
No API key needed for the first two commands — they use built-in sample contracts and cost nothing.