AI only checks what it's asked to check: a defined-terms benchmark on a 121-page MSA
Summary. We planted 25 defined-term errors in a fictional 121-page master services agreement and asked AI reviewers to find them, with and without the LexDye Definition Tracker.
- A leading chat AI, working alone, found 15 of 25. It also reported five properly defined terms as undefined.
- An AI agent, working alone, found 18 of 25 when asked to "review before signing" and 19 of 25 when asked to "check the defined terms".
- The same agent found 25 of 25 only when the prompt listed every check LexDye runs.
- LexDye alone found 25 of 25. Every AI given LexDye's report also found 25 of 25.
- Every error the AI reviewers missed was on LexDye's lists.
Conclusion: AI is strong at judgment but does not reliably run every mechanical definitions check on its own. LexDye finds them. Your AI reviews them. You don't have to.
Results
| Reviewer | LexDye report given? | Prompt | Run date | Errors found |
|---|---|---|---|---|
| A leading chat AI | No | A defined-terms audit of the agreement | November 12, 2026 | 15 / 25 |
| AI agent | No | "Review myAgreement.docx before signing and flag any issues." | November 19, 2026 | 18 / 25 |
| AI agent | No | "Check the defined terms in myAgreement.docx." | November 19, 2026 | 19 / 25 |
| AI agent | No | A detailed prompt listing every check LexDye runs | November 19, 2026 | 25 / 25 |
| LexDye Definition Tracker alone | (is the report) | None. LexDye runs the same checks every time. | November 2026 | 25 / 25 |
| Chat AI or AI agent with LexDye's report | Yes | The instructions in LexDye's report | November 2026 | 25 / 25 |
False findings: the chat AI working alone reported five defined terms as undefined and rated two of them "High" risk. All five are defined in the agreement's Schedule A definitions. None of the agent runs called a defined term undefined.
What the AI reviewers missed
Each miss below is on one of LexDye's five lists.
| Prompt | Missed errors | LexDye list that has it |
|---|---|---|
| "Check the defined terms" (6 missed) | "Change of Control" defined twice with conflicting tests (more than 50% vs. 30%); five lower-case uses of defined terms ("change control procedure", "service levels", "critical deliverable", "security incident", "customer data") | Duplicate definitions; lower-case uses |
| "Review before signing" (7 missed) | "Work Product" defined but never used; "Innovation Fund" used before its definition; "Service-Level Credit" (hyphen variant of "Service Level Credit"); two identical duplicate definitions ("Minimum Service Level", "Security Incident"); two lower-case uses ("service levels", "security incident") | Unused; used before defined; undefined; duplicate definitions; lower-case uses |
The chat AI working alone missed 10 in-scope errors, mostly misspelled or variant terms ("Key Personel", "Exit Plann", "Joint Operations Committee"), undefined terms ("Remediation Plan", "Bank Regulator", "Disaster Recovery Site"), and the conflicting "Change of Control" definitions. It noticed that "Innovation Fund" is used before its definition and then dismissed it. All 10 are on LexDye's lists.
Pattern: with everyday prompts, the agent skipped whole checks, such as lower-case uses, rather than making random errors. With the same agent and the same agreement, the prompt alone moved the score from 18 to 25.
The test agreement
- Fictional master services agreement for IT and business process outsourcing between a bank and a technology supplier. About 121 pages, a Clause 1.1 glossary, 12 schedules with their own definitions, and 372 definitions in all (LexDye count).
- 31 errors were planted. Five are outside the definitions-only scope and were not scored: two straight quotation marks around a definition (quotation style) and three glossary entries that point to the wrong clause (cross-references). One more item (E14) was dropped from scoring after drafting. That leaves 25 scored errors.
- The agreement and the full answer key, with every reviewer's result item by item, are published: answer key and test agreement.
- The agreement also contains 12 tricky-but-correct items (plurals, abbreviations defined together with their long forms, terms used with an express "as defined in" reference) that a reviewer should not flag.
| Error type | Count |
|---|---|
| Lower-case use of a defined term | 5 |
| Misspelled or variant form of a defined term | 5 |
| Defined but never used | 4 |
| Capitalized term never defined | 3 |
| Defined twice, with conflicting definitions | 3 |
| Defined twice, identically | 2 |
| Schedule-only term used in the main body without a reference | 2 |
| Used before its definition | 1 |
| Total | 25 |
Method
- Scope. Only defined-term errors were scored. Commercial points, meaning or drafting-quality points, quotation style, and section cross-references were not scored, even when a reviewer raised them correctly.
- Chat AI. A leading general-purpose chat AI was given the agreement and asked for a defined-terms audit. It worked as a chat assistant, without running its own scripts. LexDye does not name the vendor or model.
- AI agent. An AI agent that works on the user's own computer and can read files and run its own scripts. Each run started in a fresh session, in a folder that held only the agreement (myAgreement.docx). Only the prompt changed between runs. A full review took about five minutes.
- LexDye alone. The LexDye Definition Tracker (browser version, beta, November 2026) opened the Word file directly. The result is the same on every run.
- AI with LexDye. The AI received the agreement and LexDye's report, which includes line-numbered text and instructions to triage every entry. The agent run used the LexDye command-line version (beta build 2026-11-17a); it triaged all 454 report entries.
- Scoring. Each reviewer's output was checked item by item against a written answer key prepared when the errors were planted. The rescoring to the definitions-only scope (25 errors) was done on November 19, 2026.
Limitations
- One fictional agreement and a small number of runs. Results vary by document, prompt, and AI model.
- Planted errors are clean slips. Real negotiated drafts are messier.
- The AI reviewers also found real issues outside the scope of this test, such as conflicting commercial terms. That is the judgment work LexDye leaves to the AI and the lawyer.
- LexDye errs on the side of flagging for review, so its report also contains entries that are fine on inspection. The AI's job is to sort them into fix, consider, or ignore.
How LexDye is meant to be used
The LexDye Definition Tracker is a completeness layer for AI review of defined terms. It runs on the user's own computer, lists every defined term that is undefined, duplicated, unused, used before its definition, or used in lower case, and hands the AI one report to triage. The AI supplies judgment; LexDye makes sure no definitions check was skipped. See LexDye Definition Tracker: what it checks and the product page.
Citation
LexDye, Inc., "AI only checks what it's asked to check: a defined-terms benchmark on a 121-page MSA," LexDye Blog, November 19, 2026, https://www.lexdye.com/blog/defined-terms-benchmark-msa-2026-11.