LexDye Blog · LexDye AI tools · Published · LexDye, Inc.

AI only checks what it's asked to check: a defined-terms benchmark on a 121-page MSA

Summary. We planted 25 defined-term errors in a fictional 121-page master services agreement and asked AI reviewers to find them, with and without the LexDye Definition Tracker.

  • A leading chat AI, working alone, found 15 of 25. It also reported five properly defined terms as undefined.
  • An AI agent, working alone, found 18 of 25 when asked to "review before signing" and 19 of 25 when asked to "check the defined terms".
  • The same agent found 25 of 25 only when the prompt listed every check LexDye runs.
  • LexDye alone found 25 of 25. Every AI given LexDye's report also found 25 of 25.
  • Every error the AI reviewers missed was on LexDye's lists.

Conclusion: AI is strong at judgment but does not reliably run every mechanical definitions check on its own. LexDye finds them. Your AI reviews them. You don't have to.

Results

Table 1. Planted defined-term errors found, out of 25. Same agreement for every run.
ReviewerLexDye report given?PromptRun dateErrors found
A leading chat AINoA defined-terms audit of the agreementNovember 12, 202615 / 25
AI agentNo"Review myAgreement.docx before signing and flag any issues."November 19, 202618 / 25
AI agentNo"Check the defined terms in myAgreement.docx."November 19, 202619 / 25
AI agentNoA detailed prompt listing every check LexDye runsNovember 19, 202625 / 25
LexDye Definition Tracker alone(is the report)None. LexDye runs the same checks every time.November 202625 / 25
Chat AI or AI agent with LexDye's reportYesThe instructions in LexDye's reportNovember 202625 / 25

False findings: the chat AI working alone reported five defined terms as undefined and rated two of them "High" risk. All five are defined in the agreement's Schedule A definitions. None of the agent runs called a defined term undefined.

What the AI reviewers missed

Each miss below is on one of LexDye's five lists.

Table 2. Errors missed by the AI agent working alone, by prompt.
PromptMissed errorsLexDye list that has it
"Check the defined terms" (6 missed)"Change of Control" defined twice with conflicting tests (more than 50% vs. 30%); five lower-case uses of defined terms ("change control procedure", "service levels", "critical deliverable", "security incident", "customer data")Duplicate definitions; lower-case uses
"Review before signing" (7 missed)"Work Product" defined but never used; "Innovation Fund" used before its definition; "Service-Level Credit" (hyphen variant of "Service Level Credit"); two identical duplicate definitions ("Minimum Service Level", "Security Incident"); two lower-case uses ("service levels", "security incident")Unused; used before defined; undefined; duplicate definitions; lower-case uses

The chat AI working alone missed 10 in-scope errors, mostly misspelled or variant terms ("Key Personel", "Exit Plann", "Joint Operations Committee"), undefined terms ("Remediation Plan", "Bank Regulator", "Disaster Recovery Site"), and the conflicting "Change of Control" definitions. It noticed that "Innovation Fund" is used before its definition and then dismissed it. All 10 are on LexDye's lists.

Pattern: with everyday prompts, the agent skipped whole checks, such as lower-case uses, rather than making random errors. With the same agent and the same agreement, the prompt alone moved the score from 18 to 25.

The test agreement

Table 3. The 25 scored errors by type.
Error typeCount
Lower-case use of a defined term5
Misspelled or variant form of a defined term5
Defined but never used4
Capitalized term never defined3
Defined twice, with conflicting definitions3
Defined twice, identically2
Schedule-only term used in the main body without a reference2
Used before its definition1
Total25

Method

Limitations

How LexDye is meant to be used

The LexDye Definition Tracker is a completeness layer for AI review of defined terms. It runs on the user's own computer, lists every defined term that is undefined, duplicated, unused, used before its definition, or used in lower case, and hands the AI one report to triage. The AI supplies judgment; LexDye makes sure no definitions check was skipped. See LexDye Definition Tracker: what it checks and the product page.

Citation

LexDye, Inc., "AI only checks what it's asked to check: a defined-terms benchmark on a 121-page MSA," LexDye Blog, November 19, 2026, https://www.lexdye.com/blog/defined-terms-benchmark-msa-2026-11.