LexDye Blog · LexDye AI tools · Published · LexDye, Inc.

Answer key and test agreement: defined-terms benchmark on a 121-page MSA

This page is the answer key for AI only checks what it's asked to check. It lists all 31 planted errors, which 25 were scored and why, and each reviewer's result for every item. The agreement is entirely fictional; any resemblance to real companies is coincidental.

Download the test agreement: lexdye-benchmark-msa-2026-11.docx (Word, 136 KB). This is the exact file every reviewer received.

Totals

Table 1. Scored errors found, out of 25.
ReviewerFound
A leading chat AI, alone15 / 25
AI agent, alone: "Review myAgreement.docx before signing and flag any issues."18 / 25
AI agent, alone: "Check the defined terms in myAgreement.docx."19 / 25
AI agent, alone: detailed prompt listing every check LexDye runs25 / 25
LexDye Definition Tracker alone25 / 25
Chat AI or AI agent with LexDye's report25 / 25

Scope: only defined-term errors are scored. Quotation style and section cross-references are outside the scope of LexDye and of this test, so those planted items are listed but not scored.

Planted errors (31)

Table 2. Every planted error, with each reviewer's result. Columns: Chat AI = leading chat AI alone; Signing = agent alone, "review before signing" prompt; Defined terms = agent alone, "check the defined terms" prompt; Detailed = agent alone, detailed prompt; LexDye = LexDye alone; AI + LexDye = any AI with LexDye's report. Page numbers are approximate.
#LocationTextError typeNoteStatusChat AISigningDefined termsDetailedLexDyeAI + LexDye
E1Clause 1.1, “Earnback” (p. 7)“Earnback” has the meaning given in Clause 6.5.Pointer to wrong clauseEarnback is defined in Clause 6.4(a); Clause 6.5 is Critical Service Level Failures.Not scored: cross-reference (outside definitions-only scope)——————
E2Clause 1.1, Records (p. 12)"Records" means all books, records, documents, data and information, in any form, relating to this Agreement, the Services and the Charges.Straight quotesOnly glossary entry with straight quotes; conform to “Records”.Not scored: quotation style (outside scope)——————
E3Clause 1.1, “Work Product” (p. 17)“Work Product” means all reports, analyses, presentations and other written materials prepared by Supplier Personnel in the course of performing the Services.Defined but never usedNever used; Clause 20 uses Developed Materials. Delete or use.ScoredFoundMissedFoundFoundFoundFound
E4Clause 5.5(c) (p. 22); defined Clause 12.1(d) (p. 31)"...from the Innovation Fund, and shall report the remaining balance of the Innovation Fund in each Monthly Performance Report."Used before its (distant) inline definitionDefined only in Clause 12.1(d); no glossary pointer and no "(as defined in Clause 12.1(d))" at first use.ScoredMissedMissedFoundFoundFoundFound
E5Clause 6.3(b) (p. 24)"...shall not exceed the At-Risk Amount."Schedule-only term used in main bodyAt-Risk Amount is defined only for Schedule B; no glossary pointer; Clause 1.4 confines it to Schedule B.ScoredFoundFoundFoundFoundFoundFound
E6Clause 6.4(a) (p. 24)"Where Supplier has incurred a Service-Level Credit in respect of a Critical Service Level..."Variant of defined termDefined term is “Service Level Credit” (no hyphen).ScoredMissedMissedFoundFoundFoundFound
E7Clause 13.2(c) (p. 33)"Where a payment is made under a Local Service Agreement, ..."Variant of defined termDefined term is “Local Services Agreement”.ScoredMissedFoundFoundFoundFoundFound
E8Clause 17.3(c) (p. 40)"...Supplier shall deliver to Customer a Remediation Plan within ten (10) Business Days..."Capitalized term never definedUndefined; the Agreement uses “Corrective Action Plan”.ScoredMissedFoundFoundFoundFoundFound
E9Clause 19.3(b) (p. 42)"...restore each affected Service within the applicable RTO..."Schedule-only term used in main bodyRTO is defined only in Schedule I; no pointer or attribution (contrast Schedule J ¶4.2).ScoredFoundFoundFoundFoundFoundFound
E10Clause 22.2(f) (p. 47)"...other than in accordance with the change control procedure;"Lower-case use of defined termShould be “Change Control Procedure”.ScoredFoundFoundMissedFoundFoundFound
E11Clause 26.1(d) (p. 52)"any Bank Regulator requires or requests Customer to take action..."Capitalized term never definedUndefined; glossary defines “Regulator”.ScoredMissedFoundFoundFoundFoundFound
E12Clause 27.5(a) (p. 54)"For the purposes of this Clause 27.5, a “Change of Control” occurs where any person ... acquires beneficial ownership of thirty percent (30%) or more of the voting securities of Supplier..."Defined twice, conflictingClause 1.1 uses a more-than-50% / substantially-all-assets test.ScoredMissedFoundMissedFoundFoundFound
E13Annex A-2 ¶4.2 (p. 64)"...Supplier shall meet the service levels applicable to each affected In-Scope Application..."Lower-case use of defined termShould be “Service Levels”.ScoredFoundMissedMissedFoundFoundFound
E14Annex A-3 ¶3.4 (p. 66)"Supplier shall cooperate with the Service Integrator and with each Managed Third Party..."Capitalized term never definedDropped from scoring after the key was drafted. "Service Integrator" is not defined in the final text, so a reviewer who flags it is correct, but the item counts for no one.Not scored: dropped after drafting——————
E15Schedule B ¶2, “Service Level Credit” (p. 69)“Service Level Credit” means ... the Pool Allocation Percentage for the affected Critical Service Level multiplied by the At-Risk Amount, calculated solely by reference to the Monthly Charges for the Service Tower to which the failed Critical Service Level relates.Defined twice, conflictingClause 1.1 bases the credit on total Monthly Charges for all Service Towers.ScoredFoundFoundFoundFoundFoundFound
E16Schedule B ¶2, “Minimum Service Level” (p. 69)“Minimum Service Level” means, in respect of a Service Level, the minimum level of performance specified for that Service Level in Schedule B, failure to achieve which constitutes a Service Level Failure.Identical duplicateWord-for-word repeat of Clause 1.1.ScoredFoundMissedFoundFoundFoundFound
E17Schedule C ¶2, “Business Day” (p. 73)“Business Day” means a day other than a Saturday, a Sunday or a day that is a public holiday in New York, New York or in Bengaluru, India.Defined twice, conflictingClause 1.1 uses New York and London bank closing days; affects Schedule C ¶5 and ¶9 dates.ScoredFoundFoundFoundFoundFoundFound
E18Schedule C ¶2, “Benchmark Adjustment” (p. 73)“Benchmark Adjustment” has the meaning given in Clause 14.1.Pointer to wrong clauseDefined inline in Clause 14.2(a); 14.1 defines Benchmark Review.Not scored: cross-reference (outside definitions-only scope)——————
E19Schedule D ¶1, “Parallel Run” (p. 77)“Parallel Run” means a period during which the incumbent arrangements of the Customer Group and the arrangements of Supplier are operated simultaneously...Defined but never usedNever used.ScoredFoundFoundFoundFoundFoundFound
E20Schedule D ¶5.3 (p. 79)"If any critical deliverable is not accepted by its Milestone Date, ..."Lower-case use of defined termShould be “Critical Deliverable”.ScoredFoundFoundMissedFoundFoundFound
E21Schedule E ¶1, “Tower Review Board” (p. 81)“Tower Review Board” means the board of senior representatives of the Parties convened annually to review the strategic direction of each Service Tower.Defined but never usedNo such body in the governance structure or Escalation Levels.ScoredFoundFoundFoundFoundFoundFound
E22Schedule E ¶4.3 (p. 82)"The Joint Operations Committee shall meet at least monthly, ..."Variant of defined termDefined term is “Joint Operating Committee” / “JOC”.ScoredMissedFoundFoundFoundFoundFound
E23Schedule F ¶3.2 (p. 84)"Supplier shall not remove or reassign any Key Personel during the Minimum Tenure Period..."Misspelled defined termShould be “Key Personnel”.ScoredMissedFoundFoundFoundFoundFound
E24Schedule G ¶1, “Security Incident” (p. 85)“Security Incident” means any actual or reasonably suspected (a) unauthorised access to, or acquisition, use, disclosure, modification, loss or destruction of, Customer Data, or (b) compromise of ... any System used to Process Customer Data.Identical duplicateWord-for-word repeat of Clause 1.1.ScoredFoundMissedFoundFoundFoundFound
E25Schedule G ¶9.2 (p. 87)"...within one (1) hour after becoming aware of any security incident affecting Customer Systems, ..."Lower-case use of defined termShould be “Security Incident”.ScoredFoundMissedMissedFoundFoundFound
E26Schedule H ¶5.1 (p. 89)"...appropriate technical and organisational measures to protect customer data against unauthorised or unlawful Processing..."Lower-case use of defined termShould be “Customer Data”.ScoredFoundFoundMissedFoundFoundFound
E27Schedule I ¶4.2 (p. 92)"Supplier shall maintain a Disaster Recovery Site located not less than two hundred and fifty (250) miles from the primary Data Centre, ..."Capitalized term never definedOnly Disaster, Disaster Recovery Plan and Disaster Recovery Services are defined.ScoredMissedFoundFoundFoundFoundFound
E28Schedule J ¶1, “Termination Assistance Period” (p. 93)“Termination Assistance Period” has the meaning given in Clause 27.2.Pointer to wrong clauseClause 27.2 is Termination Charges; the period is in Clause 28.1 (glossary correctly says so).Not scored: cross-reference (outside definitions-only scope)——————
E29Schedule J ¶3.1 (p. 94)"Supplier shall update the Exit Plann within thirty (30) days after the end of each Contract Year..."Misspelled defined termShould be “Exit Plan”.ScoredMissedFoundFoundFoundFoundFound
E30Schedule K ¶1, Emergency Change (p. 96)"Emergency Change" means a Change that must be implemented urgently to prevent or resolve a Severity Level 1 Incident, a Security Incident or a breach of Applicable Law.Straight quotesAll other definitions use curly quotes.Not scored: quotation style (outside scope)——————
E31Schedule L ¶1, “Representative Sample” (p. 98)“Representative Sample” means a sample of not fewer than three (3) comparable transactions drawn from the Comparison Group.Defined but never usedNever used; benchmarking uses the Comparison Group.ScoredFoundFoundFoundFoundFoundFound

Tricky-but-correct items (12)

A careful reviewer should not flag these. They were not scored; flagging them is a false positive.

Table 3. Items that look like errors but are correct.
#LocationItemWhy it is correct
T1Clause 6.3(c) and throughoutPlurals/possessives: Service Level Credits, Statements of Work, Local Services Agreements, Critical Deliverables, Supplier’s, Customer Group’sInflected forms of defined terms.
T2Clauses 2.1, 2.3, 6.1, 10.2, 10.3, 12.1, 13.1; Schedules B, C, E, KSOW / Statement of Work; ESC; JOC; LSA; KPI; ARC; RRC; Tax / TaxesBoth forms defined together.
T3Clause 6.5(a)Critical Service Level Failure / Critical Service Level / Minimum Service Level / Measurement Periods in one sentenceNested defined terms, each correct.
T4Clause 4.5(a) (and all Part 2C pairs except E1)“Delay Payment” pointer → Clause 4.5, inline definition therePointer and definition agree.
T5Clause 30.1"other natural disasters"Ordinary-language phrase, not the defined “Disaster”.
T6Schedule J ¶4.2"Critical Business Function (as defined in Schedule I) ... Recovery Time Objective (as defined in Schedule I)"Cross-Schedule use with explicit attribution (contrast E9).
T7Clause 8.2(b)"Minimum Tenure Period (as defined in Schedule F)"Schedule term used in main body with attribution, permitted by Clause 1.4 (contrast E5).
T8Clause 6.3(a)Pool Allocation Percentage (also Severity Level, Resource Unit, Baseline Volume, Base Charges, ARC, RRC, KPI, Benchmarker, Standard Contractual Clauses)Schedule-defined but with a correct Clause 1.1 pointer.
T9Clause 1.2(f); “Personal Data” entry“include”, “including”, “without limitation”; “personal data”, “personal information”, “nonpublic personal information”Quoted words, not definitions; statutory quotation.
T10Clause headings; Clauses 22.3 and 29.5ALL-CAPS disclaimer and jury waiver containing SERVICES, DELIVERABLES, DISPUTEConspicuousness requirement; defined terms still correct.
T11Clause 28.1; Clause 20; Clauses 5.7/15.2Termination Assistance Services vs Termination Assistance Period vs Termination Date vs Termination Notice; Supplier Materials vs Supplier Proprietary Software; Customer Facilities vs Supplier FacilitiesDistinct defined terms used in their own senses.
T12Clause 12.1(c); Schedule C ¶11"Benchmark Adjustment (as defined in Clause 14.2)"; "Innovation Fund (as defined in Clause 12.1(d))"Inline-only terms used elsewhere with explicit cross-reference (contrast E4).

Reusing this test

To run this test yourself, give the agreement to an AI, LexDye, or another tool and score the result with this key. Please cite: LexDye, Inc., "Answer key and test agreement: defined-terms benchmark on a 121-page MSA," LexDye Blog, November 19, 2026, https://www.lexdye.com/blog/defined-terms-benchmark-msa-2026-11-answer-key. Results vary by document, prompt, and AI model.