Lexical Similarity

Explanation

Lexical Similarity using ROUGE checks how much of a trusted reference answer is captured in the AI system's output, which matters for RAG and summarisation because a summary or response that omits key reference content is incomplete and erodes user trust. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is recall-oriented: it measures the proportion of word sequences (n-grams) from the reference text that also appear in the model's output, rewarding outputs that cover the expected material. To implement it, maintain an evaluation set of inputs with reference outputs (for example, reference summaries of loan agreements or KYC documents), score model responses with a ROUGE scorer in your evaluation pipeline, and record per-item and aggregate scores as monitoring evidence, run both before release and on sampled live traffic. The thresholds are a target of at least 0.90, with investigation triggered below 0.85 and remediation required below 0.80 (proposed). A score in the investigate band prompts a reviewer to check whether outputs are dropping important content; a score in the remediate band requires corrective action such as retraining, prompt fixes, or retrieval tuning. Because ROUGE rewards coverage of reference wording, read it alongside semantic and factuality measures so that valid paraphrases are not wrongly penalised.