Lexical Similarity

Explanation

Lexical Similarity using BLEU checks how closely an AI system's generated text overlaps, word-for-word, with a trusted reference answer, which matters because in RAG and summarisation use cases a drop in similarity to the expected output is an early sign that responses are degrading, drifting, or hallucinating. BLEU (Bilingual Evaluation Understudy) scores the output by counting matching word sequences (n-grams) between the model's text and one or more reference texts, rewarding outputs that reuse the same phrasing as the reference. To implement it, assemble a curated evaluation set of inputs paired with reference outputs (for example, gold-standard answers to common customer queries or reference summaries of policy documents), run the model's responses through a BLEU scorer in your evaluation harness, and log per-item and aggregate scores as evidence; this runs both pre-deployment and as ongoing monitoring on sampled production traffic. The threshold is set as a target per control objective with an alert on breach (proposed), so you should set a risk-based threshold appropriate to the use case; when scores fall below it, the control raises an alert prompting investigation of the model or its retrieval inputs. Note BLEU rewards surface wording, so pair it with semantic and factuality checks since a correct answer phrased differently can still score low.