Semantic Similarity

Explanation

Semantic Similarity using Cosine Similarity checks whether an AI system's output means the same thing as a trusted reference answer, even when the wording differs, which matters in RAG because a response can be correct yet phrased differently from the reference, and word-overlap metrics alone would unfairly penalise it. The metric converts both the model output and the reference into embeddings (numeric vectors that capture meaning) and computes the cosine similarity between them, where a value near 1 indicates the two texts are semantically close and a low value indicates the answer has drifted in meaning. To implement it, build an evaluation set of inputs with reference answers, embed both the reference and the model output with a consistent embedding model, compute the cosine similarity in your evaluation harness, and log per-item and aggregate scores as evidence; run this in pre-deployment testing and as ongoing monitoring on sampled production responses. The threshold is set as a target per control objective with an alert on breach (proposed), so set a risk-based threshold suited to the use case; a score below it raises an alert prompting review of whether the model's meaning has drifted from the expected answer or its retrieval context. Use it alongside factuality checks, since two texts can be semantically similar yet still contain a factual error.