Retrieval Stability

Explanation

Retrieval Stability checks that a RAG system's vector embeddings stay consistent over time so that the same query keeps retrieving the right documents — embeddings can silently shift when the embedding model, tokenizer, document corpus, or preprocessing changes, causing a bank's knowledge assistant to suddenly surface stale or irrelevant policy documents even though the user's question is unchanged. It is measured by the Embedding Drift Score, a similarity measure (closer to 1.0 means embeddings are stable) comparing current embeddings or retrieval behaviour against a trusted baseline. No explicit formula is given, so implement it as a periodic similarity comparison — for example cosine similarity between baseline and current embeddings of a fixed probe set, or overlap of top-k retrieved documents — and log the score, the probe set, and the model/index version each run. Run it on a schedule and after any re-indexing or model upgrade in the embedding/retrieval layer. The thresholds are: ≥ 0.90 is the healthy target, scores below 0.85 trigger investigation, and scores below 0.80 require remediation such as re-indexing, rolling back the embedding model, or rebuilding the vector store before the degraded index continues serving production traffic.