Decision Integrity

Explanation

Decision Integrity checks whether a Retrieval-Augmented Generation (RAG) system gives answers that are consistent with the context it actually retrieved, rather than contradicting its own source material — for a bank, think of a customer-policy chatbot that retrieves the current overdraft-fee schedule but then states a different fee, a dangerous inconsistency that could mislead a customer or breach disclosure rules. It is measured by the Context Conflict Rate: the share of generated responses whose claims conflict with the retrieved evidence. No explicit calculation method is supplied, so compute it as the proportion of sampled responses flagged as conflicting against their retrieved chunks, using an automated entailment/contradiction checker (e.g. an NLI model or LLM-as-judge that compares each answer sentence to the retrieved passages) with human spot-checks. Instrument this at the post-generation stage of the RAG pipeline: log the query, the retrieved chunks, the final answer, and the per-response conflict verdict so every flag is auditable. The threshold is ≤ 1% (target 0), with an alert on any breach: whenever the conflict rate rises above 1%, the control fires, routing affected responses to review and prompting investigation of the retriever, prompt, or grounding logic.