Factuality

Explanation

The Factuality control tracks how often a model's outputs contain factual errors — statements that are wrong, unsupported or fabricated — which is essential where customers or staff act on AI answers about products, fees, eligibility or regulations and a confident falsehood can cause financial loss or mis-selling. It aligns with RBI Safety/Resilience, NIST MANAGE 2.1/3.3 and ISO Clause 6.1.2. The metric is Factual Errors, with a proposed threshold of staying at or below a defined tolerance and alerting on breach. No calculation method is provided, so measure it as the rate of responses (or claims) judged factually incorrect against a ground-truth or evidence baseline — for RAG and summarisation systems this means checking each answer is grounded in retrieved sources. To implement, sample outputs continuously and score factuality using a mix of automated grounding checks (does the claim trace to a cited source?), reference comparisons and periodic human review; log each evaluated output, its verdict and the supporting evidence. Operate trend dashboards and per-release regression checks. Because no fixed number is given, set a risk-based tolerance; whenever the factual-error rate breaches it, the control alerts, prompting investigation of the failure mode, possible rollback, and tightening of grounding or guardrails before continued use.