Factuality

Explanation

Factuality via DistanceScore checks how far an AI system's output diverges from a trusted reference or its grounding source, providing a measurable signal of factual drift across document-extraction and RAG use cases where extracted or generated content must match the underlying record. The DistanceScore expresses the closeness between the model output and the reference such that higher values indicate the output stays well within an acceptable distance of the expected content. To implement it, pair each output with its reference or source, compute the DistanceScore in your evaluation harness, and log per-item and aggregate scores as monitoring evidence; run it during pre-deployment validation and continuously on sampled production output so that gradual degradation is caught early. The thresholds are a target of at least 0.90, with investigation triggered below 0.85 and remediation required below 0.80 (proposed): a score in the investigate band prompts a reviewer to examine why outputs are diverging from the reference, while a score in the remediate band requires corrective action such as adjusting extraction logic, retrieval, or model configuration. Because this is rated Immediate priority, treat sustained scores in the remediate band as a trigger to constrain reliance on the system until accuracy is restored.