Confidence Score Calibration

Explanation

Confidence score calibration checks whether the model's stated confidence actually matches how often it is right — for example, when an extraction or retrieval model says it is 90% confident, it should be correct about 90% of the time. This matters because banks use confidence to decide what to auto-process versus route to a human; a model that is overconfident will silently push wrong results straight through (a misread loan amount auto-booked), while an underconfident one floods reviewers needlessly. It is measured by the Calibration Error Rate, computed as |Expected Accuracy - Observed Accuracy|: the absolute gap between the confidence the model claims for a bucket of predictions and the accuracy it actually achieves on them. Implement it by bucketing recent predictions by their confidence band, comparing predicted versus measured accuracy on labelled or human-verified samples, and logging per-band gaps as evidence; run this regularly on production traffic for Document Extraction and RAG. The target is a calibration error of less-than-or-equal-to 1% (ideally 0); any breach raises an alert, prompting recalibration (for example temperature scaling) or a review of the auto-processing confidence cutoff before more decisions are trusted.