Overconfidence & poor calibration
Model Accuracy & ReliabilityDescription
Output confidence is not calibrated to correctness, so users cannot tell reliable answers from unreliable ones.
A risk-rating assistant expresses high confidence on edge cases it actually gets wrong, driving bad approvals.
Multiple published evaluations (MedQA, LegalBench, FinanceBench) have systematically demonstrated that general-purpose LLMs underperform domain specialists on regulated-domain tasks and often produce generic or overconfident responses. IBM Watson's oncology failures also partly reflected domain knowledge gaps. The risk is well-evidenced in research and field pilots but mass-harm production incidents specifically attributable to domain gaps alone are not cleanly documented.
Primary mitigations
- Calibrated confidence/uncertainty estimates
- abstention on low confidence
- confidence display
- selective prediction.
Detection signals
Calibration error (ECE); confidence-vs-accuracy curves; abstention-rate monitoring.