Search the Atlas

Search risks, controls, and glossary terms

MediumReasoning ErrorRealized

Quantitative & arithmetic reasoning errors

Model Accuracy & Reliability

Description

The model makes calculation, unit, or multi-step reasoning errors in numerically sensitive financial tasks.

Example scenario

An EMI/affordability summary miscomputes interest, misstating a customer's eligibility.

Real-world evidenceRealized

The COVID-19 pandemic caused massive distribution shifts that visibly degraded deployed ML models in credit scoring, demand forecasting, fraud detection, and clinical decision support — a widely reported and studied real-world phenomenon. Multiple academic papers and industry post-mortems documented how models trained on pre-pandemic data failed materially when deployed against pandemic-era data.

Primary mitigations

  • Tool-assisted calculation (deterministic compute)
  • verifier models
  • constrained numeric output
  • human review of figures.

Detection signals

Numeric-accuracy benchmarks; recomputation checks; discrepancy alerts vs system of record.

Mitigating controls

10
Non-agentic controls

Related risks in Model Accuracy & Reliability