AI Model Performance Benchmarking in Banking: Metrics, Baselines, and Regulatory Evidence

Technical Benchmarking and Governance-Grade Benchmarking Are Not the Same Exercise

Most banks measure whether a model performs well enough to deploy. Fewer banks produce evidence that the model was assessed against thresholds set before anyone saw the results, on data the model never trained on, with segment-level performance broken out separately from the aggregate number, and retained in a form that proves it wasn't altered afterward. That second version is what an examiner is actually looking for, and the gap between the two is where most benchmarking-related findings originate.

What to Benchmark, by Domain

Credit AI needs discrimination (Gini/AUC-ROC), stability (PSI), and calibration (expected vs. actual default rate by score band) assessed together — a model with strong discrimination and poor calibration is not fit for pricing or provisioning decisions, full stop. Fraud detection needs precision, recall, and false positive rate measured at the actual deployment operating threshold, not the theoretical ROC optimum, plus p95 latency against the payment system's SLA. AML/transaction monitoring needs alert rate, false positive rate (alerts that never produce a SAR), and typology coverage — and a lower alert rate is only good news if detection quality against known typologies hasn't quietly degraded alongside it. LLM customer service AI needs domain accuracy on a BFSI-specific evaluation set, a safety pass rate across adversarial categories, financial-advice-boundary precision, and latency — a model that's accurate but fails safety benchmarks is not cleared for customer-facing deployment regardless of its accuracy score. Document AI/OCR needs field-level accuracy broken out per document type, because a dominant document type's strong accuracy can mask a minority document type performing badly — and for KYC fields specifically (name, date of birth, ID number), that masked failure is a sanctions-screening and KYC-failure risk, not just a UX annoyance. Agentic AI needs task completion rate, scope adherence, and authorization compliance, plus adversarial scenarios testing whether the agent can be manipulated through its own tool responses into exceeding its authorization — a benchmark category with no real equivalent in standard ML evaluation.

Baselines Set After Seeing the Results Are Not Baselines

The three baseline failures that show up repeatedly in BFSI AI governance: thresholds set after the model's results are already known (which makes the threshold fit the outcome instead of the other way around), thresholds derived from development data instead of a genuinely held-out set, and thresholds defined only at the aggregate level with no segment breakdown. Any one of these three turns a benchmark from a governance control into a post-hoc rationalization exercise, and an examiner who spots reversed sequencing — threshold documented after the validation run — treats it as a governance failure on its own, independent of whether the model actually performed well.

The specific numeric thresholds commonly cited in BFSI benchmarking practice — PSI below 0.1 as stable, 0.25 as a review trigger; Gini above 0.35 as a reasonable credit-model floor; sub-1% FPR for customer-impacting fraud blocks — are industry convention banks have converged on, not figures any regulator has mandated by rule. Document your own thresholds, justify them, and set them before you see results; don't cite the convention number as if citing a regulation.

What the Examination-Ready Benchmark Record Actually Contains

A slide deck of benchmark outcomes is not evidence — it proves nothing about whether thresholds were pre-defined or the test data was genuinely independent. The record that survives an examination contains: the pre-defined benchmark plan (metric definitions, thresholds, test data description) dated before the validation run; the test dataset's sample period, size, and segment composition; raw per-metric, per-segment results; pass/fail against each pre-defined threshold; an explanation and disposition for anything that failed; and the validation function's dated sign-off — retained in a hash-chained form that proves nothing was edited afterward, not in an editable spreadsheet.

India BFSI Context: Challenger Models, PMJDY Segments, and FIU-India Typologies

Indian banks face benchmarking obligations that generic international guidance doesn't spell out. RBI's model risk and internal-model validation expectations carry the same challenger-model logic found in international MRM practice — for credit AI, that typically means comparing the model against a simpler, explainable baseline like logistic regression, and being ready to explain why the added complexity earned its place. ****

Credit AI deployed against PMJDY accounts or new-to-credit customers needs its benchmark performance reported on that segment specifically — a model posting a 0.45 Gini overall but 0.18 on the PMJDY segment is failing exactly the population RBI examination attention is most focused on, and an aggregate-only benchmark would hide that completely. For AML, FIU-India publishes money-laundering typologies Indian institutions are expected to detect; typology coverage — what share of published typologies the model can actually catch — is a binary coverage benchmark distinct from statistical false-positive-rate benchmarking, and it needs its own typology-by-typology test evidence. The benchmark record should also double as the baseline for ongoing production monitoring comparisons — store it so a monitoring system can compare against it directly, not archived somewhere that requires manual reconstruction months later.

Action Checklist for Your Next Benchmarking Cycle

1. Document and freeze thresholds before the validation run starts — treat any threshold set after seeing results as invalid.
2. Break out every benchmark by segment (PMJDY, priority sector, geography, channel) in addition to the aggregate number.
3. Retain the full benchmark record — plan, data description, raw results, pass/fail, sign-off — in a tamper-evident, hash-chained form.
4. Maintain a documented FIU-India typology coverage matrix for every AML model, updated when FIU-India updates its typology list.
5. Store deployment-time benchmarks in a form your ongoing monitoring system can compare against directly, not a static report that requires manual lookup.