Language/Script Bias

Explanation

Language/script bias checks whether a document-extraction or RAG system performs equally well across the different languages and scripts a bank serves — for example Devanagari, Tamil, Bengali, and Latin scripts on KYC forms or statements — so customers writing in one script are not disadvantaged by weaker accuracy. It matters in multilingual markets like India because uneven extraction quality silently produces more errors and rejections for some communities. It is measured by the Script Equity Score, computed as max(F1_group) − min(F1_group): the gap between the best- and worst-performing language/script groups on extraction F1. To implement it, assemble labelled test sets per language/script, compute F1 for each group, take the maximum-minus-minimum gap, and log per-group F1 scores and sample errors as evidence on a fairness dashboard. Run it pre-deployment and on a recurring cadence, re-testing after model or OCR changes. The proposed thresholds set a target of at least 0.90, with the gap to investigate below 0.85 and remediate below 0.80; crossing the 0.85 investigate level prompts root-cause analysis and breaching 0.80 triggers mandatory remediation — improve training data or models for the weak script, re-test, and document — aligning with the proposed RBI Fairness and Equity and NIST MEASURE 2.11 expectations.