Every credit AI vendor claims explainability. Feature importance charts, SHAP waterfall plots, LIME summaries — these are standard outputs from any commercial model platform now. Having the outputs doesn't mean a bank has met its explainability obligations, and three gaps show up constantly. Global explainability (how the model works generally) gets confused with local explainability (why this specific decision happened for this specific customer) — validation needs the former, the Fair Practices Code needs the latter. Technical output in model notation isn't a customer-facing explanation: a SHAP value of -0.18 for "bureau_utilization_pct" is correct and also useless to the customer reading the rejection letter. And LLM-generated explanations are fluent but may not reflect the model's actual reasoning at all, which makes them regulatorily unreliable wherever explanation accuracy actually matters.
Explainability methods: when each one earns its place
SHAP computes each feature's marginal contribution to a specific prediction, relative to the model's average output. It's mathematically grounded in Shapley values from game theory and model-agnostic, which makes it the most defensible local explanation method available for banking credit AI — use it for per-customer rejection explanations, adverse-action reasons, loan officer override support, and examination evidence of local explainability. The catch: it's computationally expensive on tree models with many features, and someone still has to translate the technical output into plain language before a customer ever sees it.
LIME approximates model behavior near one prediction by fitting a simpler linear model on perturbed data around that point. It's faster than SHAP for some model types and works on architectures SHAP can't reach directly, including neural networks — but it's a looser approximation that may not faithfully represent the model's true local decision surface, and it's less defensible than SHAP if an examiner pushes on methodology. Use it where SHAP's computational cost is prohibitive, or as a supplementary cross-check alongside SHAP rather than a replacement for it.
Attention weights show which tokens a transformer model focused on when generating its output. This is the most natural explanation method for text AI — document classification, NLP-based fraud detection — because the relevant explanatory signal is locatable directly in the text. The limitation is real and easy to forget under deadline: attention isn't causation, and high-attention tokens aren't necessarily what actually drove the output, so this should never stand alone as the explanation for a credit decision.
Counterfactual explanations answer "what would need to change for this decision to flip?" For a rejected application: "if your outstanding debt were 20% lower, this application would have been approved." This is the most actionable explanation format for a customer and increasingly the standard regulators reference for actionability — but the suggested change has to be genuinely achievable within realistic ranges, and it has to be checked that the suggested change isn't itself a proxy for a discriminatory factor.
Regulatory explainability requirements by use case
| AI use case | Explainability type required | Regulatory basis | Common compliance gap |
|---|---|---|---|
| Credit rejection | Local, plain-language, counterfactual, in the customer's preferred language where applicable | RBI's Fair Practices Code (reasons for rejection must be communicated); DPDP Act data accuracy duty under Section 8(3) for the underlying data, distinct from any explanation right | SHAP values produced but never translated to plain language; English-only explanation for regional-language customers; global feature importance substituted for a per-decision explanation |
| Credit pricing (risk-based) | Local — why this rate, which risk factors drove pricing above base rate | Fair Practices Code; fair lending principles | Pricing treated as proprietary and not explained; factors described generically instead of per-customer |
| Fraud decline / transaction hold | Partial local — reason for the flag, without revealing the specific features that would let a fraudster game detection | Fair Practices Code; customer service standards; Banking Ombudsman scheme for false-positive complaints | No customer explanation on fraud holds beyond "security reasons"; dispute process that never actually reviews the AI decision factors |
| Model validation | Global plus local — feature importance for conceptual soundness, local samples for explanation quality | SR 11-7 (or its 2026 successor, SR 26-2, for US institutions); RBI's model risk guidance, which expects model documentation to include an explanation of model logic | Validation report notes that explainability tooling exists without ever validating the accuracy or quality of what it produces |
| LLM customer service | Source attribution for RAG (which knowledge base entries drove the response); self-generated explanations treated as unverified, not sufficient for credit decisions | General expectation that AI outputs remain explainable, consistent with RBI's emphasis on understandable, auditable AI systems | LLM generates a plausible-sounding explanation that doesn't reflect its actual reasoning; no RAG source attribution in customer-facing output |
India-specific explainability requirements
Regional language explanation is a Fair Practices Code requirement, and it's where most banks are weakest. The Fair Practices Code requires communication in the customer's preferred language, which for credit AI means the SHAP-based explanation has to be translatable into genuinely plain Hindi, Tamil, Telugu, Bengali, Marathi, Gujarati, and whatever other languages the bank actually serves customers in. An English-only explanation pipeline fails this requirement outright for regional-language customers — this isn't a future enhancement, it's a present compliance gap at most banks with meaningful vernacular customer bases.
DPDP Act rights for AI-driven decisions are real, but they're access and correction rights, not an explanation right. Build the per-decision lineage capability regardless — it's required either way, just for a different statutory reason than originally stated.
Proxy variable explanations need to surface legitimate factors, not the proxy itself. If pincode, UPI counterparty patterns, or device type sit in a credit model's feature set, telling a customer "applicants in your pincode cluster have higher default rates" effectively discloses that the model uses geography as a proxy — which raises both a disclosure problem and a proxy-discrimination disclosure problem at the same time. Build explanation pipelines that surface the legitimate credit factors (payment history, income, existing debt) instead of the proxy-correlated ones, even when the proxy features carry real statistical weight in the model.
What to fix in the explainability pipeline now
- Build a plain-language translation layer between SHAP/LIME output and the customer-facing adverse-action notice — don't let technical output reach customers directly.
- Stand up regional-language explanation testing for every language the bank actually serves, not just English and Hindi.
- Correct internal documentation that cites a "DPDP right to explanation" — reframe credit explanation obligations around the Fair Practices Code, and DPDP access/correction rights around per-decision data lineage.
- Audit every customer-facing explanation for proxy variable leakage — rewrite any that would reveal a geographic, UPI-pattern, or device-based proxy to the customer.
- For any LLM-based customer service explanation, require RAG source attribution before treating the explanation as reliable enough for a regulated use case.
Zytra's explainability module generates per-decision SHAP-based explanations at inference time, translates them into plain-language customer reasons in English and regional Indian languages, and produces counterfactual explanations — with explanation accuracy validated, not just existence. Request a demo to see per-decision explanations stored alongside decision records for DPDP access-request fulfillment and Fair Practices Code compliance together.
Related: AI fairness and explainability
Per-decision SHAP explanations in regional Indian languages — Fair Practices Code compliant, DPDP Act explanation right fulfillment
Zytra generates per-decision SHAP-based explanations at inference time, translates them into plain language in regional Indian languages, produces counterfactual explanations, and stores decision-level explanations for DPDP Act individual explanation right fulfillment. Explanation accuracy is validated, not just existence.