AI Explainability Tools in Banking: SHAP, LIME, Integrated Gradients, and Regulatory Adequacy

The Governance Question Explainability Tools Actually Have to Answer

An explainability method that produces compelling output but doesn't answer the question a regulator, a customer, or a board member will actually ask is false assurance dressed up as governance. Choosing an explainability tool is a governance decision, not a modeling preference — it determines whether the bank can actually meet its accountability obligations when someone asks "why was this declined" or "could this be discriminating against someone."

Which Tool Does What, and Where It Falls Short

SHAP is the closest thing to a gold standard for tabular models — credit scoring, fraud detection, AML — because its game-theoretic foundation produces consistent, additive attributions that translate directly into customer language ("income contributed +0.18; recent missed payments contributed -0.41"). TreeSHAP makes the computational cost tractable for tree-based credit models. It remains the strongest choice for individual decision accountability in structured-data use cases.

LIME is model-agnostic and faster than full SHAP, but its local surrogate-model approach is unstable — near-identical inputs can produce meaningfully different explanations, which is a real governance problem when a bank needs consistent explanations for similar cases. Treat LIME as a supplementary tool, not the primary explanation method for individual credit decisions.

Integrated Gradients satisfies the completeness axiom and works where SHAP's cost is prohibitive — deep learning models, LLM token attribution. Its governance strength is that the baseline choice makes the attribution interpretation explicit; its weakness is that a poorly chosen baseline produces attributions that are technically valid but practically misleading, so the baseline choice itself needs documentation and justification.

Counterfactual explanations answer the question the customer actually cares about — "what would need to change for this to be approved" — rather than the analytical question a developer cares about. This makes them the strongest fit for adverse-action-style obligations, because they give the individual an actionable path rather than a statistical attribution they can't do anything with.

Attention weights are widely used in LLM tooling but are not causally faithful — research has repeatedly shown the tokens that receive the highest attention are not necessarily the ones causing the output. They're a developer debugging artifact, not regulatory-adequate explanation evidence, and a bank that presents an attention heatmap to an examiner as explainability evidence is showing a mechanistic artifact as if it were causal accountability.

TCAV tests whether a model's predictions are sensitive to a human-defined concept — useful for fairness audits asking whether a model has implicitly learned a prohibited or proxy concept, rather than just which raw features it uses. It needs domain expertise to define the concepts worth testing; it isn't something you can run without deciding in advance what you're looking for.

Which Tool Satisfies Which Obligation

LLM Explainability Is Still an Open Problem — Design Around It, Don't Pretend It's Solved

Standard tools built for tabular models don't transfer cleanly to LLMs: integrated gradients produces token-level attributions that become impractical at the scale of a full context window, and attention weights aren't causally adequate on their own. The practical answer for LLM-based customer service or document analysis combines three things — reviewing the LLM's natural-language self-explanation for accuracy, systematically testing behavior on known input categories, and restricting the LLM's output types to ones where explanation stays tractable (a system that retrieves and summarizes documented policy is far more explainable than one that synthesizes novel advice). For LLM-assisted underwriting, the more defensible architecture uses the LLM to extract and structure information while an explainable ML model on structured inputs actually makes the decision — the decision stays accountable through SHAP or counterfactual explanation even though an LLM touched the pipeline.

India BFSI Context: What India's Framework Actually Requires — and Where It Doesn't

**** Given that gap, the operative explainability obligation for credit decisions in India currently comes from RBI's Fair Practices Code and general consumer protection principles rather than DPDP.

India's proxy variable risk is real and underappreciated: a model that has implicitly learned pincode correlates with socioeconomic or community patterns, or that surname correlates with religious or caste community, can produce a SHAP explanation that looks clean — "location," "name" — while hiding a discriminatory pathway entirely. TCAV testing for concept sensitivity to caste- or religion-correlated concepts is the tool built for exactly this gap; global SHAP alone will not surface it. For regional-language deployments, the explanation obligation — whatever its legal basis — has to extend to the language of interaction, with a verified translation pipeline, since an explanation technically correct in English but never delivered in the customer's language serves nobody.

Action Checklist for Your Explainability Stack

1. Use SHAP or counterfactual explanations as your primary tool for individual credit decision accountability — not LIME, and not attention weights.
2. Document, for every production AI system, which explainability method you use, what governance purpose it serves, and why it's adequate for that purpose.
3. Run TCAV concept-sensitivity testing for caste-, religion-, and language-correlated proxy concepts on every India credit model — global SHAP alone will not catch this.
4. Build verified translation pipelines so explanations reach customers in their language of interaction, not just English.
5. Stop citing DPDP Act as a right-to-explanation basis for automated decisions — it isn't one; cite the Fair Practices Code provisions that actually apply to your product line.