AI Explainability in Banking: SHAP, Counterfactuals, and the Regulatory Requirements That Break Post-Hoc Methods

A relationship manager denying a loan has to tell the customer why, in one or two sentences a human being can act on. That requirement — not a research paper's definition of interpretability — is the actual bar banking AI explainability has to clear, and it's why the same explainability method that works perfectly well for a credit-scoring model falls apart the moment you point it at a large language model.

SHAP is the right tool for gradient-boosted credit models — and the wrong one for LLMs

SHAP (Shapley Additive Explanations) attributes a prediction to the input features that drove it, and for a gradient-boosted or logistic credit model, that attribution maps cleanly onto the reason codes RBI's Fair Practices Code effectively requires for an adverse-action explanation — "income-to-EMI ratio" and "recent delinquency" are features a customer and a loan officer can both understand. . Apply SHAP to a neural network with several transformation layers and you get an explanation that looks locally coherent and is often globally misleading, because the attribution is approximating a much more complex internal computation than the simple feature-weighting it's built to explain.

Post-hoc methods don't just weaken on LLMs — they fail in a specific, predictable way

SHAP, LIME, attention-visualization, and counterfactual methods all assume there's a relatively stable mapping from input features to output you can approximate after the fact. An LLM's output depends on context, phrasing, and token-level interactions across a much larger and more entangled computation, so a post-hoc method applied to it produces an explanation that can look plausible on the screen and have essentially no relationship to what actually drove the specific output — attention weights, in particular, are frequently misread as "what the model focused on" when they don't reliably correspond to causal influence on the output at all.

The fix for LLMs is designing for explainability, not explaining after the fact

For an LLM-based system, the explainability work has to happen at build time: constrain the prompt and output structure so the model cites its sources, ground every factual claim in a retrieved document the customer or examiner can independently check (RAG-grounded attribution), and structure outputs so a claim traces back to a specific retrieved passage rather than an opaque generation. This is a genuinely different engineering discipline than bolting SHAP onto a finished model, and banks that try to retrofit post-hoc explainability onto an already-deployed LLM are solving the wrong problem at the wrong stage.

Four frameworks, four different explainability bars — and a bank needs to clear all of them with one system

RBI's Fair Practices Code wants a customer-readable reason code. The EU AI Act wants a human overseer to be able to meaningfully interpret the system well enough to intervene. . And SR 11-7/OCC guidance wants technical documentation a model risk validator can assess for conceptual soundness. A single explainability layer has to serve a customer, a regulator-facing human overseer, and a technical validator simultaneously — which is why "we have SHAP values" is not actually an answer to "is this system explainable," it's an answer to one of four different questions.

India's explainability bar is specific: per-application, not per-model

RBI's expectation isn't "the model is explainable in general" — it's that each individual credit rejection comes with factors specific to that application, which is a materially harder bar than publishing a model-level feature-importance report once at deployment. DPDP Act 2023 gives customers a right to request human review of an automated decision, which only functions if the human reviewer actually has an interpretable basis to review against — a black-box score with no reason codes attached makes that statutory right unexercisable in practice. Regional-language LLM deployments make this harder still: a RAG-grounded citation in Hindi or Tamil needs the same traceability back to source documents as the English version, and testing explainability quality separately per language is not optional if the bank actually wants to honor the DPDP review right for non-English-speaking customers.

What a bank should actually do

1. Keep SHAP (or an equivalent attribution method) for traditional ML credit and risk models — it's the right tool there and doesn't need replacing.
2. Stop applying post-hoc explanation methods to LLMs and switch to intrinsic design: constrained prompts, structured outputs, and RAG-grounded source citation built in from the start.
3. Generate per-application, customer-readable reason codes for every AI-assisted credit decision — a model-level feature-importance report does not satisfy this.
4. Build one explainability layer that can answer a customer's question, a regulator's human-oversight requirement, and a validator's technical-documentation requirement without three separate bolt-ons.
5. Test RAG-grounded explanation quality and traceability separately in every regional language the bank actually deploys in, not just English.