SR 11-7 AI Compliance: Five Gaps Traditional MRM Doesn't Cover for Machine Learning and LLMs — and What Examiners Now Expect

SR 11-7 was written in 2011, for models that produce a number from a formula — a PD, an LGD, a VaR estimate. The five pillars it established — inventory, risk tiering, independent validation, ongoing monitoring, governance — still hold up as a structure. What doesn't hold up, unmodified, is the specific validation and monitoring technique SR 11-7 assumes, because an LLM doesn't fail the way a logistic regression fails. Examiners with AI literacy are already probing the five places where the 2011 framework runs out of road, and a bank that hasn't extended its MRM program to cover them is going to find that out in the exam room, not before it.

Gap one: performance metrics can stay strong while an LLM's behavior quietly drifts

A statistical model's drift shows up in its prediction quality — accuracy degrades, calibration slips, you can see it in a dashboard. An LLM can hold strong accuracy and user-satisfaction scores while becoming steadily more willing to answer a question it should refuse, or generating a materially different explanation for two functionally identical cases. Standard SR 11-7 performance monitoring will not catch this, because it isn't measuring the right thing. Behavioral consistency testing — a fixed set of adversarial probes covering known boundary conditions, run on a schedule against production and compared to a baseline set at validation — is the only way to see the drift before a customer or a journalist does.

Gap two: fairness testing has to go beyond demographic parity, and further still for India exposure

SR 11-7 itself is silent on fairness, but ECOA, the Fair Housing Act, and increasingly pointed examiner questions create an obligation in practice that the 2011 text never spelled out. Demographic parity across race and gender is the floor, not the ceiling — US examiners are now asking about intersectional analysis and subpopulation disaggregation beyond binary categories. For any Indian book under RBI's fairness expectations, that floor rises again: proxy-variable testing for caste, religion, language, and disability, categories no standard US fair-lending toolkit was built to test. A validation package that stops at race and gender parity will satisfy neither bar.

Gap three: a vendor's own test results are not independent validation of its model

Banks have gotten comfortable treating a vendor's SOC 2 report or internal test documentation as sufficient evidence for the independent validation SR 11-7 requires on Critical and High tier models. It isn't, and examiners are starting to say so directly — the bank carries the model risk for vendor AI, not the vendor, so the vendor testing its own product cannot be the independent half of the equation. The fix is a bank-commissioned validation, separate from anything the vendor hands over, for every vendor model sitting at Critical or High tier. This is the gap most banks haven't closed, largely because it's the most expensive one to close and the one that's gone unchallenged the longest.

Gap four: SR 11-7 governs predictions, not actions — and agents take actions

An autonomous agent executing loan disbursement or claims settlement isn't producing a prediction for a human to act on; it's acting. SR 11-7's validation questions — is the methodology sound, is the data representative — don't ask what they need to ask of an agent: what is it authorized to do, who approved that authorization scope, and what happens if a prompt injection pushes it past that scope mid-workflow. None of this has SR 11-7 precedent because none of it existed as a risk category in 2011. The extension has to include a written tool-authorization manifest, a human checkpoint above a defined dollar or risk threshold for irreversible actions, and an action-level audit trail — not just a model-output log.

Gap five: reconstructing why an LLM said something requires more than a log line

SR 11-7's documentation standard assumes you can reconstruct a statistical model's output from its inputs and parameters. An LLM's output depends on the prompt, any retrieved context, the conversation history, the model version, and the inference configuration — and a standard application log typically keeps the output, not the full context state that produced it. Rebuilding that context after the fact, for an examiner's investigation or a customer complaint, needs inference-level logging that captures all of it at the moment of inference, plus a forensic replay capability to reconstruct any single past inference on demand, hash-chained so the record can't be quietly edited later.

The examination finding you'll hear if you've skipped this

The most common finding for banks running production LLMs: strong accuracy and satisfaction numbers, and no answer to "how would you detect this model producing an output that violates your own compliance policy?" That's no longer a hypothetical examiner question — it's becoming the standard one for any LLM in a material business function, and the honest answer for most banks today is "we wouldn't, until a customer told us."

Action checklist for extending SR 11-7 to AI and LLMs

1. Build a behavioral consistency baseline for every production LLM at validation, then re-test against it on a fixed schedule — weekly for customer-facing, monthly for internal.
2. Extend disparate-impact testing beyond race and gender parity to intersectional analysis, and add India proxy-variable testing for any Indian book.
3. Commission independent, bank-side validation for every vendor model at Critical or High tier — stop accepting vendor attestations as satisfying this requirement.
4. Write a tool-authorization manifest and human-checkpoint threshold for every agentic AI system before it goes live, not after an incident forces the question.
5. Stand up inference-level logging with forensic replay for every LLM, hash-chained, before an examiner or a regulator asks you to reconstruct a specific decision you can't currently reconstruct.