AI Model Drift in Banking: Why SR 11-7 Misses Context Drift — and What Regulators Now Expect

A credit model's Gini coefficient can hold steady for a year while the LLM sitting next to it in production quietly starts answering questions it was never designed to answer. Both are "drift." Only one of them shows up on a dashboard built for the other, and that gap — not a lack of monitoring effort — is why banks keep discovering LLM governance failures months after they started.

Three kinds of drift, and only two of them are on anyone's dashboard

Data drift is the familiar one: input feature distributions move away from training-time conditions — post-pandemic credit behavior, a new fraud pattern, a demographic shift in the customer base. SR 11-7's population-stability-index monitoring was built for exactly this, and it works. Concept drift is the second familiar case: the relationship between inputs and outcome itself changes, as fraudsters adapt around a detection model or credit risk factors shift during a rate cycle, and SR 11-7's revalidation-trigger framework handles this too, assuming the monitoring thresholds are actually tuned to catch it.

Context drift is the one SR 11-7 has no answer for, because it isn't a statistical phenomenon at all — it's a deployment phenomenon. The model's weights haven't moved. What's moved is the system prompt wording, the RAG corpus feeding it, or simply how customers have learned to use the tool. A model that was compliant on day one can be out of its approved envelope by month six with zero code change and zero retraining event to trigger a revalidation review, because nothing about a prompt edit or a document upload looks like the kind of "model change" SR 11-7's change-management process was built to catch.

Why performance metrics don't see this coming

SR 11-7's monitoring vocabulary — accuracy, precision, recall, Gini, the KS statistic — assumes a ground-truth label exists to measure against, which works perfectly for a probability-of-default model and fails completely for a chatbot, because there's no single number that captures whether a response was factually correct, regulatory-boundary-compliant, appropriately toned, and resistant to manipulation, all at once.

What context drift actually looks like in a live banking deployment

Scope expansion is the most common version: customers find an edge case and start asking a policy-Q&A bot for product advice it was never scoped to give, and the model's behavior hasn't changed at all — only what it's being asked has. RAG corpus drift is the quieter version: a credit policy document gets updated in the retrieval corpus, and the model's underwriting answers shift accordingly, with no code deployment and therefore no governance trigger under a framework built around model versioning. System prompt erosion happens incrementally and usually with good intentions — a hard constraint gets softened by an engineering team trying to reduce false refusals, and nobody tracks that the softening is itself a governance-relevant change. Adversarial adaptation is the slowest-moving and most dangerous: a model robust against known jailbreak patterns at launch can be vulnerable to evolved variants a few months later, and without scheduled adversarial replay, that erosion is invisible until something gets through.

What a monitoring program actually needs to catch this

A behavioral baseline, established at deployment with a fixed adversarial and edge-case test suite, is the prerequisite for everything else — without one, there's no reference point to measure drift against later, which makes "we'll monitor for drift" a hollow commitment on its own. Daily automated regression against that baseline, using semantic similarity scoring to flag meaningful shifts, catches scope expansion and prompt erosion early; a cosine-similarity threshold around 0.85 is a reasonable starting point in practice, not a number either SR 11-7 or RBI's draft eMRM guidance specifies, so tune it against your own baseline's natural variance rather than treating it as a pass/fail regulatory line.

Treating system-prompt updates and RAG corpus changes as material model changes — triggering the same drift assessment a model version change would — closes the governance gap that context drift otherwise slips through entirely. Monthly adversarial replay of past incidents against the current model turns "we tested this once at launch" into an ongoing control, and a regression on a previously-caught attack is exactly the kind of alert a monitoring program should be built to surface.

What to do this quarter

• Establish a documented behavioral baseline for every production LLM this month if one doesn't already exist — nothing else in this list works without it.
• Reclassify system-prompt edits and RAG corpus updates as material changes requiring a drift assessment, not routine engineering updates.
• Set up daily automated regression against your baseline and tune the similarity threshold against your own system's variance rather than importing a number from a vendor deck.
• Schedule monthly adversarial replay of past incidents against the current model version, and treat any regression as a monitoring alert, not a one-off surprise.
• Verify every regulatory section citation in your monitoring documentation (SR 11-7, eMRM) against the actual source text before it appears in an examination response.