What are the main risks of generative AI in banking?

The main risks of generative AI in banking fall into eight categories that don't exist or manifest differently in traditional statistical model risk: (1) Hallucination — LLMs generate plausible but factually incorrect outputs, including incorrect regulatory information, incorrect product terms, incorrect customer account information; (2) Prompt injection — adversarial inputs override system constraints, causing the model to give prohibited advice or disclose unauthorized information; (3) Context drift — model behavior changes over time as system prompts are updated, RAG corpus changes, or customer usage patterns evolve; (4) Cross-tenant data leakage — shared inference infrastructure creates risk that one bank's customer data contaminates outputs for another bank's customers; (5) Excessive agency — agentic LLMs with access to banking systems execute actions outside their authorized scope; (6) Output unpredictability — unlike statistical models, LLM outputs for similar inputs can vary significantly, making performance guarantees difficult; (7) Knowledge cutoff — LLMs don't know about regulatory changes after their training cutoff, creating compliance risk for frequently-updated regulations; (8) Copyright and confidentiality — LLMs can inadvertently reproduce training data including confidential content.

How should banks govern generative AI differently from traditional AI?

Governing generative AI in banking requires governance architecture that extends traditional model risk management in five areas: (1) Real-time guardrails — unlike statistical models where governance is primarily pre-deployment, LLMs require production-time enforcement at every inference; guardrails classify and block prohibited outputs before they reach customers; (2) Behavioral monitoring — performance metrics don't capture LLM behavioral drift; banks need behavioral consistency testing against fixed baselines, run regularly in production; (3) Adversarial evaluation — regular red-teaming against banking-specific attack scenarios (OWASP LLM Top 10, FinProof benchmark), not just standard performance benchmarks; (4) Agentic scope controls — for LLMs with tool access, authorization boundary enforcement at the tool layer, not just in the system prompt; (5) Forensic capability — ability to reconstruct what the model saw and why it produced a given output, for incident investigation and regulatory reporting. Banks that apply traditional MRM governance to LLMs are measuring the wrong things and missing the highest-risk failure modes.

Is generative AI covered by RBI's model risk management guidelines?

Yes, RBI's draft eMRM guidelines explicitly extend to generative AI and LLMs. RBI's 2024 AI Circular establishes that all production AI systems — including generative AI — are subject to board-approved governance frameworks and audit trail requirements. The FREE-AI Framework's Explainability pillar and Evaluation pillar apply to LLMs, requiring banks to be able to explain LLM outputs and conduct regular evaluation including adversarial testing. The draft eMRM guidelines extend model risk management validation requirements to include LLMs, with validation depth proportionate to the model's risk tier. Banks using LLMs in credit decisions, fraud detection, or customer service — the three most common generative AI deployments in Indian banking — must have governance documentation for these models. Banks that deployed generative AI under the assumption that it was not covered by model risk management requirements are in a governance gap.

Generative AI Risk in Banking: What Makes LLMs Different and How to Govern Them

A credit scorecard fails in ways a model risk committee has spent fifteen years learning to catch: it drifts, it loses discriminatory power, it shows disparate impact on a protected class. An LLM fails in ways that same committee has no muscle memory for — it can look perfectly healthy on every metric the committee tracks while quietly giving out wrong information, leaking one customer's data into another's session, or talking itself past a constraint nobody removed from the system prompt. Generative AI isn't a bigger version of the model risk problem banks already solved. It's a different problem wearing the same org chart.

Eight failure modes, and which ones actually get missed

Hallucination is the one every vendor demo addresses and the one that still causes real harm, because a wrong interest rate or a wrong regulatory deadline stated confidently to a customer is a mis-selling and compliance problem, not just a quality bug. Prompt injection and excessive agency are the pair that turn a chat interface into a financial-control failure: once an LLM has tool access to a payment or account-modification API, a system-prompt instruction not to exceed its scope is advice, not enforcement, and the only thing that actually stops excessive agency is authorization logic at the tool-call layer itself.

Cross-tenant leakage through shared inference infrastructure deserves the attention this piece gives it, because it produces no anomaly signature and typically surfaces only when a customer notices something that shouldn't be in their chat window.

Context drift is the failure mode that most exposes the limits of inherited SR 11-7 thinking, because a model's underlying weights haven't changed at all — a system prompt edit, a RAG corpus update, or a shift in how customers actually use the tool is enough to move behavior out of its approved envelope with zero code deployed and therefore zero change-management trigger under a traditional model-governance process. Knowledge cutoff risk is specific and real for India: an LLM trained before a FREE-AI-related update or a fresh RBI circular will confidently give a customer regulatory guidance that's already stale, and nothing about the model's confidence level signals that to the customer or the compliance officer reviewing the transcript.

Output unpredictability and sensitive-information memorization round out the list, and both point to the same uncomfortable fact for banks fine-tuning models on proprietary data: a model that memorized a customer's account history during fine-tuning can be made to reproduce it later through adversarial querying, and no amount of access control on the model's API changes that if the memorization happened at training time.

Why the existing four-component governance model misses all eight

Model inventory, independent validation, ongoing monitoring, and governance review are the right four components for a statistical model and genuinely insufficient for an LLM, for a structural reason worth stating plainly: every one of those four components was designed around the assumption that a model's behavior is fixed between retraining events. An LLM's behavior isn't fixed between anything — it can shift with a system prompt edit, a RAG corpus update, or simply because enough customers found a new way to use it. Pre-deployment validation catches none of that, because the risk is inherently a production-time risk, not a design-time one.

The governance architecture that actually covers this

The risk-to-control mapping in the original is reasonable in substance: pre-deployment adversarial testing paired with runtime injection detection for prompt injection; tool-layer authorization enforcement (not system-prompt instructions) for excessive agency; KV-cache tenant isolation for cross-tenant leakage; behavioral baselining and automated regression testing for context drift. The common thread across all four is that none of them are one-time activities — every one requires ongoing, production-time monitoring that a pre-deployment validation checklist, however thorough, cannot substitute for.

What to do this quarter

Build a documented behavioral baseline for every production LLM before assuming your existing performance monitoring covers it — it doesn't, and without a baseline there's nothing to measure drift against later.
Move agentic authorization enforcement to the tool-call layer for every system with payment or account-modification access, and stop relying on system-prompt instructions as a security control.
Treat any RAG corpus update or system-prompt change as a material model change requiring a drift assessment, not an operational tweak that skips governance review.
Add a regulatory-freshness check to any customer-facing LLM that could give advice tied to RBI, SEBI, or DPDP Act requirements — a model trained before the last guidance update is a live compliance risk, not a theoretical one.
Instrument KV-cache or shared-inference infrastructure for cross-tenant isolation violations now, rather than waiting for a customer complaint to surface the first one.

Related: generative AI risk and LLM governance

LLM GOVERNANCE

LLM governance for banking

LLM-specific governance requirements

HALLUCINATION

LLM hallucination management

6 BFSI types, 3-layer mitigation

LLM SECURITY

LLM security in banking

6 threat categories, guardrails architecture

SR 11-7

SR 11-7 for generative AI

SR 11-7 gaps for LLM governance

Generative AI governance built for BFSI — not adapted from generic frameworks

Zytra covers hallucination, prompt injection, cross-tenant leakage, excessive agency, and context drift — the eight failure modes traditional MRM governance doesn't address.

Request a demo SR 11-7 GenAI gaps →

Generative AI Risk in Banking: Eight Failure Modes That Don't Exist in Traditional AI — and the Governance Architecture That Addresses Them