KV Cache Governance for Banking AI: The Risk No Framework Covers Yet

Every bank running shared LLM infrastructure carries a governance risk that SR 11-7, RBI's FREE-AI report, and the DPDP Act all create obligations around without any of them naming it directly: KV cache cross-tenant data leakage. When a shared inference cache stores context from one customer's session and that context becomes accessible to the next request, you have a data boundary violation that standard audit logs cannot see, let alone prove didn't happen. This article covers the mechanism, the regulatory exposure, and what KV cache governance with Forensic Replay actually has to do to close the gap.

How KV cache creates cross-tenant data leakage

LLMs compute attention over every token in the current context window, and KV cache stores the resulting key-value pairs so the model doesn't recompute them for the next request sharing the same prefix — typically the system prompt every customer shares. The performance case for this is real: shared system-prompt caching commonly cuts inference latency by something in the 30–50% range for banking AI deployments, though the exact number depends heavily on prefix length and traffic mix, so treat any single figure as a starting estimate rather than a guarantee. The risk is just as real. Walk the failure through step by step: Customer A queries their account balance, and the context includes PAN, account number, and transaction history. That context gets cached. If cache isolation fails, Customer B's next request hits a cache carrying residual state from A's session, and B's response ends up shaped by A's financial data — a DPDP Act violation and a potential financial-harm event in one incident.

The leakage is often partial rather than a clean data dump: the model's response to B is subtly shaped by A's cached context in ways that don't jump out of the output text but are detectable in the model's attention patterns if you know to look. Standard application logs capture inputs and outputs, not cache state, which is exactly why a standard log-based audit trail cannot detect or investigate this failure mode. We'd flag this as the governance gap most banks don't know they have, precisely because nothing in their existing audit tooling would surface it even during a thorough review.

Why this matters for Indian banking specifically

KV cache leakage runs straight into the DPDP Act's purpose-limitation and consent architecture. When Customer A's financial data influences Customer B's session, that data is being processed for a purpose beyond its original collection, without B's consent and without A's consent to cross-customer use — Section 4 ties lawful processing to a specified purpose, and that's the provision actually in play. In a system processing thousands of sessions per hour, a systemic isolation failure doesn't produce one violation; it produces a continuous stream of them, one per contaminated session.

For banks running inference on AWS Bedrock, Azure OpenAI, or Google Vertex, the cloud provider manages the underlying cache isolation mechanics. That doesn't let the bank off the hook: RBI's payment-system data-localisation mandate and the DPDP Act both put the governance burden on the bank, not the vendor . A vendor's compliance questionnaire answer that isolation is "enforced" is not examination evidence. We have yet to see an internal audit team actually request cache-isolation test results from an inference vendor rather than taking the attestation at face value — that's the exact gap an examiner will probe first.

Regulatory mapping: where existing frameworks create obligations

KV cache governance: what it actually requires

Isolation. Tenant isolation has to be architecturally enforced, not just policy-level. In multi-tenant banking AI, that means inference infrastructure with per-tenant cache partitioning that prevents accidental cross-contamination by design, not a configuration setting someone could leave off.

Monitoring. Cache poisoning — adversarial inputs designed to pollute the shared cache with malicious context, effectively injecting instructions into other users' sessions through the cache — needs its own detection layer, watching for anomalous cache-write patterns that diverge from legitimate system-prompt updates.

Audit. Forensic Replay logging has to capture every cache access in a format that supports full reconstruction of the model's context state at any inference point. This is the only thing that makes post-incident investigation, regulatory examination, or a DPDP Act complaint adjudication possible, because standard application logs capture inputs and outputs, not the cache state delta that actually explains what happened.

Lifecycle. DPDP Act minimisation duties require that customer context not sit in cache longer than necessary. Cache TTL policy, context clearing between sessions, and logged evidence that retention limits are actually enforced — not just configured — are what an examiner expects to see on file.

Testing. Cross-tenant extraction testing has to run against the production inference configuration, not a separate test environment, because isolation failures are frequently configuration-specific and won't reproduce in a sandboxed setup that doesn't match production traffic and routing.

Zytra: KV cache governance and Forensic Replay

Zytra includes KV cache governance as a named platform capability: tenant isolation enforcement, cache poisoning detection, and Forensic Replay reconstruction of the exact model context state at any inference point from cache access logs, with examination evidence exportable on demand.

What to do before your next inference infrastructure review

1. Request tenant-isolation attestation in writing from your cloud inference vendor (Bedrock, Azure OpenAI, Vertex), then independently test it — a vendor attestation alone is not examination evidence.
2. Run cross-tenant extraction testing against your actual production inference configuration, not a staging environment that won't reproduce configuration-specific isolation failures.
3. Set a cache TTL and context-clearing policy that satisfies DPDP Act storage-limitation obligations, and log proof that it's enforced, not just configured in a settings file.
4. Stand up Forensic Replay logging before you need it for an incident — standard application logs don't capture cache state and can't reconstruct what happened after the fact.
5. Add KV cache isolation testing to your model validation checklist as a named reproducibility control under your SR 11-7 (or equivalent) program, not an informal side-check.