AI Cost Governance in Banking: LLM Inference Costs, Compute Budgets, and Financial Oversight

Why AI Costs Break the Budget Model Banks Already Have

Traditional bank IT budgeting assumes largely fixed infrastructure costs that get allocated once a year and revisited quarterly. AI spend doesn't behave that way — it's usage-driven, scales with transaction volume, and shifts with prompt design and model selection choices made by individual engineers, often without anyone in finance in the loop. A bank that budgets AI the way it budgets a core banking license will be wrong within a quarter, in either direction.

Three Governance Failures That Show Up Almost Everywhere

Costs get pooled into one opaque line item, which hides which specific use case is actually expensive and which is cheap. Per-use-case spend goes untracked against the original business case that justified the project, so nobody can say whether a use case is still earning its budget. And prompt engineering decisions — which model tier, how much context, how many retries — get made by developers with no financial visibility into the fact that those choices can produce 3–10x cost variation for functionally similar output. That last one is the quiet killer: a developer optimizing for "it works" has no reason to also optimize for "it works for a reasonable price," unless cost visibility is built into their workflow.

Six Categories to Actually Track

LLM inference tokens (the most visible and usually the fastest-growing line), compute infrastructure utilization, model retraining expense, RAG and vector database operations, human-in-the-loop oversight labor, and safety guardrail infrastructure. That last category is easy to undercount — guardrail and safety-filtering infrastructure layered onto an LLM pipeline can meaningfully increase the base inference cost, and a cost model that only counts the core model call will consistently understate what a safety-compliant deployment actually costs to run.

India-Specific Cost Dimensions That Generic Frameworks Skip

Forex exposure is real and underweighted: most frontier LLM pricing is USD-denominated, so rupee depreciation is a direct, uncontrollable cost driver for any India-based bank running meaningful LLM volume, and that exposure belongs in the same conversation as any other foreign-currency cost line, not treated as a rounding error. Regional language support carries a cost premium over English-only deployment — tokenization efficiency, translation infrastructure, and smaller optimized model availability all push costs up for non-English, non-Latin-script languages, and banks should expect and budget for this rather than being surprised by it mid-rollout. PMJDY-segment economics operate at a different scale entirely: low average transaction values mean the AI cost per customer interaction has to be managed far more tightly than for a premium banking segment, or the unit economics simply don't work. And payment-related data localization requirements mean India-specific infrastructure costs — compute and storage that must stay onshore — are a structural cost floor, not a vendor negotiation variable.

Concentration risk also has a cost dimension worth naming directly: a bank that routes the overwhelming majority of its AI inference volume through a single provider is exposed to that provider's pricing changes with no real negotiating leverage, which is the same concentration risk RBI's outsourcing framework worries about for operational resilience — just showing up on the cost side of the ledger instead.

What the Board Actually Needs on AI Cost

Four things, reported together rather than scattered across different functions' updates: AI costs as a percentage of total technology spend, with the trend over time; a variance explanation for any use case whose cost moved meaningfully from its budgeted baseline; ROI metrics that actually compare operating cost against documented value delivered, not just adoption or usage volume; and a provider concentration assessment that flags how exposed the bank is to a single vendor's pricing or availability. A board that only sees total AI spend, with no breakdown of where it's going or why it moved, cannot tell the difference between a use case that's scaling successfully and one that's quietly becoming unaffordable.

Action Checklist for AI Cost Governance

1. Tag every AI cost — inference, compute, retraining, RAG infrastructure, oversight labor, guardrails — to a specific registered use case, not a shared pool.
2. Give developers cost visibility at the point of model- and prompt-design decisions, not after the monthly bill arrives.
3. Track USD/INR forex exposure on LLM spend as its own line item, with a defined tolerance band that triggers finance review.
4. Budget regional-language deployment at a premium over English-only, based on your own measured token and infrastructure overhead rather than assuming parity.
5. Report AI cost as a percentage of technology spend, with variance explanations and provider concentration risk, to the board alongside — not separately from — AI risk reporting.