Inference Cost Stability

Explanation

Inference Cost Stability checks whether the cost of running the model has shifted materially over time, normalised so that changes in usage volume do not mask unit-cost changes. This matters because rising per-unit inference cost — from larger prompts, model upgrades, vendor price changes, or inefficient retrieval in RAG and chat workloads — can quietly erode the economics of an AI service and signals a need to optimise or renegotiate. It is measured by Cost-per-1k-tokens Drift, tracking how the cost per thousand tokens moves period over period. To implement it, instrument the serving pipeline to capture token counts and billed cost per request, aggregate to a cost-per-1k-tokens figure per period, compare each month against the prior month to compute month-over-month drift, and log the underlying usage, pricing, and computed drift as evidence. The threshold is less than 10% month-over-month drift. As a Short-term-priority control, exceeding 10% MoM drift should trigger review of the cost drivers — prompt and context size, model selection, caching, and retrieval efficiency — and corrective optimisation or commercial action to bring unit cost back within the stable band.