Latency SLA

Explanation

Latency SLA checks that the system responds quickly enough for its use case, measured at the tail rather than the average so that the slowest experiences are governed. This matters because in customer-facing chat and document-processing workflows, slow responses degrade user experience and can breach operational service commitments; tail latency is what frustrated users and downstream systems actually feel. It is measured by P95 Latency — the response time below which 95% of requests complete — computed per capability. To implement it, instrument end-to-end request timing at the serving layer, aggregate the 95th-percentile latency over rolling windows broken out by capability (interactive chat versus document extraction), and log per-request timings and the computed P95 as evidence for monitoring and measurement. The threshold differs by workload: P95 under 2 seconds for chat and under 5 seconds for extraction. As a Short-term-priority control, exceeding the relevant P95 threshold should trigger investigation of the bottleneck (model latency, retrieval, context size, or infrastructure) and performance remediation such as scaling, caching, or model/prompt optimisation to restore the service level.