Session Drift

Explanation

Session Drift checks whether a conversation stays within its intended scope or wanders off-topic over the course of a session — for example a loan-servicing assistant that starts answering general financial-advice, legal, or unrelated questions, or is gradually nudged outside the use case it was approved for. This matters because out-of-scope turns increase the chance of incorrect, non-compliant, or unsafe answers and can be a precursor to manipulation. It is measured by the Session Scope / Topic-Drift Rate: the proportion of turns in a session that fall outside the system's defined scope. To implement it, define the in-scope topic/intent boundary for each deployed assistant, classify each turn (using intent detection or topic models) as in- or out-of-scope, and log per-turn scope verdicts alongside the session id so the drift rate can be monitored continuously. The threshold is less than 5% out-of-scope turns. As a Short-term-priority monitoring control, exceeding 5% should trigger review and corrective action such as tightening scope guardrails, adding redirection or refusal behaviour for off-topic requests, and feeding findings back into prompt or routing design.