Abuse & misuse detection

Explanation

Abuse and misuse detection checks whether the system can reliably recognise harmful, toxic or abusive content — whether arriving from users trying to misuse a banking chatbot or generated by the model itself — so it can be blocked before causing harm or reputational damage. It is measured by a Misuse detection score based on a binary classification of toxicity, where Toxicity = 1 if the text contains toxic content and 0 if the text is non-toxic; the control then evaluates how well the detector catches truly toxic content. To implement and operate it, place a content-classification layer on both inputs and outputs of RAG and chat capabilities, train or tune it on relevant abuse, toxicity and misuse categories, run a labelled evaluation set to measure detection performance, and log every flagged item, the classifier's decision and the action taken (block, mask, escalate) as evidence; periodically re-evaluate against fresh adversarial examples. The threshold is detection recall ≥ 0.95 with a false-negative alert, meaning the detector must catch at least 95% of genuinely toxic content; recall falling below 0.95 — i.e. too many toxic items slipping through as false negatives — is an immediate-priority breach that triggers an alert, retraining or rule-tightening, and review of any harmful content that reached users.