Adversarial Testing

Explanation

Adversarial Testing checks how well the system detects and blocks prompt-injection and other adversarial inputs that try to subvert its instructions — for a bank's RAG or chat assistant this guards against attacks where hidden instructions in a document or user message attempt to exfiltrate data, bypass policy, or make the assistant take unauthorised actions. It is measured by the Injection Detection and blocking Score, computed exactly as Detection Score = True Positives / (True Positives + False Negatives) — that is, of all genuine injection attempts, the fraction the system correctly catches (recall). Implement it by maintaining a labelled corpus of known and red-teamed injection attacks, running them regularly against the live detection/guardrail layer, and recording true positives and false negatives to compute the score; instrument the input-handling and pre-generation stages so every blocked and missed attempt is logged with the payload, verdict, and outcome for audit, and refresh the attack set as new techniques emerge. The threshold is a > 95% detection rate: if detection falls to or below 95%, the control is breached, which — given its Immediate priority — should trigger an alert, tightening of the injection filters or guardrails, and re-testing before the gap can be exploited in production.