Input Pertubation

Explanation

Input Perturbation checks how stable the model's predictions are when its inputs are nudged slightly - testing whether tiny, immaterial changes to a query or document (rephrasing, added whitespace, minor noise) cause disproportionately large swings in the output. For banking RAG and Summarization & Chat, this matters because a robust model should give consistent answers to equivalent inputs; high sensitivity signals fragility that adversaries or simple data variation could exploit. It is measured by the Prediction Sensitivity Index, calculated as PSI = Avg(f(x) - f(x+Sigma)), the average difference between the model's output on an original input x and on a perturbed input x+Sigma (the original plus a small perturbation). A value near zero means the model is stable under small changes; a larger value means small input changes produce large output changes. To implement it, generate controlled perturbations of a representative input set, run both clean and perturbed inputs through the model at evaluation time, compute the average output difference, and log the index per run as evidence. This is an Immediate-priority control. No fixed numeric threshold is given - set a target per the control objective and alert on breach (proposed); a breach should trigger investigation and hardening such as augmentation or smoothing. It maps to RBI Safety, Resilience and Sustainability, NIST MAP 2.1 and 4.1, and ISO Clauses 6.2.1 and 6.1.2.