Contextual Fairness

Explanation

Contextual fairness checks that a model's recommendations or rankings stay stable when only the surface context changes — for example the order in which options are presented — so that a customer is not advantaged or disadvantaged purely by prompt arrangement. It matters in RAG-driven advisory and comparison flows because positional or ordering artefacts can systematically favour whichever product or candidate appears first, an unfair and non-substantive influence. It is measured by the Positional Bias Score, computed as P(recommended A first) − P(recommended B first): the difference in how often the model recommends item A versus B when their presentation positions are swapped. To implement it, run the same underlying inputs through controlled prompt permutations (reordering options, swapping positions), record which item the model favours in each arrangement, and log the permutation set and outcomes as evidence; track the position-driven swing on a fairness dashboard. Run it pre-deployment and after prompt or model changes. The threshold requires less than 5% output change across prompt permutations; if outputs shift by 5% or more it triggers a breach — investigate the ordering sensitivity, add position-randomisation or normalisation safeguards, re-test, and record remediation, supporting RBI Fairness and Equity and NIST MAP 2.1/4.1.