Discriminatory output
Fairness & BiasDescription
Generated guidance, explanations, or decisions vary by protected attribute, producing unfair or unlawful treatment.
Identical loan queries differing only by name/gender yield systematically less favourable language.
The New York Department of Financial Services opened a formal investigation into Apple Card / Goldman Sachs in 2019 after documented gender-disparate credit limit decisions. Amazon scrapped its AI recruiting tool in 2018 after internal audits confirmed it systematically downgraded resumes from women. Both are confirmed production deployments with regulatory or internal investigation outcomes confirming systematic bias against protected groups.
Primary mitigations
- Counterfactual fairness testing
- protected-attribute redaction
- fairness guardrails
- bias-bounded prompting
- human review.
Detection signals
Statistical parity / equal-opportunity gaps; counterfactual output testing; complaint-pattern analysis.