MediumTraining Data Privacy Violation●Realized
Memorised PII Exposure
Privacy & Data RiskDescription
LLM training data memorisation surfaces confidential, private, or copyrighted data in agent outputs. Model inversion attacks extract training examples; membership inference exposes whether specific data was in the training set.
Example scenario
Agent reproduces verbatim private emails from its training set when queried with partial text matches.
Real-world evidence●Realized
In March 2023, OpenAI confirmed a production bug where ChatGPT exposed conversation titles and payment details of one user to other active users, constituting a confirmed cross-context information flow violation in a live deployment. The Samsung internal code leakage via ChatGPT also demonstrates this at an organizational level.
Primary mitigations
- Differential privacy in training
- training data auditing
- memorisation detection
- model output filtering.
Detection signals
Memorisation rate; training data extraction success rate.
Mitigating controls
2 Dual coverage