LowSelf-harm/Violence●Realized

Violent or self-harm content

Description

The model generates content that encourages violence or self-harm.

Example scenario

A distressed customer in a chat receives an unsafe response instead of a helpline referral.

Real-world evidence●Realized

The Apple Card gender-bias complaint (2019) triggered a formal NYDFS investigation into differential credit limits driven by an algorithmic model, and Reuters reported Amazon scrapped an AI hiring tool in 2018 after it systemically downgraded women. Both are confirmed production compliance failures of algorithmic scoring systems.

Apple Card algorithm investigated for gender bias in credit limitsNew York Department of Financial Services · 2019 Amazon scraps secret AI recruiting tool that showed bias against womenReuters · 2018

Primary mitigations

Crisis-content classifiers
safe-completion with helpline redirection
human escalation
strict refusals.

Detection signals

Self-harm/violence benchmark scoring; escalation-trigger monitoring.

Mitigating controls

Non-agentic controls

ZYC-SEC-001Non-Agentic

Adversarial Testing

Security

ZYC-ETH-001Non-Agentic

Non-Maleficence

Ethical Governance

ZYC-ACCU-001Non-Agentic

Lexical Similarity

Accuracy

ZYC-LEGAL-001Non-Agentic

AI Regulation Compliance

Legal Compliance

Risk scoring

Severity

High

score 3

Likelihood

Low

score 1

Risk

Low

Control depth

4 mapped (below avg 5.2)

Framework refs

5/8 taxonomies

Framework mappings

Regulatory4/5

EU AI Act: Annex III Sec. 5(b) (AI systems for creditworthiness assessment and credit scoring on natural persons are explicitly listed as high-risk — this is the most directly applicable provision); Art. 9 (mandatory risk management system covering accuracy, bias, and explainability risks); Art. 10 (data governance — training data must be examined for biases that could lead to discriminatory credit decisions); Art. 13 (transparency — deployers and affected persons must receive sufficient information to understand AI-assisted credit decisions); Art. 14 (human oversight — human review of AI credit decisions must be meaningful and effective); Art. 15 (accuracy, robustness); Art. 17 (quality management system).
DORA: — Not covered by DORA. Credit model explainability, fairness, and accuracy are prudential/consumer protection obligations (CRD, Consumer Credit Directive, EBA Guidelines) outside DORA's ICT operational resilience scope.
RBI: Safety, Resilience and Sustainability
NIST AI RMF: MEASURE 2.6
ISO/IEC 42001: Clause 6.2

Risk taxonomies3/4

MIT AI Risk Repository: Domain 1 – Discrimination & toxicity
IBM AI Risk Atlas: Harmful output
NIST ARIA: Dangerous/Violent/Hateful Content
MAESTRO: — Not mapped

Security2/4

OWASP: — Not covered by OWASP LLM 2025
MITRE ATLAS: — Not covered by MITRE ATLAS
Cisco AI Risk: Safety taxonomy: Self-harm
Palo Alto AI Risk: Prisma AIRS: Toxic Content

Capability‑pattern applicability

Classification

GenAI‑specific

Capability pattern

Content-generation

Pattern scope

Content

Why

Generative-AI risk realised without agency/tool-use — applies to chat, RAG, summarization, classification, and content-generation deployments.

Ownership & threshold

Owner (RACI)

Responsible AI Lead [A]

Threshold

Unsafe-content rate = 0; safety-classifier recall ≥ 0.95

Inherent vs residual

Inherent

Low

Residual

1.8

Low

Mapped controls: 4
Assumed control effectiveness: 40%

Modeled, not measured — design-stage; controls not yet tested (reduction capped 40%). Residual applies a uniform assumed reduction to the inherent score; it is not evidence of tested control performance.

Search the Atlas

Violent or self-harm content

Description

Primary mitigations

Detection signals

Mitigating controls

Related risks in Content Safety & Integrity