HighSystem Prompt Leakage◐Demonstrated

System-prompt leakage

Description

The model discloses its system prompt, revealing instructions, secrets, or guardrail logic that enable further attacks.

Example scenario

An attacker extracts the system prompt and learns exactly how to bypass the bank's controls.

Real-world evidence◐Demonstrated

Tramèr et al. (2016) formally demonstrated model extraction against production ML APIs (BigML, Amazon ML) by querying to reconstruct functionally equivalent models. No confirmed large-scale theft of a BFSI production model has been publicly documented, keeping this at Demonstrated rather than Realized.

Stealing Machine Learning Models via Prediction APIsUSENIX Security 2016 (Tramèr et al.) · 2016 Knockoff Nets: Stealing Functionality of Black-Box ModelsIEEE CVPR 2019 (Orekondy et al.) · 2019

Primary mitigations

No secrets in prompts
prompt-leak resistant design
output filters for prompt content
secrets in vault not prompt.

Detection signals

Prompt-extraction testing; system-content-in-output detection.

Mitigating controls

Non-agentic controls

ZYC-SEC-001Non-Agentic

Adversarial Testing

Security

ZYC-ROB-001Non-Agentic

Out-of-Distribution Detection

Robustness

ZYC-SCV-001Non-Agentic

Third-Party component risk

Supply Chain Vulnerabilities

Risk scoring

Severity

High

score 3

Likelihood

High

score 3

Risk

High

Control depth

3 mapped (below avg 5.2)

Framework refs

7/8 taxonomies

Framework mappings

Regulatory5/5

EU AI Act: Art. 9 (risk management system must cover cybersecurity threats including model extraction); Art. 15(1) (providers of high-risk AI must ensure robustness and cybersecurity against adversarial attacks seeking to alter outputs or reconstruct the model); Art. 11 (technical documentation must address security measures). Applies where system falls under Annex III high-risk categories.
DORA: Art. 8 (identification and classification of ICT risks — systematic query attacks constitute an identifiable ICT threat); Art. 9(2) (ICT security policies must protect against unauthorised access and data/asset exfiltration); Art. 10 (detection mechanisms for anomalous query patterns indicative of model extraction activity).
RBI: Safety, Resilience and Sustainability
NIST AI RMF: MANAGE 2.1
ISO/IEC 42001: Annex A

Risk taxonomies3/4

MIT AI Risk Repository: Domain 2 – Privacy & security
IBM AI Risk Atlas: Prompt leaking
NIST ARIA: Information Security
MAESTRO: — Not mapped

Security4/4

OWASP: LLM07 System Prompt Leakage
MITRE ATLAS: AML.T0056
Cisco AI Risk: AI Defense: Sensitive-data guardrail
Palo Alto AI Risk: Prisma AIRS: Sensitive Data Leak

Capability‑pattern applicability

Classification

GenAI‑specific

Capability pattern

Chat

Pattern scope

Pattern-agnostic (GenAI)

Why

Generative-AI risk realised without agency/tool-use — applies to chat, RAG, summarization, classification, and content-generation deployments.

Ownership & threshold

Owner (RACI)

AI Security Lead [A]

Threshold

Attack-success / robustness-failure rate ≤ tolerance; alert on breach

Inherent vs residual

Inherent

High

Residual

5.4

Medium

Mapped controls: 3
Assumed control effectiveness: 40%

Modeled, not measured — design-stage; controls not yet tested (reduction capped 40%). Residual applies a uniform assumed reduction to the inherent score; it is not evidence of tested control performance.

Related risks in Security & Robustness

ZNR-SR-001Critical

Direct prompt injection

ZNR-SR-002Critical

Indirect prompt injection via retrieved content

ZNR-SR-003Critical

Jailbreak / guardrail bypass

ZNR-SR-005Medium

Adversarial / evasion inputs

Search the Atlas

System-prompt leakage

Description

Primary mitigations

Detection signals

Mitigating controls

Related risks in Security & Robustness