LowModel Theft◐Demonstrated

Model extraction / theft

Description

Systematic querying or insider access lets attackers replicate or steal a proprietary fine-tuned model.

Example scenario

A competitor reconstructs the bank's fine-tuned underwriting assistant via bulk queries.

Real-world evidence◐Demonstrated

Carlini et al. (2021) empirically extracted verbatim training records from GPT-2, and Shokri et al. (2017) demonstrated membership inference against commercial ML services. These are controlled research demonstrations, not confirmed incidents involving exfiltration of sensitive BFSI customer records from a production system.

Extracting Training Data from Large Language ModelsUSENIX Security 2021 (Carlini et al.) · 2021 Membership Inference Attacks Against Machine Learning ModelsIEEE S&P 2017 (Shokri et al.) · 2017

Primary mitigations

Query rate limiting
watermarking
output throttling
access controls
extraction detection.

Detection signals

Query-pattern anomaly detection; extraction-attack simulation.

Mitigating controls

Non-agentic controls

ZYC-SEC-001Non-Agentic

Adversarial Testing

Security

ZYC-ROB-001Non-Agentic

Out-of-Distribution Detection

Robustness

ZYC-SCV-001Non-Agentic

Third-Party component risk

Supply Chain Vulnerabilities

Risk scoring

Severity

High

score 3

Likelihood

Low

score 1

Risk

Low

Control depth

3 mapped (below avg 5.2)

Framework refs

6/8 taxonomies

Framework mappings

Regulatory5/5

EU AI Act: Art. 10 (data governance — training data minimisation and controls over sensitive data; Art. 10(5) allows sensitive data processing only under strict conditions for bias monitoring); Art. 15(1) (cybersecurity measures must protect against attacks that reconstruct training records or infer membership); Art. 9 (risk management must enumerate privacy inference attack vectors).
DORA: Art. 9(2) (ICT security policies must ensure confidentiality and integrity of data held or processed by financial entity ICT systems — membership inference attacks breach data confidentiality and fall within DORA's ICT threat perimeter); Art. 10 (detection of unauthorised probing of model outputs). Note: the underlying GDPR compliance obligation is outside DORA's scope.
RBI: Safety, Resilience and Sustainability
NIST AI RMF: MANAGE 1.3
ISO/IEC 42001: Annex A

Risk taxonomies3/4

MIT AI Risk Repository: Domain 4 – Malicious actors & misuse
IBM AI Risk Atlas: Model theft
NIST ARIA: Information Security
MAESTRO: — Not mapped

Security3/4

OWASP: LLM03 Supply Chain
MITRE ATLAS: AML.T0048 Exfiltration via ML Inference
Cisco AI Risk: Safety taxonomy: IP compromise
Palo Alto AI Risk: — Not covered by Palo Alto Prisma AIRS

Capability‑pattern applicability

Classification

Foundational

Capability pattern

Foundational

Pattern scope

Pattern-agnostic (GenAI)

Why

Generative-AI risk realised without agency/tool-use — applies to chat, RAG, summarization, classification, and content-generation deployments.

Ownership & threshold

Owner (RACI)

AI Security Lead [A]

Threshold

Attack-success / robustness-failure rate ≤ tolerance; alert on breach

Inherent vs residual

Inherent

Low

Residual

1.8

Low

Mapped controls: 3
Assumed control effectiveness: 40%

Modeled, not measured — design-stage; controls not yet tested (reduction capped 40%). Residual applies a uniform assumed reduction to the inherent score; it is not evidence of tested control performance.

Related risks in Security & Robustness

ZNR-SR-001Critical

Direct prompt injection

ZNR-SR-002Critical

Indirect prompt injection via retrieved content

ZNR-SR-003Critical

Jailbreak / guardrail bypass

ZNR-SR-004High

System-prompt leakage