HighOver-reliance●Realized

Over-reliance / automation bias

Description

Users over-trust AI outputs and stop applying independent judgement, propagating errors.

Example scenario

Officers rubber-stamp AI summaries without reading the underlying file.

Real-world evidence●Realized

Algorithmic and AI-adjacent system failures causing critical financial infrastructure disruptions are a confirmed pattern, including flash crashes triggered by high-frequency trading algorithms and a single software deployment error destroying a major trading firm. These are documented production incidents with material financial impact.

The 2010 Flash Crash: How Trading Algorithms Destabilized MarketsSEC / CFTC Joint Report · 2010 Knight Capital Group $440M Loss from Algorithmic Trading Software FailureSEC Enforcement Action · 2012

Primary mitigations

Confidence cues & disclaimers
friction on high-impact actions
reviewer training
periodic human-only checks.

Detection signals

Override-rate & blind-acceptance monitoring; decision-audit sampling.

Mitigating controls

Non-agentic controls

ZYC-OREL-001Non-Agentic

Inference with Human Oversight Rate

Over-reliance

ZYC-ETH-001Non-Agentic

Non-Maleficence

Ethical Governance

ZYC-ACCT-001Non-Agentic

Human-in-the-Loop

Accountability & Oversight

ZYC-ETH-002Non-Agentic

Autonomy Preservation

Ethical Governance

ZYC-ETH-003Non-Agentic

Beneficence

Ethical Governance

ZYC-OREL-002Non-Agentic

Factuality

Over-reliance

Risk scoring

Severity

High

score 3

Likelihood

High

score 3

Risk

High

Control depth

6 mapped (above avg 5.2)

Framework refs

4/8 taxonomies

Framework mappings

Regulatory5/5

EU AI Act: Annex III, point 2 (AI systems used as safety components in management and operation of critical digital infrastructure classified as high-risk); Art. 9 (Risk management system mandatory for critical infrastructure AI); Art. 15 (Accuracy, robustness and cybersecurity requirements — resilience against errors, faults and attacks)
DORA: Art. 5 (ICT risk management framework for critical functions); Art. 9 (Protection and prevention of ICT infrastructure); Art. 10 (Detection of anomalous activity); Art. 11 (Response and recovery — business continuity and disaster recovery); Art. 17 (Incident management process); Art. 24-26 (Digital operational resilience testing including threat-led penetration testing for critical systems)
RBI: Conduct and Customer Protection
NIST AI RMF: MEASURE 2.6
ISO/IEC 42001: Clause 9.1

Risk taxonomies3/4

MIT AI Risk Repository: Domain 5 – Human-computer interaction
IBM AI Risk Atlas: Over-reliance
NIST ARIA: Human-AI Configuration
MAESTRO: — Not mapped

Security1/4

OWASP: LLM09 Misinformation
MITRE ATLAS: — Not covered by MITRE ATLAS
Cisco AI Risk: — Not covered by Cisco AI Defense
Palo Alto AI Risk: — Not covered by Palo Alto Prisma AIRS

Capability‑pattern applicability

Classification

GenAI‑specific

Capability pattern

Chat

Pattern scope

Pattern-agnostic (GenAI)

Why

Generative-AI risk realised without agency/tool-use — applies to chat, RAG, summarization, classification, and content-generation deployments.

Ownership & threshold

Owner (RACI)

AI Governance Office [A]

Threshold

Programme KRI within board-approved tolerance

Inherent vs residual

Inherent

High

Residual

5.4

Medium

Mapped controls: 5
Assumed control effectiveness: 40%

Modeled, not measured — design-stage; controls not yet tested (reduction capped 40%). Residual applies a uniform assumed reduction to the inherent score; it is not evidence of tested control performance.

Related risks in Societal & Economic

ZNR-SE-002Medium

Workforce displacement

ZNR-SE-003Medium

Environmental / compute & energy cost

ZNR-SE-004Medium

Erosion of human skill / deskilling

ZNR-SE-005Medium

Accessibility & digital-divide exclusion

Search the Atlas

Over-reliance / automation bias

Description

Primary mitigations

Detection signals

Mitigating controls

Related risks in Societal & Economic