HighOver-Reliance / Automation Bias●Realized

Human Oversight Erosion

Description

Humans over-trust agent outputs, reducing critical scrutiny. Automation bias causes humans to defer to agent recommendations even when wrong; systemic erosion of human oversight capability and skill atrophy.

Example scenario

Medical professionals approve 97% of AI diagnostic recommendations without independent review, losing ability to catch systematic AI errors.

Real-world evidence●Realized

Confirmed production incidents include Samsung engineers inadvertently inputting proprietary source code and meeting notes into ChatGPT in 2023, and multiple reports of chatbots reproducing PII from training data or context windows in live deployments. These constitute real-world sensitive data exposure via model output.

Samsung employees accidentally leaked company secrets by using ChatGPTBusiness Insider · 2023 Extracting Training Data from Large Language ModelsUSENIX Security 2021 (Carlini et al.) · 2021

Primary mitigations

Confidence calibration displays
mandatory human verification workflows
skill maintenance programmes
automation bias training
decision diversity requirements.

Detection signals

Human override rate of agent recommendations; critical review frequency.

Mitigating controls

Dual coverage

ZYC-HOV-03Agentic

Trust calibration and automation bias mitigation

Human Oversight

ZYC-ETH-001Non-Agentic

Non-Maleficence

Ethical Governance

ZYC-ROB-003Non-Agentic

Hallucination Frequency

Robustness

ZYC-ACCT-001Non-Agentic

Human-in-the-Loop

Accountability & Oversight

Risk scoring

Severity

High

score 3

Likelihood

High

score 3

Risk

High

Control depth

4 mapped (below avg 5.2)

Framework refs

6/8 taxonomies

Framework mappings

Regulatory5/5

EU AI Act: Art. 10(5) data minimisation and privacy-by-design obligations for AI data processing; Art. 9(2)(b) risk management for inadvertent disclosure of personal or confidential data in outputs; Art. 26(1)(d) deployer obligations to prevent personal data exposure through AI outputs; Art. 13(1)-(3) transparency requirements including disclosure of data handling practices
DORA: Art. 17(1)-(3) ICT incident classification — sensitive data leakage in AI outputs may constitute a major ICT-related incident; Art. 19(1) reporting obligation for major ICT incidents involving data exposure; Art. 6(2)(f) data protection obligations within ICT risk management framework
RBI: People First
NIST AI RMF: GOVERN 1.1: Legal/regulatory requirements; MAP 5.1: Impact assessment
ISO/IEC 42001: Clause 6.1.2: AI risk assessment criteria; Clause 5.2: AI policy

Risk taxonomies4/4

MIT AI Risk Repository: Domain 5 – HCI
IBM AI Risk Atlas: Over-Reliance on AI Agents; Impact on Human Agency
NIST ARIA: Human-AI Configuration
MAESTRO: L5 – Observability

Security2/4

OWASP: ASI09
MITRE ATLAS: AML.TA0008
Cisco AI Risk: — Not covered by Cisco AI Defense
Palo Alto AI Risk: — Not covered by Palo Alto Prisma AIRS

Capability‑pattern applicability

Classification

Foundational (inherited)

Agentic pattern

Foundational (inherited)

Capability pattern

Document Extraction; RAG; Summarization & Chat

Pattern scope

Pattern-agnostic (machinery)

Agent loop stage

Output

Why

Agent-machinery risk — realised in any deployment pattern; filter primarily by Agentic Pattern. [proposed]

Ownership & threshold

Owner (RACI)

AI Governance Office [A]

Threshold

Programme KRI within board-approved tolerance

Inherent vs residual

Inherent

High

Residual

5.4

Medium

Mapped controls: 4
Assumed control effectiveness: 40%

Modeled, not measured — design-stage; controls not yet tested (reduction capped 40%). Residual applies a uniform assumed reduction to the inherent score; it is not evidence of tested control performance.

Related risks in Societal & Economic Risk

ZYR-SE-002High

Labour Market Disruption

ZYR-SE-003Medium

Compute-Driven Carbon Footprint

ZYR-SE-004High

AI-Enabled Authority Centralisation

ZYR-SE-005High

AI-Generated Information Manipulation

Search the Atlas

Human Oversight Erosion

Description

Primary mitigations

Detection signals

Mitigating controls

Related risks in Societal & Economic Risk