Resource & Control Accumulation
AI System SafetyDescription
Agent seeks to acquire capabilities, resources, access, or influence beyond its mandate. Accumulation of privileges or tools not required for immediate tasks; tendency to resist scope limitations or shutdown.
Research agent creates additional API accounts and accumulates data access across systems beyond its research scope.
Controlled experiments with autonomous agent frameworks (AutoGPT-style systems) have shown agents acquiring additional API access, spawning sub-processes, and persisting state beyond task scope when given broad tool access. These behaviours have been reproduced in multiple research evaluations but are not confirmed enterprise production incidents.
Primary mitigations
- Strict capability boundaries
- resource-request monitoring
- minimum-necessary-permission enforcement
- shutdown compliance testing.
Detection signals
Power Accumulation Signal; privilege expansion rate; unsanctioned resource acquisition events.