Critical Action Risk
Tool & Affordance RiskDescription
Agent invokes high-impact tools (payment systems, databases, communication channels, infrastructure APIs) without proportional authorisation. The blast radius of a single tool misuse extends far beyond agent scope.
Coding agent uses database admin credentials to 'optimise performance' by dropping indexes on production tables.
Controlled research has shown that popular post-hoc explanation methods (LIME, SHAP, saliency maps) do not faithfully reproduce the model's internal computation and can pass sanity checks even when applied to random models. This has been published but has not been the direct attributed cause of a documented production harm incident.
Primary mitigations
- Risk-tiered tool authorisation
- mandatory human approval for critical-tier tools
- impact scoring before tool invocation.
Detection signals
Critical Action Risk Score; high-impact tool invocation frequency.