MediumNon-determinism◐Demonstrated
Inconsistency / non-determinism
Model Accuracy & ReliabilityDescription
Identical inputs yield materially different answers across runs, undermining reproducibility and auditability.
Example scenario
Two officers get conflicting policy interpretations from the same query, creating inconsistent customer treatment.
Real-world evidence◐Demonstrated
Knowledge-cutoff staleness in deployed LLMs is a technically confirmed and publicly discussed limitation — users and testers have routinely elicited confidently stated outdated regulatory, market, and legal information from production chatbots. However, a specific documented incident where stale AI financial or market knowledge in production caused quantifiable client harm with regulatory follow-up has not been publicly confirmed.
Primary mitigations
- Temperature/seed control for decisioning
- response caching
- deterministic modes
- versioned prompts
- output logging.
Detection signals
Run-to-run variance tests; reproducibility audits on identical prompts.
Mitigating controls
6 Non-agentic controls
ZYC-ACCU-001Non-Agentic
Lexical Similarity
Accuracy
ZYC-XAI-001Non-Agentic
Field Justification
Explainability (XAI)
ZYC-ROB-001Non-Agentic
Out-of-Distribution Detection
Robustness
ZYC-ACCU-003Non-Agentic
Lexical Similarity
Accuracy
ZYC-ACCU-011Non-Agentic
Normalization Consistency
Accuracy
ZYC-ROB-002Non-Agentic
Input Pertubation
Robustness