Guardrail Model Benchmark Matrix
Zytra Aval v1.4 and Prism (Guardian+GEPA) versus LlamaGuard 3, Granite Guardian 3.3, WildGuard 7B, and ShieldGemma across 31 datasets — including FinProof BFSI Track 1, prompt injection, and general harm detection benchmarks.
Cross-Model Performance Comparison
Metric per cell: R = Recall · F1 = F1 score · Acc = accuracy on benign set · FPR = false positive rate (lower is better, scale inverted). "—" = not evaluated on this benchmark.
| Dataset | Type | n | Zytra · Aval v1.4Reference Baseline | Prism · ZytraGuardian · 8B | LlamaGuard 3Meta · 8B | Granite G. 3.3IBM · 8B | WildGuard 7BAllen AI · 7B | ShieldGemmaGoogle · 9B/2B† |
|---|
Sources
¹ Zytra Aval v1.4: Zytra internal evaluation, Oct 2026. F1 metric on harmful sets; Acc on benign sets.
² Semalith v1.4 and LlamaGuard 3-8B comparisons: Addagada, Semalith v1.4, arXiv:2607.22545 (Jul 2026), Tables 4–6. Metric is Recall (R) unless noted as F1.
³ Granite Guardian 3.3-8B (PI/Jailbreak axis): arXiv:2607.22545 §5.5. Metric is Recall (R) unless noted.
⁴ WildGuard 7B, ShieldGemma 2B (HarmBench/BeaverTails): Raj et al., Benchmarking Open-Source Safety Guard Models, arXiv:2605.28830 (ICLR 2026 workshop). F1 metric. Note: these tests used ShieldGemma 2B — BFSI Track 1 tested ShieldGemma 9B.
⁵ BFSI Track 1: FinProof Track 1 Leaderboard, finproof.ai, Oct 2026. B-01–B-07 Tier 4 withheld set (2,608 attacks + 1,345 BFSI benign). BFSI F1 metric.
⁶ MossCap: Semalith paper tests L6/L7/L8 levels (n=3,000 each); values shown are weighted-average recall across levels.
⁷ OR-Bench, JailbreakBench Benign, AgentHarm Benign: low Acc reflects BFSI-calibration over-refusal on out-of-domain general benign prompts — expected for a domain-specialized classifier. AgentHarm Benign FPR for LlamaGuard/Granite from arXiv:2607.22545 (n=208 holdout).
⁸ Totals row (all datasets): Zytra Aval v1.4 weighted avg F1=0.906, Pr=0.905, Rc=0.924, Acc=0.681.
⁹ Prism BFSI F1=0.727 is from the GEPA fine-tuning evaluation on FinProof v1 (13 optimization rounds). Granite base on the same GEPA eval set scored F1=0.540 before fine-tuning; the leaderboard score of 0.813 uses the sandboxed Tier 4 test partition. Aval v1.4 (mmBERT-based) was not part of the GEPA pipeline. GEPA improvements: Recall +20.1 pp, FPR −0.19 pp, AUROC +5.3 pp.
Submit your model to the FinProof leaderboard
13 attack categories, 36 subtypes drawn from real financial services incidents. Sandboxed evaluation, reproducible harness.