Attack detection is the wrong first question
When a risk committee evaluates an AI guardrail, the instinct is to ask how many jailbreaks it catches. That's the wrong starting question for a banking deployment, because in production the metric that decides whether a guardrail survives contact with real customers is the false-positive rate — how often it blocks someone who was never attacking anything.
So what: if your evaluation scorecard leads with attack recall and buries false-positive rate in a footnote, you're optimizing for the wrong failure mode. Flip the order in your next vendor evaluation.
Scale turns a small percentage into an operational crisis
A retail bank assistant handling a million customer messages a day is, overwhelmingly, handling balance checks, card activation, and payment queries — attacks are rare by comparison. At that volume, a guardrail with a 0.5% false-positive rate wrongly blocks roughly 5,000 customers a day. One with a 30% false-positive rate blocks roughly 300,000.
Put a rupee figure on that: if each wrongly-blocked interaction converts into even a 10% chance of an escalated complaint that a contact-centre agent has to resolve, 300,000 false blocks a day generates tens of thousands of avoidable complaint-handling events — on top of whatever RBI Fair Practices Code exposure comes from customers being denied service without explanation. A 30% false-positive rate doesn't protect the bank. It breaks the product and invites a regulatory grievance pattern in the same motion.
Why generic safety models run hot on financial language
General-purpose safety models are trained to treat language that sounds risky as risky. Financial conversation is full of words that sound risky out of context — "transfer," "account access," "override the limit," "bypass" — and a model that hasn't specifically learned the difference between a fraud attempt and a frustrated customer asking to raise their UPI limit will flag both.
That's not a tuning problem you fix with a stricter threshold. It's a training-data problem: the model has never seen enough legitimate BFSI traffic to learn where the real boundary sits, so it defaults to suspicion on anything that pattern-matches a watchlist of scary-sounding words.
So what: before you shortlist a guardrail vendor, ask what fraction of their training data is actual BFSI conversation versus general open-domain safety data. If they can't answer with a number, assume the answer is "very little."
What a BFSI-tuned classifier changes
Aval is trained specifically on BFSI adversarial and benign traffic, which is what lets it learn the boundary between attack intent and a legitimate financial request rather than reacting to surface vocabulary. In our own benchmarking, Aval runs a 0.5% false-positive rate on the AgentHarm benchmark while holding a 0.994 HackaPrompt R score, at 184M parameters and 11.6ms inference latency — figures we publish so they can be checked against your own traffic rather than taken on faith. By comparison, one widely used open-source guard model posted a 96.9% false-positive rate on AgentHarm in our internal testing — meaning it flagged nearly every real banking query as an attack, which makes its attack-recall numbers irrelevant in practice.
So what: ask any vendor for their false-positive rate on realistic, in-domain BFSI traffic specifically, not a generic benchmark. Anything in double digits is not a production candidate, no matter how good its attack-recall score looks on a slide.
A four-point evaluation checklist before you shortlist a guardrail
- Request the false-positive rate on BFSI-specific traffic — not a generic safety benchmark — and insist on seeing the underlying test set.
- Reject anything with a double-digit false-positive rate outright, regardless of attack-recall performance.
- Confirm the model's training data includes real adversarial and benign financial-services conversation, not just open-domain safety data.
- Test latency under your actual peak concurrent load, not the vendor's lab conditions — a model that's accurate at 11ms and unusable at 200ms under load isn't production-ready.