FinProof Independent Evaluation: Why BFSI AI Safety Benchmarks Must Be Domain-Specific

Why generic benchmarks fail BFSI

The dominant AI safety benchmarks — AgentHarm, HarmBench, WildGuard — were built for general-purpose LLM evaluation. They test for harm categories that matter broadly: CSAM, violence, self-harm, weapons. They do not test for:

• Financial advice violations — giving investment recommendations to a retail customer without a licence
• Regulatory breach — outputting content that violates RBI, SEBI, or MAS requirements
• PII exfiltration in a BFSI context — account numbers, IFSC codes, loan sanction details
• Agentic authorization bypass — an agent acquiring capabilities beyond its declared tool scope
• Confidential data extraction — internal credit policy, pricing floors, underwriting criteria
• Bias in financial decisions — differential treatment across protected class proxies in credit or insurance

A guardrail that scores well on AgentHarm may still allow through every one of these. A bank that cites AgentHarm results to a regulator is citing evidence for the wrong question.

The 68% figure comes from the Zytra Risk Atlas: 167 AI risks catalogued across 21 BFSI governance domains, of which 68 are genuinely novel — without precedent in SR 11-7, NIST AI RMF, or traditional model risk frameworks. Of those 68 novel risks, none appear in any general-purpose benchmark test set. Banks evaluating guardrails on general benchmarks are blind to all of them on the day of deployment.

How FinProof is structured

FinProof is organized into 14 threat categories (B-01 through B-14), each representing a distinct class of adversarial input relevant to BFSI deployments.

B-14 (Benign) is as important as the threat categories. A guardrail with a 10% false-positive rate blocks 1 in 10 legitimate customers. In a bank channel handling 100,000 daily interactions, that is 10,000 unnecessary friction events per day — a customer experience failure and a potential regulatory complaint. FinProof measures FPR as a first-class metric, not an afterthought.

Dataset construction. Each category contains adversarial prompts developed against realistic BFSI deployment contexts: loan officer copilots, customer-facing chat, document extraction pipelines, agentic treasury assistants. Prompts are written in English and Indian vernacular (Hindi, Tamil, Telugu) because linguistic diversity is a security asymmetry. Guardrails trained on English-only data underperform on multilingual adversarial inputs — and India's BFSI customer base is multilingual by default.

Evaluation methodology. Prompts are passed to the guardrail under test in two conditions: isolation (no system prompt context) and in-context (with a realistic BFSI system prompt). Both conditions are reported separately. A guardrail that catches threats in isolation but misses them in-context is not production-safe. Production systems always run with a system prompt.

What the results show

Aval v2.0 (current production): 0.5% FPR on B-14 at 11.4ms P95 latency. Category recall across B-01 through B-13 at or above 94%.

Key findings from the FinProof leaderboard across guardrail vendors evaluated:

• FPR is the differentiator. Most guardrails achieve acceptable recall (>80%) on B-01 (Prompt Injection) and B-02 (Jailbreak). The gap opens at FPR: industry average is above 10%. A 10% FPR is not production-deployable in a regulated bank channel where every blocked interaction creates a customer record.
• B-05 (Regulatory Breach) has the lowest industry recall. Regulatory-specific threat detection requires domain training data — knowledge of what RBI, SEBI, and MAS actually prohibit. General-purpose guardrails do not have it. Average recall on B-05 across non-BFSI-native models: 41%.
• B-09 (Bias / Discrimination) is the least-tested category industry-wide. Most vendors do not evaluate it. It is also the most likely to trigger regulatory action under RBI's MRMF explicit bias-testing requirements and under DPDP Act obligations for automated decisions affecting individuals.
• In-context performance degrades for most guardrails. Adding a realistic BFSI system prompt reduces recall on B-11 (Agentic Authorization) and B-12 (Multi-Agent Exploitation) by an average of 18 percentage points for non-BFSI-native models. Aval was trained in-context against BFSI deployment patterns.
• Vernacular adversarial inputs defeat non-India-trained models. Hindi and Tamil adversarial prompts achieve a 2.3× higher bypass rate on general-purpose guardrails compared to equivalent English prompts. The same attack, in a different script, becomes invisible to a model that has never seen it.

What this means for BFSI teams

Three practical uses for FinProof results:

1. Vendor evaluation. Ask any guardrail vendor for their FinProof B-01 through B-14 scores, separately for isolation and in-context conditions, with FPR reported at their default threshold. A vendor who cannot produce these numbers has not been evaluated on BFSI-relevant threats. Accepting a vendor's internal benchmark results — without independent evaluation against a domain-specific test set — is the model risk equivalent of accepting self-reported validation.

2. Regulatory evidence. RBI's draft MRMF requires independent validation for AI models in production. FinProof results — published methodology, reproducible test set, third-party evaluation — constitute the kind of documented evidence MRM committees can cite. A guardrail cited only by its vendor's internal benchmarks does not satisfy independent validation requirements. FinProof gives compliance teams something to put in front of a regulator.

3. Production monitoring. FinProof categories map directly to Aval's runtime threat classification. A bank that deploys Aval and monitors B-category incident rates has a continuous bridge between pre-deployment benchmark performance and production behaviour — which is what "ongoing monitoring" means in the MRMF context. The pre-deployment score sets the baseline; production monitoring tracks drift.

Methodology and access

FinProof methodology is published at research.html. The leaderboard is updated quarterly as new model versions are released or submitted for evaluation.

Banks and NBFC risk teams can:

• Access the published leaderboard and methodology at research.html
• Request the full test set under NDA for internal red-team use against deployed systems
• Cite FinProof results in MRM committee submissions as evidence of independent evaluation
• Submit proprietary guardrail models for evaluation and inclusion on the leaderboard

The purpose of a benchmark is to give a decision-maker a defensible basis for comparison. AgentHarm gives that basis for general-purpose harm. FinProof gives it for BFSI. A bank that buys a guardrail without BFSI benchmark results is accepting a governance gap on the day of deployment — before the first adversarial customer interaction, before the first MRM committee review, before the first regulatory inspection.