Normalization Consistency

Explanation

Normalization consistency checks whether values that a document-extraction pipeline pulls out are converted into a single, uniform format every time — so a date is always YYYY-MM-DD, an amount always has the same currency symbol and decimal handling, a PAN or IFSC code always follows its canonical pattern. This matters in BFSI because downstream systems (core banking, KYC, reconciliation) silently break or misjudge records when the same field arrives in mixed shapes. It is measured by the Format Consistency Score, computed as Consistent_Values / Total_Values: of all values extracted for a field, the fraction that conform to the expected normalized format. Implement it by defining canonical format rules per field, running a validator immediately after extraction and normalization in the pipeline, tagging each value as consistent or not, and logging the per-field score, sample offending values, and the source document for evidence. Operate it as a continuous monitor against the target of greater-than-or-equal-to 0.90; scores below 0.85 should be investigated to find the drifting source or format rule, and scores below 0.80 trigger remediation such as fixing parsers or quarantining affected batches before they reach systems of record.