Template Generalization

Explanation

Template Generalization checks whether the model's document-extraction quality holds up on layouts it has never seen before, rather than only on the templates it was trained on - essential in banking where new forms, statement formats, and third-party documents constantly appear in Document Extraction and the model must generalise rather than memorise. It is measured by the Template F1 Drop, calculated as F1_seen minus F1_unseen, where F1_seen is the average F1 score on document templates used in training and validation and F1_unseen is the average F1 score on newly introduced, unseen templates. A small drop means the model generalises well; a large drop signals overfitting to familiar layouts. To implement it, hold out a set of genuinely unseen templates, run extraction on both seen and unseen sets, compute the average F1 on each and their difference at evaluation time, and log the results as evidence per release. The thresholds are tiered: an unseen-template F1 of at least 0.90 is the target, performance below 0.85 should be investigated, and below 0.80 should be remediated (proposed) - for example by broadening training data or improving layout-agnostic extraction. Crossing these levels should trigger the corresponding alert and corrective action before new templates are processed in production. It maps to RBI Safety, Resilience and Sustainability, NIST MEASURE 2.7 robustness and security, and ISO Clause 6.1.2 with 9.1 monitoring (all proposed).