Document classification, extraction, validation and exception handling for banking operations, measured per field, weighted by consequence, and cross-checked against the systems that already hold the answer.
A document with forty fields extracted at ninety-five percent accuracy per field is a document that is almost never entirely correct. The errors compound, and they land wherever they land.
Document AI is bought on a single accuracy percentage. That number is almost always an average across fields, across document types and across input quality, and averaging is what makes it useless.
Two things are hidden inside it, and both decide whether the automation works.
Accuracy is quoted per document, not per field
Which hides both the compounding and the consequence.
A value was read correctly into the wrong field
Which the confidence score reports as a confident success.
Every field is treated as equally important
So an account number and a salutation get the same threshold.
Confidence is produced and then ignored
The model knew it was unsure and nothing acted on it.
Nothing is cross-checked against the core
When the bank already holds the correct value for several fields.
Tables are read as text
So a figure arrives without its row and column meaning.
Input quality is accepted as given
When fixing capture would raise the ceiling more than the model.
The exception queue arrives unsorted
So a seventy percent automation rate moved the work rather than reducing it.
Take one document type and measure accuracy field by field. Pick a hundred real documents of one type, including the bad ones, and measure extraction accuracy for each field separately rather than as a document average. Then mark each field with what it is used for: a decision input, a stored record, or a convenience. The pattern is consistent across institutions. A handful of fields are both consequential and hard, most are easy, and the overall percentage everybody was quoting describes none of them. That measurement takes a few days and it is the only sound basis for a threshold, a business case or a vendor comparison.
The pipeline is not the interesting part and it is well understood. What determines whether it works is the field taxonomy, the thresholds and the validation, and those are decided with your operations and risk functions rather than configured.
Step 1
Assess the workflow and where the documents actually arrive
Step 2
Define the field taxonomy and what each field is used for
Step 3
Establish the validation sources before designing extraction
Step 4
Design classification first, since everything downstream depends on it
Step 5
Design extraction per document type rather than one pipeline for all
Step 6
Handle tables as structure, not as text
Step 7
Set confidence thresholds per field with your risk function
Step 8
Design the exception queue as a product, not a remainder
Step 9
Integrate to the systems of record for validation and for output
Step 10
Deploy in parallel against human output to get a real baseline
Step 11
Monitor per field, per document type and per channel, and improve both
Split your exception volume by channel before blaming the model. Take a month of exceptions and attribute each one to the channel the document arrived through: branch scan, customer upload, email, post, partner feed. In most institutions the distribution is strikingly uneven, and one channel generates a disproportionate share. That finding redirects the work from model tuning to capture, which is faster, cheaper and permanent. It is also a question nobody asks, because exceptions are reported by document type rather than by origin.
Grouped by stage. The first group is commodity and the later groups are where the outcome is decided, which is the opposite of how these capabilities are usually weighted in a proposal.
Read the document
Optical character recognition across print, poor scans and handwriting, with quality assessed and reported rather than assumed
Document classification, including the long tail of variants and a defined behaviour for a type it does not recognise
Intelligent extraction per document type, with a confidence value per field
Table recognition that preserves rows, columns, headers, merged cells and figures split across pages
Decide whether to trust it
Validation against systems of record, against other documents in the pack, against arithmetic and against format rules
Reconciliation across a document set, where the same value appears in several places and the disagreement is the finding
Confidence thresholds per field, owned by your risk and operations functions, with field-level routing rather than document-level
Provenance on every extracted value: document, page and region, so a reviewer checks in seconds
Handle what is left
Exception handling designed as a work product: what failed, what was extracted, what to check and why it stopped
Review interfaces that show the value beside the region of the page it came from
Straight-through processing for the cases that clear validation, at a rate your risk function has set
Feedback capture, so a correction becomes training data and a recurring correction becomes a defect
Document comparison, identifying what differs between versions of an agreement or between a submitted record and the one on file
The third column is the point of the table. The extraction problem is broadly similar across these. What differs is where the validation comes from, and that is what sets the achievable automation rate.
| Estate | The documents | What sets the automation rate |
|---|---|---|
| KYC | Identity, address and status documents | Validation against the core and against vendors |
| Customer onboarding | Forms, proofs, declarations, mandates | Capture quality at the channel of submission |
| Loan processing | Applications, income proof, bank statements | Cross-document agreement on a few key figures |
| Mortgages | Valuations, titles, conditions, insurance | A long tail of formats from many third parties |
| Financial statements | Balance sheets, income statements, schedules | Table extraction, and arithmetic as validation |
| Trade finance | Letters of credit, invoices, bills, certificates | Consistency checks across the document set |
| Insurance claims | Claim forms, invoices, reports, evidence | Structured fields validated against policy data |
| Compliance documents | Attestations, filings, regulatory correspondence | Classification accuracy more than extraction |
Document AI is unusual in that the techniques available differ enormously in cost and in explainability, and the expensive one is not better at everything. A pipeline that applies a language model to every field is slower, dearer and harder to defend than one that escalates.
Optical character recognition and layout analysis, which establish what is on the page and where, and are the foundation everything else depends on
Template and positional extraction for documents that genuinely are consistent, which is fast, deterministic and trivially explainable
Computer vision for layout, tables, signatures, stamps and checkbox states, which is where structure is recovered
Trained extraction models for semi-structured documents, where position varies and the field set does not
Natural language processing and large language models for genuinely unstructured content, free text, qualifiers and interpretation
Business rules and validation on top of all of it, which is the layer doing the most work for the least money
Recognition and vision: OCR engines, layout analysis, table structure recognition, signature and stamp detection
Models: trained extraction models, natural language processing, large language models for unstructured content
Document AI platforms, both cloud services and specialist products, selected per document estate
Orchestration: workflow engines, business rules systems, exception queue and review tooling
Integration: enterprise APIs, event-driven processing, core banking, lending, payments and document management platforms
Ask what happens to a document type the classifier has never seen. Send an unusual but legitimate document through the pipeline, a variant from a small third party or an old format still in circulation, and watch what happens. Three outcomes are possible: it is correctly identified as unrecognised and routed, it is misclassified and extracted against the wrong schema, or it is processed with confident nonsense. The second and third are common and both produce silent errors in a system of record. A classifier with no honest unknown state is the most under-tested part of most document pipelines.
Which regulations apply to a given use case, how it is classified, and what model governance is required are determined by your compliance, legal and risk functions, based on your operating markets. We do not interpret those requirements and this page does not state which apply to you.
Technology supports a compliance programme. It does not independently establish compliance, and no extraction pipeline makes an institution compliant.
Traceability
- The source document retained and linked to every value derived from it, for the retention period your records function sets
- Provenance per value: document, version, page and region, stored rather than reconstructed
- Extraction model and version recorded against each processed document, so a result can be attributed to a configuration
- Audit trails over processing, corrections, overrides and reprocessing, that a user cannot alter
Privacy and access
- Customer data privacy and minimisation, including what is extracted at all, since an unnecessary field is unnecessary exposure
- Role-based access to documents and extracted data, resolved per user, with identity documents frequently needing tighter control than the file they support
- Encryption in transit and at rest with key ownership stated, and controlled handling of any third-party processing
- Data residency established before build, including where documents are processed and whether any leave the jurisdiction
Human oversight
- Confidence thresholds per field owned by your risk and operations functions, with the trade-off between automation and error made explicit
- Review recorded as a workflow step with the reviewer, what they saw and what they changed, rather than as an edit to a value
- A defined behaviour for an unrecognised document type, which is to route it rather than to process it against the closest schema
- No determination affecting a customer made from extracted data alone, with the accountable function unchanged
Model and resilience
- Model risk management, with extraction models in the inventory and classified, as our AI Agents page describes
- Evaluation against a labelled reference set per document type, rerun on every model or pipeline change
- Bias and fairness assessment where document handling could affect customer treatment differently across groups, with the method set by your risk function
- Operational resilience including a manual fallback that has been exercised, since document volumes do not pause for an outage
Aggregate document AI reporting is the problem this page opens with, and it reappears in monitoring. An overall accuracy figure that holds steady can conceal a collapse in one document type or one channel.
| Failure | How it presents |
|---|---|
| One document type degraded | Aggregate accuracy holds and one estate is failing |
| A third party changed their layout | Accuracy drops on one source with no change on your side |
| Capture quality fell on one channel | Exceptions rise and the model is blamed |
| Thresholds drifted toward automation | Straight-through rate improved and correction rate rose |
| Correction rate rising on one field | The model is wrong there and nobody has looked |
| Unknown types processed anyway | Confident extraction against the wrong schema |
| Validation source became unavailable | Validation silently skipped and values accepted |
| Corrections not fed back | The same error is corrected by hand every week |
| Cost per document rising | An expensive technique is being applied where a cheap one would do |
The outcomes below follow from measuring per field, validating against what the institution already knows, and designing the exception as a work product.
Processing time reduced, measured per document type rather than as a blended average
Errors reduced where they matter, because the fields with consequence are the ones that are validated and thresholded
Cost per document lowered, including by using the cheapest sufficient technique per field rather than one model everywhere
Productivity improved for the reviewers, since a useful exception takes a fraction of the time an unsorted one does
Compliance support strengthened, with the source document retained and every value traceable to its page
An automation rate your risk function has set and can defend, which is the outcome that actually survives review
We will benchmark one document type field by field, identify which fields can be validated against your systems, propose thresholds by consequence, and tell you the automation rate that follows. You get the benchmark whether or not we build anything.
CaliberFocus builds document processing for banks that measures accuracy per field, weights it by consequence and validates against the systems that already hold the answer. We design classification with an honest unknown state, extract tables as structure, route individual fields rather than whole documents, and trace every value to its page and region.
- Per-field confidence thresholds owned by your risk function
- Validation against systems of record, other documents and arithmetic
- Table extraction that preserves rows, columns, units and periods
- Exceptions delivered as a work product, with what failed and why
Security & Compliance
