Document AI and intelligent processing for life sciences, measured the way the work is actually consumed.
Every document AI proposal quotes a field-level accuracy figure, and the number is usually honest. It is also the wrong measurement. If a system extracts forty fields from a document at ninety-eight percent accuracy per field, and the errors are roughly independent, then the probability that a given document comes through with every field correct is about forty-five percent. More than half of your documents contain at least one error somewhere, and nothing in the accuracy figure tells you which half. That arithmetic is not a criticism of the technology, which is genuinely good. It is the reason document AI programmes deliver less than the business case promised, and it is almost never done before the contract is signed.
Document AI business cases in this industry follow a pattern. Current state: a person spends eleven minutes transcribing each document. Future state: the system extracts it, somebody glances at it, two minutes. Multiply by volume, calculate the saving, approve the project. The arithmetic is sound and the assumption underneath it is not, because the glance only takes two minutes if the person can trust what they are looking at. Once more than half the documents contain an error they cannot locate without checking, the glance becomes a verification, the verification takes most of the original time, and the saving largely disappears.
What makes this difficult rather than merely disappointing is that the error distribution is uneven and unobvious. Accuracy is not uniform across fields: dates, identifiers and handwritten entries fail far more often than printed headers. It is not uniform across sources either, because one site or supplier submits on a layout the model handles poorly. And it is not stable over time, since a new format arrives and performance on that subset collapses while the overall figure barely moves. A programme measuring one aggregate accuracy number will not see any of this, and will conclude that the technology underperformed when what actually happened was that nobody measured the thing that determined the outcome.
Field accuracy conceals document accuracy
Independent errors compound across fields, and the compounding is where the business case goes.
Accuracy is not uniform across fields
Dates, identifiers and handwriting fail far more often than printed text, and they are usually the important ones.
One source can be much worse than the average
A single site or supplier with an awkward layout can carry most of the error volume.
Review was scoped as a fallback
It is where the throughput actually lands, and it is usually designed last or not at all.
Confidence scores are trusted without calibration
A number from the model is not a probability about your documents until somebody checks that it is.
Backlogs are the stated problem and variability is the real one
A million identical pages is an easy problem. Ten thousand documents in forty layouts is not.
Do the arithmetic before you choose a vendor. Take the accuracy figure you have been quoted, count the fields you actually need from a document, and raise one to the power of the other. That single calculation reframes most proposals in this category, and it is the question nobody asks in the demonstration. Then ask the more useful follow-up: what is the accuracy on the five fields that matter most, what proportion of documents are perfect on those five, and how was that measured. A vendor who can answer those questions specifically is one worth talking to.
A document AI system is an extraction engine and a review workflow, and the second one determines whether the first one saves anybody time. Designing it first sounds backwards and it is the sequence that produces a working economic case rather than a disappointed one.
Step 1
Establish what the fields are actually worth
Step 2
Measure the variability, not the volume
Step 3
Build the labelled sample before anything else
Step 4
Measure per document as well as per field
Step 5
Calibrate confidence against that sample
Step 6
Design the review experience for speed of verification
Step 7
Set the routing thresholds against the economics
Step 8
Integrate into the system of record properly
Step 9
Validate proportionate to what the data does
Step 10
Monitor by source and by field group
Build the labelled sample before you evaluate anything. A few hundred real documents with verified correct values for every field, drawn deliberately from your worst sources as well as your best, is the asset this entire discipline rests on. It is how accuracy becomes a measurement rather than a claim, how confidence gets calibrated, how two vendors can be compared honestly, how a model change is assessed, and how degradation is detected later. It takes a few weeks of unexciting work, it is almost always skipped, and the programmes that have one are the programmes that know what they are getting.
Capture and understand
Ingestion across scanned, photographed, native digital, email and system-generated documents
Image quality handling: skew, rotation, contrast, resolution and the phone photograph taken at an angle
Text recognition across print, degraded scans and handwriting, with the limits of each stated honestly
Document classification, including the case where one file contains several documents that need splitting
Layout and structure understanding, so a table remains a table and a form field keeps its label
Multimodal reading where meaning sits in the layout, a figure, a signature block or a marginal annotation
Extract and verify
Field extraction with per-field confidence, returned at a granularity the review workflow can route on
Validation against format, range, reference data and the other fields on the same document
Cross-document reconciliation, where the same fact appears in several places and should agree
Reconciliation against the system of record, which catches errors no document-internal check can
Exception classification, so a reviewer knows why something was flagged rather than only that it was
Confidence calibration per document type, revisited after every model or format change
Review and deliver
A review interface built for verification speed, with the source visible and the field highlighted in place
Routing by calibrated confidence and by field importance, since those are two different reasons to review
Queue management across volume, priority and deadline, which is an operational problem as much as a technical one
Integration into LIMS, CTMS, eTMF, EDC, RIM, QMS and ERP with the source linkage preserved
Verification status and attribution carried into the record, so the receiving system knows what was checked
The original document preserved and reachable, since structure never captures the annotations, signatures and layout that explain it
Feedback capture from corrections, which is the cheapest improvement signal available and usually unused
The third column is what determines effort, and it is rarely the volume. Scoping from it produces a realistic estimate; scoping from page counts produces the business case that disappoints.
| Stream | What arrives | What makes it difficult |
|---|---|---|
| Safety case intake | Reports from many channels and formats | Free text, handwriting, and every field is consequential |
| Clinical source documents | Site records, lab reports, charts | Site-by-site variability, scans, personal data throughout |
| Regulatory correspondence | Agency letters, commitments, submissions | Dense PDFs where the obligation sits in one sentence |
| Quality records | Deviations, batch records, logbooks | Handwritten entries that change the meaning of printed text |
| Laboratory output | Instrument reports, certificates of analysis | Many layouts from many suppliers, numbers that must be exact |
| Supplier and vendor documents | Specifications, certificates, qualification packs | No control over format, and formats change without notice |
| Study documentation | Protocols, amendments, plans, approvals | Long documents where structure carries meaning |
Document processing is a pipeline and a pipeline multiplies its stage reliabilities. A failure early is unrecoverable later, which is why capture quality matters more than model selection and why the first stage deserves the attention usually given to the fourth.
| Stage | What it does | What a failure here causes |
|---|---|---|
| Capture | Image acquisition, quality correction, text recognition | Everything downstream works on text that was never there |
| Classify and split | Identify document type, separate bundled files | Correct extraction applied to the wrong kind of document |
| Extract | Locate and read the required fields | The visible failure, and usually the smallest contributor |
| Validate | Format, range, cross-field and cross-source checks | Errors that validation should have caught reach a human or a record |
| Deliver | Write to the system of record with linkage and status | Correct data arriving without its provenance, which is a compliance gap |
Reconcile against what you already know. The most effective validation available is almost always external to the document. Does this subject identifier exist in the study. Does this lot number match a batch that was manufactured. Does this product code resolve. Does this date fall inside the study period. Does this value agree with what the same supplier sent last month. Checks of this kind catch errors that no amount of internal format validation will find, they are cheap to implement because the reference data already exists in your systems, and they are routinely omitted because the pipeline was built as a document problem rather than as an enterprise one.
When a system reads a value from a document and writes it into a record the quality system controls, that is the creation of regulated data by an automated process, and the ALCOA+ expectations attach to it the way they attach to anything else. This is workable and it has implications that are easy to miss when the activity is framed as document processing rather than as record creation. Applicability and validation strategy are determinations for your quality and regulatory functions; we build the controls and the evidence to whatever they set.
Frameworks the controls may need to support
- GxP and GAMP 5, with the assurance approach proportionate to what the extracted data actually does
- 21 CFR Part 11 and EU Annex 11 where the resulting records fall within scope
- ICH GCP where the documents are clinical source material, with the obligations that carries
- HIPAA and GDPR where documents contain personal data, which in this domain is most of them
ALCOA+ applied to extraction
- Attributable, which requires knowing whether a human verified a value and which human it was
- Distinct records for extraction, correction and approval, rather than one undifferentiated system action
- Legible and contemporaneous, with the extraction event recorded when it happened
- Original, preserved as the source document with a durable link from every value derived from it
- Accurate, which is the measured figure rather than the vendor figure, per field group
Traceability
- Every extracted value linked to its source document, page and region
- Model and configuration version recorded at extraction, since both change the output
- Confidence at extraction retained, so a routing decision can be reconstructed later
- Correction history preserved, showing the original extraction alongside the verified value
Validation
- Scoped to what the data does, so a field driving a regulated decision is treated differently from one that fills a report
- Performance established on your documents using the labelled sample, not on a vendor benchmark
- Human verification documented as part of the control rather than as a mitigation added afterwards
- Re-evaluation on model change, format change and new source, since all three alter performance
Everything that goes wrong in a document AI system goes wrong in a subset: one source, one format, one field type, one language, one scanner. The overall accuracy figure moves by a fraction of a percent and nothing alerts anybody. Monitoring that is not segmented is monitoring that will find out from a complaint.
| Failure | How it presents |
|---|---|
| Correct value, wrong sample | Every character read correctly and attached to the wrong subject, batch or specimen |
| A table row or column shifts | The value is real and belongs to the line above it |
| A unit is dropped | A number arrives without the unit that makes it interpretable, and looks complete |
| A handwritten correction is ignored | The printed value is captured and the authorised amendment beside it is not |
| Extraction is partial and reports success | Required fields are missing and the document is marked processed |
| A source changes its template | Extraction on that source degrades sharply and the overall figure barely moves |
| High confidence on a wrong field | The score was never calibrated, so the threshold let it straight through |
| Two documents disagree | The system picks one and no exception is raised |
| A reviewer correction is lost | The verified value does not reach the record, or the history of it does not |
| Delivery drops the provenance | Correct values arrive in the record with nothing linking them to a document |
Document AI programmes are reported as pages processed and accuracy achieved. The outcomes that matter are how much human time the work now takes end to end, including review, and whether anybody can show where a value came from.
| Category | What we measure | Why it matters |
|---|---|---|
| Document-level accuracy | Proportion of documents with every required field correct | The number the business case actually depends on |
| Accuracy on the fields that matter | Measured separately for the fields that drive decisions | The aggregate hides exactly the fields you care about |
| Total human time per document | Including review, correction and exception handling | The only honest comparison with the manual process |
| Straight-through rate at safe thresholds | Proportion auto-processed, with error rate on that population | Throughput is only meaningful alongside what it let through |
| Confidence calibration currency | When each document type was last calibrated | An uncalibrated threshold is an unmanaged risk setting |
| Provenance completeness | Values in records with source, page, version and verification status | What converts an extracted number into a defensible one |
| Performance by source | Accuracy per submitting site, supplier or channel, trended | Where degradation appears first and where remediation pays |
We will measure what you are actually getting at document level, build the labelled sample that makes every later number real, calibrate the thresholds that determine both your workload and your error rate, design the review experience that decides the economics, and carry provenance all the way into your systems.
CaliberFocus is a life sciences AI partner delivering document AI and intelligent processing measured the way the work is actually consumed. From a labelled sample of your real documents and document-level accuracy measurement to calibrated confidence, a review interface built for verification speed and provenance carried into every system of record, we design the review queue as part of the system so the business case holds.
- Accuracy measured per document, not only per field
- Confidence calibrated against your own labelled sample
- Review queue designed for verification speed
- Every extracted value linked back to its source
Security & Compliance
