Somewhere in most growing businesses, a person is retyping. Invoice totals into the accounting system. Form fields into a CRM. Statement lines into a spreadsheet. Delivery notes into an inventory record.
It is high volume, repetitive, universally disliked, and it has an unusual property for an AI candidate: there is a correct answer, printed on the document, that anybody can check.
That property is why document extraction succeeds where vaguer AI projects stall. It can be measured, so it can be improved, and a claim about how well it is working can be verified rather than debated.
Why this is a good candidate
- Correctness is objective. The invoice total is either 4,182.60 or it is not.
- Volume is usually high enough for the saving to be measurable within weeks.
- The people doing it now will not defend the task, so adoption is not a negotiation.
- Errors are containable, if the exception path is designed — which is the whole of the difficulty.
- The baseline is easy to establish: how long it takes now, and how often it is wrong now.
That last point is worth acting on before anything is built. Manual data entry has an error rate, and businesses systematically assume it is zero. Measuring it gives you the bar the automation actually has to clear, which is usually lower than the one people imagine they are holding it to.
The part that actually decides the outcome
Extraction accuracy is what gets demonstrated. The exception path is what determines whether the system is usable, and it is almost never designed.
No extraction system is correct on every document. Some will be unclear, badly scanned, in an unexpected format, or from a supplier who changed their template. What happens to those decides everything.
| Design | Result |
|---|---|
| Reject uncertain documents to a queue nobody owns | The queue grows; the process is not automated, it is deferred |
| Accept everything and let errors through | Wrong data in the system, discovered later, trust destroyed |
| Route uncertain cases to a person with the document alongside | Works — and the corrections are training data |
Only the third is a working system. It also requires the thing most implementations omit: the reviewer must see the document and the extracted values together, with the uncertain field highlighted. Reviewing an extraction without the source is not review, it is guessing.
Confidence matters more than accuracy
A headline accuracy figure is close to useless on its own. Ninety-five percent accuracy means one document in twenty is wrong, and if you cannot tell which, every document must be checked — which is the manual process with extra steps.
What makes automation work is the system knowing when it is unsure. A system that is ninety percent accurate and reliably flags its own uncertain cases is far more useful than one that is ninety-five percent accurate and confident about all of it.
- Set confidence thresholds per field rather than per document. A total is worth checking at a much lower threshold than a description.
- Route on field-level uncertainty, so one unclear field sends one field for review rather than the whole document.
- Track how often high-confidence extractions turn out to be wrong. That figure tells you whether the confidence signal is trustworthy, which is more important than the accuracy figure.
- Tune thresholds against the cost of each error type. A wrong bank account number and a wrong description are not comparable.
Validate against what you already know
The strongest accuracy improvement in document automation is usually not a better model. It is checking the extraction against data the business already holds.
- Does the supplier on the invoice exist in the supplier list, and does the bank account match the one on record?
- Do the line items sum to the stated total, and does the tax calculation hold?
- Does the purchase order referenced exist, and do the quantities correspond?
- Is the date plausible, and is this invoice number one already processed?
- Is the amount within the range this supplier normally invoices?
These checks are ordinary arithmetic and lookups, they are deterministic, and they catch errors the extraction was confident about. They also catch something the extraction is not looking for at all: the duplicate invoice, and the invoice whose bank details have changed — which is the most common route for invoice fraud and is caught by a comparison rather than by a model.
That is worth stating directly. Document automation done with validation is a fraud control as well as an efficiency measure, and done without it, it removes the human who would have noticed.
Implementing it
- 1
Start with one document type from a few sources
One supplier's invoices, not all invoices. Narrow scope produces a working result in weeks and reveals the exception patterns early.
- 2
Measure the current process first
Time per document and current error rate. This is the bar, and it is not zero.
- 3
Build the review interface early
Document and extracted values side by side, uncertain fields highlighted, correction in one action.
- 4
Add the validation checks
Against existing data. This is usually a larger accuracy gain than anything model-related and it is deterministic.
- 5
Run in parallel before switching
Extract and compare against what the person entered, without acting on the extraction. This measures real accuracy on real documents at no risk.
- 6
Set the confidence thresholds from that data
Rather than from a default. You will have the evidence by this point.
- 7
Feed corrections back
Every reviewer correction is a labelled example. A system that does not capture them stays at its launch accuracy permanently.
- 8
Monitor the exception rate over time
A rising rate means something changed — a new supplier template, a scanner setting, a format. That signal is the maintenance trigger.
Where this is the wrong approach
- Where the document is already structured data in disguise. If the supplier can send a machine-readable file, ask for it. Extracting from a PDF that was generated from a database is solving a problem that did not need to exist.
- Where an integration is available. A direct feed is more reliable than any extraction, and cheaper to maintain.
- Where volume is low. A dozen documents a month does not repay the build or provide enough data to tune thresholds.
- Where the document requires judgement rather than extraction — assessing whether a claim is valid is not the same problem as reading what it says.
The first is the most commonly missed. A substantial amount of document automation exists to undo a conversion that another business performed on structured data it already had, and one conversation is sometimes cheaper than the whole project.
Is document extraction a good AI project for a business?
Yes, unusually so, because there is an objectively correct answer printed on the document that anyone can check. That makes it measurable, therefore improvable, and any claim about how well it is working can be verified rather than debated. It is also high volume, universally disliked by whoever does it now, and has an easily established baseline.
What decides whether document automation works?
The exception path, not the extraction accuracy. No system is correct on every document, so the design question is what happens to the uncertain ones. Rejecting them to an unowned queue defers the work rather than automating it; accepting everything puts wrong data into systems and destroys trust. The only working design routes uncertain cases to a person who sees the document and the extracted values together with the uncertain field highlighted.
Why does confidence matter more than accuracy?
Because an accuracy figure alone does not tell you which documents are wrong. Ninety-five percent accuracy with no uncertainty signal means every document must be checked, which is the manual process with extra steps. A system that is ninety percent accurate and reliably flags its own uncertain cases is more useful, because the confident cases can pass through unreviewed.
How can extraction accuracy be improved without a better model?
By validating against data the business already holds. Does the supplier exist and do the bank details match the record; do the line items sum to the total; does the referenced purchase order exist; has this invoice number already been processed; is the amount in the supplier's normal range. These are deterministic checks that catch errors the extraction was confident about — and they catch duplicate invoices and changed bank details, which is the most common route for invoice fraud.
When is document automation the wrong solution?
When the document is structured data in disguise. A PDF generated from a supplier's database can often be requested as a machine-readable file instead, and a direct integration is more reliable and cheaper to maintain than any extraction. It is also wrong at low volume, where the build cost is not repaid and there is insufficient data to tune thresholds, and for tasks requiring judgement rather than reading — assessing whether a claim is valid is a different problem from extracting what it says.
Sources and further reading
- Choosing your first AI project— why this shape scores well on the selection criteria
- AI for internal business operations— the wider set of internal applications
- Business process automation: what to automate and what to fix— the ask-for-a-feed-instead test, generalised
- API integration for modern businesses— the direct feed that beats extraction whenever it is available
- AI evaluation: how to know whether it is working— parallel running and threshold tuning as evaluation
- CRM data migration and reconciliation— the validation-against-existing-data pattern in another setting
- OWASP Top 10 for LLM Applications— risks that apply when documents from outside the business are processed
Services This Relates To
Written by KYCONNECTS Engineering. Client names are withheld under confidentiality.