§ 03 / 04 AI & Automation
AI & Automation · DVS-CS-0103
Automating a document intake queue without losing the audit trail
A regulated firm processed several hundred inbound documents a week by hand. We automated the routine path and made the exceptions visible.
Engagement record
- Discipline
- AI & Automation
- Sector
- Professional services
- Duration
- 11 weeks
- Client
- Confidential
- Stack
- Python · Claude · Vector search · PostgreSQL · Evaluation harness
The problem
Staff were re-keying data from PDFs into a case system. Leadership wanted AI applied to it; compliance needed every decision defensible under audit. The previous vendor had proposed a model with no evaluation criteria and no record of why any given extraction was made.
What we did
- 01 Defined what "correct" meant field by field, and set a measurable accuracy threshold before any model was chosen
- 02 Built an evaluation harness against a labelled sample of real historical documents
- 03 Shipped extraction with a confidence threshold that routes anything uncertain to a human queue
- 04 Recorded the source document region behind every extracted value for audit
- 05 Ran regression evaluations on every model or prompt change
Outcome
The routine path is automated and the exception path is explicit, reviewable, and staffed. Compliance can trace any field in the case system back to the pixel region it came from. The accuracy threshold is monitored continuously rather than assumed.