How the test works
What we are testing
Conduit's AI assist will let a requester attach a vendor quote or invoice and have the request pre-filled: the vendor, the document details, and every line item with quantity, unit and price. Before we build it, we test whether a model can do the reading reliably, on the client's real documents, without anything leaving Albert's machine for the reading step.
What would make this worth building
- A local model can read document facts (vendor, number, date, subtotal, tax, total) correctly on at least 9 documents in 10.
- It can list the real line items with at least 95% precision and recall, with quantity, unit and price exactly right.
- It never states, with high confidence, a number the document does not print.
- It finds every value Conduit's forms can use today.
- Where it fails, the failures fall into a small number of fixable patterns rather than random noise.
If any of 1 to 4 still fails after three rounds, it is reported as a known limitation and the build plan must design around it.
How a document moves through the test
After the loop:
Fresh documents every round
Labelling a document carefully from its page images takes time, so each round is scored on a sample, not on all 160. The sample is drawn by a script with a fixed seed, so anyone can reproduce it.
- Equal numbers of quotes, proposals and invoices.
- Scanned documents are always represented, because OCR is where reading is hardest.
- The longest documents are included, because they stress the model the most.
- No vendor appears twice, so one vendor's layout cannot dominate the score.
- Exact duplicates and earlier revisions of the same document are removed first.
- A document used to score one round is never used to score a later round. Changes are made by studying the errors in round N; round N+1 is scored on documents those changes never saw.
Samples drawn so far
What counts as passing
These thresholds were set in the approved plan before any results existed. They do not move.
| Name on site | Threshold | Why this bar |
|---|---|---|
| Documents with every fact correct | at least 90% | Facts pre-fill the vendor and the document details. If more than one document in ten needs a fact corrected, people stop trusting the pre-fill. |
| Line precision (strict) | at least 95% | An extra line would add something to a purchase order that nobody is buying. |
| Line recall (strict) | at least 95% | A missing line would under-order. Strict means the quantity, unit and price must also be exactly right. |
| High-confidence numbers not on the page | zero | In the planned feature, low-confidence values will be flagged 'Check this' for the requester. A wrong number marked high confidence would not be flagged, so none is allowed. |
| Conduit-mappable values found | 100% | These are the values Conduit's forms can use today: vendor phone, email and GST number, payment terms, the customer purchase order number and delivery dates. |
With 16 documents, one document is about 6 points of the facts score. Passing 90% therefore means at most one document with a wrong fact.
Dev checks versus acceptance
While making round 2 changes, the agents re-ran the round 1 sample to see whether each change helped. Those dev checks look much better than round 1, but they are measured on the very documents the changes were designed around. That is like marking your own homework after seeing the answers. They are shown on the results page, clearly marked, and they never fill an acceptance slot. Round 2 passes or fails only on its fresh sample.
Limits of this test
- Labels can be wrong too. The correct answers are written by a model reviewer (Claude Opus) from page images, not by the client's staff. Each correction cites the page it came from. Where a label needed a judgement call, such as whether a zero-priced row is a line item, the rule is written down and applied the same way every round.
- Some field scores are an upper bound. Labels start from the model's output and are corrected where wrong. A field the reviewer did not touch keeps the model's value, so per-field accuracy on matched lines may be slightly overstated. The pass or fail thresholds count whole documents and whole lines, which limits this effect.
- PDFs only. Requesters will often attach phone photos. Photos are not part of this round of testing.
- Small samples. 16 to 24 documents per round gives a clear signal on large failures, not a precise rate. Results are reported as counts so readers can judge for themselves.
- Three documents are not tested. They were too large for the download connector. They stay out of scoring until full copies are available.
How the client's documents are protected
- The documents are kept only on Albert's computer, outside any code repository, and are never placed on a public server.
- Claude agents downloaded the documents through the shared-folder connector and read them while building and checking the pipeline. They never publish document content.
- The extraction itself runs locally. Documents are not sent to a cloud service for the reading step.
- For scoring, Claude Opus also reviews page images of the sampled documents to write the correct answers.
- This site shows counts, scores and timings only. Documents appear under anonymous labels such as 'Quote 07'. No names, prices, document text or images are published.
- The data file behind this site is produced by a script that only copies approved fields and refuses to publish if any vendor or customer name, id, filename, email, phone number or dollar amount appears in it.