TMLS Agentic Hackathon · Toronto Tech Week · May 2026
Digitizing invoices for restaurants. Open-source and proprietary models, visual language models, multi-modal pipelines, and an agentic orchestration loop. Built from scratch during the hackathon, starting Monday.
19 preselected invoices, each picked for a distinct failure mode. Four provider columns scored against gold. Live at invoice-agent.ideaplaces.com.
Invoice extraction is solved as OCR. It is not solved as a workflow. Flat-priced vendors charge the same on invoice 1 and invoice 10,000 from the same supplier. Faxed and messy tail invoices still drop accuracy. AP teams spend their time chasing arithmetic, not paying suppliers.
A multi-agent pipeline (Docling layout, VLM extraction, deterministic validators) wired behind a viewer that compares four provider columns against gold, per invoice and per field. You see exactly where each model wins and where it fails.
The shape of what runs in doc-extractor/ for every invoice. The viewer surfaces the outputs (extracted fields with confidences, validator results, per-provider diff against gold).
Identify the supplier and load any memory we have for them.
Pull header fields, line items, and any custom fields the user asked for.
Deterministic checks: line items sum to subtotal, subtotal plus tax equals total, plausible tax rate, valid dates.
Final verdict. Confident or needs review, with failing checks called out so a human can act.
On approval, bank the corrected invoice as a few-shot example. The supplier warms from cold to warming to specialized.
Built in backend/app/memory.py: every corrected invoice banks a few-shot example for that supplier, so the next one from the same supplier extracts more accurately and more cheaply. No open-source project we surveyed has this, and no flat-priced vendor cares to.
First time seen. Generic extraction.
Learning this supplier from your corrections.
Few-shot from banked invoices for this supplier.
Our cost per invoice falls as we learn a supplier. Flat vendors never move.
Five pipeline shapes mapped before picking. The bake-off Thursday picks the production extractor. Diagrams below are the real shapes from EXPERIMENTS.md.
PDF goes through Docling for layout and tables, then a VLM sees both the page image and the parsed text and emits the field schema.
Tradeoff: Baseline today. Strong on clean docs, slower and more expensive than text-only when accuracy matches.
OCR once, then a text-only LLM extracts. Only low-confidence fields trigger a targeted vision re-read on that region.
Tradeoff: Faster and cheaper if accuracy holds. Risks: OCR errors propagate, stylized logos may not read.
Parallel narrow agents per field (distributor, date, invoice number, lines, totals). A thin orchestrator assembles the Invoice.
Tradeoff: Lets us pick a cheap model for easy fields and an expensive one only where it pays. Wins on per-field F1 if specialists beat single-call.
An LLM with a toolkit (search_text, get_table_column, check_arithmetic, lookup_distributor, re_ocr_region) iterates until validators pass.
Tradeoff: Most flexible, most expensive per invoice. Right for the hard ten percent where single-shot returns zero.
Hash the bbox layout. If the fingerprint matches a known template for this distributor, apply the cached field map. If not, fall back and store a new template.
Tradeoff: Layered speed-up, not a replacement. Only kicks in after a few invoices per supplier. Marginal cost falls as the corpus grows.
Four columns against five curated failure modes. Each cell scored by an LLM judge plus deterministic compare. Winner wires behind the same Extractor interface so the frontend never needs to know.
Baseline-good. The bar at 75% on the small Qwen.
Worst case (0%). Distributor confused with a BBQ pit logo.
French-Canadian date format and distributor failure.
Invoice number grabbed a product SKU instead.
Pre-tax vs post-tax confusion on total_extended_price and total_invoice_amount.
Two-minute tour of /inspect. Every step lands a different point on the scoring rubric.
19 preselected docs, each picked for a distinct failure mode. Start with SAQ (pre-tax vs post-tax confusion) or BREWERS (distributor confused with a BBQ pit logo) if you want the hard cases.
Self-hosted pdf.js with a magnifier loupe. Zoom and rotate as needed. The same page the extractor saw.
Default view shows the gold reference: what the correct extraction looks like for this doc. Every field has the right answer plus its source on the page.
Tabs A/B/C/D run Azure GPT-5.4, Qwen-VL 72B, Pixtral Large, and Mistral OCR + LLM. Click any tab to see that provider's prediction diffed against gold, field by field, with the misses highlighted.
Wrong field, wrong format, missing, hallucinated. Each miss carries the category so you can see whether the model grabbed an adjacent value, returned the wrong shape, or invented one.
Line items rendered as a table, full extraction as raw JSON, both per-provider. The numbers behind every claim on this landing live here.
Each criterion is scored 1 to 5 with equal weight. Here is the one line per row.
Real AP pain at FGF scale. Roughly $34k a year saved on a 200k-invoice book once the top 80% of suppliers specialize. Validator catches mis-read totals before they hit the ledger.
Four-agent pipeline (supplier-id, extractor, validator, reviewer) with a memory loop. Deterministic arithmetic and date validators. Extractor sits behind a one-method interface so any provider drops in.
Self-hosted PDF viewer with magnifier loupe, side-by-side gold vs prediction, four provider tabs to flip through, per-field miss categories (wrong-field, format, missing, hallucinated), full JSON and tables panels. Deterministic, never flakes.
Per-supplier memory plus a deterministic arithmetic validator. No open-source project we surveyed has either. Flat-priced vendors do not specialize. See the landscape doc for the full survey.
Deployed at invoice-agent.ideaplaces.com. FastAPI plus Next.js packaged as one Docker image, persisted ports 7625 and 6473. Real extraction pipeline (Docling, Qwen-VL, optional Azure GPT-5.4 vision) lives in doc-extractor and gets wired in behind the same seam.
Most teams skip this section. Stating the tradeoffs out loud is the shipping mindset point.
Each provider tab in /inspect renders a real bake-off output from doc-extractor (real Docling, real VLM, real judge). Deliberately cached so a 5-minute table demo never flakes on a slow API call. The same pipeline runs live on new uploads through the same Extractor interface.
backend/app/memory.py banks corrected invoices and warms suppliers cold to specialized. The UI that exercises it lived in an earlier demo page that we cut to keep the focus on /inspect. The memory module ships as-is; the UI seam is small.
Today the verdict uses one set of thresholds for every supplier. Per-distributor thresholds are next, and registry.py already holds the right anchor for them.
We picked 19 invoices for distinct failure modes, not coverage. After the bake-off picks a winner, the next run is the winner against OmniDocBench for citable cross-comparison numbers.