Document Intelligence Agent

The brief advanced

You've almost certainly watched this happen. Someone on the finance team has a PDF invoice open on one screen and the ERP open on the other, and they are literally reading a number off the left and typing it into the right. Multiply that by a few hundred documents a week and you have a full-time job that is pure transcription — slow, mind-numbing, and quietly generating typos that surface as reconciliation problems weeks later.

The right way to picture what you're building is a very careful, very fast data-entry clerk who never gets bored — but with one rule the human clerk never had: when it isn't sure, it raises its hand instead of guessing. That single discipline is what separates a document-intelligence agent worth deploying from a demo that looks great until it silently fat-fingers a total into your books.

Why a project like this earns a proper brief — more than a chattier assistant would — is that its value is measurable and boring: hours of manual entry converted to seconds, at a known accuracy, with a known escape hatch. When you write the brief, name the document types, the handful of fields that actually matter, and above all the cost of getting one wrong. That last number is what justifies the human-in-the-loop machinery in every section below. The common mistake is to pitch this as "AI reads your documents" and skip the cost-of-error conversation — then the first wrong invoice that reaches accounting becomes your incident instead of a designed-for, expected outcome.

"Stop paying people to retype documents into our systems." Finance keys invoices into the ERP; legal pulls terms out of contracts; ops processes forms. It's slow, expensive, and error-prone. An agent that reads the document and emits clean, validated fields — flagging anything it's unsure about — turns hours into seconds while keeping a human on the risky bits.

1 · Discovery — where does the time go? advanced

Here is the trap that sinks half of these projects. Someone gets excited, points the model at the messiest, most ambiguous scans in the pile — faded thermal receipts, handwritten annotations, a form nobody has a template for — and declares the whole idea unworkable when it stumbles. They spent all their leverage on the 5% of documents that are hard even for a human, and ignored the 80% that are dead simple and being retyped by hand every single day.

Discovery is the deliberate act of resisting that trap. Think of it like triage in an emergency room: you don't treat patients in the order they walked in, you sort by where attention pays off most. The question isn't "what's the hardest document we get?" — it's "where does the time actually go, and where does the agent add the most value per hour saved?" Usually the answer is the boring, high-volume, well-structured documents; the exotic ones are a rounding error you route to a person.

The table below is that triage laid out. Read it as a ranking, not a checklist: the ⭐⭐⭐ rows are where you start, because that's where transcription hours pile up and where a model that generalizes across layouts beats a brittle rules-based OCR pipeline. The ⭐ row — judgment calls on bad scans — isn't a failure to fix; it's designed to go to a human, and admitting that up front is what keeps the project honest.

2 · Architecture advanced

The instinct with anything labelled "agent" is to reach for a loop. You imagine the model reasoning, calling a tool, looking at the result, calling another — a little autonomous worker chipping away at the document. For extraction, that instinct is usually wrong, and following it makes the system slower, costlier, and much harder to test than it needs to be.

The cleaner mental model is an assembly line, not a thinker. A document rides down a conveyor through a fixed sequence of stations — get the text out, hand it to the model to fill a schema, run code to check it — and at the very end a gate decides which of two bins it drops into: straight-through to the system, or over to a human. Each station does one job and passes the work along; nothing loops back. That linearity is a feature: a straight pipeline is something you can reason about, unit-test station by station, and reason about the cost of.

Why it matters that the model call is exactly one box in the middle: everything before it (extracting text) and everything after it (validating, routing) is ordinary deterministic code — no AI, no randomness, fully testable for free. You've quarantined the probabilistic part to a single step and wrapped it in checks. The common mistake is to blur that boundary — letting the model also decide whether to submit, or having it "double-check" its own arithmetic — which hands the safety-critical decisions back to the least predictable component. Keep the fork in code; let the model only fill the form.

Often not an agent loop at all — frequently a single well-structured call per document (Tier 1). Text comes out of the PDF (native text, OCR, or vision for scans), the LLM extracts into a Pydantic schema, code validates, and confidence routes each result to auto-ingest or a human review queue.

3 · Risk & safety model advanced

Picture the worst thing this system can do. It's not a crash — a crash is loud, someone notices, someone fixes it. The worst thing is far quieter: the agent reads $1,240.00 as $12,400.00, its confidence looks fine, and that wrong number flows straight into accounting, gets paid, and nobody spots it until a reconciliation a month later. A confidently-wrong extraction that no human ever looked at is the failure mode that gets projects like this shut down.

So the safety model here is built around a single inversion of instinct: being right or abstaining beats being confidently wrong. A smoke detector that occasionally chirps at burnt toast is annoying; one that stays silent during a real fire is lethal. You want your agent tuned like the good smoke detector — quick to raise its hand, because a document sent to a human by mistake costs a minute, while a wrong one sent to the ERP costs an incident.

The table maps each concrete risk to the control that catches it, and you'll notice they layer rather than compete. Schema constraints stop malformed shapes at the door; per-field confidence plus validation rules stop silently-wrong values; requiring a source span stops invented numbers; a PII policy stops sensitive data leaking through the review UI. No single control is the answer — the safety comes from stacking cheap deterministic checks so that anything uncertain, malformed, or unsupported is routed to a person before it can be committed, never after.

4 · The extraction schema — the heart of the project advanced

If you've ever asked a model to "extract the invoice details" and gotten back a chatty paragraph — or JSON that was almost-but-not-quite the shape your database expected — you've felt why this section exists. Free-text prompting for structured data is a coin flip on every field name, every date format, every "oh, this one used a comma instead of a period." Downstream code that expects total: float and receives "total": "one hundred and ten dollars" doesn't gracefully degrade; it explodes.

A schema flips that relationship. Think of it as a fill-in-the-blank form handed to the model rather than a blank page: instead of "tell me about this invoice," you say "put a number here, one of exactly these three currency codes there, and a value between 0 and 1 in that box." The model isn't composing prose anymore — it's populating slots you defined, and anything that doesn't fit the slots is rejected mechanically before it can travel further.

This is why the schema, not the prompt, is the real specification of the project. The precise point: a well-designed schema makes an entire class of errors unrepresentable — a currency outside the allowed set literally cannot come back, a confidence of 2.0 literally cannot come back. The common mistake is treating the schema as a loose suggestion and doing the real validation afterward in hand-written if statements; that scatters your guarantees and lets shape-errors slip through. Put the constraints in the types, and the model is forced to hand you something already half-checked. The lab below builds exactly that, field by field.

The schema is the spec. Define exactly the fields you need, with types and constraints, and the model must fill it (or you validate and reject).

invoice schema (Pydantic)from pydantic import BaseModel, Field
from typing import Literal, Optional

class LineItem(BaseModel):
    description: str
    quantity: float
    unit_price: float
    amount: float

class Invoice(BaseModel):
    invoice_number: str
    vendor: str
    date: str                          # ISO 8601
    currency: Literal["USD","EUR","GBP"]
    line_items: list[LineItem]
    subtotal: float
    tax: float
    total: float
    confidence: float = Field(ge=0, le=1)
    needs_review: bool                 # the escalation flag

5 · Tool surface (often none — or a few) advanced

Coming off other agent projects, you might expect a long list of tools here — search this, call that, look up the other. Document intelligence is refreshingly different, and the surprise is worth sitting with: most of the time the "tool surface" is nearly empty, because the work is one model call bracketed by plain code, not a toolbelt the model reaches into mid-thought.

The useful way to think about a tool is that every one you add is a door you cut into the wall of your system — convenient, but each door is something that can be walked through in a way you didn't intend, and something you now have to guard. So the discipline is to add the fewest doors that do the job. Getting text out of a PDF, running OCR, checking totals in code — none of those need to be "tools" the model decides to call; they're just steps in the pipeline. The only thing that genuinely earns a guarded door is the one action with consequences: writing a record into the real system.

That's why the table sorts capabilities by risk, and why submit_to_system is the one marked gated. The precise point: read-only and deterministic operations (extract, OCR, validate, look up a vendor name) are cheap and safe to expose; the single write is the one that must never fire on a low-confidence or rule-failing result. The common mistake is to reflexively wrap everything in an agent loop because the project is called an "agent" — but a loop only pays for itself when a document genuinely needs multi-step reasoning (cross-referencing pages, chasing a lookup). For the ordinary invoice, one structured call is cheaper, faster, and far easier to test.

Many document projects need no agent loop — a single structured call per document is enough (and cheaper, faster, more testable). Add a loop only if the doc requires multi-step reasoning (e.g. cross-referencing pages or looking things up).

6 · Evaluation expert

On most agent projects, evaluation is the part everyone dreads — you're grading free-flowing prose, arguing about whether an answer was "good enough," standing up an LLM-as-judge and then wondering whether to trust the judge. Extraction is the happy exception, and it's the reason this project is one of the easiest to make genuinely production-grade: the right answer is a fact, not an opinion.

Because the output is structured and there's a ground truth, evaluating it is like marking a multiple-choice test against an answer key rather than grading an essay. Did invoice_number come back INV-2026-0455, yes or no? Exact-match, per field, deterministic — cheap, fast, and unarguable. That's a luxury most of the course's evals don't have, so lean into it: a golden set of ~30 carefully-labelled documents lets you measure per-field accuracy to a precise percentage.

But field accuracy alone would let a dangerous system pass, and this is the point the table circles: the metric that actually protects you is abstention correctness — when the agent flagged needs_review, was it genuinely uncertain, and crucially, did it flag the ones it got wrong? A model that's 98% accurate but silently confident on the 2% it botches is worse than one that's 95% accurate and reliably raises its hand on the rest. So you track two numbers in tension: the straight-through rate (the business win — how many docs skip review) and abstention correctness (the safety floor). Push the first without watching the second and you've optimized your way straight back into the confidently-wrong failure the whole design exists to prevent.

7 · Phased rollout expert

The tempting move is to flip the switch and let the agent auto-ingest everything on day one. Resist it. You have no golden set yet, no proven accuracy, and no earned trust from the finance team whose numbers you're about to touch — turning on full automation now is asking people to bet their books on a system that has never been measured. The first wrong auto-committed invoice would poison the project's reputation before it ever showed its value.

A phased rollout is how you earn autonomy instead of assuming it, and the mechanism is a dial, not a switch. Think of a self-driving feature that starts by only suggesting lane changes with a human's hands on the wheel, then takes over more as the data proves it safe. Here the dial is the confidence threshold: start with it effectively at "review everything," and lower it — widening the band of documents that auto-ingest — only as your evals earn the right.

That's exactly what the rungs below describe. Phase 1 pre-fills fields but sends every document to a human — which is still faster than typing from scratch, and it quietly builds the labelled golden set you need to measure anything. Phase 2 lets the confident, rule-passing extractions flow straight through while the rest go to review. Phase 3 widens coverage to more layouts as accuracy proves out. And the final rung is a line you never cross: a low-confidence or rule-failing extraction is never auto-committed, at any phase, no matter how good the numbers look. The phases move the threshold; they never remove the floor.

Skills & course map expert

Build-along plan expert

1. Define the schema (Ch 2 + P4): the exact fields you need, with types, confidence, and needs_review.
2. Extract from a text doc: feed a plain-text invoice, get a validated object out with messages.parse().
3. Add PDF/scan input: handle a real PDF (native text, then a scanned image via vision/OCR).
4. Add a validator (P4): totals must reconcile; failure sets needs_review=True.
5. Evals (Ch 5): label ~30 docs; measure per-field accuracy and abstention correctness; tune the confidence threshold.
6. Harden (Ch 6): PII policy, the review-queue UI, and a gated submit_to_system for the confident ones.

What you'll build expert

An extractor that turns an invoice into a validated Invoice object, catches arithmetic errors automatically, and sends anything uncertain to a human — never silently entering a wrong number. Often no agent loop at all: one structured call per document.

Step 1 · The schema + validator (the heart of it) expert

agent/schemas.pyfrom pydantic import BaseModel, Field, model_validator

class Invoice(BaseModel):
    invoice_number: str
    vendor: str
    date: str
    currency: Literal["USD", "EUR", "GBP"]
    line_items: list[LineItem] = Field(default_factory=list)
    subtotal: float
    tax: float
    total: float
    confidence: float = Field(ge=0, le=1)
    needs_review: bool = False

    @model_validator(mode="after")
    def check_totals(self):
        if abs((self.subtotal + self.tax) - self.total) > 0.01:
            self.needs_review = True      # flag, don't reject — a human checks
        return self

Step 2 · Sample documents expert

Step 3 · Extract + route expert

agent/engine.pydef extract(text):
    inv = client.messages.parse(model=MODEL, max_tokens=800, system=SYSTEM,
        output_format=Invoice,
        messages=[{"role":"user","content":f"Invoice document:\n{text}"}]).parsed_output
    if inv.confidence < CONFIDENCE_THRESHOLD:
        inv.needs_review = True            # low confidence also -> review
    return inv

def route(inv):
    return "REVIEW QUEUE" if inv.needs_review else "AUTO-INGEST"

agent/engine.pydef extract(text):
    inv = client.responses.parse(model=MODEL, max_output_tokens=800, instructions=SYSTEM,
        text_format=Invoice,
        input=f"Invoice document:\n{text}").output_parsed
    if inv.confidence < CONFIDENCE_THRESHOLD:
        inv.needs_review = True            # low confidence also -> review
    return inv

def route(inv):
    return "REVIEW QUEUE" if inv.needs_review else "AUTO-INGEST"

terminalpython agent/engine.py       # needs key

invoice_good.txt: INV-2026-0455 total=110.0 conf=0.96 -> AUTO-INGEST
invoice_bad.txt:  INV-777 total=90.0 conf=0.74 -> REVIEW QUEUE
   (flagged: subtotal 70.0 + tax 7.0 != total 90.0, or low confidence)

Step 4 · Extending to real PDFs & scans expert

real PDF (concept)# base64-encode the PDF and send it as a document block (Ch 1)
messages=[{"role":"user", "content": [
    {"type":"document", "source":{"type":"base64",
        "media_type":"application/pdf", "data": b64_pdf}},
    {"type":"text", "text":"Extract this invoice."},
]}
# same output_format=Invoice, same validator, same routing

5 · Tests (no key — the validator is the safety net) expert

The safety net is plain arithmetic, not the model — so the tests run offline, with no API key. A test invoice hits the validator, and its needs_review flag decides whether it lands in the pass bin or the flagged bin. The diagram shows that guard.

terminalpython -m pytest tests/ -v

test_reconciling_invoice_is_not_flagged PASSED
test_bad_totals_are_flagged_for_review PASSED
test_rounding_is_tolerated PASSED
test_currency_must_be_valid PASSED
4 passed

6 · Evals (needs key) expert

Evals measure the model's output against known-correct answers, so this step needs a key. A golden set of labelled documents is run through extraction, each result is scored per field and on its needs_review flag, and a threshold gate decides pass or fail — with the bad invoice's flag as a hard-fail. The diagram traces that pipeline.

terminalpython evals.py

good invoice field accuracy: 3/3
good invoice needs_review: False (expect False)
bad invoice needs_review:  True (expect True)
✅ evals passed

Troubleshooting expert

from pydantic import BaseModel, Field
from typing import Literal

class LineItem(BaseModel):
    description: str
    quantity: float
    unit_price: float
    amount: float

class Invoice(BaseModel):
    invoice_number: str
    vendor: str
    date: str
    currency: Literal["USD", "EUR", "GBP"]
    line_items: list[LineItem]
    subtotal: float
    tax: float
    total: float
    confidence: float = Field(ge=0.0, le=1.0)
    needs_review: bool = False

inv = Invoice(invoice_number="A-1", vendor="Acme", date="2026-01-01",
              currency="USD", line_items=[LineItem(description="Widget",
              quantity=2, unit_price=5.0, amount=10.0)],
              subtotal=10.0, tax=0.8, total=10.8, confidence=0.95)
print(inv.total, inv.needs_review)   # 10.8 False

from pydantic import model_validator

class Invoice(Invoice):   # extend the Step-1 model
    @model_validator(mode="after")
    def check_totals(self):
        line_sum = round(sum(li.amount for li in self.line_items), 2)
        if abs(line_sum - self.subtotal) > 0.01:
            raise ValueError(f"line items {line_sum} != subtotal {self.subtotal}")
        if abs(self.subtotal + self.tax - self.total) > 0.01:
            raise ValueError("subtotal + tax != total")
        return self

try:
    Invoice(invoice_number="B-2", vendor="X", date="2026-01-02", currency="USD",
            line_items=[LineItem(description="a", quantity=1, unit_price=1, amount=1)],
            subtotal=1.0, tax=0.1, total=99.0, confidence=0.9)
except ValueError as e:
    print("rejected:", e)      # rejected: subtotal + tax != total

CONFIDENCE_THRESHOLD = 0.85

def extract(text):
    # --- needs API key: schema-constrained extraction ---
    # r = client.messages.parse(model="claude-opus-4-8", max_tokens=800,
    #     system="Extract invoice fields. Set confidence honestly.",
    #     output_format=Invoice,
    #     messages=[{"role":"user","content":text}])
    # inv = r.parsed_output
    # OpenAI equivalent:
    # r = client.responses.parse(model="gpt-5.5", max_output_tokens=800,
    #     instructions="Extract invoice fields. Set confidence honestly.",
    #     text_format=Invoice, input=text)
    # inv = r.output_parsed
    inv = _fake_extract(text)                      # offline stand-in
    if inv.confidence < CONFIDENCE_THRESHOLD:
        inv.needs_review = True
    return inv

def route(inv):
    return "REVIEW QUEUE" if inv.needs_review else "AUTO-INGEST"

def _fake_extract(text):
    conf = 0.95 if "TOTAL" in text.upper() else 0.6   # deterministic for tests
    return Invoice(invoice_number="A-1", vendor="Acme", date="2026-01-01",
        currency="USD", line_items=[LineItem(description="w", quantity=1,
        unit_price=10, amount=10)], subtotal=10, tax=0, total=10, confidence=conf)

print(route(extract("clear TOTAL 10")))      # AUTO-INGEST
print(route(extract("blurry scan")))         # REVIEW QUEUE

def build_content(doc):
    kind = doc["kind"]
    if kind == "text":
        return [{"type": "text", "text": doc["text"]}]
    if kind == "pdf":
        # --- needs API key: real document block (base64 PDF) ---
        return [{"type": "document",
                 "source": {"type": "base64", "media_type": "application/pdf",
                            "data": doc["b64"]}}]
    if kind == "scan":
        return [{"type": "image",
                 "source": {"type": "base64", "media_type": "image/png",
                            "data": doc["b64"]}}]
    raise ValueError(f"unknown input kind: {kind}")

# offline: verify the dispatcher builds the right block shape
for d in [{"kind":"text","text":"INV 1"},
          {"kind":"scan","b64":"iVBOR..."}]:
    blocks = build_content(d)
    print(d["kind"], "->", blocks[0]["type"])
# text -> text
# scan -> image

GOLD = [
    {"text": "clear TOTAL 10", "should_review": False, "total": 10},
    {"text": "blurry scan",    "should_review": True,  "total": 10},
]

def evaluate(gold):
    field_hits = auto = missed_review = 0
    for g in gold:
        inv = extract(g["text"])
        field_hits += (abs(inv.total - g["total"]) < 0.01)
        if not inv.needs_review:
            auto += 1
            if g["should_review"]:
                missed_review += 1        # a bad invoice auto-ingested = critical
    return {"field_acc": field_hits/len(gold),
            "straight_through": auto/len(gold),
            "missed_review": missed_review}

m = evaluate(GOLD)
print(m)
assert m["missed_review"] == 0, "CRITICAL: bad invoice auto-ingested"
print("eval passed")

from datetime import datetime

VENDOR_FIXUPS = {}

def fixup(vendor):
    def deco(fn): VENDOR_FIXUPS[vendor] = fn; return fn
    return deco

@fixup("Globex")
def _globex(inv):
    # Globex writes dates DD/MM/YYYY; normalise to ISO
    d = datetime.strptime(inv.date, "%d/%m/%Y")
    inv.date = d.strftime("%Y-%m-%d")
    return inv

def extract_with_fixups(text, vendor):
    inv = extract(text)
    fn = VENDOR_FIXUPS.get(vendor)
    return fn(inv) if fn else inv

inv = extract("clear TOTAL 10")
inv.date = "31/12/2026"
print(_globex(inv).date)     # 2026-12-31

Knowledge check check yourself