Document Intelligence Agent

This is the schema — the exact shape of the data you want back from the model. In this project the schema is the specification: you describe every field once, and the model is forced to fill exactly that structure (or you reject it). This is Pydantic, the same tool from Python P4.

1. class LineItem(BaseModel) describes one row of an invoice — a description, a quantity, a unit price and an amount. Each field has a type (str, float) so a wrong type is rejected automatically.
2. class Invoice(BaseModel) is the whole document. line_items: list[LineItem] means "a list of those rows" — a nested model inside a model.
3. currency: Literal["USD","EUR","GBP"] only allows those three exact strings — anything else fails validation. That's how you stop garbage values at the door.
4. confidence: float = Field(ge=0, le=1) forces the number between 0 and 1 (ge = greater-or-equal, le = less-or-equal). needs_review: bool is the escalation flag — the switch that later sends a document to a human instead of straight into the system.

What the output means: Nothing runs yet — this is a definition. But once it exists, any data that doesn't match (wrong currency, missing field, confidence of 2.0) is rejected before it can reach your database.

Try this: Imagine adding a due_date field. You'd write due_date: str here, and from then on every extracted invoice would be required to include it.

This is the buildable version of the schema, now with a validator — a small piece of code that runs automatically after the fields are filled in, to check they make arithmetic sense. Together the schema and validator are your safety net, and neither one needs a model call.

1. The fields are the same idea as before. line_items: list[LineItem] = Field(default_factory=list) means "if no line items are found, start with an empty list" instead of crashing. needs_review: bool = False defaults to "don't escalate" unless something trips the flag.
2. @model_validator(mode="after") is a decorator that says "run this function after all the fields are set." The function check_totals then gets the finished object as self.
3. if abs((self.subtotal + self.tax) - self.total) > 0.01: checks whether subtotal plus tax actually equals the total, allowing a 1-cent rounding wobble. abs(...) makes the difference positive so the comparison works either way.
4. If the totals don't reconcile, it sets self.needs_review = True — it flags the document rather than throwing it away, because a human should look at it. Then return self hands the (possibly flagged) object back.

What the output means: A valid, reconciling invoice comes out with needs_review=False; one whose numbers don't add up comes out with needs_review=True — automatically, with no AI involved.

Try this: Picture an invoice where subtotal=70, tax=7, total=90. Since 70+7=77, not 90, the difference is 13 > 0.01, so needs_review flips to True. That's the arithmetic guard catching a bad document.

Now the two functions that actually do the work: extract asks the model to fill in the schema from a document's text, and route decides where the result goes. This is the single structured model call the whole project is built around.

1. client.messages.parse(...) is the key line: parse (not the plain create) makes the model return data that matches your schema. output_format=Invoice tells it which shape to use, and .parsed_output hands you back a real Invoice Python object.
2. The messages list contains one user turn: an f-string that pastes the document's text after the label "Invoice document:" so the model knows what it's reading.
3. if inv.confidence < CONFIDENCE_THRESHOLD: is the second safety check. Even if the arithmetic was fine, a model that's unsure (low confidence) gets its needs_review flag set too. So a document is escalated if either the totals fail or confidence is low.
4. route(inv) reads that final flag and returns "REVIEW QUEUE" when needs_review is true, otherwise "AUTO-INGEST". That one line is the fork in the diagram.

What the output means: For the good invoice you'll see total=110.0 conf=0.96 -> AUTO-INGEST; for the bad one, total=90.0 conf=0.74 -> REVIEW QUEUE with a note that the subtotal+tax didn't match the total, or confidence was low.

Try this: Raise CONFIDENCE_THRESHOLD close to 1.0 and re-run — more invoices get sent to review because the bar for auto-ingesting is now stricter.

This shows how to feed a real PDF (or a scan) instead of plain text — and the big lesson is how little changes. Only the input format is different; the schema, validator and routing all stay exactly the same.

1. Instead of a plain string, the message content is now a list of blocks. The first block has "type":"document" and carries the PDF itself.
2. "source":{"type":"base64", "media_type":"application/pdf", "data": b64_pdf} — base64 is a way of turning a binary file into plain text so it can travel inside a JSON request. b64_pdf is your PDF encoded that way.
3. The second block is a normal "type":"text" instruction — "Extract this invoice." — telling the model what to do with the attached document.
4. The comment at the end is the whole point: same output_format=Invoice, same validator, same routing. You swap how the content arrives; every guarantee downstream is untouched.

Try this: For a scanned image instead of a PDF, you'd send an image block the same way. Because the schema is the contract, the input can be text, PDF or image and the rest of your code never notices.

This runs the automated tests — and the exciting part is that they need no API key. Because the validator is plain arithmetic, you can prove the safety net works without ever calling the model, which makes the tests fast, free and deterministic.

1. python -m pytest tests/ -v tells pytest (the standard Python test runner) to run every test in the tests/ folder. -v means "verbose" — print each test name and its result.
2. Each line ending in PASSED is one check that behaved correctly. test_bad_totals_are_flagged_for_review proves the validator catches an invoice whose numbers don't add up — the most important guarantee in the project.
3. test_rounding_is_tolerated confirms the 0.01 tolerance stops the validator from raising false alarms on cent-level rounding, and test_currency_must_be_valid confirms the Literal[...] enum rejects an unknown currency.

What the output means: 4 passed at the bottom means all four checks succeeded. If any failed, pytest would print FAILED with the exact assertion that broke, so you'd know what to fix.

Try this: Temporarily loosen the tolerance in the validator to something huge (say 100) and re-run — test_bad_totals_are_flagged_for_review should now FAIL, showing you the test is genuinely guarding the arithmetic.

This runs the evals — measurements of how good the extraction actually is against known-correct answers. Tests check the code; evals check the model's output quality. This one needs an API key because it makes real extraction calls.

1. python evals.py runs the eval script. It feeds in documents whose correct answers you already know (a "golden set") and compares what the model produced.
2. good invoice field accuracy: 3/3 means all three checked fields matched the labels exactly. Because the output is structured, this is a deterministic exact-match score — the cheapest and most trustworthy kind of eval.
3. The two needs_review lines are the safety eval: the clean invoice must come back False (don't escalate) and the broken one must come back True (do escalate). Getting the bad one wrong would be the dangerous failure.

What the output means: ✅ evals passed means field accuracy hit its target and the bad invoice was correctly flagged. The flagging check is a hard-fail: a bad invoice that slips through un-flagged fails the eval no matter how good the accuracy number is.

Try this: Add a fourth labeled invoice to the golden set and re-run. Growing the golden set is exactly how you build confidence to let more documents auto-ingest without review.