Coding & Code-Review Agent

Before writing any logic, we declare the exact shape of the data we want back from the model. These are Pydantic models — think of each as a form with typed blanks. If the model returns something that doesn't fit the form, Pydantic rejects it, so bad data never flows downstream. This is the project's contract.

1. class Finding(BaseModel) is one reported problem. Each field is a labelled blank with a type: file: str, line: int, an issue description, and a suggested_fix.
2. severity: Literal["low", "medium", "high"] means severity must be one of exactly those three words — nothing else is accepted. That's how we stop the model inventing junk categories.
3. confidence: float = Field(ge=0, le=1) is a number the model must keep between 0 and 1 (ge = greater-or-equal, le = less-or-equal) — a 0-to-1 sureness score.
4. class Review bundles a plain-English summary with a findings list; default_factory=list just means "start empty if there are none". class Verdict is the verifier's yes/no answer later: is_real plus a reason.

What the output means: Nothing prints — this file only defines the shapes. Their payoff shows up when .parse(...) forces the model's reply into these exact forms.

Try this: Imagine the model returns severity: "urgent". Because it isn't in the Literal list, Pydantic raises a ValidationError — the schema catching bad data for you, before it can cause a bug.

To build and test a review agent you need something for it to review. Rather than wire up a live repo, this file hard-codes a small diff (the text showing what a PR changed) with three bugs deliberately planted in it — so we always know the right answer to check the agent against.

1. DIFF is a multi-line string (the '''...''' triple quotes let it span lines). Lines starting with + are newly added code — exactly what a real git diff shows.
2. The # BUG: comments mark the three planted problems: charging with no positive-amount check, a TypeError from adding a float to a string, and (further down) using a value that might be None.
3. KNOWN_ISSUES = ["amount", "TypeError", "None"] is the answer key: three short keywords, one per planted bug. Step 6's eval later checks the agent's findings mention each of these.

What the output means: This file just holds data — the diff to review and the list of bugs it contains. In production this text would come from git diff or the PR API instead.

Try this: Add a fourth planted bug to DIFF and a matching keyword to KNOWN_ISSUES. Now the eval expects the agent to catch four — a quick way to make the test harder.

This is the first of two model calls: the review pass. Its job is to cast a wide net — surface every problem it suspects. We deliberately tell it to over-report, because a second, stricter pass (Step 4) will throw out the false alarms. Recall first, precision later.

1. REVIEW_SYSTEM is the system prompt — the reviewer's standing instructions. Notice it literally says "Report everything you suspect — a separate verification step will filter", setting up the two-pass design.
2. def review_pr(diff): takes the diff text and returns findings. It calls client.messages.parse(...) — the .parse method (not plain .create) forces the reply into a shape.
3. output_format=Review is the key line: it tells the model "answer as a Review object" (the schema from Step 1), so you get a summary and a typed list of findings, not free-form prose.
4. .parsed_output at the end hands back that validated Review object directly — ready to loop over, no manual parsing.

What the output means: A Review object: a one-line change summary plus a list of Findings (file, line, severity, issue, fix) — likely including some over-eager guesses that the next step will prune.

Try this: Read the two-sentence REVIEW_SYSTEM aloud. That single instruction — "report everything, something else will filter" — is the whole philosophy of the review half of the pipeline.

This is the second model call — the verify pass — and it's the heart of the project. A brand-new, skeptical reviewer looks at each suspected bug on its own and answers one question: is this a real, defensible bug? Only survivors are kept. This is the precision lever that stops the tool from becoming noise.

1. VERIFY_SYSTEM is a deliberately harsh prompt: "Default to is_real=false if unsure. Precision, not volume." When in doubt, it drops the finding — the opposite bias from the review pass.
2. def verify(diff, finding): asks the model about one finding at a time. output_format=Verdict forces a clean yes/no answer, and .parsed_output.is_real pulls out just the True/False.
3. def review_and_verify(diff): wires the two passes together. First review = review_pr(diff) gets all the candidates.
4. confirmed = [f for f in review.findings if verify(diff, f)] is the filter (the # the filter comment): it keeps a finding only if verify(...) returns True. This is a list comprehension — a compact for-loop that builds a new, shorter list.

What the output means: The SUMMARY line, then 3 confirmed finding(s) — each printed as file:line [severity] description. The false positives from Step 3 have been silently dropped; only defensible bugs remain.

Try this: Count the calls: one review call, then one verify call per finding. That's the cost of precision. If you find the reviewer too noisy, the fix is almost always to make VERIFY_SYSTEM stricter — not to touch the review pass.

These tests check the parts that don't need the model at all — the schema and the fixture — so they run instantly with no API key. Fast, free tests like these guard the foundations every other step stands on.

1. python -m pytest tests/ -v runs the test suite. -v (verbose) lists each test by name with PASSED or FAILED.
2. test_finding_schema_validates builds a well-formed Finding and confirms Pydantic accepts it — the happy path works.
3. test_finding_rejects_bad_severity feeds a severity outside the Literal list and confirms Pydantic rejects it — proving the guard from Step 1 actually bites.
4. test_diff_contains_the_known_bugs checks the fixture from Step 2 really holds the planted bugs, so the eval has something real to catch.

What the output means: 3 passed — all three green. Because none of these call the API, they pass the same way on any machine, key or no key.

Try this: Temporarily change a severity in schemas.py to allow "urgent" and re-run — test_finding_rejects_bad_severity will fail, showing you the test is genuinely enforcing the rule.

The eval answers the question that decides whether this agent is worth shipping: does it actually catch the bugs? It runs the full review→verify pipeline on the planted diff and scores the confirmed findings against the KNOWN_ISSUES answer key from Step 2. This one needs an API key because it calls the model.

1. python evals.py runs the whole pipeline once and then grades it.
2. confirmed findings: 3 is how many bugs survived the verify filter. Each [HIT] known issue: ... line means a planted bug (amount / TypeError / None) was matched by a confirmed finding.
3. recall on known bugs: 100% means it found all three planted bugs. Recall = of the bugs that exist, what fraction did we catch — the headline quality number.
4. The note explains the two failure modes the eval watches for: it fails if it misses most bugs (recall too low) and warns if it reports far more than three (a sign the verify step has stopped filtering).

What the output means: A per-bug hit list, a recall percentage, and ✅ evals passed. Together with Step 5's schema tests, this is your evidence the pipeline both works and stays precise.

Try this: After adding a fourth bug in Step 2's 'Try this', re-run — recall is now out of four. This is exactly how you'd grow a real golden set: every bug the agent misses becomes a new case in the eval.