Specialized Topics · Part T2

Multimodal AI

The whole course so far is text-in, text-out. But real systems read images, documents with layout, and audio. This part adds those input types step by step: how a vision model actually "sees," how to send images to Claude, building document-intelligence and multimodal-RAG pipelines, and where audio fits. Basic → advanced, with runnable code and diagrams.

⏱️ ~2 hours🎯 Basic → Advanced🖼️ beyond textrunnable

⚙️ To run this for realThe code here is complete and correct as written. To actually execute it you'll need:

an Anthropic API key (ANTHROPIC_API_KEY) + pip install anthropic

Reading and learning works without any of this — run when you're ready. IDs/ARNs in examples are placeholders; swap in your own.

Learning objectives

Understand how a vision-language model turns pixels into tokens the LLM can reason over.
Send images to Claude (base64 & URL) and ask questions about them.
Extract structured data from documents (invoices, forms, screenshots).
Build multimodal RAG — retrieve over images + text.
Know where audio (speech-to-text) slots into a pipeline.

1 · How a vision model "sees" basic essential

A vision-language model doesn't process pixels directly with the LLM. An image encoder (a vision transformer) splits the image into patches, turns each patch into a vector, and projects those into the same embedding space as text tokens. The LLM then attends over image-patch tokens and text tokens together — one stream, two modalities.

Images become tokens too. The encoder converts patches into embeddings in the LLM's token space, so the model reasons over image and text jointly. Practical upshot: images cost tokens (often hundreds–thousands each), which drives latency and price.

🗺️ How to read this diagram

This picture answers a beginner's first question: how does a text model look at a picture at all? The trick is that the image is turned into the same kind of tokens the model already uses for words, so it can read pixels and text together.

Start on the left: the image is chopped into a grid of small squares called patches (the little boxes labelled image patches).
The encoder box in the middle is a vision model that turns each patch into a vector of numbers — an embedding — that lives in the model's token space.
On the right, those become img tok (image tokens) that sit right next to the text tok (your words). Follow the arrows left to right.
The LLM at the far right then reads image tokens and text tokens as one single stream — that's why it can answer questions about a picture.

In short: Because a picture becomes many tokens (often hundreds to thousands), images cost real money and add latency. Bigger image = more patches = more tokens = higher bill.

2 · Sending an image to Claude basic → intermediate essential

Images go in a message's content as an image block — either base64-encoded bytes or a URL — alongside a text block asking your question.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Ask a question about a local image

pythonimport base64, anthropic
client = anthropic.Anthropic()

with open("chart.png", "rb") as f:
    data = base64.standard_b64encode(f.read()).decode()

msg = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {
                "type": "base64", "media_type": "image/png", "data": data}},
            {"type": "text", "text": "What trend does this chart show? Give the peak value."},
        ],
    }],
)
print(msg.content[0].text)
# URL form: {"type":"image","source":{"type":"url","url":"https://..."}}

▶ How this works

This is the core recipe for asking Claude about a picture on your computer. There is only one new idea versus a normal text call: you read the image file, turn its raw bytes into text with base64, and drop it into the message as an image block.

open("chart.png", "rb") opens the file in binary read mode ("rb") because an image is raw bytes, not letters.
base64.standard_b64encode(...).decode() is the key step: it converts those bytes into a plain-text string of safe characters, because the API sends everything as text (JSON). base64 is just "bytes written as letters".
Inside messages, the content is a list of blocks. The first block has "type": "image" and carries the base64 data plus its media_type (here image/png so Claude knows the format).
The second block is a normal text block with your question. Sending the image and the question together is what makes this multimodal.
The last comment shows the alternative: instead of base64 bytes you can pass a "url" and let Claude fetch the image itself.

What the output means: msg.content[0].text prints Claude's answer in words — e.g. a sentence describing the chart's trend and naming its peak value.

Try this: Swap in your own .png and change the question text. If your file is a JPEG, remember to change media_type to "image/jpeg" to match.

Practical tipsSupported types: PNG, JPEG, WebP, GIF. Very large images are downscaled — resize to what's needed (bigger ≠ better, just costlier). You can send multiple images in one message (e.g. "compare these two screenshots"). Count image tokens when budgeting (A5/A8).

3 · Document intelligence — structured extraction advanced

The highest-value multimodal use case: turn a messy document (invoice, form, receipt, screenshot) into structured data. Combine a vision message with structured output (A6) so you get a validated object, not prose.

Vision + Pydantic = a document parser. Ask the model to read the image and return your schema; validate the result. This replaces brittle OCR-plus-regex pipelines for semi-structured documents.

🗺️ How to read this diagram

This diagram shows the single most useful multimodal job: turning a messy document (an invoice, form, or receipt photo) into clean structured data your code can trust — not a paragraph you'd have to parse by hand.

Left: the raw invoice — an image or PDF page. Unstructured; a human can read it but a program can't reliably.
Middle: vision + schema means you send the image to the vision model and tell it the exact shape you want back (the schema).
Right: out comes a tidy object like {invoice_no, date, total, line_items[]} — named fields with correct types, ready to store in a database.

In short: Read it left to right as "picture in, structured object out." This one pattern replaces brittle old OCR + regex pipelines for semi-structured documents.

Extract an invoice into a validated model

pythonfrom pydantic import BaseModel

class LineItem(BaseModel):
    description: str
    qty: int
    amount: float

class Invoice(BaseModel):
    invoice_no: str
    date: str
    total: float
    items: list[LineItem]

# messages.parse() forces schema-valid output (see Ch 2 / A6)
# resp = client.messages.parse(
#     model="claude-opus-4-8", max_tokens=2048, output_format=Invoice,
#     messages=[{"role":"user","content":[image_block, {"type":"text",
#         "text":"Extract this invoice into the schema."}]}])
# invoice = resp  # a validated Invoice object

▶ How this works

This is the code behind that diagram. You describe the shape of the data you want using Pydantic classes, then ask the model to fill it in from the image. Pydantic is your contract: the result is guaranteed to match, or it errors — no guessing.

class LineItem(BaseModel) declares one row of the invoice: a description (text), a qty (whole number), and an amount (decimal). The type after each name is the promise.
class Invoice(BaseModel) is the whole document: an invoice number, a date, a total, and items: list[LineItem] — a list of the rows above. Schemas nest like this to mirror real documents.
The commented-out call shows the payoff: client.messages.parse(..., output_format=Invoice, ...) sends the image block plus a short instruction, and the .parse method forces the reply to fit your Invoice shape.
What you get back is a real, validated Invoice Python object — you can read invoice.total directly instead of hunting through text.

What the output means: No console output here — the lines after # are commented out so the snippet is safe to read. Live, it would hand you a filled-in Invoice object with every field typed and checked.

Try this: Add a currency: str field to Invoice. The model will start returning it too — you shape the output just by editing the class.

🔗 In this courseThis is the heart of the Document Intelligence project (P4 in the gallery). Combine with the chunking/parsing from A5 for long PDFs, and structured output from A6.

4 · Multimodal RAG advanced

Standard RAG (Ch 3) retrieves text. Multimodal RAG retrieves images too — e.g. "find the diagram that answers this." Two common designs:

Caption-then-embed: use a vision model to describe each image in text, embed the caption, retrieve as normal text. Simple, works with any text vector store.
Native multimodal embeddings: a model (e.g. CLIP-style) embeds images and text into one shared vector space, so a text query can directly match an image. More powerful, needs a multimodal embedder.

Caption-then-embed pipeline (works today with any vector DB)

Setup to run this snippet

class _Any:
    '''stands in for any undefined demo value; supports call/attr/index/
    iteration and basic arithmetic (as 0.7) so demo snippets run.'''
    def __call__(self, *a, **k): return _Any()
    def __getattr__(self, k): return _Any()
    def __getitem__(self, k): return _Any()
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __contains__(self, o): return True
    def __enter__(self, *a): return _Any()
    def __exit__(self, *a): return False
    def __float__(self): return 0.7
    def __int__(self): return 1
    def __lt__(self, o): return True
    def __gt__(self, o): return False
    def __le__(self, o): return True
    def __ge__(self, o): return False
    def __add__(self, o): return o
    def __radd__(self, o): return o
    def __bool__(self): return True
    def __repr__(self): return 'demo'
    def __str__(self): return 'demo'
def call_vision(*a, **k):  # demo stub
    return _Any()

python# 1. For each image, generate a rich caption with the vision model
def caption(image_block):
    # ask: "Describe this image in detail for search: contents, text, chart values"
    return call_vision(image_block, "Describe for retrieval...")

# 2. Embed the caption (A4/A7) and store with a pointer back to the image
#    vector_db.add(embed(caption(img)), metadata={"image_path": path})

# 3. At query time: retrieve top-k captions, then feed the ACTUAL images
#    to the model for the final grounded answer.

▶ How this works

This is the simplest way to make image search work with tools you already have. The idea: you can't easily search pictures, but you can search text — so first turn each image into a written description (a caption), then do ordinary text RAG over the captions.

Step 1 — caption(image_block) asks the vision model to describe an image in detail (its contents, any text, chart values). That description is plain text.
Step 2 (the comment) — you embed that caption into a vector and store it in a vector DB, keeping a metadata={"image_path": path} pointer back to the original picture.
Step 3 (the comment) — at query time you retrieve the top matching captions, then feed the actual images (not just the captions) to the model for the final answer, so it reasons over the real pixels.

What the output means: This is a skeleton, not a runnable program (call_vision is a demo stub), so nothing prints. It shows the flow: caption → embed → retrieve → answer.

Try this: Compare with the alternative in the text above — native multimodal embeddings (CLIP-style) put images and text in one shared space, skipping the caption step.

🔗 In this courseBuilds directly on Ch 3 RAG and the vector-DB design in A7 — same retrieve→rerank→generate loop, just with images in the corpus.

5 · Audio & the full pipeline intermediate

LLMs are text models, so audio enters via a speech-to-text (ASR) step first (e.g. Whisper), then the transcript flows through the normal text pipeline; a text-to-speech step can voice the reply. The LLM sits in the middle.

Audio bookends the text pipeline. Transcribe first (ASR), run your normal LLM/RAG/agent logic on the text, optionally synthesize speech for the reply. Everything you learned about text still applies in the middle.

🗺️ How to read this diagram

This diagram shows where audio fits. LLMs only understand text, so voice has to be converted to text on the way in and (optionally) back to voice on the way out. The model you already know sits unchanged in the middle.

Follow the arrows left to right: 🎤 audio comes in first.
ASR (STT) means Automatic Speech Recognition / Speech-To-Text (e.g. Whisper) — it turns the spoken words into a plain text transcript.
LLM (text) in the centre is your normal text pipeline — everything from earlier chapters works here unchanged, because it's just text now.
TTS (Text-To-Speech) turns the model's text reply back into 🔊 reply audio, if you want a spoken answer.

In short: Audio just bookends the text pipeline: transcribe first, do your usual LLM work in the middle, synthesize speech last. No new LLM magic — two extra converters.

🎯 Interview practice interview

The interview questions this topic gets asked — worked, with code. For the full pattern catalog see A9 · Big Tech AI-engineering patterns.

System design: extract data from invoices/PDFs

Vision + structured output beats OCR+regex for semi-structured docs.

python# PIPELINE: image/PDF -> vision model with a Pydantic schema -> validated object
# SCALE:    batch pages; cache by document hash; fall back to human review on low confidence
# COST:     images cost tokens (hundreds-thousands each) -> resize, budget

▶ How this works

These three comment lines are a whiteboard answer to a common interview question: "design a system to pull data out of invoices/PDFs." Each line names one axis an interviewer probes — the core approach, how it scales, and what it costs.

PIPELINE — the core design: image/PDF goes to a vision model with a Pydantic schema, and you get a validated object out. This is the same idea as section 3 above.
SCALE — how to handle volume: process pages in batches, cache by document hash so you never re-pay for the same file, and route low-confidence results to a human reviewer instead of trusting them blindly.
COST — the money reality: images cost tokens (hundreds to thousands each), so resize images and budget deliberately.

Try this: In an interview, say the headline first — "vision + structured output beats OCR+regex" — then walk these three axes. Naming scale and cost trade-offs is what senior looks like.

Send an image to the model

Images ride in the message content as a base64 (or URL) image block next to your question.

pythonimport base64
data = base64.standard_b64encode(open("chart.png","rb").read()).decode()
content = [
    {"type":"image","source":{"type":"base64",
        "media_type":"image/png","data":data}},
    {"type":"text","text":"What is the peak value in this chart?"},
]

▶ How this works

A tight, from-memory version of "send an image to the model" — the kind of snippet you might whiteboard. It's the same recipe as the first lab, squeezed to the essentials.

base64.standard_b64encode(open("chart.png","rb").read()).decode() does the whole read-and-encode in one line: open the file as bytes, base64-encode them, decode to a text string.
content is the list of two blocks again: an image block carrying the base64 data and its media_type, then a text block with the question.
That's the entire pattern — everything else (the client.messages.create call) is identical to a normal text request.

Try this: If you can reproduce these two blocks from memory — image block + text block in a content list — you can send any picture to the model. That's the whole interview answer.

Checkpoint advanced

Explain how images become tokens and why they cost more.
Send base64/URL images to Claude and ask questions about them.
Extract structured data from a document with vision + a Pydantic schema.
Design multimodal RAG (caption-then-embed or native embeddings).
Place ASR/TTS correctly around the text LLM pipeline.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Pick the right media_type from a filename (offline)Beginner

Context: An image content block must declare the correct media_type, and a JPEG announced as PNG can be rejected. Deriving the MIME type from the filename is the small, offline first step.

Your task: Write media_type_for(path) that returns the right MIME string for the image types the model supports (PNG/JPEG/WebP/GIF) and raises on anything else. Fully runnable offline.

Requirements:

Map each supported extension to its MIME string (stdlib os only)
Both .jpg and .jpeg map to image/jpeg
Lowercase the extension so chart.PNG resolves correctly
Raise ValueError on an unsupported type such as .pdf
Show a couple of successful lookups and one rejection

💡 Hint: Split the extension with os.path.splitext, normalize case, and look it up in a small dict of the supported types.

Show solution

Runnable, stdlib only:

import os

SUPPORTED = {
    ".png": "image/png",
    ".jpg": "image/jpeg", ".jpeg": "image/jpeg",
    ".webp": "image/webp",
    ".gif": "image/gif",
}

def media_type_for(path):
    ext = os.path.splitext(path)[1].lower()
    if ext not in SUPPORTED:
        raise ValueError(f"unsupported image type: {ext!r}")
    return SUPPORTED[ext]

print(media_type_for("chart.PNG"))   # image/png
print(media_type_for("scan.jpeg"))   # image/jpeg
try:
    media_type_for("notes.pdf")
except ValueError as e:
    print(e)                         # unsupported image type: '.pdf'

Matching media_type to the actual bytes is what lets the model decode the image; a JPEG sent as image/png can be rejected.

Exercise 2 · Base64-encode local bytes and build the image block (offline)Intermediate

Context: Sending a local image means reading its bytes, base64-encoding them, and shaping the exact content-block dict the SDK expects — all doable offline before any live call.

Your task: Write image_block(path) that reads a file's bytes, base64-encodes them, and returns the SDK's image content-block dict, then verify it round-trips by decoding back.

Requirements:

Read the file in binary and base64-encode with base64 (stdlib)
Return the block shape {"type": "image", "source": {"type": "base64", "media_type": ..., "data": ...}}
Reuse the media-type lookup so the block declares the correct MIME
Demonstrate offline: write raw bytes, encode, then decode the data field to prove the round-trip
Note this is the exact shape handed to the messages API — only the final call differs

💡 Hint: The only difference from the live call is that this dict later becomes one entry in the message's content list.

Show solution

Runnable offline (creates a tiny file, encodes it, decodes back to prove it round-trips):

import base64, os

def media_type_for(path):
    return {"png":"image/png","jpg":"image/jpeg","jpeg":"image/jpeg"}[
        os.path.splitext(path)[1].lower().lstrip(".")]

def image_block(path):
    with open(path, "rb") as f:
        data = base64.standard_b64encode(f.read()).decode()
    return {"type": "image",
            "source": {"type": "base64",
                       "media_type": media_type_for(path),
                       "data": data}}

# --- demo: write raw bytes, encode, verify round-trip ---
with open("tiny.png", "wb") as f:
    f.write(b"\x89PNG\r\n\x1a\n demo bytes")
blk = image_block("tiny.png")
print(blk["source"]["media_type"])                       # image/png
print(base64.b64decode(blk["source"]["data"])[:4])       # b'\x89PNG'

This is the exact shape the message API consumes; the only difference from the live call is that you hand this block to client.messages.create.

Exercise 3 · Ask Claude a question about a local image (needs an API key)Advanced

Context: The core vision recipe is one idea beyond a text call: the message content is a list of blocks — an image block plus a text block. This rung needs an API key and network.

Your task: Combine the encode step with a real vision call: send an image block plus a text question and print the model's answer.

Requirements:

Base64-encode a local image and place it in an image content block
Send a message whose content list holds the image block plus a text question
Read the answer from the first content block of the response
Requires ANTHROPIC_API_KEY and the anthropic package (network needed)
The offline encode block from the previous rung is the runnable half

💡 Hint: The whole novelty over a text call is that content is a list of blocks rather than a single string.

Show solution

Needs an API key (ANTHROPIC_API_KEY) and the anthropic package. This is the lesson's core recipe, verbatim in shape:

import base64, anthropic

client = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY from env

with open("chart.png", "rb") as f:
    data = base64.standard_b64encode(f.read()).decode()

msg = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {
                "type": "base64", "media_type": "image/png", "data": data}},
            {"type": "text", "text": "What trend does this chart show? Give the peak value."},
        ],
    }],
)
print(msg.content[0].text)

Only one idea beyond a text call: the content is a list of blocks — an image block plus a text block. msg.content[0].text is the answer.

Exercise 4 · Estimate image token cost before you send (offline)Expert

Context: Images cost tokens roughly by area, so a huge image quietly inflates your bill and latency. Budgeting the cost before you send is the practical discipline — bigger is not better.

Your task: Write approx_image_tokens(w, h) using a simple tile heuristic and a maybe_downscale(w, h, budget) that decides whether to shrink a large image before sending. Runnable offline.

Requirements:

Estimate cost by tiling: ceil(w/tile) × ceil(h/tile) × tokens-per-tile
Use math only; note the exact numbers vary by provider and the point is budgeting
maybe_downscale returns as-is when the estimate fits the budget
When over budget, scale dimensions down (square-root of the ratio) and re-estimate
Show a large image getting downscaled and a small one sent unchanged

💡 Hint: Scaling both dimensions by sqrt(budget/estimate) shrinks the tile count roughly to the budget in one step.

Show solution

Runnable, stdlib only. Uses a simple, documented-style tile heuristic (exact numbers vary by provider; the point is budgeting before you pay):

import math

TILE = 512          # px per tile edge
TOKENS_PER_TILE = 170

def approx_image_tokens(w, h):
    tiles = math.ceil(w / TILE) * math.ceil(h / TILE)
    return tiles * TOKENS_PER_TILE

def maybe_downscale(w, h, budget=700):
    est = approx_image_tokens(w, h)
    if est <= budget:
        return (w, h, est, "send as-is")
    # scale so estimated tokens fit the budget
    scale = math.sqrt(budget / est)
    nw, nh = int(w * scale), int(h * scale)
    return (nw, nh, approx_image_tokens(nw, nh), "downscaled")

print(approx_image_tokens(512, 512))     # 170
print(maybe_downscale(2048, 2048))       # (..., 'downscaled') -- big image trimmed
print(maybe_downscale(400, 300))         # (400, 300, 170, 'send as-is')

Bigger ≠ better: more patches = more tokens = higher bill and latency. Budgeting up front is the practical discipline the lesson calls out.

Exercise 5 · Caption-then-embed for multimodal RAG (offline pipeline shape)Professional

Context: The multimodal RAG pattern is caption-then-embed: turn each image into a text caption, then embed the caption into a vector store. Because the index is text, it works today with any text vector DB.

Your task: Model the caption-then-embed pipeline offline with a fake captioner and a deterministic embedder, keeping the real vision call swappable, and retrieve the best-matching image for a text query.

Requirements:

A caption() stub stands in where a real vision call would go
A deterministic embed() (e.g. bag-of-words) turns caption text into a vector
A cosine() similarity ranks captions against the query vector
Build an index of (image, caption-embedding) pairs and return the closest image for a query
Explain that swapping caption() for a vision call and embed() for a real model makes it production-ready

💡 Hint: Because images are represented by their caption embeddings, retrieval is just text-vs-text cosine over the captions.

Show solution

Runnable offline. The caption() is a stub where a vision call would go; the embed/search half is a real, tiny cosine index:

import math

def caption(image_path):
    # OFFLINE STUB. Real version: client.messages.create(...) on the image,
    # then use msg.content[0].text as the caption.
    return {"invoice.png": "invoice total 42 dollars acme",
            "chart.png":   "line chart sales peak q3"}[image_path]

def embed(text):                       # toy bag-of-words vector (deterministic)
    vocab = ["invoice","total","chart","sales","peak","acme"]
    return [text.count(w) for w in vocab]

def cosine(a, b):
    dot = sum(x*y for x, y in zip(a, b))
    na = math.sqrt(sum(x*x for x in a)); nb = math.sqrt(sum(y*y for y in b))
    return dot / (na*nb) if na and nb else 0.0

# index images by their captions
index = [(p, embed(caption(p))) for p in ["invoice.png", "chart.png"]]
q = embed("what was the sales peak")
best = max(index, key=lambda it: cosine(q, it[1]))
print("best match:", best[0])          # chart.png

Because images are embedded via their captions, this works today with any text vector DB — swap caption() for a real vision call and embed() for your embedding model.

Exercise 6 · Invoice extraction with a validated schema (needs an API key)Industry scenario

Context: The highest-value multimodal use case turns a messy invoice image into a validated object, not prose. The trust comes from schema validation, not from the model's free-form text.

Your task: Combine a vision message with structured output to extract invoice fields, then validate the returned JSON against a schema and reject on missing or mistyped fields. The live call needs an API key; the validation half runs offline.

Requirements:

The vision call asks for JSON with specific keys (vendor, total, currency) and JSON only
Parse the model text as JSON with the stdlib json module
A validate_invoice() checks every required key is present and correctly typed
Raise on a missing field (e.g. total) and show a valid object passing
Requires ANTHROPIC_API_KEY for extraction; the schema check is runnable offline
Frame the rule: never trust free-form model text as data — validate before it enters your system

💡 Hint: Keep a small required-fields map of name → expected type and check presence and isinstance before returning the object.

Show solution

The vision call needs an API key; the schema/validation logic below it is runnable offline and is what makes the result trustworthy:

import base64, json, anthropic          # SDK part needs an API key

def extract_invoice(path):
    client = anthropic.Anthropic()
    with open(path, "rb") as f:
        data = base64.standard_b64encode(f.read()).decode()
    msg = client.messages.create(
        model="claude-opus-4-8", max_tokens=1024,
        messages=[{"role": "user", "content": [
            {"type": "image", "source": {"type": "base64",
                "media_type": "image/png", "data": data}},
            {"type": "text", "text":
                "Extract JSON with keys: vendor (str), total (number), "
                "currency (str). Reply with JSON only."},
        ]}],
    )
    return json.loads(msg.content[0].text)

# ---- offline: validate whatever came back (runs as-is) ----
REQUIRED = {"vendor": str, "total": (int, float), "currency": str}

def validate_invoice(obj):
    for key, typ in REQUIRED.items():
        if key not in obj:
            raise ValueError(f"missing field: {key}")
        if not isinstance(obj[key], typ):
            raise TypeError(f"{key} has wrong type")
    return obj

print(validate_invoice({"vendor": "Acme", "total": 42.0, "currency": "USD"}))
try:
    validate_invoice({"vendor": "Acme"})
except ValueError as e:
    print("rejected:", e)              # rejected: missing field: total

Design note: never trust free-form model text as data — validate against a schema and fail on missing/mistyped fields before it enters your system (invoices flow to ledgers).

Knowledge check check yourself

✓ Knowledge check

According to the lesson, how does a vision-language model "see" an image, and what practical cost consequence follows from that mechanism?

Show answer

An image encoder (a vision transformer) splits the image into patches, turns each patch into a vector, and projects them into the same embedding space as text tokens, so the LLM attends over image and text tokens as one stream. Because a picture becomes many tokens (often hundreds to thousands), images cost real money and add latency — bigger image = more patches = more tokens = higher bill, so you resize to what's needed.

✓ Knowledge check

For turning invoices/PDFs into structured data, the lesson recommends vision + a Pydantic schema over OCR-plus-regex. What does pairing vision with a schema (via messages.parse) buy you?

Show answer

You get a validated object whose fields have the right types (e.g. {invoice_no, date, total, items[]}) instead of prose you'd parse by hand — the schema is a contract, so the reply must match or it errors. This replaces brittle OCR+regex pipelines for semi-structured documents and lets you read invoice.total directly.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

1 · How a vision model "sees" basic essential

2 · Sending an image to Claude basic → intermediate essential

3 · Document intelligence — structured extraction advanced

4 · Multimodal RAG advanced

5 · Audio & the full pipeline intermediate

🎯 Interview practice interview

Checkpoint advanced

🪜 Practice ladder beginner → industry

Knowledge check check yourself

More in Specialized topics