Multimodal AI

1 · How a vision model "sees" basic essential

Start with the confusion that trips up almost everyone new to this. You paste a screenshot into a chat window, the model describes it perfectly, and it feels like the LLM grew eyes. So the first time your image bill is ten times what you expected, or a "simple" photo question is inexplicably slow, it makes no sense — a picture is just one attachment, right? The mistake is imagining the model looks at pixels the way you do. It doesn't. Understanding what really happens is what turns those cost and latency surprises into things you can predict and control.

Here's the analogy that fixes it: the model can only read one language — tokens — the same numbered chunks it uses for words. It has no separate "eyes." So before an image can be reasoned about, it has to be translated into that same token language, exactly like translating a foreign document into English before an English-only reader can understand it. The translator is a separate piece of machinery bolted in front of the language model, and the length of the translation is what you pay for.

Concretely, that translator does three things. First it chops the image into a grid of small square patches (think of laying a sheet of graph paper over the picture). Second, an encoder — a vision transformer — turns each patch into a vector of numbers, and crucially projects those vectors into the same embedding space the LLM uses for text. Third, the LLM simply reads that combined stream: image-patch tokens sitting shoulder-to-shoulder with your text tokens, one sequence, attended over jointly. That "same space" projection is the whole trick — it's why a text model can suddenly answer questions about a chart without being retrained from scratch.

Now the practical consequence, which is the whole reason this section is first. A picture doesn't become one token — it becomes many, often hundreds to thousands, because every patch is a token. More patches means more tokens means more money and more time. That gives you three levers you now understand rather than guess at:

• Image size drives cost. A bigger image is more patches is more tokens is a bigger bill. Resizing a huge photo down to what the task actually needs can cut the token count dramatically for the same answer — bigger is not sharper eyes, just a longer translation.
• Detail you don't need is money you don't spend. If you only need to read a total off a receipt, a modest resolution is plenty; reserve high resolution for genuinely fine print or dense diagrams.
• Latency tracks tokens too. Those image tokens sit in the same sequence the model processes, so a heavy image slows the response the same way a very long prompt would.

Keep that mental model — patches become tokens in text's space — as you read the diagram below; every cost and design choice in the rest of this part flows from it.

A vision-language model doesn't process pixels directly with the LLM. An image encoder (a vision transformer) splits the image into patches, turns each patch into a vector, and projects those into the same embedding space as text tokens. The LLM then attends over image-patch tokens and text tokens together — one stream, two modalities.

2 · Sending an image to Claude basic → intermediate essential

You already know how to make a text call — so the hurdle here is smaller than it looks, but it's exactly where beginners freeze. Your first instinct is to reach for some special "vision endpoint" or a separate upload API, and you go hunting for one that doesn't exist. Then you try the obvious thing — jam the file path or the raw image bytes into the content string — and get a cryptic error, because a message body is JSON and JSON is text; raw binary has no place in it. The good news: sending an image is the same call you already make, with one new shape to learn.

Here's the mental model. Think of the message content not as a single sentence but as an envelope you can put several cards into — a list of "blocks." A normal text call happens to hold one card (your question). A vision call holds two: an image block and a text block. That's the entire conceptual leap — content becomes a list — and once it clicks, multimodal stops feeling exotic.

The image block itself needs two things the model can't guess. First, the bytes of the picture, rewritten as text — that's what base64 does: it re-expresses raw binary as a string of safe letters and digits so it can ride inside JSON. (You aren't compressing or changing the image, just spelling it out in an alphabet text can carry.) Second, a media type like image/png or image/jpeg, which tells the model how to decode those bytes back into a picture. Getting the media type wrong — announcing a JPEG as PNG — is the single most common beginner failure and can get the image rejected outright, so it must match the real file.

You have two ways to supply the picture, and the choice is a genuine trade-off:

• base64 bytes — you read the file, encode it, and embed it directly in the request. Works for local files and private images the model's servers can't reach; the cost is a larger request body.
• a URL — you pass a link and let the model fetch the image itself. Lighter request, but the URL must be publicly reachable, which rules out anything behind a login or on your laptop.

One more thing the envelope model buys you: you can drop several image cards into one message and ask the model to compare them ("which of these two screenshots has the error?"). And because you now know from section 1 that each image is really many tokens, you'll want to size images sensibly before encoding — the recipe below is deliberately minimal so the one new idea, content is a list of blocks, stands out clearly.

Images go in a message's content as an image block — either base64-encoded bytes or a URL — alongside a text block asking your question.

pythonimport base64, anthropic
client = anthropic.Anthropic()

with open("chart.png", "rb") as f:
    data = base64.standard_b64encode(f.read()).decode()

msg = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {
                "type": "base64", "media_type": "image/png", "data": data}},
            {"type": "text", "text": "What trend does this chart show? Give the peak value."},
        ],
    }],
)
print(msg.content[0].text)
# URL form: {"type":"image","source":{"type":"url","url":"https://..."}}

3 · Document intelligence — structured extraction advanced

Picture the pipeline that has ruined more engineering weekends than almost any other. The task sounds trivial: read the total off a stack of invoices. So you bolt on an OCR library, get back a wall of text, and start writing regexes to fish out the invoice number and the amount. It works on the ten samples you tested. Then vendor #11 puts the total in the top-right instead of the bottom, vendor #12 uses "Amount Due" instead of "Total," a scan comes in rotated, and your regexes quietly return the wrong number — or nothing. This is the classic OCR-plus-regex trap: brittle because it treats a layout problem as a string problem.

The reframe that makes this section the highest-value one in the whole part: a vision model doesn't need you to describe where the total lives, because it reads the document the way a person does — seeing that a number near the words "Total Due" in the bottom-right is the total, regardless of the exact pixels. So instead of extracting text and then hunting through it, you ask the model to look at the page and hand you the fields directly.

But "hand you the fields" hides the real trick, and it's worth slowing down for. If you just ask "what's the invoice number and total?" you get back a sentence — prose you'd have to parse, which is the regex nightmare all over again, one layer up. The fix is to pair vision with a schema: you declare, up front, the exact shape you expect — an object with invoice_no, date, total, and a list of line items, each with its own types — and force the reply to match it. Think of the schema as a form with labelled, typed boxes: the model must fill in this box with text, that one with a number. You don't get a paragraph; you get a validated object.

Why the validation matters so much here: total declared as a number isn't decoration — it's a contract. If the model returns something that can't be a number, the parse fails loudly instead of silently feeding "$1,2O0" (with a letter O) into your accounting system. That is the difference between a demo and something you'd trust with real invoices. Two habits fall out of this:

• Model the document's real structure. Nest schemas the way the paper nests — an invoice contains a list of line items — so one call returns the whole thing already organized, not a flat blob you reshape afterward.
• Treat the schema as your source of truth. Need a new field like currency or tax? Add it to the schema and the extraction follows; you shape the output by editing the contract, not by writing more parsing code.

This is a genuine paradigm shift from the old pipeline, and the diagram below draws it as one clean arrow — picture in, typed object out — replacing the fragile OCR + regex chain entirely for semi-structured documents.

The highest-value multimodal use case: turn a messy document (invoice, form, receipt, screenshot) into structured data. Combine a vision message with structured output (A6) so you get a validated object, not prose.

pythonfrom pydantic import BaseModel

class LineItem(BaseModel):
    description: str
    qty: int
    amount: float

class Invoice(BaseModel):
    invoice_no: str
    date: str
    total: float
    items: list[LineItem]

# messages.parse() forces schema-valid output (see Ch 2 / A6)
# resp = client.messages.parse(
#     model="claude-opus-4-8", max_tokens=2048, output_format=Invoice,
#     messages=[{"role":"user","content":[image_block, {"type":"text",
#         "text":"Extract this invoice into the schema."}]}])
# invoice = resp  # a validated Invoice object

4 · Multimodal RAG advanced

Here's the wall you hit the moment your knowledge base isn't pure text. You built RAG in Chapter 3, it retrieves passages beautifully, and then a user asks "which diagram shows the network topology?" or "find the slide with the Q3 revenue chart." Your retriever is helpless — it can only compare the text of the query against the text of your chunks, and a diagram isn't text. The valuable answer is sitting right there in your corpus as an image, and standard RAG can't reach it. Multimodal RAG is how you close that gap.

To see the fix, remember what retrieval actually is: turning things into vectors — points in a high-dimensional space — so that "similar in meaning" becomes "close together," and finding relevant items becomes finding nearby points. Text RAG only works because both the query and the chunks land in the same space, so distance is meaningful. The whole challenge with images is getting a picture and a text query into a comparable space at all. There are two honest ways to do it, and choosing between them is a real engineering decision:

• Caption-then-embed — the pragmatic route. You first ask a vision model to describe each image in rich text (a caption), then embed and store that caption like any other text chunk. You've converted the image problem into a text problem, so it works with any text vector store you already run — nothing new to buy. The catch: retrieval quality is only as good as the caption. If the caption says "a bar chart" but the query needs the exact peak value, and the captioner didn't mention it, the match fails. So you write captions for search — deliberately dense with the contents, on-image text, and numbers people might query.
• Native multimodal embeddings — the powerful route. A model trained to place images and text in one shared vector space (the CLIP family works this way) lets a text query match an image directly, no caption in between. Nothing is lost in translation to words, but you take on a dependency: a multimodal embedder and a store that holds its vectors.

One subtlety that separates a working system from a toy: even with caption-then-embed, the caption is only for finding the right image — at answer time you feed the actual image back to the model, not its caption. The caption is an index card in the drawer; once you've located the right card you pull the real document. That two-phase move — retrieve on a cheap text proxy, then reason over the true pixels — is what keeps answers grounded in what's really in the picture rather than in a lossy summary of it.

Standard RAG (Ch 3) retrieves text. Multimodal RAG retrieves images too — e.g. "find the diagram that answers this." Two common designs:

• Caption-then-embed: use a vision model to describe each image in text, embed the caption, retrieve as normal text. Simple, works with any text vector store.
• Native multimodal embeddings: a model (e.g. CLIP-style) embeds images and text into one shared vector space, so a text query can directly match an image. More powerful, needs a multimodal embedder.

class _Any:
    '''stands in for any undefined demo value; supports call/attr/index/
    iteration and basic arithmetic (as 0.7) so demo snippets run.'''
    def __call__(self, *a, **k): return _Any()
    def __getattr__(self, k): return _Any()
    def __getitem__(self, k): return _Any()
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __contains__(self, o): return True
    def __enter__(self, *a): return _Any()
    def __exit__(self, *a): return False
    def __float__(self): return 0.7
    def __int__(self): return 1
    def __lt__(self, o): return True
    def __gt__(self, o): return False
    def __le__(self, o): return True
    def __ge__(self, o): return False
    def __add__(self, o): return o
    def __radd__(self, o): return o
    def __bool__(self): return True
    def __repr__(self): return 'demo'
    def __str__(self): return 'demo'
def call_vision(*a, **k):  # demo stub
    return _Any()

python# 1. For each image, generate a rich caption with the vision model
def caption(image_block):
    # ask: "Describe this image in detail for search: contents, text, chart values"
    return call_vision(image_block, "Describe for retrieval...")

# 2. Embed the caption (A4/A7) and store with a pointer back to the image
#    vector_db.add(embed(caption(img)), metadata={"image_path": path})

# 3. At query time: retrieve top-k captions, then feed the ACTUAL images
#    to the model for the final grounded answer.

5 · Audio & the full pipeline intermediate

A voice assistant sounds like it must be some fundamentally new kind of AI — and that assumption is exactly what over-complicates people's designs. You imagine you need an "audio model" that ingests sound and speaks back, and you go looking to rebuild your whole stack around it. For the common case, that's the wrong picture, and it leads to architectures far hairier than the job requires. The reality is almost anticlimactic: the same text LLM you've used all course sits at the centre, untouched, with a converter on each side.

The key fact to internalize is that a language model, at heart, still thinks in text tokens — the same currency from section 1. It doesn't natively hear. So audio has to be turned into text on the way in and, if you want a spoken reply, back into audio on the way out. Two small translators bookend the pipeline you already own:

• ASR / STT (speech-to-text) on the way in — Automatic Speech Recognition, e.g. a model like Whisper, converts the spoken audio into a plain-text transcript. From that instant on, it's just text, and everything you learned about prompting, RAG, and agents applies with zero changes.
• TTS (text-to-speech) on the way out — optional — turns the model's text answer back into a voice. If your product only needs to read voice (a meeting-notes summarizer), you skip this entirely; you add it only when you need to speak back.

Think of the LLM as a brilliant colleague who only reads and writes notes: a receptionist (ASR) transcribes what callers say into notes, your colleague reasons over the notes exactly as always, and a narrator (TTS) reads the written reply aloud. The colleague's job never changed — you just added a transcriber and a narrator at the doors. This "bookend" framing is powerful because it tells you where every hard problem lives: transcription accuracy and speech quality are the audio pieces, while correctness, grounding, and tool use remain the text pieces you already know how to build and test.

Where this mental model quietly breaks — and it's worth flagging. Bookending is right for turn-based tasks (transcribe a recording, answer, maybe speak). It starts to strain for truly real-time, low-latency conversation — think interruptions, back-channel "mm-hm," barge-in — where the round trip of transcribe → think → synthesize adds noticeable lag, and newer end-to-end speech systems blur the three boxes into one. For nearly everything you'll build first, though, the pipeline below is the correct and simplest answer; reach for the fancier stuff only when latency, not capability, becomes the bottleneck.

LLMs are text models, so audio enters via a speech-to-text (ASR) step first (e.g. Whisper), then the transcript flows through the normal text pipeline; a text-to-speech step can voice the reply. The LLM sits in the middle.

🎯 Interview practice interview

The interview questions this topic gets asked — worked, with code. For the full pattern catalog see A9 · Big Tech AI-engineering patterns.

python# PIPELINE: image/PDF -> vision model with a Pydantic schema -> validated object
# SCALE:    batch pages; cache by document hash; fall back to human review on low confidence
# COST:     images cost tokens (hundreds-thousands each) -> resize, budget

pythonimport base64
data = base64.standard_b64encode(open("chart.png","rb").read()).decode()
content = [
    {"type":"image","source":{"type":"base64",
        "media_type":"image/png","data":data}},
    {"type":"text","text":"What is the peak value in this chart?"},
]

Checkpoint advanced

import os

SUPPORTED = {
    ".png": "image/png",
    ".jpg": "image/jpeg", ".jpeg": "image/jpeg",
    ".webp": "image/webp",
    ".gif": "image/gif",
}

def media_type_for(path):
    ext = os.path.splitext(path)[1].lower()
    if ext not in SUPPORTED:
        raise ValueError(f"unsupported image type: {ext!r}")
    return SUPPORTED[ext]

print(media_type_for("chart.PNG"))   # image/png
print(media_type_for("scan.jpeg"))   # image/jpeg
try:
    media_type_for("notes.pdf")
except ValueError as e:
    print(e)                         # unsupported image type: '.pdf'

import base64, os

def media_type_for(path):
    return {"png":"image/png","jpg":"image/jpeg","jpeg":"image/jpeg"}[
        os.path.splitext(path)[1].lower().lstrip(".")]

def image_block(path):
    with open(path, "rb") as f:
        data = base64.standard_b64encode(f.read()).decode()
    return {"type": "image",
            "source": {"type": "base64",
                       "media_type": media_type_for(path),
                       "data": data}}

# --- demo: write raw bytes, encode, verify round-trip ---
with open("tiny.png", "wb") as f:
    f.write(b"\x89PNG\r\n\x1a\n demo bytes")
blk = image_block("tiny.png")
print(blk["source"]["media_type"])                       # image/png
print(base64.b64decode(blk["source"]["data"])[:4])       # b'\x89PNG'

import base64, anthropic

client = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY from env

with open("chart.png", "rb") as f:
    data = base64.standard_b64encode(f.read()).decode()

msg = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {
                "type": "base64", "media_type": "image/png", "data": data}},
            {"type": "text", "text": "What trend does this chart show? Give the peak value."},
        ],
    }],
)
print(msg.content[0].text)

import math

TILE = 512          # px per tile edge
TOKENS_PER_TILE = 170

def approx_image_tokens(w, h):
    tiles = math.ceil(w / TILE) * math.ceil(h / TILE)
    return tiles * TOKENS_PER_TILE

def maybe_downscale(w, h, budget=700):
    est = approx_image_tokens(w, h)
    if est <= budget:
        return (w, h, est, "send as-is")
    # scale so estimated tokens fit the budget
    scale = math.sqrt(budget / est)
    nw, nh = int(w * scale), int(h * scale)
    return (nw, nh, approx_image_tokens(nw, nh), "downscaled")

print(approx_image_tokens(512, 512))     # 170
print(maybe_downscale(2048, 2048))       # (..., 'downscaled') -- big image trimmed
print(maybe_downscale(400, 300))         # (400, 300, 170, 'send as-is')

import math

def caption(image_path):
    # OFFLINE STUB. Real version: client.messages.create(...) on the image,
    # then use msg.content[0].text as the caption.
    return {"invoice.png": "invoice total 42 dollars acme",
            "chart.png":   "line chart sales peak q3"}[image_path]

def embed(text):                       # toy bag-of-words vector (deterministic)
    vocab = ["invoice","total","chart","sales","peak","acme"]
    return [text.count(w) for w in vocab]

def cosine(a, b):
    dot = sum(x*y for x, y in zip(a, b))
    na = math.sqrt(sum(x*x for x in a)); nb = math.sqrt(sum(y*y for y in b))
    return dot / (na*nb) if na and nb else 0.0

# index images by their captions
index = [(p, embed(caption(p))) for p in ["invoice.png", "chart.png"]]
q = embed("what was the sales peak")
best = max(index, key=lambda it: cosine(q, it[1]))
print("best match:", best[0])          # chart.png

import base64, json, anthropic          # SDK part needs an API key

def extract_invoice(path):
    client = anthropic.Anthropic()
    with open(path, "rb") as f:
        data = base64.standard_b64encode(f.read()).decode()
    msg = client.messages.create(
        model="claude-opus-4-8", max_tokens=1024,
        messages=[{"role": "user", "content": [
            {"type": "image", "source": {"type": "base64",
                "media_type": "image/png", "data": data}},
            {"type": "text", "text":
                "Extract JSON with keys: vendor (str), total (number), "
                "currency (str). Reply with JSON only."},
        ]}],
    )
    return json.loads(msg.content[0].text)

# ---- offline: validate whatever came back (runs as-is) ----
REQUIRED = {"vendor": str, "total": (int, float), "currency": str}

def validate_invoice(obj):
    for key, typ in REQUIRED.items():
        if key not in obj:
            raise ValueError(f"missing field: {key}")
        if not isinstance(obj[key], typ):
            raise TypeError(f"{key} has wrong type")
    return obj

print(validate_invoice({"vendor": "Acme", "total": 42.0, "currency": "USD"}))
try:
    validate_invoice({"vendor": "Acme"})
except ValueError as e:
    print("rejected:", e)              # rejected: missing field: total

Knowledge check check yourself