Vision & speech

The media AI services intermediate

Here's the trap almost every team falls into first. A product manager says "we need the app to understand photos and voice notes," and an engineer immediately starts scoping a computer-vision model to train, a speech dataset to label, GPUs to rent. Three months and a five-figure bill later they have something that recognises cats 80% of the time. Meanwhile the competitor shipped the same feature in an afternoon by calling four AWS APIs that already do all of that — trained on far more data than any single team could gather. The lesson of this chapter: for common vision and speech tasks, you almost never build the model. You rent one, by the request.

Think of these four services as specialist contractors you can hire by the minute. You don't teach the electrician how electricity works; you hand them the job and they hand you back the result. Rekognition looks at a picture and tells you what's in it. Transcribe listens to audio and writes down the words. Translate takes text in one language and rewrites it in another. Polly takes text and speaks it aloud. Each does exactly one narrow thing, extremely well, for a fraction of a cent per call — because AWS amortises the training cost across millions of customers.

So why is this chapter in an AI engineering course if you're not building the models? Because the interesting work has moved up a level. These services are perception — they turn the messy analog world (pixels, sound waves) into clean text and structured data. But they don't reason. Rekognition can tell you a photo contains a "Truck" and a "Person"; it can't tell you "this looks like an insurance claim for a rear-end collision." That reasoning step is where Claude comes in. The pattern you'll build over and over: a narrow perception API converts the world into text, then an LLM reasons over that text, then maybe another narrow API turns the answer back into speech or another language. The engineering skill isn't training models — it's composing them.

The common mistake here is reaching for the LLM too early. Claude can read an image directly (you saw that in the Files/vision chapters), so why bother with Rekognition? Because for high-volume, well-defined tasks — moderating ten thousand uploads a day, transcribing every support call — a purpose-built service is cheaper, faster, and returns structured fields (confidence scores, timestamps, bounding boxes) that an LLM's free-text answer won't. Use the cheap narrow service for the bulk perception work; save the expensive general reasoner for the judgement calls. Knowing which tool to point at which part of the problem is the whole game.

Four pay-per-use services cover vision and speech: Rekognition (images/video), Transcribe (speech→text), Translate (text→text across languages), and Polly (text→speech). Like Textract, they are narrow and cheap — and pair well with Claude for the reasoning step.

Rekognition: see an image intermediate

Picture the feature nobody wants to own. Your app lets users upload profile photos, and one morning a moderator's queue has ten thousand new images in it. A human can eyeball maybe a few hundred an hour before their judgement goes numb — and the ones you actually need to catch (violence, nudity, hate symbols) are buried in a sea of selfies and sunsets. You cannot hire your way out of this, and you cannot ignore it, because the one bad image you miss is the one that ends up on the news. This is the problem Rekognition was built for: machine vision that never gets tired.

The mental model is simple. Rekognition is a pre-trained pair of eyes attached to an API. You show it an image (by pointing at a file already sitting in S3, or by sending the raw bytes), and it hands back a list of labels — the things it recognises — each with a confidence score from 0 to 100. "Truck, 99.4" means it is 99.4% sure the picture contains a truck. You didn't train it; AWS did, on a corpus larger than any one company could assemble. That's the entire value proposition: object and scene recognition you can add in an afternoon, priced per image.

Notice the two different questions the lab asks the same image, because they map to two different jobs. detect_labels answers "what is in this picture?" — useful for auto-tagging a photo library, routing an image to the right team, or making a catalogue searchable. detect_moderation_labels is a separate call that answers "is this picture safe?" — it looks specifically for categories like violence, nudity, and drugs. Keeping these as distinct calls is deliberate: tagging and safety are different concerns with different cost and latency budgets, and you often want the moderation gate to run on every upload while richer labelling runs only when needed.

The confidence score is where beginners go wrong, so anchor on it now. A label is not a yes/no fact; it is a probability. If you treat "Weapon, 55.0" the same as "Weapon, 99.0" you will either block innocent photos or wave through dangerous ones. The correct pattern — which the practice ladder makes you build — is to pick a threshold (say, 80.0) and turn the score into a decision: flag anything at or above the line, let the rest through. Choosing that threshold is a product judgement, not an AWS setting: a children's app sets it low and paranoid; a photo-editing tool sets it high and permissive. Rekognition gives you the number; you own where the line sits.

rekognition.pyimport boto3
rek = boto3.client("rekognition", region_name="us-east-1")
img = {"S3Object": {"Bucket": "my-media", "Name": "photo.jpg"}}

labels = rek.detect_labels(Image=img, MaxLabels=5)
print([(l["Name"], round(l["Confidence"], 1)) for l in labels["Labels"]])

# content moderation — useful as an upload guardrail
mod = rek.detect_moderation_labels(Image=img)
print("flagged:", [m["Name"] for m in mod["ModerationLabels"]])

[('Truck', 99.4), ('Vehicle', 99.4), ('Wheel', 92.1), ('Machine', 92.1), ('Person', 88.0)]
flagged: []

Transcribe + Translate advanced

The bug that catches everyone the first time looks like this: you write result = transcribe(audio), print result, and get back… almost nothing. No transcript, no error, just an object saying the job is "IN_PROGRESS." You assume the call failed and start debugging code that was working fine. The real issue is that you expected a synchronous answer from an asynchronous service — and the difference between those two words is the single most important idea in this section.

Here's the analogy that makes it stick. Translate is a vending machine: you put in an English sentence, the Spanish one drops out immediately, in the same call. Transcribe is a laundromat: you drop off your audio, get a ticket (the job name), leave, and come back later to collect the finished transcript. Nothing is wrong when the drop-off returns without your clothes — that's how it's supposed to work. Long audio can take seconds or minutes to process, far longer than a single web request should block, so Transcribe deliberately hands the work off to the background and lets you check on it later.

That "check on it later" is called polling, and it's a pattern you'll use for every long-running AWS job. You call start_transcription_job with a unique name and the S3 location of the audio; it returns right away. Then you call get_transcription_job in a loop, reading its status field, until that status flips to COMPLETED (or FAILED). Only then is the transcript ready — and even then it isn't in the response, it's written to a file in S3 that you fetch separately. Three steps, not one: start, poll, fetch. Beginners write the first step and wonder where their text went.

Why does the design split perception (Transcribe) from language (Translate) into two services at all? Because they're different problems with different shapes. Turning sound waves into words is a heavy, batch-friendly job that benefits from running async on big audio files. Rewriting finished text from one language to another is light and instant. Chaining them — transcribe a call, then translate the transcript — is a taste of the composition pattern that runs through this whole chapter: each service does one narrow thing, and the automation is how you wire them together. The common trap is forgetting the language codes differ between services (Transcribe wants a locale like en-US; Translate wants a bare language like en), a small mismatch that produces confusing errors until you notice it.

Transcribe is asynchronous: start a job over an S3 audio file, poll for completion, read the transcript from S3. Then Translate can localize it.

transcribe.pyimport boto3
ts = boto3.client("transcribe", region_name="us-east-1")

ts.start_transcription_job(
    TranscriptionJobName="call-001",
    Media={"MediaFileUri": "s3://my-media/support-call.mp3"},
    MediaFormat="mp3",
    LanguageCode="en-US",
)
# poll get_transcription_job until COMPLETED, then fetch the transcript JSON from S3

tr = boto3.client("translate", region_name="us-east-1")
out = tr.translate_text(Text="The database is down.",
                        SourceLanguageCode="en", TargetLanguageCode="es")
print(out["TranslatedText"])

La base de datos está caída.

Polly: speak expert

Consider the accessibility ticket you can't close by hand. Your product needs to speak — status alerts read aloud in a warehouse where nobody can look at a screen, an IVR phone menu, a language-learning app that pronounces words, an audiobook version of every article. The old way was to book a voice actor, record every phrase, and re-record the whole library every time the copy changed. That doesn't scale to text that's generated on the fly, and it certainly doesn't scale to the summaries a Claude-powered agent writes in real time. You need speech you can generate from arbitrary text, on demand. That's Polly.

Polly is the exact mirror image of Transcribe, and holding the two side by side is the fastest way to understand both. Transcribe goes sound → text; Polly goes text → sound. You hand it a string, tell it which voice to use (Polly ships dozens — Joanna and Matthew are US-English, and there are voices across many languages) and what audio format you want, and it synthesises spoken audio. Unlike Transcribe it answers synchronously — the vending-machine model again — so there's no job to poll.

The one detail that trips people up is what comes back: not a file, not a URL, but a stream of raw bytes. Audio isn't text, so you can't just print it or write it in text mode. The idiom is to open a file in binary mode ("wb"), .read() the whole audio stream, and write those bytes to disk as, say, an .mp3. Get the mode wrong — open in text mode — and Python will mangle the bytes and hand you a file that won't play. That "binary, not text" instinct is worth internalising; it applies to every media byte-stream you'll ever save.

Zoom back out and Polly is the closing bracket on the pattern this chapter keeps circling. The perception services (Transcribe, Rekognition) turn the world into text; Claude reasons over the text; Polly (and Translate) turn the answer back into the human's world — spoken, and in their language. That's what the "full loop" callout below is describing: transcribe a call, analyse it, summarise it with Claude, then speak the summary back. Each service is one narrow contractor; the product is the composition. Once you can see that shape, you can assemble a startling amount of "AI" functionality out of a handful of cheap, boring, reliable APIs — which is exactly how real production systems are built.

polly.pyimport boto3
polly = boto3.client("polly", region_name="us-east-1")

audio = polly.synthesize_speech(
    Text="Deployment complete. All systems healthy.",
    OutputFormat="mp3", VoiceId="Joanna",
)
with open("status.mp3", "wb") as f:
    f.write(audio["AudioStream"].read())
print("wrote status.mp3")

import boto3

rek = boto3.client("rekognition", region_name="us-east-1")
resp = rek.detect_labels(
    Image={"S3Object": {"Bucket": "my-media", "Name": "photo.jpg"}},
    MaxLabels=5,
)
for lbl in resp["Labels"]:
    print(lbl["Name"], round(lbl["Confidence"], 1))

import boto3

tr = boto3.client("translate", region_name="us-east-1")
resp = tr.translate_text(
    Text="Where is the nearest station?",
    SourceLanguageCode="en", TargetLanguageCode="es",
)
print(resp["TranslatedText"])

import boto3

polly = boto3.client("polly", region_name="us-east-1")
resp = polly.synthesize_speech(
    Text="Your order has shipped.",
    OutputFormat="mp3", VoiceId="Joanna",
)
with open("out.mp3", "wb") as f:
    f.write(resp["AudioStream"].read())
print("wrote out.mp3")

import boto3

ts = boto3.client("transcribe", region_name="us-east-1")
ts.start_transcription_job(
    TranscriptionJobName="call-42",
    Media={"MediaFileUri": "s3://my-media/call.mp3"},
    MediaFormat="mp3", LanguageCode="en-US",
)
# poll until COMPLETED:
job = ts.get_transcription_job(TranscriptionJobName="call-42")
print(job["TranscriptionJob"]["TranscriptionJobStatus"])

def publish_ok(moderation_labels, threshold=80.0):
    flagged = [m["Name"] for m in moderation_labels
               if m["Confidence"] >= threshold]
    return {"publish": not flagged, "flagged": flagged}

# offline with a sample Rekognition-shaped response:
labels = [{"Name": "Violence", "Confidence": 92.5},
          {"Name": "Tobacco",  "Confidence": 40.0}]
print(publish_ok(labels))   # publish False, flagged ['Violence']
# In prod: rek.detect_moderation_labels(Image=...)['ModerationLabels']

VOICE_BY_LANG = {"es": "Lucia", "en": "Joanna", "de": "Vicki"}

def pipeline_plan(caller_lang):
    return [
        ("transcribe.start_transcription_job", {"LanguageCode": caller_lang}),
        ("poll get_transcription_job", "until COMPLETED"),
        ("translate.translate_text", {"from": caller_lang, "to": "en"}),
        ("bedrock.converse", "classify intent"),
        ("translate.translate_text", {"from": "en", "to": caller_lang}),
        ("polly.synthesize_speech", {"VoiceId": VOICE_BY_LANG.get(caller_lang, "Joanna")}),
    ]

for step, arg in pipeline_plan("es"):
    print(step, arg)

Knowledge check check yourself