AI EngineeringZero to ProductionHome·About·Contact
AWS AI Automation · Chapter W11

Vision & speech

Rekognition, Transcribe, Translate, and Polly are narrow, cheap media APIs. Compose them with Claude for reasoning and you have a full vision/speech automation.

⏱️ ~1.5 hours🧪 3 labs🎯 Intermediate
⚙️ To run this for realThe code here is complete and correct as written. To actually execute it you'll need:
  • AWS credentials (aws configure) + pip install boto3
Reading and learning works without any of this — run when you're ready. IDs/ARNs in examples are placeholders; swap in your own.

Learning objectives

  • Detect labels, text, and moderation flags in images with Rekognition.
  • Transcribe audio to text and translate it across languages.
  • Synthesize speech with Polly.
  • Compose these into a media-processing automation.
▶ Runnable companionEvery code block in this lesson is also a standalone file under code/aws11-media-vision-speech/ in the course, with a README on how to run it. Read here, run there — no need to copy-paste.

The media AI services intermediate

Four pay-per-use services cover vision and speech: Rekognition (images/video), Transcribe (speech→text), Translate (text→text across languages), and Polly (text→speech). Like Textract, they are narrow and cheap — and pair well with Claude for the reasoning step.

Rekognition: see an image intermediate

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.
Lab W11.1
rekognition.pyimport boto3
rek = boto3.client("rekognition", region_name="us-east-1")
img = {"S3Object": {"Bucket": "my-media", "Name": "photo.jpg"}}

labels = rek.detect_labels(Image=img, MaxLabels=5)
print([(l["Name"], round(l["Confidence"], 1)) for l in labels["Labels"]])

# content moderation — useful as an upload guardrail
mod = rek.detect_moderation_labels(Image=img)
print("flagged:", [m["Name"] for m in mod["ModerationLabels"]])
[('Truck', 99.4), ('Vehicle', 99.4), ('Wheel', 92.1), ('Machine', 92.1), ('Person', 88.0)]
flagged: []
▶ How this works

Rekognition is AWS's vision service: you hand it an image and it tells you what's in the picture. This block points it at a photo sitting in an S3 bucket (AWS's file storage) and asks two questions — "what objects are here?" and "is this image safe?". Nothing is trained by you; the model already knows how to recognise common things.

  1. boto3.client("rekognition", region_name="us-east-1") creates the connection object (AWS calls it a client) that lets Python send commands to Rekognition in the us-east-1 data-centre region. boto3 is the official AWS library for Python.
  2. img = {"S3Object": {"Bucket": "my-media", "Name": "photo.jpg"}} does not upload anything — it just says "the picture is the file photo.jpg inside the bucket named my-media". You point at where the file already lives.
  3. rek.detect_labels(Image=img, MaxLabels=5) asks "what do you see?" and caps the answer at the 5 most confident guesses. Each guess ("label") comes with a confidence score from 0–100.
  4. The print([(l["Name"], round(l["Confidence"], 1)) ...]) line loops over the labels and builds a tidy list of (name, confidence) pairs, rounding the score to one decimal.
  5. rek.detect_moderation_labels(Image=img) is a separate check for unsafe content (violence, nudity, etc.). It returns a list of flags — useful as a guardrail before you let a user's upload through.

What the output means: The first line lists what Rekognition saw, most-confident first — e.g. ('Truck', 99.4) means it is 99.4% sure the photo contains a truck. flagged: [] is an empty list, meaning nothing was flagged as unsafe — the image passed the moderation check.

Try this: Change MaxLabels=5 to MaxLabels=2 and you'll get only the top two guesses. A non-empty flagged list is your cue to reject or hold the upload for review.

Transcribe + Translate advanced

Transcribe is asynchronous: start a job over an S3 audio file, poll for completion, read the transcript from S3. Then Translate can localize it.

Lab W11.2
transcribe.pyimport boto3
ts = boto3.client("transcribe", region_name="us-east-1")

ts.start_transcription_job(
    TranscriptionJobName="call-001",
    Media={"MediaFileUri": "s3://my-media/support-call.mp3"},
    MediaFormat="mp3",
    LanguageCode="en-US",
)
# poll get_transcription_job until COMPLETED, then fetch the transcript JSON from S3

tr = boto3.client("translate", region_name="us-east-1")
out = tr.translate_text(Text="The database is down.",
                        SourceLanguageCode="en", TargetLanguageCode="es")
print(out["TranslatedText"])
La base de datos está caída.
▶ How this works

This block chains two services. Transcribe turns recorded speech into written text (speech-to-text). Translate then rewrites text from one language into another. Transcribe is asynchronous: instead of answering instantly, you start a job, it works in the background, and you check back later — the way you might drop clothes at a laundromat and return when they're done.

  1. ts.start_transcription_job(...) kicks off the job. TranscriptionJobName="call-001" is a label you invent so you can find this job again later. Media={"MediaFileUri": "s3://..."} tells it which audio file (in S3) to transcribe.
  2. MediaFormat="mp3" and LanguageCode="en-US" tell it the file is an MP3 and the speech is US English — hints that make transcription more accurate.
  3. The comment line # poll get_transcription_job until COMPLETED marks the step you'd normally add: polling means repeatedly asking "done yet?" in a loop until the status reads COMPLETED, then reading the finished transcript (returned as a JSON file in S3).
  4. tr = boto3.client("translate", ...) opens a second client, this time for Translate.
  5. tr.translate_text(Text="The database is down.", SourceLanguageCode="en", TargetLanguageCode="es") asks it to turn that English sentence (en) into Spanish (es). The result lives under the "TranslatedText" key of the reply.

What the output means: La base de datos está caída. — the Spanish translation of "The database is down." Only the Translate step prints here; the Transcribe job runs in the background and is fetched separately.

Try this: Change TargetLanguageCode="es" to "fr" (French) or "de" (German) and re-run just the Translate part. The source stays English; only the target language changes.

Polly: speak expert

Lab W11.3
polly.pyimport boto3
polly = boto3.client("polly", region_name="us-east-1")

audio = polly.synthesize_speech(
    Text="Deployment complete. All systems healthy.",
    OutputFormat="mp3", VoiceId="Joanna",
)
with open("status.mp3", "wb") as f:
    f.write(audio["AudioStream"].read())
print("wrote status.mp3")
▶ How this works

Polly is the mirror image of Transcribe: it turns written text into spoken audio (text-to-speech). This block hands Polly a sentence and saves the spoken version as an MP3 file you can play.

  1. polly = boto3.client("polly", region_name="us-east-1") opens the connection to the Polly service.
  2. polly.synthesize_speech(Text="Deployment complete...", OutputFormat="mp3", VoiceId="Joanna") is the core call: give it the words, ask for MP3 audio, and pick a voiceJoanna is one of Polly's built-in US-English voices. It returns the audio as a stream of bytes, not a file yet.
  3. with open("status.mp3", "wb") as f: opens a new file for writing in binary mode ("wb") — binary because audio is raw bytes, not text. The with block automatically closes the file when done.
  4. f.write(audio["AudioStream"].read()) reads all the audio bytes Polly sent back and writes them into status.mp3 on your disk.

What the output means: It prints wrote status.mp3 and leaves a real, playable status.mp3 file in your folder that says "Deployment complete. All systems healthy." in Joanna's voice.

Try this: Swap VoiceId="Joanna" for "Matthew" (a male US voice) and re-run — you'll get the same words in a different voice. Change the Text= to hear your own message.

The full loopTranscribe a support call → Comprehend the sentiment (W10) → summarize with Claude (W2) → Polly a spoken summary. Each service does one thing; the automation is the composition. W13 wires it with Step Functions.

Exercise W11.1 — Voice-of-customer pipeline

Context: Support recordings are a goldmine, but raw audio is unsearchable and full of PII. The voice-of-customer pattern chains media services: transcribe the call, analyze and redact it, summarize it, and even speak the summary back.

Your task: Transcribe a recorded call, detect sentiment and PII with Comprehend, redact the PII, and have Claude write a one-paragraph summary — then output both the text and a Polly audio version.

Requirements:

  • Transcribe the call audio to text (async Transcribe job, then read the transcript)
  • Run Comprehend on the transcript for both sentiment and PII entities
  • Redact the detected PII from the text before it goes any further downstream
  • Have Claude on Bedrock write a one-paragraph summary of the redacted transcript
  • Render that summary to speech with Polly and save the audio
  • Produce both artifacts: the summary text and the audio file

💡 Hint: Redact off the PII spans Comprehend returns before you send anything to Claude, so no personal data reaches the summary or the audio.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Detect labels in an imageBeginner

Context: Rekognition is managed computer vision: hand it an image and it returns what it sees, with a confidence for each label. It is the fastest way to add object/scene understanding without training a model.

Your task: Use Rekognition detect_labels on an S3 image and print each detected label alongside its confidence.

Requirements:

  • Call detect_labels on a boto3 rekognition client with Image pointing at an S3Object
  • Cap the results with MaxLabels so you get the top few, not everything
  • Iterate the returned Labels list
  • Print each label's Name and its Confidence (round the confidence for readability)

💡 Hint: The response is a ranked Labels list; each item already carries both Name and Confidence.

Show solution

Rekognition returns Labels with Name and Confidence; MaxLabels caps the count.

import boto3

rek = boto3.client("rekognition", region_name="us-east-1")
resp = rek.detect_labels(
    Image={"S3Object": {"Bucket": "my-media", "Name": "photo.jpg"}},
    MaxLabels=5,
)
for lbl in resp["Labels"]:
    print(lbl["Name"], round(lbl["Confidence"], 1))
Exercise 2 · Translate text synchronouslyIntermediate

Context: Translate is a synchronous, one-call service: unlike Transcribe it returns immediately, which makes it the simplest building block for reaching users in their own language.

Your task: Use Translate translate_text to convert an English sentence to Spanish and print the translation.

Requirements:

  • Call translate_text on a boto3 translate client
  • Pass the source Text plus SourceLanguageCode="en" and TargetLanguageCode="es"
  • Read the result off the TranslatedText field of the single response
  • Understand this call is synchronous — no job to start or poll

💡 Hint: One request, one response — there is no async job here; the answer is on TranslatedText.

Show solution

Translate is synchronous: one call returns TranslatedText immediately.

import boto3

tr = boto3.client("translate", region_name="us-east-1")
resp = tr.translate_text(
    Text="Where is the nearest station?",
    SourceLanguageCode="en", TargetLanguageCode="es",
)
print(resp["TranslatedText"])
Exercise 3 · Text to speech with PollyAdvanced

Context: Polly is the mirror of Transcribe: it turns text into natural speech. It streams back raw audio bytes, so you choose a voice and format and write the stream to a file yourself.

Your task: Use Polly synthesize_speech to render text to an MP3 and save the returned audio stream to a file.

Requirements:

  • Call synthesize_speech on a boto3 polly client with the Text to speak
  • Request OutputFormat="mp3" and pick a VoiceId (e.g. Joanna)
  • The response's AudioStream is a readable byte stream, not a URL
  • Open a file in binary mode and write AudioStream.read() to disk

💡 Hint: The audio comes back as a streaming body; .read() it and write the bytes with "wb".

Show solution

Polly returns an AudioStream of bytes; pick a VoiceId and write the stream out.

import boto3

polly = boto3.client("polly", region_name="us-east-1")
resp = polly.synthesize_speech(
    Text="Your order has shipped.",
    OutputFormat="mp3", VoiceId="Joanna",
)
with open("out.mp3", "wb") as f:
    f.write(resp["AudioStream"].read())
print("wrote out.mp3")
Exercise 4 · Start an async transcription jobExpert

Context: Long audio can't be transcribed in a single blocking call, so Transcribe is asynchronous: you start a job, it works in the background, and the transcript eventually lands in S3. You must design for the wait.

Your task: Use Transcribe start_transcription_job on an S3 audio file, then show how you check progress with get_transcription_job — making the async, poll-for-completion nature explicit.

Requirements:

  • Call start_transcription_job on a boto3 transcribe client with a unique TranscriptionJobName
  • Point Media.MediaFileUri at the S3 audio and set MediaFormat and LanguageCode
  • Recognize the call returns immediately without a transcript
  • Poll get_transcription_job and read TranscriptionJob.TranscriptionJobStatus until it is COMPLETED
  • Note the finished transcript is delivered to S3, not inline in the start call

💡 Hint: Starting the job and getting the transcript are two different calls; treat status as something you poll, not something you receive up front.

Show solution

Transcribe is asynchronous: start the job, then poll for completion; the transcript lands in S3.

import boto3

ts = boto3.client("transcribe", region_name="us-east-1")
ts.start_transcription_job(
    TranscriptionJobName="call-42",
    Media={"MediaFileUri": "s3://my-media/call.mp3"},
    MediaFormat="mp3", LanguageCode="en-US",
)
# poll until COMPLETED:
job = ts.get_transcription_job(TranscriptionJobName="call-42")
print(job["TranscriptionJob"]["TranscriptionJobStatus"])
Exercise 5 · Content-moderation gate before publishingProfessional

Context: User-generated images are a liability the moment you publish them. Rekognition's moderation labels plus a confidence threshold turn raw scores into a hard publish/block decision — a guardrail you own in code.

Your task: Use Rekognition detect_moderation_labels and refuse to publish an image if any moderation label exceeds a confidence threshold. Keep the gate logic runnable offline.

Requirements:

  • The gate takes a list of moderation labels (each with Name and Confidence) and a threshold (e.g. 80.0)
  • Flag every label whose Confidence meets or exceeds the threshold
  • Return a decision such as {"publish": not flagged, "flagged": […]}
  • Publish only when nothing is flagged
  • Prove it offline with a Rekognition-shaped sample response; note the live call is detect_moderation_labels(…)["ModerationLabels"]

💡 Hint: Separate the decision from the API: the threshold comparison is pure and testable offline, and the real call just supplies the list of labels.

Show solution

Moderation acts as a guardrail; a threshold turns confidence scores into a publish/block decision.

def publish_ok(moderation_labels, threshold=80.0):
    flagged = [m["Name"] for m in moderation_labels
               if m["Confidence"] >= threshold]
    return {"publish": not flagged, "flagged": flagged}

# offline with a sample Rekognition-shaped response:
labels = [{"Name": "Violence", "Confidence": 92.5},
          {"Name": "Tobacco",  "Confidence": 40.0}]
print(publish_ok(labels))   # publish False, flagged ['Violence']
# In prod: rek.detect_moderation_labels(Image=...)['ModerationLabels']
Exercise 6 · A multilingual voice-note support pipelineIndustry scenario

Context: Callers leave voice notes in many languages, and a good support experience answers each one in the caller's own language. That means composing several narrow media services around Claude — speech-to-text, translate, reason, text-to-speech.

Your task: Design the multilingual voice-note pipeline: Transcribe (async) → Translate to English → Claude for intent → Polly reply in the caller's language. Model the language routing and the async wait offline.

Requirements:

  • Lay out the ordered steps as data (service call + its key arguments) so the plan is inspectable
  • Transcribe in the caller's language and show the poll-until-COMPLETED wait as an explicit step
  • Translate caller→English before reasoning, then English→caller before replying
  • Route the Polly VoiceId from the caller's language via a lookup, falling back to a default voice
  • Runs offline — it emits the plan, it does not make AWS calls

💡 Hint: Each stage is a distinct AWS service; model the flow as a list of (step, args) tuples and drive the voice choice off a language→voice map.

Show solution

Compose narrow media services around Claude: speech-to-text, translate, reason, text-to-speech. Each step is a distinct AWS service.

VOICE_BY_LANG = {"es": "Lucia", "en": "Joanna", "de": "Vicki"}

def pipeline_plan(caller_lang):
    return [
        ("transcribe.start_transcription_job", {"LanguageCode": caller_lang}),
        ("poll get_transcription_job", "until COMPLETED"),
        ("translate.translate_text", {"from": caller_lang, "to": "en"}),
        ("bedrock.converse", "classify intent"),
        ("translate.translate_text", {"from": "en", "to": caller_lang}),
        ("polly.synthesize_speech", {"VoiceId": VOICE_BY_LANG.get(caller_lang, "Joanna")}),
    ]

for step, arg in pipeline_plan("es"):
    print(step, arg)

✓ Checkpoint — you can move on when you can…

  • Detect labels and moderation flags with Rekognition.
  • Run an async Transcribe job and translate the result.
  • Synthesize speech with Polly.
  • Compose media services + Claude into one pipeline.

Knowledge check check yourself

✓ Knowledge check

Transcribe and Translate are chained in the lab, but they behave differently. Which one is asynchronous and what does that require of your code?

Show answer
Transcribe is asynchronous: you start_transcription_job, poll get_transcription_job until COMPLETED, then read the transcript JSON from S3. Translate answers synchronously in the call, returning TranslatedText immediately.
✓ Knowledge check

The chapter frames these media APIs as narrow services composed with Claude. Describe the 'full loop' example and the principle it illustrates.

Show answer
Transcribe a support call -> detect sentiment/PII with Comprehend -> summarize with Claude -> speak the summary with Polly. Each service does one narrow thing well; the automation is the composition of them (wired with Step Functions in W13).
© 2026 studybydoing.in · AI Engineering: Zero to Production · All rights reserved. · About · Privacy Policy · Terms · Contact
Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.
© studybydoing.in