Vision & speech
Rekognition, Transcribe, Translate, and Polly are narrow, cheap media APIs. Compose them with Claude for reasoning and you have a full vision/speech automation.
- AWS credentials (
aws configure) +pip install boto3
Learning objectives
- Detect labels, text, and moderation flags in images with Rekognition.
- Transcribe audio to text and translate it across languages.
- Synthesize speech with Polly.
- Compose these into a media-processing automation.
code/aws11-media-vision-speech/ in the course, with a README on how to run it. Read here, run there — no need to copy-paste.The media AI services intermediate
Four pay-per-use services cover vision and speech: Rekognition (images/video), Transcribe (speech→text), Translate (text→text across languages), and Polly (text→speech). Like Textract, they are narrow and cheap — and pair well with Claude for the reasoning step.
Rekognition: see an image intermediate
rekognition.pyimport boto3
rek = boto3.client("rekognition", region_name="us-east-1")
img = {"S3Object": {"Bucket": "my-media", "Name": "photo.jpg"}}
labels = rek.detect_labels(Image=img, MaxLabels=5)
print([(l["Name"], round(l["Confidence"], 1)) for l in labels["Labels"]])
# content moderation — useful as an upload guardrail
mod = rek.detect_moderation_labels(Image=img)
print("flagged:", [m["Name"] for m in mod["ModerationLabels"]])
[('Truck', 99.4), ('Vehicle', 99.4), ('Wheel', 92.1), ('Machine', 92.1), ('Person', 88.0)]
flagged: []
Rekognition is AWS's vision service: you hand it an image and it tells you what's in the picture. This block points it at a photo sitting in an S3 bucket (AWS's file storage) and asks two questions — "what objects are here?" and "is this image safe?". Nothing is trained by you; the model already knows how to recognise common things.
boto3.client("rekognition", region_name="us-east-1")creates the connection object (AWS calls it a client) that lets Python send commands to Rekognition in theus-east-1data-centre region.boto3is the official AWS library for Python.img = {"S3Object": {"Bucket": "my-media", "Name": "photo.jpg"}}does not upload anything — it just says "the picture is the filephoto.jpginside the bucket namedmy-media". You point at where the file already lives.rek.detect_labels(Image=img, MaxLabels=5)asks "what do you see?" and caps the answer at the 5 most confident guesses. Each guess ("label") comes with a confidence score from 0–100.- The
print([(l["Name"], round(l["Confidence"], 1)) ...])line loops over the labels and builds a tidy list of(name, confidence)pairs, rounding the score to one decimal. rek.detect_moderation_labels(Image=img)is a separate check for unsafe content (violence, nudity, etc.). It returns a list of flags — useful as a guardrail before you let a user's upload through.
What the output means: The first line lists what Rekognition saw, most-confident first — e.g. ('Truck', 99.4) means it is 99.4% sure the photo contains a truck. flagged: [] is an empty list, meaning nothing was flagged as unsafe — the image passed the moderation check.
Try this: Change MaxLabels=5 to MaxLabels=2 and you'll get only the top two guesses. A non-empty flagged list is your cue to reject or hold the upload for review.
Transcribe + Translate advanced
Transcribe is asynchronous: start a job over an S3 audio file, poll for completion, read the transcript from S3. Then Translate can localize it.
transcribe.pyimport boto3
ts = boto3.client("transcribe", region_name="us-east-1")
ts.start_transcription_job(
TranscriptionJobName="call-001",
Media={"MediaFileUri": "s3://my-media/support-call.mp3"},
MediaFormat="mp3",
LanguageCode="en-US",
)
# poll get_transcription_job until COMPLETED, then fetch the transcript JSON from S3
tr = boto3.client("translate", region_name="us-east-1")
out = tr.translate_text(Text="The database is down.",
SourceLanguageCode="en", TargetLanguageCode="es")
print(out["TranslatedText"])
La base de datos está caída.
This block chains two services. Transcribe turns recorded speech into written text (speech-to-text). Translate then rewrites text from one language into another. Transcribe is asynchronous: instead of answering instantly, you start a job, it works in the background, and you check back later — the way you might drop clothes at a laundromat and return when they're done.
ts.start_transcription_job(...)kicks off the job.TranscriptionJobName="call-001"is a label you invent so you can find this job again later.Media={"MediaFileUri": "s3://..."}tells it which audio file (in S3) to transcribe.MediaFormat="mp3"andLanguageCode="en-US"tell it the file is an MP3 and the speech is US English — hints that make transcription more accurate.- The comment line
# poll get_transcription_job until COMPLETEDmarks the step you'd normally add: polling means repeatedly asking "done yet?" in a loop until the status readsCOMPLETED, then reading the finished transcript (returned as a JSON file in S3). tr = boto3.client("translate", ...)opens a second client, this time for Translate.tr.translate_text(Text="The database is down.", SourceLanguageCode="en", TargetLanguageCode="es")asks it to turn that English sentence (en) into Spanish (es). The result lives under the"TranslatedText"key of the reply.
What the output means: La base de datos está caída. — the Spanish translation of "The database is down." Only the Translate step prints here; the Transcribe job runs in the background and is fetched separately.
Try this: Change TargetLanguageCode="es" to "fr" (French) or "de" (German) and re-run just the Translate part. The source stays English; only the target language changes.
Polly: speak expert
polly.pyimport boto3
polly = boto3.client("polly", region_name="us-east-1")
audio = polly.synthesize_speech(
Text="Deployment complete. All systems healthy.",
OutputFormat="mp3", VoiceId="Joanna",
)
with open("status.mp3", "wb") as f:
f.write(audio["AudioStream"].read())
print("wrote status.mp3")
Polly is the mirror image of Transcribe: it turns written text into spoken audio (text-to-speech). This block hands Polly a sentence and saves the spoken version as an MP3 file you can play.
polly = boto3.client("polly", region_name="us-east-1")opens the connection to the Polly service.polly.synthesize_speech(Text="Deployment complete...", OutputFormat="mp3", VoiceId="Joanna")is the core call: give it the words, ask for MP3 audio, and pick a voice —Joannais one of Polly's built-in US-English voices. It returns the audio as a stream of bytes, not a file yet.with open("status.mp3", "wb") as f:opens a new file for writing in binary mode ("wb") — binary because audio is raw bytes, not text. Thewithblock automatically closes the file when done.f.write(audio["AudioStream"].read())reads all the audio bytes Polly sent back and writes them intostatus.mp3on your disk.
What the output means: It prints wrote status.mp3 and leaves a real, playable status.mp3 file in your folder that says "Deployment complete. All systems healthy." in Joanna's voice.
Try this: Swap VoiceId="Joanna" for "Matthew" (a male US voice) and re-run — you'll get the same words in a different voice. Change the Text= to hear your own message.
Exercise W11.1 — Voice-of-customer pipeline
Context: Support recordings are a goldmine, but raw audio is unsearchable and full of PII. The voice-of-customer pattern chains media services: transcribe the call, analyze and redact it, summarize it, and even speak the summary back.
Your task: Transcribe a recorded call, detect sentiment and PII with Comprehend, redact the PII, and have Claude write a one-paragraph summary — then output both the text and a Polly audio version.
Requirements:
- Transcribe the call audio to text (async Transcribe job, then read the transcript)
- Run Comprehend on the transcript for both sentiment and PII entities
- Redact the detected PII from the text before it goes any further downstream
- Have Claude on Bedrock write a one-paragraph summary of the redacted transcript
- Render that summary to speech with Polly and save the audio
- Produce both artifacts: the summary text and the audio file
💡 Hint: Redact off the PII spans Comprehend returns before you send anything to Claude, so no personal data reaches the summary or the audio.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Rekognition is managed computer vision: hand it an image and it returns what it sees, with a confidence for each label. It is the fastest way to add object/scene understanding without training a model.
Your task: Use Rekognition detect_labels on an S3 image and print each detected label alongside its confidence.
Requirements:
- Call
detect_labelson aboto3rekognitionclient withImagepointing at anS3Object - Cap the results with
MaxLabelsso you get the top few, not everything - Iterate the returned
Labelslist - Print each label's
Nameand itsConfidence(round the confidence for readability)
💡 Hint: The response is a ranked Labels list; each item already carries both Name and Confidence.
Show solution
Rekognition returns Labels with Name and Confidence; MaxLabels caps the count.
import boto3
rek = boto3.client("rekognition", region_name="us-east-1")
resp = rek.detect_labels(
Image={"S3Object": {"Bucket": "my-media", "Name": "photo.jpg"}},
MaxLabels=5,
)
for lbl in resp["Labels"]:
print(lbl["Name"], round(lbl["Confidence"], 1))
Context: Translate is a synchronous, one-call service: unlike Transcribe it returns immediately, which makes it the simplest building block for reaching users in their own language.
Your task: Use Translate translate_text to convert an English sentence to Spanish and print the translation.
Requirements:
- Call
translate_texton aboto3translateclient - Pass the source
TextplusSourceLanguageCode="en"andTargetLanguageCode="es" - Read the result off the
TranslatedTextfield of the single response - Understand this call is synchronous — no job to start or poll
💡 Hint: One request, one response — there is no async job here; the answer is on TranslatedText.
Show solution
Translate is synchronous: one call returns TranslatedText immediately.
import boto3
tr = boto3.client("translate", region_name="us-east-1")
resp = tr.translate_text(
Text="Where is the nearest station?",
SourceLanguageCode="en", TargetLanguageCode="es",
)
print(resp["TranslatedText"])
Context: Polly is the mirror of Transcribe: it turns text into natural speech. It streams back raw audio bytes, so you choose a voice and format and write the stream to a file yourself.
Your task: Use Polly synthesize_speech to render text to an MP3 and save the returned audio stream to a file.
Requirements:
- Call
synthesize_speechon aboto3pollyclient with theTextto speak - Request
OutputFormat="mp3"and pick aVoiceId(e.g.Joanna) - The response's
AudioStreamis a readable byte stream, not a URL - Open a file in binary mode and write
AudioStream.read()to disk
💡 Hint: The audio comes back as a streaming body; .read() it and write the bytes with "wb".
Show solution
Polly returns an AudioStream of bytes; pick a VoiceId and write the stream out.
import boto3
polly = boto3.client("polly", region_name="us-east-1")
resp = polly.synthesize_speech(
Text="Your order has shipped.",
OutputFormat="mp3", VoiceId="Joanna",
)
with open("out.mp3", "wb") as f:
f.write(resp["AudioStream"].read())
print("wrote out.mp3")
Context: Long audio can't be transcribed in a single blocking call, so Transcribe is asynchronous: you start a job, it works in the background, and the transcript eventually lands in S3. You must design for the wait.
Your task: Use Transcribe start_transcription_job on an S3 audio file, then show how you check progress with get_transcription_job — making the async, poll-for-completion nature explicit.
Requirements:
- Call
start_transcription_jobon aboto3transcribeclient with a uniqueTranscriptionJobName - Point
Media.MediaFileUriat the S3 audio and setMediaFormatandLanguageCode - Recognize the call returns immediately without a transcript
- Poll
get_transcription_joband readTranscriptionJob.TranscriptionJobStatusuntil it isCOMPLETED - Note the finished transcript is delivered to S3, not inline in the start call
💡 Hint: Starting the job and getting the transcript are two different calls; treat status as something you poll, not something you receive up front.
Show solution
Transcribe is asynchronous: start the job, then poll for completion; the transcript lands in S3.
import boto3
ts = boto3.client("transcribe", region_name="us-east-1")
ts.start_transcription_job(
TranscriptionJobName="call-42",
Media={"MediaFileUri": "s3://my-media/call.mp3"},
MediaFormat="mp3", LanguageCode="en-US",
)
# poll until COMPLETED:
job = ts.get_transcription_job(TranscriptionJobName="call-42")
print(job["TranscriptionJob"]["TranscriptionJobStatus"])
Context: User-generated images are a liability the moment you publish them. Rekognition's moderation labels plus a confidence threshold turn raw scores into a hard publish/block decision — a guardrail you own in code.
Your task: Use Rekognition detect_moderation_labels and refuse to publish an image if any moderation label exceeds a confidence threshold. Keep the gate logic runnable offline.
Requirements:
- The gate takes a list of moderation labels (each with
NameandConfidence) and a threshold (e.g.80.0) - Flag every label whose
Confidencemeets or exceeds the threshold - Return a decision such as
{"publish": not flagged, "flagged": […]} - Publish only when nothing is flagged
- Prove it offline with a Rekognition-shaped sample response; note the live call is
detect_moderation_labels(…)["ModerationLabels"]
💡 Hint: Separate the decision from the API: the threshold comparison is pure and testable offline, and the real call just supplies the list of labels.
Show solution
Moderation acts as a guardrail; a threshold turns confidence scores into a publish/block decision.
def publish_ok(moderation_labels, threshold=80.0):
flagged = [m["Name"] for m in moderation_labels
if m["Confidence"] >= threshold]
return {"publish": not flagged, "flagged": flagged}
# offline with a sample Rekognition-shaped response:
labels = [{"Name": "Violence", "Confidence": 92.5},
{"Name": "Tobacco", "Confidence": 40.0}]
print(publish_ok(labels)) # publish False, flagged ['Violence']
# In prod: rek.detect_moderation_labels(Image=...)['ModerationLabels']
Context: Callers leave voice notes in many languages, and a good support experience answers each one in the caller's own language. That means composing several narrow media services around Claude — speech-to-text, translate, reason, text-to-speech.
Your task: Design the multilingual voice-note pipeline: Transcribe (async) → Translate to English → Claude for intent → Polly reply in the caller's language. Model the language routing and the async wait offline.
Requirements:
- Lay out the ordered steps as data (service call + its key arguments) so the plan is inspectable
- Transcribe in the caller's language and show the poll-until-
COMPLETEDwait as an explicit step - Translate caller→English before reasoning, then English→caller before replying
- Route the Polly
VoiceIdfrom the caller's language via a lookup, falling back to a default voice - Runs offline — it emits the plan, it does not make AWS calls
💡 Hint: Each stage is a distinct AWS service; model the flow as a list of (step, args) tuples and drive the voice choice off a language→voice map.
Show solution
Compose narrow media services around Claude: speech-to-text, translate, reason, text-to-speech. Each step is a distinct AWS service.
VOICE_BY_LANG = {"es": "Lucia", "en": "Joanna", "de": "Vicki"}
def pipeline_plan(caller_lang):
return [
("transcribe.start_transcription_job", {"LanguageCode": caller_lang}),
("poll get_transcription_job", "until COMPLETED"),
("translate.translate_text", {"from": caller_lang, "to": "en"}),
("bedrock.converse", "classify intent"),
("translate.translate_text", {"from": "en", "to": caller_lang}),
("polly.synthesize_speech", {"VoiceId": VOICE_BY_LANG.get(caller_lang, "Joanna")}),
]
for step, arg in pipeline_plan("es"):
print(step, arg)
✓ Checkpoint — you can move on when you can…
- Detect labels and moderation flags with Rekognition.
- Run an async Transcribe job and translate the result.
- Synthesize speech with Polly.
- Compose media services + Claude into one pipeline.
Knowledge check check yourself
Transcribe and Translate are chained in the lab, but they behave differently. Which one is asynchronous and what does that require of your code?
Show answer
The chapter frames these media APIs as narrow services composed with Claude. Describe the 'full loop' example and the principle it illustrates.