AI EngineeringZero to ProductionHome·About·Contact
AWS AI Automation · Chapter W13

Serverless orchestration

The automation payoff is things happening without you. Lambda runs the code, EventBridge routes events, and Step Functions sequences multi-step AI pipelines with retries.

⏱️ ~2 hours🧪 3 labs🎯 Advanced
⚙️ To run this for realThe code here is complete and correct as written. To actually execute it you'll need:
  • AWS credentials (aws configure) + pip install boto3
Reading and learning works without any of this — run when you're ready. IDs/ARNs in examples are placeholders; swap in your own.

Learning objectives

  • Trigger AI processing from an S3 upload with Lambda + EventBridge.
  • Orchestrate a multi-step AI pipeline with Step Functions.
  • Handle the async services (Textract/Transcribe) without polling by hand.
  • Reason about retries, error paths, and idempotency in AI automation.
▶ Runnable companionEvery code block in this lesson is also a standalone file under code/aws13-serverless-orchestration/ in the course, with a README on how to run it. Read here, run there — no need to copy-paste.

Event-driven AI advanced

The automation payoff is things happen without you. A document lands in S3 → a pipeline extracts, classifies, and stores it, with no human trigger. Lambda runs the code, EventBridge routes the events, and Step Functions sequences multi-step flows with built-in retries.

S3 → Lambda trigger advanced

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.
Lab W13.1 
s3_trigger.py# Lambda triggered by an S3 put. It kicks off async Textract and returns;
# a second trigger (Textract -> SNS -> Lambda) handles the result.
import boto3, urllib.parse
tx = boto3.client("textract")

def lambda_handler(event, context):
    rec = event["Records"][0]["s3"]
    bucket = rec["bucket"]["name"]
    key = urllib.parse.unquote_plus(rec["object"]["key"])

    job = tx.start_document_analysis(
        DocumentLocation={"S3Object": {"Bucket": bucket, "Name": key}},
        FeatureTypes=["FORMS", "TABLES"],
        NotificationChannel={
            "SNSTopicArn": "arn:aws:sns:us-east-1:123456789012:textract-done",
            "RoleArn": "arn:aws:iam::123456789012:role/textract-sns",
        },
    )
    return {"jobId": job["JobId"]}
▶ How this works

This is a Lambda function — a small piece of code AWS runs for you, automatically, with no server to manage. It runs the instant a file is uploaded to an S3 bucket (S3 is AWS's file storage). Its job here is to start reading the document, then get out of the way: document analysis can take a while, so it kicks off the work and returns immediately instead of waiting.

  1. tx = boto3.client("textract") — boto3 is the AWS toolkit for Python. This line opens a connection to Textract, the AWS service that reads text, forms, and tables out of documents (PDFs, scans).
  2. def lambda_handler(event, context): is the entry point AWS calls when the function fires. AWS hands it an event — a bundle of data describing what happened. For an S3 upload, that event contains which bucket and which file.
  3. The next three lines dig the details out of the event: the bucket name and the file key (its path/name). unquote_plus undoes URL-encoding so a name like my%20file.pdf becomes my file.pdf.
  4. tx.start_document_analysis(...) tells Textract "start reading this file." FeatureTypes=["FORMS", "TABLES"] asks it to also pull out form fields and tables, not just plain text.
  5. The NotificationChannel block is the clever part: instead of waiting, we tell Textract "when you're done, post a message to this SNS topic" (SNS is AWS's notification/messaging service). A separate Lambda listens on that topic and handles the finished result later.

What the output means: The function returns {"jobId": job["JobId"]} — just a tracking number for the Textract job. It does not return the extracted text; that arrives later via the SNS notification. Returning fast like this is normal and correct for event-driven code.

Try this: Trace what happens if two files are uploaded at once: AWS simply runs two copies of this Lambda in parallel, each with its own event. That automatic scaling is the whole point of "serverless."

No hand-rolled pollingIn W10 we noted big-document Textract is async. Here SNS notifies a second Lambda when the job finishes — event-driven, not a while True: sleep() loop. This is the production shape.

Step Functions: sequence the steps expert

A Step Functions state machine is your pipeline as JSON: extract → classify with Claude → store, with automatic retries and a catch path per state. It is the durable orchestration layer you would otherwise hand-code.

Lab W13.2 
pipeline.asl.json{
  "Comment": "Doc intelligence pipeline",
  "StartAt": "Extract",
  "States": {
    "Extract":  { "Type": "Task", "Resource": "arn:aws:lambda:...:extract",
                  "Retry": [{"ErrorEquals": ["States.ALL"], "MaxAttempts": 3}],
                  "Next": "Classify" },
    "Classify": { "Type": "Task", "Resource": "arn:aws:lambda:...:classify-with-claude",
                  "Catch": [{"ErrorEquals": ["States.ALL"], "Next": "DeadLetter"}],
                  "Next": "Store" },
    "Store":    { "Type": "Task", "Resource": "arn:aws:lambda:...:store", "End": true },
    "DeadLetter": { "Type": "Task", "Resource": "arn:aws:lambda:...:dlq", "End": true }
  }
}
▶ How this works

This is not program code — it's a configuration file (JSON) that describes a Step Functions state machine. Think of it as a flowchart written as text: AWS reads it and runs your steps in order, one after another, keeping track of where it is. Each step is a "state." This one wires up a document pipeline: Extract → Classify → Store, with a safety exit if something breaks.

  1. "StartAt": "Extract" names the first step. "States" then lists every step by name.
  2. Each step has "Type": "Task" (do a unit of work) and a "Resource" — the ARN (AWS's unique ID) of the Lambda function that does that step's work. So Extract runs one Lambda, Classify runs another (classify-with-claude), and so on.
  3. "Next": "Classify" is the arrow between steps: after Extract finishes, go to Classify. The last real step uses "End": true to stop.
  4. "Retry" on Extract means: if it fails, automatically try again up to "MaxAttempts": 3 times before giving up. You get this reliability for free — no retry code to write.
  5. "Catch" on Classify is the safety net: "ErrorEquals": ["States.ALL"] means "on any error, jump to DeadLetter" instead of crashing the whole pipeline. DeadLetter is a step that parks failed documents somewhere safe for a human to look at.

What the output means: Nothing prints — this file is uploaded to AWS as the definition of the pipeline. When you later start an execution, AWS walks these states in order and shows you a live diagram of which step is running, succeeded, retried, or failed.

Try this: Follow the two possible paths with your finger: the happy path is Extract → Classify → Store → End; the failure path is Classify → (error) → DeadLetter. Every state either points to a "Next" or says "End": true.

Invoke and observe expert

Lab W13.3 
start_sfn.pyimport boto3, json
sfn = boto3.client("stepfunctions", region_name="us-east-1")

run = sfn.start_execution(
    stateMachineArn="arn:aws:states:us-east-1:123456789012:stateMachine:doc-intel",
    input=json.dumps({"bucket": "my-docs", "key": "invoice.pdf"}),
)
print("execution:", run["executionArn"])
▶ How this works

This tiny script manually starts the pipeline you defined in Lab W13.2. In production the pipeline usually starts on its own (an upload event), but this is how you'd trigger one run yourself — for testing, or from another program.

  1. sfn = boto3.client("stepfunctions", region_name="us-east-1") opens a connection to the Step Functions service in a specific AWS region.
  2. sfn.start_execution(...) says "run one instance of this pipeline now." stateMachineArn is the ID of the pipeline (the state machine from Lab W13.2).
  3. input=json.dumps({...}) is the starting data handed to the first step. json.dumps turns the Python dictionary into a JSON text string, because that's the format Step Functions expects. Here we tell it which document to process: bucket my-docs, file invoice.pdf.
  4. print("execution:", run["executionArn"]) shows the ID of this specific run so you can find it in the AWS console.

What the output means: You get one line like execution: arn:aws:states:...:execution:doc-intel:abc123. That ARN is a receipt for this run — paste it into the Step Functions console to watch the Extract → Classify → Store steps light up in real time.

Try this: Change the "key" to a different filename and start it again — each call creates a brand-new, independently tracked execution with its own ARN.

Design for idempotencyEvents can fire more than once. Make each step idempotent (e.g. key DynamoDB writes by document id) so a retry never double-processes. This is the single most common bug in event-driven AI pipelines.

Exercise W13.1 — Wire the pipeline

Context: A pipeline is only real once it survives the console: an upload triggers it end to end, and a forced failure proves the error path actually catches. This exercise wires and then breaks the flow on purpose.

Your task: Connect S3 upload → Lambda → Step Functions (extract → Claude classify → DynamoDB store). Upload a document, watch the execution in the Step Functions console, then force an error in the classify step and confirm the catch path sends it to the dead-letter state.

Requirements:

  • An S3 upload triggers a Lambda that starts the Step Functions execution
  • The state machine runs extract → Claude classify → DynamoDB store in order
  • Watch a successful execution graph in the Step Functions console
  • Deliberately force a failure in the classify state
  • Confirm the Catch routes the failed run to the dead-letter state
  • Verify the DynamoDB write only happens on the success path, not the failed one

💡 Hint: Prove both paths: one clean run that reaches the store state, and one forced-error run whose Catch diverts it to dead-letter before any write.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Parse an S3 trigger eventBeginner

Context: Event-driven pipelines usually begin when a file lands in S3. The very first thing your Lambda must do is read the bucket and key out of the event S3 delivers — the shape of that event is fixed and worth knowing cold.

Your task: Write a Lambda handler that reads the bucket name and object key from an S3 event record. Runs offline as a pure handler.

Requirements:

  • Accept the standard (event, context) handler signature
  • S3 delivers records under event["Records"]; read the first record's s3 block
  • Pull the bucket from s3.bucket.name and the key from s3.object.key
  • Return (or print) the extracted bucket and key
  • Testable offline by passing a hand-built S3-shaped event dict

💡 Hint: The event nesting is Records[0].s3.bucket.name and Records[0].s3.object.key; hard-code a sample event to test without AWS.

Show solution

S3 delivers events under event["Records"][0]["s3"]; pull bucket and object key from there.

def lambda_handler(event, context):
    rec = event["Records"][0]["s3"]
    bucket = rec["bucket"]["name"]
    key = rec["object"]["key"]
    print("new object:", bucket, key)
    return {"bucket": bucket, "key": key}

evt = {"Records": [{"s3": {"bucket": {"name": "docs"},
                          "object": {"key": "in/a.pdf"}}}]}
print(lambda_handler(evt, None))   # {'bucket':'docs','key':'in/a.pdf'}
Exercise 2 · Async Textract with an SNS notificationIntermediate

Context: Multi-page documents can't be OCR'd in one synchronous call, so Textract offers an async job that notifies you on completion via SNS — no hand-rolled polling loop, which is the serverless-friendly way to wait.

Your task: Start Textract with start_document_analysis and a NotificationChannel so SNS fires when the job completes.

Requirements:

  • Call start_document_analysis on a boto3 textract client with DocumentLocation pointing at the S3 object
  • Request the FeatureTypes you need (e.g. FORMS, TABLES)
  • Supply a NotificationChannel with an SNSTopicArn and the RoleArn Textract assumes to publish
  • Capture the returned JobId for correlating the later completion notification
  • Understand this replaces polling: SNS signals completion instead

💡 Hint: The async start returns only a JobId; the transcript of work arrives later, and the NotificationChannel is what pushes you the "done" signal.

Show solution

For multi-page docs, kick off the async job and let SNS notify you instead of hand-rolled polling.

import boto3

tx = boto3.client("textract", region_name="us-east-1")
resp = tx.start_document_analysis(
    DocumentLocation={"S3Object": {"Bucket": "docs", "Name": "in/a.pdf"}},
    FeatureTypes=["FORMS", "TABLES"],
    NotificationChannel={"SNSTopicArn": "arn:aws:sns:us-east-1:123:tx-done",
                         "RoleArn": "arn:aws:iam::123:role/textract-sns"},
)
print(resp["JobId"])
Exercise 3 · Author a Step Functions state machineAdvanced

Context: Step Functions lets you chain Lambdas into a resilient workflow declared as data (ASL JSON), with retries and error catches built in — so failure handling lives in configuration, not scattered try/except code.

Your task: Write the Amazon States Language JSON for a linear flow Extract → Classify → Store, with a retry and a dead-letter catch on the first state. Validate it offline.

Requirements:

  • Set StartAt and define each stage as a Task state under States
  • Chain the stages with Next, and mark the terminal state with End: true
  • On the Extract state, add a Retry block (e.g. States.ALL, MaxAttempts)
  • On the Extract state, add a Catch that routes failures to a dead-letter Fail state
  • Confirm it is valid JSON offline (e.g. serialize it) — no execution needed

💡 Hint: ASL is just a dict you can build in Python and json.dumps; Retry and Catch are per-state lists that give resilience with zero custom code.

Show solution

ASL chains Task states with Next/End; Retry and Catch give resilience without custom code.

import json

asl = {
  "StartAt": "Extract",
  "States": {
    "Extract": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123:function:extract",
      "Retry": [{"ErrorEquals": ["States.ALL"], "MaxAttempts": 3}],
      "Catch": [{"ErrorEquals": ["States.ALL"], "Next": "DeadLetter"}],
      "Next": "Classify"},
    "Classify": {"Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123:function:classify",
      "Next": "Store"},
    "Store": {"Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123:function:store",
      "End": True},
    "DeadLetter": {"Type": "Fail", "Error": "ExtractFailed"},
  },
}
print(json.dumps(asl)[:60], "...")   # valid JSON
Exercise 4 · Start an executionExpert

Context: Once a state machine exists, you kick off a run with a single API call and a JSON input. That call returns an execution ARN you use to track and reference the specific run.

Your task: Use Step Functions start_execution to launch the state machine with a JSON input.

Requirements:

  • Call start_execution on a boto3 stepfunctions client
  • Pass the stateMachineArn of the workflow to run
  • Serialize the run input to a JSON string for the input argument
  • Read the returned executionArn that identifies this run

💡 Hint: The input is a JSON string, not a dict — json.dumps your payload; the response's executionArn handles this specific execution.

Show solution

start_execution takes the state-machine ARN and a JSON-string input and returns an executionArn.

import boto3, json

sfn = boto3.client("stepfunctions", region_name="us-east-1")
resp = sfn.start_execution(
    stateMachineArn="arn:aws:states:us-east-1:123:stateMachine:doc-pipeline",
    input=json.dumps({"bucket": "docs", "key": "in/a.pdf"}),
)
print(resp["executionArn"])
Exercise 5 · Idempotent processingProfessional

Context: Serverless triggers deliver at-least-once, so the same event can fire twice. Any step with side effects must be idempotent — keying writes by document id makes a re-delivery a harmless no-op instead of a double-post.

Your task: Implement process_once(doc_id, store) so re-processing the same document id is a no-op. Runs offline.

Requirements:

  • Check whether doc_id is already recorded in the store before doing work
  • On a first sighting, record the id and return a processed result
  • On a repeat, return a skipped_duplicate result and do not write again
  • The store is keyed by id (a dict/set stands in for DynamoDB) so the guard is O(1)
  • Demonstrate the second call on the same id is skipped, offline

💡 Hint: The dedupe key is the document id; the presence check must happen before the write so a redelivery never repeats the effect.

Show solution

At-least-once delivery means duplicates; keying by id makes reprocessing safe.

def process_once(doc_id, store):
    if doc_id in store:
        return {"status": "skipped_duplicate", "id": doc_id}
    store[doc_id] = {"processed": True}      # idempotent write keyed by id
    return {"status": "processed", "id": doc_id}

seen = {}
print(process_once("doc-1", seen))   # processed
print(process_once("doc-1", seen))   # skipped_duplicate
Exercise 6 · A resilient invoice-ingest pipelineIndustry scenario

Context: Invoices land in S3 at unpredictable volume, and the pipeline must survive transient failures without ever posting an invoice twice. That means composing triggers, Step Functions retry/catch, and an idempotency guard into one hands-off flow.

Your task: Design the event-driven pipeline (S3 → Lambda → async Textract → SNS → Step Functions) and encode the failure policy: retry transient errors 3×, dead-letter after that, and never double-post. Model the decision offline.

Requirements:

  • The decision takes a document id, attempt number, and error state (plus the dedupe store)
  • Guard idempotency first: a known doc_id yields skip_duplicate
  • A transient error under the attempt cap yields retry (Step Functions Retry)
  • An error at/after the cap yields dead_letter (the Catch to a DLQ)
  • A clean run records the id and returns posted
  • Runs offline — it models the retry/idempotency decision, it doesn't call AWS

💡 Hint: Order the checks so the idempotency guard wins over retry, and retry wins over dead-letter; the attempt count versus a max is what separates "try again" from "give up".

Show solution

Compose triggers + Step Functions retry/catch + idempotency for a hands-off, resilient pipeline.

def handle(doc_id, attempt, error, store, max_attempts=3):
    if doc_id in store:
        return "skip_duplicate"               # idempotency guard
    if error and attempt < max_attempts:
        return "retry"                        # transient -> Step Functions Retry
    if error:
        return "dead_letter"                  # exhausted -> Catch to DLQ
    store[doc_id] = True
    return "posted"

store = {}
print(handle("A1", 1, "Throttling", store))   # retry
print(handle("A1", 3, "Throttling", store))   # dead_letter
print(handle("A1", 1, None, store))           # posted
print(handle("A1", 1, None, store))           # skip_duplicate

✓ Checkpoint — you can move on when you can…

  • Trigger a Lambda from an S3 upload.
  • Handle async Textract via SNS instead of polling.
  • Express a multi-step pipeline as a Step Functions state machine with retries + catch.
  • Explain why idempotency matters in event-driven AI.

Knowledge check check yourself

✓ Knowledge check

In the S3 -> Lambda -> Textract flow, why does the Lambda start_document_analysis and return immediately instead of waiting, and how is the result handled?

Show answer
Big-document Textract is async and can take a while, so the Lambda kicks off the job and returns fast (returning just the jobId). Textract posts to an SNS topic when finished, and a second Lambda listens on that topic to handle the result -- event-driven, not a hand-rolled polling loop.
✓ Knowledge check

In the Step Functions state machine, what is the difference between a state's Retry and its Catch, and why is idempotency stressed for event-driven pipelines?

Show answer
Retry automatically re-runs a failing state up to MaxAttempts; Catch routes any error to another state (e.g. a DeadLetter) instead of crashing the pipeline. Because events (and retries) can fire more than once, each step must be idempotent -- e.g. keying DynamoDB writes by document id -- so a replay never double-processes.

More in AWS AI automation

W1AWS AI foundations W10Textract & Comprehend W11Vision & speech W12Kendra & Amazon Q W14Observability & cost W2Bedrock & Claude W3Tool use & structured output W4Knowledge Bases (managed RAG)
© 2026 studybydoing.in · AI Engineering: Zero to Production · All rights reserved. · About · Privacy Policy · Terms · Contact
Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.
© studybydoing.in