Serverless orchestration

Event-driven AI advanced

The automation payoff is things happen without you. A document lands in S3 → a pipeline extracts, classifies, and stores it, with no human trigger. Lambda runs the code, EventBridge routes the events, and Step Functions sequences multi-step flows with built-in retries.

Picture the version you're trying to escape. You wrote an AI pipeline that works beautifully — you run it by hand, it downloads the file, calls Textract, calls Claude, writes the row. So you put it on a cron job that fires every five minutes, wakes up, lists the bucket, and asks "anything new?" Most of the time the answer is "no," and you've paid for a server to sit there asking a question with no answer. Worse: when 500 invoices arrive at 9am on the 1st of the month, your one poor script processes them one at a time while the queue backs up, and if the box crashes halfway through you have no idea which files it already handled. That whole shape — a machine you keep running, polling for work it usually doesn't have, that doesn't scale and doesn't remember — is exactly what event-driven serverless deletes.

The mental shift is from pull to push. Instead of your code repeatedly asking "did anything happen?", AWS tells your code the instant something does. The upload itself becomes the trigger. Nothing runs — and nothing is billed — until a real event arrives, and then as many copies as needed run at once. That's the deep meaning of "serverless": not that there are no servers, but that you never provision, scale, or babysit one. You hand AWS a function and a rule for when to run it, and it handles the rest.

Three services do the heavy lifting, and it helps to hold each one's job clearly in your head before the labs:

• Lambda is the worker — a single function that runs on demand, scales to zero when idle, and spins up a fresh copy per event. It's where your actual code lives.
• EventBridge (and native triggers like S3 notifications) is the wiring — the rules that say "when this happens, run that." It decouples the thing that produces an event from the thing that reacts to it, so you can add or swap reactions without touching the source.
• Step Functions is the conductor — it sequences several Lambdas into one durable, multi-step workflow, remembering where each run is and retrying failures for you. This is what keeps a pipeline reliable across steps that each might fail.

The common beginner mistake here is reaching for these too early — wiring five Lambdas and a state machine for a job a single script could do. Event-driven design earns its complexity when work is bursty, multi-step, and must not be lost; for a steady trickle of simple tasks, a plain queue or scheduled job is often the honest answer. The rest of this chapter shows the shape you graduate to, and when it's worth it.

S3 → Lambda trigger advanced

This is the single most common serverless pattern in AWS, and it's worth understanding down to the mechanics. The story is always the same: a file appears in a bucket, and you want something to happen — extract its text, resize the image, index it, kick off a pipeline. The question is how your code learns the file arrived. The naive answer is to poll the bucket on a timer, which we just saw is wasteful and slow. The serverless answer is a bucket notification: you configure S3 once with a rule — "when an object is created here, invoke this Lambda" — and from then on the upload itself calls your function, within moments, automatically.

The piece that trips people up is what the Lambda actually receives. AWS doesn't hand your function the file — it hands it a small JSON event describing what happened. For an S3 trigger that event carries the bucket name and the object key (the file's path within the bucket), buried in a fixed nested shape: event["Records"][0]["s3"]. There can be more than one record, which is why it's a list. Your first job in almost every S3-triggered Lambda is to reach into that structure and pull out the bucket and key — and then, if you need the bytes, call S3 yourself to fetch them. Learn that event shape cold; you will type it hundreds of times.

One subtlety worth flagging before the lab, because it's a real production gotcha: object keys arrive URL-encoded in the event. A file literally named Q3 report.pdf shows up as Q3+report.pdf or Q3%20report.pdf. If you pass that raw string back to S3 or Textract, the service looks for a file with a plus sign or percent-twenty in its name and reports "not found" — a bug that only surfaces once a real user uploads a file with a space or a non-ASCII character. That's why the lab runs the key through urllib.parse.unquote_plus before using it.

The other idea the lab quietly teaches is don't wait when you don't have to. Reading a large multi-page document is slow, and a Lambda that blocks while Textract grinds is a Lambda you're paying for by the millisecond to do nothing. So instead of waiting, this handler starts the analysis and returns immediately, letting AWS notify a second function when the result is ready. Fire-and-hand-off, not sit-and-wait — that instinct is what separates a serverless design that scales cheaply from one that quietly runs up a bill.

s3_trigger.py# Lambda triggered by an S3 put. It kicks off async Textract and returns;
# a second trigger (Textract -> SNS -> Lambda) handles the result.
import boto3, urllib.parse
tx = boto3.client("textract")

def lambda_handler(event, context):
    rec = event["Records"][0]["s3"]
    bucket = rec["bucket"]["name"]
    key = urllib.parse.unquote_plus(rec["object"]["key"])

    job = tx.start_document_analysis(
        DocumentLocation={"S3Object": {"Bucket": bucket, "Name": key}},
        FeatureTypes=["FORMS", "TABLES"],
        NotificationChannel={
            "SNSTopicArn": "arn:aws:sns:us-east-1:123456789012:textract-done",
            "RoleArn": "arn:aws:iam::123456789012:role/textract-sns",
        },
    )
    return {"jobId": job["JobId"]}

Step Functions: sequence the steps expert

A Step Functions state machine is your pipeline as JSON: extract → classify with Claude → store, with automatic retries and a catch path per state. It is the durable orchestration layer you would otherwise hand-code.

Here's the pain this section removes. One Lambda triggered by one upload is easy. But real AI pipelines are chains: extract the text, then classify it with Claude, then write the result to a database — three steps where each depends on the last, and each can fail on its own. The tempting first draft is to cram all three into one giant Lambda with nested try/except blocks, a manual retry loop, and a variable tracking "which step am I on." That code is fragile: if the Lambda times out at step two, you've lost all state and have no clean way to resume. And you've hand-rolled retry logic that you'll get subtly wrong. Chaining steps reliably, across failures, is genuinely hard — and it's exactly the problem AWS packaged up as Step Functions.

The key mental move is that a Step Functions workflow is data, not code. You don't write a program that calls the steps; you write a JSON document — in a dialect called Amazon States Language (ASL) — that describes the steps and the arrows between them, and AWS executes that description for you. Think of it as a flowchart you can version-control. Each box in the flowchart is a state: most are Task states that run one Lambda, and each state names its successor with Next (or declares itself the finish with End: true). AWS runs the states in order, remembers exactly where each execution is, and shows you a live diagram lighting up as the run progresses.

The reason this earns its own service — rather than being a fancy loop — is two per-state safety features that would otherwise be scattered, error-prone code:

• Retry re-runs a failing state automatically, up to MaxAttempts, with back-off between tries. Transient hiccups (a throttled API, a blip) heal themselves with zero code from you. This is for errors you expect to pass on a second try.
• Catch is the escape hatch for errors that won't heal: instead of crashing the whole pipeline, it routes the failed run to a different state — typically a "dead-letter" step that parks the document safely for a human to inspect. This is for errors you expect to stay broken.

The distinction between those two is the thing to internalize: Retry says "try the same thing again," Catch says "give up on this path and go somewhere else." Get them backwards — catching what you should retry — and a momentary blip permanently dead-letters a perfectly good document. Reading them in the JSON below, notice that failure handling lives in configuration, declaratively, right next to the step it protects — not buried in imperative try/except code three functions deep.

pipeline.asl.json{
  "Comment": "Doc intelligence pipeline",
  "StartAt": "Extract",
  "States": {
    "Extract":  { "Type": "Task", "Resource": "arn:aws:lambda:...:extract",
                  "Retry": [{"ErrorEquals": ["States.ALL"], "MaxAttempts": 3}],
                  "Next": "Classify" },
    "Classify": { "Type": "Task", "Resource": "arn:aws:lambda:...:classify-with-claude",
                  "Catch": [{"ErrorEquals": ["States.ALL"], "Next": "DeadLetter"}],
                  "Next": "Store" },
    "Store":    { "Type": "Task", "Resource": "arn:aws:lambda:...:store", "End": true },
    "DeadLetter": { "Type": "Task", "Resource": "arn:aws:lambda:...:dlq", "End": true }
  }
}

Invoke and observe expert

You've described a pipeline; now you need to run one and watch it. In steady-state production the pipeline usually starts on its own — an upload event kicks it off — but you'll constantly want to fire a run by hand: to test a change, to reprocess a specific document, or to trigger the flow from another program. Starting a Step Functions workflow is a single API call, start_execution, and understanding the two things it takes and the one thing it returns is the whole of this section.

The distinction to hold onto is state machine vs. execution. The state machine is the blueprint — the ASL definition you wrote once, identified by its stateMachineArn. An execution is one live run of that blueprint against one specific input. You can have hundreds of executions of the same state machine running at once, each processing a different document, each tracked independently. It's the difference between a recipe and the individual dinners you cook from it — one recipe, many meals, each with its own outcome.

Two details in the call matter. First, the input is a JSON string, not a Python dictionary — that's why you wrap your payload in json.dumps. Forgetting this is the classic first-run error: pass a raw dict and the SDK rejects it. Second, the call hands back an executionArn — a unique receipt for this run. That ARN is how you find the run later: paste it into the Step Functions console and you get a live graph of your states lighting up green as they succeed, amber as they retry, red if they fail into the Catch path. That visual audit trail — being able to see exactly where any given document got to, and why it failed — is a large part of why teams reach for Step Functions instead of stitching Lambdas together by hand. Observability isn't bolted on; it comes free with the orchestration.

start_sfn.pyimport boto3, json
sfn = boto3.client("stepfunctions", region_name="us-east-1")

run = sfn.start_execution(
    stateMachineArn="arn:aws:states:us-east-1:123456789012:stateMachine:doc-intel",
    input=json.dumps({"bucket": "my-docs", "key": "invoice.pdf"}),
)
print("execution:", run["executionArn"])

def lambda_handler(event, context):
    rec = event["Records"][0]["s3"]
    bucket = rec["bucket"]["name"]
    key = rec["object"]["key"]
    print("new object:", bucket, key)
    return {"bucket": bucket, "key": key}

evt = {"Records": [{"s3": {"bucket": {"name": "docs"},
                          "object": {"key": "in/a.pdf"}}}]}
print(lambda_handler(evt, None))   # {'bucket':'docs','key':'in/a.pdf'}

import boto3

tx = boto3.client("textract", region_name="us-east-1")
resp = tx.start_document_analysis(
    DocumentLocation={"S3Object": {"Bucket": "docs", "Name": "in/a.pdf"}},
    FeatureTypes=["FORMS", "TABLES"],
    NotificationChannel={"SNSTopicArn": "arn:aws:sns:us-east-1:123:tx-done",
                         "RoleArn": "arn:aws:iam::123:role/textract-sns"},
)
print(resp["JobId"])

import json

asl = {
  "StartAt": "Extract",
  "States": {
    "Extract": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123:function:extract",
      "Retry": [{"ErrorEquals": ["States.ALL"], "MaxAttempts": 3}],
      "Catch": [{"ErrorEquals": ["States.ALL"], "Next": "DeadLetter"}],
      "Next": "Classify"},
    "Classify": {"Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123:function:classify",
      "Next": "Store"},
    "Store": {"Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123:function:store",
      "End": True},
    "DeadLetter": {"Type": "Fail", "Error": "ExtractFailed"},
  },
}
print(json.dumps(asl)[:60], "...")   # valid JSON

import boto3, json

sfn = boto3.client("stepfunctions", region_name="us-east-1")
resp = sfn.start_execution(
    stateMachineArn="arn:aws:states:us-east-1:123:stateMachine:doc-pipeline",
    input=json.dumps({"bucket": "docs", "key": "in/a.pdf"}),
)
print(resp["executionArn"])

def process_once(doc_id, store):
    if doc_id in store:
        return {"status": "skipped_duplicate", "id": doc_id}
    store[doc_id] = {"processed": True}      # idempotent write keyed by id
    return {"status": "processed", "id": doc_id}

seen = {}
print(process_once("doc-1", seen))   # processed
print(process_once("doc-1", seen))   # skipped_duplicate

def handle(doc_id, attempt, error, store, max_attempts=3):
    if doc_id in store:
        return "skip_duplicate"               # idempotency guard
    if error and attempt < max_attempts:
        return "retry"                        # transient -> Step Functions Retry
    if error:
        return "dead_letter"                  # exhausted -> Catch to DLQ
    store[doc_id] = True
    return "posted"

store = {}
print(handle("A1", 1, "Throttling", store))   # retry
print(handle("A1", 3, "Throttling", store))   # dead_letter
print(handle("A1", 1, None, store))           # posted
print(handle("A1", 1, None, store))           # skip_duplicate

Knowledge check check yourself