AWS AI Automation · Chapter W9

SageMaker pipelines

A SageMaker Pipeline turns a one-off training script into the automated, repeatable MLOps lifecycle: preprocess → train → evaluate → register, with approval-gated promotion.

⏱️ ~2 hours🧪 3 labs🎯 Advanced→Expert

⚙️ To run this for realThe code here is complete and correct as written. To actually execute it you'll need:

AWS credentials with SageMaker access + pip install sagemaker
AWS credentials (aws configure) + pip install boto3

Reading and learning works without any of this — run when you're ready. IDs/ARNs in examples are placeholders; swap in your own.

Learning objectives

Assemble a SageMaker Pipeline of processing → training → register steps.
Register a trained model into the Model Registry with approval status.
Explain how this automates the MLOps lifecycle from your LLMOps chapters.
Trigger a pipeline run and inspect its execution.

▶ Runnable companionEvery code block in this lesson is also a standalone file under code/aws9-sagemaker-pipelines/ in the course, with a README on how to run it. Read here, run there — no need to copy-paste.

Pipelines automate the lifecycle intermediate

A one-off training script is a demo. A SageMaker Pipeline is the automated, repeatable lifecycle: preprocess, train, evaluate, and register — versioned and re-runnable. This is the MLOps discipline, in AWS primitives.

Define the pipeline intermediate

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Lab W9.1

pipeline.pyfrom sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.steps import ProcessingStep, TrainingStep
from sagemaker.sklearn.processing import SKLearnProcessor
from sagemaker.estimator import Estimator

ROLE = "arn:aws:iam::123456789012:role/sm-pipeline-role"

# Build a processor and an estimator (real args elided), then wire them
# into a repeatable DAG. This is the whole point of a Pipeline.
processor = SKLearnProcessor(framework_version="1.2-1", role=ROLE, instance_count=1,
                             instance_type="ml.m5.large")
estimator = Estimator(image_uri="<training-image>", role=ROLE, instance_count=1,
                      instance_type="ml.m5.large")

pipeline = Pipeline(
    name="churn-model-pipeline",
    steps=[
        ProcessingStep(name="Preprocess", processor=processor),
        TrainingStep(name="Train", estimator=estimator),
    ],
)
pipeline.upsert(role_arn=ROLE)

▶ How this works

Instead of running a training script by hand every time, this builds a pipeline: a fixed sequence of steps AWS SageMaker can run for you, over and over, the same way. Think of it as an assembly line for building a machine-learning model — first clean the data, then train on it. This file just describes that assembly line; it doesn't run it yet.

The from sagemaker... lines pull in the building blocks: a Pipeline (the assembly line), the step types (ProcessingStep, TrainingStep), and the workers that do the actual jobs (a SKLearnProcessor for data prep, an Estimator for training).
ROLE is an AWS IAM role ARN — a permission badge that says "this pipeline is allowed to use AWS resources on your behalf". The value here is a placeholder; you'd swap in your own.
processor and estimator define who runs each job and on what hardware. instance_type="ml.m5.large" picks the size of the cloud machine; instance_count=1 means one machine.
Pipeline(name=..., steps=[...]) is the key line: it lists the steps in order — first Preprocess, then Train. That ordered list is the assembly line (engineers call it a DAG, a dependency graph).
pipeline.upsert(role_arn=ROLE) sends this definition up to AWS and saves it. "Upsert" = update if it already exists, otherwise create it. After this, the pipeline exists in the cloud, ready to be triggered.

What the output means: Nothing visible prints. The result lives in AWS: a saved pipeline named churn-model-pipeline that you (or an automated schedule) can run on demand.

Try this: Add a third step to the steps=[...] list — for example an Evaluate step between train and register. The pipeline runs steps in the order you list them, so where you put it matters.

Model Registry advanced

Training outputs a model artifact; the Model Registry versions it, tracks metrics, and gates promotion with an approval status. Your deploy step only ships Approved versions — an eval-gated rollout, exactly like O3.

Lab W9.2

approve.pyimport boto3
sm = boto3.client("sagemaker", region_name="us-east-1")

# promote a specific model version to Approved so deployment can pick it up
sm.update_model_package(
    ModelPackageArn="arn:aws:sagemaker:us-east-1:123456789012:model-package/churn/3",
    ModelApprovalStatus="Approved",
)

▶ How this works

Training can produce many versions of a model over time. The Model Registry is AWS's catalogue of those versions. Before a model is allowed into production, a human (or an automated check) marks it Approved — a safety gate. This tiny script flips that switch on one specific version.

import boto3 loads the AWS SDK for Python — the library that lets your code talk to AWS services.
sm = boto3.client("sagemaker", region_name="us-east-1") opens a connection to SageMaker in a specific AWS region. sm is now your handle for giving it commands.
sm.update_model_package(...) is the command that changes a model version's status.
ModelPackageArn="...model-package/churn/3" names exactly which version — here, version 3 of the churn model group. The ARN is AWS's unique address for that item.
ModelApprovalStatus="Approved" is the actual change: it promotes that version. Now a deploy step configured to ship only Approved models will pick this one up.

What the output means: No printed output. Behind the scenes, version 3 of the churn model flips from pending to Approved in the registry, unblocking its deployment.

Try this: Change "Approved" to "Rejected". That's how you'd block a bad model version — the same gate, used to keep something out of production.

Run and inspect expert

Lab W9.3

run_pipeline.pyimport boto3
sm = boto3.client("sagemaker", region_name="us-east-1")

ex = sm.start_pipeline_execution(PipelineName="churn-model-pipeline")
print("execution:", ex["PipelineExecutionArn"])

steps = sm.list_pipeline_execution_steps(PipelineExecutionArn=ex["PipelineExecutionArn"])
for s in steps["PipelineExecutionSteps"]:
    print(s["StepName"], s["StepStatus"])

▶ How this works

The pipeline from Lab W9.1 is saved in AWS but idle. This script triggers a run of it and then checks on each step's progress — like pressing Start on the assembly line and watching a status board.

sm.start_pipeline_execution(PipelineName="churn-model-pipeline") kicks off one run of the pipeline by name. AWS begins working through the steps in the background.
The call returns an ex object. ex["PipelineExecutionArn"] is the unique ID of this particular run — you'll need it to ask about progress. The print shows it to you.
sm.list_pipeline_execution_steps(PipelineExecutionArn=...) asks AWS: "for this run, what are the steps and how are they doing?" It hands back a list.
The for s in steps["PipelineExecutionSteps"]: loop walks that list and prints each step's name and status — e.g. whether it's Executing, Succeeded, or Failed.

What the output means: You'll see the run's ARN, then one line per step, such as Preprocess Succeeded and Train Executing. Run it again a minute later and the statuses will have moved forward.

Try this: Wrap the last three lines in a loop that re-checks every 30 seconds until every step shows Succeeded — that's the beginning of monitoring a pipeline automatically.

Exercise W9.1 — Eval-gated pipeline

Context: The capstone ties evaluation to promotion: a model earns registration only by beating a baseline, and earns deployment only by explicit approval. This register-if-good, deploy-if-approved contract is the backbone of safe automated rollout.

Your task: Add an evaluation step that computes a metric and only registers the model if it beats a baseline, register it as PendingManualApproval, then approve it and confirm a deploy step would pick it up.

Requirements:

Compute an evaluation metric in a dedicated step (the Ch 8d pattern)
Register the model only when the metric beats the baseline
Register as PendingManualApproval, not auto-approved
Approve the version and confirm a deploy step would select it
Needs AWS creds; keep register-if-good and deploy-if-approved as two gates

💡 Hint: Reuse the industry rung's condition-gated RegisterModel, then perform approval as a separate update_model_package call to keep the human in the loop.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Why a pipeline beats a one-off scriptBeginner

Context: A one-off training script is a demo; a SageMaker Pipeline is the automated, versioned, re-runnable lifecycle behind real MLOps. Understanding what the pipeline buys you — and the four stages it wires together — is the foundation for everything else in the lesson.

Your task: In your own words, explain what a SageMaker Pipeline gives you over running a training script by hand, and name the four lifecycle stages the lesson lists.

Requirements:

Contrast a manual script with a repeatable, versioned, schedulable DAG of steps
Name the four stages in order: preprocess → train → evaluate → register
Explain that the same steps re-run identically every time
Mention approval-gated promotion at the end of the flow
Runs offline (a conceptual exercise)

💡 Hint: Frame it as MLOps discipline expressed in AWS primitives — the value is repeatability and version tracking, not any single step.

Show solution

A one-off training script is a demo; a SageMaker Pipeline is the automated, repeatable, versioned lifecycle you (or a schedule) can re-run the same way every time. It turns training into a DAG of steps rather than manual button-pushing.

The four lifecycle stages from the lesson:

preprocess -> train -> evaluate -> register

This is the MLOps discipline expressed in AWS primitives: the same steps, wired together, re-runnable and version-tracked, with approval-gated promotion at the end.

Exercise 2 · Define and upsert a two-step pipelineIntermediate

Context: Defining a pipeline is just describing an ordered list of steps and who runs each; the ordered list is the DAG. upsert then ships that definition to AWS (create or update) without running anything yet.

Your task: Using the SageMaker Workflow SDK, build a pipeline named churn-model-pipeline with a preprocessing step then a training step, and register it with upsert.

Requirements:

Create a processor (e.g. SKLearnProcessor) and an Estimator, each with a role and instance type
Wrap them in a ProcessingStep and a TrainingStep
List the steps in order in Pipeline(steps=[...]) — that is the DAG
Call pipeline.upsert(role_arn=ROLE) to register (create or update)
Nothing runs yet; needs AWS creds

💡 Hint: upsert only registers the definition — think of it as saving the recipe, not cooking; a separate call triggers a run.

Show solution

from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.steps import ProcessingStep, TrainingStep
from sagemaker.sklearn.processing import SKLearnProcessor
from sagemaker.estimator import Estimator

ROLE = "arn:aws:iam::123456789012:role/sm-pipeline-role"

processor = SKLearnProcessor(framework_version="1.2-1", role=ROLE, instance_count=1,
                             instance_type="ml.m5.large")
estimator = Estimator(image_uri="<training-image>", role=ROLE, instance_count=1,
                      instance_type="ml.m5.large")

pipeline = Pipeline(
    name="churn-model-pipeline",
    steps=[
        ProcessingStep(name="Preprocess", processor=processor),
        TrainingStep(name="Train", estimator=estimator),
    ],
)
pipeline.upsert(role_arn=ROLE)

The processor and estimator define who runs each job and on what hardware; Pipeline(steps=[...]) lists them in order — that ordered list is the DAG. ROLE is an IAM role ARN (a placeholder here) granting the pipeline permission to use AWS resources. pipeline.upsert(...) sends the definition up to AWS: update if it exists, otherwise create. Nothing prints and nothing runs yet — the pipeline just now exists in the cloud, ready to trigger. Needs AWS credentials to run (SageMaker access + pip install sagemaker).

Exercise 3 · Promote a model version in the Model RegistryAdvanced

Context: The Model Registry versions every trained model and tracks a ModelApprovalStatus per version. That single field is the gate a deploy step reads: flipping it to Approved ships a version; the same call with one word changed keeps a bad version out.

Your task: Write a boto3 script that promotes version 3 of the churn model group to Approved via update_model_package, and explain the one-word change that would block it instead.

Requirements:

Call update_model_package with the version-3 ModelPackageArn (ending in /churn/3)
Set ModelApprovalStatus="Approved" to unblock deployment
Explain that a deploy step ships only approved versions
Note that "Rejected" is the one-word change that blocks it
Needs AWS creds

💡 Hint: The ARN's trailing /churn/3 pins exactly one version — the approval status is the only thing you are changing.

Show solution

import boto3
sm = boto3.client("sagemaker", region_name="us-east-1")

# promote a specific model version to Approved so deployment can pick it up
sm.update_model_package(
    ModelPackageArn="arn:aws:sagemaker:us-east-1:123456789012:model-package/churn/3",
    ModelApprovalStatus="Approved",
)

The Model Registry versions each trained model and tracks a ModelApprovalStatus. update_model_package flips the status of one specific version — the ModelPackageArn ending in /churn/3 names exactly version 3 of the churn group. Setting it to "Approved" unblocks a deploy step configured to ship only approved versions. Changing the single value to "Rejected" uses the same gate to keep a bad version out of production. Needs AWS credentials to run (aws configure + pip install boto3).

Exercise 4 · Trigger a run and inspect per-step statusExpert

Context: Starting a run and reading its progress are two different calls: one kicks off an execution by name, the other queries per-step status using the execution ARN as the handle. This is the basis of every monitoring script you will build on top.

Your task: Start a run of churn-model-pipeline, print its execution ARN, then list each step with its status.

Requirements:

Call start_pipeline_execution(PipelineName=...)
Print the returned PipelineExecutionArn
Pass that ARN to list_pipeline_execution_steps
Loop and print each step's StepName and StepStatus
Needs AWS creds and an already-upserted pipeline

💡 Hint: The execution ARN is the handle for this run — everything you query about progress keys off it, and statuses advance if you re-run a minute later.

Show solution

import boto3
sm = boto3.client("sagemaker", region_name="us-east-1")

ex = sm.start_pipeline_execution(PipelineName="churn-model-pipeline")
print("execution:", ex["PipelineExecutionArn"])

steps = sm.list_pipeline_execution_steps(PipelineExecutionArn=ex["PipelineExecutionArn"])
for s in steps["PipelineExecutionSteps"]:
    print(s["StepName"], s["StepStatus"])

start_pipeline_execution kicks off one run by name; AWS works through the steps in the background. The returned ex["PipelineExecutionArn"] uniquely identifies this run and is the handle you pass to list_pipeline_execution_steps to fetch progress. The loop prints each step's name and status — e.g. Preprocess Succeeded, Train Executing. Re-run a minute later and the statuses will have advanced. Needs AWS credentials to run (boto3 + a pipeline already upserted).

Exercise 5 · Poll an execution until every step succeedsProfessional

Context: "Kick it off and hope" is not monitoring. Turning the trigger script into a poller that re-checks step status until every step reaches a terminal state is the seed of the automated pipeline monitoring you would wire into alerting.

Your task: Extend the run script into a monitor: after triggering the pipeline, re-check step status every 30 seconds and stop when all steps have finished (Succeeded or Failed).

Requirements:

Start the execution and capture its ARN
Define a set of terminal states (e.g. Succeeded, Failed, Stopped)
Loop, re-querying list_pipeline_execution_steps and sleeping ~30s between checks
Guard against the brief empty-steps window right after start
Exit only when all steps are terminal, then report overall success/failure; needs AWS creds

💡 Hint: Steps can be momentarily empty right after start, so require a non-empty list and all-terminal before breaking, or you will exit early.

Show solution

import boto3, time
sm = boto3.client("sagemaker", region_name="us-east-1")

ex = sm.start_pipeline_execution(PipelineName="churn-model-pipeline")
arn = ex["PipelineExecutionArn"]
print("execution:", arn)

TERMINAL = {"Succeeded", "Failed", "Stopped"}
while True:
    steps = sm.list_pipeline_execution_steps(PipelineExecutionArn=arn)["PipelineExecutionSteps"]
    statuses = {s["StepName"]: s["StepStatus"] for s in steps}
    print(statuses)
    if steps and all(v in TERMINAL for v in statuses.values()):
        break
    time.sleep(30)

print("done:", "FAILED" if "Failed" in statuses.values() else "ALL SUCCEEDED")

This is the lesson's "Try this" turned into real monitoring: the loop re-queries list_pipeline_execution_steps every 30 seconds and exits once every step reaches a terminal state, then reports overall success or failure. Note steps can be empty for a moment right after start, so the steps and ... guard prevents a premature exit. This is the seed of automated pipeline monitoring you'd wire into alerting. Needs AWS credentials to run (boto3 + an upserted pipeline).

Exercise 6 · Eval-gated pipeline with manual approvalIndustry scenario

Context: Production promotion is never automatic on faith. The eval-gated pattern registers a model only if it beats a baseline metric, and even then lands it as PendingManualApproval so a human clears it before any deploy step picks it up.

Your task: Implement W9.1: add an evaluation step that registers the model only if it beats a baseline, register it as PendingManualApproval, then approve it so a deploy step would pick it up.

Requirements:

An evaluate step emits a metric (e.g. accuracy)
A ConditionStep with ConditionGreaterThanOrEqualTo gates registration on beating the baseline (e.g. 0.80)
RegisterModel uses approval_status="PendingManualApproval" (not auto-shipped)
Wire preprocess → train → evaluate → gate into the pipeline and upsert
Then flip the version to Approved with update_model_package; needs AWS creds

💡 Hint: The condition's if_steps is where the register lands — below-baseline runs simply skip it, and approval stays a deliberate second, human step.

Show solution

from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.steps import ProcessingStep, TrainingStep
from sagemaker.workflow.condition_step import ConditionStep
from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo
from sagemaker.workflow.step_collections import RegisterModel

ROLE = "arn:aws:iam::123456789012:role/sm-pipeline-role"

# ... processor, estimator, and an 'evaluate' ProcessingStep that emits accuracy ...
register = RegisterModel(
    name="RegisterChurn",
    estimator=estimator,
    model_package_group_name="churn",
    approval_status="PendingManualApproval",   # not auto-shipped
)
gate = ConditionStep(
    name="GateOnAccuracy",
    conditions=[ConditionGreaterThanOrEqualTo(left=eval_accuracy, right=0.80)],
    if_steps=[register],          # only register if it beats the 0.80 baseline
    else_steps=[],
)
pipeline = Pipeline(name="churn-model-pipeline",
                    steps=[preprocess, train, evaluate, gate])
pipeline.upsert(role_arn=ROLE)

The condition step is the eval gate: the model is only registered when its accuracy clears the baseline, and even then it lands as PendingManualApproval rather than auto-approved. A human (or automated check) then promotes it, exactly as in Lab W9.2:

import boto3
sm = boto3.client("sagemaker", region_name="us-east-1")
sm.update_model_package(
    ModelPackageArn="arn:aws:sagemaker:us-east-1:123456789012:model-package/churn/3",
    ModelApprovalStatus="Approved",
)

Together this is the eval-gated rollout the lesson describes: register only if it beats a baseline, deploy only Approved versions. Needs AWS credentials to run (SageMaker SDK + boto3).

✓ Checkpoint — you can move on when you can…

Assemble a multi-step SageMaker Pipeline.
Register a model version and set its approval status.
Explain how the Model Registry enables eval-gated rollouts.
Trigger a run and inspect per-step status.

Knowledge check check yourself

✓ Knowledge check

What does a SageMaker Pipeline give you over a one-off training script, and what does pipeline.upsert do?

Show answer

It turns training into an automated, repeatable, versioned DAG of steps (preprocess -> train -> evaluate -> register) that you or a schedule can re-run. upsert sends the pipeline definition to AWS, creating it if new or updating it if it already exists.

✓ Knowledge check

How does the Model Registry enable an eval-gated rollout?

Show answer

The registry versions each trained model and tracks a ModelApprovalStatus. A deploy step ships only Approved versions, so promotion is gated: a human or automated check sets a version to Approved (or Rejected) before it can reach production.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

Pipelines automate the lifecycle intermediate

Define the pipeline intermediate

Model Registry advanced

Run and inspect expert

Exercise W9.1 — Eval-gated pipeline

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in AWS AI automation