AWS AI Automation · Chapter W8

SageMaker deploy

SageMaker is for when you need to own the model. Deploy an open model from JumpStart to an endpoint, invoke it from boto3 — and learn the delete-the-endpoint discipline.

⏱️ ~2 hours🧪 2 labs🎯 Advanced

⚙️ To run this for realThe code here is complete and correct as written. To actually execute it you'll need:

AWS credentials with SageMaker access + pip install sagemaker
AWS credentials (aws configure) + pip install boto3

Reading and learning works without any of this — run when you're ready. IDs/ARNs in examples are placeholders; swap in your own.

Learning objectives

Explain where SageMaker fits vs. Bedrock (own/host models vs. managed FMs).
Deploy an open model from JumpStart to a real-time endpoint.
Invoke the endpoint from boto3 and read predictions.
Avoid the classic cost trap: delete endpoints you are not using.

▶ Runnable companionEvery code block in this lesson is also a standalone file under code/aws8-sagemaker-deploy/ in the course, with a README on how to run it. Read here, run there — no need to copy-paste.

SageMaker vs. Bedrock intermediate

Bedrock gives you managed foundation models by API. SageMaker is the platform for when you need to own the model: host an open-weight LLM, fine-tune, or serve a classic ML model. More control, more responsibility — including the instance bill.

Need	Use
Call Claude / an FM by API	Bedrock
Host an open-weight model yourself	SageMaker endpoint
Train / fine-tune a custom model	SageMaker training + pipelines
Serve a classic ML model (xgboost…)	SageMaker endpoint

Deploy from JumpStart intermediate

JumpStart is a catalog of pre-packaged models you can deploy in a few lines. The SDK builds the endpoint config and provisions the instance.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Lab W8.1

deploy.pyfrom sagemaker.jumpstart.model import JumpStartModel

# Deploy an open text-generation model to a real-time endpoint.
model = JumpStartModel(model_id="huggingface-llm-mistral-7b-instruct")
predictor = model.deploy(
    instance_type="ml.g5.2xlarge",   # GPU instance — billed per hour while it runs
    initial_instance_count=1,
)
print("endpoint:", predictor.endpoint_name)

▶ How this works

SageMaker is the AWS service for hosting your own machine-learning model on servers you rent by the hour. JumpStart is a built-in catalog of ready-made models — you pick one by name and SageMaker downloads it, puts it on a server, and gives you a private web address (an endpoint) to send requests to. This block does the whole deploy in four lines.

from sagemaker.jumpstart.model import JumpStartModel pulls in the helper class that knows how to deploy a catalog model. The sagemaker library talks to your AWS account for you.
JumpStartModel(model_id="huggingface-llm-mistral-7b-instruct") picks which model to deploy — here the open Mistral 7B chat model. The model_id is just its catalog name; nothing runs or costs money yet, you've only described what you want.
model.deploy(...) is the step that actually rents a server and loads the model onto it. instance_type="ml.g5.2xlarge" asks for a GPU machine (needed to run an LLM fast), and initial_instance_count=1 says start with one of them. This call takes several minutes and the meter starts running the moment the server is up.
It hands back a predictor object — your remote control for the live endpoint. predictor.endpoint_name is the unique name AWS assigned it, which you print so you can find it again later.

What the output means: After a few minutes you'll see something like endpoint: hf-llm-mistral-7b-.... That name proves a live GPU server is now running your model and waiting for requests.

Try this: Notice there is no delete step here — that's the trap. A running ml.g5.2xlarge bills every hour whether you use it or not, so the moment you're done you must call predictor.delete_endpoint() to stop the charge.

The endpoint bills per hour, awake or idleUnlike Bedrock (pay-per-token, scales to zero), a real-time endpoint charges for the instance every hour it exists. Delete it when done: predictor.delete_endpoint(). An idle ml.g5.2xlarge left overnight is a real, avoidable bill.

Invoke from boto3 advanced

Lab W8.2

invoke_endpoint.pyimport boto3, json
smr = boto3.client("sagemaker-runtime", region_name="us-east-1")

resp = smr.invoke_endpoint(
    EndpointName="my-endpoint",
    ContentType="application/json",
    Body=json.dumps({"inputs": "Explain blue-green deployment in one sentence."}),
)
print(json.loads(resp["Body"].read()))

▶ How this works

Once an endpoint is live, this is how you actually ask it a question from plain Python. You don't need the SageMaker deploy library for this — just boto3, AWS's general-purpose client. You send a request to the endpoint by name and read back whatever the model generated.

boto3.client("sagemaker-runtime", region_name="us-east-1") opens a connection to the part of AWS that runs endpoints (called sagemaker-runtime). region_name must match the region where you deployed. We name it smr.
smr.invoke_endpoint(...) sends one request. EndpointName="my-endpoint" is the name from the deploy step (swap in your real one). ContentType tells the server the request is JSON.
Body=json.dumps({"inputs": "Explain blue-green deployment..."}) is the actual prompt. json.dumps turns the Python dictionary into a JSON text string, because the endpoint expects JSON — the "inputs" key is the standard field these models read the prompt from.
The reply comes back as raw bytes inside resp["Body"]. resp["Body"].read() reads those bytes and json.loads(...) parses them back into a Python object so you can use the answer.

What the output means: You'll see the model's generated text, typically as a list like [{'generated_text': 'Blue-green deployment runs two identical environments...'}]. That is the model's one-sentence answer to your prompt.

Try this: Change the sentence inside "inputs" to any question and re-run — same endpoint, new answer, no re-deploy. Each call is cheap; it's the idle server between calls that costs money.

Batch transform: no endpoint needed expert

For offline scoring of a large dataset, batch transform spins up instances, processes an S3 input, writes to S3, and tears down — no long-lived endpoint, no idle cost.

Exercise W8.1 — Deploy, invoke, delete

Context: The deploy/invoke/delete loop is the whole SageMaker hosting lifecycle in miniature, and the reason it matters is money: the only reliable proof you stopped the charge is seeing it disappear from the bill the next day.

Your task: Deploy a small JumpStart model, invoke it three times from boto3, then delete the endpoint and confirm it is gone — and check Cost Explorer the next day to prove deletion stopped the charge.

Requirements:

Deploy one small JumpStart model to a real-time endpoint
Invoke it three times against the same live endpoint
Delete the endpoint and confirm it no longer appears as in-service
Verify in Cost Explorer the following day that charges stopped
Needs AWS creds; treat the deletion as the load-bearing step

💡 Hint: Reuse the expert rung's finally-guarded cleanup so the delete happens no matter what, then let the next-day bill be the real receipt.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Choose SageMaker vs BedrockBeginner

Context: The first decision in any hosting task is which service even fits. Bedrock gives you managed foundation models by API that scale to zero; SageMaker is for when you must own the model — an open-weight LLM or a classic ML model — on instances you rent by the hour.

Your task: For each need, say whether you would reach for Bedrock or a SageMaker endpoint: (a) call Claude by API, (b) host an open-weight Mistral yourself, (c) serve a classic xgboost model — justifying each in one line.

Requirements:

Map (a) to Bedrock: managed FM by API, pay-per-token, scales to zero
Map (b) to a SageMaker endpoint: you host the open-weight model yourself
Map (c) to a SageMaker endpoint: classic ML served on rented instances
State the trade in one line: more control for more responsibility (and the hourly bill)
Runs offline (a reasoning exercise, no API calls)

💡 Hint: Ask one question per case: do I need to own and host the weights, or just call a model? Ownership pushes you to SageMaker.

Show solution

Mapping straight from the lesson's decision table:

(a) Call Claude / an FM by API        -> Bedrock
(b) Host an open-weight model yourself -> SageMaker endpoint
(c) Serve a classic ML model (xgboost) -> SageMaker endpoint

Bedrock gives you managed foundation models by API (pay-per-token, scales to zero), so it fits (a). SageMaker is for when you need to own the model — host an open-weight LLM or serve a classic ML model on instances you rent by the hour — so it fits (b) and (c). The trade is more control for more responsibility, including the instance bill.

Exercise 2 · Deploy a JumpStart model to an endpointIntermediate

Context: JumpStart is the fast path to hosting an open model: choosing the model is free and only describes what you want, while .deploy(...) is the call that actually rents a GPU server and starts the meter. Knowing which line costs money is the core lesson.

Your task: Using the SageMaker Python SDK, deploy huggingface-llm-mistral-7b-instruct from JumpStart to a single-instance real-time endpoint on a GPU instance, and print the endpoint name.

Requirements:

Build a JumpStartModel(model_id=...) for the Mistral instruct model
Call .deploy(...) with a GPU instance_type (e.g. ml.g5.2xlarge) and initial_instance_count=1
Capture the returned predictor
Print predictor.endpoint_name so you can find it again
Needs AWS creds; the meter starts once the server is up

💡 Hint: Constructing the model object is free — it is .deploy() that spins up billable hardware, so treat that line as the cost boundary.

Show solution

from sagemaker.jumpstart.model import JumpStartModel

# Deploy an open text-generation model to a real-time endpoint.
model = JumpStartModel(model_id="huggingface-llm-mistral-7b-instruct")
predictor = model.deploy(
    instance_type="ml.g5.2xlarge",   # GPU instance - billed per hour while it runs
    initial_instance_count=1,
)
print("endpoint:", predictor.endpoint_name)

Picking the model is free — JumpStartModel(model_id=...) only describes what you want. The .deploy(...) call is what actually rents a GPU server and loads the model, returning a predictor whose endpoint_name you print so you can find it again. The meter starts the moment the server is up. Needs AWS credentials to run (SageMaker access + pip install sagemaker).

Exercise 3 · Invoke the endpoint from boto3Advanced

Context: Calling a live endpoint does not need the heavyweight deploy library — just the general-purpose boto3 runtime client. The gotcha is the payload shape and reading the response back out of a byte stream.

Your task: Write a boto3 script that sends a JSON prompt to an endpoint named my-endpoint in us-east-1 via the sagemaker-runtime client and prints the parsed response.

Requirements:

Use the sagemaker-runtime client with a matching region_name
Put the prompt under the "inputs" key, JSON-encoded
Set ContentType="application/json" on invoke_endpoint
Read resp["Body"] and parse it with json.loads
Print the parsed object (typically a list with generated_text); needs AWS creds

💡 Hint: The region on the client must match where the endpoint was deployed, and the reply arrives as bytes you must .read() before parsing.

Show solution

import boto3, json
smr = boto3.client("sagemaker-runtime", region_name="us-east-1")

resp = smr.invoke_endpoint(
    EndpointName="my-endpoint",
    ContentType="application/json",
    Body=json.dumps({"inputs": "Explain blue-green deployment in one sentence."}),
)
print(json.loads(resp["Body"].read()))

You don't need the SageMaker deploy library to invoke — just the general-purpose boto3 client for the sagemaker-runtime service, whose region_name must match where you deployed. The prompt goes in the "inputs" field (the standard key these text-generation models read), JSON-encoded with json.dumps. The reply arrives as bytes in resp["Body"]; .read() plus json.loads(...) parses it into a Python object, typically [{'generated_text': '...'}]. Needs AWS credentials to run (aws configure + pip install boto3).

Exercise 4 · Deploy, invoke three times, then deleteExpert

Context: An idle real-time endpoint bills every hour whether or not it is used, so the professional discipline is deploy → invoke → delete — and the delete must happen even if an invocation blows up, or you leave a GPU billing overnight.

Your task: Implement W8.1: deploy a small JumpStart model, invoke it three times from boto3, then delete the endpoint so the per-hour charge stops — cleaning up even if invocation fails.

Requirements:

Deploy once and reuse the same live endpoint for all three invocations
Loop three prompts through invoke_endpoint, printing each parsed reply
Wrap the invocations so cleanup is guaranteed on any error
Call predictor.delete_endpoint() in a finally block
Needs AWS creds; confirm the endpoint is gone afterward

💡 Hint: Put the delete in finally, not after the loop — that is the one placement that survives an exception mid-invocation.

Show solution

import json, boto3
from sagemaker.jumpstart.model import JumpStartModel

model = JumpStartModel(model_id="huggingface-llm-mistral-7b-instruct")
predictor = model.deploy(instance_type="ml.g5.2xlarge", initial_instance_count=1)
name = predictor.endpoint_name
print("endpoint:", name)

smr = boto3.client("sagemaker-runtime", region_name="us-east-1")
try:
    for q in ["What is CI/CD?", "Define idempotency.", "What is a canary deploy?"]:
        resp = smr.invoke_endpoint(
            EndpointName=name,
            ContentType="application/json",
            Body=json.dumps({"inputs": q}),
        )
        print(q, "->", json.loads(resp["Body"].read()))
finally:
    predictor.delete_endpoint()        # stop the per-hour charge no matter what
    print("deleted:", name)

This is the full deploy -> invoke -> delete discipline. The three invocations reuse the same live endpoint (no re-deploy per call — each call is cheap; it's the idle server between calls that costs money). The finally block guarantees predictor.delete_endpoint() runs even if an invocation raises, which is exactly how you avoid an ml.g5.2xlarge left billing overnight. Needs AWS credentials to run (SageMaker + boto3).

Exercise 5 · Score a dataset with batch transform (no endpoint)Professional

Context: For a one-off offline scoring run, a real-time endpoint is exactly the wrong tool: it bills for every idle hour it exists. Batch transform spins up instances, reads S3, writes S3, and tears itself down — no long-lived endpoint, no idle cost.

Your task: Explain why a real-time endpoint is wrong for a large one-off scoring job, then sketch a batch transform job with the SageMaker SDK that reads from S3, writes to S3, and tears itself down.

Requirements:

State the cost argument: endpoints bill per hour awake or idle; batch transform does not linger
Build a transformer(...) off the model with an S3 output_path
Call transformer.transform(data=...) pointing at an S3 input prefix
Call transformer.wait(); instances tear down when the job finishes
Needs AWS creds; no endpoint is left running

💡 Hint: The mental model: batch transform is a job that ends, not a server that waits — which is why it dodges the per-hour endpoint trap.

Show solution

A real-time endpoint bills for its instance every hour it exists, awake or idle — wasteful for a one-off offline scoring run. Batch transform spins up instances, processes an S3 input, writes results to S3, and tears down: no long-lived endpoint, no idle cost.

from sagemaker.jumpstart.model import JumpStartModel

model = JumpStartModel(model_id="huggingface-llm-mistral-7b-instruct")

transformer = model.transformer(
    instance_count=1,
    instance_type="ml.g5.2xlarge",
    output_path="s3://my-bucket/scored/",
)
transformer.transform(
    data="s3://my-bucket/inputs/",
    content_type="application/json",
)
transformer.wait()   # instances are torn down when the job finishes

The key operational point from the lesson: for offline scoring of a large dataset, batch transform avoids the classic per-hour endpoint cost trap because nothing stays running after the job completes. Needs AWS credentials to run (SageMaker access + pip install sagemaker).

Exercise 6 · Enforce the no-idle-endpoint policyIndustry scenario

Context: Orgs get surprise bills from endpoints engineers forget to delete. Because SageMaker charges per hour with no scale-to-zero, you cannot trust people to remember delete_endpoint() — you need an automated sweep plus a written policy.

Your task: Design a lightweight guardrail: a boto3 script that lists all InService endpoints and flags ones older than a threshold, plus a one-line policy statement.

Requirements:

List endpoints with list_endpoints(StatusEquals="InService")
Compute each endpoint's age from its CreationTime vs now (UTC)
Flag any endpoint older than a threshold (e.g. 12 hours)
Include a commented hard-enforcement path (delete_endpoint) and a one-line policy
Note it is meant to run on a schedule (EventBridge → Lambda); needs AWS creds

💡 Hint: Compare timezone-aware datetimes (both UTC) so the age math is correct, and keep auto-delete behind a comment until the policy is agreed.

Show solution

import boto3, datetime as dt

sm = boto3.client("sagemaker", region_name="us-east-1")
MAX_AGE_HOURS = 12
now = dt.datetime.now(dt.timezone.utc)

resp = sm.list_endpoints(StatusEquals="InService")
for ep in resp["Endpoints"]:
    age_h = (now - ep["CreationTime"]).total_seconds() / 3600
    flag = "FLAG" if age_h > MAX_AGE_HOURS else "ok"
    print(f"{ep['EndpointName']:40} {age_h:6.1f}h  {flag}")
    # optional hard enforcement:
    # if age_h > MAX_AGE_HOURS:
    #     sm.delete_endpoint(EndpointName=ep['EndpointName'])

Policy: "Real-time endpoints are ephemeral — every endpoint older than 12 hours is auto-flagged (and, off-hours, auto-deleted); anything long-lived must be an approved, tagged production endpoint or moved to batch transform." This operationalizes the lesson's core warning: unlike Bedrock's pay-per-token scale-to-zero, a SageMaker endpoint charges per hour whether used or not, so the org needs an automated sweep rather than trusting engineers to remember delete_endpoint(). Run it on a schedule (EventBridge -> Lambda). Needs AWS credentials to run (SageMaker access + boto3).

✓ Checkpoint — you can move on when you can…

State when to use SageMaker instead of Bedrock.
Deploy a JumpStart model to a real-time endpoint.
Invoke an endpoint from boto3.
Explain the per-hour cost trap and how batch transform avoids it.

Knowledge check check yourself

✓ Knowledge check

When should you choose SageMaker over Bedrock?

Show answer

Use Bedrock to call a managed foundation model by API. Use SageMaker when you need to own the model: host an open-weight LLM yourself, train/fine-tune a custom model, or serve a classic ML model -- more control at the cost of more responsibility (including the instance bill).

✓ Knowledge check

What is the classic cost trap of a SageMaker real-time endpoint, and how does batch transform avoid it?

Show answer

A real-time endpoint bills for its instance every hour it exists, awake or idle, so you must delete_endpoint when done (unlike Bedrock's pay-per-token, scale-to-zero). Batch transform instead spins up instances to score an S3 dataset, writes results to S3, and tears down -- no long-lived endpoint and no idle cost.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

SageMaker vs. Bedrock intermediate

Deploy from JumpStart intermediate

Invoke from boto3 advanced

Batch transform: no endpoint needed expert

Exercise W8.1 — Deploy, invoke, delete

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in AWS AI automation