SageMaker deploy

SageMaker vs. Bedrock intermediate

Here's a mistake that shows up in cloud bills every month. A team needs to run a language model. They've used Bedrock before — call an API, pay per token, done. So they reach for SageMaker the same way, spin up an endpoint to host an open model "just to try it," get pulled into a meeting, and forget about it. Bedrock trained them that "not calling the model" means "not paying." SageMaker does not work that way, and the difference is the single most important thing on this whole page. The two services look like they do the same job — "give me a model" — but they bill on completely opposite principles, and confusing them is how you wake up to a four-figure surprise.

The cleanest way to hold the distinction is a rental analogy. Bedrock is a taxi: you get in, you pay for the ride, you get out, the meter stops. AWS owns the car, keeps it running, and you never think about it between trips — that's a managed foundation model, called by API, priced per token, scaling to zero when idle. SageMaker is leasing a car: you sign for the vehicle and you pay for it whether you drive it or leave it in the garage. In exchange you get the keys — you can put any model on it (an open-weight LLM off Hugging Face, a model you fine-tuned yourself, or a plain old XGBoost classifier that isn't an LLM at all), tune it, and control exactly how it runs. That's the trade in one line: Bedrock trades control for zero-maintenance; SageMaker trades maintenance for control.

So the real question isn't "which is better" — it's "do I need to own the model, or just talk to one?" If a hosted Claude or Titan answers your need, calling it through Bedrock is almost always the right default: less code, no servers, no idle cost. You cross over to SageMaker only when ownership is the actual requirement — the model isn't in Bedrock's catalog, you fine-tuned custom weights, compliance demands the model live inside your own account, or it's a classic ML model that was never a foundation model to begin with. The word to listen for is "host." The moment a task says "host the weights yourself," you've left Bedrock's territory and taken on the responsibility (and the hourly bill) that comes with the keys. The table below is that decision boiled down to a lookup.

Deploy from JumpStart intermediate

Think about what "host your own model" actually involves if you did it by hand. You'd rent a GPU server, install the right CUDA drivers, download tens of gigabytes of model weights, load them into GPU memory, stand up a web server that accepts requests and streams tokens back, wire up health checks, and configure networking so nothing leaks to the public internet. That's a week of fiddly, error-prone work before you've answered a single prompt — and it's exactly the work that makes people give up on self-hosting and just call an API. JumpStart exists to make that week disappear.

JumpStart is best understood as an app store for models. Instead of assembling the plumbing yourself, you pick a model off a shelf by its catalog name, and SageMaker already knows the recipe: which serving container it needs, how to load its weights, what instance size it wants, how to expose it. You name the model; AWS builds the rest. What you get at the end is an endpoint — a private, always-on web address inside your AWS account that takes a request and returns the model's answer. Mentally, an endpoint is "your model, running on a server, reachable at a URL only your account can hit."

There's one distinction in the code below that is worth slowing down for, because getting it wrong is the whole cost lesson: describing a model and deploying it are two different acts, and only one of them costs money. Constructing the JumpStartModel(...) object is free — it's just you writing down "I want this model." Nothing has been rented; no weights have moved. The .deploy(...) call is the line that reaches into AWS, provisions a physical GPU instance, pulls the weights onto it, and starts the server. That call takes several minutes and, the instant the server reports healthy, the meter starts ticking by the hour — and it will keep ticking until you explicitly tear the endpoint down. So when you read the four lines below, read them as "three lines of describing, one line of spending." The instance type you pass to .deploy matters too: an LLM needs a GPU (that's why the example uses a ml.g5-class machine, not a cheap CPU box), and bigger models need bigger, pricier instances. Pick the smallest instance that fits the model, because you are paying for every hour it exists, not every prompt it answers.

deploy.pyfrom sagemaker.jumpstart.model import JumpStartModel

# Deploy an open text-generation model to a real-time endpoint.
model = JumpStartModel(model_id="huggingface-llm-mistral-7b-instruct")
predictor = model.deploy(
    instance_type="ml.g5.2xlarge",   # GPU instance — billed per hour while it runs
    initial_instance_count=1,
)
print("endpoint:", predictor.endpoint_name)

Invoke from boto3 advanced

A live endpoint is useless until something talks to it, and the first time people try, they reach for the same sagemaker library they used to deploy — and hit friction. That library is heavy: it's built for managing models (creating them, deploying them, tearing them down), and it's not what your application code should carry just to send a prompt. The realization that unlocks this section is that deploying and invoking are separate jobs done by separate tools. Deploying is an admin task you do once. Invoking is what your app does a thousand times a second in production — and for that you want the lightweight, general-purpose AWS client, boto3.

Here's a useful mental split: there are two "SageMakers" as far as the API is concerned. There's sagemaker (the control plane) — the office that sets up and manages endpoints — and there's sagemaker-runtime (the data plane) — the counter you actually walk up to and hand your request. Invoking a model is a data-plane operation, so you open a boto3.client("sagemaker-runtime"). This is the same pattern you'll see across AWS: the tooling that builds a resource is deliberately kept separate from the high-traffic path that uses it, so your request path stays fast and dependency-light.

Two small details in the code below trip up almost everyone, so name them now. First, region is not optional. An endpoint lives in exactly one AWS region, and if your client points at a different region, AWS will honestly tell you the endpoint doesn't exist — because from that region's point of view, it doesn't. The region_name on the client must match where you deployed. Second, the response comes back as raw bytes, not a tidy Python object. SageMaker hands you a stream, and you have to .read() those bytes and then json.loads(...) them yourself — forgetting this is why beginners print something that looks like <botocore.response.StreamingBody> instead of an answer. Also note the payload shape: these text-generation containers expect your prompt under an "inputs" key in JSON. That key is a convention of the serving container, not a law of SageMaker — a different model might expect a different field — but for the JumpStart LLMs here, "inputs" is what they read.

invoke_endpoint.pyimport boto3, json
smr = boto3.client("sagemaker-runtime", region_name="us-east-1")

resp = smr.invoke_endpoint(
    EndpointName="my-endpoint",
    ContentType="application/json",
    Body=json.dumps({"inputs": "Explain blue-green deployment in one sentence."}),
)
print(json.loads(resp["Body"].read()))

Batch transform: no endpoint needed expert

Picture the wrong version of this first, because it's a real and expensive mistake. You have a million product reviews sitting in S3 and you need a sentiment score for each one — a one-time job, no user waiting on the other end. The instinct, fresh off the last two sections, is to deploy an endpoint and loop your million rows through invoke_endpoint. It works. But that endpoint is a server that sits awake the entire time, and — critically — it doesn't turn itself off when the last row is scored. It keeps billing by the hour until a human remembers to delete it. You've used a real-time tool for an offline job, and the mismatch is money.

The fix is to stop thinking "server" and start thinking "job." A real-time endpoint is a shop that stays open waiting for walk-in customers; batch transform is a work crew you hire for one project. The crew shows up, does the work, and goes home — you pay for the hours worked, not for a storefront sitting empty overnight. Concretely, batch transform takes a whole dataset from S3, spins up one or more instances, runs every record through the model, writes the results back to S3, and then automatically tears the instances down when the job finishes. There is no endpoint left running afterward, which means there is nothing to forget to delete and no idle-hour bill to accrue. That self-teardown is the entire point.

The way to decide between the two is to ask "is someone waiting for this answer right now?" If yes — a chat UI, an API your app calls live — you need the always-on endpoint and its low per-request latency, and you accept the discipline of deleting it. If no — you're scoring a file, backfilling a table, running a nightly report — batch transform is almost always the cheaper, safer choice precisely because it can't be left running. It also parallelizes naturally: point it at more instances and a huge dataset finishes faster, then everything shuts down. Same model, same weights; the only thing that changed is whether you provisioned a server or a job.

(a) Call Claude / an FM by API        -> Bedrock
(b) Host an open-weight model yourself -> SageMaker endpoint
(c) Serve a classic ML model (xgboost) -> SageMaker endpoint

from sagemaker.jumpstart.model import JumpStartModel

# Deploy an open text-generation model to a real-time endpoint.
model = JumpStartModel(model_id="huggingface-llm-mistral-7b-instruct")
predictor = model.deploy(
    instance_type="ml.g5.2xlarge",   # GPU instance - billed per hour while it runs
    initial_instance_count=1,
)
print("endpoint:", predictor.endpoint_name)

import boto3, json
smr = boto3.client("sagemaker-runtime", region_name="us-east-1")

resp = smr.invoke_endpoint(
    EndpointName="my-endpoint",
    ContentType="application/json",
    Body=json.dumps({"inputs": "Explain blue-green deployment in one sentence."}),
)
print(json.loads(resp["Body"].read()))

import json, boto3
from sagemaker.jumpstart.model import JumpStartModel

model = JumpStartModel(model_id="huggingface-llm-mistral-7b-instruct")
predictor = model.deploy(instance_type="ml.g5.2xlarge", initial_instance_count=1)
name = predictor.endpoint_name
print("endpoint:", name)

smr = boto3.client("sagemaker-runtime", region_name="us-east-1")
try:
    for q in ["What is CI/CD?", "Define idempotency.", "What is a canary deploy?"]:
        resp = smr.invoke_endpoint(
            EndpointName=name,
            ContentType="application/json",
            Body=json.dumps({"inputs": q}),
        )
        print(q, "->", json.loads(resp["Body"].read()))
finally:
    predictor.delete_endpoint()        # stop the per-hour charge no matter what
    print("deleted:", name)

from sagemaker.jumpstart.model import JumpStartModel

model = JumpStartModel(model_id="huggingface-llm-mistral-7b-instruct")

transformer = model.transformer(
    instance_count=1,
    instance_type="ml.g5.2xlarge",
    output_path="s3://my-bucket/scored/",
)
transformer.transform(
    data="s3://my-bucket/inputs/",
    content_type="application/json",
)
transformer.wait()   # instances are torn down when the job finishes

import boto3, datetime as dt

sm = boto3.client("sagemaker", region_name="us-east-1")
MAX_AGE_HOURS = 12
now = dt.datetime.now(dt.timezone.utc)

resp = sm.list_endpoints(StatusEquals="InService")
for ep in resp["Endpoints"]:
    age_h = (now - ep["CreationTime"]).total_seconds() / 3600
    flag = "FLAG" if age_h > MAX_AGE_HOURS else "ok"
    print(f"{ep['EndpointName']:40} {age_h:6.1f}h  {flag}")
    # optional hard enforcement:
    # if age_h > MAX_AGE_HOURS:
    #     sm.delete_endpoint(EndpointName=ep['EndpointName'])

Knowledge check check yourself