OpenAI Batch API

1 · What the Batch API is essential

Not every request needs an answer in two seconds. Overnight evals, bulk classification of a backlog, embedding a whole corpus, generating thousands of product descriptions — this is work measured in throughput, not latency. Paying the premium realtime price (and babysitting rate limits) for work nobody is waiting on is a waste. The Batch API is OpenAI's answer: submit a big pile of requests as one job, let it process asynchronously within a window, and pay less for the privilege.

The deal is explicit. You give up speed — a batch completes within a 24-hour window, not immediately — and in exchange you get two things: roughly 50% off the per-token price, and a much higher throughput ceiling because the work doesn't count against your realtime rate limits the same way. For work that isn't time-sensitive, that's close to free money.

The common mistake is reaching for batch when a user is actually waiting (a chat reply, a live search) — the 24-hour window makes that a terrible experience — or, conversely, hammering the realtime endpoint in a for loop for a million-row offline job and getting rate-limited into oblivion. Match the tool to the deadline: a human waiting → realtime; a cron job → batch.

2 · The batch lifecycle essential

A batch is a file-in, file-out job, and the lifecycle has four moves. First you build a JSONL file — one line per request, each line carrying a custom_id you choose plus the request body. You upload that file (client.files.create with purpose="batch"), then create the batch pointing at the uploaded file id. OpenAI processes it asynchronously; you poll its status until it's completed, then download the output file — another JSONL, one result line per request, each echoing the custom_id so you can line results back up with inputs.

That custom_id is the thread that ties the whole thing together. Results don't necessarily come back in order, so the id is how you say "this answer belongs to row 4,217 of my dataset." Choose it meaningfully (a database row id, a document hash) and matching results back is trivial.

3 · Recipe 1 — submit a batch essential

Build the JSONL, upload it, and create the batch. Each line's body is exactly what you'd pass to responses.create, and url names the endpoint the batch runs against.

submit.pyimport json
from openai import OpenAI
client = OpenAI()

# 1. build a JSONL file — one request per line, each with a custom_id
rows = ["Summarize photosynthesis.", "Explain TCP in one line."]
with open("requests.jsonl", "w") as f:
    for i, prompt in enumerate(rows):
        f.write(json.dumps({
            "custom_id": f"req-{i}",          # YOUR id — how results map back
            "method": "POST",
            "url": "/v1/responses",          # the endpoint this batch runs
            "body": {"model": "gpt-5.5", "input": prompt},
        }) + "\n")

# 2. upload the file
upload = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")

# 3. create the batch pointing at the uploaded file
batch = client.batches.create(
    input_file_id=upload.id,
    endpoint="/v1/responses",
    completion_window="24h",            # the only window today
)
print(batch.id, batch.status)

4 · Recipe 2 — poll until it ends intermediate

A batch isn't instant, so you poll its status until it reaches a terminal state. The lifecycle runs validating → in_progress → finalizing → completed (or failed/expired/cancelled). Poll gently — this is a job that takes minutes to hours, so check every so often, not in a tight loop.

poll.pyimport time
from openai import OpenAI
client = OpenAI()

def wait_for(batch_id, every=30):
    while True:
        b = client.batches.retrieve(batch_id)
        print(b.status, b.request_counts)        # completed/total progress
        if b.status in {"completed", "failed", "expired", "cancelled"}:
            return b
        time.sleep(every)                         # poll gently — this takes a while

5 · Recipe 3 — retrieve & match by custom_id intermediate

When the batch is completed it has an output_file_id. Download that file, parse it as JSONL, and use each line's custom_id to line results back up with your inputs.

collect.pyimport json
from openai import OpenAI
client = OpenAI()

def collect(batch):
    content = client.files.content(batch.output_file_id).text
    results = {}
    for line in content.splitlines():
        row = json.loads(line)
        cid = row["custom_id"]                 # your id, echoed back
        if row.get("error"):
            results[cid] = ("ERROR", row["error"])
        else:
            body = row["response"]["body"]       # the full Responses object
            results[cid] = ("OK", body["output_text"])
    return results

6 · Error handling per item advanced

A batch of 10,000 rarely comes back 10,000 clean. Some rows will error — a malformed body, a content-policy refusal, a token overflow — and the critical property of batch is that one bad row doesn't sink the job. Each line in the output carries its own success-or-error, so you handle failure per item, not per batch. The pattern: collect the OK results, bucket the errors by custom_id, and re-submit just the failures as a new, smaller batch.

7 · Batch vs realtime vs streaming advanced

Three delivery modes, one decision rule — who, if anyone, is waiting? Streaming (ox2 §5) is for a human watching tokens appear right now. Realtime (a normal responses.create) is for a request a program needs answered in-line to continue. Batch is for work with no live consumer — the cheaper, higher-throughput tier for anything you can collect later.

8 · Cost math (runs offline) professional

The batch discount is only worth claiming if you can quantify it. This estimator is pure Python — no key needed — and makes the saving a number you can put in a design doc.

batch_savings.py# illustrative rates $/1M tokens — check current pricing
RT_IN, RT_OUT = 5.0, 15.0
BATCH_DISCOUNT = 0.5                       # ~50% off

def cost(reqs, in_tok, out_tok, batch=False):
    c = reqs * (in_tok*RT_IN + out_tok*RT_OUT) / 1_000_000
    return c * (BATCH_DISCOUNT if batch else 1)

n, i, o = 100_000, 600, 300
print(f"realtime ${cost(n,i,o):,.0f}  batch ${cost(n,i,o,batch=True):,.0f}")

9 · Tech-lead — production batch pipelines tech-lead

A production batch pipeline is a data pipeline that happens to call an LLM. The LLM call is the easy part; the engineering is everything around it: chunk a large dataset into batches under the file-size and request-count limits, persist each batch id and its custom_id→row mapping to a database, poll from a scheduled job (not a blocking process), reconcile counts on completion, and auto-resubmit the error rows. Treat the whole thing as idempotent and resumable — a batch that fails at hour 20 shouldn't mean re-running the first 19.

The capstone for this track (the batch document pipeline project) builds exactly this end to end.

Knowledge check check yourself