Batch API discounts: when to use them

A batch API lets you submit a large set of requests as a single asynchronous job instead of calling the synchronous endpoint one request at a time. In exchange for giving up an immediate response — results typically land within a defined processing window rather than instantly — the provider charges a flat discount on both input and output tokens. It's one of the few genuinely free discounts in this market: the same model, the same output quality, a lower price, with the only real cost being latency you often don't need in the first place.

What belongs in a batch job

• Bulk classification or tagging runs over an existing dataset — support ticket triage, content moderation sweeps, lead scoring.
• Offline summarization or extraction jobs that feed a report, not a live user session.
• Eval and regression runs against a fixed prompt set during development.
• Nightly re-processing jobs — re-embedding a corpus, re-scoring a queue — that already run on a schedule and don't need to be instant.

What doesn't

Anything a user is waiting on in real time is disqualified by definition — chat interfaces, live agent tool calls, interactive coding assistants. Batch also isn't a fit for workloads with unpredictable, bursty submission patterns where you need to react within seconds of a new item arriving; the processing window makes that architecture impractical even if the discount is appealing.

The discount varies by provider — check the model page, not a rule of thumb

It's tempting to treat the batch discount as a fixed, provider-agnostic number, but it isn't — the percentage is set per provider and sometimes varies by model within the same provider's lineup. Two providers advertising "batch discounts" can mean noticeably different savings once you compare the actual figure on the model page. Before you build a cost model around batch pricing, pull the discount percentage for the specific model you intend to use rather than assuming it matches what a competitor offers.

Processing windows aren't identical either

The window a provider commits to — how long a batch job can take before results are guaranteed — differs by provider and isn't always published as a hard SLA. If your job has a real deadline (a report due by morning, a queue that needs to clear before the next business day), confirm the provider's stated window against that deadline rather than assuming "batch" means the same turnaround everywhere.

How the discount compounds with other levers

Batch pricing and prompt caching solve different problems and can often be combined: a batch job that also repeats a static prefix across its requests gets both discounts at once, since caching affects the input rate and batch affects the whole request. If you're running a large offline job with a shared system prompt or reference document across every item, check whether your provider allows caching within batch requests — it's the largest compounding saving available on this dataset.

The batch discount percentage is a fixed multiplier off standard pricing, not a separate rate — you can compute the batch cost of any workload yourself once you know the standard cost and the discount on the model page.

A simple test before you migrate a job to batch

Ask whether anyone is actually looking at the result within minutes of submission. If the honest answer is hours or the next business day, it's a batch candidate regardless of how the code currently calls the API. Migrating an existing synchronous integration usually just means swapping the endpoint and adding a poll-or-webhook step to collect results — the request payload and model choice don't need to change at all.

It's also worth revisiting periodically, not just once at launch. A workload that started as interactive can drift into batch-eligible territory as the product evolves — an internal tool that used to need instant answers might now feed a nightly digest instead. Whenever you're reviewing spend, that's the right moment to ask the same question again: does anything downstream of this job actually need it synchronously, or has that requirement quietly gone away?