Batch inference (batch API)

What is batch inference?

Batch inference is sending a large set of model requests to a provider as one job and letting it work through them in its own time, instead of answering each one on the spot. You hand over a list of prompts, you get a job id back, and somewhere in the next 24 hours the answers land in a file. In return for the wait, the provider charges half the normal price per token.

The deal is about capacity. Real-time inference has to answer while someone watches a spinner, so the provider keeps compute ready for the peak. A batch job can be slotted into the quiet hours, and part of that saving comes back to you.

In data engineering the word batch already means a nightly job that processes the day's records in one pass. Batch inference is the language-model version, sold as a feature of the model API itself: the Message Batches API at Anthropic, the Batch API at OpenAI, the Gemini Batch API at Google.

How a batch job runs

1. You build the request list. Each line is a complete request as you would send it normally, plus an id you choose, custom_id at Anthropic and OpenAI, key at Google, so you can match answers back to your rows. OpenAI and Google want a JSONL file you upload first.

2. You get a job id back. Anthropic validates each request only while processing it, so a mistake in your request shape turns up in the results rather than at submission. Try one request through the normal API first.

3. You wait. Poll every few minutes until the status reads ended or completed, or register a webhook: OpenAI sends batch.completed, batch.failed, batch.expired and batch.cancelled events.

4. You download the results. A JSONL file with one line per request, holding the answer or an error for that one request. A failed row does not fail the job, and the lines can come back in any order.

Price and window, as of September 2026

From the providers' own documentation on 4 September 2026, in dollars per million tokens, input first and output second.

• Anthropic, Message Batches API. Half price on both, so Claude Haiku 4.5 costs 0.50 and 2.50 in batch against 1 and 5 live. Up to 100,000 requests or 256 MB per batch. Anything unprocessed after 24 hours expires and is not billed, and results stay downloadable for 29 days.

• OpenAI, Batch API. A 50 percent discount, so GPT-5.6 Luna drops from 0.20 and 1.20 to 0.10 and 0.60. Up to 50,000 requests in a file of at most 200 MB, and 24 hours is the only window on offer. When it closes, unfinished requests are cancelled and you pay for the ones that completed. Result files are deleted 30 days later.

• Google, Gemini Batch API. Half the standard cost, so Gemini 3.5 Flash-Lite goes from 0.30 and 2.50 to 0.15 and 1.25. Inline under 20 MB, or a JSONL file of up to 2 GB. A job still pending after 48 hours expires, and results are kept for six weeks.

Google has announced that most Gemini 3 prices rise on 1 January 2027, so look the numbers up again on the day you build the business case.

What to batch, and what to keep live

One question decides it: is anyone waiting for this answer? Classifying a year of support tickets into queues, once, for a report or a training set. Enriching a product catalogue with descriptions, attributes or translations. Monthly summaries per customer, generated overnight on the first. Running your evals over a few thousand test cases every time you change a prompt.

The same question rules things out. A chatbot has a person waiting. An agent cannot plan its next step until the previous call returns, so a 24-hour window per step makes no sense. Anything with a deadline inside a few hours is a gamble: the providers promise the window, not the hour, and Anthropic warns that under heavy demand more requests expire.

The cheapest per row: batch, caching and the small tier

Batch is one of three discounts that multiply. Prompt caching cuts the repeated front of the prompt to a tenth of the input price on Anthropic, and the model tier sets the base price the percentages work on. Anthropic states that its batch and caching discounts stack. OpenAI lists batch and cached input separately without saying whether they combine.

A worked example on Anthropic's price list of 4 September 2026. You classify 100,000 tickets, each request 500 input tokens (400 of instructions and labels, 100 of ticket) and 50 output tokens, so 50 million input and 5 million output tokens.

• Live, Claude Sonnet 5 at 2 and 10: 100 plus 50, so 150 dollars.

• Live, Claude Haiku 4.5 at 1 and 5: 50 plus 25, so 75 dollars.

• Batch, Haiku 4.5 at 0.50 and 2.50: 25 plus 12.50, so 37.50 dollars.

• Batch and caching together, if the 400 instruction tokens hit the cache every time: 40 million cache reads at 0.05, 10 million uncached input at 0.50, 5 million output at 2.50, so about 20 dollars.

Treat the last line as a best case: Anthropic says cache hits inside a batch are best effort, with hit rates between 30 and 98 percent, and recommends the one-hour cache because a job can run longer than the five-minute default. Budget on the 37.50. The heaviest step is the tier, not the discount: the same job on Claude Fable 5.1 live comes to 750 dollars.

Real-time versus batch inference: who is waiting

Real-time inference has a person or a running process on the other end. The answer comes back in seconds, and the system around it is built for that: rate limits per minute, retries with backoff, a fallback model for when the provider is slow. You pay full price for the promise.

Batch inference has a queue on the other end, so the provider schedules the work and you can send 100,000 requests in one call without touching your per-minute limits. OpenAI gives batches a separate pool of much higher rate limits, and Anthropic counts batch requests against a separate queue limit. That throughput is the second reason to batch, after price.

What to watch out for with batch inference

Make the id do the work. Use the ticket number or the product SKU as your custom_id, not a counter. Resubmit the failed rows tomorrow and the same id lands the answer on the same row.

Plan for partial results. A batch can end with 99,400 answers, 500 errors and 100 expired rows. Have your loader upsert the successes and collect the failures into a new, smaller batch. Anthropic and OpenAI leave expired and errored requests off the bill, but noticing them is your job.

Never join on the order. Anthropic and OpenAI both say the output order may not match the input order. Join on the id, never on the line number.

Download before the results disappear. A job nobody looked at until an audit two months later has nothing left to show.