Inference
Inference is the act of running a trained model to produce output — the step you pay for each time a request is made, as distinct from the one-off cost of training the model.
Inference is billed per token, separately for input and output, and output typically costs several times more than input. This is why the cheapest way to reduce an AI bill is usually to request tighter output rather than to change model.
Two further levers apply on most platforms: batch processing, which commonly halves the rate for work that need not be real time, and prompt caching, which bills the stable portion of a repeated prompt at a fraction of the standard input rate.
How it works
When a request arrives, the provider loads the model onto accelerators, processes the input tokens in one pass, then generates the output one token at a time, each step reading everything before it. Input processing is fast and parallel; output generation is sequential, which is why long answers take longer and cost more.
Latency and price depend on the model's size, the length of input and output, and the hardware it runs on, which is why the same task can cost very different amounts on different models.
Example
A team summarising a thousand support tickets overnight can send them as a batch job at a reduced rate, keep the shared instructions identical so they are cached, and ask for a three-line summary rather than a paragraph, cutting the bill several times over without changing model.
Common mistakes
- Using the largest model for every task when a smaller one handles most of them well.
- Asking for long, open-ended output when a short, structured answer would do.
- Running real-time requests for work that could wait for cheaper batch processing.
