In short
Choose by running your own evaluation set on your real inputs, not by benchmark position — public benchmarks measure tasks that are unlikely to resemble yours. Narrow the field first by hard constraints: data handling, latency, context length, and cost at your expected volume. Then design so the model can be swapped, because the landscape changes faster than most products do.
Narrow by constraints first
Before comparing quality, eliminate models that cannot work. This usually reduces a confusing field to two or three candidates.
Data handling. Where are requests processed, what is retained, and does the provider train on your inputs by default? Check the terms for your specific plan rather than the marketing page. For client work or regulated data, this can decide everything on its own.
Latency. A user-facing feature that must feel instant rules out slower reasoning models regardless of quality. Test with your real prompt length, since latency scales with input.
Context length. If you routinely send long documents, models below that threshold are out — and note that many providers charge a higher rate above a long-context threshold, so this is a cost question too.
Cost at your volume. Rates vary by more than an order of magnitude. As of September 2026, current tiers run from around $0.20 to $10 per million input tokens depending on model and provider. Estimate your monthly tokens and calculate; the answer sometimes eliminates the frontier tier immediately.
Capabilities. Structured output, tool calling, images, audio, streaming. If you need reliable JSON, that narrows things.
Benchmarks are weak evidence
Public leaderboards measure performance on standardised tasks. Yours is not a standardised task.
They are also gamed, saturate quickly, and rank models within a point or two of each other in ways that do not survive contact with real inputs. A model two places higher on a leaderboard is not reliably better at classifying your support tickets.
Use benchmarks for one thing: generating a shortlist. Then discard them and test properly.
Test with your own evaluation set
This is the only comparison worth trusting.
Take twenty to fifty real inputs, define what a good answer looks like, and run every candidate against them. Keep the prompt identical, and record cost and latency alongside quality — the evaluation discipline you need anyway.
What this usually reveals: the difference between tiers is smaller than expected on ordinary tasks and larger than expected on hard ones. Which means the right answer is often a cheaper model for most requests, with escalation for the difficult minority.
Also test the failure modes. How does each handle ambiguous input, out-of-scope requests, and adversarial phrasing? Refusal behaviour differs substantially between providers and matters more in production than benchmark scores.
Start cheaper than you think
The most common expensive mistake is defaulting to the frontier tier because it is safest, then never revisiting.
Start on the balanced tier. Move up only when your evaluation shows a specific failure you can measure. Move down for high-volume, low-judgement work — classification, extraction, routing, tagging — which frequently runs fine on the cheapest option.
With a twenty-five-fold spread between tiers, this decision has more effect on your bill than any other optimisation, as covered in controlling AI costs.
Hosted or self-hosted
Hosted is right for almost everyone starting out: no infrastructure, immediate access to current models, and someone else handling scaling.
Self-hosted open models make sense when data cannot leave your environment, when volume is high enough that per-token pricing exceeds infrastructure cost, when you need a model that will not change underneath you, or when you need to fine-tune deeply.
Be honest about the operational cost. Running inference well — GPU capacity, batching, monitoring, updates — is real work, and teams frequently underestimate it because the model download is free.
Design for switching
The strongest position is not picking correctly. It is being able to change your mind cheaply.
Put the model behind an interface so provider specifics live in one adapter rather than throughout the codebase.
Keep prompts in configuration, not scattered inline.
Maintain the evaluation set, so a candidate can be tested in an afternoon.
Avoid provider-specific features where equivalents exist, unless the benefit is substantial.
The landscape changes faster than most products. A team that can evaluate and switch in a week is in a better position than one that chose optimally a year ago and cannot move.
Reviewing the decision
Re-run your evaluation quarterly, and whenever a provider announces something significant. Prices fall, models improve, and the tier that was necessary last year is often unnecessary now.
Also re-run it when nothing has changed on your side, because providers update models in place. That is the only way to catch a regression before your users do.
If you are choosing a model for something that will run at volume, book a call — the evaluation is usually a day and it changes the answer more often than not.
Common questions
How do I choose which LLM to use?
Narrow by hard constraints first — data handling terms, latency, context length, cost at your volume, and required capabilities — which usually reduces the field to two or three. Then run your own evaluation set of real inputs against each, keeping the prompt identical and recording cost and latency alongside quality.
Are LLM benchmarks useful for choosing a model?
Only for generating a shortlist. Public leaderboards measure standardised tasks that are unlikely to resemble yours, they saturate quickly, and they separate models by margins that do not survive contact with real inputs. A model two places higher is not reliably better at your specific job.
Should I start with the most capable model?
No — start on the balanced tier and move up only when your evaluation shows a measurable failure. With rates spanning more than an order of magnitude between tiers, defaulting to the frontier model is the most common way teams overspend, and high-volume low-judgement work usually runs fine on the cheapest option.
When does self-hosting an open model make sense?
When data cannot leave your environment, when volume makes per-token pricing exceed infrastructure cost, when you need a model that will not change underneath you, or when you need deep fine-tuning. Be realistic about the operational cost — running inference well is real work that teams underestimate.
How do I avoid being locked into one provider?
Put the model behind an interface so provider specifics live in one adapter, keep prompts in configuration rather than inline, avoid provider-specific features where equivalents exist, and maintain an evaluation set so a candidate can be tested in an afternoon.
