Self-Hosting an LLM vs Using an API: The Cost Math

The GPU is the cost everyone calculates. The on-call engineer, idle hours, upgrades and evaluation work are the costs that actually decide this.

Self-Hosting an LLM vs Using an API: The Cost Math — Troiana insight cover

In short

For most teams, an API is cheaper and simpler until volume is very high or data rules demand otherwise. A single rented GPU can cost $248–$650 a month before staffing, which equals hundreds of millions of tokens on budget APIs. Self-host when privacy, compliance, customisation or steady high volume justify the engineering work, and start by measuring your real usage.

The question people ask, and the better one

“Is it cheaper to host it ourselves?” is usually answered by comparing a GPU's rental price with an API's token price. That leaves out most of the cost. The better question is: what's the total cost of running this reliably, at the quality we need?

Our shorter guide, what is the cheapest way to run an LLM?, covers all the options. This piece looks at the decision in more depth, for teams thinking seriously about self-hosting.

The full cost of self-hosting

CostExample or question
GPU computeRunPod lists an RTX 4090 from $0.34/hour, and an H100 from $1.99/hour
RedundancyA second instance, so that you can survive one failing
Idle timePaying through nights and weekends unless you scale down
Engineering timeServing, scaling, updates, drivers, security
On-callWho gets woken if inference fails at 3am?
EvaluationTesting each new model before you switch
MonitoringLatency, throughput, errors and quality
Opportunity costWhat your team isn't building meanwhile

A worked comparison

Let's use current list prices from November 2026.

One RTX 4090, running continuously (about 730 hours):

  • Community Cloud at $0.34: about $248
  • Secure Cloud at $0.89: about $650

Two H100s on Secure Cloud, for redundancy and more capacity (at $2.89 each for PCIe):

  • 730 × $2.89 × 2 = about $4,220 a month

A share of an engineer's time. Suppose someone spends a fifth of their time on this. Use your real salary figure, but even at a modest assumption, this could easily be larger than the compute bill for smaller setups.

The API, for comparison. DeepSeek's Flash model lists at $0.15 input and $0.60 output per million tokens off-peak. At a 5:1 mix, that's about $0.225 per million tokens. So:

  • $650 buys roughly 2.9 billion tokens a month
  • $4,220 buys roughly 18.8 billion tokens a month

Top-tier models cost much more. At $2 input and $10 output, the same mix works out at about $3.33 per million, so $4,220 buys about 1.27 billion tokens. So the break-even depends heavily on which API you'd otherwise use. It also matters whether your self-hosted model can really do the same job, since a model that fits your hardware might be weaker.

Compute per month, before staffing, log scale
One RTX 4090, Community CloudAbout $248
One RTX 4090, Secure CloudAbout $650
Two H100s, Secure CloudAbout $4,220
$100$1,000$10,000
730 hours at the list prices above. The same money buys roughly 2.9 billion and 18.8 billion tokens on the budget API used in this comparison.

Reasons to self-host other than price

Data control. If contracts or regulation mean that data can't leave your environment, then the price comparison doesn't matter much. Be aware that big cloud providers also offer hosted models within your own cloud account, which may satisfy this need without you running the serving stack. See is it safe to send customer data to an LLM?

Customisation. Fine-tuned models, unusual architectures or custom inference settings might not be offered by any provider. See should you fine-tune or just prompt better?

Stability. You choose when a model changes. Hosted models can be deprecated, or change behaviour, which matters if you've carefully tested an application against one version.

Predictable latency. A dedicated instance isn't competing with other customers, although you still have to size it correctly.

Independence. You're insulated from provider price changes, outages or policy decisions. An open-weight model gives you that fallback, even if you never use it day to day.

Reasons to stay on an API

  • Quality. Frontier models may be ahead of what you can host, especially on harder tasks.
  • Elasticity. Traffic spikes aren't your problem, and quiet periods cost nothing.
  • Speed. You can have a new model in production with a config change.
  • Caching and batching discounts. These can reduce costs substantially, with no work from you.
  • Focus. Your customers don't care who runs the inference.

What running it properly involves

Self-hosting is more than installing a tool on a server. For production traffic, you'll need:

  1. A serving engine designed for concurrency, such as vLLM, rather than a desktop tool. See vLLM vs Ollama.
  2. Autoscaling or capacity planning, plus rate limiting and queueing.
  3. Authentication and network controls. An open inference endpoint is a real risk.
  4. Monitoring and alerts for latency and errors.
  5. An evaluation suite to check quality on your tasks before upgrading. See how to evaluate an LLM feature.
  6. A fallback, often to a hosted API, when your own service is down.

A sensible path

  1. Start on an API. Ship, and find out if anybody wants the feature.
  2. Keep your code portable. Use an OpenAI-compatible interface and keep model names in configuration.
  3. Log real usage: tokens, cost per task, latency and quality.
  4. Reduce API costs first, through caching, routing and shorter prompts. See how to control AI API costs.
  5. Pilot self-hosting on one low-risk workload, such as batch processing, where uptime is less critical.
  6. Move further only if the numbers hold up, and keep a hosted fallback.

Mixed setups often work best. A team might run a self-hosted model for high-volume classification, or for sensitive documents, while using hosted models for everything else.

When the numbers do favour self-hosting

As a rough checklist, self-hosting begins to make sense when most of these are true:

  • You have steady, high volume, with little idle time
  • An open model meets your quality bar, proven by testing
  • You have or can hire people who can run it
  • Your API bill is large enough to be worth an engineer's attention
  • You have a data, control or customisation reason as well as cost

If you're only meeting one of those, you're probably better off staying on an API for now.

Want a second opinion on your numbers? Book a call.

Common questions

Is self-hosting an LLM cheaper than an API?

Usually not at low or moderate volume, once you count idle time and engineering work. It can win at steady, very high volume, especially against premium API prices. Compare your full costs at your own real usage and quality bar.

What do I need to self-host an LLM in production?

A GPU server, a serving engine built for concurrency such as vLLM, authentication, monitoring, autoscaling or capacity planning, plus an evaluation process for new models. A hosted fallback is sensible too. A desktop tool isn't enough for production traffic.

Can I keep data private without self-hosting?

Often, yes. Enterprise APIs and models hosted in your own cloud account may meet your requirements, with suitable contract terms. Check data retention, training use and processing locations with your legal team before assuming you have to run models yourself.

Which models can I self-host?

Open-weight models from various labs, with licences that differ on commercial use and scale. Check each licence and test quality on your own tasks. The right choice changes quickly, so keep your stack flexible enough to swap models.

Have something worth building right?