Self-Hosting Open LLMs vs Paying for an API: How to Work Out the Real Cost
· 7 min read · Aureus Global Systems
Key takeaway
Paying per token and paying for GPUs are two different cost models, and comparing them honestly means looking at utilisation, peak load and people, not just the price sheet. This guide walks through a break-even formula with illustrative numbers, the hidden costs of running your own inference, and when an API or a hybrid setup is the better call.
Once an LLM feature moves from prototype to production, the monthly API invoice starts getting attention. Open-weight models have become good enough for many workloads, and the obvious question follows: would it be cheaper to run the model ourselves? The honest answer is that it depends on a handful of numbers you can estimate in an afternoon. This article shows how to do that estimate properly, and where most back-of-envelope comparisons go wrong.
Two different cost models
A hosted API charges per token. Your cost scales linearly with usage, there is no cost when traffic is zero, and the provider absorbs capacity planning, redundancy and model upgrades. Input and output tokens are usually priced differently, and some providers discount cached prompt prefixes or batch (non-real-time) requests.
Self-hosting turns that into a mostly fixed cost. You pay for GPU hours whether or not they are busy, plus the engineering time to run the stack. Your marginal cost per extra token is close to zero until you hit capacity, at which point it jumps by a whole GPU (or a whole node). That step-function shape is the single most important thing to understand: self-hosting rewards high, steady utilisation and punishes spiky or low traffic.
| Hosted API | Self-hosted open model | |
|---|---|---|
| Cost shape | Variable, per token | Mostly fixed, per GPU-hour, in steps |
| Cost at zero traffic | Zero | Full GPU and ops cost |
| Capacity planning | Provider handles it (within rate limits) | You size for peak, plus headroom |
| Model choice | Provider catalogue | Any open-weight model you can license and fit |
| Data residency and control | Depends on provider and region | Fully in your environment |
| Upgrades and evals | Provider ships new versions | You test, roll out and roll back |
Estimating break-even
The break-even point is the daily token volume at which a month of self-hosting costs the same as a month of API usage. You need five inputs:
- T: tokens per day (input plus output, or model them separately if prices differ a lot).
- P: the blended API price per million tokens, weighted by your actual input/output ratio.
- R: sustained throughput per GPU in tokens per second, measured on your model, your prompt lengths and your latency target.
- H: the GPU hourly rate you will actually pay (on-demand, reserved or committed).
- E: monthly engineering and operations cost attributable to running inference.
Monthly API cost is roughly T x 30 x P / 1,000,000. For self-hosting, first work out how many GPUs you need: take your peak tokens per second (average tokens per second multiplied by a peak-to-average factor), divide by R, round up, then add redundancy. Monthly self-hosting cost is then GPUs x H x 730 hours + E. Break-even daily volume is the self-hosting monthly cost divided by (30 x P / 1,000,000).
A worked example with illustrative numbers
- Volume: assume 200 million tokens per day. At an assumed blended API price of $2 per million tokens, that is about $400 per day, or roughly $12,000 per month.
- Average load: 200,000,000 / 86,400 seconds is about 2,300 tokens per second.
- Peak load: traffic is rarely flat. With an assumed peak-to-average ratio of 3, you need to serve about 6,900 tokens per second at peak.
- GPU count: assume a measured sustained throughput of 2,500 tokens per second per GPU at your latency target. 6,900 / 2,500 rounds up to 3 GPUs; add one for redundancy and rolling deploys, giving 4.
- GPU cost: at an assumed $2.50 per GPU-hour, 4 x 2.50 x 730 is about $7,300 per month.
- People: assume a quarter of an engineer's time for operations, upgrades and on-call, costed at $4,000 per month.
- Total self-hosted: about $11,300 per month versus about $12,000 for the API.
Break-even in this example is $11,300 / (30 x $2 / 1,000,000), or about 188 million tokens per day. Above that, self-hosting wins on direct cost; below it, the API does. Notice how thin the margin is, and how sensitive it is to every assumption. Halve the peak-to-average ratio and you need two fewer GPUs. Double the API price (for example by using a larger frontier model) and the case becomes clear-cut. Pick a smaller open model that runs twice as fast and the GPU line drops sharply.
Also look at utilisation. In this example average demand is about 2,300 tokens per second against 10,000 tokens per second of provisioned capacity, so the fleet is around 23% busy on average. That idle capacity is the real cost of self-hosting, and it is why batch workloads that can be scheduled into quiet hours often improve the economics more than any engine tuning.
The hidden costs that break naive comparisons
Engineering and operations time
Someone has to choose and tune the serving engine, build container images, manage drivers and CUDA versions, wire up autoscaling, monitoring and alerting, and be on call. This is rarely zero, and for a first deployment it can be the largest line item. If no one on the team has run GPU workloads before, budget for a learning curve.
Idle capacity and peak sizing
You size for peak, not average. Autoscaling GPU nodes helps, but GPU capacity is not always available on demand, model weights take time to load, and cold starts can blow latency targets. Many teams keep a warm minimum fleet, which is exactly the idle cost the API would have absorbed.
Redundancy and reliability
A single GPU is a single point of failure. Production usually means at least N+1 capacity, possibly across zones, plus a fallback path when a node or a driver update misbehaves. Matching a major provider's availability is a real engineering project.
Evaluations and model lifecycle
Switching from a frontier API model to an open model is not free even if the hardware is. You need an evaluation set that reflects your real tasks, a way to compare quality before and after quantisation, and a process for testing and rolling out new model versions. Without evals, cost savings can quietly turn into quality regressions that show up as support tickets.
Things that are easy to forget
- Storage and network egress for model weights, logs and traces.
- Load testing time and the GPU hours it consumes.
- Security review, licence review of the model weights, and compliance sign-off.
- Commitment risk: a one- or three-year GPU reservation looks cheap until a better, smaller model makes half of it unnecessary.
When the API is still the right answer
- Low or unpredictable volume, where you would struggle to keep GPUs busy.
- Tasks that genuinely need frontier-level reasoning that available open models do not yet match on your evals.
- Early-stage products where requirements, prompts and even the model choice are still changing weekly.
- Teams without anyone who wants to own GPU infrastructure, where the opportunity cost of engineering time outweighs the savings.
- Workloads that can use a provider's batch or prompt-caching discounts, which can change the per-token maths substantially.
Self-hosting tends to make sense when volume is high and steady, when a smaller open model meets the quality bar for a well-defined task (classification, extraction, summarisation, routing, embeddings), when data must stay inside your environment, or when you need control over latency and model versions that an API cannot give you.
Hybrid routing: usually the practical answer
It is rarely all or nothing. A common pattern is to put a thin gateway in front of your models and route by task. High-volume, well-understood requests go to a self-hosted open model sized for steady baseline load. Hard or rare requests, and overflow during peaks, go to an API. The gateway gives you one place for authentication, logging, rate limits, cost attribution and fallbacks.
- Route by task type: extraction and classification to a small self-hosted model, open-ended reasoning to a frontier API.
- Route by load: serve baseline from your fleet and burst to an API instead of provisioning for peak.
- Route by confidence: let the small model answer, and escalate to a larger model when a validator or confidence check fails.
- Keep the interface consistent: most open-source serving engines expose an OpenAI-compatible endpoint, which makes swapping backends far less painful.
Hybrid routing also changes the break-even maths in your favour. Because the self-hosted fleet only has to cover baseline load, its utilisation is higher, and you stop paying for idle GPUs to absorb a peak that the API can handle.
A practical way to decide
- Pull a month of real usage: tokens in and out, by feature, by hour.
- Build an eval set for the highest-volume task and test one or two candidate open models against your current API model.
- Benchmark throughput for the candidate on your own prompts at your latency target. Do not rely on published figures.
- Run the break-even formula with your measured numbers and at least two scenarios (expected and pessimistic).
- If the case holds, start with a hybrid setup for one task, measure, then expand.
Aureus Global Systems helps teams run this analysis and, where it makes sense, build and operate self-hosted inference alongside existing API usage. If you want a second pair of eyes on your numbers, our AI inference engineering team is happy to talk it through.
Need help with this?
Our AI Inference Engineering team does this every week — as a project or as engineers on your team.
Keep reading
Migrating from OpenAI APIs to Open-Source LLMs: A Step-by-Step Playbook
Moving off a proprietary LLM API saves money only when it is done workload by workload, with an evaluation set built before any model is chosen. This playbook covers the nine steps we follow — from measuring what you spend today to a staged rollout with a fallback to the API — and the mistakes that erase the savings.
vLLM vs TensorRT-LLM vs SGLang: Choosing an LLM Inference Engine in 2026
vLLM, TensorRT-LLM and SGLang are the three serving engines most teams shortlist for self-hosted LLMs, and each makes different trade-offs between flexibility, peak performance and workload fit. This guide explains what each one is good at, when to pick which, and why quantisation, batching and right-sizing usually matter more than the engine itself.