Cost tool

LLM inference cost calculator

Three ways to serve an open model — a managed API, a rented cloud GPU, or your own hardware — priced side by side over three years. The comparison includes the costs teams usually forget: setup, maintenance, power, and the engineering time to keep it running.

How to read this comparison

The three options fail in different directions. A managed API costs nothing when idle and everything at scale, because you pay per token forever. A cloud GPU flips that: the meter runs on time, not tokens, so it rewards steady utilization and punishes bursty traffic — an instance at 5% utilization still bills for 100% of the hour. Self-hosting has the lowest marginal cost per token but the highest fixed cost and the longest payback, and it only wins when volume is both high and predictable.

The break-even figures are the part worth acting on. If your volume sits far below the crossover point, the decision is already made — use an API and spend the engineering time elsewhere. If you are near it, remember that the estimate assumes your GPU stays busy; halve the utilization and the crossover moves substantially. Before committing, confirm the memory footprint of your exact model and context length in the VRAM Calculator, check which card actually fits with Can I Run It, and read Open vs Closed Models for the governance side of the same decision.

How these numbers are calculated

Each option is built from a different formula, which is why they cross over rather than scale together.

1. Managed API

Monthly cost is simply (tokens ÷ 1,000) × rate. The rate is not fixed: serverless hosts charge by model size, so we derive a parameter count from the FP16 footprint (roughly params ≈ VRAM ÷ 2.3, allowing two bytes per parameter plus about 15% overhead) and step the rate across the observed 2026 range — from $0.0001 per 1K tokens under 5B parameters to $0.00104 at 70B-class. Hugging Face Inference is modelled at 2× that base and Replicate at 2.5×, with a frontier closed API at $0.0036 per 1K tokens for contrast. There is no setup or maintenance line, because there is no infrastructure.

2. Rented cloud GPU

Here the meter runs on time, not tokens: hourly rate × hours per day × days per month, plus a one-off setup cost and a recurring monthly maintenance line. The calculator picks the cheapest instance whose GPU actually holds the model — a T4 at $0.526/hr up to 12 GB, an L4 at $0.85/hr or A10G at $1.006/hr up to 20 GB, an A100 at $4.098/hr up to 35 GB. Because the term is hours rather than tokens, an instance sitting idle costs exactly as much as one saturated with traffic.

3. Self-hosted hardware

This is the only option with real capital cost, and it has four terms: hardware selected by VRAM requirement and depreciated over 36 months; supporting infrastructure at a flat $2,800 for chassis, networking, UPS, and cooling; recurring running costs of power plus connectivity, maintenance, and backup; and personnel. Power is metered honestly at (watts ÷ 1,000) × 24 × 30 × $0.12/kWh. Personnel assumes a quarter of a DevOps engineer at $120,000 and a tenth of an MLOps engineer at $140,000 — about $3,667 a month, and almost always the dominant term.

A worked example

Take Llama 3 8B at roughly 16 GB in FP16, serving 50 million tokens a month — a real but modest internal workload. On the API path, 16 GB implies about 7B parameters, so the rate is $0.0002 per 1K tokens: 50,000 thousand-token units × $0.0002 = $10 a month, or $120 for the year.

On a rented L4 running around the clock, 720 hours × $0.85 = $612 a month. Add $500 a month of maintenance and $200 of setup and year one lands at $13,544.

Self-hosting looks cheapest on hardware and is not close overall. A 16 GB card costs $600 and draws 160 W, so power is about $14 a month; with connectivity, maintenance, and backup the running total is $364 a month. Add $2,800 of infrastructure and $44,000 of partial headcount and year one is $51,768 — over four hundred times the API bill, and the GPU is 1% of it.

That is the whole lesson of the tool. At this volume the decision is not close, and it does not become close by buying a cheaper card. Self-hosting starts winning when token volume climbs far enough that the per-token API line overtakes a fixed staffing cost you were going to pay anyway — which is why teams that already run infrastructure reach that point much sooner than teams that would be staffing up for it.

What this model does not capture

These are planning estimates, and four limits are worth stating plainly. First, per-token pricing moves constantly and varies several-fold between providers; the rates here were current in July 2026 and your input/output ratio shifts the effective cost further. Second, the model assumes steady utilization — bursty traffic makes the rented-GPU column look far worse in reality than it does here, and a reserved or spot instance makes it look better. Third, the personnel figures are US-market salaries at an assumed split; a team already carrying that headcount should read the self-hosted column as marginal cost, not total. Fourth, nothing here prices latency, data residency, or the risk of a provider deprecating a model underneath you — for many teams those decide the question before cost does.

Use the shape of the comparison and the break-even point rather than the absolute figures, then confirm against current provider pricing before committing budget.

Frequently asked questions