LLM Self-Hosting Cost Calculator

About this LLM self-hosting calculator

This free, browser-based calculator estimates whether an open-weight LLM will run on your own GPU or server, how fast it will generate tokens, and whether self-hosting is cheaper than a comparable hosted API. Start from your hardware — pick a GPU or accelerator, a quantization (FP16, FP8, INT8, NVFP4, MXFP4 or INT4), and how many GPUs you plan to use — and it shows the models that fit, their VRAM breakdown (weights + KV cache + overhead), single-stream and served tokens per second, and an estimated cost per 1M tokens.

Because efficiency changes with scale, you can set the total number of GPUs. The calculator forms data-parallel replicas (replicas = total GPUs ÷ GPUs needed per replica), scales throughput almost linearly, and prices the whole fleet including any idle GPUs — so you can directly compare running 2 GPUs versus 100. It then weighs your electricity- or rental-based self-hosting cost against the per-token price of a spec-similar commercial API (Claude, GPT, Gemini, Grok, or hosted open models) to tell you which is cheaper at your utilization.

Step 2 also pairs each self-host model with the commercial API of similar measured intelligence using the Artificial Analysis Intelligence Index — a composite that averages roughly ten public benchmarks — so you can see, for example, which open model lands near the GPT-5.6, Claude Opus 4.8 or Grok 4.5 tier. Every estimate is computed locally in your browser from public specifications; no prompt, key or data is uploaded.

Quantization (precision) guide — vs typical API models

This calculator's quantization setting mirrors how hosted APIs actually run models. Commercial API providers typically serve at FP8 or BF16, so choosing the same precision keeps the self-host-vs-API comparison fair. Use this guide:

How long a context do you actually need? (reference)

• Short chat / simple Q&A — 2K–8K tokens is plenty.
• RAG or single-document Q&A — 16K–32K covers most retrieved chunks.
• Long documents, multi-file code, or agent runs — 64K–128K.
• Whole codebases, books, or very long chat sessions — 256K and up.

Context length is just the working-memory window per request — longer isn't 'smarter', it only lets the model see more at once, and it grows KV-cache VRAM linearly. Pick the smallest size that fits your task.

What does 'concurrent requests' mean — and why 16?

• It's how many requests the GPU serves in parallel (the batch size). It raises TOTAL throughput — tokens/sec added up across all users — but it does NOT make a single answer better, smarter, or faster.
• 16 is a sensible middle default: a single GPU can usually batch roughly 8–32 sequences before VRAM headroom for the KV cache runs out. Push it higher and total throughput keeps rising until the VRAM headroom is gone — after that it stops helping (and the calculator caps the effective batch at what fits).
• Set it to how many users you realistically expect at the same time: 1–4 for a personal single-user setup, 16–64 for a shared team endpoint. It only affects the throughput and cost-per-token numbers, never quality.

Inference speed vs serving throughput

Two different numbers matter, and mixing them up is a common mistake. Inference speed is how fast a single request generates tokens (tokens/sec); it is fixed by how many GPUs share one copy of the model through tensor parallelism, so adding more total GPUs does not make one answer faster. Serving throughput is the total tokens/sec across all concurrent requests; it scales almost linearly as you add GPUs, because each additional group runs another data-parallel replica. In short, more GPUs buy you more total throughput and a lower cost per token — not a quicker single response.

If you BUY the hardware — ownership cost (TCO)

The rows above use rental or electricity only. This breakdown assumes you purchase the GPU instead: its street price is straight-line depreciated over 1 year (8760h, running 24/7), plus an estimated host PC/server and electricity — giving a more complete cost per token.

Scores use the Artificial Analysis Intelligence Index (July 2026 snapshot — a composite averaging ~10 public benchmarks). Cross-check against the live leaderboards: artificialanalysis.ai · benchlm.ai · GPU street-price references (approximate, 2026-07): gpucost.org

Frequently asked questions