The local-AI field guide

§01 — the math nobody tells you

Your "32 GB" is not 32 GB.

VRAM is shared between the model's weights, the framework's CUDA graphs, activation & multimodal buffers, per-sequence state, and the KV cache that grows with every token of context. Before you pick a model, do the subtraction — here's a 32 GB card serving a 30B model in NVFP4 at long context:

weights graphs + buffers KV free

Two things that trip everyone up: weights-in-VRAM ≠ checkpoint-on-disk (a "~18 GB" model loaded clocks ~21.5 GiB), and on a Mamba-hybrid the KV cache is not the context bottleneck — only the few attending layers cost KV, so the headline "256K context" is bounded by total VRAM, not by KV. The same 30B in FP8 (~29 GiB of weights alone) wouldn't fit at all — which is exactly why NVFP4 earns its place. I had this math 14× wrong once and wrote it down: the KV-cache correction and why 32 GB is a budget, not a suggestion.

§02 — what fits

Will it fit? Do the subtraction.

Enter a card and a model. The verdict is honest about headroom — a model that "fits" with nothing to spare will fragment, swap, and crawl under real load.

weights = params × bytes/param × quant-overhead · KV = KB/token × context × sequences · total = weights + KV + overhead. Switching architecture changes KV by ~40× — that's the whole point of the KV post. KV cost and overhead are pre-filled estimates; edit them for your model.

§03 — the memory tier map

What runs where.

Usable VRAM by card, and the class of model it opens up. The jump from 16 GB to 24 GB is where local AI gets genuinely useful; 32 GB is where MoE models (only a few B active per token) start to shine.

The 32 GB row is measured: Nemotron (FP8 reasoning), a 20B image hero at ~28 GiB peak, 30B-A3B MoE — all on one RTX 5090, sequentially, never co-loaded. Smaller-card rows are estimates using the ~70% usable heuristic; a unified-memory Mac is a different regime (the OS reserves a chunk and it's shared with the display).

§04 — how many users

One card, how many people?

Continuous batching means throughput does not divide by user count — but KV-cache memory caps how many sequences can generate at once. Measured on the 5090: single stream ~276 tok/s, 2 users ≈ 1.4×, 4 ≈ 2.4×, and it knees at ~8 concurrent.

Past the knee, aggregate throughput is flat and latency climbs — adding users just makes everyone wait. Rule of thumb on 32 GB: ~10–15 heavy or ~30–40 light chat users, latency under ~2.5 s. Your model, context, and quant move the knee.

§05 — cloud vs local

When does owning the rig pay off?

Cloud API spend is a tax that never stops. A local rig is a one-time cost that depreciates — and your data never leaves the building. Enter your usage to see roughly when local wins. Estimates, not a quote — every number is editable and the assumptions are shown.

Assumes the rig draws ~0.5 kW under sustained AI load and runs ~6 h/day of actual inference. Local image/token costs are electricity only (machine cost per job is cents — the rig is the real number). Doesn't price your time, or the value of data never leaving your building, which for regulated work is often the whole point. Want this deployed?

§06 — what a clip actually costs

The machine cost of a creative job is cents.

Every job below is a measured wall-clock run from the case studies — a 34.5-second avatar clip, a real object-removal shot. Pick one, and see what the machine actually spends on it: electricity plus a fair share of hardware depreciation. Spoiler: the GPU is never the expensive part.

That gap — cents of machine cost vs a listed price in the tens or hundreds — is not margin on electricity. It's the operator: quality tiers chosen honestly, a QA pass, the failure modes known in advance, and a redo when the footage is hard. The other arithmetic worth knowing: at this utilisation the machine's entire monthly cost — depreciation and electricity together — is covered by roughly ten billable hours of work a month, and a modest steady workload pays the whole rig off in about 13 months. Owning the hardware is not the risky part of this business.