GPU memory & the model

1 · Where the VRAM actually goes essential

A GPU's memory (VRAM) is a hard ceiling — an A100 has 40 or 80 GB, an H100 80 GB. When people say a model "doesn't fit", they mean the sum of four things exceeds that ceiling. Learn the four line items and you can predict fit before you launch:

Inference pays for weights + KV cache (plus a little working memory). Training adds activations + gradients + optimizer state — which is why the same model needs far more VRAM to train than to serve.

2 · Precision — bytes per parameter essential

Every number in the model occupies a fixed number of bytes set by its precision. This single constant drives the whole weight-memory calculation. Memorize the table:

fp16 and bf16 are both 2 bytes — they differ in how the 16 bits split between exponent and mantissa (bf16 trades precision for fp32's dynamic range, which is why it's the training default). For memory math they're identical. Quantization (IC2) is precision applied to the weights after training; here we care about the bytes it buys you.

vram_weights.pyGIB = 1024 ** 3  # bytes in a gibibyte

def vram_for_model(params_b, bytes_per_param):
    """Weight memory in GiB. params_b = billions of params."""
    total_bytes = params_b * 1e9 * bytes_per_param
    return total_bytes / GIB

BYTES = {"fp32": 4, "fp16": 2, "bf16": 2, "fp8": 1, "int8": 1, "int4": 0.5}

for size in (7, 13, 70):
    line = [f"{size:>3}B:"]
    for fmt in ("bf16", "int8", "int4"):
        gib = vram_for_model(size, BYTES[fmt])
        line.append(f"{fmt}={gib:6.1f} GiB")
    print("  ".join(line))

  7B:  bf16=  13.0 GiB  int8=   6.5 GiB  int4=   3.3 GiB
 13B:  bf16=  24.2 GiB  int8=  12.1 GiB  int4=   6.1 GiB
 70B:  bf16= 130.4 GiB  int8=  65.2 GiB  int4=  32.6 GiB

The formula is just params × bytes/param. A 70B model in bf16 is 70e9 × 2 bytes ≈ 130.4 GiB of weights alone — already over a single 80 GB card. In int4 (0.5 bytes) it's ≈ 32.6 GiB, which fits. That is the whole reason quantization exists for serving large models.

3 · The KV cache — memory that grows with tokens intermediate

During generation the model caches the key and value vectors for every token it has already seen, at every layer, so it never recomputes attention over the past (IC3). That cache is pure VRAM overhead on top of the weights, and it grows linearly with sequence length and batch size. The formula:

kv_bytes = 2 × n_layers × n_kv_heads × head_dim × seq_len × batch × bytes_per_elem

The leading 2 is for K and V. Note n_kv_heads, not query heads — grouped-query attention (GQA) shrinks the cache by sharing KV across query heads, which is exactly why modern large models use it.

kv_cache.pyGIB = 1024 ** 3

def kv_cache_gb(n_layers, n_kv_heads, head_dim, seq_len, batch, bytes_per_elem=2):
    """KV-cache VRAM in GiB. Factor 2 is for K and V."""
    elems = 2 * n_layers * n_kv_heads * head_dim * seq_len * batch
    return elems * bytes_per_elem / GIB

# 70B-class model with grouped-query attention (8 KV heads), fp16 cache
cfg = dict(n_layers=80, n_kv_heads=8, head_dim=128, seq_len=8192)

for batch in (1, 8, 32):
    gib = kv_cache_gb(batch=batch, **cfg)
    print(f"batch={batch:>2} @ 8k ctx -> KV cache = {gib:6.2f} GiB")

batch= 1 @ 8k ctx -> KV cache =   2.50 GiB
batch= 8 @ 8k ctx -> KV cache =  20.00 GiB
batch=32 @ 8k ctx -> KV cache =  80.00 GiB

For a 70B-class model (80 layers, 8 KV heads via GQA, head_dim 128) a single 8k-token sequence in fp16 is only ~2.5 GiB — but 32 concurrent sequences is ~80 GiB, the whole card on its own. KV cache, not weights, is usually what OOMs a busy inference server.

4 · Activations & the training memory blow-up advanced

Inference is memory-light: weights + KV cache and you're done. Training is a different regime. Backprop must keep the forward activations around to compute gradients, and the optimizer keeps its own state per parameter. The four-way split for Adam-style training in mixed precision:

That's roughly 14 bytes/param just for weights + gradients + optimizer state in the classic mixed-precision Adam recipe (4 + 2 + 4 + 4) — before a single activation. A 7B model is thus ~91 GiB of state, which is why it does not train on one 80 GB card without sharding (MS2) or memory-saving tricks. Activations are extra on top and scale with batch × seq × hidden × layers; gradient checkpointing trades compute to shrink them.

train_memory.pyGIB = 1024 ** 3

def inference_state_gb(params_b, bytes_per_param=2):
    """Just the weights (bf16 serving)."""
    return params_b * 1e9 * bytes_per_param / GIB

def training_state_gb(params_b):
    """Mixed-precision Adam persistent state, bytes/param:
       fp32 master weights (4) + grads (2) + Adam m (4) + Adam v (4) = 14."""
    bytes_per_param = 4 + 2 + 4 + 4
    return params_b * 1e9 * bytes_per_param / GIB

for size in (7, 13):
    inf = inference_state_gb(size)
    tr = training_state_gb(size)
    print(f"{size:>2}B: serve(bf16)={inf:6.1f} GiB | "
          f"train(Adam)={tr:6.1f} GiB | {tr/inf:.1f}x more")

 7B: serve(bf16)=  13.0 GiB | train(Adam)=  91.3 GiB | 7.0x more
13B: serve(bf16)=  24.2 GiB | train(Adam)= 169.5 GiB | 7.0x more

The same 7B model: ~13 GiB to serve in bf16, ~91 GiB of persistent state to train with Adam — a 7× gap, and that's before activations. This is the single most important reason training needs clusters while inference often needs one card.

5 · Does it fit? Batch & sequence vs the ceiling professional

Capacity planning is one question: does weights + KV cache + headroom fit the card? Weights are fixed; KV cache is the dial you actually turn with batch size and context length. The professional move is to compute the answer, not to try-and-OOM in production.

fits_80gb.pyGIB = 1024 ** 3

def vram_for_model(params_b, bytes_per_param):
    return params_b * 1e9 * bytes_per_param / GIB

def kv_cache_gb(n_layers, n_kv_heads, head_dim, seq_len, batch, bytes_per_elem=2):
    elems = 2 * n_layers * n_kv_heads * head_dim * seq_len * batch
    return elems * bytes_per_elem / GIB

def fits_80gb(params_b, bytes_per_param, kv_gb, headroom=0.10):
    """Does weights + KV cache fit an 80 GB card with headroom to spare?"""
    card = 80.0
    usable = card * (1 - headroom)
    need = vram_for_model(params_b, bytes_per_param) + kv_gb
    return need, usable, need <= usable

# 13B bf16 serving 8 users at 4k ctx
kv = kv_cache_gb(n_layers=40, n_kv_heads=40, head_dim=128, seq_len=4096, batch=8)
need, usable, ok = fits_80gb(13, 2, kv)
print(f"13B bf16 + KV {kv:.1f} GiB -> need {need:5.1f} / {usable:.1f} usable -> "
      f"{'FITS' if ok else 'OOM'}")

# 70B bf16, weights alone
need, usable, ok = fits_80gb(70, 2, 0.0)
print(f"70B bf16 weights only     -> need {need:5.1f} / {usable:.1f} usable -> "
      f"{'FITS' if ok else 'OOM'}")

# 70B int4, weights alone
need, usable, ok = fits_80gb(70, 0.5, 0.0)
print(f"70B int4 weights only     -> need {need:5.1f} / {usable:.1f} usable -> "
      f"{'FITS' if ok else 'OOM'}")

13B bf16 + KV 25.0 GiB -> need  49.2 / 72.0 usable -> FITS
70B bf16 weights only     -> need 130.4 / 72.0 usable -> OOM
70B int4 weights only     -> need  32.6 / 72.0 usable -> FITS

Read the output as a capacity plan: a 13B model in bf16 on an 80 GB card leaves plenty of room for KV cache, so you can push batch/context high; a 70B model in bf16 doesn't fit at all and must be quantized or sharded. The knobs — precision, batch size, max sequence length — are exactly what you tune to stay under the ceiling while maximizing throughput (IC1/IC4).

6 · Debugging an OOM methodically professional

"CUDA out of memory" is not a mystery once you know the four consumers. Debug it as a budget problem, in order — attribute the memory, then attack the biggest line item:

OOM debugging methodology

1. Read the allocator, don't guess. Log torch.cuda.memory_allocated() and max_memory_allocated() around load and around a forward pass to see what's resident vs. peak.
2. Attribute it to a consumer. Weights are constant; if memory climbs with concurrency it's KV cache; if it spikes only during .backward() it's activations/optimizer.
3. Attack the biggest line item. KV cache → cap batch/context, use GQA/paged attention, quantize the KV cache. Weights → quantize (IC2) or shard (MS2). Optimizer → LoRA/QLoRA (FT3), 8-bit optimizer, or shard.
4. Reduce activations if training. Enable gradient checkpointing, lower the micro-batch, use gradient accumulation to keep the effective batch while cutting peak memory.
5. Fix fragmentation last. If allocated is well under total but you still OOM, set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and avoid wild batch-size swings.

7 · Measuring it for real on a GPU advanced

Everything above is arithmetic you can do offline. On real hardware you confirm it with the CUDA allocator. This snippet is real and correct but needs a GPU + PyTorch — it will not run in this offline environment; it's here so you know exactly what to measure:

measure_gpu.py# >>> needs a GPU + PyTorch: pip install torch transformers accelerate
# This is REAL and correct but will NOT run in an offline/CPU-only environment.
import torch
from transformers import AutoModelForCausalLM

def mib(n):
    return n / (1024 ** 2)

# Load weights in bf16 to halve the memory vs fp32.
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.3",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)

# Confirm the arithmetic from vram_weights.py against the real allocator.
print(f"allocated after load: {mib(torch.cuda.memory_allocated()):.0f} MiB")

ids = torch.randint(0, 32000, (1, 512), device="cuda")
with torch.no_grad():
    _ = model(ids)  # forward pass builds a little KV/working memory

print(f"peak allocated:       {mib(torch.cuda.max_memory_allocated()):.0f} MiB")
# ~14 GiB resident for a 7B model in bf16 -> matches vram_for_model(7, 2).

8 · Tech-lead — VRAM as a capacity-planning discipline tech-lead

A lead doesn't launch instances to find out what fits. They keep a VRAM budget model — the four consumers, parameterized by precision, batch, and context — and use it to choose hardware, set max_model_len and batch limits, and forecast cost per token (IC1). The calculators in this lesson are that model in miniature: they turn "will it fit?" from an expensive experiment into a line of arithmetic.

The strategic levers a lead reaches for, in order of cheapness: quantize weights (IC2) to cut the fixed cost; cap context and batch to bound the KV cache; GQA / paged KV / KV quantization to serve more concurrency per card; and only then shard across GPUs (MS2) when a single card genuinely can't hold the model. Sharding adds communication cost, so it's the last resort, not the first reach.

def weight_gib(params_b, bytes_per):
    total_bytes = params_b * 1e9 * bytes_per
    return total_bytes / (1024 ** 3)

for name, p in [("7B", 7), ("13B", 13), ("70B", 70)]:
    bf16 = weight_gib(p, 2)
    int8 = weight_gib(p, 1)
    int4 = weight_gib(p, 0.5)
    print(f"{name:>4}:  bf16={bf16:6.1f} GiB  int8={int8:6.1f} GiB  int4={int4:6.1f} GiB")
# 7B: bf16 ~13.0, int8 ~6.5, int4 ~3.3 -- halving bytes halves the footprint

def kv_gib(n_layers, n_kv_heads, head_dim, seq, batch, bytes_per=2):
    elems = 2 * n_layers * n_kv_heads * head_dim * seq * batch
    return elems * bytes_per / (1024 ** 3)

# a 70B-ish config: 80 layers, 8 KV heads, head_dim 128, bf16
for b in (1, 8, 32):
    gib = kv_gib(n_layers=80, n_kv_heads=8, head_dim=128, seq=8192, batch=b)
    print(f"batch={b:>2} @ 8k ctx -> KV cache = {gib:6.2f} GiB")
# batch 1 -> 2.50, batch 8 -> 20.00, batch 32 -> 80.00 : linear in batch AND seq

def serve_gib(params_b):        # bf16 weights only
    return params_b * 1e9 * 2 / (1024 ** 3)

def train_gib(params_b):        # bf16 weight + fp32 grad + fp32 m + fp32 v
    bytes_per = 2 + 4 + 4 + 4   # = 14 bytes/param (Adam, mixed precision)
    return params_b * 1e9 * bytes_per / (1024 ** 3)

for name, p in [("7B", 7), ("13B", 13)]:
    s, t = serve_gib(p), train_gib(p)
    print(f"{name}: serve(bf16)={s:6.1f} GiB | train(Adam)={t:6.1f} GiB | {t/s:.1f}x more")
# ~7x more -- optimizer state, not the weights, dominates a training budget

def usable(card_gib, headroom=0.10):
    return card_gib * (1 - headroom)

def fits(need_gib, card_gib, headroom=0.10):
    u = usable(card_gib, headroom)
    verdict = "FITS" if need_gib <= u else "OOM"
    return f"need {need_gib:6.1f} / {u:4.1f} usable -> {verdict}"

W = lambda p, bp: p * 1e9 * bp / (1024 ** 3)
print("13B bf16 + KV 25.0 GiB    ->", fits(W(13, 2) + 25.0, 80))
print("70B bf16 weights only     ->", fits(W(70, 2), 80))
print("70B int4 weights only     ->", fits(W(70, 0.5), 80))
# 13B+KV fits; 70B bf16 OOMs (130 GiB); 70B int4 fits (33 GiB)

def triage(dominant, over_by_gib):
    # dominant term from a memory profile: 'kv', 'weights', or 'activations'
    plan = []
    if dominant == "kv":
        plan += ["lower max batch / concurrent seqs",
                 "cap context length or enable paged-KV (vLLM)"]
    elif dominant == "activations":
        plan += ["enable gradient/activation checkpointing",
                 "reduce micro-batch, use grad accumulation"]
    elif dominant == "weights":
        plan += ["quantize weights (bf16 -> int8 -> int4)",
                 "shard weights across GPUs (tensor parallel / FSDP)"]
    plan.append(f"still {over_by_gib:.0f} GiB over? move to a bigger card")
    return plan

for step in triage("kv", 12):
    print("-", step)

import math

def kv_per_user_gib(n_layers=80, n_kv_heads=8, head_dim=128, seq=8192, bytes_per=2):
    return 2 * n_layers * n_kv_heads * head_dim * seq * bytes_per / (1024 ** 3)

def plan(users, card_gib=80, weight_gib=32.6, headroom=0.10, gpu_hr=2.0):
    usable = card_gib * (1 - headroom)
    per_user = kv_per_user_gib()
    users_per_replica = int((usable - weight_gib) // per_user)
    if users_per_replica < 1:
        return "weights + one user do not fit -- need a bigger card"
    replicas = math.ceil(users / users_per_replica)
    monthly = replicas * gpu_hr * 24 * 30
    return (f"KV/user={per_user:.2f} GiB, users/replica={users_per_replica}, "
            f"replicas={replicas}, ~${monthly:,.0f}/mo")

print(plan(500))
# real per-card headroom must be MEASURED (torch.cuda.max_memory_allocated) -- needs GPU