Full fine-tune vs PEFT

When you'd go full intermediate

Here's a decision that has quietly burned real budgets. A team decides their model needs to "learn their domain," so they reach for the biggest hammer they know — full fine-tuning — spin up a multi-GPU cluster, and three days and several thousand dollars later they have a model that's marginally better than what a $40 LoRA run would have produced in an afternoon on a single card. The mistake wasn't the tuning; it was skipping the question this whole section exists to answer: do I actually need to move every weight, or just nudge a few?

The plain way to hold the two options in your head is a renovation analogy. PEFT (LoRA/QLoRA) is like adding a smart-home overlay to a house: you leave the walls, wiring, and plumbing exactly as they are and bolt on a small module that changes how the place behaves. Full fine-tuning is knocking the house down to the studs and rebuilding — enormously more powerful if you genuinely need a different structure, but wildly overkill if all you wanted was to dim the lights differently. Because the base model already knows grammar, world facts, reasoning, and instruction-following from its original training, most tasks are a "behavior overlay," not a "rebuild." That is why PEFT is the default and full FT is the exception, not a matter of taste.

The common mistake is treating "full fine-tuning" as the serious, professional choice and PEFT as the budget shortcut — the intuition is exactly backwards. On nearly every practical axis PEFT is the better engineering decision: it trains under 1% of the parameters, fits on hardware you can actually rent, produces a small swappable artifact instead of a fresh multi-gigabyte model, and — as you'll see two sections down — is far less prone to catastrophic forgetting, the failure mode where a model gets so absorbed in your narrow task that it forgets how to do everything else it used to be good at. Full fine-tuning touches every weight, so it can overwrite that general competence; PEFT freezes the original weights, so the model literally cannot forget what it knew. The honest cases for going full are narrow and specific: you need a large behavior shift (a new language, a new modality, a genuinely different style of reasoning — not just a tone tweak), you have enough clean data to justify moving billions of parameters without overfitting, and you have the multi-GPU hardware to pay the memory bill the next section computes. Miss any one of those and PEFT is not the compromise — it's the correct answer. Read the comparison table below as a scorecard where every column but one favors PEFT; that lopsidedness is the point.

The memory math intermediate

The single most useful thing you can do before any fine-tuning project is a back-of-the-envelope memory calculation — because "will this even fit on the GPU I have?" is the question that decides everything else, and the answer surprises almost everyone the first time. People assume a 7-billion-parameter model needs "about 7B worth of memory" to train. It doesn't. It needs roughly six times that, and understanding why turns fine-tuning from guesswork into arithmetic you can do in your head.

Here's the intuition, and it's worth internalizing because it explains the entire existence of PEFT. Think of each parameter not as a single number but as a small entourage that travels together during training. The parameter's actual value (the weight) is just one member. To learn, the optimizer also has to remember, for that same parameter, which direction to nudge it (the gradient) and — because modern training uses the Adam optimizer — two running averages of past gradients (Adam's "moments," usually called m and v, which smooth the updates so training doesn't thrash). So one parameter you can see drags along three invisible companions you have to store. In mixed precision the rough tally is: weight ≈ 2 bytes, gradient ≈ 2 bytes, and Adam's two moments ≈ 4 + 4 bytes — about 12 bytes per parameter, of which the weight itself is a mere sixth. The gradient and optimizer state, not the weights, are what blow up your budget.

Multiply that out and the picture is stark: a 7B model at ~12 bytes/param is roughly 78 GB just for the training state, before you count activations or the batch — comfortably past any single card, which is precisely why full fine-tuning means renting a multi-GPU box. The common misconception to kill here is that quantization or a smaller batch will rescue a full fine-tune on one GPU; it won't, because the optimizer entourage scales with the number of trainable parameters, and in full FT that number is all of them. This is the exact hinge on which PEFT turns: if you freeze the base weights, they need no gradient and no optimizer state — they become read-only ballast you can even store in 4-bit (that's the "Q" in QLoRA). You then pay the full 12-byte entourage cost only for the tiny adapter — under 1% of the parameters — so the training state collapses from ~78 GB to a few GB that fits on one consumer card. The formula below is the whole argument made numeric; the takeaway is that PEFT wins on memory not by being clever about the same work, but by refusing to do most of the work at all.

Merging and serving adapters advanced

You've trained an adapter — now comes the question that trips people up at deploy time: do you glue it onto the base model permanently, or keep it as a detachable part you snap on at request time? This isn't a trivial packaging detail; it quietly decides how many models you have to store, how fast each request is, and how painful it is to ship an update. Getting it wrong is the difference between a serving stack that scales to a dozen tuned behaviors on one GPU and one that needs a dozen separate model copies.

The analogy that makes it click is camera lenses. A LoRA adapter is a lens; the base model is the camera body. Merging (merge_and_unload()) is like welding one lens permanently onto the body — you get a single, purpose-built instrument that's dead simple to hand someone and has zero attach-time overhead, but it's now one camera that does one thing. Keeping the adapter separate is the normal interchangeable-lens setup: one body, a bag of lenses, and you click on whichever one this shot needs. Mechanically, merging works because a LoRA adapter is just a small low-rank update to the weights — mathematically it's a matrix you can add back into the original weight matrices, after which the adapter has no independent existence; the numbers have been folded in. Once merged, the model loads and runs exactly like any ordinary model, with no adapter code in the serving path at all.

So when do you weld versus keep-detachable? Merge when you're serving a single tuned behavior, or when latency is critical — a merged model has no per-request adapter-loading step, so it's both the simplest artifact to ship (one folder, drop it onto vLLM or TGI) and the fastest to run. The common mistake is to merge reflexively because it "feels cleaner," and then regret it the moment you have a second behavior to serve. Keep adapters separate when one base model must serve many behaviors — a support-tone adapter, a code-style adapter, a summarizer — because you can load the base once and hot-swap the right small adapter per request, which saves enormous memory (one 14 GB base instead of five) and, just as importantly, lets you retrain and redeploy one behavior without rebuilding or re-testing the others. That decoupling — update one lens without touching the camera or the other lenses — is often the deciding factor in a real platform, well beyond the raw memory savings. The lab below shows the merge step; the callout after it shows when to resist merging.

merge.pyfrom peft import AutoPeftModelForCausalLM

# Load base + adapter, merge into one set of weights, save a standalone model.
model = AutoPeftModelForCausalLM.from_pretrained("out/adapter", device_map="auto")
merged = model.merge_and_unload()
merged.save_pretrained("out/merged-model")   # now serve it like any HF model (IC6/LM3)

rows = {
    "params trained": ("all (billions)", "<1%"),
    "memory":         ("very high",      "low (QLoRA: one GPU)"),
    "artifact":       ("whole new model","small adapter"),
    "swappable":      ("no",             "yes — many adapters, one base"),
}
print(f"{'dimension':16}{'full':18}PEFT")
for k,(full,peft) in rows.items():
    print(f"{k:16}{full:18}{peft}")
print("\nDEFAULT: PEFT (LoRA/QLoRA) — cheaper, faster, swappable")

def full_ft_gb(params_b, w=2, grad=2, adam_m=4, adam_v=4):
    bytes_per = w + grad + adam_m + adam_v    # ~12 bytes/param, mixed precision
    return params_b * bytes_per / 1024**3

for p in (1e9, 7e9, 13e9):
    print(f"{p/1e9:.0f}B params -> {full_ft_gb(p):.0f} GB")
# 1B -> 11 GB   7B -> 78 GB   13B -> 145 GB

def compare(params_b, trained_frac=0.004):
    full = params_b * 12 / 1024**3
    base_4bit = params_b * 0.5 / 1024**3            # frozen, quantized
    adapter = params_b * trained_frac * 12 / 1024**3 # only the adapter pays optimizer
    qlora = base_4bit + adapter
    return round(full,1), round(qlora,1), round(full/qlora,1)

full, qlora, ratio = compare(7e9)
print(f"full FT : {full} GB")     # 78.2
print(f"QLoRA   : {qlora} GB")    # ~3.5
print(f"QLoRA is ~{ratio}x lighter")  # ~22x

def serve_mode(num_task_adapters, latency_critical, ops_simplicity):
    if num_task_adapters >= 2 and not latency_critical:
        return "HOT-SWAP adapters — one base serves many tasks, load per request"
    if ops_simplicity or latency_critical:
        return "MERGE — bake adapter into base -> one standalone model, simplest & fastest"
    return "MERGE — default for a single-task deployment"

print(serve_mode(1, True, True))    # MERGE
print(serve_mode(5, False, False))  # HOT-SWAP

from peft import AutoPeftModelForCausalLM

# Load base + trained adapter together, fold the adapter into the weights, save.
model = AutoPeftModelForCausalLM.from_pretrained("out/adapter", device_map="auto")
merged = model.merge_and_unload()          # adapter baked into base weights
merged.save_pretrained("out/merged-model") # now serve like any HF model (IC6)

def full_ft_gate(big_behavior_shift, lots_of_quality_data, have_multigpu):
    need = {"large behavior shift": big_behavior_shift,
            "lots of quality data": lots_of_quality_data,
            "multi-GPU hardware":   have_multigpu}
    missing = [k for k,v in need.items() if not v]
    return ("GO — full FT justified" if not missing
            else "NO-GO — PEFT instead; missing: " + ", ".join(missing))

print(full_ft_gate(True, True, True))    # GO
print(full_ft_gate(True, False, True))   # NO-GO ... missing: lots of quality data

Knowledge check check yourself