NLP & Transformers · Chapter K3

Sentiment Analysis & Text Classification

Classification — assigning a label to text — is the most common NLP task, and sentiment analysis is its poster child. This chapter builds a classifier the classic way (features + a model), evaluates it honestly, then compares it head-to-head with the LLM approach so you know exactly when each wins.

⏱️ ~55 min🏷️ Foundations + code🎯 Intermediate

Learning objectives

Frame problems as text classification (binary, multi-class, multi-label).
Build a classic classifier: features (K2) → model → prediction.
Evaluate honestly with a train/test split and the right metrics — not just accuracy.
Compare classic ML vs LLM classification on cost, data, and accuracy.
Choose the right approach for a real classification problem.

You already built the LLM version — now see the classic oneIn Chapter 2 you built a sentiment classifier with a prompt + structured output, and Chapter 5 taught evaluating it. This chapter builds the classic ML version on K2's features, so you can compare the two approaches with real understanding — and know when a lightweight trained model beats calling a frontier LLM.

Framing a classification problem intermediate

Classification assigns one or more labels from a fixed set to a piece of text. A huge share of real NLP work is "just" classification once you frame it right.

Type	Labels	Example
Binary	2 classes	Spam / not spam
Multi-class	N mutually-exclusive classes	Sentiment: positive / neutral / negative
Multi-label	Any subset of N classes	A ticket tagged both "billing" AND "urgent"

Many "AI" tasks are classification in disguiseRoute a support ticket, detect toxicity, flag a document type, decide if an email needs a reply, tag content — all classification. Recognizing that gives you a well-understood toolbox (this chapter) and clear evals (Ch 5). It's also why the capstone's risk classification (read_only/reversible/irreversible) is a classification problem at its core (Ch 2, Ch 8).

Lab K3.1 · The classic classifier intermediate

The classic recipe: turn text into features (K2), feed them to a trained model, get a label. Simple, fast, and it's how sentiment analysis worked for years.

Features → model → label. Vectorize the text (TF-IDF or embeddings, K2), train a standard classifier (logistic regression, naive Bayes, or a linear SVM) on labeled examples, and predict. The representation choice (K2) matters as much as the model — good features make a simple model shine.

🗺️ How to read this diagram

This picture is the whole recipe for a classic (non-LLM) text classifier, read left to right. It's an assembly line: raw text goes in one end, a label comes out the other.

text (first box) — the raw sentence you want to label, e.g. "this product is amazing". A computer can't do math on words directly, so it can't stay as text.
features (K2) — the text is turned into numbers a model can use. TF-IDF (from the previous chapter) counts which words appear and how distinctive they are, producing a row of numbers per sentence.
trained model — a standard classifier (logistic regression, naive Bayes, or SVM — the caption lists them) that has already learned from labeled examples which number-patterns go with which label.
label (last box) — the model's answer, one class from a fixed set (e.g. positive). The arrows show data only flows one way, left to right.

In short: The big idea in the caption: good features make a simple model shine. Getting the text-to-numbers step (K2) right often matters more than which model you pick.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Lab K3.1

Requires: pip install scikit-learn

classify.pyfrom sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.2)

clf = make_pipeline(TfidfVectorizer(), LogisticRegression())   # features + model
clf.fit(X_train, y_train)                    # learn from labeled examples

preds = clf.predict(X_test)                  # label unseen text
print(clf.predict(["this product is amazing"]))   # ['positive']

▶ How this works

This is the classic classifier from start to finish in ~8 lines using scikit-learn, the standard Python machine-learning library. It turns text into numbers, trains a model on labeled examples, then predicts labels for text it has never seen.

The from sklearn... import lines pull in the four tools: TfidfVectorizer (text → numbers), LogisticRegression (the model), make_pipeline (glues steps together), and train_test_split (splits your data honestly).
train_test_split(texts, labels, test_size=0.2) randomly holds back 20% of your data as a test set the model will never train on. The other 80% (X_train, y_train) is for learning; the 20% (X_test, y_test) is saved to grade it fairly later. X is the text, y is the correct labels.
make_pipeline(TfidfVectorizer(), LogisticRegression()) chains the two steps into one object — vectorize, then classify — so you never have to run them separately.
clf.fit(X_train, y_train) is the actual training: the model looks at the 80% of examples and their known labels and learns the word-patterns for each class.
clf.predict(X_test) asks the trained model to label the held-out text. The last line predicts a brand-new sentence — the comment shows it returns ['positive'].

What the output means: preds holds the model's guessed labels for the test set, and the final line prints ['positive'] — the model's label for "this product is amazing". The prediction comes back in a list because you can classify many texts at once.

Try this: Change test_size to 0.5 (a 50/50 split) — you train on less data, so accuracy usually drops. Then change the sample sentence to something negative and see if the predicted label flips.

A few classic models, and when each fitsNaive Bayes — dead simple, strong baseline for text, near-instant. Logistic regression — the reliable default; interpretable, fast, great with TF-IDF. Linear SVM — historically top for text classification. All train in seconds on a laptop and run in microseconds. That speed/cost profile is exactly why they still beat an LLM for some jobs (see below).

Lab K3.2 · Evaluating honestly intermediate

The critical discipline: measure on data the model hasn't seen, and use the right metric. Accuracy alone lies — especially on imbalanced data.

The accuracy trapIf 95% of emails aren't spam, a model that predicts "not spam" for everything scores 95% accuracy — and catches zero spam. Accuracy is meaningless on imbalanced classes. You need precision and recall to see what's really happening. This is the single most important idea in classification evaluation — and it's the same lesson as Chapter 5's evals.

Metric	Answers	Care when…
Accuracy	% correct overall	Classes are balanced (and only then)
Precision	Of predicted-positive, how many were right?	False positives are costly (flagging good email as spam)
Recall	Of actual-positives, how many did we catch?	False negatives are costly (missing actual fraud)
F1	Balance of precision & recall	You need one number and both matter
Confusion matrix	Which classes get confused for which	Always — it shows where it fails

Lab K3.2

Requires: pip install scikit-learn

evaluate.pyfrom sklearn.metrics import classification_report, confusion_matrix

preds = clf.predict(X_test)
print(classification_report(y_test, preds))   # precision, recall, F1 per class
print(confusion_matrix(y_test, preds))        # what gets confused for what

▶ How this works

Training a model is easy; knowing whether it's actually good is the real skill. These two scikit-learn functions grade the model on the held-out test set — the data it never trained on — so the score reflects how it'll do on real, unseen text.

preds = clf.predict(X_test) gets the model's guesses for the test set (the same 20% you set aside in Lab K3.1). We now compare these guesses to the true answers y_test.
classification_report(y_test, preds) prints, for each class, three numbers: precision (of the items it labeled X, how many really were X), recall (of the items that truly were X, how many it caught), and F1 (a single score balancing the two). This is why plain accuracy isn't enough — see the 'accuracy trap' note above.
confusion_matrix(y_test, preds) prints a grid showing which classes get mistaken for which — e.g. how often 'neutral' was wrongly called 'positive'. It shows you where the model fails, not just how often.

What the output means: Two printouts: a table of precision/recall/F1 per class (numbers from 0 to 1, higher is better), and a square grid where the diagonal is correct predictions and off-diagonal cells are mistakes. Reading the grid tells you exactly which two classes the model confuses.

Try this: After running, look for the row with the lowest recall — that's the class the model misses most. In spam detection, low recall on 'spam' means real spam is slipping through, which is usually worse than the occasional false alarm.

Never evaluate on training dataA model scored on data it trained on will look great and fail in production — it memorized, it didn't generalize. Always hold out a test set it never saw (and a separate validation set if you're tuning). This train/test discipline is the foundation of trustworthy evaluation — and it maps directly onto Chapter 5's golden set and the eval gate (O3/I4).

Classic ML vs LLM classification advanced

Here's the comparison this whole chapter builds toward. You can classify text two ways now — a trained classic model, or a prompt to an LLM. Neither is universally better.

Two ways to classify, different profiles. Classic ML needs labeled training data but is tiny, instant, cheap, and deterministic. An LLM needs no training (zero/few-shot from a prompt), handles nuance and new classes on the fly, but costs more per call and is slower. The right choice depends on volume, budget, data, and how much nuance the task needs.

🗺️ How to read this diagram

This is a side-by-side scorecard of the two ways you can now classify text: a classic trained model (left) versus asking an LLM with a prompt (right). Neither wins everywhere — the point is to see the trade-offs at a glance.

Left — Classic ML. It needs labeled data to train, but once trained it's tiny, fast, and cheap to run, and gives the same answer every time (deterministic). The catch: to change what it does, you must retrain it.
Right — LLM (prompt). It needs no training data (zero/few-shot: you just describe the task or show a few examples), and it handles subtle, nuanced language well. The catch: it's slower and costs more per item.
Read each row as a trade: 'needs labeled data' vs 'no training', 'fast·cheap·tiny' vs 'slower·pricier', 'deterministic' vs 'handles nuance', 'retrain to change' vs 'edit the prompt'. Every strength on one side is a weakness on the other.

In short: Rule of thumb from the caption: pick the classic model when you have labels + huge volume + tight budget; reach for the LLM when you have little data or need nuance. The table just below the diagram spells out each case.

Factor	Classic ML wins	LLM wins
Training data	You have thousands of labeled examples	You have few/none — zero/few-shot
Volume & cost	Millions of items, tight budget/latency	Moderate volume; per-call cost acceptable
Nuance	Clear, well-separated classes	Sarcasm, context, subtle categories
Flexibility	Fixed label set, rarely changes	Labels/criteria change often (edit the prompt)
Explainability/determinism	Regulated, must be reproducible	Flexibility matters more than determinism

The pragmatic hybridA common production pattern: use the LLM to bootstrap labels (few-shot classify a few thousand examples), then train a cheap classic model on those labels to run at scale for pennies. You get the LLM's nuance encoded into a model that's fast and cheap enough for millions of items. It's model-tiering (C1) taken to its logical end — the frontier model teaches; the small model serves.

The LLM classifier, recalled advanced

You built this in Chapter 2 — worth seeing side by side with the classic version. No training set, no feature engineering; the "model" is a prompt plus a schema.

Uses the client / objects set up earlier in this lesson, and needs ANTHROPIC_API_KEY set. Run the earlier blocks first.

llm_classify.pyfrom pydantic import BaseModel
from typing import Literal

class Sentiment(BaseModel):
    label: Literal["positive", "neutral", "negative"]
    confidence: float

r = client.messages.parse(model="claude-haiku-4-5", max_tokens=128,   # small model — easy task (C1)
    messages=[{"role":"user","content": "honestly the best purchase this year"}],
    output_format=Sentiment)
print(r.parsed_output.label)     # positive — no training data required

Two things transfer straight from Chapter 2Structured output (a Literal enum) makes the label safe to consume — the model literally can't return an off-set class. And model tiering: sentiment is easy, so use Haiku, not the frontier model (C1). Even when you choose the LLM route, the classic evaluation discipline (K3.2 — test set, precision/recall/F1) is exactly how you check it (Ch 5).

▶ How this works

This is the LLM version of the exact same task — the sentiment classifier you built back in Chapter 2 — shown here so you can compare it with the classic one above. Notice there's no training set and no feature engineering: the entire 'model' is a prompt plus a required output shape.

class Sentiment(BaseModel) uses Pydantic to declare the exact answer shape you want back. label: Literal["positive", "neutral", "negative"] forces the answer to be one of those three words — the model literally cannot return anything else — and confidence: float asks for a certainty score.
client.messages.parse(...) sends the text to the model and, thanks to output_format=Sentiment, makes it reply in your Sentiment shape instead of a free-form paragraph.
model="claude-haiku-4-5" deliberately picks a small, cheap model — the comment notes sentiment is an easy task, so you don't pay for a big one (model tiering, C1).
r.parsed_output.label pulls the clean, validated label out of the reply as a normal Python value you can store or branch on — no parsing text by hand.

What the output means: positive prints. There was no training step at all — the model classified "honestly the best purchase this year" straight from the prompt, which is the whole point of the comparison with the classic pipeline above.

Try this: Swap the message to something mixed like "it works but shipping was slow" and watch the confidence value — ambiguous text should lower it. Even for the LLM route, you'd still grade it with the K3.2 metrics (test set, precision/recall/F1).

Choosing an approach advanced

Classification decision sketch if a regex/keyword rule fully captures it → rules (K1) — cheapest elif lots of labeled data + huge volume + tight budget → classic ML classifier elif little/no labeled data OR nuanced classes → LLM (zero/few-shot) elif nuance AND scale → LLM to label → train classic model always → evaluate on a held-out test set with the right metric

Same "match the tool to the task" refrainThis is model tiering (C1), build-vs-adopt (I1), and classic-vs-LLM (K1) all over again. The reflex "throw an LLM at it" is often wrong for classification: if you have the labels and the volume, a logistic-regression model is 1000× cheaper and just as accurate on clear-cut classes. Reach for the LLM when nuance or missing data justifies it.

Common pitfalls expert

Pitfall	Fix
Reporting accuracy on imbalanced data	Use precision/recall/F1 and a confusion matrix
Evaluating on training data	Hold out a test set the model never saw
Defaulting to an LLM for high-volume classification	A classic model is far cheaper if you have labels
Free-text LLM labels breaking downstream	Constrain with a `Literal` enum (Ch 2)
Frontier model for an easy classify task	Use a small model (C1) — or a classic one
Ignoring class imbalance in training	Rebalance, reweight, or resample; watch recall

Exercises expert

Exercise K3.1 — Classic sentiment classifier

Context: A TF-IDF + logistic-regression pipeline is the workhorse classic sentiment classifier; its confusion matrix tells you exactly which classes it muddles.

Your task: Train the TF-IDF + logistic-regression pipeline on a labeled sentiment dataset, then report per-class precision/recall/F1 and the confusion matrix.

Requirements:

Fit TF-IDF features into a logistic-regression classifier
Report precision, recall, and F1 per class
Produce the confusion matrix
Identify the most-confused class and reason about why

💡 Hint: The off-diagonal cells of the confusion matrix point straight at the classes the model conflates.

Exercise K3.2 — Classic vs LLM, head to head

Context: On the same held-out set, accuracy is often a wash — it is cost×volume, not accuracy, that flips the deploy decision between a classic model and an LLM.

Your task: Score your classic classifier and the LLM classifier on one held-out test set with identical metrics, then compare accuracy, cost per 1000 items, and latency.

Requirements:

Evaluate both models on the identical test set and metrics
Measure cost per 1000 items and per-item latency for each
Decide which to deploy at 10 items/day
Decide which to deploy at 10M items/day and explain the flip

💡 Hint: At tiny volume the LLM's zero-training convenience wins; at scale the classic model's near-zero marginal cost decides it.

Show what to look for

On clear-cut sentiment both may score similarly, but the classic model costs ~nothing and runs in microseconds — decisive at 10M/day. At 10 items/day the LLM's zero-training convenience wins. The metric that flips the decision is cost×volume, not accuracy.

Exercise K3.3 — Bootstrap-and-distill

Context: The hybrid pattern uses an LLM to manufacture labels, then trains a cheap classic model on them — buying most of the LLM's quality at a fraction of the cost.

Your task: Use the LLM to label 500 unlabeled examples, train a classic classifier on those labels, and evaluate it against a human-labeled test set.

Requirements:

Have the LLM label a batch of previously-unlabeled examples
Train a classic classifier on the LLM-generated labels
Evaluate on a separate human-labeled test set
Report how close the distilled model gets to the LLM

💡 Hint: This is bootstrap-and-distill: the LLM is the teacher, the classic model is the cheap student you actually ship.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Frame the problem: labels & featuresBeginner

Context: Framing turns a task into (features → label). Before any model, you define the label set and check class balance, because a rare class shapes every downstream choice.

Your task: Given raw examples, produce a clean (text, label) dataset and report the class balance.

Requirements:

Separate the examples into parallel texts and labels
Report the count per class (e.g. with Counter)
Flag rare classes at framing time
Note the total number of examples

💡 Hint: Class balance drives stratified splits and possible resampling later — decide it now, not after a model surprises you.

Show solution

Framing first: define the label set and check balance. Runnable:

from collections import Counter

raw = [("Refund my order now!", "billing"),
       ("App crashes on login", "bug"),
       ("How do I reset password", "howto"),
       ("Charged twice this month", "billing"),
       ("Login button does nothing", "bug")]

texts  = [t for t, _ in raw]
labels = [y for _, y in raw]
print("classes:", dict(Counter(labels)))
print("n examples:", len(texts))
# imbalance check: billing=2 bug=2 howto=1 -- flag rare classes early

Class balance drives everything downstream: a rare class needs stratified splits and possibly resampling, decided at framing time.

Exercise 2 · A multinomial Naive Bayes by handIntermediate

Context: Naive Bayes is the classic text classifier: class priors times per-word likelihoods, with add-1 smoothing so an unseen word never zeroes out a class.

Your task: Implement multinomial Naive Bayes — training (priors + smoothed word likelihoods) and prediction — in the standard library.

Requirements:

Estimate class priors from the label counts
Estimate per-class word likelihoods with add-1 (Laplace) smoothing
Predict by summing log-probabilities to avoid underflow
Return the highest-scoring class for a new document

💡 Hint: Work in log-space and smooth every likelihood; the smoothing is what keeps an unseen word from sending a class probability to zero.

Show solution

Train counts, apply Laplace smoothing, predict via log-probabilities. Runnable:

import math
from collections import Counter, defaultdict

def train_nb(data):
    classes = Counter(y for _, y in data)
    word_counts = defaultdict(Counter)
    vocab = set()
    for text, y in data:
        for w in text.lower().split():
            word_counts[y][w] += 1; vocab.add(w)
    return classes, word_counts, vocab

def predict_nb(text, model):
    classes, wc, vocab = model
    total = sum(classes.values()); V = len(vocab)
    best, best_lp = None, -math.inf
    for c in classes:
        lp = math.log(classes[c] / total)
        n_c = sum(wc[c].values())
        for w in text.lower().split():
            lp += math.log((wc[c][w] + 1) / (n_c + V))   # add-1 smoothing
        if lp > best_lp:
            best_lp, best = lp, c
    return best

data = [("cheap meds buy now", "spam"), ("win money now", "spam"),
        ("meeting at noon", "ham"), ("lunch tomorrow", "ham")]
m = train_nb(data)
print(predict_nb("cheap money now", m))   # spam
print(predict_nb("meeting lunch", m))      # ham

Naive Bayes is fast, tiny, and interpretable — a strong baseline that classic NLP still reaches for on small labeled datasets.

Exercise 3 · Evaluate honestly: precision/recall/F1Advanced

Context: Accuracy lies on imbalanced data — a majority-only predictor can score 90% while catching zero minority cases. Per-class precision, recall, and F1 expose what accuracy hides.

Your task: Implement a confusion matrix and per-class precision, recall, and F1 from true/predicted label lists.

Requirements:

Count TP, FP, and FN per class
Precision = TP / (TP + FP); recall = TP / (TP + FN)
F1 is the harmonic mean of precision and recall
Guard against zero denominators and report metrics per class

💡 Hint: Report each class separately: a strong overall accuracy can hide a minority class whose recall is near zero.

Show solution

Compute TP/FP/FN per class, then derive the metrics. Runnable:

from collections import Counter

def metrics(y_true, y_pred):
    labels = sorted(set(y_true) | set(y_pred))
    out = {}
    for c in labels:
        tp = sum(t == c and p == c for t, p in zip(y_true, y_pred))
        fp = sum(t != c and p == c for t, p in zip(y_true, y_pred))
        fn = sum(t == c and p != c for t, p in zip(y_true, y_pred))
        prec = tp / (tp + fp) if tp + fp else 0.0
        rec  = tp / (tp + fn) if tp + fn else 0.0
        f1 = 2*prec*rec/(prec+rec) if prec+rec else 0.0
        out[c] = dict(precision=round(prec,2), recall=round(rec,2), f1=round(f1,2))
    return out

y_true = ["spam","spam","ham","ham","ham","ham"]
y_pred = ["spam","ham", "ham","ham","ham","spam"]
for c, m in metrics(y_true, y_pred).items():
    print(c, m)
# accuracy hides that spam recall is only 0.5 -- per-class metrics expose it

On a 90/10 split, a "predict majority" model scores 90% accuracy while catching zero minority cases — which is why you report per-class precision/recall/F1.

Exercise 4 · Stratified train/test splitExpert

Context: Honest evaluation needs a split that preserves class ratios in every fold; a naive random split can starve the test set of a rare class entirely.

Your task: Implement a stratified train/test split in the standard library and verify the class balance holds in both folds.

Requirements:

Group examples by label first
Split each label group by the same fraction
Combine the per-class slices into train and test sets
Verify the class ratios are preserved in both folds

💡 Hint: Stratify by splitting within each class, so the rare class shows up in the test fold at the same rate as in the data.

Show solution

Group by label, split each group by the same fraction, then combine. Runnable:

import random
from collections import Counter, defaultdict

def stratified_split(data, test_frac=0.4, seed=0):
    rng = random.Random(seed)
    by_class = defaultdict(list)
    for item in data:
        by_class[item[1]].append(item)
    train, test = [], []
    for c, items in by_class.items():
        items = items[:]; rng.shuffle(items)
        cut = round(len(items) * test_frac)
        test += items[:cut]; train += items[cut:]
    return train, test

data = [("t%d" % i, "A" if i % 3 else "B") for i in range(15)]
tr, te = stratified_split(data)
print("train balance:", dict(Counter(y for _, y in tr)))
print("test balance :", dict(Counter(y for _, y in te)))
# ratios preserved in both folds, unlike a naive random split

A random split can starve the test set of a rare class; stratification keeps evaluation honest by preserving the class distribution in every fold.

Exercise 5 · Classic vs LLM classifier: the trade routerProfessional

Context: The lesson contrasts a trained classic model (fast, cheap, needs labels) with an LLM zero-shot classifier (no labels, slower, pricier). A router makes the labels-vs-latency-vs-volume trade explicit.

Your task: Encode classifier_choice(...) that routes between a trained classic model and an LLM based on available labels, latency, volume, and label churn.

Requirements:

Ample labels plus a tight latency budget favours a trained classic model
Scarce labels or a frequently-changing label set favours an LLM zero/few-shot
Very high daily volume tips toward the cheap trained model at scale
Otherwise, use the LLM to bootstrap labels then distill to a classic model

💡 Hint: The pragmatic pattern: let the LLM bootstrap labels or cover the long tail, then distill into a cheap classic model once the label set stabilizes.

Show solution

Route on labels-available, latency, volume, and label churn. Runnable:

def classifier_choice(labeled_examples, latency_ms, volume_per_day, labels_change_often):
    if labeled_examples >= 500 and latency_ms < 50:
        return "trained classic model (NB/linear/small transformer)"
    if labeled_examples < 50 or labels_change_often:
        return "LLM zero/few-shot (no training data needed)"
    if volume_per_day > 1_000_000:
        return "trained model (cost at scale)"
    return "LLM to bootstrap labels, then distill to a classic model"

print(classifier_choice(2000, 20, 5_000_000, False))
print(classifier_choice(10, 400, 500, True))

The pragmatic pattern: use an LLM to bootstrap labels or handle the long tail, then distill into a cheap classic model once you have data and the label set stabilizes.

Exercise 6 · Ship a text classifier with a confidence gateIndustry scenario

Context: A shippable classifier abstains when unsure and routes to a human — turning "always guess" into "answer when confident, escalate when not" and bounding the error the business sees.

Your task: Wrap Naive Bayes to return a label plus a normalized confidence, and defer low-confidence cases to human review.

Requirements:

Convert the per-class log-probabilities into probabilities via a stable softmax
Take the top class and its normalized confidence
Abstain and route to a human when confidence falls below a threshold
Return both the decision and the confidence score

💡 Hint: Subtract the max log-prob before exponentiating for a numerically stable softmax, then gate the answer on the resulting top probability.

Show solution

Turn log-probs into a softmax confidence; abstain below a threshold. Runnable:

import math
from collections import Counter, defaultdict

def train_nb(data):
    classes = Counter(y for _, y in data); wc = defaultdict(Counter); vocab = set()
    for text, y in data:
        for w in text.lower().split():
            wc[y][w] += 1; vocab.add(w)
    return classes, wc, vocab

def predict_conf(text, model, threshold=0.65):
    classes, wc, vocab = model
    total = sum(classes.values()); V = len(vocab)
    lps = {}
    for c in classes:
        lp = math.log(classes[c] / total); n_c = sum(wc[c].values())
        for w in text.lower().split():
            lp += math.log((wc[c][w] + 1) / (n_c + V))
        lps[c] = lp
    m = max(lps.values())
    exps = {c: math.exp(v - m) for c, v in lps.items()}   # softmax, stable
    Z = sum(exps.values())
    probs = {c: e / Z for c, e in exps.items()}
    label = max(probs, key=probs.get)
    if probs[label] < threshold:
        return ("ABSTAIN -> human review", round(probs[label], 2))
    return (label, round(probs[label], 2))

data = [("buy cheap now", "spam"), ("win prize now", "spam"),
        ("lunch at noon", "ham"), ("meeting tomorrow", "ham")]
m = train_nb(data)
print(predict_conf("cheap prize now", m))
print(predict_conf("noon meeting lunch", m))
print(predict_conf("hello", m))            # low confidence -> abstain

A confidence gate is what makes a classifier shippable: it turns "always guess" into "answer when sure, escalate when not", bounding the error the business sees.

✓ Checkpoint — you can move on when you can…

Frame a problem as binary/multi-class/multi-label classification.
Build a classic features→model classifier.
Evaluate with a test set and precision/recall/F1 (not just accuracy).
Compare classic ML vs LLM classification on the key factors.
Choose (and justify) an approach — including the hybrid.

🏗️ Toward the capstoneThe capstone's risk classifier — is this action read_only, reversible, or irreversible? — is a text/structured classification problem, and it's evaluated exactly as this chapter teaches: a held-out set of real actions with precision/recall on the dangerous class (you cannot afford a false "safe"). Classic classification discipline is what keeps that safety gate honest (Ch 8d). Next, K4: how neural networks learned to model sequences. Next: neural sequence models →

Knowledge check check yourself

✓ Knowledge check

What is the 'accuracy trap' on imbalanced data, and which metrics does the lesson say you need instead?

Show answer

If 95% of emails aren't spam, a model that always predicts 'not spam' scores 95% accuracy while catching zero spam — accuracy is meaningless on imbalanced classes. You need precision and recall (and a confusion matrix) to see what's really happening.

✓ Knowledge check

The lesson describes a 'bootstrap-and-distill' hybrid pattern. What is it, and why is it attractive at very high volume?

Show answer

Use the LLM to few-shot label a few thousand examples, then train a cheap classic model on those labels to run at scale. You encode the LLM's nuance into a model that's fast and cheap enough to serve millions of items — model tiering taken to its end: the frontier model teaches, the small model serves.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

Framing a classification problem intermediate

Lab K3.1 · The classic classifier intermediate

Lab K3.2 · Evaluating honestly intermediate

Classic ML vs LLM classification advanced

The LLM classifier, recalled advanced

Choosing an approach advanced

Common pitfalls expert

Exercises expert

Exercise K3.1 — Classic sentiment classifier

Exercise K3.2 — Classic vs LLM, head to head

Exercise K3.3 — Bootstrap-and-distill

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in NLP & transformers