Sentiment Analysis & Text Classification

Framing a classification problem intermediate

Start with the moment this pays off. A product team comes to you drowning in support tickets and says "we need AI to help." That sounds like a research project — until you notice the real request is "sort these into billing, bug, and how-to so the right person sees them first." The instant you spot that, the fog clears: this isn't open-ended generation, it's classification, a task the industry has understood for decades and can measure to two decimal places. Framing is the cheap step that turns a vague ambition into a solved problem — and getting it wrong is how teams end up throwing a frontier LLM at something a five-line model would nail.

Here is the plain idea: classification assigns one or more labels from a fixed, known-in-advance set to a piece of text. The "fixed set" part is the whole game. If you can write down the possible answers before you see the input — spam or not, positive/neutral/negative, this-department or that-one — you are classifying. If the answer is free-form ("summarize this", "write a reply"), you are not; that's generation, a different and usually more expensive tool. Think of classification as sorting mail into a fixed wall of pigeonholes: every letter goes into exactly one slot (or, for multi-label, gets a few stamps), and the slots were built before any mail arrived.

The three shapes in the table below are worth internalizing because they change everything downstream — the model, the loss function, and especially how you score it. Binary is one yes/no decision. Multi-class picks exactly one of N options that are mutually exclusive (a review can't be both positive and negative). Multi-label is the sneaky one: the classes are not exclusive, so a ticket can be tagged both "billing" and "urgent" at once — which means it's really N independent yes/no questions wearing a trench coat, not one N-way choice.

The common beginner mistake is forcing a multi-label problem into a multi-class box. If your tags can co-occur and you build a single "pick one" model, you've quietly made it impossible to ever return two labels, and you'll spend a week confused about why "billing + urgent" tickets keep losing one tag. Decide the shape first, out loud, before you touch a model — it's a one-sentence choice that saves days. A huge share of real NLP work is "just" classification once you frame it right, and framing costs nothing but a moment's honesty about what the answer actually looks like.

Lab K3.1 · The classic classifier intermediate

Before LLMs, this is how every spam filter, sentiment dashboard, and news categorizer on earth actually worked — and an enormous number of them still do, quietly, because it's fast and nearly free. It matters that you learn it not as history but as a live option: the day someone asks you to label ten million reviews on a budget, the classic pipeline is the answer, and you'll want to have built one with your own hands. This is the workhorse the whole chapter compares the LLM against, so meet it properly.

The recipe is three steps, and it helps to see it as a translation problem. A model can't do arithmetic on the word "amazing" — it needs numbers. So step one turns text into features (the K2 material: TF-IDF, which is just "how often does each word appear, weighted by how distinctive it is"). Step two feeds those numbers to a model that has already studied thousands of labeled examples and learned which number-patterns go with which label. Step three hands you the label. Picture a language teacher who's graded a mountain of essays: show them a new one and they instantly sense "this reads positive" — not by magic, but from patterns absorbed over all those examples. The classic classifier is that teacher, distilled into a few kilobytes of learned weights.

Here's the insight that separates people who ship good classic classifiers from people who fight them: the features usually matter more than the model. Beginners obsess over "should I use logistic regression or an SVM?" when the real leverage is in the text-to-numbers step — clean tokenization, sensible TF-IDF, maybe swapping in embeddings. Good features make a simple model shine; bad features can't be rescued by a fancy one. That's why K2 came first and why the diagram below spends a whole box on "features." The model is often the least interesting decision you'll make.

One honest caveat so the "simple and fast" framing doesn't oversell it: classic classifiers see words, not meaning. TF-IDF treats "not good" as the words "not" and "good" sitting near each other — it has no built-in sense that "not" flips the sentiment. That literal-mindedness is exactly the weakness the LLM comparison later in this chapter exploits. But for clear-cut, well-separated classes — is this email spam, is this review positive — that literal-mindedness is plenty, and it comes at a thousandth of the cost. Simple, fast, and it's how sentiment analysis worked for years.

classify.pyfrom sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.2)

clf = make_pipeline(TfidfVectorizer(), LogisticRegression())   # features + model
clf.fit(X_train, y_train)                    # learn from labeled examples

preds = clf.predict(X_test)                  # label unseen text
print(clf.predict(["this product is amazing"]))   # ['positive']

Lab K3.2 · Evaluating honestly intermediate

This is the section that separates a demo from a product, and it's the one people skip. Here's the failure that happens over and over: a team trains a classifier, it reports 97% accuracy, everyone celebrates, it ships — and in production it's a disaster, because that 97% was measured wrong or measured on the wrong thing. Evaluating honestly is not a formality you do at the end; it's the discipline that tells you whether you have anything at all. If you take one thing from this whole chapter, take this: a number you can't trust is worse than no number, because it makes you confident while you're wrong.

Two ideas do all the work. The first is where you measure: always on data the model has never seen. A model graded on its own training data is like a student who was given the exam answers to study — of course they ace it, and of course that tells you nothing about a real test. The second is which number you look at, and this is where the famous trap lives. Accuracy — plain "percent correct" — sounds like the obvious metric, and on balanced data it's fine. But the moment your classes are lopsided, accuracy becomes a liar, and most real problems (spam, fraud, defects, the dangerous-action class in the capstone) are wildly lopsided.

The fix is to split "correct" into two questions that pull in opposite directions. Precision asks: of the things I flagged, how many were real? (High precision = few false alarms.) Recall asks: of the things that were actually there, how many did I catch? (High recall = few misses.) A useful analogy is a smoke detector. Crank its sensitivity to max and it catches every fire (perfect recall) but shrieks at burnt toast (terrible precision); make it sleepy and it never false-alarms (perfect precision) but might miss a real fire (terrible recall). You can't max both — pushing one usually drops the other — so you pick based on which mistake actually hurts. F1 is the single number that balances the two (their harmonic mean, which punishes lopsidedness), and the confusion matrix is the honest picture underneath it all: a grid showing exactly which classes get mistaken for which, so you see where you fail, not just how often. Use the right metric, on data the model hasn't seen — those two habits are the entire foundation of trustworthy evaluation, and the table below tells you which metric to reach for when.

evaluate.pyfrom sklearn.metrics import classification_report, confusion_matrix

preds = clf.predict(X_test)
print(classification_report(y_test, preds))   # precision, recall, F1 per class
print(confusion_matrix(y_test, preds))        # what gets confused for what

Classic ML vs LLM classification advanced

This is the decision that lands on your desk in real life, and getting it wrong costs real money. Someone needs text classified. You now know two completely different ways to do it: train a classic model on labeled examples, or write a prompt and ask an LLM. The reflex in 2026 is "just use the LLM" — it's easy, it's impressive, it needs no training data. And for a lot of tasks that reflex is right. But choose the LLM for a high-volume job that a logistic-regression model would handle, and you can burn a thousand times the budget for the same accuracy. Choose the classic model for a nuanced, low-data task and you'll spend weeks hand-labeling data to build something that underperforms a five-minute prompt. Neither is universally better — the skill is knowing which trade you're making.

The cleanest way to hold this in your head is that the two approaches have mirror-image strengths: nearly every advantage of one is a weakness of the other. A useful analogy is hiring a specialist temp versus a brilliant generalist consultant. The classic model is the specialist you trained yourself — cheap to run, lightning-fast, does the exact one job you drilled it on, and gives the identical answer every time. But it's rigid: it only knows what you taught it, and changing its job means retraining from scratch. The LLM is the consultant on call — needs no training, understands sarcasm and context, handles brand-new categories the moment you describe them in the prompt. But it's slower, costs a fee per question, and might phrase things a little differently each time.

The variables that decide it are volume, budget, data, and nuance — and they interact. Low volume tilts toward the LLM because per-call cost barely matters when you're doing ten a day; sky-high volume tilts hard toward the classic model because a fraction-of-a-cent per item, multiplied by millions, becomes a real line on the bill while the trained model runs for essentially nothing. Plenty of labeled data unlocks the classic route; little or none forces the LLM's zero/few-shot magic. Clean, well-separated classes are classic-model territory; sarcasm, subtle context, and fuzzy boundaries are where the LLM earns its keep. The mistake to avoid is treating this as a technology preference ("we're an LLM shop now") instead of a per-task calculation. Read the scorecard below and the table under it as a way to make that calculation explicit — and remember the two aren't mutually exclusive, which the hybrid pattern right after exploits.

The LLM classifier, recalled advanced

The stakes here are conceptual: seeing these two side by side is what makes the trade-off real instead of abstract. You could nod along to "the LLM needs no training data" and not feel it — until you look at the code and realize the entire classifier is a prompt and a schema. There's no .fit(), no training loop, no labeled dataset, no feature engineering. Where the classic pipeline needed thousands of examples before it could say a single word, this version classifies its first sentence cold. That's the superpower and the price tag in one glance, and it's why we recall it here rather than in a separate lesson.

The mental shift is that the "model" isn't something you build — it already exists, pre-trained on a large slice of the internet, and you're just describing the job in words. Think of the difference between training a new employee for weeks (classic ML) versus handing a competent contractor a one-line brief and getting work back immediately (the LLM). The contractor knew how to read sentiment before you showed up; your job is only to specify what you want and to constrain the answer so it's safe to use downstream. That constraint is the quietly critical part: left to free-form, the model might reply "This seems quite positive!" which is useless to code that expects one of three exact strings.

Two ideas from Chapter 2 do the heavy lifting and are worth naming before you read the code. First, structured output: by declaring a Literal of the allowed labels, you make it impossible for the model to return an off-menu answer — the label comes back as a clean, validated value, not a sentence you have to parse. Second, model tiering: sentiment is an easy task, so you deliberately pick a small, cheap model (Haiku) rather than a frontier one — paying frontier prices for "is this review happy?" is exactly the reflex this chapter argues against. Keep both in view as you read; they're what make the LLM route production-grade rather than a demo. You built this in Chapter 2 — worth seeing side by side with the classic version.

Uses the client / objects set up earlier in this lesson, and needs the matching API key set (ANTHROPIC_API_KEY for Claude, OPENAI_API_KEY for OpenAI). Run the earlier blocks first.

Zero-training LLM classification — in Claude or OpenAI

The Sentiment model and the Literal enum are identical for both vendors — only the call differs. In Claude you call client.messages.parse(output_format=Sentiment) and read r.parsed_output; in OpenAI the same thing is client.responses.parse(text_format=Sentiment) and you read r.output_parsed. Either way the schema forces an on-menu label. Toggle the tab to switch SDK.

llm_classify.pyfrom pydantic import BaseModel
from typing import Literal

class Sentiment(BaseModel):
    label: Literal["positive", "neutral", "negative"]
    confidence: float

r = client.messages.parse(model="claude-haiku-4-5", max_tokens=128,   # small model — easy task (C1)
    messages=[{"role":"user","content": "honestly the best purchase this year"}],
    output_format=Sentiment)
print(r.parsed_output.label)     # positive — no training data required

llm_classify.pyfrom pydantic import BaseModel
from typing import Literal

class Sentiment(BaseModel):
    label: Literal["positive", "neutral", "negative"]
    confidence: float

r = client.responses.parse(model="gpt-5.5", max_output_tokens=128,   # small/easy task (C1)
    input="honestly the best purchase this year",
    text_format=Sentiment)
print(r.output_parsed.label)     # positive — no training data required

Choosing an approach advanced

Everything above becomes useless if you freeze at the actual decision, so here it is boiled down to a checklist you can run in your head. The sketch below isn't a rigid algorithm to memorize; it's the order in which a seasoned engineer asks questions, cheapest option first. That ordering is the real lesson: you don't start by reaching for the most powerful tool, you start by reaching for the simplest one that could possibly work and only climb the ladder when the task forces you to.

Read it top to bottom as a series of "can I get away with something cheaper?" gates. First: could a plain keyword or regex rule settle it? If every urgent ticket literally contains the word "outage," you may not need machine learning at all — a rule is free, instant, and perfectly explainable, and it's embarrassing how often people skip straight past it. If rules can't capture it but you have lots of labels and high volume, the classic model wins on cost. If you have little data or the classes are nuanced, the LLM's zero/few-shot ability justifies its price. And if you have both nuance and scale, the hybrid — LLM to label, classic model to serve — gives you most of the quality at a fraction of the running cost.

The one gate with no elif, the one that fires no matter which branch you land in, is the last: evaluate on a held-out test set with the right metric. This is deliberate. Whichever tool you pick, you are not done until you've measured it honestly (K3.2) — the choice of classifier is a guess until the numbers confirm it. Treat the sketch as advisory and the evaluation as mandatory, and you'll rarely choose wrong.

Common pitfalls expert

Every row in this table is a mistake that has shipped to production and cost someone a bad week — they're not hypothetical foot-guns, they're the specific ways classification projects go quietly wrong. The reason to study them as a set is that they share a personality: each one looks fine right up until it doesn't. The model reports a great score, the demo works, everyone moves on — and the failure only surfaces later, in the numbers that matter, when it's expensive to unwind.

Notice the two villains that recur. The first is measuring the wrong thing: reporting accuracy on imbalanced data, or grading on training data, both give you a comforting number that describes a world you don't live in. The fix in both cases is the K3.2 discipline — right metric, unseen data — which is why that section was the heart of the chapter. The second villain is reaching for the wrong-sized tool: defaulting to an LLM for high-volume work you have labels for, or using a frontier model where a small one or a classic one would do. That's the cost-blindness this whole chapter is trying to cure. Skim the table now, then treat it as a pre-flight checklist you actually run before you call a classifier "done" — most of these are one honest question away from being caught.

Exercises expert

from collections import Counter

raw = [("Refund my order now!", "billing"),
       ("App crashes on login", "bug"),
       ("How do I reset password", "howto"),
       ("Charged twice this month", "billing"),
       ("Login button does nothing", "bug")]

texts  = [t for t, _ in raw]
labels = [y for _, y in raw]
print("classes:", dict(Counter(labels)))
print("n examples:", len(texts))
# imbalance check: billing=2 bug=2 howto=1 -- flag rare classes early

import math
from collections import Counter, defaultdict

def train_nb(data):
    classes = Counter(y for _, y in data)
    word_counts = defaultdict(Counter)
    vocab = set()
    for text, y in data:
        for w in text.lower().split():
            word_counts[y][w] += 1; vocab.add(w)
    return classes, word_counts, vocab

def predict_nb(text, model):
    classes, wc, vocab = model
    total = sum(classes.values()); V = len(vocab)
    best, best_lp = None, -math.inf
    for c in classes:
        lp = math.log(classes[c] / total)
        n_c = sum(wc[c].values())
        for w in text.lower().split():
            lp += math.log((wc[c][w] + 1) / (n_c + V))   # add-1 smoothing
        if lp > best_lp:
            best_lp, best = lp, c
    return best

data = [("cheap meds buy now", "spam"), ("win money now", "spam"),
        ("meeting at noon", "ham"), ("lunch tomorrow", "ham")]
m = train_nb(data)
print(predict_nb("cheap money now", m))   # spam
print(predict_nb("meeting lunch", m))      # ham

from collections import Counter

def metrics(y_true, y_pred):
    labels = sorted(set(y_true) | set(y_pred))
    out = {}
    for c in labels:
        tp = sum(t == c and p == c for t, p in zip(y_true, y_pred))
        fp = sum(t != c and p == c for t, p in zip(y_true, y_pred))
        fn = sum(t == c and p != c for t, p in zip(y_true, y_pred))
        prec = tp / (tp + fp) if tp + fp else 0.0
        rec  = tp / (tp + fn) if tp + fn else 0.0
        f1 = 2*prec*rec/(prec+rec) if prec+rec else 0.0
        out[c] = dict(precision=round(prec,2), recall=round(rec,2), f1=round(f1,2))
    return out

y_true = ["spam","spam","ham","ham","ham","ham"]
y_pred = ["spam","ham", "ham","ham","ham","spam"]
for c, m in metrics(y_true, y_pred).items():
    print(c, m)
# accuracy hides that spam recall is only 0.5 -- per-class metrics expose it

import random
from collections import Counter, defaultdict

def stratified_split(data, test_frac=0.4, seed=0):
    rng = random.Random(seed)
    by_class = defaultdict(list)
    for item in data:
        by_class[item[1]].append(item)
    train, test = [], []
    for c, items in by_class.items():
        items = items[:]; rng.shuffle(items)
        cut = round(len(items) * test_frac)
        test += items[:cut]; train += items[cut:]
    return train, test

data = [("t%d" % i, "A" if i % 3 else "B") for i in range(15)]
tr, te = stratified_split(data)
print("train balance:", dict(Counter(y for _, y in tr)))
print("test balance :", dict(Counter(y for _, y in te)))
# ratios preserved in both folds, unlike a naive random split

def classifier_choice(labeled_examples, latency_ms, volume_per_day, labels_change_often):
    if labeled_examples >= 500 and latency_ms < 50:
        return "trained classic model (NB/linear/small transformer)"
    if labeled_examples < 50 or labels_change_often:
        return "LLM zero/few-shot (no training data needed)"
    if volume_per_day > 1_000_000:
        return "trained model (cost at scale)"
    return "LLM to bootstrap labels, then distill to a classic model"

print(classifier_choice(2000, 20, 5_000_000, False))
print(classifier_choice(10, 400, 500, True))

import math
from collections import Counter, defaultdict

def train_nb(data):
    classes = Counter(y for _, y in data); wc = defaultdict(Counter); vocab = set()
    for text, y in data:
        for w in text.lower().split():
            wc[y][w] += 1; vocab.add(w)
    return classes, wc, vocab

def predict_conf(text, model, threshold=0.65):
    classes, wc, vocab = model
    total = sum(classes.values()); V = len(vocab)
    lps = {}
    for c in classes:
        lp = math.log(classes[c] / total); n_c = sum(wc[c].values())
        for w in text.lower().split():
            lp += math.log((wc[c][w] + 1) / (n_c + V))
        lps[c] = lp
    m = max(lps.values())
    exps = {c: math.exp(v - m) for c, v in lps.items()}   # softmax, stable
    Z = sum(exps.values())
    probs = {c: e / Z for c, e in exps.items()}
    label = max(probs, key=probs.get)
    if probs[label] < threshold:
        return ("ABSTAIN -> human review", round(probs[label], 2))
    return (label, round(probs[label], 2))

data = [("buy cheap now", "spam"), ("win prize now", "spam"),
        ("lunch at noon", "ham"), ("meeting tomorrow", "ham")]
m = train_nb(data)
print(predict_conf("cheap prize now", m))
print(predict_conf("noon meeting lunch", m))
print(predict_conf("hello", m))            # low confidence -> abstain

Knowledge check check yourself