Numerical Computing & Data Analysis with NumPy and Pandas

Why not just lists and dicts? essential

Picture the moment this bites you. You've built a little prototype that reads a CSV of customer orders into a list of dictionaries — one dict per row — and loops over them to total the revenue. On the 500-row sample it's instant, so you ship it. Then the real export lands: 4 million rows. The same loop now churns for a minute, your laptop fan spins up, and a colleague asks why the "total revenue" number takes so long to appear. Nothing in your code is wrong — it's the data structure that gave out under scale. This section is about the two libraries built precisely to survive that moment.

Here's the plain reason a list of dicts struggles. In Python, every value is a full object with its own header, type tag, and reference count — a single integer isn't 8 bytes of raw number, it's a boxed object sitting somewhere in memory. A list of dicts is therefore a pile of pointers to pointers: to add up a column of prices you touch a million little objects living in a million scattered places, and the CPU spends most of its time chasing addresses rather than doing arithmetic. NumPy flips this: it stores a whole column as one contiguous block of raw, same-type numbers — like a single long ruler instead of a scatter of loose beads — so the processor can stream through it and even use special wide instructions that crunch several numbers per clock tick.

The analogy that makes it stick: a Python list of dicts is a filing cabinet where each folder can hold anything and lives in a random drawer; a NumPy array (and the Pandas column built on it) is a printed spreadsheet where every cell in a column is the same kind of thing, lined up edge to edge. You don't rifle through drawers — you read straight down the page. That one layout choice — typed, contiguous, columnar — is what buys you both the speed and the one-line queries.

The mistake beginners make is assuming "it worked on my test data, so it'll scale." It won't — but the flip side matters too: a list of dicts is genuinely the right tool for a config file or a handful of records, and dragging in Pandas for a three-item list is its own kind of over-engineering. The skill is knowing where the line sits: reach for NumPy/Pandas when you have tabular data you'll compute over, and stay with plain Python when you don't. The table below is the honest side-by-side of what you trade and what you gain when you cross that line.

NumPy: the array engine essential

Every time you've felt a data script drag, the fix was probably one idea: stop looping, start vectorizing. NumPy is where that idea lives. It's the quiet engine under almost the entire scientific-Python world — Pandas, scikit-learn, PyTorch tensors, the embedding math in your RAG pipeline — all of it bottoms out in NumPy arrays. So the half-hour you spend getting comfortable here pays off in every library you touch later.

The one concept to internalize is vectorization: writing operations that apply to a whole array at once instead of to one element at a time in a Python loop. Why does that matter so much? Because a Python for loop pays a tax on every single iteration — the interpreter checks types, boxes and unboxes objects, updates counters — and on a million rows that tax dwarfs the actual arithmetic. When you write a * 10, NumPy hands the entire job to a tight, pre-compiled C loop that already knows every element is a number and just multiplies straight down the memory block. Same answer, but the per-item overhead vanishes, which is where the routine 10–100× speedups come from.

The analogy: a Python loop is you paying a toll booth once per car; vectorization is waving the whole convoy through on one pre-cleared pass. The cars (the numbers) are the same — you've just stopped stopping. The habit worth building now: whenever your fingers start typing for x in ... over numeric data, pause and ask "is there an array operation that does this to the whole thing at once?" Nine times out of ten there is, and it's both shorter and faster.

Two sub-ideas in the lab will follow you everywhere, so meet them by name up front. A boolean mask — a[a > 2] — first builds an array of True/False the same shape as your data, then keeps only the True positions; it's the seed of every row filter you'll write in Pandas. And axis is just "which direction do I collapse?": on a 2-D grid, axis=0 squashes the rows together to give you a per-column answer, and axis=1 squashes the columns to give a per-row answer. That axis convention trips up nearly everyone at first, so the plain reading — "axis=0 goes down, axis=1 goes across" — is worth pinning to memory before you read the code.

numpy_basics.pyimport numpy as np

a = np.array([1, 2, 3, 4])
print(a * 10)              # [10 20 30 40] — vectorized, no loop
print(a.mean(), a.sum())    # 2.5 10 — fast aggregates
print(a[a > 2])            # [3 4] — boolean mask selection

m = np.array([[1, 2], [3, 4]])
print(m.shape)              # (2, 2)
print(m.mean(axis=0))       # [2. 3.] — mean down each column

Pandas: the DataFrame essential

If NumPy gave you a fast column of numbers, the DataFrame is what makes that column usable by a human. A raw NumPy array has no idea that column 3 is "price" and column 0 is "customer name" — it's just numbers with positions. The moment your data has named fields of different types — a text name, an integer age, a float spend — you've outgrown a bare array, and that's exactly the gap the DataFrame fills. This is the structure you'll actually live in for the rest of your data career, so it's worth building a solid mental picture rather than memorizing method names.

Here's the picture: a DataFrame is a spreadsheet you drive with code instead of a mouse. It has named, typed columns down the page and a row index — the labels running down the left margin, like the row numbers in Excel. One single column, pulled out on its own, is a Series: a labeled 1-D array. So a DataFrame is really just a bundle of aligned Series that share one index. Grasping that "column = Series, table = DataFrame, index = the row labels that keep everything aligned" is genuinely 90% of the fluency; almost every Pandas operation is either "do something to a column (Series)" or "do something to the whole table (DataFrame)."

Why the index earns its own mention — it's the piece newcomers overlook and then get bitten by. The index isn't data; it's the row's name tag, and Pandas uses it to line up rows automatically when you combine, join, or assign between tables. Add two Series together and Pandas matches them by index label, not by position — which is a gift when your data lines up and a baffling source of surprise NaNs when it doesn't. The common mistake is treating the index as decorative and being shocked when a reset_index() or a mismatched index changes your results; the fix is to always know what your index is. Keep that in mind as you read the diagram below — the skinny leftmost strip isn't a column, it's the addressing system for everything else.

Lab B1.2 · The core loop: load → inspect → clean → transform → aggregate intermediate

Here's the reassuring secret of data work: it looks like a thousand different tasks, but underneath it's almost always the same five moves in the same order. Load the data, look at it, clean it, reshape it, summarize it. Once you've internalized that rhythm, a CSV you've never seen before stops being intimidating — you already know the first five things you're going to do to it. This section is that rhythm, and it's worth treating the order as a checklist you run every single time, not a suggestion.

The step people skip — and pay for — is inspect. The temptation is to jump straight from "load" to "compute the thing I care about," because inspecting feels like busywork. It isn't. Think of it like a cook tasting and smelling ingredients before throwing them in the pot: head() shows you what the rows actually look like, info() reveals the sneaky column that's stored as text when you assumed it was a number, and isna().sum() tells you where the holes are. Ten seconds of looking saves you from the classic disaster where you confidently average a column that was secretly full of nulls, or divide by a "price" that's actually the string "$1,200". The bug isn't loud — it produces a plausible-but-wrong number, which is the most dangerous kind.

The other mindset to carry through every stage is the vectorized one from the last section. Notice that when the loop reaches "transform," the code writes df["total"] = df["price"] * df["qty"] — a whole-column operation — rather than looping row by row. That's deliberate: the same "operate on columns, not rows" habit you learned in NumPy is what keeps this pipeline fast even when the CSV grows from a thousand rows to ten million. And the final move, aggregate, is where the payoff lands — a groupby collapses all that cleaned detail into the one-line answer a human (or your boss) actually asked for. Read the numbered comments in the lab as the stages of that pipeline; each block is one move, and the order is the whole lesson.

pipeline.pyimport pandas as pd

# 1 · LOAD
df = pd.read_csv("sales.csv")

# 2 · INSPECT — always look before you leap
df.head()          # first rows
df.info()          # columns, dtypes, non-null counts
df.describe()      # numeric summary stats
df.isna().sum()    # missing values per column

# 3 · CLEAN
df = df.dropna(subset=["price"])           # drop rows missing a price
df["region"] = df["region"].fillna("unknown")   # fill missing categories
df["date"] = pd.to_datetime(df["date"])  # fix a dtype

# 4 · TRANSFORM — vectorized, no loop
df["total"] = df["price"] * df["qty"]

# 5 · AGGREGATE
by_region = df.groupby("region")["total"].sum().sort_values(ascending=False)
print(by_region)

Lab B1.3 · Selecting & filtering (and the gotchas) intermediate

This is the operation you'll perform more than any other, so it's worth getting right in your fingers. "Give me the EU rows where price is over 100" is the bread and butter of analysis, and Pandas gives you a small, consistent set of tools to say exactly that. The good news is there are only a handful of patterns to learn. The catch is that a couple of them have traps sitting right next to the correct version — traps that don't crash, they just quietly give you the wrong answer — so this section spends as much time on the gotchas as on the happy path.

Start with the cleanest mental split. Selecting columns uses square brackets with names: one name gives you a Series (df["price"]), a list of names gives you a smaller DataFrame (df[["name","price"]]) — the double brackets are just "a list, inside the indexing brackets," not magic. Selecting rows uses a boolean mask, the exact same idea you met in NumPy: df["price"] > 100 builds a column of True/False, and df[that] keeps the True rows. If masks felt abstract in the NumPy section, this is where they earn their keep.

Two traps deserve a warning before you write a single line. First, when you combine conditions you cannot use Python's plain and/or — you must use & and |, and you must wrap each condition in parentheses: df[(df["price"] > 100) & (df["region"] == "EU")]. Why? Because and tries to boil a whole array down to a single True/False and throws an error, while & combines the masks element by element the way you meant. Second — and this is the famous one — never assign through a filter in two steps. Writing df[df.x > 0]["y"] = 1 looks reasonable but silently edits a throwaway copy and leaves your real DataFrame untouched (Pandas even prints a "SettingWithCopy" warning you'll learn to respect). The fix is to say it in one breath with .loc: df.loc[df.x > 0, "y"] = 1.

That last point leads to the final distinction: .loc versus .iloc. Think of it as "by name" versus "by seat number." .loc selects by labels or conditions — index labels and column names, or a boolean mask — which is what you want the vast majority of the time. .iloc selects by raw integer position, like slicing a list, ignoring whatever the labels say. Reach for .iloc only when position genuinely is what you mean (the first five rows, regardless of their labels); reach for .loc for everything else. Keep that pairing straight and the rest of the lab is mechanical.

select.pydf["price"]                 # one column → Series
df[["name", "price"]]      # several columns → DataFrame

# boolean-mask filtering (the NumPy idea, on rows)
df[df["price"] > 100]                          # rows where price > 100
df[(df["price"] > 100) & (df["region"] == "EU")]  # combine with & / | and parens

# label- vs position-based access
df.loc[df["region"] == "EU", "total"]   # .loc = labels/conditions
df.iloc[0:5]                             # .iloc = integer positions

Group-by: the analytical workhorse intermediate

Almost every real question anyone asks of data has the same secret shape: "per X, what's the Y?" Revenue per region. Average latency per endpoint. Error count per customer. Ticket volume per day. Once you notice that pattern, you realize a single Pandas move answers all of them — and that move is groupby. This is the tool that turns a raw table into an insight, which is why it earns the "analytical workhorse" title. Master this one operation and you can answer most of what a business will ever ask you.

The formal name is split-apply-combine, and it's worth saying slowly because the three words are the algorithm. Split: chop the rows into groups that share a key — all the EU rows here, all the US rows there. Apply: run the same summary calculation on each pile independently — sum the totals in EU, sum the totals in US. Combine: stack those per-group answers into one small result table, one row per group. Picture a mail sorter dumping a bin of letters into labeled pigeonholes (split), weighing each pigeonhole (apply), and writing the weights on a single sheet (combine). The diagram below draws exactly that flow left to right.

The beautiful part is how little the code changes as the question changes. df.groupby("region")["total"].sum() reads almost like the English sentence "for each region, sum the total." Want the average instead of the total? Swap .sum() for .mean(). Want how many orders per region? .count(). Want several statistics at once? .agg([...]). The shape of the operation is fixed; you're just turning two dials — which column to group by, and which summary to apply — to ask an entirely new business question. If you've met SQL, this is precisely GROUP BY (T5); the mental model transfers straight across, and knowing both lets you pick whichever tool the data already lives in.

Lab B1.4 · From messy data to LLM-ready records advanced

This is the section that explains why a Pandas chapter is sitting in an AI-engineering course at all. A language model is only as good as what you feed it, and models don't want a DataFrame — they want clean, well-shaped text or JSON, one tidy record at a time. Garbage in the table becomes garbage in the prompt, which becomes a confidently wrong answer out of the model. So the humble cleaning and reshaping you've been practicing turns out to be the load-bearing step of the whole pipeline: Pandas is the bridge between the raw, messy world and the pristine input an LLM can actually reason over.

The pattern to hold in your head is a round trip: in and out. On the way in, you clean a column of free text — drop the empty rows, strip the whitespace, filter out the junk — and then convert the surviving rows into a list of small dictionaries with to_dict(orient="records"). That list-of-dicts shape ([{"id": 1, "review": "..."}, ...]) is the natural hand-off point, because it's trivially JSON-serializable and trivially loopable — you iterate over it, sending each record to the model. On the way out, the model's structured answers get packed straight back into a new DataFrame and joined onto your cleaned rows, so predictions sit right beside the data that produced them, ready to aggregate or chart. Think of Pandas as the airport on both ends of the flight: it handles departures and arrivals.

One technique in the lab is worth calling out because it's what senior Pandas code looks like: the method chain. Instead of reassigning df over and over, you thread the operations together — df.dropna(...).assign(...).loc[...] — so the data flows through each step like a pipeline, and the code reads top to bottom as the sequence of transformations you're applying. It's cleaner to read and it sidesteps the SettingWithCopy trap from the last section entirely, because you're never assigning back into a filtered slice. The common mistake here is the opposite extreme: shaping data after it's already gone to the model, or handing the model rows you never cleaned — an empty or malformed record wastes a call and pollutes your results. Clean first, shape second, call the model last. Read the lab as that in-and-out bridge, and notice it's the exact loop the Data Analyst Agent project (P5) automates.

to_llm.py# clean + shape a table, then hand rows to an LLM step
clean = (df
    .dropna(subset=["review"])
    .assign(review=lambda d: d["review"].str.strip())
    .loc[lambda d: d["review"].str.len() > 0]
)

records = clean[["id", "review"]].to_dict(orient="records")
# records = [{"id": 1, "review": "..."}, ...] — ready to batch through a classifier (Ch 2)

# and the reverse: model results back into a DataFrame for analysis / a chart
results = pd.DataFrame([classify(r["review"]) for r in records])
merged = clean.reset_index(drop=True).join(results)

Common pitfalls advanced

Here's the honest truth about Pandas bugs: the dangerous ones don't crash. A syntax error is a gift — the program stops and points at the line. The pitfalls that actually hurt you in production are the silent ones: the code runs, prints a number, and everyone believes it, but the number is wrong. This table is a field guide to the handful of quiet failures that catch nearly everyone at least once, so you can recognize the smell before it costs you a bad decision.

Read down the list and you'll notice they all share one root cause — a mismatch between what you think Pandas is doing and what it's actually doing. You think a loop is fine; it's just slow (and gets slower as data grows). You think you cleaned the data; a column is secretly still a string. You think your assignment stuck; it edited a copy. You think you averaged the real values; you averaged around a pile of hidden NaNs. The defense against every one of them is the same discipline this whole chapter has been drilling: inspect before you trust, vectorize instead of loop, assign through .loc, and know your missing data on purpose. Treat the table as a checklist to run in your head whenever a result looks even slightly off — the fix is almost always one of these six.

Exercises advanced

import numpy as np

usd = np.array([10.0, 25.5, 100.0, 3.75])
rate = 0.92                      # USD -> EUR
eur = usd * rate                 # vectorized: no Python loop
print(np.round(eur, 2))          # [ 9.2  23.46 92.   3.45]

# the loop version does the same thing, slower and noisier:
eur_loop = [p * rate for p in usd]
print([round(x, 2) for x in eur_loop])

import pandas as pd
import numpy as np

df = pd.DataFrame({"amount": ["$1,200", "$350", None, "$45"]})

# strip $ and commas, coerce to number, fill missing, cast to int
df["amount_num"] = (
    df["amount"]
      .str.replace(r"[$,]", "", regex=True)   # "$1,200" -> "1200"
      .pipe(pd.to_numeric, errors="coerce")   # bad/None -> NaN
      .fillna(0)
      .astype(int)
)
print(df)
print("total:", df["amount_num"].sum())       # total: 1595

import pandas as pd

df = pd.DataFrame({
    "region":  ["US", "EU", "US", "APAC", "EU", "US"],
    "revenue": [100, 80, 120, 60, 40, 30],
})

summary = (
    df.groupby("region")["revenue"]
      .agg(total="sum", mean="mean", orders="count")
      .sort_values("total", ascending=False)
)
print(summary)
# region  total  mean  orders
# US        250  83.3       3
# EU        120  60.0       2
# APAC       60  60.0       1

import pandas as pd

orders = pd.DataFrame({"order_id": [1, 2, 3], "customer_id": [10, 11, 99]})
customers = pd.DataFrame({"customer_id": [10, 11], "name": ["Ana", "Ben"]})

merged = orders.merge(customers, on="customer_id", how="left")   # keep all orders
merged["name"] = merged["name"].fillna("<UNKNOWN>")
merged["orphan"] = merged["name"] == "<UNKNOWN>"
print(merged)
#    order_id  customer_id      name  orphan
# 0         1           10       Ana   False
# 1         2           11       Ben   False
# 2         3           99  <UNKNOWN>    True
assert merged["orphan"].sum() == 1

import pandas as pd
import json

df = pd.DataFrame({
    "Ticket ID": [1, 2, 3],
    "Subject":   ["Login fails", "Refund?", None],
    "Priority":  ["high", "low", "high"],
})

clean = (
    df.rename(columns={"Ticket ID": "id", "Subject": "subject", "Priority": "priority"})
      .dropna(subset=["subject"])          # subject is required for the LLM
)
records = clean.to_dict(orient="records")  # list[dict], one per row
print(json.dumps(records, indent=2))
# [
#   {"id": 1, "subject": "Login fails", "priority": "high"},
#   {"id": 2, "subject": "Refund?",     "priority": "low"}
# ]
assert all("subject" in r and r["subject"] for r in records)

import pandas as pd

def region_totals(rows):
    # rows: list of dicts with 'region' and 'revenue' (revenue may be str/None).
    # Returns {region: total_revenue}, skipping rows we can't clean.
    df = pd.DataFrame(rows)
    if df.empty:
        return {}
    df["revenue"] = pd.to_numeric(df.get("revenue"), errors="coerce")
    df = df.dropna(subset=["region", "revenue"])          # drop malformed
    out = df.groupby("region")["revenue"].sum().astype(float)
    return out.to_dict()

data = [
    {"region": "US", "revenue": "100"},
    {"region": "US", "revenue": 50},
    {"region": "EU", "revenue": None},     # dropped
    {"region": None, "revenue": 10},       # dropped
    {"region": "EU", "revenue": "80"},
]
result = region_totals(data)
assert result == {"US": 150.0, "EU": 80.0}, result
print(result)   # {'US': 150.0, 'EU': 80.0}

Knowledge check check yourself