Numerical & Tensor Computing

An array (numpy calls it an ndarray) is a grid of numbers that are all the same type, packed tightly in memory. Two words you'll see everywhere: shape (how many numbers along each dimension) and dtype (what kind of number — here 32-bit floats). This block just builds a few arrays and asks them about themselves.

1. np.array([1.0, 2.0, 3.0]) makes a 1-D array (a vector). v.shape is (3,) — three numbers in a single row — and v.dtype is float32 because we asked for it.
2. np.zeros((3, 4)) makes a 3-by-4 grid (a matrix) filled with zeros. .nbytes is the total memory it uses: 3×4 = 12 numbers × 4 bytes each = 48 bytes.
3. emb is a fake embedding matrix: 1000 rows, each a 384-number vector — the shape a real embedding model produces for 1000 texts.
4. emb[0] and emb[:10] grab row 0 and the first 10 rows. These are views — windows onto the same memory, not copies — so they're instant and use no extra RAM.

What the output means: (3,) float32, then (3, 4) 48, then the embedding matrix's shape and size (about 1500 KB). The point: shape and dtype together tell you exactly how much memory an array costs.

Try this: Change np.float32 to np.float64 and re-check .nbytes — it doubles, because each number now takes 8 bytes instead of 4.

The single most important numpy habit: do math on whole arrays at once instead of looping over elements yourself. This is called vectorization. The loop still happens, but down in fast compiled C code, not in slow Python — often 100× faster for the same result.

1. a and b are each a million random numbers.
2. The commented-out line shows the slow way: a Python for loop doing a[i]*b[i] + 1 one element at a time (a million interpreted steps).
3. out = a * b + 1 is the fast way: numpy multiplies every pair and adds 1 across the whole array in one call. This is elementwise math — same position in each array combines together.
4. .sum(), .mean(), .max() are reductions: they crunch the whole array down to one number, also in fast C.
5. a[a > 0.9] is a boolean mask: a > 0.9 makes an array of True/False, and indexing with it keeps only the elements that were True — replacing a filter loop.

What the output means: No printout here — the point is speed. The vectorized a * b + 1 produces the exact same result as the loop would, but in a fraction of the time.

Try this: Wrap both the loop version and the vectorized version in time.perf_counter() calls and print the difference. You'll typically see 50–100× speedup.

Broadcasting is numpy's rule for combining arrays of different shapes without writing a loop or making copies. It virtually stretches the smaller array to fit the bigger one. Here we use it to scale every row of a matrix to length 1 (a unit vector), which is exactly what you do before comparing embeddings.

1. np.linalg.norm(emb, axis=1, keepdims=True) measures the length of each row. axis=1 means "go along the columns, once per row"; keepdims=True keeps the result shaped (1000, 1) instead of collapsing to (1000,).
2. unit = emb / norms divides a (1000, 384) matrix by a (1000, 1) column. Broadcasting stretches that single column across all 384 columns, so every row gets divided by its own length.
3. The comments at the bottom state the rule: compare shapes right-to-left; each pair of dimensions must be equal, or one of them must be 1 (the 1 is the one that stretches).

What the output means: ~1.0 — after dividing each row by its length, every row is a unit vector (length 1). That's the normalization step that turns a dot product into cosine similarity (next section).

Try this: Print norms.shape and emb.shape side by side, then try emb / np.linalg.norm(emb, axis=1) without keepdims=True — you'll get a shape-mismatch error, which shows why the (1000, 1) shape matters.

This is the heart of semantic search / RAG. Every document and the query are turned into vectors; the most similar documents are the answer. When vectors are normalized to length 1, similarity is just their dot product — multiply matching positions and add them up. A single matrix multiply scores the entire corpus at once.

1. normalize(x) divides each vector by its length (the broadcasting trick from section 3), so every vector has length 1.
2. corpus is 10,000 normalized document vectors; query is one normalized query vector, both of dimension 384.
3. scores = corpus @ query — the @ is matrix multiplication. It takes the dot product of the query with every one of the 10,000 rows, giving 10,000 similarity scores. Because the vectors are unit-length, each dot product is the cosine similarity (1.0 = identical direction, 0 = unrelated).
4. np.argpartition(-scores, k)[:k] finds the indices of the top 5 scores cheaply, without fully sorting all 10,000. np.argsort(-scores[idx]) then sorts just those 5 into ranked order. (The minus signs flip it to largest-first.)

What the output means: Prints the indices of the 5 best-matching documents and their similarity scores, best first. This is precisely what a vector store returns for a query.

Try this: Set query = corpus[7] (copy an existing document as the query) and re-run — document 7 should come back on top with a score very close to 1.0, because a vector is most similar to itself.

Two staples of scientific computing. Sparse matrices store only the non-zero entries — a huge memory win when most of the grid is zeros (like a word-counts-per-document table). And np.linalg.solve solves a system of linear equations, the workhorse behind regressions and least-squares fitting.

1. sparse.csr_matrix(...) takes a normal array that's mostly zeros and stores it compactly. tdm.nnz is the number of non-zeros — here 2, even though the grid has 6 cells. Only the 2 real values cost memory.
2. A and b describe the equations 3x + y = 9 and x + 2y = 8 in matrix form (A x = b).
3. np.linalg.solve(A, b) finds the x that satisfies both at once — no algebra by hand.

What the output means: tdm.nnz prints 2, and the solve returns [2., 3.] — meaning x=2, y=3. Plug those back in: 3(2)+3 = 9 and 2+2(3) = 8. Correct.

Try this: Change b to [9.0, 9.0] and re-solve to see how the answer shifts. This same solve is what fits a line to data in least-squares regression.

This tiny Value class is PyTorch in miniature — the mechanism that lets neural nets learn. Each Value holds a number and remembers how it was built. After a calculation you call .backward() and it works out the gradient: how much each input affected the result. That's automatic differentiation (autograd).

1. Every Value stores its data (the number), its grad (the gradient, starting at 0), and its _parents (the Values it came from) — so the whole calculation forms a graph.
2. __add__ and __mul__ define + and *. Each builds a new Value and stashes a small _back function saying how to push gradient to its inputs. The comments give the calculus: for a+b each input's share is 1; for a*b, a's share is b and vice versa.
3. backward() seeds the final gradient at 1.0, orders the nodes so each is handled after its parents (a topological sort), then runs every _back in reverse — the chain rule, applied backwards through the graph.
4. The demo computes f = a*b + a with a=2, b=3, then f.backward() fills in the gradients.

What the output means: 4.0 2.0. By hand: df/da = b + 1 = 3 + 1 = 4 and df/db = a = 2. The engine derived those automatically — that is exactly how a model figures out which way to nudge each weight.

Try this: Change a, b to Value(5.0), Value(-2.0) and predict a.grad (it's b+1 = -1) before running.

Quantization shrinks a model by storing its weights in fewer bits — here going from 32-bit floats to 8-bit integers, 4× smaller. The trick is to find one scale factor that maps the range of float values onto the integers -127…127, then multiply back when you need the (approximate) original.

1. weights is 1000 random float32 numbers standing in for trained model weights.
2. scale = np.abs(weights).max() / 127 finds how much one integer step is worth: the biggest magnitude weight maps to 127.
3. q = np.round(weights / scale).astype(np.int8) is the quantize step — divide by the scale and round to whole int8 values. This q is what actually gets stored, at 1 byte each.
4. dequant = q.astype(np.float32) * scale is the de-quantize step — multiply back to recover a close-but-not-exact float. err measures the average difference lost to rounding.

What the output means: 1000 4000 (bytes for int8 vs float32 — 4× smaller) and a tiny mean err. The small error is the price of the memory savings; for inference it's usually negligible.

Try this: Multiply weights by 100 before quantizing and watch mean err grow — a wider value range means each int8 step covers more, so rounding loses more precision.

The commented lines sketch the token-generation loop every LLM runs: a forward pass produces logits (a raw score per possible next token), softmax turns those scores into probabilities, and the model samples one token, appends it, and repeats. The runnable part is softmax itself.

1. softmax converts a list of arbitrary numbers into probabilities — all positive and summing to 1 — by exponentiating each and dividing by the total.
2. z = z - z.max() is the one line interviewers look for: subtracting the largest value first prevents np.exp from overflowing on big logits. It doesn't change the answer because the shift cancels in the division.
3. temp (temperature) divides the logits first: higher temperature flattens the probabilities (more random/creative), lower sharpens them (more focused) — the same knob the API exposes.
4. e / e.sum() normalizes the exponentiated values so they add up to 1.

What the output means: [0.659 0.242 0.099] — the three input scores [2.0, 1.0, 0.1] became probabilities that sum to 1, with the biggest logit getting the biggest share. The model would most likely pick token 0.

Try this: Call softmax(np.array([2.0, 1.0, 0.1]), temp=0.5) and compare — the lower temperature makes the top probability even larger (more confident, less random).

The bare interview version of softmax — turn scores into probabilities that sum to 1. Interviewers want to see the numerical-stability trick, not fancy code.

1. z = logits - np.max(logits) subtracts the largest score first. This is the make-or-break detail: without it, np.exp of a large logit overflows to infinity and you get nan. The subtraction cancels out in the final ratio, so the result is identical — just safe.
2. np.exp(z) raises e to each value (all now ≤ 0, so all safely between 0 and 1), and e / e.sum() scales them to add up to exactly 1.

Try this: Feed in np.array([1000.0, 1001.0]). This function handles it fine; a naive version without the - np.max line would return nan. That's the whole point of the question.

Scaled dot-product attention is the operation inside a Transformer (and thus every LLM). Plain-English version: for each query, score how well it matches every key, turn those scores into weights with softmax, then take a weighted average of the values. This function is that formula in four lines.

1. d = Q.shape[-1] reads the vector dimension — used to scale the scores.
2. scores = Q @ K.T / np.sqrt(d) — every query dotted with every key gives a match score. Dividing by √d keeps the numbers from getting huge as dimension grows, so softmax stays well-behaved.
3. The next two lines are softmax applied per row: subtract each row's max for stability, exponentiate, then divide by the row's sum so each row of weights adds to 1. axis=-1, keepdims=True makes that happen independently for each query.
4. w @ V uses those weights to take a weighted average of the value vectors — the attention output: each query's answer is a blend of the values it attended to most.

What the output means: Returns one output vector per query — each a weighted mix of the V rows, weighted toward the keys that matched that query best. Stack this operation many times and you have the core of a Transformer.

Try this: Make Q, K, V small (e.g. shape (2, 4)) and print w — confirm each row sums to 1. Those rows are the attention weights.