Information theory & optimization

1 · Entropy — expected surprise intermediate

Start with a bet, because that’s what a probability really is. Imagine two coins. One is fair — call heads or tails and you’re right half the time, forever unpredictable. The other is weighted 90/10; you learn to just say ‘heads’ and you’re rarely wrong. Now here’s the question that names this whole chapter: how surprised are you, on average, by each flip? The fair coin surprises you constantly — you never see the outcome coming. The biased coin barely surprises you at all. Entropy is the number that puts a precise value on “how surprised, on average” — and it turns out to be the exact tool that measures how hard a thing is to predict, which is the whole game in machine learning.

The intuition to hold onto: rare = surprising, certain = boring. If something you were sure would happen happens, you’ve learned nothing — zero surprise. If something you thought was nearly impossible happens, you’re shocked — huge surprise. Information theory makes this literal: the surprise of an outcome with probability p is −log p. When p is close to 1, −log p is near 0 (no news). When p is tiny, −log p is large (big news). Entropy is just the average of that surprise across all the outcomes, weighted by how often each one happens. Use base-2 logs and the answer comes out in bits — the number of yes/no questions you’d need, on average, to pin down the outcome.

The mistake almost everyone makes at first is thinking entropy measures the amount of data, or its size in bytes. It doesn’t — it measures unpredictability. A gigabyte of the letter ‘A’ repeated has almost zero entropy (utterly predictable, compresses to nothing); a gigabyte of random noise has maximum entropy (nothing to predict, won’t compress at all). That connection is not a coincidence: entropy is exactly the floor that compression can’t beat, the fewest bits per symbol any code could ever achieve. Keep that link in mind — a model that predicts text well is, in a real sense, a compressor of that text, and the loss it’s trained on is measured in these same units.

Entropy H(p) = −Σ p_i log p_i is the average surprise of a distribution — in bits when the log is base 2. A rare event (small p) carries more surprise (−log p is large); a certain event carries none. Entropy is maximized by the uniform distribution (maximum uncertainty) and zero for a certain outcome. It is the information-theoretic floor: the fewest bits per symbol any code could achieve, and the baseline the loss functions below are measured against.

python# stdlib only
import math

def entropy(p, base=2):
    return -sum(pi * math.log(pi, base) for pi in p if pi > 0)   # 0*log0 := 0

print("fair coin   :", entropy([0.5, 0.5]))          # 1.0 bit
print("biased coin :", round(entropy([0.9, 0.1]), 5))# 0.469 bits (less uncertain)
print("uniform of 4:", entropy([0.25]*4))            # 2.0 bits
print("certain     :", entropy([1.0, 0.0]))          # -0.0 bits (no surprise)

fair coin   : 1.0
biased coin : 0.469
uniform of 4: 2.0
certain     : -0.0

A fair coin needs exactly 1 bit; a 90/10 coin only 0.469 bits (you can usually guess ‘heads’); a uniform 4-way choice needs 2 bits; a certain outcome needs none. Entropy quantifies how much a distribution can surprise you — the reference point for measuring a model against reality next.

2 · Cross-entropy & KL divergence — THE loss advanced

This is the single most important equation in the whole course, so it earns a story. Every time an LLM is trained — every one of the trillions of steps that built the model you talk to — a number gets computed that says “how wrong were you?”, and the model nudges itself to make that number smaller. That number is cross-entropy. If you understand this section, you understand what “training” literally is: shrinking cross-entropy. Everything else — GPUs, backprop, learning rates — is machinery in service of driving this one quantity down.

Build the intuition on top of last section’s surprise. Entropy measured your average surprise when you knew the true odds. But a model doesn’t know the true odds — it guesses a distribution q while reality follows p. Cross-entropy asks: if you plan your life (or your compression code) around the model’s guesses q, but the world actually behaves like p, how surprised will you be on average? The analogy: you pack for a city’s weather using a forecast (q) but experience the real weather (p). A great forecast leaves you rarely caught out; a bad one has you soaked in a t-shirt. Cross-entropy is your average “caught out” cost — and crucially, it’s lowest exactly when your forecast matches reality. That’s why minimizing it makes the model’s guesses match the data.

KL divergence is the honest accountant sitting next to cross-entropy. It answers the follow-up question: of that total surprise, how much was unavoidable (the world is genuinely random — that’s the entropy H(p)) versus how much was your model’s fault for guessing wrong? KL is exactly the second part: the extra cost you paid purely because q ≠ p. That gives the identity that ties the section together: cross-entropy = unavoidable floor + your penalty. Since the floor is fixed by the data and you can’t change it, driving cross-entropy down is identical to driving your penalty (the KL) to zero.

The trap to avoid: people conflate KL with a distance, so they assume D(p‖q) and D(q‖p) are interchangeable. They are not — KL is asymmetric, and the warning box below spells out how choosing the wrong direction quietly changes what your model learns. For now, just hold the shape of the idea: cross-entropy is what you minimize, KL is why minimizing it works, and entropy is the wall you can never push past.

Cross-entropy H(p, q) = −Σ p_i log q_i is the average surprise when reality is p but you encoded for q — your model’s predicted distribution. KL divergence D(p‖q) = Σ p_i log(p_i/q_i) is the extra cost of that mismatch. The identity that matters: H(p, q) = H(p) + D(p‖q). Since H(p) is fixed by the data, minimizing cross-entropy = minimizing KL = making the model’s distribution match reality. When the label is one-hot, cross-entropy collapses to −log q(correct) — the negative log-likelihood that classifiers and LLMs train on.

python# stdlib only
import math

def entropy(p, base=2):        return -sum(pi*math.log(pi, base) for pi in p if pi > 0)
def cross_entropy(p, q, base=2): return -sum(pi*math.log(qi, base) for pi, qi in zip(p, q) if pi > 0)
def kl(p, q, base=2):          return sum(pi*math.log(pi/qi, base) for pi, qi in zip(p, q) if pi > 0)

# One-hot label -> cross-entropy is just -log(prob of correct class):
true = [1, 0, 0]
pred = [0.7, 0.2, 0.1]
print("CE (one-hot, nats):", round(cross_entropy(true, pred, math.e), 5))  # 0.35667
print("-ln(0.7)          :", round(-math.log(0.7), 5))                      # 0.35667

# General case: verify H(p,q) = H(p) + KL(p||q)
p = [0.5, 0.3, 0.2]
q = [0.4, 0.4, 0.2]
print("H(p)      :", round(entropy(p), 5))          # 1.48548
print("H(p,q)    :", round(cross_entropy(p, q), 5)) # 1.52193
print("KL(p||q)  :", round(kl(p, q), 5))            # 0.03645
print("H(p,q)-H(p):", round(cross_entropy(p, q) - entropy(p), 5))  # 0.03645

CE (one-hot, nats): 0.35667
-ln(0.7)          : 0.35667
H(p)      : 1.48548
H(p,q)    : 1.52193
KL(p||q)  : 0.03645
H(p,q)-H(p): 0.03645

With a one-hot label, cross-entropy is exactly −ln(0.7) = 0.357 — the model is penalized purely by the probability it gave the correct class. In the general case H(p,q) − H(p) equals KL(p‖q) = 0.03645 to the digit, confirming the decomposition. This is why the training loss for every classifier and LLM is cross-entropy: minimizing it drives the model’s predictions toward the true distribution (and equals the MLE of A11).

3 · Perplexity — what the LLM number means advanced

Open any language-model paper and you’ll trip over the word “perplexity” in the first page. It’s the number researchers quote to say “our model is better,” the number that dropped headline after headline as GPT-2 gave way to GPT-3 and beyond. But it looks like an intimidating piece of jargon, and most people nod along without knowing what a perplexity of 8 versus 20 actually means. Here’s the reassuring truth: perplexity is not a new idea at all — it’s cross-entropy from the last section, wearing friendlier clothes.

The problem it solves is human, not mathematical. Cross-entropy comes out in nats or bits — 2.08 nats, 3.00 nats — and those units mean nothing to your gut. So we exponentiate the loss to turn it into a count you can feel: an effective branching factor. The analogy that makes it click: imagine the model, at each word, is playing a guessing game where it must pick the next token. A perplexity of 8 means it’s doing about as well as if it were choosing blindly among 8 equally likely options at every step. A perplexity of 20 means it’s flailing among 20. Lower perplexity = fewer plausible branches = a model that’s genuinely narrowed down what comes next. When you read “perplexity 8 beats perplexity 20,” picture a confident reader who’s down to a short list of likely words versus a confused one still juggling twenty.

The common mistake is treating perplexity as an absolute grade, like a test score out of 100. It isn’t — it’s only meaningful relative to the same test set and the same tokenizer. Change the vocabulary, the domain, or how you split words into tokens and the number shifts, so comparing your model’s perplexity to someone else’s on a different dataset is meaningless. The one anchor that always holds: a model that knows nothing beyond a uniform guess over a vocabulary of size V has perplexity exactly V. That’s the ceiling of cluelessness, and every bit of real learning pulls the number below it — which the sanity check in the lab demonstrates to the digit.

Perplexity = exp(average negative log-likelihood) = exp(cross-entropy in nats). It is the cross-entropy loss made interpretable: a perplexity of K means the model is, on average, as uncertain as if it were choosing uniformly among K equally likely tokens. Lower is better. A model that predicts the actual next token with high probability has low perplexity; a clueless uniform model over a vocabulary of size V has perplexity V. It is the standard intrinsic metric for language models.

python# stdlib only
import math

# The probability the model assigned to each ACTUAL next token of a sequence:
token_probs = [0.5, 0.1, 0.25, 0.4]
avg_nll = -sum(math.log(pt) for pt in token_probs) / len(token_probs)
ppl = math.exp(avg_nll)
print("avg NLL (nats):", round(avg_nll, 5))    # 1.32458
print("perplexity    :", round(ppl, 5))         # 3.7606

# Sanity check: a uniform model over vocab V has perplexity V.
V = 100
uniform_probs = [1.0 / V] * 10
ppl_u = math.exp(-sum(math.log(pt) for pt in uniform_probs) / len(uniform_probs))
print("uniform(V=100):", round(ppl_u, 5))       # 100.0

avg NLL (nats): 1.32458
perplexity    : 3.7606
uniform(V=100): 100.0

The model’s effective ‘branching factor’ on this sequence is 3.76 — as uncertain as picking among ~3.76 equally likely tokens each step. The sanity check confirms the meaning: a model that knows nothing beyond a uniform choice over 100 tokens has perplexity exactly 100. When you read ‘this LLM has perplexity 8 on the test set,’ it means an average branching factor of 8 — and it is just exp of the cross-entropy loss from §2.

4 · Mutual information intermediate

Here’s a question that sounds simple but is worth real money: does this feature actually tell me anything about what I’m trying to predict? You’ve got a hundred columns in a dataset and a label you want to guess. Some columns are gold; some are noise dressed up as signal. Feed the noise into your model and you waste compute, invite overfitting, and slow everything down. Mutual information is the principled tool that scores each column’s answer to exactly that question — “how much does knowing this reduce my uncertainty about the label?”

The intuition builds directly on surprise and entropy. Uncertainty about Y is its entropy — how surprised you’d be by Y on average. Now someone whispers the value of X in your ear. If X and Y are related, your uncertainty about Y drops: you can guess it better now. Mutual information is precisely that drop — the bits of uncertainty about Y that vanish once you know X (and, beautifully, it’s symmetric: knowing Y tells you exactly as much about X). The analogy: knowing the sky is cloudy sharpens your guess about rain; the two share information. Knowing today’s lottery numbers tells you nothing about tomorrow’s coin flip; they share none.

The clean way to see it — and the mistake it guards against. Mutual information is the KL divergence between the joint distribution p(x,y) and the product of the marginals p(x)p(y). That product is what the joint would look like if X and Y were independent, so MI measures exactly how far reality departs from independence. When they truly are independent, the joint equals the product, KL is zero, and I(X;Y) = 0 — on the nose. The trap here is confusing MI with plain correlation: correlation only catches linear straight-line relationships and will happily report zero for a strong curved or XOR-style dependence. Mutual information catches any statistical relationship, linear or not — which is why it’s the honest score for “is this feature useless?” when the pattern might be weird.

Mutual information I(X; Y) = Σ p(x,y) log(p(x,y)/(p(x)p(y))) measures how much knowing one variable reduces uncertainty about the other — it is the KL divergence between the joint and the product of marginals. I = 0 exactly when X and Y are independent. In AI it drives feature selection (keep features informative about the label), diagnostics, and information-theoretic probes of what a representation encodes.

python# stdlib only
import math

# Joint P(X,Y) over {0,1}x{0,1}: X and Y agree 80% of the time.
joint = [[0.4, 0.1],
         [0.1, 0.4]]
px = [sum(row) for row in joint]                       # marginal of X
py = [sum(joint[i][j] for i in range(2)) for j in range(2)]  # marginal of Y

mi = 0.0
for i in range(2):
    for j in range(2):
        if joint[i][j] > 0:
            mi += joint[i][j] * math.log(joint[i][j] / (px[i]*py[j]), 2)
print("marginals:", px, py)         # [0.5, 0.5] [0.5, 0.5]
print("MI (bits):", round(mi, 5))   # 0.27807  (>0 -> dependent)

marginals: [0.5, 0.5] [0.5, 0.5]
MI (bits): 0.27807

Each variable alone is a fair coin (marginals 0.5/0.5), but they agree 80% of the time, so knowing X tells you 0.278 bits about Y — positive mutual information means dependence. Make the joint the product of marginals (0.25 everywhere) and MI drops to 0. This is the quantity a feature selector maximizes to keep inputs that actually inform the label.

5 · Gradient descent from scratch advanced

We’ve spent four sections learning how to measure wrongness. Now comes the payoff: how do you fix it? A model has a knob — millions or billions of knobs, really — and some setting of those knobs makes the loss small. The catch is that nobody can solve for the right setting directly; the loss landscape is far too tangled. So instead we do something almost embarrassingly simple, and it’s the beating heart of all modern AI training: we feel which way is downhill and take a small step that way. Repeat a few million times. That’s gradient descent, and it is how every neural network that has ever worked got trained.

The analogy everyone uses because it’s the right one: you’re on a foggy mountainside trying to reach the valley. You can’t see the bottom, but you can feel the slope under your feet. So you take a step in the steepest downhill direction, feel the slope again, step again. The gradient is that felt slope — a vector pointing in the direction of steepest uphill, which is why we step in its negative. The size of each step is the learning rate η, and it’s the one knob you’ll fuss over most. Tiny steps and you’ll be on the mountain till nightfall; giant steps and you’ll leap clean over the valley and end up higher than you started.

That last failure is the mistake worth burning into memory, because you will hit it. A learning rate that’s too large doesn’t just slow things down — it makes each step overshoot the minimum by more than the last, so the parameter bounces outward with ever-growing swings until the numbers blow up. In real training you see this as the loss suddenly printing NaN or inf and everyone groaning. The lab below shows it happen on a trivial one-parameter loss: a good learning rate glides to the answer, a bad one explodes to −112 and keeps going. Watching it diverge on a function this simple is the fastest way to build a permanent instinct for “my learning rate is too high.”

Training minimizes a loss by gradient descent: repeatedly step the parameters in the direction of the negative gradient (steepest downhill), x ← x − η·∇f(x), where η is the learning rate. The gradient points uphill; negating it descends. The learning rate is the make-or-break knob: too small and it crawls, too large and it overshoots and diverges.

python# stdlib only
def f(x):    return (x - 3)**2
def grad(x): return 2 * (x - 3)          # df/dx, by hand

def descend(lr, steps):
    x = 0.0
    for _ in range(steps):
        x -= lr * grad(x)                # the update rule
    return x

good = descend(lr=0.1, steps=50)
print("lr=0.1 ->", round(good, 6), "f =", round(f(good), 8))   # ~3.0, ~0.0

bad = descend(lr=1.1, steps=20)          # too large: overshoots
print("lr=1.1 ->", round(bad, 3))        # blows up -> -112.013

lr=0.1 -> 2.999957 f = 0.0
lr=1.1 -> -112.013

With η=0.1 the parameter converges to the minimum x=3 (loss ~0). With η=1.1 each step overshoots the minimum by more than it started, so the iterate explodes to −112 and keeps growing — the classic diverging-loss symptom you see as NaN in training when the learning rate is too high. Tuning η (and schedules/warmup) is a core part of getting training to converge (see MS2).

6 · Convexity, SGD vs batch advanced

Gradient descent always walks downhill — so why doesn’t it always reach the bottom? That question separates the toy losses in the last section from the reality of training a real network, and the answer is the shape of the landscape. Some loss surfaces are a single clean bowl; others are a mountain range full of dips, ridges and false valleys. Whether your foggy-mountain walk is guaranteed to find the true bottom or might get stranded in a roadside ditch comes down to one property: convexity.

A convex function is the friendly case: one bowl, no tricks. Picture pouring water anywhere on it — it always rolls to the same lowest point. There’s exactly one valley, so “keep walking downhill” cannot fail; any minimum you find is the global minimum. The clean test is that the graph curves up everywhere (its second derivative stays ≥ 0). Our friend (x−3)² is convex — a perfect parabola bowl. The sobering fact is that deep networks are emphatically not convex: their loss surface is a wild, high-dimensional terrain of countless dips. That’s exactly why training them is part science, part craft — and why practitioners obsess over initialization, learning-rate schedules, and warmup instead of just running “the algorithm.”

The second idea here answers a practical worry: computing the gradient over your entire dataset every single step is brutally expensive. If you have ten million examples, one honest step means touching all ten million — and you need thousands of steps. That’s batch gradient descent: maximally accurate, painfully slow. The opposite extreme, stochastic gradient descent (SGD), estimates the downhill direction from a single example — wildly cheap, but jittery, since one example is a noisy stand-in for the whole. The mistake beginners make is assuming that noise is purely bad; in fact the jitter can bounce you out of shallow bad valleys that a perfectly smooth gradient would sink into. The winning compromise everyone actually ships is mini-batch: a handful of examples per step — enough to smooth the noise, few enough to fit in memory and step often. Every framework you’ll ever use does this by default.

A function is convex if its graph curves up everywhere (second derivative ≥ 0) — then any local minimum is the global minimum and gradient descent is guaranteed to find it. (x−3)² is convex; deep networks are not, which is why training is an art. Batch GD uses the whole dataset per step (accurate gradient, expensive); stochastic GD (SGD) uses one example (noisy but cheap and can escape shallow traps); mini-batch — a handful at a time — is the practical middle ground every framework uses.

python# stdlib only
# Convex: second derivative constant and positive.
def f(x): return (x - 3)**2
h = 1e-5
def second_deriv(x):
    return (f(x+h) - 2*f(x) + f(x-h)) / h**2
print("f''(x) ~", round(second_deriv(0.0), 3), "-> convex (>0)")   # 2.0

# Linear model loss over 4 points; compare batch grad vs single-sample grad.
xs = [1.0, 2.0, 3.0, 4.0]
ys = [3.0, 5.0, 7.0, 9.0]           # y = 2x + 1
w = b = 0.0
# Batch gradient (averaged over ALL points):
n = len(xs)
gw = sum(2*((w*x+b)-y)*x for x, y in zip(xs, ys)) / n
gb = sum(2*((w*x+b)-y)   for x, y in zip(xs, ys)) / n
print("batch grad (w,b):", round(gw, 3), round(gb, 3))     # -35.0 -12.0
# SGD gradient (ONE random-ish point, here the first):
x0, y0 = xs[0], ys[0]
sgw, sgb = 2*((w*x0+b)-y0)*x0, 2*((w*x0+b)-y0)
print("sgd  grad (w,b):", round(sgw, 3), round(sgb, 3))    # -6.0 -6.0

f''(x) ~ 2.0 -> convex (>0)
batch grad (w,b): -35.0 -12.0
sgd  grad (w,b): -6.0 -6.0

The numeric second derivative is 2 everywhere — positive, so the loss is convex and GD will find the global minimum. The batch gradient averages all four points (−35, −12); the SGD gradient from a single point (−6, −6) is a cheaper, noisier estimate of the same direction. Real training uses mini-batches: enough samples to smooth the noise, few enough to fit in memory and step often (MS1).

7 · The chain rule & backprop — tiny, end-to-end expert expert

Gradient descent needs the gradient — but where does it come from? On a one-parameter toy loss we just wrote the derivative down by hand. A real network has billions of parameters chained through dozens of layers; there is no chance of differentiating that by hand, and doing it naively (perturbing each parameter one at a time to see how the loss moves) would take billions of forward passes per step. Backpropagation is the breakthrough that makes it feasible: it computes the gradient with respect to every parameter in essentially a single backward sweep. Without it there is no deep learning — full stop. This section builds it on the smallest possible model so you can watch every number, then tells you honestly that the mechanism is identical at trillion-parameter scale.

The whole trick is one rule from calculus you already half-know: the chain rule. Think of a factory assembly line. Raw input goes in one end, passes through station after station, and a finished product — the loss — comes out the other. Now something’s wrong with the final product and you want to know which station to adjust and by how much. The chain rule says: figure out how sensitive the output is to the last station, then how sensitive that station’s input is to the station before it, and multiply those sensitivities as you walk backwards up the line. Each parameter’s gradient is just the product of the local sensitivities on the path from it to the loss. Backprop is exactly this — a backward walk multiplying local derivatives — done systematically so shared work is never repeated.

The mistake to avoid is imagining backprop as something separate and magical from what you’ve already learned. It is not a different algorithm; it is simply how you obtain the gradient that gradient descent then consumes. The full loop — and this is the punchline of the entire chapter — is always the same four beats: run the forward pass to get a prediction, compute the loss (cross-entropy for an LLM, squared error here), backprop the gradients via the chain rule, and step the parameters downhill. The lab below runs exactly that loop on a 1-feature line and recovers y = 2x + 1 from a cold start. Swap the hand-derived derivatives for an autograd engine (the tensor library of A4) and the very same four beats train a model with a hundred billion parameters. Nothing about the idea changes — only the number of knobs.

Backpropagation is the chain rule applied backwards through a computation to get the gradient of the loss with respect to every parameter. For the loss L = (w·x + b − y)², the chain rule gives ∂L/∂w = 2(pred−y)·x and ∂L/∂b = 2(pred−y). Below we train a 1-feature linear model end-to-end with these hand-derived gradients — deliberately the smallest case so you can see every number, but the exact mechanism (forward pass → loss → gradients → update) that scales to billion-parameter nets (the autograd engine of A4).

python# stdlib only  (deliberately a 1-feature model so every step is visible)
xs = [1.0, 2.0, 3.0, 4.0]
ys = [3.0, 5.0, 7.0, 9.0]           # target relationship: y = 2x + 1

w, b = 0.0, 0.0
lr = 0.01
for epoch in range(1000):
    dw = db = 0.0
    n = len(xs)
    for x, y in zip(xs, ys):
        pred = w * x + b            # forward pass
        err  = pred - y             # dL/dpred = 2*err (MSE)
        dw += 2 * err * x / n       # chain rule: dL/dw
        db += 2 * err / n           # chain rule: dL/db
    w -= lr * dw                    # gradient-descent update
    b -= lr * db

loss = sum((w*x + b - y)**2 for x, y in zip(xs, ys)) / len(xs)
print("learned w, b:", round(w, 4), round(b, 4))   # ~2.0, ~1.0
print("final MSE   :", round(loss, 6))             # ~0.0

learned w, b: 2.0049 0.9857
final MSE   : 3.4e-05

Starting from w=b=0, 1000 gradient steps recover w≈2.00 and b≈0.99 — the true y = 2x + 1 relationship — driving the MSE to essentially zero. Every step did exactly what an LLM does per batch: run the forward pass, compute the loss, backprop the gradients via the chain rule, and nudge the parameters downhill. Swap the hand-derived gradients for an autograd engine and this scales to the whole network.

Checkpoint expert

import math
def entropy(p):
    return -sum(pi*math.log(pi, 2) for pi in p if pi > 0)

print(entropy([0.5, 0.5]))         # 1.0
print(entropy([0.25]*4))           # 2.0
print(round(entropy([0.9, 0.1]), 4))  # 0.469

1.0
2.0
0.469

import math
def ce(true, pred):
    return -sum(t*math.log(q) for t, q in zip(true, pred) if t > 0)

print(round(ce([1,0,0], [0.7,0.2,0.1]), 5))   # 0.35667  (-ln 0.7)
print(round(ce([1,0,0], [0.4,0.3,0.3]), 5))   # 0.91629  (-ln 0.4, worse)
print(round(-math.log(0.7), 5))               # 0.35667

0.35667
0.91629
0.35667

import math
def perplexity(token_probs):
    nll = -sum(math.log(p) for p in token_probs) / len(token_probs)
    return math.exp(nll)

print(round(perplexity([0.5, 0.1, 0.25, 0.4]), 5))   # 3.7606
V = 50
print(round(perplexity([1/V]*8), 5))                 # 50.0

3.7606
50.0

def grad(x): return 2*(x-3)
def run(lr, steps=60):
    x = 0.0
    for _ in range(steps): x -= lr*grad(x)
    return x

for lr in (0.01, 0.1, 0.9, 1.1):
    print(lr, round(run(lr), 4))

0.01 2.1073
0.1 3.0
0.9 3.0
1.1 -169039.5431

import math
def mi(joint):
    px = [sum(r) for r in joint]
    py = [sum(joint[i][j] for i in range(len(joint))) for j in range(len(joint[0]))]
    tot = 0.0
    for i,row in enumerate(joint):
        for j,pij in enumerate(row):
            if pij > 0: tot += pij*math.log(pij/(px[i]*py[j]), 2)
    return tot

print(round(mi([[0.4,0.1],[0.1,0.4]]), 5))   # 0.27807 (dependent)
print(round(mi([[0.25,0.25],[0.25,0.25]]), 5))  # 0.0 (independent)

0.27807
0.0

xs = [1.0, 2.0, 3.0, 4.0]
ys = [3.0, 5.0, 7.0, 9.0]        # y = 2x + 1
w = b = 0.0
lr = 0.01
for _ in range(2000):
    n = len(xs)
    dw = sum(2*((w*x+b)-y)*x for x,y in zip(xs,ys)) / n
    db = sum(2*((w*x+b)-y)   for x,y in zip(xs,ys)) / n
    w -= lr*dw; b -= lr*db
loss = sum((w*x+b-y)**2 for x,y in zip(xs,ys)) / len(xs)
print(round(w,4), round(b,4), round(loss,8))

2.0002 0.9993 8e-08