Environments, packaging & dependencies

1 · Virtual environments — isolation essential

Here's the afternoon that turns everyone into a venv believer. You finish project A, which pins an old version of some library. You switch to project B, run pip install for something it needs, and pip quietly upgrades that shared library to satisfy B. A week later you open A again — and it's broken. Nothing in A's code changed; a package it relied on got upgraded out from under it by an unrelated project. You lose an hour bisecting versions before realising the two projects were fighting over one shared toolbox the whole time.

The everyday analogy is a shared kitchen. If every cook uses one set of pans, someone searing fish in the good skillet ruins the next person's pancakes. The fix isn't better etiquette — it's giving each cook their own counter with their own pans. A virtual environment is that private counter: a folder holding one project's packages and nothing else, so project A can keep its old library while project B pins a new one and neither ever notices the other exists.

Why it works is almost anticlimactic: a venv is mostly a folder plus a tweak to where your shell looks for python and pip. When you "activate" it, your terminal's search path is rewired so those commands point inside the venv instead of at the system Python. That single path change is the isolation — installs land in the project's box, and when you deactivate, the path flips back. The common mistake is skipping the venv "just for a quick test" and installing globally; that one shortcut is the single most common root cause of the dreaded "but it works on my machine." The rule is boring and absolute: one project, one venv, created before you install anything.

A venv is a private package folder per project so versions don't collide. Every project starts with one.

venv.shpython3 -m venv .venv
source .venv/bin/activate           # prompt shows (.venv)
pip install requests
pip freeze > requirements.txt       # snapshot exact versions
deactivate

2 · Dependencies & requirements essential

A venv keeps your project's packages private — but privacy alone doesn't make them repeatable. Picture handing your project to a teammate with the note "just pip install the stuff it uses." They install the latest of everything, a minor version has shifted since you built it, and now a function you relied on behaves differently. The code is identical on both machines; the versions underneath it are not. The gap between "it has the packages" and "it has the same packages I tested" is exactly what this section closes.

Think of it like a recipe. "Add some flour and a bit of sugar" works when you cook it, because your hands know the amounts — but it's useless to anyone else, and useless to future you who's forgotten. A real recipe says "240g flour, 100g sugar." A requirements.txt is that precise recipe for your environment: it doesn't say "install requests-ish," it says requests==2.32.3, down to the patch. Anyone with the file can reproduce your exact kitchen.

The mechanic that makes this honest is pip freeze, and here's the detail beginners miss: it records everything installed, not just the handful of packages you named. When you pip install fastapi, fastapi drags in a dozen packages of its own (its dependencies), and those have versions too. Freeze captures the whole tree, so the rebuild is complete rather than "the top-level names, and whatever versions of the rest happen to be latest today." The common mistake is hand-writing requirements.txt with only the packages you typed — that leaves the transitive dependencies floating, and floating is precisely where "works on my machine" hides. Freeze the full set; that's the recipe that actually reproduces.

requirements.shpip install fastapi uvicorn
pip freeze > requirements.txt
pip install -r requirements.txt     # reproduce exactly, elsewhere/CI/container
cat requirements.txt                # fastapi==0.115.0  uvicorn==0.30.0 ...

3 · Semantic versioning intermediate

You've pinned your versions — so now the question is how to read them. A number like 2.3.1 looks arbitrary until you learn it's a promise, and once you see the promise you can answer the question every upgrade forces on you: is this safe to take, or is it about to break my code? Almost everyone learns this the hard way — you bump a library to grab a bug fix, the install succeeds, and an import that worked yesterday now throws because the number quietly told you it might and you didn't listen.

The everyday analogy is the way we talk about house renovations. Repainting a wall (a patch) is safe — walk in tomorrow and everything's where you left it. Adding a new room (a minor change) gives you more without touching what already worked. Knocking down a load-bearing wall (a major change) means the old floor plan is gone; your muscle memory for where the kitchen was is now wrong. Semantic versioning encodes exactly that in three numbers: MAJOR.MINOR.PATCH. The leftmost number is the one that can hurt you.

Why this matters for the specs you write: the version rule you put in a requirements file is a bet on how much change you'll tolerate. ==2.3.1 says "exactly this, never surprise me" — right for an app you deploy, wrong for a shared library that would then clash with everyone else's pins. >=2.3,<3 says "any 2.x from 2.3 up, but never cross into 3.0" — you're trusting the promise that anything below the major bump won't break you, while slamming the door on the one number that's allowed to. The common mistake is treating a major bump as routine because "the install worked." Installing is not the same as running; a green install with a major bump is precisely the moment to read the changelog and test, not to ship on Friday afternoon.

Versions are MAJOR.MINOR.PATCH. MAJOR = breaking, MINOR = new-but-compatible, PATCH = bugfix. This tells you whether an upgrade is safe, and which spec to write.

semver.pydef parse(v): return tuple(int(x) for x in v.split(".")[:3])
def is_compatible(have, spec_min, spec_max_major):
    h = parse(have)
    return parse(spec_min) <= h and h[0] < spec_max_major

print("2.3.1 satisfies >=2.3,<3 :", is_compatible("2.3.1", "2.3.0", 3))
print("3.0.0 satisfies >=2.3,<3 :", is_compatible("3.0.0", "2.3.0", 3))  # breaking major
print("major bump = review before upgrading")

2.3.1 satisfies >=2.3,<3 : True
3.0.0 satisfies >=2.3,<3 : False
major bump = review before upgrading

4 · Advanced — dependency resolution & conflicts advanced

Sooner or later pip will look you dead in the eye and say ResolutionImpossible, and it feels like the tool is broken. It isn't. You asked for library A and library B, both of which you legitimately need, and it turns out A insists on urllib3>=2 while B insists on urllib3<2. There is no single version of urllib3 that makes both happy, and since only one copy can be installed, pip has no move to make. The first time it happens you'll stare at the error convinced you typed something wrong; understanding what's actually going on turns a lost afternoon into a five-minute diagnosis.

Here's the analogy that makes it click. Imagine planning a dinner for two friends: one is vegan, the other is on a strict all-meat diet. Any single dish that satisfies one insults the other — the requirements don't overlap, so no menu exists. That's a dependency conflict exactly: each requirement carves out a range of acceptable versions, and a solution only exists where those ranges intersect. When the intersection is empty, no amount of cleverness conjures a version into being. This shape has a name — the diamond dependency — because A and B both point down at the same shared package from different directions.

What a resolver does is search: it walks the available versions of each package looking for one combination that lands inside every requirement's range at once, and it reports failure only when it can prove no such combination exists. Modern pip does this properly (older pip used to just install the last one it saw and leave you with a silently broken environment — worse than an honest error). The common mistake is reading "ResolutionImpossible" as "pip is buggy" and forcing the install with flags; that just recreates the old silent breakage. The real fix is to widen someone's range — upgrade B to a release that accepts the new urllib3, or hold A back — so the intervals overlap again. The lab below is a miniature resolver so you can watch that intersection logic run.

When two of your dependencies need different versions of a third, you get a conflict. Understanding this saves hours. A resolver finds a version satisfying all constraints — or fails. Here's the core idea, runnable.

resolver.pydef resolve(constraints, available):
    """constraints: list of (min_major, ...). available: sorted versions.
    Return the highest version satisfying ALL constraints, or None."""
    def ok(v, c): return c["min"] <= v < c["lt_major"]
    for v in sorted(available, reverse=True):
        if all(ok(v, c) for c in constraints):
            return v
    return None

# pkgA needs >=1.2,<2 ; pkgB needs >=1.5,<2 -> highest 1.x that fits
avail = [1.1, 1.3, 1.6, 1.9, 2.0, 2.1]
cons = [{"min": 1.2, "lt_major": 2.0}, {"min": 1.5, "lt_major": 2.0}]
print("resolved to:", resolve(cons, avail))            # 1.9
# now a conflict: pkgC demands >=2.0
cons2 = cons + [{"min": 2.0, "lt_major": 3.0}]
print("with pkgC>=2.0:", resolve(cons2, avail))        # None -> CONFLICT

resolved to: 1.9
with pkgC>=2.0: None

5 · Advanced — a clean project layout advanced

Every messy Python project starts the same innocent way: one script.py at the top of a folder. Then a second script, a helpers.py, a stray test.py, a data.csv, three notebooks, and a requirements.txt someone added once and never updated. It runs fine on your laptop — right up to the day you try to install it somewhere else and discover the imports only worked because you happened to be standing in that exact folder. Nothing was wrong with the code; it was never arranged to be shippable.

The analogy is a kitchen versus a junk drawer. In a junk drawer everything is technically present — but finding the tape means digging past batteries and takeout menus. A professional kitchen has a place for knives, a place for pans, a place for spices, and any cook who walks in already knows where to look. A standard project layout is that shared floor plan: your importable code lives under src/, your tests live under tests/, and a single control file (pyproject.toml) sits at the top describing the whole thing. The value isn't tidiness for its own sake — it's that tools and teammates already know the layout, so packaging, testing, and imports all just work without special-casing your project.

The one non-obvious piece worth understanding is why the code goes under src/ rather than loose at the top. With a flat layout, import my_project can accidentally succeed by picking up the folder you're standing in — even before the package is properly installed — so you test something subtly different from what your users will install. Pushing the code down into src/ removes that shortcut: the only way to import it is to install it, which means you test the installed package, exactly as a teammate would get it. The common mistake is dismissing all this as ceremony and leaving scripts at the root; that flat pile is precisely what can't be pip-installed, can't be tested in isolation, and quietly imports the wrong thing. The tree below is the shape to grow into.

layout.txtmy_project/
├── README.md
├── .gitignore
├── pyproject.toml          # metadata + deps + entry points (modern standard)
├── src/my_project/
│   ├── __init__.py
│   └── core.py
└── tests/                  # your Testing section lives here
    └── test_core.py

6 · Professional — package with pyproject.toml professional

There's a quiet line every project eventually needs to cross: from "a folder of scripts I run" to "a thing other people install." The moment a teammate asks "how do I use your tool?" and your honest answer is "clone the repo, cd into it, activate the venv, then run python scripts/thing.py from this specific directory" — you've hit the wall. That's not a product, it's a set of instructions people will get wrong. Turning the folder into a real package is what lets them type pip install once and then just run your command from anywhere, like they run black or pytest.

The everyday analogy is the difference between a home-cooked meal and a boxed product on a shelf. Both are food, but the boxed one has a label listing exactly what's inside, how much, what it needs, and how to prepare it — so a total stranger can use it without phoning you. pyproject.toml is that label for your code. In one declarative file it states the package's name, its version, which Python it needs, what it depends on, and even what command it should expose once installed. It's written in TOML, a deliberately boring key = value format grouped into [sections] so both humans and tools can read it.

Two ideas in this file repay understanding. First, [project.scripts] is the magic that makes your-tool a real terminal command: it maps a name to a function, and after install, typing that name runs that function — that's literally how the CLI tools you use every day get their commands. Second, pip install -e . (an editable install) installs the package while pointing back at your source files, so your edits take effect immediately without reinstalling — the sweet spot for developing a package you're also using. The common mistake is reaching for the old setup.py you saw in a five-year-old tutorial; the modern standard (PEP 621) is this single declarative pyproject.toml, and it's both simpler and what current tooling expects.

pyproject.toml[project]
name = "my-project"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = ["requests>=2.30,<3"]

[project.scripts]
mytool = "my_project.core:main"      # pip install . -> `mytool` command

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
# build a wheel:  python -m build   ;  editable install:  pip install -e .

7 · Professional — inspect the environment professional

The most maddening bug reports are the ones that end with "…but it works fine here." A user's agent crashes on a version mismatch you can't reproduce, and the whole conversation is two people describing invisible environments to each other. The way out is to stop guessing and have the program tell you what's actually installed on the machine where it's running. That single habit — logging your real environment — collapses a day of back-and-forth into one glance at a log line.

The analogy is the label inside a device that reads "model X, firmware 2.1." When something misbehaves, a technician doesn't interrogate you about what you think you have — they read the label. Your running Python process has exactly such a label, and you don't need to shell out to pip list or parse text to read it: the standard library's importlib.metadata asks the environment directly, in code, with no extra packages to install. It can hand you the version of any single package or enumerate everything installed.

Why do this from inside your program rather than at a terminal? Because the terminal only tells you, once, right now. Reading metadata in code means your app can print its own package versions into its startup logs, an agent can check that a critical dependency meets a minimum before it does anything risky, and a diagnostic endpoint can report its environment on demand. The common mistake is scraping pip freeze output as text and pattern-matching it — brittle, slow, and it breaks the moment the format shifts. Query the metadata API instead; it's the same source of truth, structured and reliable. This lab reads it, and it's the exact building block the tech-lead CI check in §8 turns into an automated guardrail.

metadata.pyfrom importlib.metadata import version, distributions
print("pip:", version("pip"))
some = sorted(d.metadata["Name"] for d in distributions())[:5]
print("installed (sample):", some)

pip: 24.0
installed (sample): ['pip', 'requests', 'setuptools', ...]

8 · Tech-lead — reproducibility across the team tech-lead

Every "works on my machine" story ends the same way, and as a lead it eventually becomes your story to end. A deploy fails at 6pm, three engineers pile onto a call, and forty minutes later someone finds it: one machine had a dependency that had floated to a newer version between two installs. No one changed the code. No one changed the requirements on purpose. A single unpinned line drifted, and the whole team paid for it. The junior-engineer reaction is to fix that one version and move on; the tech-lead reaction is to make the whole class of bug impossible to merge again.

The analogy is the shift from "we're all careful" to "the door won't open unless it's safe." Verbal agreements — "everyone pin your deps, okay?" — fail exactly when it matters, because they rely on every person remembering under deadline pressure. What scales instead is turning reproducibility from a hope into a checkable property: a pinned lockfile committed to the repo, a documented one-command setup, and an automated gate in CI that reads the environment and refuses to merge anything that has drifted. The rule stops living in people's heads and starts living in the pipeline, where it can't be forgotten.

Concretely, "reproducible" means: same lockfile plus same Python version yields the same bytes installed, on a laptop, a CI runner, or inside a container — no exceptions. That's why apps commit a fully pinned lock (exact versions, ideally with hashes) while libraries stay loose and let the installing app resolve. The CI check in the lab below is the enforcement arm: it audits what's actually installed against what's required and exits non-zero on any mismatch, failing the build before the mismatch reaches production. The common mistake is trusting a requirements.txt full of >= ranges to be reproducible — it isn't; two installs a week apart can land on different versions. Ranges express intent; a pinned lock plus this gate is what delivers reproducibility, and enforcing that distinction is squarely the lead's job.

A lead guarantees everyone runs the same environment: pinned lockfiles, a documented setup, and a CI check that catches drift before it causes "works on my machine" bugs.

env_audit.pyfrom importlib.metadata import version, PackageNotFoundError
REQUIRED = {"pip": None}   # in real use: {"requests":"2","pytest":"8", ...} pinned majors

def audit(required):
    problems = []
    for pkg, want_major in required.items():
        try: have = version(pkg)
        except PackageNotFoundError: problems.append(f"MISSING: {pkg}"); continue
        if want_major and have.split(".")[0] != want_major:
            problems.append(f"{pkg}: want major {want_major}, have {have}")
    return problems

issues = audit(REQUIRED)
print("environment OK" if not issues else "DRIFT:\n" + "\n".join(issues))

environment OK

python3 -m venv .venv          # create the env in ./.venv
source .venv/bin/activate      # activate (Windows: .venv\Scripts\activate)
python -m pip install requests
python -c "import requests; print(requests.__version__)"
which python                   # points inside .venv, not /usr/bin
deactivate                     # leave the env

pip freeze > requirements.txt      # exact versions of everything installed
# recreate elsewhere:
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

requests==2.32.3      # exact -- reproducible, what freeze writes
flask>=3.0            # floor -- may pull a newer minor, less reproducible

def parse(v):
    return tuple(int(x) for x in v.split("."))

def upgrade_kind(old, new):
    (a1, b1, c1), (a2, b2, c2) = parse(old), parse(new)
    if a2 > a1:            return "major -- may break, read changelog"
    if a2 == a1 and b2 > b1: return "minor -- new features, backward-compatible"
    if (a2, b2) == (a1, b1) and c2 > c1: return "patch -- bug fixes only"
    return "same or downgrade"

print(upgrade_kind("1.4.0", "1.5.2"))   # minor -- backward-compatible
print(upgrade_kind("1.4.0", "2.0.0"))   # major -- may break

def satisfies(version, lower=None, upper=None):
    v = tuple(int(x) for x in version.split("."))
    if lower and v < tuple(int(x) for x in lower.split(".")): return False
    if upper and v >= tuple(int(x) for x in upper.split(".")): return False
    return True

# is there ANY version both accept? check candidate versions
candidates = ["1.26.0", "2.0.0", "2.2.0"]
a_ok = [v for v in candidates if satisfies(v, lower="2.0.0")]     # A: >=2
b_ok = [v for v in candidates if satisfies(v, upper="2.0.0")]     # B: <2
common = set(a_ok) & set(b_ok)
print("compatible versions:", common or "NONE -- conflict")

# pyproject.toml
[build-system]
requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta"

[project]
name = "reportkit"
version = "0.1.0"
dependencies = ["requests>=2.31"]

[project.scripts]
report = "reportkit.cli:main"      # `report` on the CLI calls cli.main()

pip install -e .        # editable install: code changes take effect immediately
report --help           # the entry point now works as a command

import re, sys

def audit(requirements_text):
    floating = []
    for line in requirements_text.splitlines():
        line = line.split("#")[0].strip()
        if not line:
            continue
        # a reproducible line has an exact == pin
        if "==" not in line:
            floating.append(line)
    return floating

reqs = """
requests==2.32.3
flask>=3.0
numpy
pydantic==2.7.1
"""
bad = audit(reqs)
print("unpinned:", bad)          # ['flask>=3.0', 'numpy']
sys.exit(1 if bad else 0)        # non-zero fails the CI gate

Knowledge check check yourself