The command line & automation

1 · Navigation — where am I, what's here essential

The first time you open a terminal, it feels like being dropped into a dark house with no light switch. There's a blinking cursor, no icons, no folders you can see — just a prompt waiting. Almost everyone's instinct is to freeze, because a graphical file browser shows you where you are, and the terminal doesn't. But that missing map is exactly what these three commands hand back to you, and within five minutes the dark house lights up room by room.

Here's the mental model that makes it click: the shell is always standing in exactly one folder, called the working directory. Think of it like a person walking through a building — at any moment they're in one room, they can look around that room, and they can walk through a door into another. They can't be in two rooms at once. Every command you type runs as if you were standing in that folder, which is why "where am I?" is the single most important question on the command line — the answer silently changes what everything else does.

The common beginner mistake is confusing the two kinds of address a file can have. An absolute path (/home/you/x) starts from the very top of the filesystem — it's like a full postal address, unambiguous from anywhere on Earth. A relative path (./x, ../x) is measured from where you're currently standing — it's like saying "next door" or "upstairs", which only makes sense once you know which building you're in. The dot shorthands are worth burning into memory now: . means "here" and .. means "the folder above me". People lose hours debugging a script that "can't find a file", when the real problem is that a relative path pointed somewhere else because the working directory wasn't what they assumed.

Three commands cover most movement: pwd (where), ls (what), cd (go). Paths are either absolute (/home/you/x) or relative (./x, ../x).

nav.shpwd                 # /home/you
ls -la              # long listing, incl. hidden (dot) files
cd projects         # into a dir;  cd ..  up;  cd ~  home;  cd -  previous
tree -L 1           # (if installed) a visual of the current level

2 · Files — create, copy, move, remove essential

In a graphical file manager, every file operation comes with a safety net you never think about. You drag a file to the wrong place, you drag it back. You delete something, it sits in the Trash for weeks until you empty it. You right-click, you get "Undo Move". The command line strips all of that away — and the first time it bites you is unforgettable. There's a well-worn horror story every senior engineer has some version of: a stray space or a wrong variable turned rm -rf $DIR/ into rm -rf /, and a career's worth of files evaporated with no confirmation, no trash, no undo. That's not a reason to fear the terminal; it's a reason to understand exactly what each command does before you press Enter.

The way to hold these commands in your head is as a small vocabulary of verbs, each doing one blunt thing: mkdir makes a folder, touch creates an empty file, cp copies, mv moves or renames, rm removes. There's no "are you sure?" dialog between you and any of them — the shell assumes you meant what you typed. That directness is the trade: total speed and scriptability, zero hand-holding.

The one distinction that trips up almost every beginner is the two arrows: > versus >>. A single > means "point this output into that file" — and if the file already has contents, they are silently obliterated and replaced. A double >> means "add this output onto the end". Think of > as replacing the whole page and >> as writing another line at the bottom. Beginners reach for > out of habit, run it twice on a log file they meant to append to, and watch the first half of their data vanish. Read that difference carefully in the lab below, because it is the single most common way people destroy data they didn't mean to touch.

files.shmkdir -p demo/sub && cd demo
touch a.txt
echo "hello" > a.txt        # write (overwrite)
echo "world" >> a.txt       # append
cat a.txt; wc -l a.txt      # show; count lines
cp a.txt b.txt; mv b.txt sub/c.txt; rm a.txt

3 · Pipes, redirection & search essential

Picture the moment a teammate pastes you a 400,000-line log file and asks "which IP addresses hit us the most yesterday?" In a spreadsheet, that's an afternoon of imports, formulas, and pivot tables. On the command line, it's one line you can type in fifteen seconds — and that line, the first time it works, is when the terminal stops feeling like a chore and starts feeling like a superpower. This section is where the whole chapter turns a corner: not "how do I move around" but "how do I compose an answer out of nothing but small, dumb tools."

Here's the philosophy underneath it, and it's genuinely beautiful once you see it. Unix was built on a bet: instead of one giant program that tries to do everything, make dozens of tiny programs that each do one thing perfectly, then give people a way to snap them together. grep only filters lines. sort only sorts. wc only counts. None of them knows what the others do — and that's the point. The connector that makes them a team is the pipe, the | symbol, which takes whatever the command on its left prints and feeds it straight into the command on its right as if you'd typed it.

The analogy that makes pipes stick is a factory assembly line. Raw material (your file's text) goes in one end; each station along the belt does one small transformation and passes the result to the next; a finished answer rolls out the far end. cat access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head is literally six stations in a row — extract the IP, line them up, count the duplicates, order by count, take the top few. You didn't write a program; you arranged existing tools in a line.

The mistake beginners make is trying to build the whole line at once, then staring at a wrong answer with no idea which stage broke. The fix is a habit: build the pipe one stage at a time, run it, look at the output, then add the next |. Also keep two things separate in your mind — a pipe (|) sends output to another command, while redirection (>, >>, 2>) sends output to a file. They look similar and get confused constantly; the difference is whether the destination is another program or a file on disk.

The Unix superpower: small single-purpose tools chained with pipes (|). grep filters, sort/uniq/wc transform, >/>> redirect to files.

pipes.shgrep "ERROR" app.log | wc -l                 # how many errors?
grep -rn "TODO" src/                          # recursive, with line numbers
cat access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head   # top IPs
ls -la | grep "\.py$" > py_files.txt          # save matches to a file
some_cmd 2> errors.txt                         # redirect stderr separately

4 · File & OS work from Python (pathlib) intermediate

Shell one-liners are wonderful right up until the day they aren't. You write a tidy pipeline of grep and awk on your Mac, it works perfectly, you hand it to a teammate on Windows — and it detonates, because the path separators are backslashes, the tools aren't installed, and the quoting rules are different. Or your clever one-liner grows a third condition and an error case, and suddenly you're fighting shell syntax that was never meant to hold real logic. That's the wall this section gets you over: when file work becomes real automation, you move it into Python, where it's portable, readable, and easy to guard with checks and logging.

The tool for that job is pathlib, and the mental shift it asks of you is small but powerful: stop thinking of a path as a string, and start thinking of it as an object. Old-style code treated /home/you/logs/app.log as text you glued together with + and os.path.join — fragile, easy to get separators wrong, littered with special cases. A Path is a thing that knows it's a location: it can tell you its own name, its size, whether it exists, what's inside it. Think of the difference between having a scrap of paper with an address scribbled on it versus having a GPS pin you can actually navigate to.

The one piece of pathlib that surprises everyone is that it overloads the / operator to join paths: base / "logs" / "app.log" builds a path. The first time you see it, it looks like you're dividing folders, and that's a fair double-take — but it reads left-to-right exactly like the path it produces, and crucially Python inserts the right separator for whatever operating system you're on. That's the portability win in a single character. The common trap here is the opposite habit — reaching for string concatenation or hard-coded / or \\ out of muscle memory — which is exactly the brittleness pathlib exists to kill. The demo below runs entirely inside a throwaway temp directory, so you can execute it without any risk to your real files.

Real automation does file work in Python, portably, with pathlib. Runs top-to-bottom in a temp dir — 100% safe to execute.

pathlib_ops.pyfrom pathlib import Path
import tempfile

base = Path(tempfile.mkdtemp())                # throwaway dir
(base / "logs").mkdir()
f = base / "logs" / "app.log"
f.write_text("INFO ok\nERROR boom\nINFO fine\nERROR bad\n")

lines = f.read_text().splitlines()
errors = [l for l in lines if l.startswith("ERROR")]
print("total lines:", len(lines), "| errors:", len(errors))

for child in (base / "logs").glob("*.log"):    # like `ls *.log`
    print("found:", child.name, child.stat().st_size, "bytes")

total lines: 4 | errors: 2
found: app.log 34 bytes

5 · Driving the shell from Python (subprocess) intermediate

Sooner or later you hit a tool that only exists on the command line. There's no Python library for it — it's a binary someone else wrote: git, docker, ffmpeg, a company deploy script. You need to run it from your Python program, catch what it printed, and decide what to do based on whether it worked. This is the bridge between the two worlds you've been learning, and it's the beating heart of every build tool, deploy script, and CI runner on the planet: they are, at bottom, Python (or similar) orchestrating shell commands and reacting to the results.

The mental model is a manager delegating a task. Your Python code is the manager; subprocess.run hands a job to a worker (the command), then waits for the worker to come back with two things: what they produced (the output, on stdout) and whether they succeeded (the exit code). That exit code is the part beginners forget exists, and it matters enormously: by universal convention 0 means "success" and any non-zero number means "something went wrong." A command can print perfectly normal-looking output and still have failed — the only reliable way to know is to check the code. Code that runs a command and blindly trusts it worked is how a deploy script cheerfully continues past a failed build and ships nothing.

Now the safety issue, and this is the one that turns into a security incident if you get it wrong. There are two ways to hand a command to subprocess: as a list of separate arguments (["rm", name]), or as a single string with shell=True (f"rm {name}"). They look interchangeable. They are not. With the list form, your arguments go straight to the program with no shell in between — a filename containing spaces or semicolons or && is just a weird filename, harmless. With shell=True, a full shell parses your string first, and if any part of it came from a user, an attacker can smuggle in extra commands — a filename like foo; rm -rf ~ becomes two commands, and the second one is a disaster. The rule to internalize now, before the danger callout below spells it out: always pass a list, and never combine shell=True with input you didn't write yourself.

Build tools, deploy scripts, and CI runners all run commands from code and check the result. subprocess.run captures output and the exit code; check it and fail loudly.

subprocess_run.pyimport subprocess

def run(cmd):
    """Run a command (list form!), return stdout, raise on failure."""
    r = subprocess.run(cmd, capture_output=True, text=True)
    if r.returncode != 0:
        raise RuntimeError(f"{' '.join(cmd)} failed ({r.returncode}): {r.stderr.strip()}")
    return r.stdout.strip()

print("python:", run(["python3", "--version"]))
print("files: ", run(["ls"]).split()[:3])       # first 3 entries here

6 · Advanced — a real CLI with argparse advanced

Every useful script eventually escapes your laptop. You write a quick thing that counts lines in a file, a teammate asks for it, and now it needs to take options — which files, print a total or a breakdown, output JSON or text. The tempting move is to reach into sys.argv and start pulling arguments out by position: "the first one is the filename, the second is the mode…" That path leads directly to pain. You'll hand-write help text that drifts out of date, forget to validate types, produce cryptic errors when someone passes the wrong thing, and reinvent — badly — a wheel Python already ships perfectly round. argparse is that wheel.

Think of argparse as the difference between a handwritten note and a proper order form. A note ("bring milk maybe eggs?") is ambiguous and easy to misread. An order form has labelled fields, marks which are required, rejects nonsense before it reaches the kitchen, and comes with its own instructions printed on it. You describe your tool's inputs once — this argument is required, that one is a flag, this one must be an integer — and argparse handles the parsing, the type-checking, the error messages, and a generated --help screen for free. Users of professional tools like pip or git get that polished --help because those tools declared their arguments the same way you're about to.

The deeper lesson in this section isn't argparse itself — it's a design split that will follow you into every serious codebase: separate the logic from the wiring. Notice the code keeps the real work (count_lines) in a plain function that knows nothing about the command line, and lets a thin main() do nothing but translate command-line arguments into calls to that function. Why bother? Because a pure logic function is testable — you can call it directly with a list and check the result, no terminal needed — and reusable — another tool can import it. The beginner mistake is stuffing file-reading, argument-parsing, and printing all into one tangled function; it works once and can never be tested or reused. Keep the brain (logic) separate from the mouth and ears (I/O), and everything downstream gets easier.

Professional command-line tools parse flags, show --help, and return proper exit codes. argparse is the standard way. Keep the logic in a pure function (testable) and let main() wire the CLI.

linecount.pyimport argparse, sys
from pathlib import Path

def count_lines(paths):                          # PURE logic -> unit-testable
    return {p: len(Path(p).read_text().splitlines())
            for p in paths if Path(p).exists()}

def main(argv=None):
    ap = argparse.ArgumentParser(prog="linecount", description="Count lines in files.")
    ap.add_argument("paths", nargs="+")
    ap.add_argument("-t", "--total", action="store_true", help="print grand total only")
    args = ap.parse_args(argv)
    counts = count_lines(args.paths)
    if args.total:
        print(sum(counts.values()))
    else:
        for p, n in counts.items(): print(f"{n:6}  {p}")
    return 0

Path("/tmp/df_cli.txt").write_text("a\nb\nc\n")
main(["/tmp/df_cli.txt"])                         # normally: python linecount.py file ...
print("logic is testable:", count_lines(["/tmp/df_cli.txt"]))

     3  /tmp/df_cli.txt
logic is testable: {'/tmp/df_cli.txt': 3}

7 · Advanced — robust shell scripts advanced

There's a specific, gut-dropping failure mode that has burned nearly every ops engineer once. A backup script runs on a schedule for months. One night a disk fills up, the cd into the backup folder fails — but the script doesn't notice, and the very next line, which was written assuming that cd worked, runs in the wrong directory. By default, bash's reaction to a failed command is a shrug: it prints an error and keeps going to the next line anyway. So the script marches on through steps that no longer make sense, and you discover the damage days later when you actually need the backup that was never made. Robust shell scripting is, more than anything, the discipline of refusing to let that happen.

The reason this catches people is that the shell's defaults are optimized for interactive use, not scripts. When you're typing commands by hand and one fails, you see it, so bash continuing is convenient. But a script has no human watching each line — so "keep going after an error" turns from convenient into dangerous. A good analogy is an assembly line with no quality check between stations: if station three jams and nobody stops the belt, stations four through ten keep bolting parts onto a broken product. You want the belt to halt the instant anything goes wrong, while the mistake is still one step, not ten.

That halt is exactly what the famous incantation set -euo pipefail buys you, and it belongs at the top of essentially every serious bash script. In plain terms: -e stops the script the moment any command fails, -u stops it if you reference a variable you never set (which catches the classic typo bug where $SOURCE becomes an empty string and rm -rf $SOURCE/ becomes rm -rf /), and pipefail makes a pipeline count as failed if any stage in it failed, not just the last one. The mistake here is philosophical, and worth naming: shell is superb glue for wiring tools together, but the moment your logic grows real branching, data structures, or error handling, that's your signal to switch to Python (section 5) — don't try to build a cathedral out of duct tape.

When the job is genuinely shell (glue, ops), write it defensively. set -euo pipefail makes it fail fast instead of limping on with bad state.

backup.sh#!/usr/bin/env bash
set -euo pipefail                           # exit on error / unset var / failed pipe

SRC="${1:?usage: ./backup.sh <folder>}"     # require arg 1, else error
[ -d "$SRC" ] || { echo "not a directory: $SRC" >&2; exit 1; }

STAMP="$(date +%Y%m%d-%H%M%S)"
DEST="backup-${STAMP}.tar.gz"
tar -czf "$DEST" "$SRC"
echo "backed up $SRC -> $DEST ($(du -h "$DEST" | cut -f1))"

8 · Professional — logging & observability in tools professional

Every tool starts its life littered with print statements. You sprinkle in print("got here") and print("value is", x) to see what's happening, it works, you ship it. Then it's 2 a.m., the tool has failed on a server you can't see, someone forwards you a wall of output, and you have no idea which line came from where, when it happened, or whether a given message was a routine note or the actual catastrophe. Every line looks equally important because print gives them all the same flat voice. This section is about the habit that separates a script from a tool: you stop printing and start logging.

The difference is like a scribbled sticky note versus a proper flight recorder. print just throws text at the screen. Logging attaches context to every message automatically: a timestamp (so you can line events up with an outage), a level that says how much this message matters — DEBUG for noise you only want while developing, INFO for normal progress, WARNING for "this is odd but survivable", ERROR for "something broke" — and a name saying which component spoke. The payoff is that in production you can turn the volume up or down without touching the code: run at WARNING and you hear only problems; flip to DEBUG when you're chasing a bug and the same tool narrates every step.

Then there's a subtlety that feels like trivia until the day it saves you: where the logs go. A Unix program has two output channels — stdout (standard out) for its real results, and stderr (standard error) for diagnostics. They look identical on your screen, so beginners assume they're the same thing. They're not, and the distinction is the whole game: if your tool's answer goes to stdout and its log chatter goes to stderr, then someone can pipe your tool into another (yourtool | wc -l) or redirect its output to a file, and get only the clean data — the timestamps and progress messages stay on the terminal where a human can watch them, without contaminating the machine-readable result. That's why the config below deliberately sends logs to stderr. The mistake is logging your diagnostics to stdout and quietly poisoning every pipeline your tool ever feeds.

Real tools log (not print) so ops can trace them: levels, timestamps, and to stderr so stdout stays parseable. This is the same discipline your services use.

logging_tool.pyimport logging, sys

logging.basicConfig(
    level=logging.INFO, stream=sys.stderr,
    format="%(asctime)s %(levelname)s %(message)s", datefmt="%H:%M:%S")
log = logging.getLogger("mytool")

def process(items):
    log.info("starting, %d items", len(items))
    done = 0
    for it in items:
        try:
            done += 1
        except Exception:
            log.exception("failed on %s", it)     # logs the traceback
    log.info("done, processed %d", done)
    return done

print("result:", process(["a", "b", "c"]))

result: 3

9 · Tech-lead — ship the tool to the team tech-lead

Here's how a genuinely useful internal tool usually dies. You build something great. To run it, a colleague has to clone a repo, find the right folder, remember to activate a virtual environment, and type python scripts/tools/linecount.py with the full path every single time. Two people bother. Everyone else keeps doing the task by hand, because the friction of invoking your tool is higher than the pain it removes. The tool isn't bad — it's just not shippable, and a tech-lead's job is precisely to close that last mile so the whole team actually adopts it.

The goal is to make your tool behave like the commands people already trust. When you type git or black or pip, you don't specify where the program lives, don't activate anything, don't think about Python at all — you just type its name and it runs, from any directory. That "just type its name anywhere" magic isn't built into those tools; it's declared, in a package config file, by a single line called an entry point. It's the difference between a dish you cooked once and a recipe printed in a book anyone can follow: packaging is what turns your private script into shared infrastructure.

The mechanism is a pyproject.toml — the standard file that describes a Python package — and specifically its [project.scripts] section, which maps a command name to a function in your code. After a one-time pip install ., that name lands on everyone's PATH and behaves like any built-in command. Two things quietly make this possible, and they're the payoff of everything that came before: the tool must be a proper package (the packaging discipline you'll go deeper on in DF5), and its logic must be cleanly reachable through a main() entry point — which is exactly why section 6 insisted on separating pure logic from the CLI wiring. The common mistake is treating packaging as an afterthought bolted on at the end; in reality, the testable, importable structure you chose early is what makes shipping trivial now.

A lead makes the tool usable by everyone: an entry point so pip install puts it on the PATH, and the logic separated so it's testable (TQ). This is the DF5-packaging + testing payoff.

pyproject.toml[project]
name = "team-tools"
version = "1.0.0"
requires-python = ">=3.10"

[project.scripts]
linecount = "team_tools.cli:main"    # pip install . -> `linecount` on everyone's PATH

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

pwd              # print working directory: where am I?
ls -la           # list all (-a incl. hidden) in long form (-l: perms, size, date)
cd project       # move into ./project
pwd              # now .../project
cd ..            # back up one level
cd -             # jump back to the previous directory

grep ERROR app.log | wc -l        # count ERROR lines (grep filters, wc -l counts)
grep ERROR app.log > errors.txt   # write matching lines to a file (overwrites)
grep ERROR app.log | sort | uniq -c   # bonus: how many of each distinct ERROR line

from pathlib import Path
import tempfile

# set up a temp tree so this is self-contained and runnable
root = Path(tempfile.mkdtemp())
(root / "a.txt").write_text("hello")
(root / "sub").mkdir()
(root / "sub" / "b.txt").write_text("world!!")
(root / "c.log").write_text("ignore me")

for p in sorted(root.rglob("*.txt")):        # recursive glob, only .txt
    print(p.relative_to(root), p.stat().st_size)
# a.txt 5
# sub/b.txt 7

import subprocess

# safe: args as a list, no shell parsing
r = subprocess.run(["echo", "hello from subprocess"],
                   capture_output=True, text=True)
print("stdout:", r.stdout.strip())
print("exit code:", r.returncode)

# handle failure without crashing:
r2 = subprocess.run(["false"], capture_output=True, text=True)  # 'false' exits 1
if r2.returncode != 0:
    print("command failed, code:", r2.returncode)

# DANGEROUS (do not do with untrusted input):
# subprocess.run(f"grep {user_input} file", shell=True)  # shell injection risk

import argparse

def build_parser():
    p = argparse.ArgumentParser(description="Greet someone N times.")
    p.add_argument("--name", required=True, help="who to greet")
    p.add_argument("--count", type=int, default=1, help="how many times")
    return p

def main(argv=None):
    args = build_parser().parse_args(argv)
    for _ in range(args.count):
        print(f"Hello, {args.name}!")

# normally: main()  reads sys.argv. Here we pass args to keep it runnable/testable:
main(["--name", "Sam", "--count", "2"])
# Hello, Sam!
# Hello, Sam!

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
log = logging.getLogger("mytool")

log.info("starting run")
log.warning("input file was empty, using default")
try:
    1 / 0
except ZeroDivisionError:
    log.error("computation failed", exc_info=True)   # logs the traceback
log.info("done")

#!/usr/bin/env bash
set -euo pipefail        # -e: exit on error, -u: error on unset vars,
                         # pipefail: a failing command in a pipe fails the script
IFS=$'\n\t'            # safer word-splitting

Knowledge check check yourself