Advanced Strings, Tokenization & Data Parsing

Why string mechanics bite in production motivation

Here's the uncomfortable truth about LLM systems: the model is almost never the part that breaks. Your prompt engineering is fine, your retrieval is tuned, your evals pass — and then a user in Munich types their own name, "Müller", and your dedup logic files them as a brand-new customer because the "ü" arrived in a byte form your database had never seen. Nothing crashed. No error surfaced. You just quietly created a duplicate, and you won't find out until someone's support history is split across two accounts. That's the shape of a text bug: silent, plausible-looking, and discovered by a user before it's discovered by you.

The reason this whole layer is treacherous is that text pretends to be simple. A string looks like "just letters," the way a map looks like "just the ground." But underneath, a single line of text is really three different things at once — the characters a human reads, the bytes a disk stores, and the tokens a model actually consumes — and they almost never line up. Most bugs in this chapter come from an engineer reaching for the wrong one of those three: measuring token cost with len(str), comparing names without normalizing, splitting a CSV on commas. Each mistake feels reasonable in the moment and is invisible until scale or a foreign character exposes it.

The common mistake worth naming up front is trusting your eyes. If two strings look identical on screen, you assume they're equal; if a document looks like plain paragraphs, you assume you can split it anywhere; if English averages four characters a token, you assume the estimate holds for a customer ID or a Japanese sentence. Every one of those assumptions is false in a way that won't announce itself. This part exists to replace "looks fine" with "I measured it" — because getting text right is unglamorous, rarely praised, and the difference between a system that works on your laptop and one that survives real users.

An accented name that won't match, a CSV that breaks on a quoted comma, a prompt that silently blew the context window, a chunk that split a sentence mid-word and wrecked retrieval — every one of these is a text-handling bug, and they're among the most common in real AI systems. Getting text right is unglamorous and high-leverage.

1 · Unicode & encodings — str vs bytes intermediate

Picture the last time you opened a file and saw caf� or a wall of é where an accent should be. That garbled text isn't corruption — it's a translation error. Somewhere, text that was written down in one alphabet-of-numbers got read back using a different one, and the two didn't agree. Understanding str versus bytes is understanding exactly where that translation happens, so you can stop it going wrong.

The cleanest analogy is ideas versus ink. A str is the idea of the text: the sequence of abstract characters "c, a, f, é" — what a human means, independent of how it's written down. bytes is the ink: the actual physical marks (numbers 0–255) you commit to disk or send down a wire. To get from idea to ink you must pick a writing system — that's what an encoding is. Encoding (str→bytes) is writing the idea down; decoding (bytes→str) is reading it back. And here's the catch that causes every garble: the reader has to use the same writing system the writer used. If you write with UTF-8 and someone reads with Latin-1, the ink is fine but the interpretation is wrong — hence é.

Why does this bite so hard in AI work specifically? Because your text comes from everywhere — scraped web pages, user uploads, PDFs, other people's APIs — and each source may have used a different writing system. UTF-8 is the modern standard (it can write every character Unicode defines, and ASCII is a subset of it), so "always assume and specify UTF-8" is the single habit that prevents most of this pain. The common mistake is letting the encoding default: calling open(path) with no encoding= argument means Python guesses based on the operating system, so the same code reads correctly on your Mac and mangles text on a Windows server — a bug that only appears in production. Name the encoding explicitly, every time, and the guessing stops. Now, the precise definitions.

A Python str is a sequence of Unicode code points (abstract characters). bytes is raw encoded data. You encode str→bytes to send/store, and decode bytes→str to read. UTF-8 is the standard encoding. Confusing the two is the source of endless UnicodeDecodeErrors.

pythons = "café ☕"              # a str: 6 code points
b = s.encode("utf-8")         # bytes: 'é' and '☕' take multiple bytes
print(len(s), len(b))          # 6 9  — chars != bytes!
print(b)                       # b'caf\xc3\xa9 \xe2\x98\x95'
print(b.decode("utf-8"))       # 'café ☕' back to str

# Reading files: ALWAYS specify encoding, or you get platform-dependent bugs
# with open("doc.txt", encoding="utf-8") as f: text = f.read()

# Handling bad bytes gracefully instead of crashing:
messy = b"caf\xff"
print(messy.decode("utf-8", errors="replace"))   # 'caf?'

2 · Normalization & cleaning advanced

This is the bug from the intro, and it's worth meeting head-on because it will fool you. Type "café" one way and "café" another way, print them side by side, and they are pixel-for-pixel identical. Then ask Python if they're equal and it says False. You'll stare at the screen convinced the language is broken. It isn't — you've hit the single sneakiest fact about Unicode: the same visible character can be built more than one way, and the different constructions are genuinely different data even though they render the same.

The analogy is spelling versus handwriting. The letter "é" can be written as one pre-composed symbol (the code point U+00E9, "e-with-acute"), or as two pieces stacked — a plain "e" followed by a separate "combining acute accent" that draws itself on top of the previous letter. Your screen renders both as a single "é" because that's what they mean to a reader. But to a computer comparing raw code points, one is a single item and the other is a list of two, so they can't be equal — the way "color" and "colour" mean the same thing but are not the same string. Unicode normalization is the fix: it rewrites text into one canonical spelling so that things which mean the same actually are the same. NFC composes pieces back into single characters; NFKC goes further and also folds "compatibility" look-alikes (a full-width "Ａ", a ligature "fi") down to their plain forms.

Why does an AI engineer care so much about a two-vs-one accent? Because so much of what we do is using text as a key: deduplicating documents, caching by prompt, matching a user's name, joining retrieved chunks. Every one of those silently assumes "equal-looking means equal." The mistake — and it's a data-quality landmine, not a crash — is keying on raw, un-normalized text: your cache misses on a query it has already answered, your dedup keeps two copies of the same paragraph, your user gets a second account. None of it throws an error; it just quietly corrupts your data at scale, and the entry point (macOS filenames, copy-paste from Word, certain keyboards) produces decomposed forms all the time. Normalize before you compare, hash, or store, and the whole class of bug disappears. The precise version follows.

The same text can have multiple byte representations — "é" as one code point or "e" + a combining accent. They look identical but compare unequal. Unicode normalization (NFC/NFKC) canonicalizes them, essential before dedup, search, or using text as a key (D3).

pythonimport unicodedata

a = "café"                          # 'é' = single code point U+00E9
b = "café"                    # 'e' + combining acute U+0301
print(a == b)                        # False! (look identical, differ in bytes)
print(unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b))  # True

def clean(text):
    text = unicodedata.normalize("NFKC", text)   # canonical + compatibility
    text = " ".join(text.split())              # collapse whitespace/newlines/tabs
    return text.strip()

print(clean("  Hello\t\n  world  "))         # 'Hello world'

3 · Tokenization & BPE — how LLMs actually see text expert intermediate

Every time you've seen an LLM confidently miscount the letters in "strawberry," you've watched this section's idea in action. The model doesn't fumble because it's dumb — it fumbles because it never sees the letters. It sees "strawberry" as maybe two or three opaque chunks, the way you might read the word "the" as a single shape without ever registering the individual t-h-e. If you don't know how text is chopped up before the model touches it, half of what LLMs do — and half of what they cost — stays mysterious. This is the section that lifts that veil.

Here's the mental model. Imagine you had to design a fixed dictionary of, say, 100,000 "pieces" and promise that any text on the internet could be spelled using only those pieces. You'd be silly to fill it with whole words (you'd run out of slots and still miss every typo and proper noun) and silly to use only single letters (then every word is a long, wasteful sequence). The clever middle ground is subword pieces: give common words their own single piece ("the", " and", "python"), and let rare words fall back to smaller fragments ("tokenization" → "token" + "ization"). These pieces are called tokens, and how you build that dictionary is the tokenizer's whole job. The dominant recipe is Byte-Pair Encoding (BPE), and its logic is almost embarrassingly simple: start with individual characters, find the most frequent adjacent pair across all your text, glue that pair into a new piece, and repeat thousands of times. Frequency does the work — the pairs that show up constantly (like "e"+"s", "t"+"h") get merged early and become the dense, cheap tokens; the rare stuff never gets merged and stays fragmented.

The consequence you must internalize is that tokens, words, and characters are three different currencies with no fixed exchange rate. Common English prose is dense — roughly a token every four characters — because the frequent pieces earned their own slots. But a UUID, a chunk of minified code, a long number, or a sentence of Thai was never frequent in the training text, so it shatters into many tokens. The classic mistake is reasoning about the model in words: "my prompt is 500 words, that's fine" tells you almost nothing about whether it fits the context window or what it'll cost, because a 500-word English paragraph and a 500-"word" config dump can differ several-fold in tokens. Tokens are the unit the model, the context limit, and your bill are all denominated in — so tokens are the unit you must learn to think in. The from-scratch trainer below makes the merging concrete.

LLMs don't read characters or words — they read tokens: subword units from a fixed vocabulary. Modern tokenizers use Byte-Pair Encoding (BPE): start from bytes/characters and repeatedly merge the most frequent adjacent pair into a new token, learning a vocabulary where common words are one token and rare words split into pieces. This is why token count ≠ word count, and why "strawberry" might be 2–3 tokens.

pythonfrom collections import Counter

def bpe_train(words, num_merges):
    # represent each word as a list of chars + end marker
    vocab = {w: list(w) + ["</w>"] for w in words}
    merges = []
    for _ in range(num_merges):
        pairs = Counter()
        for sym, freq in [(vocab[w], words[w]) for w in vocab]:
            for i in range(len(sym) - 1):
                pairs[(sym[i], sym[i+1])] += freq   # count adjacent pairs
        if not pairs: break
        best = pairs.most_common(1)[0][0]        # most frequent pair
        merges.append(best)
        for w in vocab:                          # merge it everywhere
            s, out = vocab[w], []
            i = 0
            while i < len(s):
                if i < len(s)-1 and (s[i], s[i+1]) == best:
                    out.append(s[i] + s[i+1]); i += 2
                else:
                    out.append(s[i]); i += 1
            vocab[w] = out
    return merges

words = {"low": 5, "lower": 2, "newest": 6, "widest": 3}
print(bpe_train(words, 5))     # learns merges like ('e','s'), ('es','t'), ('l','o')...

4 · Counting tokens for budget & cost advanced

Two things in an LLM app are metered by the token, and getting the count wrong breaks both. The first is your bill — you pay per token in and per token out, so an off-by-40% estimate is an off-by-40% cost forecast. The second is the hard wall of the context window: every model can only read so many tokens at once, and the request that sails through in testing with three retrieved chunks will throw an error (or worse, silently truncate the end of your prompt) the day a user's query pulls in eight. Counting tokens isn't bookkeeping pedantry; it's how you keep an app from either surprising you on the invoice or falling over under a heavier-than-average request.

The trap here follows directly from the last section. Because English runs about four characters to a token, it is so tempting to write len(text) // 4 and call it a token count. As a back-of-envelope sanity check that's fine — but the moment you rely on it for anything real, it lies. Feed it code, a table of numbers, JSON, or any non-English language and the four-chars-per-token rule collapses; those inputs fragment into far more tokens than the heuristic predicts, so your "safe" prompt blows the window and your cost estimate is fiction. The rule of thumb is only calibrated on the one kind of text (ordinary English prose) that behaves.

The honest answer is to ask the real tokenizer, and the mistake to avoid is treating all tokenizers as interchangeable — they aren't. Each model family was trained with its own vocabulary and merge rules, so the same string counts differently across them; a number that's known exactly for Claude means using Claude's tokenizer, not OpenAI's tiktoken and not a heuristic. For Anthropic models the source of truth is the count_tokens endpoint, which returns the server's exact count for the model you name — no guessing. Reserve the char/4 estimate for cheap local guardrails ("is this obviously too big?") and reach for the provider's counter whenever money or the context limit is actually on the line. The code contrasts both.

You must know a request's token count to stay under context limits and estimate cost. Use the provider's tokenizer/counting rather than guessing from characters.

python# The Anthropic SDK provides server-side token counting:
#   from anthropic import Anthropic
#   client = Anthropic()
#   n = client.messages.count_tokens(
#           model="claude-opus-4-8",
#           messages=[{"role": "user", "content": text}],
#       ).input_tokens
#
# For OpenAI-family / local models, tiktoken does it locally:
#   import tiktoken; enc = tiktoken.get_encoding("cl100k_base")
#   n = len(enc.encode(text))

# A rough offline estimate ONLY for sanity checks (never for billing):
def rough_tokens(text):
    return max(1, len(text) // 4)      # ~4 chars/token for English prose

print(rough_tokens("The quick brown fox jumps."))   # ~6

5 · Advanced regex — powerful, use with restraint advanced

Regex is the chainsaw of text processing: astonishing when the job fits, dangerous when it doesn't. Reach for it to rip a timestamp, an error level, and a service name out of a million log lines and it'll do in one expression what a hand-written loop couldn't touch. Point it at nested, quoted, recursive structure — JSON, HTML, a CSV with commas inside fields — and it will turn on you: fragile, unreadable, and occasionally able to hang your process outright. The skill this section teaches is half syntax and half judgment about which of those two situations you're in.

The way to hold it in your head: a regular expression is a tiny pattern language for describing the shape of text so the engine can find every place that shape occurs. The beginner tools (P2) let you say "some digits, then a dash, then more digits." The advanced tools let you do two extra things that matter a lot in practice. Named capture groups — (?P<level>\w+) — let you pull out labelled pieces and read them back by name instead of counting anonymous positions, which is the difference between self-documenting extraction and a mess of m.group(3) calls nobody can maintain. Lookarounds — lookahead (?=...) and lookbehind (?<=...) — let you match a spot because of what surrounds it without swallowing that context: "grab the digits that come right after a $, but leave the $ out of the result." They assert a condition at a position rather than consuming characters.

Two mistakes are worth burning into memory. The first is using regex on formats that have real parsers. HTML, JSON, and CSV are recursive and quote-aware; a regex sees flat text and will get quoted commas, escaped quotes, and nesting wrong in ways that pass every test you write and fail on the input you didn't imagine. Use json, csv, an HTML library — that's literally the next section. The second, subtler danger is catastrophic backtracking (ReDoS): a seemingly innocent pattern with nested quantifiers like (a+)+ can, on a maliciously crafted string, force the engine to try exponentially many ways to match and effectively freeze — a real denial-of-service vector when the input comes from users. Keep patterns simple, anchored, and free of nested "one-or-more of one-or-more," and regex stays the sharp, safe tool it should be. The lab shows the good features and flags the traps.

Regex is the right tool for structured pattern extraction (log lines, IDs, citations). Beyond the basics (P2), the advanced features are groups, named captures, and lookarounds — and knowing when a real parser beats a regex.

pythonimport re

# Named capture groups -> readable, self-documenting extraction
log = "2026-09-02 14:03:11 ERROR checkout-api: timeout after 30s"
m = re.match(r"(?P<date>\S+) (?P<time>\S+) (?P<level>\w+) (?P<svc>[\w-]+): (?P<msg>.*)", log)
print(m.group("level"), m.group("svc"))     # ERROR checkout-api

# Lookahead (?=...) / lookbehind (?<=...): match position, don't consume
price = "$1,299.00"
digits = re.findall(r"(?<=\$)[\d,]+", price)   # ['1,299'] — the part after $

# Compile once if reused in a loop (perf)
CITATION = re.compile(r"\[(\d+)\]")
print(CITATION.findall("See [1] and [12]."))   # ['1', '12']

6 · Chunking text for RAG expert advanced

Ask anyone who has debugged a disappointing RAG system and they'll eventually tell you: the retrieval was fine, the chunking was the problem. It's the least glamorous step in the pipeline and the one that quietly caps how good everything downstream can be. If you cut your documents into the wrong pieces before you ever embed them, no amount of clever similarity search or re-ranking can recover the answer — because the answer, as a coherent unit, no longer exists in your index. Chunking is where retrieval quality is won or lost, long before a query arrives.

Think of chunking like cutting a long document into index cards, where each card has to make sense on its own because that's how it'll be pulled and shown to the model. That framing exposes the tension immediately. Make the cards too big — cram three topics onto one — and a search for topic B drags along the noise of A and C, diluting the relevance signal and eating your token budget when the chunk is retrieved. Make them too small — a single sentence per card — and you shred context: "It raised revenue 40%" is useless on its own if "the pricing change" that it refers to landed on a different card. Good chunking splits along the seams the text already has — paragraph and sentence boundaries — so each piece is a self-contained thought rather than an arbitrary slice through the middle of one.

The move that separates a toy chunker from a real one is overlap: letting the end of one chunk repeat at the start of the next. This is the same sliding-window instinct from the arrays track (D1), applied to prose, and it's insurance against the boundary problem — a fact that straddles two chunks survives intact in at least one of them instead of being cleanly severed from its context. Two mistakes to avoid: chunking with zero overlap (which guarantees some ideas get guillotined at a boundary) and measuring chunk size in characters when the real constraint is tokens (§4) — a chunk that's "800 characters" can be wildly different token counts depending on whether it's English or code, so production chunkers count tokens and split on real structure (headings, code fences, table rows), never blindly at a character offset. The lab builds a sentence-aware chunker with overlap so you can see both ideas working together.

Retrieval quality starts with chunking: too big and you dilute relevance and blow token budgets; too small and you lose context. Good chunking respects structure (paragraphs/sentences) and uses overlap so ideas spanning a boundary aren't lost.

pythonimport re

def chunk_text(text, max_chars=800, overlap=150):
    # split into sentences (simple heuristic; use nltk/spacy for hard cases)
    sentences = re.split(r"(?<=[.!?])\s+", text.strip())
    chunks, cur = [], ""
    for s in sentences:
        if len(cur) + len(s) <= max_chars:
            cur = (cur + " " + s).strip()
        else:
            if cur: chunks.append(cur)
            # start next chunk with the tail of the previous (overlap)
            cur = (cur[-overlap:] + " " + s).strip() if cur else s
    if cur: chunks.append(cur)
    return chunks

doc = "RAG grounds answers. It retrieves chunks. Then it cites them. " * 10
cs = chunk_text(doc, max_chars=120, overlap=30)
print(len(cs), "chunks; first:", cs[0][:60])

7 · Parsing data formats robustly advanced

The demo always works. Production is where the data fights back. You test your CSV loader on a clean file and it's flawless; then a real export arrives with a company name like "Doe, John" — a comma inside a field — and your naive line.split(",") silently shifts every column one to the right for that row. You ask an LLM for JSON and get pristine JSON in the notebook; then in production the model wraps it in a friendly "Sure! Here's your data:" and a ```json fence, and json.loads throws on the very first character. Robust parsing is the discipline of writing for the messy input you'll actually receive, not the tidy one you tested with.

The single most important principle here is use the real parser, never hand-roll one. Formats like JSON and CSV look simple enough that splitting strings by hand feels reasonable — and that instinct is exactly the mistake. These formats have rules you'll forget: CSV allows quoted fields, commas and even newlines inside quotes, and doubled quotes ("") to mean a literal quote character; JSON has nested structure, escaping, and Unicode. Python's csv and json modules already implement every one of those rules correctly, so csv.DictReader reads "Doe, John" as one field without you thinking about it. Reaching for str.split on structured data is re-writing a parser badly — it's the same lesson as "don't parse HTML with regex," and it's why the previous section handed off to this one.

Two more habits make parsing production-grade. First, expect LLM output to be dirty and defend against it: models editorialize and fence their answers, so a helper that strips prose and slices out the outermost { … } before calling json.loads turns a fragile call into a reliable one — though the truly robust move is to ask for structured output or tool use (covered in A6) so clean JSON arrives by contract and you never scrape it out of chatter at all. Second, know your line-oriented cousin JSONL (one JSON object per line), the standard for datasets and logs: you parse it line by line, which has the happy property that one malformed record can be skipped instead of killing the entire batch. The lab shows all three — rescued JSON, quote-aware CSV, and JSONL — done the safe way.

pythonimport json, csv, io

# 1. Robust JSON: LLMs sometimes wrap JSON in prose or ```json fences
def extract_json(text):
    text = text.strip()
    if text.startswith("```"):
        text = text.split("```")[1].removeprefix("json").strip()
    start = text.find("{")
    if start >= 0:
        text = text[start:text.rfind("}") + 1]     # slice the outermost object
    return json.loads(text)
# (Better: ask for structured output / tool use so you never parse prose — see A6.)

# 2. CSV — never split on commas! The module handles quoting/embedded commas.
raw = 'name,note\n"Doe, John","says ""hi"""\n'
for row in csv.DictReader(io.StringIO(raw)):
    print(row["name"], "|", row["note"])     # Doe, John | says "hi"

# 3. JSONL — one JSON object per line (the standard for datasets/logs)
def read_jsonl(text):
    return [json.loads(line) for line in text.splitlines() if line.strip()]

8 · Tabular data with pandas intermediate → advanced

The moment your text becomes a table, stop writing loops. Eval results, request logs, token-usage records, a text-to-SQL result set — all of it lands as rows and columns, and the reflex to iterate over it with a for loop and a running dictionary is both slower and far more bug-prone than the tool built for exactly this shape. That tool is pandas, and learning to think in it is what turns "let me write forty lines to compute the pass rate per model" into a single, readable line.

The core object is the DataFrame — picture a spreadsheet living in memory: named columns, indexed rows, and a whole vocabulary of operations that act on entire columns at once instead of one cell at a time. The reason this isn't just a nicer syntax is the same reason NumPy (A4) is fast: pandas is vectorized, so an operation like "average the latency column" runs in optimized C over the whole column rather than in a Python loop over each value. The three verbs you'll use constantly map to questions you already ask of data: filter ("show me only the slow requests") via a boolean mask, group ("break this down per model") via groupby, and aggregate ("give me the mean, the count, the quartiles") via functions like .mean() and .describe(). Chain those three and most day-to-day data questions answer themselves.

The mistake to unlearn is the loop-and-accumulate habit carried over from general Python: manually walking rows, maintaining counters, building result dicts by hand. It works on ten rows and quietly becomes both a performance problem and a correctness problem as the data grows — off-by-one filters, forgotten edge cases, mutable state you have to reason about. A boolean mask (df[df["latency"] > 1.0]) is the same idea you met with NumPy arrays, and groupby(...).mean() expresses "split, compute, recombine" declaratively so there's simply less code to get wrong. For the eval summaries and cost analyses in this course, that clarity is the whole point. The lab analyzes a small eval-results table with exactly these operations.

When data is tabular (eval results, logs, the data-analyst project), pandas is the tool: load, filter, group, aggregate — vectorized on top of numpy (A4).

pythonimport pandas as pd

df = pd.DataFrame([
    {"case": "q1", "model": "opus", "passed": True,  "latency": 1.2},
    {"case": "q2", "model": "opus", "passed": False, "latency": 2.1},
    {"case": "q3", "model": "haiku", "passed": True, "latency": 0.4},
])
print(df["passed"].mean())              # overall pass rate: 0.667
print(df.groupby("model")["passed"].mean())   # pass rate per model
slow = df[df["latency"] > 1.0]             # boolean mask (like numpy)
print(df["latency"].describe())          # count/mean/std/min/max/quartiles

Exercises expert

🎯 Interview practice interview

The interview questions this topic gets asked — worked, with code. For the full pattern catalog see A9 · Big Tech AI-engineering patterns.

pythonimport numpy as np
def top_p_sample(probs, p=0.9):
    idx = np.argsort(-probs)
    cum = np.cumsum(probs[idx])
    cutoff = np.searchsorted(cum, p) + 1
    keep = idx[:cutoff]
    return int(np.random.choice(keep, p=probs[keep]/probs[keep].sum()))

pythonimport json
def extract_json(text):
    text = text.strip()
    if text.startswith("```"):
        text = text.split("```")[1].removeprefix("json").strip()
    start, end = text.find("{"), text.rfind("}")
    return json.loads(text[start:end+1])

Checkpoint expert

s = "caf\u00e9"          # 'café', 4 characters
b = s.encode("utf-8")
print(len(s), "chars")     # 4
print(len(b), "bytes")     # 5  (é is 2 bytes in UTF-8)
print(b)                    # b'caf\xc3\xa9'
print(b.decode("utf-8") == s)  # True

import unicodedata

composed = "\u00e9"          # é as one code point
decomposed = "e\u0301"       # e + combining acute
print(composed == decomposed)                 # False!
print(len(composed), len(decomposed))         # 1 2

a = unicodedata.normalize("NFC", composed)
b = unicodedata.normalize("NFC", decomposed)
print(a == b)                                 # True

import unicodedata, re

def clean(text):
    text = unicodedata.normalize("NFC", text)
    # drop control chars (category starting with 'C') except newline/tab
    text = "".join(c for c in text
                   if c in "\n\t" or not unicodedata.category(c).startswith("C"))
    text = re.sub(r"[ \t]+", " ", text)      # collapse spaces/tabs
    text = re.sub(r"\n{3,}", "\n\n", text)    # cap blank lines
    return text.strip()

dirty = "Hello\u200b   world\x00 !\n\n\n\nDone"
print(repr(clean(dirty)))   # 'Hello world !\n\nDone'

def apply_merge(symbols, pair, merged):
    out, i = [], 0
    while i < len(symbols):
        if i < len(symbols)-1 and (symbols[i], symbols[i+1]) == pair:
            out.append(merged)
            i += 2
        else:
            out.append(symbols[i])
            i += 1
    return out

word = list("lower")                    # ['l','o','w','e','r']
word = apply_merge(word, ("l","o"), "lo")
word = apply_merge(word, ("e","r"), "er")
print(word)                              # ['lo', 'w', 'er']

def chunk(text, max_tokens=20, overlap_tokens=5):
    words = text.split()
    est = lambda n: int(n * 1.3)              # ~1.3 tokens/word proxy
    step = max(1, max_tokens - overlap_tokens)
    # convert token budget to a word budget
    max_words = int(max_tokens / 1.3)
    ov_words  = int(overlap_tokens / 1.3)
    chunks, i = [], 0
    while i < len(words):
        piece = words[i:i+max_words]
        chunks.append(" ".join(piece))
        i += max_words - ov_words
    return chunks

text = " ".join(f"w{i}" for i in range(40))
cs = chunk(text)
print(len(cs), "chunks; overlap visible:")
for c in cs:
    print("  ", c[:30], "...")

import json

def parse_logs(lines):
    good, bad = [], 0
    for ln in lines:
        ln = ln.strip()
        if not ln:
            continue
        try:
            rec = json.loads(ln)
        except json.JSONDecodeError:
            bad += 1
            continue
        # defensive coercion: latency may arrive as str or missing
        try:
            rec["latency_ms"] = float(rec.get("latency_ms", 0))
        except (TypeError, ValueError):
            rec["latency_ms"] = 0.0
        good.append(rec)
    total = len(good) + bad
    rate = good and round(len(good)/total, 3)
    return good, {"ok": len(good), "bad": bad, "success_rate": rate}

lines = [
    '{"path": "/a", "latency_ms": "12.5"}',
    'NOT JSON',
    '{"path": "/b", "latency_ms": 30}',
    '{"path": "/c"}',            # missing field -> coerced to 0.0
]
recs, stats = parse_logs(lines)
print(stats)                      # {'ok': 3, 'bad': 1, 'success_rate': 0.75}
print([r["latency_ms"] for r in recs])   # [12.5, 30.0, 0.0]

Knowledge check check yourself