Advanced Strings, Tokenization & Data Parsing

Computers store text as numbers. A Python str is the human-readable text (a sequence of characters); bytes is the same text turned into the raw numbers that get saved to a file or sent over a network. You encode to go str→bytes and decode to go bytes→str. This block shows why the two are not the same length.

1. s = "café ☕" is a normal string of 6 characters. len(s) counts characters, so it is 6.
2. s.encode("utf-8") converts it to bytes using UTF-8 (the standard encoding). Accented letters and emoji need more than one byte, so the byte count is bigger — len(b) is 9.
3. b.decode("utf-8") turns the bytes back into the original string — a clean round-trip.
4. The last line feeds broken bytes (b"caf\xff") to decode. errors="replace" means "don't crash on bad bytes — put a placeholder instead", so you get 'caf?' rather than a UnicodeDecodeError.

What the output means: print(len(s), len(b)) prints 6 9 — proof that characters and bytes are different counts. print(b) shows the raw bytes with \x.. escapes for the multi-byte characters.

Try this: Change s to plain ASCII like "cafe" and re-run — now len(s) and len(b) match, because every ASCII character is exactly one byte. The mismatch only appears with accents/emoji.

Two strings can look identical on screen yet be stored as different bytes — for example "é" as one character, versus "e" followed by a separate accent mark. Python then says they are not equal. Normalization rewrites both into one canonical form so they compare equal. This matters any time you dedupe, search, or use text as a lookup key.

1. a and b print the same but are built differently, so a == b is False — a classic, invisible bug.
2. unicodedata.normalize("NFC", …) converts both to the same standard form; after that they are equal, so the second print shows True.
3. The clean() function is a reusable tidy-up: NFKC normalizes (also folding compatibility look-alikes), " ".join(text.split()) collapses every run of spaces/tabs/newlines into single spaces, and .strip() trims the ends.

What the output means: clean(" Hello\t\n world ") returns 'Hello world' — messy whitespace becomes one clean line.

Try this: Run clean on text with double spaces and tabs and watch them collapse. Normalizing like this before storing text is how you stop "same content, different bytes" from creating duplicate records.

LLMs don't read letters or whole words — they read tokens, which are common chunks of text from a fixed vocabulary. This is a tiny from-scratch version of Byte-Pair Encoding (BPE), the algorithm that builds that vocabulary: it starts with single characters and repeatedly glues the most frequent neighboring pair into a new token. Don't worry about every line — follow the big loop.

1. Each word starts split into single characters plus a </w> end-of-word marker, e.g. low → ['l','o','w','</w>'].
2. Counter() tallies how often each adjacent pair of symbols appears across all words, weighted by how common each word is.
3. pairs.most_common(1) picks the single most frequent pair — that becomes the next merge. The inner while loop then walks through every word and fuses that pair wherever it occurs (e.g. e+s → es).
4. Repeat num_merges times. The list of merges it returns is the learned vocabulary of subword pieces.

What the output means: bpe_train(words, 5) prints the 5 merges it learned, like ('e','s') then ('es','t') — the algorithm discovered that "est" is a useful chunk because it appears in newest and widest.

Try this: Lower the merge count to 1 to see just the first, most-common pair. This is exactly why token count ≠ word count: common pieces become one token, rare words get split into several.

You pay LLM APIs per token, and each model has a maximum number of tokens it can read at once (its context window). So before sending text you often need to know how many tokens it is. The commented lines show the real way; the small function is a rough backup.

1. The commented block at the top is the accurate method: Anthropic's SDK has client.messages.count_tokens(...) which asks the server for the exact count for a given model — the source of truth for Claude.
2. For other model families, the commented tiktoken lines count tokens locally on your machine.
3. rough_tokens(text) is a quick offline guess: English prose averages about 4 characters per token, so it divides the character length by 4. max(1, …) makes sure even a tiny string counts as at least 1 token.

What the output means: rough_tokens("The quick brown fox jumps.") returns about 6 — a ballpark, not an exact bill.

Try this: Never trust the char/4 estimate for real billing — it's only a sanity check. For code, other languages, or long numbers it can be way off, which is why you measure with the actual tokenizer when money or limits are on the line.

A regular expression (regex) is a mini-pattern language for finding and pulling structured pieces out of text — great for log lines, IDs, and prices. This block shows three power features. The r"..." prefix means a raw string so backslashes are taken literally.

1. Named groups (?P<name>...) label each captured piece. After matching the log line you read fields by name — m.group("level") gives ERROR — instead of counting anonymous positions.
2. Lookbehind (?<=\$) means "only match here if a $ comes right before" — but the $ itself is not included in the result. So [\d,]+ grabs 1,299 from $1,299.00.
3. re.compile(...) builds the pattern once and reuses it, which is faster inside loops. CITATION.findall(...) returns every match as a list, here ['1', '12'].

What the output means: You get the named fields (ERROR checkout-api), the price digits (['1,299']), and all citation numbers (['1', '12']).

Try this: Regex is powerful but the wrong tool for JSON/CSV/HTML — those have real parsers (next section). Also avoid patterns like (a+)+, which can hang on some inputs (a real denial-of-service risk called ReDoS).

For RAG (retrieval), you split a long document into smaller chunks to store and search. Chunks that are too big waste tokens; too small and they lose context. This function splits on sentence boundaries and keeps a little overlap so an idea that straddles a boundary isn't cut in half.

1. re.split(r"(?<=[.!?])\s+", …) breaks the text into sentences by splitting on the space that follows a ., !, or ?.
2. The loop keeps adding sentences to the current chunk cur while it stays under max_chars.
3. When the next sentence would overflow, it saves the current chunk and starts a new one — but seeds it with cur[-overlap:], the tail of the previous chunk. That repeated overlap is the safety net for ideas spanning a boundary.
4. At the end it appends whatever is left in cur and returns the list of chunks.

What the output means: print(len(cs), …) shows how many chunks the repeated sentence produced and the first 60 characters of chunk one — you can see sentences kept whole and the overlap carried forward.

Try this: Shrink max_chars to see more, smaller chunks; raise overlap to carry more context between them. Real systems chunk on structure (headings, code blocks) and measure size in tokens, not characters.

Real data arrives in messy formats. This block shows the safe way to handle three: JSON that a model wrapped in chatter, CSV with awkward commas, and JSONL. The golden rule is use the real parser (json, csv) rather than splitting strings by hand.

1. extract_json(text) rescues JSON from an LLM reply. Models often wrap it in a ```json code fence or add prose, so the function strips the fence, finds the first { and last }, slices out that outermost object, and finally calls json.loads to turn the text into a real Python dict.
2. For CSV, note the raw line has a comma inside a quoted field ("Doe, John"). csv.DictReader understands quoting and gives you each row as a dictionary — splitting on commas yourself would break here.
3. read_jsonl handles JSONL (one JSON object per line, the standard for datasets/logs): it loops over lines and parses each non-empty one.

What the output means: The CSV loop prints Doe, John | says "hi" — the embedded comma and escaped quotes were handled correctly by the parser.

Try this: Even better than parsing prose: ask the model for structured output / tool use (covered in A6) so the JSON arrives clean and you never scrape it out of text.

When data is a table (rows and columns — like eval results or logs), pandas is the standard tool. A DataFrame is that table in memory; you can filter, group, and summarize it with one-liners instead of writing loops.

1. pd.DataFrame([...]) builds the table from a list of dictionaries — each dict is one row, each key is a column (case, model, passed, latency).
2. df["passed"].mean() averages the True/False column (True counts as 1), giving the overall pass rate, 0.667.
3. df.groupby("model")["passed"].mean() splits rows by model first, then averages within each group — the pass rate per model.
4. df[df["latency"] > 1.0] keeps only rows where latency exceeds 1.0 (a boolean mask). .describe() prints summary statistics (count, mean, min, max, quartiles).

What the output means: You get the overall pass rate (0.667), a per-model breakdown, and a statistics summary of the latency column.

Try this: Change the > 1.0 threshold to filter differently, or group by a different column. This is exactly how you'd summarize eval runs or cost logs in the course projects.

This is a classic interview question: top-p (nucleus) sampling is how an LLM picks the next token. Instead of always taking the single most likely token, it keeps just enough of the top choices to cover a probability mass p (say 90%), then samples from that shortlist — giving controlled randomness.

1. np.argsort(-probs) sorts the token indices from most to least likely (the minus flips ascending into descending).
2. np.cumsum(...) makes a running total of those sorted probabilities. np.searchsorted(cum, p) finds how many of the top tokens are needed to reach the mass p; + 1 includes the one that crosses the line.
3. keep = idx[:cutoff] is that shortlist. The final line renormalizes their probabilities so they sum to 1 (probs[keep]/probs[keep].sum()) and randomly picks one, returning its index.

What the output means: It returns the integer index of the chosen token — one of the high-probability candidates, chosen at random in proportion to its likelihood.

Try this: A smaller p (e.g. 0.5) keeps fewer candidates → safer, more repetitive output; a larger p keeps more → more varied, riskier. This is the same "top_p" knob you set on real API calls.

The other common interview task for this topic: pull clean JSON out of an LLM reply. Models like to add explanations or fence the JSON in ``` blocks, so you can't just call json.loads on the raw text — you first trim it down to the JSON object.

1. text.strip() removes surrounding whitespace. If the reply starts with a ``` fence, it splits on the fence, takes the code portion, and drops a leading json label with removeprefix.
2. text.find("{") and text.rfind("}") locate the first opening and last closing brace — the outermost JSON object, ignoring any prose around it.
3. json.loads(text[start:end+1]) parses that slice into a real Python dict.

What the output means: You get a usable Python dictionary regardless of whether the model wrapped its answer in prose or code fences.

Try this: Feed it a string like 'Sure!\n```json\n{"ok": true}\n```' and confirm it returns {'ok': True}. In production, prefer structured output (A6) so this cleanup is rarely needed.