Programs must meet the outside world

Earlier lectures used user input and introduced file operations and exceptions. This lecture develops a complete pipeline for external data, from reproducible inputs to validated results and diagnostics.

Three related things belong together here:

• randomness: programs need uncertainty (simulation, sampling, games)
• text encoding: external data is bytes, and has to become characters
• files and regular expressions: getting data in, and pulling out the parts you want

Their shared requirement is an explicit input contract and clear failure boundaries. Files go missing, encodings are wrong, formats are irregular. So L11’s exception handling appears repeatedly below.

---

1. Randomness

The basics

import random

print(random.randint(1, 6))          # 1 to 6, both ends included
print(random.randrange(0, 10, 2))    # one element of range(0,10,2)
print(random.random())               # a float in [0.0, 1.0)
print(random.uniform(1.5, 3.5))      # the upper endpoint depends on floating-point rounding

Choosing from a sequence

import random

deck = ["SA", "HK", "DQ", "CJ", "S10"]

print(random.choice(deck))                 # one at random
print(random.sample(deck, 3))              # 3 without replacement
print(random.choices(deck, k=3))           # 3 with replacement, may repeat

shuffled = deck.copy()
random.shuffle(shuffled)                   # shuffles in place
print(shuffled)
print(deck)                                # the original is untouched

choices also takes weights:

import random

random.seed(42)
outcomes = random.choices(["win", "draw", "lose"], weights=[1, 2, 7], k=20)
print(outcomes)
print("losses:", outcomes.count("lose"))

seed: making randomness reproducible

This section matters more than everything above it.

Computer “random” numbers are pseudorandom — produced by a deterministic algorithm from a starting value, the seed. With the same version, algorithm and sequence of calls, the same seed replays the same sequence. Different seeds do not guarantee that every individual output differs.

import random

random.seed(42)
print([random.randint(1, 100) for _ in range(5)])

random.seed(42)
print([random.randint(1, 100) for _ in range(5)])      # identical

random.seed(7)
print([random.randint(1, 100) for _ in range(5)])      # a different seed

Why does this matter?

Because scientific computing, machine learning, and any experiment involving randomness must be reproducible. Someone taking your code has to get your results, or there is no way to verify, compare or debug anything.

import random

def simulate(n: int, seed: int | None = None) -> float:
    """Roll a die n times and return the mean."""
    if n <= 0:
        raise ValueError("n must be positive")
    rng = random.Random(seed)
    total = 0
    for _ in range(n):
        total += rng.randint(1, 6)
    return total / n

print("no seed, usually different:")
print(f"  {simulate(1000):.4f}")
print(f"  {simulate(1000):.4f}")

print("seeded, identical each time:")
print(f"  {simulate(1000, seed=123):.4f}")
print(f"  {simulate(1000, seed=123):.4f}")

---

2. Text encoding

Characters and bytes

L2 established that computers store only bits, and that a character’s meaning comes from convention. ASCII was the first widely adopted convention: 7 bits, 128 characters.

128 slots clearly cannot cover the world’s writing. The modern answer is Unicode, assigning a number — a code point — to essentially every character.

print(ord('A'), ord('中'), ord('😀'))
print(chr(65), chr(20013), chr(128512))

But a code point is not yet bytes. Unicode says 中 is number 20013; it does not say how that number should sit in a file.

Unicode also defines character properties and related rules. len(str) counts code points, not displayed characters in general.

An encoding represents text as bytes. A widely used choice is UTF-8:

s = "A中😀"

utf8 = s.encode("utf-8")
print("text:", s)
print("UTF-8 bytes:", utf8)
print("characters:", len(s), " bytes:", len(utf8))

Look at that: 3 characters, 8 bytes. UTF-8 is variable-length:

Many common Chinese characters use three bytes, but not all. A displayed emoji may contain several code points.

for ch in "A中😀":
    b = ch.encode("utf-8")
    print(f"{ch}  code point {ord(ch):>6}  {len(b)} bytes  {b}")

How mojibake happens

Decoding bytes with the wrong encoding produces garbage.

s = "你好"
b = s.encode("utf-8")
print("decoded correctly:", b.decode("utf-8"))
print("decoded as latin-1:", b.decode("latin-1"))      # garbage

s = "你好"
b = s.encode("utf-8")
print(b.decode("gbk"))       # a different Chinese encoding — also garbage

s = "你好"
b = s.encode("utf-8")
print(b.decode("ascii"))     # ASCII cannot represent it at all

One rule for practice

Use UTF-8 for new course text; read existing files with their actual encoding, specified explicitly.

# poor: depends on the system default, and breaks on another machine
open("data.txt")

# good: explicit
open("data.txt", encoding="utf-8")

Windows often defaults to something other than UTF-8, which is a common source of “but it works on my machine”.

---

3. Files

Open, read, close

# writing
with open("demo.txt", "w", encoding="utf-8") as f:
    f.write("first line\n")
    f.write("second line\n")

# reading
with open("demo.txt", encoding="utf-8") as f:
    content = f.read()
print(repr(content))

with guarantees the file is closed even if something raises in between (L11). Do not call open and close by hand.

Modes

Three ways to read

with open("demo.txt", "w", encoding="utf-8") as f:
    f.write("apple 3\nbanana 5\norange 2\n")

# 1. everything at once, as one string
with open("demo.txt", encoding="utf-8") as f:
    print(repr(f.read()))

# 2. as a list of lines
with open("demo.txt", encoding="utf-8") as f:
    print(f.readlines())

# 3. line by line — preferred
with open("demo.txt", encoding="utf-8") as f:
    for line in f:
        print(repr(line))

A complete example

# create some data first
data = """Alex,math,87
Blake,math,92
Alex,physics,78
Chen,math,65
Blake,physics,88
"""
with open("scores.txt", "w", encoding="utf-8") as f:
    f.write(data)


def load_scores(path: str) -> dict[str, dict[str, int]]:
    """Read a scores file into {name: {subject: score}}."""
    result: dict[str, dict[str, int]] = {}
    with open(path, encoding="utf-8") as f:
        for lineno, line in enumerate(f, start=1):
            line = line.strip()
            if not line:                      # skip blank lines
                continue
            try:
                name, subject, score = line.split(",")
                value = int(score)
                if not 0 <= value <= 100:
                    raise ValueError("score outside 0..100")
                result.setdefault(name, {})[subject] = value
            except ValueError:
                print(f"{path}:{lineno}: invalid score record; input={line!r}")
    return result


scores = load_scores("scores.txt")
for name, subjects in scores.items():
    avg = sum(subjects.values()) / len(subjects)
    print(f"{name}: {subjects}  mean {avg:.1f}")

Three details there embody “the outside world is dirty”:

• skip blank lines — files usually have one at the end
• wrap parsing in try/except — one bad row should not kill the program
• report the line number — otherwise you have no idea where to look

CSV

Comma-separated data looks simple, but real CSV has traps: fields may contain commas, newlines and quotes. Use the standard library; do not split by hand.

import csv

rows = [
    ["name", "subject", "score"],
    ["Alex", "math", 87],
    ["Blake, Jr.", "physics", 92],       # a comma inside a field
]

with open("scores.csv", "w", encoding="utf-8", newline="") as f:
    writer = csv.writer(f)
    writer.writerows(rows)

with open("scores.csv", encoding="utf-8") as f:
    print(f.read())

with open("scores.csv", encoding="utf-8", newline="") as f:
    for row in csv.reader(f):
        print(row)

Look at the second data row — csv quoted the field containing a comma, and reads it back correctly. A hand-written split(",") would break here.

DictReader is nicer still:

import csv

with open("scores.csv", encoding="utf-8", newline="") as f:
    for row in csv.DictReader(f):
        print(row["name"], "→", row["score"])

Paths

from pathlib import Path

p = Path("scores.csv")
print("exists:", p.exists())
print("name:", p.name)
print("suffix:", p.suffix)
print("parent:", p.parent)
print("size:", p.stat().st_size, "bytes")

# build paths with / rather than string concatenation
data_dir = Path(".")
print(data_dir / "sub" / "file.txt")

pathlib beats gluing strings because it is cross-platform — Windows uses \, Unix uses /, and pathlib handles it.

---

4. Regular expressions

When you need them

split(",") handles tidy data. Real text often is not tidy:

2026-09-14 10:23:45 ERROR  user alice login failed
2026-09-14 10:23:47 INFO   user bob login succeeded
2026-09-14 10:24:02 ERROR  user carol login failed

Plain split() handles repeated whitespace. A regex helps when fields also have optional components, length rules or value constraints. CSV still belongs to the csv parser.

A regular expression is a language for describing patterns in text. You describe the shape; it finds the matches.

Basic syntax

import re

text = "order A12345, amount 299, date 2026-09-14"

print(re.findall(r"\d+", text))            # every run of digits
print(re.findall(r"[A-Z]\d+", text))       # a capital followed by digits
print(re.search(r"\d{4}-\d{2}-\d{2}", text).group())   # the date

The core notation:

Quantity:

Position:

The four main functions

import re

text = "cat bat rat mat"

print(re.search(r"[bc]at", text))          # the first match, as a Match object
print(re.search(r"[bc]at", text).group())  # the matched text
print(re.findall(r"[bcr]at", text))        # every match, as a list
print(re.sub(r"[bcr]at", "dog", text))     # replace
print(re.split(r"\s+", text))              # split on a pattern

Groups: extracting the parts you want

Parentheses mark the pieces you want back separately:

import re

log = "2026-09-14 10:23:45 ERROR user alice login failed"

m = re.search(r"(\d{4})-(\d{2})-(\d{2})", log)
if m:
    print("whole:", m.group())        # or m.group(0)
    print("year:", m.group(1))
    print("month:", m.group(2))
    print("day:", m.group(3))
    print("all groups:", m.groups())

Named groups read better:

import re

log = "2026-09-14 10:23:45 ERROR user alice login failed"

m = re.search(r"(?P<date>\S+) (?P<time>\S+) (?P<level>\w+) user (?P<user>\w+)", log)
if m:
    print(m.group("date"), "|", m.group("level"), "|", m.group("user"))
    print(m.groupdict())

A complete example

import re

logs = """2026-09-14 10:23:45 ERROR user alice login failed
2026-09-14 10:23:47 INFO  user bob login succeeded
2026-09-14 10:24:02 ERROR user carol login failed
2026-09-14 10:24:15 ERROR user alice login failed
a line in the wrong format
2026-09-14 10:25:00 INFO  user dave login succeeded"""

pattern = re.compile(r"^(?P<date>\S+) (?P<time>\S+) +(?P<level>\w+) +user (?P<user>\w+) (?P<action>.+)$")

failures: dict[str, int] = {}
for line in logs.split("\n"):
    m = pattern.match(line)
    if not m:
        print(f"  skipping unparseable line: {line!r}")
        continue
    if m.group("level") == "ERROR":
        user = m.group("user")
        failures[user] = failures.get(user, 0) + 1

print("failed logins:", failures)

re.compile names a reusable, testable pattern. Module functions also cache recent patterns, so explicit compilation does not necessarily make calls faster. Use fullmatch for whole-record validation. A date-shaped match does not establish calendar validity.

Greedy and lazy

import re

html = "<b>bold</b> and <i>italic</i>"

print(re.findall(r"<.*>", html))     # greedy: as long as possible
print(re.findall(r"<.*?>", html))    # lazy: as short as possible

* and + are greedy by default — they take as many characters as they can. Adding ? makes them lazy.

When not to reach for a regex

They are powerful and hard to read. A workable test:

text = "name=Alice"

# for something this simple, string methods are clearer
key, value = text.split("=", 1)
print(key, value)

# do not use one for its own sake
import re
print(re.match(r"(\w+)=(\w+)", text).groups())     # same answer, harder to read

---

5. A Complete Processing Pipeline

Success means every input record has an explicit outcome: accepted rows contribute to statistics, rejected rows retain diagnostics, and exported records survive a CSV round trip.

This example uses a smaller three-field format than the lab. A pattern checks the field shape; duration conversion and range checks validate meaning. Blank lines and comments are rejected in this example rather than silently ignored. Missing files, permission errors and decoding failures remain file-level errors for the caller; only expected record failures are recovered here.

csv.DictWriter complements DictReader: fieldnames sets the column order and writeheader() writes the header. CSV reads fields as strings, so the example retains the validated duration string. Statistics change only after validation succeeds.

"""Complete lecture example; intentionally smaller than the Lab12 contract."""
import csv
from pathlib import Path
import re

PATTERN = re.compile(
    r"(?P<level>INFO|WARN|ERROR)\s+"
    r"user=(?P<user>[a-z]+)\s+"
    r"ms=(?P<ms>[0-9]+)"
)


def process_log(path: str):
    """Reject malformed rows; file errors propagate to the caller."""
    records, diagnostics, counts = [], [], {}
    with open(path, encoding="utf-8") as f:
        for lineno, raw in enumerate(f, start=1):
            line = raw.rstrip("\n")
            match = PATTERN.fullmatch(line)
            if match is None:
                diagnostics.append((lineno, line, "invalid fields"))
                continue
            row = match.groupdict()
            try:
                duration = int(row["ms"])
                if not 1 <= duration <= 3000:
                    raise ValueError("ms outside 1..3000")
            except ValueError as exc:
                diagnostics.append((lineno, line, str(exc)))
                continue
            records.append(row)
            level = row["level"]
            counts[level] = counts.get(level, 0) + 1
    return records, diagnostics, counts


def demo():
    sample = "\n".join([
        "INFO user=alice ms=42",
        "ERROR user= ms=abc",
        "WARN user=bob ms=0",
        "ERROR user=carol ms=99",
    ])
    Path("app.log").write_text(sample, encoding="utf-8")
    records, diagnostics, counts = process_log("app.log")
    for lineno, raw, reason in diagnostics:
        print(f"app.log:{lineno}: {reason}; input={raw!r}")
    print("counts:", counts)
    fields = ["level", "user", "ms"]
    with open("parsed.csv", "w", encoding="utf-8", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=fields)
        writer.writeheader()
        writer.writerows(records)
    with open("parsed.csv", encoding="utf-8", newline="") as f:
        restored = list(csv.DictReader(f))
    assert restored == records
    assert len(records) + len(diagnostics) == len(sample.splitlines())
    print("CSV round trip:", restored == records)


if __name__ == "__main__":
    demo()

Actual output:

app.log:2: invalid fields; input='ERROR user= ms=abc'
app.log:3: ms outside 1..3000; input='WARN user=bob ms=0'
counts: {'INFO': 1, 'ERROR': 1}
CSV round trip: True

Reference Topics

The reference slides contain runnable examples. Report file and record failures separately instead of returning an unexplained empty result. Each rejected input above can be used directly as a regression test.

Summary

Randomness

• randint(a, b) includes b; randrange does not
• sample does not repeat positions; values may repeat. choices samples with replacement; shuffle works in place
• Record the seed, version and call sequence; initialize once per experiment
• Use secrets, not random, for anything cryptographic

Encoding

• Unicode numbers characters (code points); an encoding turns code points into bytes
• UTF-8 uses 1–4 bytes per scalar value; displayed characters may contain multiple code points
• Mojibake is decoding bytes with the wrong encoding
• Choose UTF-8 for new text; specify the actual encoding when reading existing files

Files

• Always use with — it guarantees closing
• "w" truncates the file
• Process large files line by line; retaining rows or growing statistics still consumes memory
• Preserve raw input for diagnostics; remove newlines and spaces only according to the data contract
• Use the csv module, never a hand-written split(",")
• Use pathlib for paths; it is cross-platform

Regular expressions

• Always write patterns as r"..."
• \d \w \s . [...] with * + ? {n,m}
• search finds the first, findall all, sub replaces
• search returns None when nothing matches — check before using
• Use named groups; clearer than counting parentheses
• * and + are greedy; add ? for lazy
• For simple cases use string methods — a regex is one more thing to debug

Exercises

1. Simulate rolling two dice 10,000 times and report the distribution of totals. Use a fixed seed so two runs agree.
2. How many bytes is "中华人民共和国" in UTF-8? In GBK? Verify with code and explain the difference.
3. Write count_lines(path: str) -> int. Requirements: use with, specify UTF-8, and raise something meaningful when the file does not exist.
4. Given a scores CSV with the header name,subject,score, use csv.DictReader to report the mean score per subject.
5. Write regular expressions extracting, from a block of text:
   • email addresses
   • dates in the form 2026-09-14
   • eleven-digit phone numbers beginning with 1
6. This is meant to match “exactly three digits” but also matches inside "12345". Why, and how would you fix it?

   re.search(r"\d{3}", "12345")

7. Do one real data-processing task: read a log file (format your own, but deliberately include some malformed lines), count occurrences of each level (INFO/ERROR/WARN), and skip and report the lines you could not parse.

The lab problems are in Lab 12. Solo assignment 3, big integers, spans weeks 12–13. Consult the schedule and submission page for the configured deadline.