Programs must meet the outside world

For eleven lectures all your data has been literal text in your source. Real programs are not like that — data arrives from files, networks, sensors and users.

Three related things belong together here:

• randomness: programs need uncertainty (simulation, sampling, games)
• text encoding: external data is bytes, and has to become characters
• files and regular expressions: getting data in, and pulling out the parts you want

What they share is this: the outside world is dirty. Files go missing, encodings are wrong, formats are irregular. So L11’s exception handling appears repeatedly below.

---

1. Randomness

The basics

import random

print(random.randint(1, 6))          # 1 to 6, both ends included
print(random.randrange(0, 10, 2))    # one element of range(0,10,2)
print(random.random())               # a float in [0.0, 1.0)
print(random.uniform(1.5, 3.5))      # a float in [1.5, 3.5]

Choosing from a sequence

import random

deck = ["SA", "HK", "DQ", "CJ", "S10"]

print(random.choice(deck))                 # one at random
print(random.sample(deck, 3))              # 3 without replacement
print(random.choices(deck, k=3))           # 3 with replacement, may repeat

shuffled = deck.copy()
random.shuffle(shuffled)                   # shuffles in place
print(shuffled)
print(deck)                                # the original is untouched

choices also takes weights:

import random

random.seed(42)
outcomes = random.choices(["win", "draw", "lose"], weights=[1, 2, 7], k=20)
print(outcomes)
print("losses:", outcomes.count("lose"))

seed: making randomness reproducible

This section matters more than everything above it.

Computer “random” numbers are pseudorandom — produced by a deterministic algorithm from a starting value, the seed. The same seed gives the same sequence.

import random

random.seed(42)
print([random.randint(1, 100) for _ in range(5)])

random.seed(42)
print([random.randint(1, 100) for _ in range(5)])      # identical

random.seed(7)
print([random.randint(1, 100) for _ in range(5)])      # a different seed

Why does this matter?

Because scientific computing, machine learning, and any experiment involving randomness must be reproducible. Someone taking your code has to get your results, or there is no way to verify, compare or debug anything.

import random

def simulate(n: int, seed: int | None = None) -> float:
    """Roll a die n times and return the mean."""
    if seed is not None:
        random.seed(seed)
    return sum(random.randint(1, 6) for _ in range(n)) / n

print("no seed, different each time:")
print(f"  {simulate(1000):.4f}")
print(f"  {simulate(1000):.4f}")

print("seeded, identical each time:")
print(f"  {simulate(1000, seed=123):.4f}")
print(f"  {simulate(1000, seed=123):.4f}")

---

2. Text encoding

Characters and bytes

L2 established that computers store only bits, and that a character’s meaning comes from convention. ASCII was the first widely adopted convention: 7 bits, 128 characters.

128 slots clearly cannot cover the world’s writing. The modern answer is Unicode, assigning a number — a code point — to essentially every character.

print(ord('A'), ord('中'), ord('😀'))
print(chr(65), chr(20013), chr(128512))
print("Unicode now covers well over 150,000 characters")

But a code point is not yet bytes. Unicode says 中 is number 20013; it does not say how that number should sit in a file.

The rule turning code points into bytes is an encoding, and the universal one is UTF-8:

s = "A中😀"

utf8 = s.encode("utf-8")
print("text:", s)
print("UTF-8 bytes:", utf8)
print("characters:", len(s), " bytes:", len(utf8))

Look at that: 3 characters, 8 bytes. UTF-8 is variable-length:

for ch in "A中😀":
    b = ch.encode("utf-8")
    print(f"{ch}  code point {ord(ch):>6}  {len(b)} bytes  {b}")

How mojibake happens

Decoding bytes with the wrong encoding produces garbage.

s = "你好"
b = s.encode("utf-8")
print("decoded correctly:", b.decode("utf-8"))
print("decoded as latin-1:", b.decode("latin-1"))      # garbage

s = "你好"
b = s.encode("utf-8")
print(b.decode("gbk"))       # a different Chinese encoding — also garbage

s = "你好"
b = s.encode("utf-8")
print(b.decode("ascii"))     # ASCII cannot represent it at all

One rule for practice

Always state the encoding explicitly, and always use UTF-8.

# poor: depends on the system default, and breaks on another machine
open("data.txt")

# good: explicit
open("data.txt", encoding="utf-8")

Windows often defaults to something other than UTF-8, which is a common source of “but it works on my machine”.

---

3. Files

Open, read, close

# writing
with open("/tmp/demo.txt", "w", encoding="utf-8") as f:
    f.write("first line\n")
    f.write("second line\n")

# reading
with open("/tmp/demo.txt", encoding="utf-8") as f:
    content = f.read()
print(repr(content))

with guarantees the file is closed even if something raises in between (L11). Do not call open and close by hand.

Modes

Three ways to read

with open("/tmp/demo.txt", "w", encoding="utf-8") as f:
    f.write("apple 3\nbanana 5\norange 2\n")

# 1. everything at once, as one string
with open("/tmp/demo.txt", encoding="utf-8") as f:
    print(repr(f.read()))

# 2. as a list of lines
with open("/tmp/demo.txt", encoding="utf-8") as f:
    print(f.readlines())

# 3. line by line — preferred
with open("/tmp/demo.txt", encoding="utf-8") as f:
    for line in f:
        print(repr(line))

A complete example

# create some data first
data = """Alex,maths,87
Blake,maths,92
Alex,physics,78
Chen,maths,65
Blake,physics,88
"""
with open("/tmp/scores.txt", "w", encoding="utf-8") as f:
    f.write(data)


def load_scores(path: str) -> dict[str, dict[str, int]]:
    """Read a scores file into {name: {subject: score}}."""
    result: dict[str, dict[str, int]] = {}
    with open(path, encoding="utf-8") as f:
        for lineno, line in enumerate(f, start=1):
            line = line.strip()
            if not line:                      # skip blank lines
                continue
            try:
                name, subject, score = line.split(",")
                result.setdefault(name, {})[subject] = int(score)
            except ValueError:
                print(f"  line {lineno} is malformed, skipping: {line!r}")
    return result


scores = load_scores("/tmp/scores.txt")
for name, subjects in scores.items():
    avg = sum(subjects.values()) / len(subjects)
    print(f"{name}: {subjects}  mean {avg:.1f}")

Three details there embody “the outside world is dirty”:

• skip blank lines — files usually have one at the end
• wrap parsing in try/except — one bad row should not kill the program
• report the line number — otherwise you have no idea where to look

CSV

Comma-separated data looks simple, but real CSV has traps: fields may contain commas, newlines and quotes. Use the standard library; do not split by hand.

import csv

rows = [
    ["name", "subject", "score"],
    ["Alex", "maths", 87],
    ["Blake, Jr.", "physics", 92],       # a comma inside a field
]

with open("/tmp/scores.csv", "w", encoding="utf-8", newline="") as f:
    writer = csv.writer(f)
    writer.writerows(rows)

with open("/tmp/scores.csv", encoding="utf-8") as f:
    print(f.read())

with open("/tmp/scores.csv", encoding="utf-8", newline="") as f:
    for row in csv.reader(f):
        print(row)

Look at the second data row — csv quoted the field containing a comma, and reads it back correctly. A hand-written split(",") would break here.

DictReader is nicer still:

import csv

with open("/tmp/scores.csv", encoding="utf-8", newline="") as f:
    for row in csv.DictReader(f):
        print(row["name"], "→", row["score"])

Paths

from pathlib import Path

p = Path("/tmp/scores.csv")
print("exists:", p.exists())
print("name:", p.name)
print("suffix:", p.suffix)
print("parent:", p.parent)
print("size:", p.stat().st_size, "bytes")

# build paths with / rather than string concatenation
data_dir = Path("/tmp")
print(data_dir / "sub" / "file.txt")

pathlib beats gluing strings because it is cross-platform — Windows uses \, Unix uses /, and pathlib handles it.

---

4. Regular expressions

When you need them

split(",") handles tidy data. Real text often is not tidy:

2026-09-14 10:23:45 ERROR  user alice login failed
2026-09-14 10:23:47 INFO   user bob login succeeded
2026-09-14 10:24:02 ERROR  user carol login failed

Suppose you want every user whose login failed. split can do it, but fragilely — one extra space and it breaks.

A regular expression is a language for describing patterns in text. You describe the shape; it finds the matches.

Basic syntax

import re

text = "order A12345, amount 299, date 2026-09-14"

print(re.findall(r"\d+", text))            # every run of digits
print(re.findall(r"[A-Z]\d+", text))       # a capital followed by digits
print(re.search(r"\d{4}-\d{2}-\d{2}", text).group())   # the date

The core notation:

Quantity:

Position:

The four main functions

import re

text = "cat bat rat mat"

print(re.search(r"[bc]at", text))          # the first match, as a Match object
print(re.search(r"[bc]at", text).group())  # the matched text
print(re.findall(r"[bcr]at", text))        # every match, as a list
print(re.sub(r"[bcr]at", "dog", text))     # replace
print(re.split(r"\s+", text))              # split on a pattern

Groups: extracting the parts you want

Parentheses mark the pieces you want back separately:

import re

log = "2026-09-14 10:23:45 ERROR user alice login failed"

m = re.search(r"(\d{4})-(\d{2})-(\d{2})", log)
if m:
    print("whole:", m.group())        # or m.group(0)
    print("year:", m.group(1))
    print("month:", m.group(2))
    print("day:", m.group(3))
    print("all groups:", m.groups())

Named groups read better:

import re

log = "2026-09-14 10:23:45 ERROR user alice login failed"

m = re.search(r"(?P<date>\S+) (?P<time>\S+) (?P<level>\w+) user (?P<user>\w+)", log)
if m:
    print(m.group("date"), "|", m.group("level"), "|", m.group("user"))
    print(m.groupdict())

A complete example

import re

logs = """2026-09-14 10:23:45 ERROR user alice login failed
2026-09-14 10:23:47 INFO  user bob login succeeded
2026-09-14 10:24:02 ERROR user carol login failed
2026-09-14 10:24:15 ERROR user alice login failed
a line in the wrong format
2026-09-14 10:25:00 INFO  user dave login succeeded"""

pattern = re.compile(r"^(?P<date>\S+) (?P<time>\S+) +(?P<level>\w+) +user (?P<user>\w+) (?P<action>.+)$")

failures: dict[str, int] = {}
for line in logs.split("\n"):
    m = pattern.match(line)
    if not m:
        print(f"  skipping unparseable line: {line!r}")
        continue
    if m.group("level") == "ERROR":
        user = m.group("user")
        failures[user] = failures.get(user, 0) + 1

print("failed logins:", failures)

re.compile compiles the pattern once, which is faster when you use it repeatedly in a loop.

Greedy and lazy

import re

html = "<b>bold</b> and <i>italic</i>"

print(re.findall(r"<.*>", html))     # greedy: as long as possible
print(re.findall(r"<.*?>", html))    # lazy: as short as possible

* and + are greedy by default — they take as many characters as they can. Adding ? makes them lazy.

When not to reach for a regex

They are powerful and hard to read. A workable test:

text = "name=Alice"

# for something this simple, string methods are clearer
key, value = text.split("=")
print(key, value)

# do not use one for its own sake
import re
print(re.match(r"(\w+)=(\w+)", text).groups())     # same answer, harder to read

---

Summary

Randomness

• randint(a, b) includes b; randrange does not
• sample never repeats, choices may, shuffle works in place
• seed() makes results reproducible — any program using randomness should expose the seed
• Use secrets, not random, for anything cryptographic

Encoding

• Unicode numbers characters (code points); an encoding turns code points into bytes
• UTF-8 is variable length: 1 byte for ASCII, 3 for CJK, 4 for emoji
• Mojibake is decoding bytes with the wrong encoding
• Always write encoding="utf-8" explicitly

Files

• Always use with — it guarantees closing
• "w" truncates the file
• Read large files with for line in f; memory use stays constant
• Every line carries its \n, so you almost always .strip()
• Use the csv module, never a hand-written split(",")
• Use pathlib for paths; it is cross-platform

Regular expressions

• Always write patterns as r"..."
• \d \w \s . [...] with * + ? {n,m}
• search finds the first, findall all, sub replaces
• search returns None when nothing matches — check before using
• Use named groups; clearer than counting parentheses
• * and + are greedy; add ? for lazy
• For simple cases use string methods — a regex is one more thing to debug

Exercises

1. Simulate rolling two dice 10,000 times and report the distribution of totals. Use a fixed seed so two runs agree.
2. How many bytes is "中华人民共和国" in UTF-8? In GBK? Verify with code and explain the difference.
3. Write count_lines(path: str) -> int. Requirements: use with, specify UTF-8, and raise something meaningful when the file does not exist.
4. Given a scores CSV with the header name,subject,score, use csv.DictReader to report the mean score per subject.
5. Write regular expressions extracting, from a block of text:
   • email addresses
   • dates in the form 2026-09-14
   • eleven-digit phone numbers beginning with 1
6. This is meant to match “exactly three digits” but also matches inside "12345". Why, and how would you fix it?

   re.search(r"\d{3}", "12345")

7. Do one real data-processing task: read a log file (format your own, but deliberately include some malformed lines), count occurrences of each level (INFO/ERROR/WARN), and skip and report the lines you could not parse.

The lab problems are in Lab 12. Solo project B is due this week.