Processing Real Text Data

What this session is for

• Read and write files safely with with and an explicit encoding
• Handle irregular real data, skipping bad rows rather than crashing
• Extract fields from semi-structured text with regular expressions
• Make a random experiment reproducible

Generating the data

Run this first to produce the file for this session:

import random
from pathlib import Path

random.seed(1602)          # fixed seed, so everyone gets identical data

Path("/tmp/lab12").mkdir(exist_ok=True)

names = ["alice", "bob", "carol", "dave", "eve"]
levels = ["INFO", "WARN", "ERROR"]
lines = []
for i in range(200):
    ts = f"2026-11-{random.randint(1, 28):02d} {random.randint(0,23):02d}:{random.randint(0,59):02d}:{random.randint(0,59):02d}"
    lvl = random.choices(levels, weights=[6, 3, 1])[0]
    user = random.choice(names)
    ms = random.randint(1, 3000)
    lines.append(f"{ts} {lvl} user={user} action=login duration={ms}ms")

# deliberately insert 12 bad lines
for _ in range(12):
    bad = random.choice([
        "",                                     # blank
        "### this is a comment",                # comment
        "2026-11-05 nothing after the timestamp",   # missing fields
        "2026-11-06 12:00:00 DEBUG user= action=login duration=abc",   # empty user, non-numeric duration
    ])
    lines.insert(random.randint(0, len(lines)), bad)

Path("/tmp/lab12/app.log").write_text("\n".join(lines), encoding="utf-8")
print("wrote /tmp/lab12/app.log with", len(lines), "lines")

Problems

12-1 An encoding experiment

Write code answering these three questions, and put both the code and its output in your notes:

1. How many bytes is "计算导论" in UTF-8? In GBK? Why the difference?
2. Decode UTF-8-encoded "你好" as latin-1. What do you get? Encode that result back as latin-1 — do you recover the original bytes?
3. Decode the same bytes as ascii. What happens, and how does that failure differ from question 2’s?

12-2 Reading a file safely

Implement:

def read_lines(path: str) -> list[str]:
    """Read every line, stripping the trailing newline.
    Raise FileNotFoundError with something useful when the file is missing.
    """

Requirements: use with, specify encoding="utf-8", and iterate line by line rather than calling readlines() and post-processing.

Answer in your notes: what would go wrong with your implementation on a 10 GB file?

12-3 Parsing the log

Implement:

def parse_line(line: str) -> dict[str, str] | None:
    """Parse one log line into {date, time, level, user, action, duration}.
    Return None when it cannot be parsed.
    """

The normal format is:

2026-11-05 14:23:45 INFO user=alice action=login duration=1234ms

Use a regular expression with named groups. Strip the ms suffix from duration and return it as a string.

Then:

def load_log(path: str) -> tuple[list[dict[str, str]], list[tuple[int, str]]]:
    """Return (parsed records, [(line number, raw text)] for the bad lines)."""

Report in your notes: how many lines in total? how many parsed? what were the bad ones?

12-4 Statistics

Building on 12-3, implement three functions:

def count_by_level(records: list[dict[str, str]]) -> dict[str, int]: ...
def busiest_user(records: list[dict[str, str]]) -> str: ...
def average_duration(records: list[dict[str, str]]) -> float: ...

Note that duration is a string and conversion can fail. Should a failed record count toward the average, or be skipped? Your decision, but explain it in your notes.

12-5 Writing CSV

Write the records from 12-3 to /tmp/lab12/parsed.csv, with a header row.

Use the csv module (not a hand-written join(",")) and pass newline="".

Then read it back with csv.DictReader and verify the record count matches. Paste the verification code and its output into your notes.

12-6 Extraction with regular expressions

Given this text, write patterns extracting three things:

text = """
Contacts: Alex alex@sjtu.edu.cn 13812345678
Blake's email is blake.wang@example.com, phone 15900001111
Chen: chen@qq.com / 13700002222
Invalid: not-an-email@ and 123456
"""

1. every email address
2. every 11-digit phone number beginning with 1
3. the domain part of every email (whatever follows the @)

Give all three patterns in your notes and explain how you confirmed they do not match anything on the “Invalid” line.

12-7 A reproducible simulation

Write a function simulating the birthday problem:

def birthday_collision(n_people: int, trials: int, seed: int | None = None) -> float:
    """Run trials simulations, each assigning n_people random birthdays (1-365),
    and return the fraction in which at least two people share one.
    """

Use it to compute the collision probability for n_people from 20 to 30, and find the first group size where the probability exceeds 50%.

Requirement: the same seed must give exactly the same result. Paste the output of two runs into your notes to demonstrate it.

Before you submit

• 12-1 has code and output for all three questions
• 12-2 uses with and an explicit encoding, and answers the 10 GB question
• 12-3 uses named groups and reports the bad lines
• 12-3 does not crash on the full dirty dataset
• 12-4 explains how failed records are handled
• 12-5 uses the csv module with newline=""
• 12-6 gives all three patterns and explains the exclusions
• 12-7 shows two runs proving reproducibility
• AI usage declaration written
• Solo project B submitted