Final Project: Multimodal Classification Challenge

1Downloads & submission

• Coursework handout (PDF) — 6 pages with figures.
• Data + starter bundle (.tar.gz, ~80 MB):
  • data/ — all CSV and NPZ files live here:
    • data/train.csv, data/val.csv, data/unlabeled.csv, data/test.csv — example_id, text_raw, labels or mask
    • data/{split}_image.npz, data/{split}_text_CLIP.npz, data/{split}_text_BERT.npz — pooled 768-d features
  • starter/ — load_data.py, evaluate.py, baseline_concat_mlp.py, submission_template.csv
  • tests/sanity_checks.py — verifies the bundle and a submission CSV
  • docs/report_template.tex & docs/report_template.docx — NeurIPS-style report templates
• Leaderboard: http://taohuang.info:3317 — register your team, upload submissions, see your rank. Register by May 29, 23:59 so the instructor can create your Canvas team in time.
• Canvas: a single zip with code, report.pdf, and slides.pdf — submitted after the leaderboard closes, by Jun 14 23:59. The leaderboard handles the test submission; Canvas handles the write-up. You can only upload the Canvas zip once the instructor has created your Canvas team from your leaderboard registration.

2Learning objectives

• Reason about multimodal fusion and choose between early, late, and gated fusion architectures based on the task structure.
• Design for a known train/test distribution gap; quantify your method's robustness via stratified ablations.
• Use a small labeled set, a large unlabeled pool, and (optionally) LLM APIs as engineering components — while disclosing all of them.
• Communicate methodology in a NeurIPS-style report and an 8-minute talk + open Q&A.

3The task & data

You receive four splits of the same underlying dataset. Every example has the same per-modality content: an image-feature vector, two text-feature vectors (CLIP- and BERT-encoded), and the raw text the features were computed from. Labels are multi-hot vectors of length 18; classes are anonymized integers label_0 through label_17.

Splits

CSV schemas

Row i of every .npz file for a split corresponds to the i-th data row of {split}.csv. The CSV's example_id column is the canonical mapping.

Test-time dropout details

When has_image=0, the row in data/test_image.npz is all zeros. When has_text=0, BOTH text feature files have zero rows there and text_raw is the empty string. The mask is provided so you can identify these examples; the zeroing is already applied.

4Setup & first run

tar -xzf student_bundle.tar.gz
cd student_bundle/
pip install numpy torch tqdm
python -m starter.baseline_concat_mlp     # ~1 minute, writes submission.csv
python -m tests.sanity_checks --submission submission.csv

The baseline is the floor you must beat — it is the simplest concat-MLP and does not know about test-time modality dropout. The remaining four weeks are about beating it.

6Submission format & leaderboard

A CSV with exactly this header and 2,000 data rows (one per test example, any order):

example_id,label_0,label_1,...,label_17
ex_00123,0.81,0.04,...,0.03
...

• Values may be probabilities in [0,1] or binaries in {0,1}; the leaderboard thresholds probabilities at 0.5 by default.
• F1-macro is the leaderboard metric. F1-micro is reported but does not affect rank.
• Each team is anonymized on the public leaderboard — only your group nickname is shown.
• 5 submissions per day per team, 50 total over the four weeks. Format-error submissions that fail to score don't count against your quota.
• Scoring is immediate. Rank by best F1-macro; ties broken by whoever reached the score first.

7Timeline

10Grading rubric (100 points)

The 8-minute presentation + Q&A on the last class is graded separately as the 10% course-level presentation component — it is not part of these 100 project points.

11Submission checklist

• Registered your team on the leaderboard (one account per team).
• sanity_checks.py passes for both the bundle and your final submission CSV.
• Final submission uploaded to the leaderboard before Jun 12, 20:00.
• Single zip uploaded to Canvas before Jun 14, 23:59 containing:
  • code/ with a runnable README and pinned random seeds
  • report.pdf — max 8 pages, all required sections including AI usage
  • slides.pdf — presentation slides