# Coursework 3: Multi-Layer Perceptron & Backpropagation

## Objective

Implement a multi-layer perceptron (MLP) with backpropagation from scratch using NumPy. Explore how hyperparameters affect performance and compare with CW2's logistic regression.

## Connection to CW2

This coursework **builds on CW2**. You will reuse and extend your CW2 implementations:

| CW2 concept | CW3 extension |
|---|---|
| `softmax()` | Reused in `CrossEntropyLoss.forward()` |
| `cross_entropy_loss()` | Wrapped into `CrossEntropyLoss` class with `forward()` + `backward()` |
| Optimizer: `dW = X.T @ grad` | Becomes `Linear.backward()`: same math + `grad_input = grad @ W.T` |
| `GradientDescent` | Extended to `SGD` with momentum |

**Key insight**: In CW2, your optimizer was secretly doing the backward pass of a single Linear layer!

## What You Need to Implement

| File | What to implement |
|---|---|
| `layers.py` | `Linear.forward()`, `Linear.backward()`, `ReLU.forward()`, `ReLU.backward()`, `Sigmoid.forward()`, `Sigmoid.backward()` |
| `model.py` | `MLP.__init__()`, `MLP.forward()`, `MLP.backward()` |
| `losses.py` | `CrossEntropyLoss.forward()`, `CrossEntropyLoss.backward()` |
| `optimizer.py` | `SGD.__init__()`, `SGD.step()`, `SGD.zero_grad()` |

## How to Run

```bash
cd cw3_mlp

# Quick mode (for debugging)
python run.py --quick

# Default configuration
python run.py

# Custom architecture
python run.py --hidden_dims 256 128 --activation relu --learning_rate 0.01

# Hyperparameter sweeps (for report)
python run.py --sweep hidden_dim
python run.py --sweep num_layers
python run.py --sweep learning_rate

# Run gradient checks
python -m tests.test_cw3
```

## How to Verify Your Implementation

1. Run `python -m tests.test_cw3` — all gradient checks should pass
2. Run `python run.py` — should reach ~88% test accuracy with default settings
3. Your MLP should clearly outperform your CW2 logistic regression

## Deliverables

1. **Code**: Completed `layers.py`, `model.py`, `losses.py`, `optimizer.py`
2. **Report** (use the template in `../../report_template/`, see example in `../../report_examples/cw3_example_report.pdf`):
   - Hyperparameter exploration (hidden dim, num layers, learning rate)
   - Comparison with CW2 logistic regression
   - Analysis of results

## Grading Rubric (100 points)

| Component | Points | Criteria |
|---|---|---|
| Linear layer forward + backward | 15 | Gradient passes numerical check |
| ReLU forward + backward | 10 | Correct zero-threshold gradient |
| Sigmoid forward + backward | 5 | Correct computation |
| CrossEntropy loss (reuse from CW2) | 10 | Correct forward/backward |
| MLP assembly (forward + backward) | 15 | Correct sequential forward, reverse-order backprop |
| SGD optimizer with momentum | 10 | Correct velocity update |
| Training convergence | 10 | 2-layer MLP >= 87% test accuracy |
| Hyperparameter exploration report | 15 | >= 3 hyperparameters explored with plots and conclusions |
| Comparison with CW2 | 5 | Discusses why MLP > logistic regression |
| Code quality | 5 | Clean, commented |

**Bonus (up to +10):** Adam optimizer (+5), dropout layer (+5)
