Study Guide 3 — Labs 6–7: Neural Network Background & Building Networks with nn.Module
md10 min readSummary of AIDE500
Method: Read → run the notebook cells → retype
class Netfrom memory → Self-check out loud.
LAB 6 — Neural Networks Background (ResNet, freezing, transfer learning)
The core definition (memorize)
- An NN is a collection of nested functions executed on input data. Their behavior is defined by parameters (weights and biases), stored in PyTorch as tensors.
- Training = two steps:
- Forward propagation — the network runs input data through each function to make its best guess.
- Backward propagation — the network adjusts parameters proportionate to the error, by traversing backwards from the output, collecting derivatives of the error w.r.t. the parameters (gradients), and optimizing with gradient descent.
- Update rule (Lab 7 states it):
weight = weight - learningRate * gradient
torchvision's 3 key components
| Component | What it gives you |
|---|---|
| Datasets | standard loaders: ImageNet, CIFAR-10, MNIST |
| Models | architectures with pre-trained weights: ResNet, VGG, MobileNet |
| Transforms | preprocessing & augmentation operations for images |
Pretrained vs from-scratch (EXAM Q37 territory)
model = resnet18(weights=ResNet18_Weights.DEFAULT) # pretrained on ImageNet-1K (1000 classes)
model = resnet18(weights=None) # random weights — train from scratchResNet18_Weights.DEFAULTdownloads the best available weights trained on ImageNet-1K = 1000 classes → that's why the last layer showsout_features=1000.weights=None= architecture only, random weights (you'll see garbage predictions like0.04, 0.02, ...).
Saving / loading a model — state_dict
torch.save(model.state_dict(), PATH) # save: a DICTIONARY {layer_name: weight_tensor}
state_dict = torch.load(PATH, map_location="cpu")
for key, value in state_dict.items():
print(f"{key}: {value.shape}") # e.g. conv1.weight: torch.Size([64, 3, 7, 7])
model = resnet18(weights=None)
model.load_state_dict(torch.load(PATH)) # load weights into a fresh architecturestate_dict= Python dict mapping each layer → its parameter tensor (weights & biases).- You can read shapes from it — e.g.
conv1.weightis(64, 3, 7, 7)= 64 filters, 3 input channels (RGB), 7×7 kernel.
Changing the classifier head (EXAM Q37 answer pattern)
model.fc = nn.Linear(model.fc.in_features, 10) # 512 -> 10 (new dataset with 10 labels)model.fcis ResNet's final fully-connected layer;in_features = 512stays,out_featuresbecomes your number of classes.
One full training step (the lab's walkthrough — worth re-typing)
inputData = torch.rand(1, 3, 64, 64) # (batch, channels, H, W): 1 RGB image 64×64
labels = torch.rand(1, n_features) # must be compatible with n_features (10)
prediction = model(inputData) # 1. FORWARD PASS -> shape (1, 10)
loss = (prediction - labels).sum() # 2. LOSS (dummy error here)
loss.backward() # 3. BACKWARD: grads -> .grad of each parameter
optim = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=0.9)
optim.step() # 4. UPDATE the weights- Optimizer creation registers all model parameters; tensors passed as parameters get
requires_grad=Trueautomatically.
Visualizing images (re-used in Lab 9's imshow)
image = torchvision.utils.make_grid(inputData) # 4D batch -> single 3D image
npimg = image.numpy()
plt.imshow(np.transpose(npimg, (1, 2, 0))) # (C,H,W) -> (H,W,C) for matplotlib- Matplotlib wants NumPy arrays and wants channels last; PyTorch images are (C, H, W) → transpose
(1,2,0).
Differentiation demo (understand, don't memorize numbers)
a = torch.tensor([2., 3.], requires_grad=True)
b = torch.tensor([6., 4.], requires_grad=True)
Q = 3*a**3 - b**2 # "error" function
Q.backward(gradient=torch.tensor([1., 1.])) # gradient arg needed because Q is a VECTOR
a.grad # = 9*a² = [36, 81] ✓ (checked in the lab)
b.grad # = -2*b = [-12, -8] ✓- Why
gradient=[1.,1.]: for a vector output you must seed it; mathematicallydQ/dQ = 1.
The computational graph (DAG)
- autograd records tensors + operations in a directed acyclic graph of
Functionobjects. Leaves = input tensors, roots = output tensors. - Forward pass does 2 things simultaneously: (1) runs the operation, (2) stores the operation's gradient function in the DAG.
.backward()on the root: computes gradients from eachgrad_fn, accumulates them into.grad, and propagates via the chain rule all the way to the leaves.- Graphs are dynamic: recreated from scratch after every
backward()→ you can use normal Python control flow (if/loops) and change shapes every iteration. - Exclusion rule:
requires_grad=Falseremoves a tensor from the DAG. But if any input hasrequires_grad=True, the output requires gradients:
a = x + y # x, y normal -> a.requires_grad = False
b = x + z # z requires grad -> b.requires_grad = True (EXAM-tested logic)Freezing & transfer learning (EXAM Q37 — the recipe)
for param in model.parameters(): # freeze EVERYTHING
param.requires_grad = False
model.fc = nn.Linear(512, 10) # replace head — new layer is UNFROZEN by default
# Now ONLY model.fc's weights/bias get gradients & get updated during training- Why freeze: performance (fewer autograd computations) + keep learned features; typical transfer learning = keep feature extractor, retrain the classifier.
torch.no_grad()is the context-manager version of this exclusion (used at inference).
LAB 7 — Neural Networks with nn.Module
The training procedure (5 lines to memorize)
- Define the network (learnable parameters)
- Iterate over the dataset
- Process input through the network (forward)
- Compute the loss (how far output is from correct)
- Propagate gradients back (
backward) and update weights:weight = weight − lr × gradient
Device selection
device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"Building blocks (know each signature + shape behavior)
| Component | Signature | Shape effect |
|---|---|---|
| nn.Module | base class of all NN components in PyTorch | — |
| nn.Linear | nn.Linear(in_features, out_features, bias=True) |
last dim: in → out; computes y = xAᵀ + b |
| nn.Conv2d | nn.Conv2d(in_ch, out_ch, kernel_size, stride, bias) |
spatial shrinks per formula below |
| F.max_pool2d | F.max_pool2d(x, kernel_size=2, stride=1, ...) |
down-sampling: max value in each window |
| F.relu | F.relu(x) |
max(0, x) — zeroes negatives (EXAM Q20 ✓) |
| nn.Softmax | nn.Softmax(dim=1) |
logits → probabilities, each in (0,1), sum to 1 |
| tensor.view | x.view(shape) |
reshape without copying (shared memory!), element count must match, -1 = infer |
nn.Parameter= a Tensor auto-registered as a parameter when assigned to a Module;requires_grad=Trueby default.torch.empty= uninitialized RAM (same garbage idea asnp.empty).- Linear demo: input
(3,2,4)within=4, out=3→ output(3,2,3): only the last dim changes. - Conv demo: input
(1,4,4),kernel=2, stride=1→(1,3,3)(because 4−2+1 = 3). - Softmax caution (EXAM Q5/Q24): Softmax is what converts logits into probabilities — Lab 9's network never applies it, so its outputs are raw logits.
F.relu≠ softmax; ReLU is just max(0,x).
The output-size formula (EXAM Q21 — you got this ✓)
P=padding,D=dilation (usually 1),K=kernel,S=stride. Simplified (P=0, D=1):H_out = H_in − K + 1(for S=1).- Pooling with
k=2, s=2(no padding): halves the size (floor division).
Building class Net — the single most important code block in the course
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(1, 6, 3) # 1 -> 6 channels, 3x3 kernel
self.conv2 = nn.Conv2d(6, 16, 3) # 6 -> 16 channels, 3x3 kernel
self.fc1 = nn.Linear(16 * 6 * 6, 120) # flatten 16x6x6 = 576 features
self.fc2 = nn.Linear(120, 84)
self.fc3 = nn.Linear(84, 10) # 10 classes
def forward(self, x):
x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2)) # conv -> relu -> pool
x = F.max_pool2d(F.relu(self.conv2(x)), 2) # conv -> relu -> pool (2 = square window)
x = x.view(-1, self.num_flat_features(x)) # flatten to 1-D per sample
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = self.fc3(x) # raw output (logits)
return x
def num_flat_features(self, x):
size = x.size()[1:] # ALL dims except the batch dimension
num_features = 1
for s in size: num_features *= s
return num_featuresKey facts:
- You write only
forward().backward()is automatically defined by autograd. ⚠️ EXAM Q9 (you answered True — wrong!). The lab states it verbatim: "You just have to define the forward function, and the backward function is automatically defined for you using autograd." net.parameters()returns all learnable tensors — this net has 10: (conv1.w, conv1.b, conv2.w, conv2.b, fc1.w, fc1.b, fc2.w, fc2.b, fc3.w, fc3.b).params[0]= conv1 weight, shape(6, 1, 3, 3);params[6]= fc2 weight, shape(84, 120).- Weights are initialized with random uninitialized values (via
torch.empty-style init) before training. - Input must be 4D:
torch.randn(1, 1, 32, 32)→ output(1, 10). - Shape trace of the lab's Net on 32×32 input (run it to believe it):
32 → conv(3) → 30 → pool(2) → 15 → conv(3) → 13 → pool(2) → 6 → view → 16·6·6 = 576 → fc → 10. The forward printstorch.Size([1, 576]). - The computation graph:
input → conv2d → relu → maxpool2d → conv2d → relu → maxpool2d → view → linear → relu → linear → relu → linear(then → MSELoss → loss).
Mini-batch rule (NOTE box)
torch.nnonly supports mini-batches — inputs arenSamples × nChannels × H × W.- Single sample? Add a fake batch dimension:
input.unsqueeze(0).
Loss functions
criterion = nn.MSELoss()
target = target.view(1, -1) # make target the SAME shape as output
loss = criterion(output, target) # scalar tensor with grad_fn=<MseLossBackward0>- Loss = how far output is from target. (Lab 9 will switch to
nn.CrossEntropyLoss.)
The gradient-zeroing lesson (⚠ EXAM Q28, Q47 — your weakest spot)
print(net.conv1.bias.grad) # some tensor (gradients from a previous backward!)
net.zero_grad() # ZERO all parameter gradients -> grad becomes None/0
print(net.conv1.bias.grad) # None / cleared
loss.backward() # now grads are populated FRESH- Why zero? Gradients accumulate — if you don't clear them, each iteration adds to the old values and your update is wrong.
- The lab's phrase: "You need to clear the existing gradients though, else gradients will be accumulated to existing gradients."
Updating weights — from raw math to torch.optim
Manual SGD:
learning_rate = 0.01
for f in net.parameters():
f.data.sub_(f.grad.data * learning_rate) # weight = weight - lr * gradientThe proper way (torch.optim — knows SGD, Nesterov-SGD, Adam, RMSProp, ...):
optimizer = optim.SGD(net.parameters(), lr=0.01) # or optim.Adam(...)
# THE TRAINING LOOP — learn by heart (order matters!)
optimizer.zero_grad() # 1. clear old gradients
output = net(input) # 2. forward
loss = criterion(output, target)
loss.backward() # 3. compute new gradients
optimizer.step() # 4. update weights⚠️ EXAM Q28 (explain-code) asks what's missing in a loop that starts with output = net(input) → the answer is optimizer.zero_grad() at the top. EXAM Q47 blanks = zero_grad / backward / step — you only filled step.
Recap box (verbatim meanings)
| Class | Role |
|---|---|
torch.Tensor |
multi-dim array; holds .grad w.r.t. tensors; supports backward() |
nn.Module |
network module — convenient encapsulation of parameters (+ GPU move, export, load helpers) |
nn.Parameter |
Tensor auto-registered as a parameter when assigned as a Module attribute |
autograd.Function |
implements forward and backward of an autograd operation; every tensor op creates ≥1 Function node encoding its history |
Displaying images (again)
def imshow(img):
img = img / 2 + 0.5 # un-normalize (reverse Normalize(0.5,0.5))
npimg = img.numpy()
plt.imshow(np.transpose(npimg, (1, 2, 0))) # (C,H,W) -> (H,W,C)Self-check — Labs 6–7
- In your own words: what are the two steps of NN training, and what does backward propagation actually collect?
resnet18(weights=ResNet18_Weights.DEFAULT)vsresnet18(weights=None)— difference?- What is
state_dict? What shape isconv1.weightin ResNet-18 and what do the 4 numbers mean? - You have a pretrained model with
fc=Linear(512, 1000)but your dataset has 20 classes. Write the two-line fix. - Freeze all params of
modelexcept the new head — code? - Why must
labelshaven_featurescolumns? What shape did the demo use? - Write
nn.Conv2dandnn.Linearsignatures. For input(1,4,4)withConv2d(1,1,kernel=2,stride=1), what's the output size? H_in=32, K=3, P=0, D=1, S=1→ H_out? Then pool k=2,s=2 → ?- What is the FULL size trace of the lab's Net for input (1,1,32,32) up to the flatten? (Answer:
1,576— derive it: 32→30→15→13→6, 16×6×6.) - True/False: (a) you must define
backward()yourself in your class. (b) gradients accumulate if you don't zero them. (c)view()copies data. (d)nn.Parameterrequires grad by default. - Fill in the training loop:
___()→output = net(input)→loss = ...→___()→___(). - What does
F.relucompute? What doesnn.Softmaxcompute, and does Lab 9's network apply it? - Why
unsqueeze(0)for a single image? - In the freezing demo, after
param.requires_grad = False+model.fc = nn.Linear(512,10), which tensors get gradients duringloss.backward()?
Answers: 1) forward = best guess; backward = collect derivatives (gradients) of loss w.r.t. each parameter, traversing output→inputs. 2) pretrained-ImageNet vs random-init. 3) dict {layer: tensor}; (64,3,7,7) = out-channels, in-channels(RGB), 7×7 kernel. 4) model.fc = nn.Linear(model.fc.in_features, 20). 5) loop model.parameters() → requires_grad=False, then replace model.fc. 6) output/label dims must match; demo (1,10). 7) see table; output (1,3,3). 8) 30 → 15. 9) 32→30→15→13→6; 16·6·6=576. 10) (a) False (auto!), (b) True, (c) False (shares), (d) True. 11) optimizer.zero_grad → forward → loss.backward → optimizer.step. 12) max(0,x); softmax = exp/Σexp → probabilities sum 1; Lab 9 does NOT apply it (outputs are logits). 13) nn needs batch dim. 14) Only model.fc's weight & bias.
Exam questions this prepares you for: Q5, Q9, Q20, Q21, Q24, Q28, Q29, Q34, Q37, Q47, plus Section 4's Q33/Q34.
Next: Study Guide 4 — Lab 8 (Transforms & DataLoaders) & Lab 9 (Training a classifier).