Lesson 16 – Training Stabilizer: Batch Normalization Mechanics From First Principles
The Trainer.fit() Trap
The fast path: drop nn.BatchNorm2d into your model, wrap everything in a Trainer.fit() call, done. Training converges faster and more smoothly than without it — BatchNorm earns its reputation immediately. This is the trap: the high-level wrapper calls model.train() before every training step and model.eval() before every evaluation step for you, so completely, invisibly correctly, that you never have to learn that BatchNorm is the one layer in your entire network whose math genuinely changes depending on which of those two modes it's in.
Here's what gets decided for you, silently, every time a Trainer object calls .train() or .eval() on your behalf:
Which statistics normalize your data. In training mode, BatchNorm normalizes using the current batch's mean and variance. In eval mode, it uses a running average accumulated across all of training. These are genuinely different numbers, computed by genuinely different code paths — not the same formula with a flag.
Whether your output depends on what else is in the batch. In training mode, yes — the same input produces a different output depending on which other examples happen to share its batch, because the normalization statistics are computed from the whole batch together. In eval mode, no — each example is normalized independently against fixed running statistics.
Which numbers are parameters and which are buffers. BatchNorm has two kinds of internal state:
gammaandbeta, learnable affine parameters thatoptimizer.step()updates via gradients; andrunning_mean/running_var, which update through direct arithmetic insideforward()and never receive a gradient at all. A library handles this distinction so cleanly you can go a long time without knowing it exists.
None of this is optional complexity to hide from you — it's the entire mechanism. Miss the difference between the two modes and you don't get an error. You get a model that appears to work and then behaves inexplicably at inference.
The Failure Mode: The Same Input, a Different Answer Every Time
Here's the bug, and it's one of the most disorienting a PyTorch practitioner can hit, precisely because nothing crashes. You finish training, save your model, load it for inference, and forget one line: model.eval(). The model is still in training mode — the default a freshly-loaded nn.Module doesn't change on its own.
You run inference on a single image. You get a prediction. You run inference on the exact same image again, this time as part of a batch with three other images. You get a different prediction — not because anything about the image changed, but because BatchNorm's normalization statistics were computed from whatever batch happened to be in front of it that call. Batch it with different companions a third time, get a third answer. The model's confidence, and sometimes its predicted class entirely, drifts based on things that have nothing to do with the input you actually care about.
Keep going with the full lesson
The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.
Already a member? Sign in