Lesson 17 — Overfit Fixer: Dropout, DropConnect & Stochastic Depth From First Principles
The Trainer.fit() Trap
The fast path: drop nn.Dropout(0.5) somewhere in your model, call .fit(), and watch validation accuracy improve a bit. This is the trap — not because dropout is complicated to use, but because a high-level wrapper never makes you confront what "regularization" is actually buying you, or what it costs to get wrong.
Here's what a one-line nn.Dropout() call decides for you, silently:
How the layer compensates for its own randomness. During training, some fraction
pof activations get zeroed. If nothing corrects for that, the expected magnitude of the layer's output during training is systematically smaller than at evaluation time, when nothing gets zeroed at all. A library'snn.Dropouthandles this with "inverted dropout" scaling — automatically, and you may never learn the scaling exists.Whether you're actually addressing overfitting, or just adding noise. Dropout, DropConnect, and Stochastic Depth are three genuinely different mechanisms — zeroing activations, zeroing weights, and skipping entire layers, respectively — and they don't have the same cost, the same test-time correction, or the same effect on a given architecture. Reaching for
nn.Dropoutby default means never learning when one of the other two would actually be the better fit.Whether your model is overfitting at all. A
Trainer.fit()call that only reports a single aggregate accuracy per epoch can hide a train/validation gap that's been widening for the last five epochs. You need to be watching for it, deliberately, to catch it before it's obvious.
None of this crashes. A model can train end-to-end, converge smoothly, and still be quietly memorizing its training set rather than learning anything that generalizes — and a high-level trainer won't tell you which one happened.
The Failure Mode: Memorization Dressed Up as Convergence
Train a moderately-sized CNN on a small dataset — a realistic constraint on a 4GB RAM laptop, where you often don't have the luxury of tens of thousands of labeled examples — with no regularization at all. The training curve looks fantastic: loss drops smoothly, training accuracy climbs past 95%, then 99%, often within the first ten or fifteen epochs. If you only watched that number, you'd call the run a success.
Validation accuracy tells a different story. It rises for the first few epochs, alongside training accuracy — genuine learning is happening early on. Then it plateaus, and on a small enough dataset, it can actually start declining while training accuracy keeps climbing toward 100%. By epoch 20, a typical run on a small synthetic dataset might show training accuracy at 99% and validation accuracy that peaked at 70% around epoch 6 and has since drifted down to 58%. The model hasn't stopped learning — it's learned the specific noise pattern of each individual training example so precisely that it's actively getting worse at generalizing while getting better at recall. This is overfitting in its purest, most measurable form: a widening train/validation gap, growing epoch over epoch, with no exception and no NaN anywhere in sight.
Keep going with the full lesson
The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.
Already a member? Sign in