Day 9: Gradient Descent Lab

The optimizer.step() Trap

Every modern deep learning tutorial eventually reaches this line:

One line. Works perfectly. And teaches you absolutely nothing about what just happened to your weights.

Here is what a beginner would do to "learn" gradient descent using a library:

Done in three lines. Scipy's minimize handles the gradient, the step size, the convergence check, the line search, the Hessian approximation — all of it, invisibly. You got the answer. You learned nothing.

The cost of that abstraction is concrete: when your loss curve explodes during real training — and it will — you will have no mental model of why. You won't know if the learning rate is too high, if the gradient is numerically unstable, if you've initialized in a flat region, or if you're stuck in a local minimum. The library absorbed all that signal and returned a single number. This lesson gives that signal back.

The Failure Mode

Let's build gradient descent naively first, so you can see exactly how it breaks.