Day12:Backprop Tracer

Backpropagation Step by Step

Backprop Tracer — AI Models From Scratch (Beginner Edition)

The loss.backward() Trap

Here is how most beginners first encounter backpropagation in PyTorch:

Three lines. Done. The network learns. You feel productive.

What you don't see: the framework constructed a directed acyclic graph of every operation performed during the forward pass, stored the local Jacobian of every node, then walked that graph in reverse — multiplying gradients together through the chain rule — and deposited the result into each parameter's .grad attribute. That is roughly 200 lines of carefully engineered C++ executing silently behind your three-line incantation.

The cost of this abstraction isn't performance — PyTorch does it fast. The cost is comprehension. When your model doesn't learn (flat loss), or explodes (NaN loss), or learns poorly (gradient vanishing), you have no map. You're debugging a black box.

This lesson gives you the map.

The Failure Mode: What Breaks Before You Even Start

Let's say you build a 3-layer network naively. You initialize all weights to zero — it seems clean, symmetric, a neutral starting point: