Day4: Derivative Driller

Derivative Driller

Calculus Intuition for AI — Seeing the Slope That Drives Learning

The sklearn.fit() Trap

Open any introductory ML tutorial and you'll find gradient descent described in one sentence: "the optimizer minimizes the loss." Then model.fit(X, y) appears, and the lesson moves on. That sentence isn't wrong — but it hides the one mechanism that every neural network, every optimizer, and every training loop depends on: the derivative.

The derivative of a loss function with respect to a weight tells the optimizer exactly how to adjust that weight to reduce the loss. Not approximately. Not heuristically. Exactly — to first order. When you call optimizer.step() in PyTorch, you are applying weight -= lr * dL/dweight, where dL/dweight is a partial derivative computed via automatic differentiation. The framework computes it for you, invisibly.

That invisibility is the trap. You can ship a working model without ever understanding what a derivative is in array form, why step size matters, or what happens when the gradient signal vanishes or explodes. This lesson strips the abstraction away. We implement numerical differentiation from scratch using NumPy, build a live function plotter with a draggable tangent line, and connect every number on screen directly to the calculus that makes optimizers work.

The Failure Mode: When h Is Wrong

The standard formula for a numerical derivative is deceptively simple:

This is the central difference approximation. The "2h" in the denominator is not optional. Here is what happens when a beginner implements the forward difference instead: