Lesson 19-20: Neural Networks from Scratch

Introduction

Most “from scratch” neural network repositories stop at a script that prints accuracy. That is useful for learning, but it is not a system engineers can iterate on: there is no stable API surface, no durable run metadata, no clear separation between reusable core logic and orchestration, and no way to observe what the system is doing while it works.

week_19_20_aiml_integrated_project is a self-contained capstone that consolidates neural-network fundamentals into one production-style application:

• An installable NumPy-first Python package (week1920_python) holding perceptron training, activation primitives, MLP forward passes, a backpropagation trainer, MNIST loading, and a forward-only classifier

• A FastAPI backend exposing two API surfaces: a lab path for bounded topic demos and a product path for experiment lifecycle management

• Neural Network Studio — a React dashboard with a guided learning stepper, live run polling, and a product experiment manager

• Postgres-backed experiment metadata (async SQLAlchemy + Alembic), Docker volumes for artifacts and MNIST cache

The goal is system-first engineering. Perceptron training is not just a notebook cell—it is a Perceptron.train call behind a lab route with convergence metadata. XOR backprop is not a blocking script—it runs in a worker thread via asyncio.to_thread so live metrics polling stays responsive. Product runs return HTTP 202 while mnist_train_job executes training or forward smoke tests in the background.

For engineers, the payoff is traceability: lab demos stay stateless and bounded; product workflows produce Run rows with metrics_json, artifact paths, and status transitions you can query without opening log files.

Core Components

The stack is layered. Each topic below is a distinct building block; together they form one coherent neural-network workbench.

Logic gate datasets

get_and_data and get_xor_data in core/data/gates.py return 2-input boolean classification arrays. AND is linearly separable; XOR is not. These datasets power perceptron separability demos and XOR backprop training in both the lab and product paths.

Perceptron classifier

Perceptron implements Rosenblatt’s update rule with configurable learning rate and epoch cap. train returns convergence status, epochs run, and per-epoch error counts; predict_batch evaluates all samples. Used by the lab perceptron endpoint to demonstrate separability limits on AND vs XOR.

Activation functions and gradient checks

core/nn/activations.py defines ReLU, LeakyReLU, Sigmoid, Tanh, and Softmax with forward and backward passes. gradient_check compares analytical gradients to numerical estimates. The lab activations endpoint runs checks across all four primary activations and returns demo curve samples.

Multilayer perceptron forward pass

MLP builds a configurable stack with Xavier-style weight init, sigmoid output activation, forward cache for backprop, param_count, and summary. The lab MLP forward endpoint accepts comma-separated layer_sizes and returns output shapes, parameter totals, and cache depth on a 4-point logic table.

Backpropagation trainer

Trainer in core/nn/trainer.py trains a fully connected network with ReLU hidden layers and sigmoid output using MSE loss. It implements forward, backward, and weight update with a consistent dense-layer convention: weights shaped (fan_in, fan_out), computation Z = A @ W + b. Used for XOR training in lab and product run jobs.

MNIST data loading and forward classifier