Lesson 29: Performance Profiler Model Benchmarking & Profiling
The %timeit Trap
Here's the beginner path into performance work: wrap the model call in a
notebook magic or a bare stopwatch.
Run that cell once and you get a number. It feels like an answer. What it
actually hides: whether that one call happened to be the model's first
call ever, whether the measurement includes Python interpreter overhead
and CPython's own dispatch cost, and — the question that matters most —
where inside the model that time went. A single wall-clock number aroundmodel(x) can tell you the forward pass took 15 milliseconds. It cannot
tell you whether 45% of that was one MaxPool2d layer or a convolution,
and that distinction decides what you'd actually change to make it
faster.
This lesson builds two tools by hand: a warmup-aware latency benchmark
that separates one-time startup cost from steady-state cost, and an
operator-level profile using torch.profiler that ranks exactly which
ATen operations consume the CPU time a naive timer only reports in
aggregate.
The Failure Mode: A Real, Measurable, Silent Inflation
Benchmarking without a warmup phase doesn't crash — it returns a number
that's simply wrong, confidently. Timing the very first call into a fresh
model on this machine, then 20 subsequent calls to the same model,
produced results like these across several runs:
A consistent 25–40% inflation on the very first call, every time — and on
one run, under contention from other processes on the same machine, that
first call took over 100 times longer than steady state. The cause is a
set of one-time costs a fresh model pays exactly once: thread-pool
spin-up, oneDNN/MKL-DNN selecting and caching which low-level kernel
implementation to use for each op shape, and first-touch memory allocation
that hasn't been paged in yet. None of these recur on the second call.
Average them into a 20-iteration benchmark anyway, and the reported mean
latency describes a cost that only the very first real request will ever
pay — everyone after that gets the faster, unreported number.
Keep going with the full lesson
The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.
Already a member? Sign in