Lesson 27 — Auto-Tuner: Hyperparameter Optimization with Optuna

AI Models Deep Learning Intermediate Edition ~6 min read 60 min hands-on

The study.optimize(...) Trap

Here's the beginner path into hyperparameter tuning: pip install optuna,
wrap your model in a function, and write four lines.

It runs. Fifty trials execute, a best_params dict comes out the other
end, and it feels like you've automated away the tedious part of deep
learning. What that four-line block hides: whether trial.report() gets
called anywhere inside train() — and if it doesn't, should_prune()
returns False on every single call, for every single trial, forever.
Optuna's pruner isn't broken; it was never given anything to prune on. A
hyperparameter combination that diverges to NaN in epoch 1 still runs
all 50 epochs of train(), because nothing ever told the study it was
allowed to stop early. You paid for 50 full trainings and only 12 of them
were ever worth running to completion.

This lesson builds the training loop study.optimize() calls internally,
by hand — architecture depth and width as search dimensions, not just
learning rate, a real pruning signal reported every epoch, and the
specific failure mode of skipping that signal made visible with a stopwatch.

The Failure Mode: A Confident Answer From an Underfed Search

A naive Optuna integration doesn't crash either. Two specific ways it goes
quietly wrong:

Pruning that never engages. MedianPruner compares a trial's
intermediate value, at a given step, against the median of every prior
trial's value at that same step — but only if something calls
trial.report(value, step) inside the training loop. Forget that one line
and trial.should_prune() is permanently False. A learning rate of 3.0
that blows up the loss in epoch 1 still trains for all 4 configured epochs
before Optuna sees a final score. Run 20 trials at 4 epochs each with a
search space that includes a few bad learning rates, and you'll burn 80
epoch-equivalents of CPU time when 30 would have been enough.

Keep going with the full lesson

The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.

Already a member? Sign in