Day 24: Tree Builder

Decision Trees from Scratch

The sklearn.fit() Trap

Here is how most beginners learn decision trees:

That one-liner is a production tool, not a teaching tool. It gives you an
answer but hides every question worth asking:

• How did it decide to split on petal_width <= 0.8 instead of sepal_length <= 5.5?

• What is Gini impurity, numerically, and why does minimizing it find a good split?

• Why does a depth-10 tree memorize training data while a depth-3 tree generalizes?

sklearn's DecisionTreeClassifier is 3,000+ lines of Cython. It is optimized
for production throughput, not for your understanding. When you call .fit(),
you are outsourcing your education to a black box. This lesson takes that box
apart.

The Failure Mode: What a Naive Implementation Gets Wrong

Suppose you attempt a from-scratch implementation the obvious way — a Python
loop that tries every possible threshold for every feature:

This works. On 150 samples (Iris), it finishes in milliseconds. Upload a
real dataset — 10,000 rows, 20 features — and your tree takes 8 seconds per
node to evaluate splits. With 200 nodes in a depth-7 tree, that is 26 minutes
to train. The math is right; the implementation is unusable.

There is also a subtler failure: the Gini impurity of an empty node.