Day18: Dataset Splitter

Train/Val/Test Splits Done Right

The train_test_split() Trap

Here is how 90% of ML tutorials teach dataset splitting:

python

One line. It works. Ship it.

Except — do you know what stratify=y actually computes? Do you know what happens when your dataset has 950 samples of class 0 and 50 samples of class 1 and you forget that argument? Do you know the difference between a validation set and a test set, and why using the test set to pick your best epoch is the single most common form of data leakage in production ML?

The one-liner hides every one of these questions. It hides them so well that most engineers using it couldn't answer them under interview pressure — let alone debug the silent accuracy inflation it causes in production.

This lesson makes the hiding stop. We implement the splitter ourselves, in NumPy, and we watch exactly what the arrays look like at every step.

• • *

The Failure Mode: The Silent Lie

Here is the exact wrong output a naive split produces. Suppose you have a fraud detection dataset: 9,800 legitimate transactions and 200 fraudulent ones.

python