Day17: Feature Engineer

The sklearn.fit() Trap

Here is how most beginners learn feature engineering:

Three lines. It works. And it costs you everything this lesson is designed to teach.

PolynomialFeatures is a black box that hands you a matrix with 0 to 153 columns (for 10 input features at degree 2) with no explanation of which column is which transformation, why some combinations blow up numerically, or what happens when a feature has zeros and you try to log-transform it. The abstraction hides the geometry of what you're doing to your data.

Here is the concrete cost: if your model underperforms after feature engineering, you have no lever to pull. You don't know if the problem is degree too high (numerical overflow), collinear features inflating the covariance matrix, or missing log transforms on skewed distributions. You're debugging a black box with a screwdriver.

This lesson removes the box entirely.

The Failure Mode

Let's say you skip the black box and try to implement polynomial expansion naively: