Day 14: News Classifier Text Classification with TF-IDF + Deep
Lesson 14 — News Classifier: TF-IDF + Deep Networks From First Principles
The Trainer.fit() Trap
The fast path: TfidfVectorizer().fit_transform(all_the_text) from a library, straight into a classifier, no second thought about what "fit" touched. Minutes later you have a news-category classifier with 91% validation accuracy. This is the trap, and in TF-IDF pipelines specifically it hides a decision that has nothing to do with your neural network at all: which documents were allowed to influence your feature statistics.
TF-IDF isn't a neural network layer — it has no weights, no gradients, nothing optimizer.step() ever touches. It's classical feature engineering: for every word, how often does it appear in this document (term frequency), and how rare is it across the whole corpus (inverse document frequency)? That second number is a corpus-wide statistic, and it's exactly the kind of thing that's trivially easy to compute over more documents than you should.
Here's what a one-line .fit_transform() call decides for you, silently:
Whether your validation set contributed to the vocabulary. If "fit" runs on train and validation documents together, the document-frequency counts underlying every IDF weight already know how rare each word is in the data you're about to evaluate on.
Whether your validation set contributed to the IDF weights themselves. Even without vocabulary leakage, if validation documents are counted when computing how many documents contain a given word, the resulting weight is not purely a property of the training distribution anymore.
How large your feature vector gets. A
.fit_transform()call defaults to using every distinct token as a feature unless you explicitly cap it — and for a real news corpus, that's tens of thousands of columns, most of which are typos, names, and words that appear exactly once.
None of this crashes. All of it quietly changes what your validation number actually measures.
The Failure Mode: A Validation Score That Lies
Here's the naive version of this bug, and it's a genuinely common one: you split your data into train and validation, but you compute TF-IDF statistics — the vocabulary and the inverse-document-frequency weights — on the union of both splits, because the vectorizer call happens once, before the split, out of convenience. Every downstream number looks completely normal. Training converges. Validation accuracy climbs to something that looks publication-ready — say, 91%.
Keep going with the full lesson
The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.
Already a member? Sign in