Lesson 25:Self-Supervised Trainer Contrastive Learning — SimCLR

AI Models Deep Learning Intermediate Edition ~8 min read 60 min hands-on

Lesson 25 — Self-Supervised Trainer: Contrastive Learning From First Principles

The Trainer.fit() Trap

The fast path: copy a SimCLR implementation from a tutorial, swap in your own images, call .fit(). Contrastive pretraining doesn't need labels, which makes it feel like it should be simpler than supervised training — and that's exactly what makes it easy to copy a loss function you don't fully understand and get a model that trains, converges, and learns almost nothing useful.

Here's what a copied contrastive loss implementation hides if you don't understand the mechanics underneath it:

  • Whether the loss can be minimized by a shortcut that has nothing to do with learning good representations. The whole idea of contrastive learning is: two augmented views of the same image should look similar in embedding space, and views of different images should look different. There's more than one way to satisfy that objective mathematically, and only one of them produces embeddings that are actually useful downstream.

  • What "similar" and "different" are even measured with. Cosine similarity and raw dot-product similarity are not interchangeable choices — they respond to completely different aspects of an embedding vector, and a network trained against one will exploit whichever one you handed it.

  • How much your batch size is actually buying you. Every other image in a contrastive batch serves as a negative example. Fewer negatives means a weaker, easier task — and the memory cost of computing all those pairwise comparisons doesn't grow the way you'd expect.

None of this crashes. A contrastive model with a subtly wrong loss function trains smoothly, its loss curve drops convincingly, and the only way to discover it learned nothing useful is to actually test what it learned — which self-supervised pretraining, by design, defers to a separate evaluation step most tutorials gloss over.

The Failure Mode: A Loss That Improves While Learning Nothing

Here's the concrete version, and it's a genuinely common one. The NT-Xent loss compares embeddings via a similarity matrix — every augmented view against every other view in the batch. The textbook version L2-normalizes every embedding first, so every pairwise score is a bounded cosine similarity between -1 and 1: the only way to increase similarity between two vectors is to point them in a more similar direction. Skip that normalization step and use raw dot products instead, and the network gains a second, much easier way to shrink the loss: dot-product magnitude scales with the length of the vectors involved, not just their direction. A network can inflate the norm of embeddings for a positive pair — a purely numerical adjustment, unrelated to whether the underlying images are actually similar — and watch its dot-product similarity climb regardless of direction.

Keep going with the full lesson

The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.

Already a member? Sign in