Day 15: Seq2Seq Translator Encoder-Decoder Architecture

AI Models Deep Learning Intermediate Edition ~7 min read 60 min hands-on

Lesson 15 — Seq2Seq Translator: Encoder-Decoder Architecture From First Principles

The Trainer.fit() Trap

The fast path: a pretrained MarianMTModel from a high-level library, .generate() called on some input text, done. This is the trap, and for encoder-decoder models specifically it hides a decision with a genuinely surprising consequence: what the decoder is allowed to see about its own past while it's being trained.

A translation decoder generates one token at a time, and each new token depends on the ones already generated. During training, you have the ground-truth target sequence sitting right there — so the obvious thing to do is feed the decoder the correct previous token at every step, regardless of what it just predicted. This is called teacher forcing, and a high-level .fit() call typically does it unconditionally, at a ratio of 100%, without ever surfacing that as a choice you made.

Here's what that silently decides for you:

  • Whether the decoder ever practices recovering from its own mistakes. At inference time, there is no ground truth to feed back in — the decoder must condition on its own previous predictions, errors included. If it never once saw its own errors during training, it has no experience handling them.

  • Whether your training loss is honest about deployment performance. Teacher-forced next-token accuracy measures something meaningfully different from full-sequence generation accuracy. A model can look excellent on the first and mediocre on the second, and a single scalar loss curve won't tell you which one you're looking at.

  • How the encoder's final state gets handed to the decoder at all. If your encoder and decoder use different hidden sizes — entirely reasonable, since they're solving different sub-problems — something has to bridge the shapes. A library picks a convention for you; build it yourself and you have to decide.

None of this crashes during training. The first two produce a working model with a training loss that looks completely normal.

The Failure Mode: A Model That Only Works With the Answer Key

Train a decoder with teacher_forcing_ratio=1.0 — always feed the true previous token, never the model's own prediction — and here's what happens. Training loss drops smoothly. Teacher-forced validation accuracy (still feeding ground truth at every step, just not updating weights) looks great, often 95%+ per-token. Everything about the numbers says the model works.

Keep going with the full lesson

The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.

Already a member? Sign in