vLLM Tool Strictness: Test the Schema and the Result Separately
A tool call can be valid and still answer the wrong question. A weather request for Chennai might contain a correctly typed city field but name a different city. Treat format validation and task correctness as separate checks when evaluating an agent’s serving setup.
Why the serving configuration deserves attention
In an October 20, 2026 engineering post, Ubicloud describes investigating malformed tool calls while serving open-weight models. It reports improving K2VV ToolCall schema accuracy from 92% to 100% after enabling constrained decoding in its setup. Those are the provider’s reported benchmark results, not independent measurements of every model, deployment or completed agent task.
Ubicloud: dated engineering account and reported benchmark results
Read the strictness level precisely
The vLLM v0.31.0 tool-calling documentation distinguishes these server-side controls for structural-tag-based tool calling:
- --tool-strict-level function constrains the tool-call envelope, including markup and function name; argument schemas depend on each tool’s strict setting.
- --tool-strict-level parameter additionally enforces argument schemas for every tool, as if each declared strict: true.
- With tool_choice="auto", constrained generation still permits a plain-text response; it does not require a tool call.
Parser support and configuration matter. The documentation says VLLM_ENFORCE_STRICT_TOOL_CALLING=false disables structural tags and takes precedence over the server-side floor. Match the guide to your installed version, selected parser and request configuration before changing a deployment. These settings are not a universal command recipe for every model.
vLLM v0.31.0: constrained decoding behavior and server-side strictness floor
Keep benchmark metrics separate
Moonshot AI’s vendor-verifier repository defines schema accuracy as the proportion of tool-call responses whose payload passes schema validation. It separately compares tool-trigger behavior with its official service. Read these as different measurements: agreement about when to call and conformity of a generated payload do not themselves establish that a business task was completed correctly.
Moonshot AI: tool-call metric definitions and evaluation methodology
Build a small test set with known answers
Use harmless test tools that return fixed responses and record attempted calls. Start with a few requests whose expected behavior you can specify before running the model. This is a proposed application test routine, not a reproduction of Ubicloud’s benchmark.
- A clear tool request: specify the expected function and argument values, then check both the schema and the intended values.
- A no-tool request: confirm that a normal conversational answer does not trigger an unnecessary external operation.
- A missing-information request: check whether the application asks for clarification or follows its stated fallback instead of inventing a required value.
- A multi-turn request: include an earlier tool response and check that the next call uses the intended context.
- A tool error: return a controlled failure and check whether the application reports it accurately, rather than claiming the task succeeded.
Record failures by stage
For each case, keep the raw response, parsed call, validation result, stubbed execution result and final answer. Separate malformed payloads from wrong argument values, unnecessary calls, tool failures and misleading completion claims. A single overall pass rate can hide which part of the workflow needs attention.
Compare a baseline with one controlled configuration change, keeping the model, prompt, tool definitions and test cases fixed. Repeat enough runs to see whether failures recur, and record latency and retries alongside correctness. Keep the tested release and configuration with the results so another reviewer can reproduce the comparison.
The useful acceptance condition belongs to your application: the call follows the declared contract, requests the intended operation, and produces an accurately reported outcome. Keep schema compliance as one observable part of that condition, with separate evidence for the rest.
