04. How Tool Schemas Change Agent Behavior

For the same read-only task, Codex produced one compound Shell command while Kimi produced two parallel Bash calls. The difference cannot be attributed only to model style because the models received different tool interfaces.

A tool schema is the structured contract containing a tool name, purpose, argument types, required fields, and value bounds. The model uses it to select an action and form arguments. The executor validates the proposed call against the same contract.

A definition shapes capability and choice

A tool named run_anything with one arbitrary string argument forces the model to construct a command and leaves little structure for pre-execution analysis. Separate read_file(path) and list_directory(path, limit) tools expose intent in their arguments.

Longer descriptions are not automatically better

A description answers what the tool does, when it applies, and when it must not be used. Too little text makes neighboring tools ambiguous. Too much text increases fixed context and scatters enforceable policy through natural language.

A directory tool should state whether it is recursive, whether it includes hidden entries, its maximum result count, the permitted root, and its symbolic-link policy. The last two cannot exist only in prose. The executor must enforce them.

Validation occurs at two layers

Structural validity is not authorization. {"path":"/etc/passwd"} can satisfy a string schema while violating workspace policy.

A controlled schema comparison

Keep model, task, and fixture unchanged:

Run each group 30 times. Record correct tool choice, argument-validation failures, model turns, and input Tokens. One trace explains the mechanism; repeated runs show whether the schema changes behavior consistently.

Results also need a contract

A plain string cannot reliably distinguish success, an empty result, truncation, and failure. A result contract includes status, duration, error class, truncation state, and content.

Kimi’s two tool messages referred to Bash_0 and Bash_1 through tool_call_id. This identifier returns each parallel result to the correct request. The tool name selects the handler; the call identifier identifies one action instance.

Schema versions belong in the trace

An unchanged tool name does not imply an unchanged contract. Adding a required field, narrowing an enum, or changing path semantics can make identical model output behave differently. Every tool.requested event records the schema version used for validation.

Compatibility tests cover three cases: adding an optional field should accept old calls; adding a required field should produce a structural error; tightening a policy range should produce a policy denial rather than a generic execution failure.

Negative cases test boundaries

A positive case shows that the model can call a tool for a clear task. Negative cases test whether it avoids calling when the user asks only for an explanation, a path is outside the workspace, limit exceeds the bound, or the task requires recursion that the tool does not provide.

Tool-selection errors, argument-structure errors, and policy denials are reported separately. They correspond to descriptions, schemas, and authorization. Combining them into one failure rate removes the information needed to improve the runtime.

The tool schema is therefore not secondary API documentation. It defines the Agent’s action space and provides the entry point for permission checks and trace replay.

A schema changes the model's action space

Under strict tool output, JSON Schema can restrict valid continuations to a smaller set than ordinary prose. The action space is the set of operations the model can express in the current state.

Strict structure does not prove semantic correctness. {"path":"README.md","limit":5} can pass every structural check while selecting the wrong resource. Structural validity and task correctness need separate metrics.

Names and descriptions also alter action distinguishability. Two tools named “search” require the model to infer whether one searches local code and another uses the internet. Executor-side typing cannot repair that selection problem.

Tool granularity creates opposing costs

Separate read_file, list_directory, and search_text tools make intent and authorization explicit. Hundreds of similar tools consume context and increase ambiguity. One generic shell tool has a short schema but delegates quoting, path handling, side effects, and error interpretation to the model.

total cost = selection cost + argument-construction cost + execution risk

Overly fine tools raise selection cost. Overly broad tools raise argument and execution risk. A useful boundary is an operation that can be authorized, retried, and reported independently.

Anthropic describes tool use as a contract between model and application: the model emits a request, while execution occurs in client or server code. Anthropic, How tool use works MCP requires a name, description, and input schema and can expose an output schema, but client authorization remains separate. MCP Tools specification

A randomized schema-ablation experiment

Run A–D variants in randomized order. Otherwise provider changes, cache state, or rate limits can coincide with the schema change. Each trace records model version, sampling settings, fixture snapshot, and schema version.

Grade every run at four levels:

1. It selects a tool capable of completing the task.
2. It produces structurally valid arguments.
3. The arguments identify the correct resource and intent.
4. Execution returns evidence needed by the task.

A call may pass level two and fail level three by reading a neighboring file. A parse-success metric would count that as success. Include a no-tool baseline: a model may guess a plausible directory listing, while the filesystem trace proves it acquired no current evidence.

Dynamic loading changes the experiment

For large toolsets, Anthropic's current guidance uses Tool Search to keep definitions out of the initial context. The model searches a catalog and matched definitions enter the trajectory later. This reduces baseline context but adds a discovery stage. Anthropic, Manage tool context

A replayable trace records catalog version, query, matches, and loaded schema versions. Saving only the final tool list cannot show whether the model lacked the tool or saw it and declined to use it.

Schema is not the entire security contract

JSON Schema expresses types, enumerations, formats, and local conditions. It does not decide whether the current principal may read a path or whether a destination may receive data derived from a secret.

Tool design should make all three independently observable rather than forcing every rule into one schema.