11. Which Layer Should Handle an Agent Error?

Claude Code returned 403 and Grok returned 402 in the capture. Neither reached tool execution. Recording only task_failed would mix these traces with a local command failure.

Classify by failure location

The layer identifies who can repair the problem. Rephrasing cannot fix account quota. Restarting an executor cannot add a missing argument.

Retry requires recoverability

Exponential backoff increases the delay after each attempt and is suitable for temporary rate limits and server failures. It is not suitable for denial, deterministic validation errors, or exhausted quota.

Logs, models, and users need different representations

The durable log stores error class, original detail, call identifier, duration, and retry count. The model receives a stable code, necessary explanation, and allowed next actions. The user sees impact, a concrete recovery action, and relevant evidence.

A raw stack trace wastes context and can expose internal paths. “Failed” alone gives the model no basis for a decision.

Partial failure preserves success

When one parallel call succeeds and another fails, both states remain. The model may complete a partial task, and the runtime retries only the failed item.

Collapsing the batch into failure discards completed work and can duplicate it during retry.

Backoff needs jitter and a budget

When many Agents receive a rate limit together, fixed delays synchronize their next attempts. Jitter adds a bounded random component to reduce that retry wave.

Honor Retry-After when the service provides it. Otherwise use exponential backoff with jitter. Each attempt records error class, delay, and remaining total retry budget so one tool cannot consume the whole task.

Error classification is tested

An adapter can misclassify 403 as a temporary server error or wrap a timeout as an argument error. A regression corpus contains real or simulated responses and asserts normalized error code, retryability, side-effect state, and user-facing action.

When classification is uncertain, an unknown non-automatic-retry state is safer than treating every unknown error as temporary.

Termination conditions

Every task limits model turns, retries per tool, wall-clock time, total Tokens, and repeated instances of the same error. Reaching a limit produces an explicit terminal state and names the incomplete step.

Without termination, a deterministic error becomes continuing cost.

The two failed CLI requests do not establish product capability. They establish that requests reached different account or organization checks. Before comparing answers, verify that traces reached the same runtime stage.

Retry answers three questions

Can the cause disappear without changing the request? Is repetition safe? Do time, attempt, and cost budgets permit it? A 429 may be temporary without making a possibly accepted write safe to resend. A validation error does not heal with time but can continue after argument correction.

Exponential backoff increases delay; jitter prevents many Agents from retrying together. Reports separate attempts, model corrections, waiting, execution, and termination reason.

Tool errors return a machine code, argument path, safe summary, and protected diagnostic reference. A full stack can expose secrets; “failed” cannot support correction.

The fault matrix covers schema, permission, missing resource, rate limit, timeout, server error, result parsing, and cancellation. Every cell specifies modify, wait, query status, request authorization, stop, or escalate.