14. Recording a Replayable Agent Trace

Codex output separated command start, command completion, Agent messages, and usage. Kimi separated tool requests, tool results, and final content. Flattening these into terminal text makes it difficult to identify which model decision produced one result.

Three levels of identity

Concurrent events also need a monotonic sequence. Timestamps alone cannot order threads and hosts with clock skew or equal timestamp precision.

Normalize events, not all content into text

A trace includes context assembly, model start and completion, tool proposal, validation, authorization, start and completion, user approval or denial, state transitions, usage, and termination reason.

Payloads differ across models and tools. The outer event envelope remains stable so queries can compare turns, tool failures, and permission denials.

OpenTelemetry’s generative-AI conventions use attributes such as gen_ai.operation.name. Its Agent examples represent invoke_agent as a parent span with model calls and tool execution as children. A span is an observable operation with a start, end, and attributes. See OpenTelemetry GenAI Observability.

Two replay levels

Execution replay is risky. Sends, writes, and creates use simulated tools or require new authorization.

Version the complete runtime

Trace metadata records model and client versions, prompt-template version, tool-schema version, permission-policy version, and project-state reference. Otherwise a changed replay cannot be attributed to model or runtime change.

A stable event envelope

Every event contains schema version, type, trace identity, parent event, sequence, occurrence time, ingestion time, principal, and payload reference. The difference between occurrence and ingestion times exposes queue delay and reordering.

Readers support older event versions or apply explicit migrations. Historical source events remain append-only; derived fields can be rebuilt.

Controlling nondeterminism

Model sampling, current time, random values, network responses, and filesystem state can change output. Read-only replay substitutes saved values. Execution replay records a new environment snapshot and links the new trace as a branch.

Replay never overwrites the original. It creates a new trace_id with source trace and replay mode so comparison can mark matching, missing, added, and changed events.

Sensitive data and debuggability

Redaction occurs before persistence. Recording secrets and deleting them later enlarges the exposure window.

A replayable trace restores causal relationships from events. A complete-looking chat transcript does not.

Decision replay is not side-effect replay

Decision replay supplies saved tool results to the model again. Side-effect replay executes tools again and requires an isolated snapshot, simulator, or idempotency. Production traces do not resend email, payment, or deletion during debugging.

An OpenTelemetry span measures a bounded operation such as a model or tool call. A domain event records a state fact such as authorization denial or result acceptance. They share trace identity but are not substitutes.

OpenTelemetry's 2026 GenAI example records models, Tokens, tools, and results while treating full prompt and result content as optional sensitive telemetry. OpenTelemetry, Inside the LLM Call

Every failed trace must answer what the model saw, which tools existed, where arguments came from, who authorized, whether action occurred, how results returned, and why execution stopped.