]> CNIC, Chinese Academy of Sciences

Beijing China

Operations and Management Operations and Management Area Working Group AgentOps agent observability anomaly detection failure attribution root-cause analysis Agentic systems execute tasks through long-running sequences of model inference, planning, delegation, tool use, state updates, verification, and recovery. Conventional metrics, logs, and traces can identify a failed request but often cannot determine whether a failure is emerging, which actor introduced it, or which earlier event was its root cause. This document specifies an AgentOps observability event model and processing requirements for interoperable early failure detection and post-failure root-cause attribution. It defines common event, anomaly, assertion, and diagnosis records; distinguishes causal origins from propagated symptoms; and provides the evidence model required by separate benchmarks of detection lead time, responsible-actor attribution, and root-cause localization. The model is transport-neutral and can be carried by existing telemetry systems.

Introduction Large language model (LLM)-based agents increasingly act as software components that plan, call tools, coordinate with other agents, update memory, inspect environmental feedback, and revise intermediate results. An execution is therefore not a single request-response pair. It is a distributed, stateful trajectory involving heterogeneous actors and a mixture of deterministic and probabilistic operations. Traditional operations telemetry remains necessary, but it is not sufficient for this setting. A late tool error or failed verification may be only the visible symptom of an earlier flawed plan, incorrect handoff, state loss, hallucinated assumption, or ignored constraint. Furthermore, telemetry generated by an agent can itself be incomplete or incorrect. Operational systems need evidence that supports two different functions:

• online detection of a deviation early enough to permit mitigation; and
• post-failure attribution of the responsible actor and earliest decisive root-cause event.

These functions MUST NOT be conflated. An anomalous event is not necessarily causal, and a causal event need not have the largest anomaly score. Long-horizon evidence also creates a practical context problem for LLM-based diagnostic systems: a complete trajectory may contain hundreds of steps, while the relevant evidence is sparse and separated from the final failure by many intervening events. This document defines a common event model that preserves semantic, topological, temporal, and responsibility-transfer evidence. It also defines interoperable records for anomaly signals, operational assertions, and final diagnoses. Detection and attribution algorithms are outside the scope of this document. The design is informed by work on agent-and-step failure attribution , anomaly-first trajectory filtering , and long-horizon root-cause benchmarks . It is intended to complement, rather than replace, distributed tracing and telemetry semantic conventions such as and .

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Scope This document specifies:

• a logical event envelope for agentic executions;
• minimum evidence needed for early failure detection;
• a representation for anomaly and policy-assertion signals;
• a representation for actor and root-cause attribution;
• rules for causal-chain roles and delegation-aware responsibility; and
• evidence fields needed by AgentOps diagnosis benchmarks.

This document does not specify a telemetry transport, storage engine, sampling algorithm, anomaly-detection model, or automated remediation policy. It does not require disclosure of model weights, hidden chain-of-thought, or plaintext user content.

Terminology

Agentic system:

A software system in which one or more model-driven actors select or execute actions over multiple steps, possibly using tools, memory, delegation, and environmental feedback.

Actor:

An entity that participates in an execution. An actor can be an agent, orchestrator, model, tool, verifier, human, or environment component.

Trajectory:

The ordered event graph associated with one agentic task or session. A trajectory can contain branches, joins, retries, and nested calls.

Root-cause event:

The earliest decisive event that introduced a defect without which the observed failure would not have occurred under the recorded execution.

Responsible actor:

The actor accountable for the root-cause event. This can be an agent, a non-agent component, or the external input or environment.

Propagation event:

An event that carries forward an already introduced defect without independently introducing the decisive defect.

Exposure event:

An event that makes a latent defect observable, such as a failed test or verifier rejection.

Terminal event:

The event at which the system declares or irrecoverably reaches failure.

Early failure detection:

Production of an anomaly or assertion signal before the terminal event, with sufficient evidence to support an operational response.

Trust level agreement (TLA):

An application-defined set of runtime confidence or safety levels and assertions that establish operational boundaries for an agentic system.

AgentOps Observability Architecture An implementation typically contains the following logical roles:

1. Event producers instrument agents, orchestrators, model gateways, tools, memory systems, verifiers, and environments.
2. A collector correlates and stores event records without changing their causal identifiers.
3. A detector compares the partial trajectory with one or more baselines and emits anomaly or assertion records.
4. An attribution engine retrieves evidence, reconstructs propagation, and emits a diagnosis record.
5. An operator or automated controller decides whether to continue, checkpoint, retry, reroute, constrain, or terminate the task.

The same component can perform more than one role. The architecture does not require a central collector, but all records used together MUST share a stable session identifier and SHOULD share distributed trace identifiers. Event producers SHOULD reuse W3C Trace Context identifiers when an event is part of a distributed trace. An AgentOps event is more specific than a generic span: it also records actor identity, agentic phase, action kind, delegation, state references, and evidence needed to distinguish causal origin from downstream manifestation.

Event Model

Event Envelope Each event record MUST contain:

• schema_version, identifying the version of this event model;
• event_id, unique within the telemetry domain;
• session_id, stable across the complete task trajectory;
• timestamp, formatted according to ;
• sequence, a monotonically increasing value within the producing actor;
• actor, identifying the actor and its operational role;
• phase, identifying the agentic lifecycle phase;
• event_type, identifying the operation represented by the event; and
• status, describing the event outcome known at emission time.

Each event SHOULD contain trace_id, span_id, and parent_span_id when distributed trace context exists. It SHOULD contain parent_event_ids when the execution topology cannot be represented by a single parent span. The sequence field orders events from one producer. Consumers MUST NOT assume that it establishes a total order across producers. Consumers SHOULD use causal links, timestamps, and producer-local sequence numbers together.

Actor and Phase The actor.kind value MUST be one of human, agent, orchestrator, model, tool, verifier, environment, or other. The actor.id value MUST be stable within the session. The actor.role value SHOULD describe the functional role rather than a vendor-specific class name. The phase value MUST be one of:

• pre-execution: request interpretation, constraint extraction, planning, actor selection, and initial state construction;
• execution: inference, delegation, tool use, memory access, state update, and intermediate verification; or
• post-execution: final verification, result publication, diagnosis, remediation, and audit.

Event Types This document defines the following initial event types: Initial AgentOps Event Types

Event type	Meaning
session.start, session.end	Task or session boundary
task.received	Input and constraints accepted
plan.created, plan.revised	Plan or decomposition changed
agent.delegated, agent.returned	Responsibility transferred or returned
model.request, model.response	Model interaction boundary
tool.call, tool.result	Tool interaction boundary
memory.read, memory.write	Persistent or session state access
checkpoint.created, checkpoint.restored	Recoverable state boundary
assertion.evaluated	TLA or policy assertion evaluated
verification.result	Intermediate or final verification
recovery.attempt	Retry, rollback, reroute, or repair action
human.intervention	Human decision or correction
anomaly.signal	Detector output
diagnosis.result	Attribution engine output

Unknown event types MUST be preserved by collectors and MUST NOT cause the containing trajectory to be rejected. Their semantics are extension data.

Evidence and Content Handling Events SHOULD record structured attributes required for diagnosis, including action name, input and output references, tool result status, model and tool versions, token usage, duration, delegation target, verifier result, and checkpoint reference when applicable. Large or sensitive content SHOULD be represented by an evidence_ref containing a content hash, media type, access-controlled location, and sensitivity label. A collector MUST NOT dereference protected evidence unless authorized for the relevant purpose. Implementations MUST NOT require hidden model chain-of-thought. An implementation MAY record a model-generated decision summary, reflection, or explanation when the model and deployment policy permit it. Such text MUST be labeled as self-reported evidence and MUST NOT be treated as an authoritative description of internal model computation.

CDDL Definition The following Concise Data Definition Language (CDDL) fragment defines the JSON-compatible logical record. JSON serialization MUST follow .

AgentOps Core Event Model any } } actor = { id: tstr, kind: "human" / "agent" / "orchestrator" / "model" / "tool" / "verifier" / "environment" / "other", ? role: tstr, ? instance: tstr, ? version: tstr } action = { kind: tstr, ? name: tstr, ? input_ref: evidence-ref, ? output_ref: evidence-ref, ? duration_ms: uint, ? token_input: uint, ? token_output: uint } delegation = { from_actor: tstr, to_actor: tstr, instruction_ref: evidence-ref, ? constraints_ref: evidence-ref } component = { name: tstr, ? provider: tstr, ? version: tstr, ? call_id: tstr } evidence-ref = { id: tstr, ? media_type: tstr, ? hash: tstr, ? location: tstr, ? sensitivity: "public" / "internal" / "confidential" / "restricted", ? provenance: "observed" / "self-reported" / "derived" } ]]>

Early Failure Detection

Baselines and Partial-Trajectory Evaluation A detector operates on a partial trajectory and compares it with an explicit baseline. The baseline can represent successful executions, policy-conforming executions, a known model or tool version, or an application-defined TLA level. A detector SHOULD consider both:

• semantic evidence, such as actor role, operation, constraint, tool result, and state transition; and
• structural evidence, such as delegation topology, call tree, branch, retry, checkpoint, and dependency relationships.

This dual view is important because an individually plausible action can be anomalous in its topological or role context, while a structurally unusual action can be semantically correct. The specification does not mandate a particular detector. Reconstruction, prediction, assertion, spectrum, statistical, and hybrid detectors can all produce conforming anomaly records.

Anomaly Signal An anomaly record MUST identify the detector, baseline, detection time, score, and candidate event set. It MUST state whether a higher or lower score indicates greater deviation. It SHOULD include the threshold, feature groups, confidence, and the earliest candidate event.

Anomaly Signal Record

The candidate_event_ids array is a ranked set. A detector SHOULD emit a compact, evidence-preserving candidate set rather than copying the complete trajectory into the anomaly record. The original events MUST remain retrievable for audit and attribution.

Assertions and Trust Levels Application assertions provide a deterministic complement to statistical detection. An assertion record MUST identify the applicable TLA level, assertion identifier, evaluation result, and events used as evidence.

Trust Level Assertion Record

The approaching result supports risk prediction before a hard boundary is crossed. Trust levels and boundaries are application-specific; this document standardizes their observability, not their values.

Root-Cause Attribution

Attribution Principles An attribution engine MUST distinguish at least four causal-chain roles: root-cause, propagation, exposure, and terminal. It MUST NOT select an exposure or terminal event solely because it is closest to the observed failure. Attribution SHOULD apply the following responsibility rules:

• If an executor faithfully implements a defective delegated instruction, the delegating actor and instruction event are the primary root-cause candidates. The executor event is propagation evidence.
• If the executor departs from a valid instruction or introduces a new defect, responsibility remains with the executor.
• A verifier that exposes a defect is not the root cause. A verifier that misses a detectable defect can be a contributing factor.
• If an external requirement is self-contradictory or an environment failure is unrecoverable, a non-agent source can be the responsible actor.
• Among causally sufficient candidates, the engine SHOULD select the earliest decisive event for which a feasible recovery existed before the failure propagated.

These rules allow responsibility to move across planner-executor-verifier handoffs without automatically blaming either the first or last actor.

Diagnosis Record A diagnosis record MUST identify the responsible actor, root-cause event, terminal event, supporting evidence, and method. It SHOULD include the failure chain, category, confidence, and validation status.

Root-Cause Diagnosis Record

The evidence_event_ids values MUST resolve to events in the original trajectory. Derived summaries can be included through rationale_ref, but they MUST NOT replace the source evidence.

Failure Categories Implementations MAY use the following initial analysis taxonomy. The category is auxiliary; responsible-actor and root-cause-event labels remain the primary interoperable targets. Initial Failure Category Classes

Prefix	Class	Example leaf categories
A	Input understanding	misunderstood intent, ignored constraint, misread observation
B	Reasoning	fabrication, flawed reasoning
C	Planning and collaboration	flawed plan, wrong handoff, cross-actor state loss
D	Execution and tools	wrong tool or arguments, unrecovered tool failure, repetition
E	Verification and completion	verification gap, premature completion
X	Non-agent	contradictory input, unrecoverable environment

Future specifications can define stable category identifiers. Collectors MUST preserve unknown categories.

Benchmarking Considerations This document standardizes observability evidence, not a task suite or scoring methodology. AgentOps early-detection and root-cause-diagnosis benchmark requirements are specified separately in . In particular, an AgentOps diagnosis benchmark is distinct from a benchmark of an agent that performs network configuration or other network operations. The latter evaluates the agent's task capability and network outcome; the former evaluates whether an operational method can detect and diagnose a failure in an agentic system. A network-management-agent trajectory can be encoded using this event model and used as a diagnosis workload, but its network-task score is not an AgentOps diagnosis score.

Operational Processing Requirements

Collection and Sampling Sampling decisions can destroy the evidence required for diagnosis. Implementations SHOULD retain task boundaries, delegation edges, assertion violations, anomaly candidates, verifier outcomes, recovery attempts, and checkpoint references whenever any event in the session is retained. When full payloads cannot be retained, producers SHOULD emit stable hashes and access-controlled evidence references. Head sampling alone is generally insufficient for failed trajectories; collectors SHOULD support a deferred or tail-based retention decision triggered by anomalies, assertion violations, or terminal failures.

Clock and Ordering All timestamps MUST follow and SHOULD be expressed in UTC. Implementations SHOULD record clock-quality information when cross-host ordering affects diagnosis. Causal parent links take precedence over wall clock order when the two conflict.

Schema Evolution Major incompatible revisions MUST change schema_version. Consumers MUST ignore unknown optional fields and preserve them when forwarding records. Required fields defined by a future major version MUST NOT be assumed by a consumer that only supports this version.

Privacy Considerations Agent trajectories can contain user prompts, personal data, credentials, proprietary source code, retrieved documents, model outputs, and behavioral profiles. Centralizing this material creates privacy risks described more generally in . Producers MUST support field-level redaction or evidence references in place of plaintext content. Access to evidence SHOULD be purpose-limited, logged, and time-bounded. Benchmark publishers MUST remove credentials and direct identifiers and SHOULD evaluate whether trajectory combinations can re-identify users or organizations. Actor identifiers SHOULD be pseudonymous outside the operational trust domain. Content hashes can still reveal low-entropy values through guessing and therefore MUST NOT be treated as anonymization by themselves. Raw prompts, outputs, memory values, and self-reported reasoning summaries SHOULD NOT cross an administrative boundary unless needed for a declared diagnostic purpose and protected by an applicable policy.

Security Considerations AgentOps telemetry is security-sensitive. It can expose instructions, system topology, tool capabilities, failure patterns, and recovery controls. The general guidance in applies. Telemetry producers and collectors MUST provide confidentiality, integrity, and origin authentication appropriate to the deployment. Diagnosis systems MUST treat agent-generated explanations and attributes as untrusted input. An agent or tool can attempt to hide a failure, frame another actor, inject instructions into a diagnostic model, or create excessive telemetry. Collectors SHOULD separate observed facts from self-reported and derived claims using the provenance field. Attribution engines SHOULD validate cited event identifiers and SHOULD prefer directly observed tool, verifier, and state-transition evidence over unsupported natural-language claims. Implementations MUST apply resource limits to record size, evidence retrieval, candidate-set expansion, and recursive delegation graphs. Automated remediation SHOULD require policy authorization proportional to impact. In particular, an anomaly score alone MUST NOT authorize destructive actions. Replay and reordering can corrupt causal reconstruction. Deployments SHOULD use tamper-evident storage, stable identifiers, producer-local sequence numbers, and duplicate detection. Cross-domain trace identifiers MUST NOT be accepted as proof of trust or identity.

IANA Considerations This document has no IANA actions.

Implementation Status This section is to be removed before publication as an RFC. The event model is based on telemetry fields used in ongoing AgentOps experiments for anomaly-first failure attribution and long-horizon root-cause localization. No interoperability implementation is claimed for version -00. Future revisions are expected to report mappings to existing OpenTelemetry representations and validation of the CDDL examples.

References Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. This document defines a date and time format for use in Internet protocols that is a profile of the ISO 8601 standard for representation of dates and times using the Gregorian calendar. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. JavaScript Object Notation (JSON) is a lightweight, text-based, language-independent data interchange format. It was derived from the ECMAScript Programming Language Standard. JSON defines a small set of formatting rules for the portable representation of structured data. This document removes inconsistencies with other specifications of JSON, repairs specification errors, and offers experience-based interoperability guidance. This document proposes a notational convention to express Concise Binary Object Representation (CBOR) data structures (RFC 7049). Its main goal is to provide an easy and unambiguous way to express structures for protocol messages and data formats that use CBOR or JSON. Informative References All RFCs are required to have a Security Considerations section. Historically, such sections have been relatively weak. This document provides guidelines to RFC authors on how to write a good Security Considerations section. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. This document offers guidance for developing privacy considerations for inclusion in protocol specifications. It aims to make designers, implementers, and users of Internet protocols aware of privacy-related design choices. It suggests that whether any individual RFC warrants a specific privacy considerations section will depend on the document's content. World Wide Web Consortium OpenTelemetry Authors Authors withheld during double-blind review Authors withheld during double-blind review

Example Records

Early Anomaly Example The following example reports a deviation at an early delegated planning event. The payload is illustrative and not derived from benchmark data.

Illustrative Early Anomaly Record

Attribution Example

Illustrative Root-Cause Attribution Record

Acknowledgements The author thanks the researchers and practitioners whose AgentOps, multi-agent failure attribution, and long-horizon diagnosis work informed this document.