Internet-Draft Agent Runtime Telemetry System July 2026
Wang, et al. Expires 7 January 2027 [Page]
Workgroup:
Network Working Group
Internet-Draft:
draft-wang-agent-runtime-telemetry-system-00
Published:
Intended Status:
Informational
Expires:
Authors:
X. Wang
CAICT
X. Liu
CAICT
Z. Li
CAICT
M. Zhao
CAICT

Agent Runtime Telemetry System

Abstract

Large language model (LLM)-driven agents are increasingly deployed in large-scale production environments, where multi-agent collaboration, chained tool invocation, and long-context reasoning have become standard execution patterns. However, existing distributed systems telemetry frameworks primarily focus on infrastructure-level telemetry, such as network traffic, host metrics, and service health, and provide limited visibility into internal agent reasoning and execution semantics.This results in several recurring challenges in production environments, including inconsistent metric definitions across platforms, lack of end-to-end execution traceability, difficulty in root cause analysis, and limited interoperability among heterogeneous telemetry systems.

This document defines a unified capability framework for Agent Runtime Telemetry Systems. It standardizes two telemetry collection planes: Network-side collection at gateways and traffic ingress layers, and Agent-side instrumentation within the agent runtime environment. It further specifies structured telemetry models, cross-layer trace correlation mechanisms, runtime metrics, and anomaly detection and remediation workflows.

The framework enables interoperable telemetry across heterogeneous agent systems and deployment environments. It supports unified telemetry ingestion, end-to-end execution trace reconstruction, behavioral telemetry, and closed-loop anomaly analysis, providing a standardized foundation for agent operations, reliability engineering, and compliance auditing.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 7 January 2027.

▲

Table of Contents

1. Introduction

With the rapid advancement of large language models, AI agents are increasingly deployed in enterprise production systems. These systems have evolved from isolated applications into complex distributed architectures composed of multiple agents, model services, tool execution services, and external system dependencies. This evolution introduces new challenges in runtime telemetry, behavior tracing, failure localization, and unified operational management.

Compared to traditional distributed applications, agent execution workflows exhibit fundamentally different runtime characteristics. A single task execution may involve multiple dynamically composed stages, including model inference, tool invocation, context management, long-term memory access, and multi-agent collaboration. These execution paths are stateful, non-linear, and tightly coupled across system components, making them difficult to observe using conventional infrastructure-centric telemetry approaches.

Existing telemetry systems for agent-based workloads are insufficient in several key aspects. First, most systems focus on coarse-grained runtime metrics such as service availability, request throughput, and system resource utilization. They lack visibility into fine-grained execution semantics, including reasoning steps, tool invocation chains, and failure causality, limiting their effectiveness for debugging and operational optimization. Second, current telemetry models are primarily designed for traditional distributed systems. They emphasize network-level requests and service metrics but do not provide a unified representation for agent-native behaviors, such as inference traces, context evolution, memory operations, and inter-agent communication. As a result, they fail to capture the semantic structure of agent execution workflows. Third, the ecosystem remains fragmented, with no widely adopted standard for agent telemetry collection, trace representation, or cross-platform telemetry interfaces. This fragmentation leads to poor interoperability across frameworks and vendors, making it difficult to implement unified telemetry in large-scale multi-agent deployments.

This document defines a standardized capability framework for Agent Runtime Telemetry System. It specifies requirements across four dimensions: Agent Data Collection, Runtime monitoring, Behavior Tracking, Anomaly Analysis & Remediation.

The framework is designed to support heterogeneous agent frameworks and multi-deployment environments. It enables consistent telemetry collection, unified metric semantics, end-to-end trace reconstruction, and closed-loop anomaly analysis, thereby providing a foundation for telemetry-driven agent operations, reliability engineering, and compliance governance.

2. Conventions and Definitions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

3. Terminology

The following terms are defined in this document:

4. Overview of Agent Runtime Telemetry System

This document defines four layers of standardized telemetry technical capabilities with hierarchical dependencies and mutual coordination, following a complete technical pipeline: data collection, real-time metric monitoring, post-hoc behavior tracing, and anomaly analysis.

This document defines the capability framework of the Agent Runtime Telemetry System, covering four dimensions: Agent Data Collection, Runtime Monitoring, Behavior Tracking, and Anomaly Analysis & Remediation.

Agent Data Collection serves as the foundational telemetry layer of the telemetry system. It standardizes two data collection planes--Network-side and Agent-side instrumentation--and defines unified constraints for identifiers, transmission mechanisms, and base data structures.This layer aligns with the global identification system defined in Chapter 7, including trace_id, span_id, and session_id, providing a unified data foundation for higher-level monitoring, tracing, and analytical capabilities.

The Runtime Monitoring captures real-time operational state and resource consumption during agent task execution, providing online situational awareness of system behavior.It consists of four subdomains: Token Usage Monitoring, Model Monitoring, Service Quality Monitoring, and Multi-Agent Collaboration Monitoring.

The Behavior Tracing reconstructs complete execution traces after task completion in a structured manner, enabling interpretability, failure reproduction, and compliance auditing.It achieves full lifecycle reconstruction of agent reasoning logic, invocation behavior, and interaction processes through decision trace tracking, tool call tracing, and session interaction observability.

The Anomaly Analysis & Remediation provides a closed-loop analytical system based on telemetry from the previous three layers, enabling anomaly detection, root cause analysis, and coordinated mitigation actions.This capability enables continuous detection and governance of anomalous agent behavior through baseline deviation analysis, root cause localization, anomaly alerting, and an anomaly-based threat library.

5. Agent Data Collection

The Agent Data Collection focuses on standardized collection mechanisms for telemetry data throughout the Agent lifecycle, forming the data foundation of the Agent Runtime Telemetry System. Its objective is to acquire various types of telemetry signals required for agent operation through unified data collection modes.

Unlike traditional distributed systems, agent systems are typically driven by large language models and include complex behavioral paths such as inference generation, tool invocation, context management, and multi-agent collaboration. The telemetry system MUST NOT only capture basic runtime information but also reflect key events, context changes and cross-component interaction relationships generated during agent task execution, thereby supporting runtime metrics (Chapter 6), behavior tracing (Chapter 7), and anomaly analysis and remediation (Chapter 8).

This document defines two standardized data collection modes: Network-side Collection and Agent-side Collection.

5.1. Network-side Collection

Network-side Collection observes the full lifecycle of agent requests within communication channels, covering key stages including request initiation, routing and forwarding, model invocation, tool invocation, response return, and security policy enforcement.

This mode is deployed at gateways or traffic ingress layers without requiring intrusion into or modification of the agent runtime environment, providing low-intrusion deployment, broad coverage, and strong centralized governance capabilities.

Network-side Collection MUST satisfy the following requirements:

Network-side Collection SHOULD provide the following capabilities:

Network-side Collection MAY include:

5.2. Agent-side Collection

Agent-side Collection is a mechanism within the agent runtime environment that structurally captures execution processes via embedded telemetry mechanisms (such as SDK instrumentation, runtime instrumentation, or proxy forwarding). Its scope includes inference behavior, tool invocation, memory access, and context evolution.This mode reconstructs the complete internal execution path of the agent and correlates it with network-side telemetry to enable end-to-end telemetry.

Agent-side Collection MUST satisfy the following requirements:

Agent-side Collection SHOULD provide the following capabilities:

Agent-side Collection MAY include:

6. Runtime Monitoring

The Runtime Monitoring focuses on telemetry of underlying Agent infrastructure and runtime workload. It serves as the fundamental data source for consumers to assess Agent availability and for operators to perform capacity planning and fault localization.

6.1. Token Usage Monitoring

Token Usage Monitoring is a core dimension for measuring Agent cost efficiency and resource consumption. It quantifies computational resources consumed by input and output text during large language model execution and provides the basis for billing, rate limiting, and performance optimization.

Token Usage Monitoring SHOULD record the following signals:

6.2. Model Monitoring

Model Monitoring is a dedicated telemetry dimension for large language model execution, used to describe how inference latency, throughput, hardware load, and version differences affect system behavior, and serves as one of the core telemetry capabilities distinguishing agents from traditional systems.

Model Monitoring SHOULD record the following signals:

6.3. Service Quality Monitoring

The Service Quality Monitoring capability domain focuses on telemetry of external service-level performance of agents and serves as the core basis for determining whether service requirements are met.

Service Quality Monitoring SHOULD record the following signals:

6.4. Multi-Agent Collaboration Monitoring

Multi-Agent Collaboration Monitoring characterizes communication behavior and collaboration efficiency among multiple agents during cooperative task execution. It serves as a key telemetry dimension for evaluating system-level autonomy and coordination capability. This dimension focuses on inter-agent message passing, task distribution, and collaborative execution outcomes.

Communication behaviors between agents within multi-agent systems SHOULD record the following signals:

7. Behavior Tracking

The Behavior Tracking focuses on the telemetry of agent decision-making processes, tool invocation behaviors, and interaction content. It provides structured reconstruction capabilities for understanding what an agent did, how it was done, and why it was performed during execution.

This dimension achieves explainability, post-hoc reproducibility, and compliance auditing of agent behavior through unified modeling of reasoning traces, tool invocation chains, session interactions, and end-to-end execution paths.

7.1. Decision Trace Tracking

Decision Trace Tracking records the complete reasoning path of an agent from task reception to final output generation. It serves as the core telemetry mechanism for understanding agent decision logic.

Each agent inference invocation SHOULD generate a standardized reasoning step record containing the following fields:

7.2. Tool Invocation Observation

The Tool Invocation Observation records the complete lifecycle of external tool, API, or service invocations performed by an agent. It is a critical capability for identifying functional failures and performance bottlenecks.

Tool invocation observations SHOULD record the following fields:

7.3. Session Interaction Observability

Session Interaction Observability targets the telemetry of dialogue behavior between agents and callers, covering the full interaction lifecycle from session establishment to termination.

The Session Interaction Observability capability domain SHOULD cover the following telemetry fields:

8. Anomaly Analysis & Remediation

The Anomaly Analysis & Remediation builds a closed-loop telemetry capability for agent runtime systems based on telemetry data collected from Agent Data Collection (Chapter 5), Runtime Monitoring (Chapter 6), and Behavior Tracking (Chapter 7).

It uses trace_id, span_id, and session_id as unified correlation keys to jointly analyze runtime metrics and behavior traces within consistent time windows, enabling anomaly detection, localization, attribution, and remediation.

8.1. Baseline Deviation Analysis

Baseline Deviation Analysis is used to identify deviations of agent runtime behavior relative to historical normal behavioral patterns and serves as the fundamental mechanism for anomaly detection.

This mechanism is built upon the complete runtime metric system defined in Chapter 6. The system performs joint observation of the above metrics within a time window and compares them against historical baseline distributions to determine whether the current runtime state exhibits significant deviation. A unified deviation score is generated to represent the overall anomaly severity. This score does not depend on any single metric but reflects the aggregated variation trends of multi-dimensional metrics within the same observation window.

In implementation, baseline models MUST support hierarchical modeling across tenant, agent, and model dimensions to accommodate differences in workload patterns and model behaviors. Meanwhile, baseline models MUST support versioned management to ensure consistency and traceability during model iteration and system evolution.Baseline models MUST also support versioned management to ensure consistency and traceability across model iterations and system evolution.When performing deviation detection, the system MUST perform joint verification using both runtime metrics and behavior traces to reduce false positives caused by numerical fluctuations alone, and SHOULD support continuous baseline updates over time to adapt to long-term environmental drift.

8.2. Root Cause Analysis

Root Cause Analysis is used to systematically attribute anomalies to their origin points, propagation paths, and impact scopes after anomaly occurrence, thereby establishing causal relationships between metric anomalies and behavior traces.

This capability relies on the trace/span execution tree structure defined in Chapter 7, combined with time-aligned changes in runtime metrics defined in Chapter 6, to perform cross-layer correlation analysis of anomalies.

During analysis, the system first identifies key metrics that exhibit significant deviations within the anomaly time window, such as increased latency, abnormal Token consumption, elevated error rates, or expanded multi-agent communication delays. These abnormal signals are then correlated with corresponding trace_id values to locate the affected execution traces.

The system then performs stepwise backtracking along the execution structure constructed via span_id to identify the earliest node where abnormal signals appear. It further analyzes whether the node belongs to the model inference stage, tool invocation stage, or multi-agent collaboration stage, thereby classifying the anomaly source as one of the following types: model anomaly, tool anomaly, system anomaly, or collaboration anomaly.

The system SHOULD also analyze the propagation path of anomalies across the execution trace to determine the scope and diffusion pattern of the impact.

The final output of Root Cause Analysis SHOULD provide a complete description of the anomaly origin node, propagation path, and affected scope, and SHOULD support cross-session and cross-agent correlation to enable interpretable reconstruction of complex agent system behaviors.

8.3. Anomaly Alerting

Anomaly Alerting converts anomalies detected through runtime metrics and baseline deviation analysis into real-time actionable alert events, enabling rapid response and risk control during system operation.

Alert triggers originate from three categories of signals: baseline deviation detection results, critical runtime metric anomalies, and behavior trace anomalies. When the system determines that anomaly severity exceeds predefined thresholds, a standardized alert event is generated.

Each alert event MUST maintain strong correlation with trace_id, session_id, and agent_id, and MUST explicitly include the key metric source that triggered the alert to ensure full contextual interpretability.

The system SHOULD support multi-level alerting mechanisms, including informational, warning, and critical levels, with dynamic severity classification based on anomaly intensity.

In implementation, the system MUST support alert deduplication and aggregation to prevent alert storms under high-frequency fluctuation scenarios. It SHALL also support alert lifecycle management, including state transitions such as triggered, acknowledged, in-progress, and resolved/closed.

8.4. Anomaly-Based Threat Library

The Anomaly-Based Threat Library is used to structurally store historical anomaly events, enabling reusable anomaly pattern recognition and continuous improvement of system detection and analysis capabilities.

This capability is built by integrating historical anomaly events and root cause analysis results, and jointly abstracting them with runtime metric patterns (Chapter 6) and behavior trace patterns (Chapter 7), forming a semantically consistent set of anomaly patterns.

These anomaly patterns include both metric-side features and behavior-side features, and establish associations with corresponding root cause types and remediation strategies, forming complete knowledge units.

During system evolution, newly observed anomaly events MAY be continuously fed back into the threat library to extend or refine existing anomaly patterns.The threat library MUST support versioned management to ensure compatibility and stability of detection rules during iterative evolution.Additionally, the system MAY support similarity-based anomaly retrieval over historical patterns to facilitate rapid identification of known issues, and MAY enable cross-system sharing and reuse of anomaly patterns.

9. Security Considerations

This document defines an agent runtime telemetry framework covering data collection, metrics monitoring, behavior tracing, and anomaly analysis. Security properties depend on the deployment environment and the trust assumptions of the telemetry pipeline.

The system introduces several security considerations, including but not limited to agent identity spoofing, telemetry data manipulation, trace and span falsification, cross-agent message forgery, and potential information leakage through telemetry data exposure.

Telemetry data SHOULD be treated as attested runtime telemetry rather than ground truth about system behavior or external execution reality.In deployments using a trusted gateway, enforcement and validation of telemetry data MAY be performed at the gateway boundary. In SDK-based deployments without external enforcement, the correctness of telemetry data is conditional on agent and runtime integrity.

This document does not address runtime compromise scenarios or provide guarantees under fully compromised execution environments.

10. Privacy Considerations

This framework may collect sensitive runtime information, including user inputs, model outputs, tool invocation parameters, and inter-agent communication data. Implementations SHOULD apply data minimization and access control to prevent unauthorized exposure.Sensitive data SHOULD be masked or anonymized when stored, transmitted, or used for analysis. Multi-tenant systems MUST enforce strict isolation of telemetry data across tenants.

11. IANA Considerations

This document has no IANA actions.

12. Normative References

[RFC2119]
Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, , <https://www.rfc-editor.org/info/rfc2119>.
[RFC8174]
Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, , <https://www.rfc-editor.org/info/rfc8174>.

Authors' Addresses

Xianghua Wang
CAICT
Huayuan North Road, HaiDian District
Beijing
Beijing, 100191
China
Xuesen Liu
CAICT
Huayuan North Road, HaiDian District
Beijing
Beijing, 100191
China
Zhe Li
CAICT
Huayuan North Road, HaiDian District
Beijing
Beijing, 100191
China
Mengjia Zhao
CAICT
Huayuan North Road, HaiDian District
Beijing
Beijing, 100191
China