| Internet-Draft | LLM Incident Metrics Terminology | August 2026 |
| Gaikwad | Expires 12 February 2027 | [Page] |
This document defines terminology for reporting incidents in Large Language Model (LLM) services in which output behavior departs from intended behavior without producing an error response. It defines the lifecycle events of such an incident, the intervals between those events, and the qualifiers a reported interval carries.¶
The intervals defined here are anchored at the point information about an occurrence reaches a responsible party. The interval preceding the first monitoring signal is named and is not defined as measurable.¶
This document defines terminology only. It specifies no data model, no reporting obligations, and no target values.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 12 February 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
An LLM service can degrade without failing. A response that is ungrounded, unnecessarily refused, or inconsistent with an identical earlier response is returned with a success status. The service remains available by every conventional measure while the output it produces departs from what the operator intends.¶
Operators respond to such degradations, and increasingly report on that response. The vocabulary available for the reporting comes from reliability engineering and from security operations, where the same term is used for intervals with different starting events. A reported interval from one operator therefore does not tell a reader what was measured.¶
This document defines a vocabulary for the events and intervals of an output behavior incident. It defines intervals by naming their endpoints, so that a reported value states what it measured.¶
This document defines terminology only. It does not define a data model, a reporting format, an obligation to report, or a target value for any interval.¶
This document does not define the behavioral indicators whose departure constitutes an incident. Indicator definitions and the fault classes constructed from them are given in [FAULT-TERMS].¶
Interval values obtained under this terminology are properties of a service and of the operator responding to it. They are not comparable between operators. The purpose of the terminology is to make a reported value intelligible to a reader.¶
[RFC9940] defines terms for network fault and problem management, including Incident, Event, Fault, and Symptom. This document uses those terms as defined there and adds the intervals between incident lifecycle events.¶
[I-D.ietf-nmop-network-incident-yang] defines a data model for network incident lifecycle management, carrying timestamps for occurrence, raising, acknowledgement, and clearing. The intervals defined here are expressible over those timestamps. Section 9 gives the correspondence.¶
[FAULT-TERMS] and [FAULT-METHOD] define terminology and procedures for characterizing a detector under controlled fault injection. Detection Coverage, defined in [FAULT-TERMS] and measured under [FAULT-METHOD], is the qualifier described in Section 7.3 of this document.¶
[LLM-TERMS] defines System Under Test boundaries for LLM serving. Those boundaries are used here to state where an incident was observed.¶
[RFC6414] defines a lifecycle of failure, detection, failover, and restoration for sub-IP protection mechanisms, together with the intervals between those events. This document takes the same shape of named events with named intervals between them. The occurrences it addresses differ in four ways, and each difference produces a term with no counterpart in that work.¶
The failure is not binary. A sub-IP path is either carrying traffic or it is not. Output behavior departs by degree, so an occurrence has a magnitude and a report states one.¶
The system emits no failure signal. A protection mechanism observes its own failure and acts on it. A service producing degraded output returns a success response, so detection is performed by an observer outside the failing component and the completeness of that observation is a property of the observer. Section 7.3 exists for this reason.¶
Detection requires accumulated evidence. A path failure is observable in a single packet. A shift in a rate-valued indicator is observable only across many responses, so the interval before detection depends on request volume and not on elapsed time alone.¶
Whether an occurrence is a fault requires judgement. A path is not down by opinion. An increase in refusals may be a departure from intended behavior or the intended effect of a policy change, and determining which is a distinct activity performed by a person. Section 5.5 and Section 7.4 exist for this reason.¶
The abbreviations MTBF and MTTR appear in IETF documents, for example in the terminology list of [I-D.ietf-teas-ietf-network-slice-definition], without a definition of the intervals they name. This document defines its intervals by naming their endpoints and does not use those abbreviations.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
Normative language in this document constrains how a value is reported. It places no requirement on how a service behaves or on how an operator responds.¶
The definition excludes error responses because those are covered by existing availability and error-rate reporting. It includes degradations that return success. The operator determines what behavior is intended, and the indicators by which departure is expressed.¶
An occurrence qualifies whether or not it produces external harm. A report may additionally classify an incident by its consequence, and such classification is out of scope here.¶
The available boundaries are the Model Engine, the Application Gateway, and the Compound System, as defined in [LLM-TERMS]. A degradation present at the Model Engine may be suppressed by output filtering before the Application Gateway. An incident observed at one boundary is not the same occurrence as an incident observed at another.¶
The intervals of Section 6 are defined over the following events. Each definition states what makes the event attestable. Events are listed in the order they commonly occur. Diagnosis is an exception and is discussed in Section 5.11.¶
Behavior Departure is defined so that it can be named in a report. It is not defined as a measurable event. Establishing it requires evidence from before the operator was aware that evidence would be needed, and the telemetry carrying that evidence commonly has a retention period shorter than the interval in question.¶
Where a service is instrumented to retain sufficient evidence, Behavior Departure MAY be established retrospectively. A report doing so MUST state the evidence and the method used.¶
Detection is a property of the monitoring mechanism and not of any person. A signal exists at Detection. Nothing has yet been concluded about whether the occurrence is genuine, what class it belongs to, or whether anyone will act on it.¶
Detection is established retrospectively in most cases, by locating the earliest signal associated with the incident once the incident is known. A report MUST state whether Detection was recorded at the time or established retrospectively.¶
An incident first learned of through a user report or an external notification has no Detection. Such an incident MUST record Detection as absent, which is distinct from recording it as equal to Notification.¶
Notification is the point at which a person can act. It is distinct from Detection, which requires no person. A signal that fires into an unattended channel produces Detection and no Notification.¶
Notification is the anchor for the intervals of Section 6, because it is the earliest event that is attestable in every incident, including those learned of through a user report or an external party.¶
Notification places information in front of a responsible party. That party is frequently not the one able to act. Routing is the point at which the occurrence reaches the team, individual, or mechanism that responds to it.¶
Responsibility may be assigned more than once before the responding party is reached. Routing is the time of the assignment to the party that responded, and not the time of the first assignment. An occurrence assigned once has a Routing time equal to that assignment.¶
Routing may precede or follow Confirmation. A responder may be identified from the affected component before anyone determines that the occurrence is genuine, and an occurrence may be confirmed by the party first notified and then handed to another party to respond.¶
A report SHOULD carry a Hop Count, being the number of parties responsibility passed through before reaching the one that responded. An occurrence reaching its responder on the first assignment has a Hop Count of one.¶
Each hop adds delay before the party able to act is engaged, and no mitigation applied by an operator can begin earlier than the final hop. Hop Count therefore bounds the intervals that follow Routing, and two operators reporting the same Notification-to-Mitigation Interval at different Hop Counts describe different response paths.¶
Detection produces a signal. Confirmation produces a conclusion. The two are separated because the work between them is substantial for output behavior incidents. Determining whether an increase in refusals is a departure from intended behavior or the intended effect of a policy change requires evidence beyond the indicator. Determining whether a groundedness decline originates in the model or in a retrieval component requires evidence from both.¶
An occurrence determined not to be an incident has a Confirmation time and no subsequent events. Such occurrences are reported under Section 7.4.¶
Confirmation establishes that an incident occurred and what class it belongs to. It does not establish why. Section 5.11 covers that.¶
Mitigation may precede Confirmation and may precede Diagnosis. Routing traffic to a previous model version, tightening a filter, and reducing sampling temperature are mitigations. A service under mitigation operates in a configuration its operator did not intend for normal service.¶
A mitigation applied by a mechanism without operator involvement may precede Notification. A mitigation applied by an operator cannot. The two carry different intervals, and an interval measured over a set containing both describes neither.¶
A reported Mitigation MUST declare its Mitigation Actor, taking one of the values automated, operator-initiated, or operator-approved automated. The third value denotes a mechanism that prepares a mitigation and applies it once an operator authorises it, and carries the authorisation delay that the first does not.¶
Where Mitigation precedes Notification, the Notification-to-Mitigation Interval MUST NOT be reported. The Detection-to-Mitigation Interval is reported instead.¶
Intervals ending at Mitigation MUST NOT be aggregated across Mitigation Actors.¶
Recovery describes the behavior a user experiences. It does not describe the configuration producing that behavior. A service operating correctly because a mitigation is holding it correct has recovered and has not reverted.¶
The term repair is not used in this document. No component is replaced in the incidents addressed here, and the interval a repair would name has no distinct endpoints.¶
Reversion is optional in the sense that an incident may be closed with a mitigation retained as the permanent configuration. Where that occurs, Reversion MUST be recorded as not applicable, with a statement that the mitigation was retained.¶
The interval a service spends recovered but not reverted is frequently the longest segment of an incident. It is reported separately under Section 6.¶
Closure ends the period during which responders are assigned to the incident. It commonly coincides with Recovery, since an operator ends the response once user-visible behavior is correct. It may follow Reversion where the operator holds the response open until the mitigation is removed.¶
Closure does not require Diagnosis. An operator may close an incident with the condition that produced it unestablished, and establish it later. Section 5.10 covers the analysis that follows Closure.¶
Analysis performed after Closure operates on a schedule unrelated to the response. It is commonly governed by a separate commitment, expressed as a period following Closure, and it is commonly the artifact carrying a Diagnosis that the response did not reach.¶
The analysis also carries the observations from which subsequent detection and mitigation are changed. A change in Mitigation Actor across incidents of a class, from operator-initiated to automated, is the observable consequence of such an analysis, and is reported through Section 5.6 rather than here.¶
Diagnosis does not occupy a fixed position in the sequence. It may precede Mitigation, follow Recovery, follow Reversion, or never occur. An operator may mitigate an incident without knowing its cause and may close it without establishing one.¶
Diagnosis is therefore reported as an event and not as a stage. Intervals ending at Diagnosis are meaningful. Intervals beginning at Diagnosis are not defined by this document, because the events that would follow it may already have occurred.¶
A Diagnosis reached before Closure was reached during the response. A Diagnosis reached after Closure was reached by retrospective analysis, and is commonly recorded in the artifact defined in Section 5.10. The two are reported separately, since a set combining them describes neither the response nor the analysis.¶
An incident closed without Diagnosis MUST record it as not established. The cause is then reported under Section 7.5 with a status of investigating.¶
Diagnosis may be performed by an operator, by an automated mechanism, or by an operator working from output produced by such a mechanism. An automated mechanism reaches an attribution on a timescale unrelated to that of an operator, and the confidence attaching to its output is a property of the mechanism. A time to Diagnosis therefore describes the actor as much as the incident.¶
A reported Diagnosis MUST declare its Diagnosis Actor, taking one of the values operator, automated, or operator-reviewed automated.¶
Intervals ending at Diagnosis MUST NOT be aggregated across Diagnosis Actors. An attribution produced by an automated mechanism and not reviewed by an operator SHOULD carry a Cause Status of preliminary.¶
Intervals are named by their endpoints. A reported interval MUST identify the events it spans and MUST carry the qualifiers of Section 7.¶
An interval MUST NOT be reported where either of its endpoints is absent or undetermined. An interval spanning an event recorded as not applicable MUST NOT be reported as zero.¶
The events other than Diagnosis and Postmortem fall on a single time axis in the order shown below. Diagnosis is omitted from the figure because it has no fixed position. Postmortem is omitted because it commonly follows closure of the incident record.¶
BD DT NT RT CF MT RC RV
| | | | | | | |
--+-------+-------+-------+-------+-------+-------+-------+--->
|<-(a)->|
|<-(b)->|
|<-(c)->|
|<-(d)->|
|<-(e)->|
|<-(f)->|
|<-(g)->|
|<-------------(h)------------->|
BD Behavior Departure (a) Departure-to-Detection
DT Detection (b) Detection-to-Notification
NT Notification (c) Notification-to-Routing
RT Routing (d) Routing-to-Confirmation
CF Confirmation (e) Confirmation-to-Mitigation
MT Mitigation (f) Mitigation-to-Recovery
RC Recovery (g) Recovery-to-Reversion
RV Reversion (h) Notification-to-Recovery
The figure shows one ordering. Routing may follow Confirmation, and Mitigation may precede Confirmation or Notification. The intervals of Section 6 are defined by their endpoints and not by the positions shown here.¶
The events of Section 5 other than Diagnosis occur in order, so each bounds those after it. An occurrence cannot be confirmed before a person is notified of it, and cannot be mitigated before it is confirmed or, where mitigation precedes confirmation, before it is notified. Detection therefore sets a lower bound on every event that follows it.¶
An improvement in detection moves the subsequent events earlier without changing the intervals between them. An operator that halves its Departure-to-Detection Interval recovers earlier by that amount and reports the same Notification-to-Recovery Interval as before. The two quantities describe different capabilities and one does not substitute for the other.¶
The Notification-to-Mitigation Interval commonly contains a component fixed by operator procedure. Approval, change control, deployment, and verification consume time that does not vary with how the occurrence was detected, how large it was, or how quickly it was confirmed. That component sets a floor the interval does not fall below, however good the detection preceding it.¶
A mitigation applied by a mechanism without operator involvement does not carry that component, and does not carry Notification as a predecessor. Such a mitigation may occur at Detection. The ordering above therefore holds for operator-initiated mitigation and does not hold for automated mitigation, which is why Section 5.6 requires the Mitigation Actor to be declared and separates the two intervals.¶
Segment intervals MUST be reported separately and MUST NOT be summed into a single reported value. A single value spanning detection and mitigation attributes procedure-bound time to detection performance, and attributes detection delay to procedure. An operator MAY additionally report a span covering several segments, and MUST report the segments alongside it.¶
Recovery describes the behavior a user experiences. Reversion describes the configuration producing it. The interval between them is the time the service spends operating correctly in a configuration its operator did not intend for normal service.¶
The two coincide where no mitigation was applied. A change that restores intended behavior directly leaves nothing to remove, so Recovery is that change and Reversion MUST be recorded as not applicable. The Recovery-to-Reversion Interval is then not reported.¶
The two are far apart where a mitigation holds behavior correct while the condition producing the incident is addressed on a longer schedule. A service running under a retained mitigation has recovered and has not reverted, and reporting only Recovery conceals that it is running in an unintended configuration.¶
Reversion does not precede Recovery. A mitigation removed while the condition persists returns the service to degraded behavior, which is a continuation of the incident and not a Reversion. Where this occurs, Recovery and Reversion are recorded at the times finally reached.¶
The Departure-to-Detection Interval spans an event that Section 5.1 does not define as measurable. Where it has not been established, it MUST be reported as undetermined. It MUST NOT be omitted, and it MUST NOT be reported as zero.¶
A reader cannot distinguish an omitted Departure-to-Detection Interval from one of zero duration. The two carry opposite information about the service and about the reporting operator.¶
Reports outside this document commonly use abbreviations of the form mean time to X. Those abbreviations are used for several of the intervals above, and the same abbreviation is used for different intervals in different reports.¶
| Abbreviation | Reported as | Intervals it has been used for |
|---|---|---|
| MTTD | Detect | Departure-to-Detection; Detection-to-Notification |
| MTTN | Notify | Detection-to-Notification |
| MTTA | Acknowledge | Detection-to-Notification; Notification-to-Confirmation |
| MTTI | Identify | Notification-to-Confirmation; Notification-to-Diagnosis |
| MTTR | Repair, recover, restore, resolve, respond | Notification-to-Mitigation; Notification-to-Recovery; Recovery-to-Reversion |
An abbreviation does not carry its endpoints, and a reader cannot recover them from the abbreviation alone. Each of these abbreviations additionally asserts that the value is a mean, and the value reported under them is frequently a percentile. Section 7.1 addresses this.¶
These abbreviations SHOULD NOT be used in a report prepared under this document. Where one is used, the interval it names and the Summary Statistic it carries MUST be stated alongside it.¶
An interval reported as an aggregate over a set of incidents carries information only alongside the following qualifiers.¶
Distributions of incident intervals are commonly heavy-tailed. A small number of long incidents moves the mean well above the typical value, so a reported mean describes neither a typical incident nor the worst one. Reports in practice therefore carry percentiles, most often a median with one or more upper percentiles.¶
The abbreviations of Section 6.4 assert a mean. The value reported under them is frequently a percentile. A report carrying a percentile under a label asserting a mean states something other than what it measured.¶
Reporting a median together with an upper percentile conveys the typical value and the tail separately. Reporting the mean alongside the median conveys how far the tail carries the distribution, since a mean substantially above the median indicates the aggregate is set by a few long incidents.¶
An aggregate interval MUST be reported with its Summary Statistic. An aggregate reported without one cannot be compared against any other aggregate.¶
An upper percentile is determined by the largest observations in the set. Where the Incident Count multiplied by the complement of the percentile is less than one, the percentile equals the largest observation in the set. Under that condition the value MUST be reported as the maximum and MUST NOT be reported as a percentile. A ninetieth percentile therefore requires an Incident Count of at least ten, and a ninety-fifth percentile requires at least twenty.¶
A median SHOULD be reported for every aggregate interval. At least one upper percentile SHOULD be reported where the Incident Count permits it.¶
An aggregate over a small number of incidents is dominated by variance. A reader cannot judge this without the count.¶
The count also determines which summary statistics the set can carry. Section 7.1 states the condition under which an upper percentile is the largest observation in the set rather than a percentile.¶
Every interval in Section 6 is computed over incidents that reached Notification. Incidents that did not reach Notification contribute to neither the aggregate nor the Incident Count. Reported intervals therefore shorten as the fraction reaching Notification declines, with no change in service behavior and no change in operator response.¶
An estimate MAY be obtained by characterizing the operator's detection apparatus under [FAULT-METHOD], by sampled retrospective review, or by comparison against incidents reported by external parties. The method MUST be stated.¶
The events and intervals of this document describe response. An operator also acts to stop incidents of a class from occurring. That activity has no event on the axis of Figure 1, because its success consists of occurrences that do not happen.¶
Recurrence is a property of a class and of an interval of time. A report states both. Two incidents are of the same class where the declared classification assigns them the same value, and classification granularity therefore determines what counts as a recurrence.¶
Recurrence is observable. It is the closest observable counterpart to the activity described in Section 8.3.¶
Prevention is defined so that it can be named in a report. It is not defined as a measurable quantity. Its effect is the absence of occurrences, and an absence cannot be attributed to the action that produced it without a counterfactual the operator does not have.¶
A decline in incidents of a class following preventive action is consistent with the action having worked. It is also consistent with a change in traffic, a change in the model, a change in detection coverage, and with the class having been rare throughout. No interval in Section 6 distinguishes these.¶
Section 8.2 is the quantity reported in its place. A report MUST NOT state a measured effect of prevention. It MAY state preventive action taken and the Recurrence Rate observed since.¶
This asymmetry has a consequence for reporting. Every event in Section 5 is attestable and prevention is not, so a report describing only what is measurable describes response and omits prevention entirely. An operator whose preventive work is effective reports fewer incidents and no evidence of the work.¶
The events of Section 5 correspond to timestamps defined in [I-D.ietf-nmop-network-incident-yang] as follows.¶
| Event | Timestamp | Note |
|---|---|---|
| Behavior Departure | occur-time | Commonly undetermined |
| Detection | (none) | No corresponding timestamp |
| Notification | raise-time | Interval anchor |
| Confirmation | ack-time | Approximate; the model's acknowledge operation is broader |
| Mitigation | (none) | No corresponding timestamp |
| Recovery | clear-time | |
| Reversion | (none) | No corresponding timestamp |
| Diagnosis | (none) | Operator state 'diagnosed' exists, without a time |
Three of the eight events map to an existing timestamp. Confirmation and Diagnosis correspond to operator states in the model's lifecycle, reached through the acknowledge and diagnose operations, and the model records the state without recording the time the state was entered. Detection, Mitigation, and Reversion have no counterpart.¶
An operator computing the intervals of Section 6 from this model records the missing times by other means. Whether an extension carrying them is warranted is left to the working group.¶
This document defines terminology and introduces no protocol mechanisms.¶
Reported values may be sensitive. A Detection Coverage Qualifier states the fraction of degradations an operator does not observe. A Confirmation Outcome Rate indicates how much noise an adversary would need to generate to occupy operator attention. Operators SHOULD consider aggregation granularity and disclosure audience when publishing these values.¶
An operator publishing intervals without a Detection Coverage Qualifier may present a shorter response profile than the evidence supports. The requirement in Section 7.3 exists in part to limit this.¶
This document has no IANA actions.¶
This document builds on the terminology separation established by the Network Management Operations Working Group in [RFC9940] and on the lifecycle structure of [I-D.ietf-nmop-network-incident-yang].¶