]> Independent

gaikwad.madhav@gmail.com

Operations and Management Benchmarking Methodology benchmarking LLM fault injection fault detection gray failure This document defines test procedures for characterizing the fault detection capability of observability systems that monitor Large Language Model (LLM) serving deployments. Procedures are given for Detection Latency, Detection Coverage, Detection Threshold Magnitude, False Detection Rate, and Boundary Masking. The Detector Under Test is the observability system. Output Behavior Faults are injected at a known time under controlled conditions, which makes detection latency directly measurable. This document is a companion to "Benchmarking Terminology for Output Behavior Fault Detection in Large Language Model Serving Systems" and is to be read alongside it. This document specifies no acceptance thresholds.

Introduction defines terminology for benchmarking the fault detection capability of observability systems monitoring LLM serving deployments. This document defines the procedures that produce those measurements. The procedures follow the pattern of and . A parameter is varied across trials, the Detector Under Test is observed, and a characteristic of the detector is reported with the conditions under which it was obtained.

Scope This document specifies test procedures, required test conditions, and a reporting format. It does not specify acceptance thresholds, target values, or service-level objectives. A result obtained by these procedures characterizes a detector. It does not establish whether that detector is adequate for any purpose. The mechanism by which a fault is produced is out of scope. Any mechanism that shifts the declared indicator by the declared magnitude at the declared time is acceptable. states the properties an injection apparatus must have. Benchmarking described by this document is confined to a test environment. states this requirement.

Applicability A result obtained by these procedures characterizes one Detector Under Test in one declared environment. The serving deployment includes a language model whose output is stochastic, so the baseline against which fault magnitude is expressed is a property of that environment. requires the baseline to be established for each environment before any fault trial. Two results are comparable when the Observation Point, the DUT configuration, the serving deployment configuration, the request population, and the offered load are the same. Results MUST NOT be compared when any of these differ. requires each to be reported so that a reader can determine whether a comparison is valid. The intended uses follow from this. A detector configuration can be compared against another configuration on the same environment. A detector can be tracked across changes to itself or to the deployment it observes. A detector can be characterized before it is relied on. These procedures are not intended to produce values that rank detectors across organizations, and a value obtained here carries no meaning outside the environment in which it was obtained. A full series under these procedures is long. Twenty trials at each combination of Fault Class, magnitude, profile, and load, each followed by a quiet period, together with the search of , occupies substantial test time. A reduced series MAY be run for screening, using fewer trials, fewer load points, or a coarser search resolution. A reduced series MUST be reported as such, with the reductions stated, and MUST NOT be reported as a result under this methodology.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. Terms defined in are used here with their definitions from that document. These include Detector Under Test, Observation Point, Fault Injection Point, Output Behavior Fault, Fault Class, Fault Profile, Fault Onset, Detection Event, False Detection, Baseline Interval, Detection Latency, Detection Coverage, Detection Threshold Magnitude, and Boundary Masking.

Test Setup

Topology The test environment comprises four components.

Traffic generator:

Offers requests to the serving deployment at a controlled rate from a declared request population.

Serving deployment:

The LLM serving stack under observation, configured to one of the System Under Test boundaries of .

Fault injection apparatus:

Applies Output Behavior Faults at the Fault Injection Point and records Fault Onset.

Detector Under Test:

The observability system, sampling at the Observation Point and emitting the declared detection signal.

The relative position of the Fault Injection Point and the Observation Point MUST be recorded and reported. A fault injected downstream of the Observation Point is not observable by the detector, and trials in that configuration measure only Boundary Masking.

Detector Under Test Configuration The DUT configuration MUST be frozen before the first trial and MUST NOT be changed during a test series. Changing thresholds, sampling rates, judge models, or alerting rules begins a new test series. The following MUST be recorded: the indicators the DUT computes, the method used to compute each, the sampling rate for any sampled indicator, the aggregation window, the alerting thresholds, and the version of any model used as a judge.

Traffic Generation Offered load MUST be constant within a trial. The request population MUST be constant across all trials in a series. Detection Latency for a rate-valued indicator depends on how quickly evidence accumulates, and evidence accumulates with requests rather than with time. Results are therefore reported against offered load. A test series SHOULD cover at least four load points spanning at least two orders of magnitude, and SHOULD include a load point representative of the intended deployment. Response caching in the serving deployment suppresses the indicator shift a fault is intended to produce. Caching MUST be disabled, or the request population MUST be constructed so that cache hits do not occur. Which approach was used MUST be reported.

Fault Injection Apparatus The apparatus MUST record Fault Onset with a resolution of one millisecond or finer, and MUST record it independently of any DUT output. The apparatus MUST be able to remove an applied fault and return the deployment to its baseline configuration. The apparatus MUST NOT emit any signal to the DUT other than through the behavior of the serving deployment. An apparatus that notifies the DUT of injection invalidates the measurement. The magnitude actually achieved MUST be verified against the magnitude intended, by measuring the affected indicator at the Fault Injection Point during application. A trial in which the achieved magnitude differs from the intended magnitude by more than ten percent MUST be discarded and repeated.

Clock Synchronization Fault Onset and Detection Event may be recorded by different hosts. Detection Latency is the difference between them, and clock offset between those hosts enters the result directly as error. The maximum clock offset between the injection apparatus and the DUT MUST be measured and reported. The offset MUST be smaller than one percent of the smallest Detection Latency reported in the series. Where this cannot be achieved, the offset MUST be reported as an error bound on every latency value in the series. Where the injection apparatus and the DUT record timestamps on the same host, this requirement does not apply and the arrangement MUST be reported.

Prerequisites

Detection Signal Declaration The signal that constitutes a Detection Event MUST be declared before the first trial. The declaration MUST state the signal type, the component that emits it, and the criterion by which a signal is attributed to an applied fault. The declaration MUST NOT be changed after any trial in the series has been run. Selecting the detection signal after observing results invalidates the series.

Baseline Characterization Before the first fault trial, a Baseline Interval MUST be run at each load point to be used in the series. The Baseline Interval MUST be at least ten times the DUT aggregation window, and MUST be at least three hundred seconds. For each indicator to be faulted, the mean and standard deviation over the Baseline Interval MUST be recorded. Fault magnitude is reported as an absolute change from this mean. A fault magnitude smaller than twice the baseline standard deviation is not distinguishable from baseline variation. Trials at such magnitudes MAY be run, and results from them MUST be reported with the baseline standard deviation alongside.

Inter-Trial Quiet Period Observability systems commonly implement alert deduplication, flap suppression, and state that persists after a condition clears. A trial begun while the DUT retains state from a previous trial does not measure detection from a resting state. Between trials, the fault MUST be removed and a quiet period MUST be observed. The quiet period MUST be long enough for the DUT to return every affected indicator to within one standard deviation of its baseline mean, and to clear any active alert state. The quiet period used MUST be reported.

Single Fault Application One Output Behavior Fault MUST be applied at a time, unless the declared attribution criterion of is able to attribute a signal to a specific fault in the presence of others. Where multiple faults are applied, the attribution criterion MUST be reported.

Test Procedures

Detection Latency Objective. Determine the interval from Fault Onset to the Detection Event, for a stated Fault Class, magnitude, Fault Profile, and offered load. Procedure.

1. Offer traffic at the selected load. Wait for the quiet period of .
2. Apply the fault. Record Fault Onset.
3. Observe the DUT until a Detection Event attributable to the fault occurs, or until the trial timeout expires.
4. Record the Detection Event time, or record the trial as undetected.
5. Remove the fault. Observe the quiet period.
6. Repeat from step 2 until the required number of trials is complete.

Trial count. At least twenty trials MUST be run at each combination of Fault Class, magnitude, profile, and load. Trial timeout. The timeout MUST be declared before the series and MUST be at least twenty times the DUT aggregation window. A trial reaching the timeout is recorded as undetected and contributes to Detection Coverage. Reported value. The median and the ninety-fifth percentile of Detection Latency across detected trials MUST be reported, together with the number of trials and the number undetected. The mean MAY be reported. Values from undetected trials MUST NOT be imputed.

Detection Coverage Objective. Determine the fraction of applied faults for which a Detection Event occurred, for a stated Fault Class, magnitude, profile, and load. Procedure. Detection Coverage is computed from the same trials as . No separate trials are required. Reported value. The number of trials in which a Detection Event occurred, divided by the number of trials run. The numerator and denominator MUST both be reported. A coverage value MUST NOT be reported without the fault magnitude at which it was obtained.

Detection Threshold Magnitude Objective. Determine the smallest fault magnitude at which the DUT achieves a stated Detection Coverage, for a given Fault Class, profile, and load. Procedure. The procedure is a binary search on magnitude, following the pattern of the throughput search in .

1. Select a target coverage level. Declare it before the search begins.
2. Establish an upper bound magnitude at which coverage meets or exceeds the target, and a lower bound at which it does not. Where the lower bound falls below twice the baseline standard deviation, record that the search floor was reached and stop.
3. Run the trials of at the midpoint magnitude.
4. If coverage at the midpoint meets or exceeds the target, the midpoint becomes the new upper bound. Otherwise it becomes the new lower bound.
5. Repeat from step 3 until the interval between bounds is smaller than the declared resolution.

Reported value. The upper bound at termination, the target coverage level, the search resolution, and the baseline standard deviation of the affected indicator. Where the search floor was reached, the result MUST be reported as below the floor and MUST NOT be reported as a magnitude.

False Detection Rate Objective. Determine the rate at which the DUT emits the declared detection signal when no fault is applied. Procedure.

1. Offer traffic at the selected load with no fault applied.
2. Observe the DUT for the declared observation period. Record every emission of the declared detection signal.
3. Repeat at each load point in the series.

Observation period. The period MUST be at least the total time spent under fault application across the whole series at that load point. A False Detection Rate measured over a shorter period than the latency trials it accompanies understates the rate. Reported value. The count of false detections and the observation period. A Detection Latency or Detection Threshold Magnitude result MUST NOT be published without an accompanying False Detection Rate obtained under the same DUT configuration and load.

Boundary Masking Objective. Determine the reduction in observed fault magnitude between the Fault Injection Point and the Observation Point. Procedure.

1. Apply a fault of known magnitude at the Fault Injection Point.
2. Measure the affected indicator at the Fault Injection Point and at the Observation Point over the same interval, using the same method at both points.
3. Remove the fault. Observe the quiet period. Repeat for the required number of trials.

Trial count. At least twenty trials MUST be run. Reported value. One minus the ratio of the magnitude observed at the Observation Point to the magnitude observed at the Fault Injection Point, reported as a median across trials. The measurement method used at both points MUST be reported. Where the two points are the same, Boundary Masking is zero by construction and MUST be reported as not applicable.

Reporting Format A result reported under this methodology MUST include the following fields. Required reporting fields

Field	Content
Observation Point	Model Engine, Application Gateway, or Compound System
Fault Injection Point	Location, and position relative to the Observation Point
DUT configuration	Indicators, methods, sampling rates, aggregation window, thresholds, judge model version
Fault Class	As defined in the terminology document
Fault magnitude	Absolute change from baseline mean, in indicator units
Fault Profile	Step, Ramp, or Intermittent, with parameters
Baseline statistics	Mean and standard deviation of the affected indicator
Offered load	Requests per second, per load point
Caching	Disabled, or request population constructed to avoid hits
Trial count	Trials run, and trials undetected
Trial timeout	Seconds
Quiet period	Seconds
Clock offset	Maximum measured offset, or same-host declaration
Detection Latency	Median and 95th percentile, seconds
Detection Coverage	Numerator and denominator
False Detection Rate	Count and observation period
Boundary Masking	Median fraction, or not applicable

Results obtained at different Observation Points, under different DUT configurations, or at different offered loads MUST NOT be combined into a single reported value.

Security Considerations The procedures in this document deliberately degrade the output behavior of a serving deployment. Applying them to a deployment serving production traffic exposes users to ungrounded, unsafe, or incorrectly refused responses. The test environment MUST be isolated from production traffic. The fault injection apparatus MUST NOT be reachable from a production control plane. The reasoning of applies to this document. A published Detection Threshold Magnitude states the degradation an operator would not observe. A published False Detection Rate indicates how much noise an adversary would need to generate to exhaust operator attention. Publishers SHOULD consider the disclosure audience for both.

IANA Considerations This document has no IANA actions.

Normative References Informative References

Acknowledgements This document follows the methodology structure established by and applied to data center benchmarking in . The binary search of follows the throughput search of . The author thanks the Benchmarking Methodology Working Group for that foundation.