Benchmarking Methodology for Output Behavior Fault Detection in Large Language Model Serving Systems

1.1. Scope

This document specifies test procedures, required test conditions, and a reporting format. It does not specify acceptance thresholds, target values, or service-level objectives. A result obtained by these procedures characterizes a detector. It does not establish whether that detector is adequate for any purpose.¶

The mechanism by which a fault is produced is out of scope. Any mechanism that shifts the declared indicator by the declared magnitude at the declared time is acceptable. Section 3.4 states the properties an injection apparatus must have.¶

Benchmarking described by this document is confined to a test environment. Section 7 states this requirement.¶

1.2. Applicability

A result obtained by these procedures characterizes one Detector Under Test in one declared environment. The serving deployment includes a language model whose output is stochastic, so the baseline against which fault magnitude is expressed is a property of that environment. Section 4.2 requires the baseline to be established for each environment before any fault trial.¶

Two results are comparable when the Observation Point, the DUT configuration, the serving deployment configuration, the request population, and the offered load are the same. Results MUST NOT be compared when any of these differ. Section 6 requires each to be reported so that a reader can determine whether a comparison is valid.¶

The intended uses follow from this. A detector configuration can be compared against another configuration on the same environment. A detector can be tracked across changes to itself or to the deployment it observes. A detector can be characterized before it is relied on. These procedures are not intended to produce values that rank detectors across organizations, and a value obtained here carries no meaning outside the environment in which it was obtained.¶

A full series under these procedures is long. Twenty trials at each combination of Fault Class, magnitude, profile, and load, each followed by a quiet period, together with the search of Section 5.3, occupies substantial test time. A reduced series MAY be run for screening, using fewer trials, fewer load points, or a coarser search resolution. A reduced series MUST be reported as such, with the reductions stated, and MUST NOT be reported as a result under this methodology.¶

3.1. Topology

The test environment comprises four components.¶

Traffic generator:

Offers requests to the serving deployment at a controlled rate from a declared request population.¶

Serving deployment:

The LLM serving stack under observation, configured to one of the System Under Test boundaries of [LLM-TERMS].¶

Fault injection apparatus:

Applies Output Behavior Faults at the Fault Injection Point and records Fault Onset.¶

Detector Under Test:

The observability system, sampling at the Observation Point and emitting the declared detection signal.¶

The relative position of the Fault Injection Point and the Observation Point MUST be recorded and reported. A fault injected downstream of the Observation Point is not observable by the detector, and trials in that configuration measure only Boundary Masking.¶

3.2. Detector Under Test Configuration

The DUT configuration MUST be frozen before the first trial and MUST NOT be changed during a test series. Changing thresholds, sampling rates, judge models, or alerting rules begins a new test series.¶

The following MUST be recorded: the indicators the DUT computes, the method used to compute each, the sampling rate for any sampled indicator, the aggregation window, the alerting thresholds, and the version of any model used as a judge.¶

3.3. Traffic Generation

Offered load MUST be constant within a trial. The request population MUST be constant across all trials in a series.¶

Detection Latency for a rate-valued indicator depends on how quickly evidence accumulates, and evidence accumulates with requests rather than with time. Results are therefore reported against offered load. A test series SHOULD cover at least four load points spanning at least two orders of magnitude, and SHOULD include a load point representative of the intended deployment.¶

Response caching in the serving deployment suppresses the indicator shift a fault is intended to produce. Caching MUST be disabled, or the request population MUST be constructed so that cache hits do not occur. Which approach was used MUST be reported.¶

3.4. Fault Injection Apparatus

The apparatus MUST record Fault Onset with a resolution of one millisecond or finer, and MUST record it independently of any DUT output.¶

The apparatus MUST be able to remove an applied fault and return the deployment to its baseline configuration. The apparatus MUST NOT emit any signal to the DUT other than through the behavior of the serving deployment. An apparatus that notifies the DUT of injection invalidates the measurement.¶

The magnitude actually achieved MUST be verified against the magnitude intended, by measuring the affected indicator at the Fault Injection Point during application. A trial in which the achieved magnitude differs from the intended magnitude by more than ten percent MUST be discarded and repeated.¶

3.5. Clock Synchronization

Fault Onset and Detection Event may be recorded by different hosts. Detection Latency is the difference between them, and clock offset between those hosts enters the result directly as error.¶

The maximum clock offset between the injection apparatus and the DUT MUST be measured and reported. The offset MUST be smaller than one percent of the smallest Detection Latency reported in the series. Where this cannot be achieved, the offset MUST be reported as an error bound on every latency value in the series.¶

Where the injection apparatus and the DUT record timestamps on the same host, this requirement does not apply and the arrangement MUST be reported.¶

4.1. Detection Signal Declaration

The signal that constitutes a Detection Event MUST be declared before the first trial. The declaration MUST state the signal type, the component that emits it, and the criterion by which a signal is attributed to an applied fault.¶

The declaration MUST NOT be changed after any trial in the series has been run. Selecting the detection signal after observing results invalidates the series.¶

4.2. Baseline Characterization

Before the first fault trial, a Baseline Interval MUST be run at each load point to be used in the series. The Baseline Interval MUST be at least ten times the DUT aggregation window, and MUST be at least three hundred seconds.¶

For each indicator to be faulted, the mean and standard deviation over the Baseline Interval MUST be recorded. Fault magnitude is reported as an absolute change from this mean.¶

A fault magnitude smaller than twice the baseline standard deviation is not distinguishable from baseline variation. Trials at such magnitudes MAY be run, and results from them MUST be reported with the baseline standard deviation alongside.¶

4.3. Inter-Trial Quiet Period

Observability systems commonly implement alert deduplication, flap suppression, and state that persists after a condition clears. A trial begun while the DUT retains state from a previous trial does not measure detection from a resting state.¶

Between trials, the fault MUST be removed and a quiet period MUST be observed. The quiet period MUST be long enough for the DUT to return every affected indicator to within one standard deviation of its baseline mean, and to clear any active alert state. The quiet period used MUST be reported.¶

4.4. Single Fault Application

One Output Behavior Fault MUST be applied at a time, unless the declared attribution criterion of Section 4.1 is able to attribute a signal to a specific fault in the presence of others. Where multiple faults are applied, the attribution criterion MUST be reported.¶

5.1. Detection Latency

Objective. Determine the interval from Fault Onset to the Detection Event, for a stated Fault Class, magnitude, Fault Profile, and offered load.¶

Procedure.¶

1. Offer traffic at the selected load. Wait for the quiet period of Section 4.3.¶
2. Apply the fault. Record Fault Onset.¶
3. Observe the DUT until a Detection Event attributable to the fault occurs, or until the trial timeout expires.¶
4. Record the Detection Event time, or record the trial as undetected.¶
5. Remove the fault. Observe the quiet period.¶
6. Repeat from step 2 until the required number of trials is complete.¶

Trial count. At least twenty trials MUST be run at each combination of Fault Class, magnitude, profile, and load.¶

Trial timeout. The timeout MUST be declared before the series and MUST be at least twenty times the DUT aggregation window. A trial reaching the timeout is recorded as undetected and contributes to Detection Coverage.¶

Reported value. The median and the ninety-fifth percentile of Detection Latency across detected trials MUST be reported, together with the number of trials and the number undetected. The mean MAY be reported. Values from undetected trials MUST NOT be imputed.¶

5.2. Detection Coverage

Objective. Determine the fraction of applied faults for which a Detection Event occurred, for a stated Fault Class, magnitude, profile, and load.¶

Procedure. Detection Coverage is computed from the same trials as Section 5.1. No separate trials are required.¶

Reported value. The number of trials in which a Detection Event occurred, divided by the number of trials run. The numerator and denominator MUST both be reported. A coverage value MUST NOT be reported without the fault magnitude at which it was obtained.¶

5.3. Detection Threshold Magnitude

Objective. Determine the smallest fault magnitude at which the DUT achieves a stated Detection Coverage, for a given Fault Class, profile, and load.¶

Procedure. The procedure is a binary search on magnitude, following the pattern of the throughput search in [RFC2544].¶

1. Select a target coverage level. Declare it before the search begins.¶
2. Establish an upper bound magnitude at which coverage meets or exceeds the target, and a lower bound at which it does not. Where the lower bound falls below twice the baseline standard deviation, record that the search floor was reached and stop.¶
3. Run the trials of Section 5.1 at the midpoint magnitude.¶
4. If coverage at the midpoint meets or exceeds the target, the midpoint becomes the new upper bound. Otherwise it becomes the new lower bound.¶
5. Repeat from step 3 until the interval between bounds is smaller than the declared resolution.¶

Reported value. The upper bound at termination, the target coverage level, the search resolution, and the baseline standard deviation of the affected indicator. Where the search floor was reached, the result MUST be reported as below the floor and MUST NOT be reported as a magnitude.¶

5.4. False Detection Rate

Objective. Determine the rate at which the DUT emits the declared detection signal when no fault is applied.¶

Procedure.¶

1. Offer traffic at the selected load with no fault applied.¶
2. Observe the DUT for the declared observation period. Record every emission of the declared detection signal.¶
3. Repeat at each load point in the series.¶

Observation period. The period MUST be at least the total time spent under fault application across the whole series at that load point. A False Detection Rate measured over a shorter period than the latency trials it accompanies understates the rate.¶

Reported value. The count of false detections and the observation period. A Detection Latency or Detection Threshold Magnitude result MUST NOT be published without an accompanying False Detection Rate obtained under the same DUT configuration and load.¶

5.5. Boundary Masking

Objective. Determine the reduction in observed fault magnitude between the Fault Injection Point and the Observation Point.¶

Procedure.¶

1. Apply a fault of known magnitude at the Fault Injection Point.¶
2. Measure the affected indicator at the Fault Injection Point and at the Observation Point over the same interval, using the same method at both points.¶
3. Remove the fault. Observe the quiet period. Repeat for the required number of trials.¶

Trial count. At least twenty trials MUST be run.¶

Reported value. One minus the ratio of the magnitude observed at the Observation Point to the magnitude observed at the Fault Injection Point, reported as a median across trials. The measurement method used at both points MUST be reported. Where the two points are the same, Boundary Masking is zero by construction and MUST be reported as not applicable.¶