KV Cache Distribution for Distributed LLM Inference: Use Case and Requirements

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 5 January 2027.¶

Copyright Notice

Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

LLM applications routinely require knowledge beyond what is contained in model parameters: conversation history for personalized assistants, enterprise data for domain-specific agents, retrieved documents for search-augmented responses. Two established approaches address this need, each with trade-offs. In-context approaches (prompt construction and retrieval-augmented generation) supply external knowledge at inference time without modifying the model, but are constrained by the model's context window and incur per-request retrieval and prefill computation. Fine-tuning adapts model parameters to new knowledge, but requires training resources and labeled data, and binds the result to specific tasks.¶

KV cache reuse occupies a useful middle ground. The key-value (KV) cache holds the attention state computed from previously processed tokens. Reusing cached state across requests avoids repeated prefill computation and reduces time-to-first-token. In distributed inference deployments, the KV cache becomes a network-distributed resource: the effectiveness of steering a request to a service instance depends not only on computing and network metrics but also on whether reusable cached state is available at or near that instance.¶

This document describes the KV cache distribution use case for Computing-Aware Traffic Steering (CATS), identifies the gaps relative to the existing CATS framework and metrics, and states requirements for cache-state metric exposure and for the distribution and synchronization of cached content across multiple cache tiers.¶

2. Terminology

KV Cache:

Intermediate attention state (keys and values) produced during inference. In disaggregated serving, the KV cache is transferred from prefill servers to decode servers.¶

Prefill:

The inference phase that processes the input prompt and produces the initial KV cache.¶

Decode:

The inference phase that generates output tokens incrementally, consuming and extending the KV cache.¶

Expert Parallelism (EP):

A parallelization strategy for mixture-of-experts models in which experts are distributed across servers, requiring all-to-all token exchange.¶

3. Problem Statement

In distributed LLM inference, the KV cache computed during prefill is a valuable resource that can be reused to avoid recomputation. When a request arrives, steering it to an instance that already holds the relevant KV cache state avoids prefill latency. However, the KV cache is distributed across service instances and cache tiers; a CATS decision that considers only computing load and network metrics cannot account for cache availability.¶

This creates a gap: the existing CATS metrics do not expose cache state, and the CATS distribution framework does not address how cached content is distributed or synchronized across instances. This document states requirements to close this gap.¶

4. Cache-State Metric Exposure

A service instance SHOULD expose cache-state metrics that allow a CATS decision function to determine whether reusable KV cache state is available. These metrics include: the set of cache keys (e.g., token-prefix identifiers or content hashes) present at the instance; the cache entry sizes; and the expected retrieval cost. The metrics MUST be expressible in the CATS metric framework so that they can be combined with computing and network metrics in steering decisions.¶

5. Distribution and Synchronization of Cached Content

For cache reuse to be effective across instances, cached content may need to be distributed or migrated. This document identifies the following requirements: (a) a mechanism to advertise cache availability across instances; (b) a mechanism to transfer KV cache state between instances or cache tiers, respecting latency and bandwidth constraints; (c) consistency handling for cache entries that are updated or evicted. The specific transfer protocol is outside the scope of this document but should build on existing transport building blocks.¶

6. Security Considerations

Cache-state metrics and cache content may reveal information about the inputs being processed. Exposure of cache metrics MUST be restricted to authorized CATS components. Transfer of KV cache state between instances SHOULD be protected for integrity and confidentiality.¶

7. IANA Considerations

This document has no IANA actions.¶

Authors' Addresses