| Internet-Draft | Network Management | March 2025 |
| Liu & Guo | Expires 4 September 2025 | [Page] |
With the rise of ChatGPT, DeepSeek, and other Large Language Models, which is short for LLMs in the remaining part, as well as the proliferation of inference applications, inference serving oriented to large-scale users has become increasingly critical. However, due to the extreme demands on computing power and communication during inference, the large-scale service deployment of LLMs poses significant challenges. To address these challenges, different vendors have adopted diverse inference service architectures, among which the vLLM proposed in 2023 is the most representative. This paper investigates mainstream inference frameworks, summarizes their core design principles, and analyzes the requirements and challenges they impose on system and network configurations. The goal is to lay a foundation for defining a unified LLM inference architecture in the future.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 4 September 2025.¶
Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Since the launch of ChatGPT in 2023, more and more product-level LLMs have emerged, with GPT-4o, Claude-Sonnet-3.5, Gemini, Kimi, and others leading the charge. In early 2025, DeepSeek-R1 reignited the LLM frenzy, and Musk's xAI recently unveiled the powerful Grok3. It is evident that LLMs will continue to reach new heights.¶
Major vendors, including OpenAI, Anthropic, DeepSeek, and Google, have deployed their LLM applications across mobile and web platforms. As the field grows, daily active users (DAUs) for these applications are expected to surge, potentially reaching hundreds of millions during peak periods. This presents significant challenges for large-scale inference services. For instance, up to now, DeepSeek still struggles with persistent "Service Busy" issues.¶
Existing large-scale inference service architectures primarily adopt two technical approaches: Prefill-Decoding (PD) Fusion and Prefill-Decoding Disaggregation, which is derived from the distinct computational characteristics of the Prefill (compute-intensive) and Decoding (memory-intensive) phases. Efficient network management and hardware coordination are essential to maximize system throughput and minimize user-perceived latency.¶
This document first introduces mainstream inference frameworks, then optimization metrics, and finally elaborates on the network and system requirements for deploying large-scale LLM inference services.¶
At present, there are two main technical routes of the mainstream LLM service systems, namely PD Fusion and PD Disaggregation. Prefill, which is to simultaneously compute all of tokens of user requests, also known as prompts, is characterized as computational intensive, computing-bound, with extremely high computing force requirements. Decoding generates user-required content based on the KV Cache and first token generated by Prefill phase. Due to the reuse of KV Cache of the tokens prior to the current token, it is characterized as memory-intensive and memory-bound, with higher requirements for memory in decoding phase. A complete LLM inference procedure is shown in Figure 1. Based on whether to decouple two stages with obviously different computing requirements, two technical routes of LLM inference serving system emerge, namely, PD Fusion and decoupled PD Disaggregation. The rest of this section describes in detail about the two technical architectures.¶
+-------------+ +-------------+ +-------------+ +-------------+
| LLM | | LLM | | LLM | | LLM |
| Iteration 1 +-+| Iteration 2 ++| Iteration 3 ++| Iteration 4 ++
+-----^-------+ |+---^------^--+|+---^-------^-+|+---^-------^-+|
| | | | | | | | | | |
| | +--+--+ | | +--+--+ | | +--+--+ | |
<Prompt:Is apple | | KV | <Yes> | | KV | <It> | | KV | <Is> |<EOS>
a fruit?>| |Cache| ^ | |Cache| ^ | |Cache| ^ | ^
| +--^--+ | | +--^--+ | | +--^--+ | | |
| | | | | | | | | | |
+----+------+ +----+-------+ +----+-------+ +--+
+-----Prefill----+--------------------Decoding----------------------+
Prefill: Processes all tokens in user prompts (Parallelizable, compute-bound, requiring high computing power).¶
Decoding: Generates output tokens sequentially based on the KV Cache from Prefill (Memory-bound, requiring high GPU memory).¶
In PD Fusion, LLM instances are deployed within a single cluster, managed by a global scheduler responsible for load balancing, KV Cache management, and resource allocation. Most frameworks adopt vLLM[vLLM]'s paged KV Cache mechanism, inspired by OS virtual memory management. This approach stores KV Cache into non-contiguous physical blocks across nodes and uses a scheduler to map logical blocks to physical memory. Additionally, prefix-sharing strategies are employed to reuse KV Cache for prompts with identical prefixes, reducing redundant computations. Remote KV Cache replication across nodes is also required to reduce duplicated computing of KV Cache of same tokens. The architecture is shown in Figure 2.¶
Request1/Prompt1
Request2/Prompt2
|
|
+------------v------------+
| |
| Scheduler/Controller |
Request1 | | Request2
+-----------+ ********************* +----------+
| | *KV Cache Management* | |
| | * Load Balancing * | |
| | * ... ... * | |
| | ********************* | |
| +-------------------------+ |
| |
| |
| |
+----v-----+ Remote +----------+ Remote +----v-----+
| Model |KVCache copy| Model | KVCache copy| Model |
|Instance 1<----------->|Instance 2|<------------>Instance 3|
+----------+ +----------+ +----------+
In PD Disaggregation, Prefill and Decoding are decoupled into separate instances to optimize hardware utilization. After Prefill computes the full KV Cache for a prompt, the data is transferred to Decoding instances for text generation. This architecture demands efficient coordination between Prefill and Decoding instances, as well as reliable high-speed data transmission. The workflow is illustrated in Figure 3.¶
Request1/Prompt1
Request2/Prompt2
|
|
+------------v------------+
| |
| Scheduler/Controller |
Request1 | | Request2
+-----------+ ********************* +----------+
| | *KV Cache Management* | |
| | * Load Balancing * | |
| | * ... ... * | |
| | ********************* | |
| +-------------------------+ |
| |
| |
| |
+----v-----+ +----v-----+
| Model | | Model |
| | Remote KVCache copy | |
| Prefill <-------------------------------------> Prefill |
|Instance 1| |Instance 2|
+----+-----+ +----+-----+
|KV Cache KV Cache|
|Transfer Transfer|
| |
+----v-----+ +----v-----+
| Model | | Model |
| | | |
|Decoding | |Decoding |
|Instance 1| |Instance 2|
+----------+ +----------+
The ultimate goals of an inference system are to maximize system goodput which reflects the serving volume of user requests and minimize user-perceived latency. For PD Disaggregation architectures, two key metrics are defined as follows:¶
TTFT (Time to First Token): The time taken by the Prefill phase to generate the first token.¶
TBT (Time Between Tokens): The interval between consecutive token generations in the Decoding phase.¶
Optimization aims to minimize both TTFT and TBT under resource constraints and SLO constraints.¶
To achieve large-scale LLM service deployment, frameworks MUST address the following challenges in both control plane and data plane.¶
Both PD Fusion and PD Disaggregation architectures require dynamic load balancing to prevent server overload. For PD Disaggregation, schedulers MUST consider compute constraints (Prefill) and memory constraints (Decoding) when distributing requests.¶
Effective KV Cache management is critical. Most frameworks adopt vLLM’s paged KV Cache mechanism, schedulers are REQUIRED to handle memory allocation, cross-request KV cache sharing, and KV cache replacement policies. Future optimizations must address exponential user growth and ensure efficient cache synchronization across clusters or nodes.¶
PD Disaggregation architectures demand high-speed, reliable transmission of KV Cache data between Prefill and Decoding instances. The network MUST provide low-latency, high-bandwidth channels to ensure seamless coordination.¶
TBD.¶