Network Working Group H. Pan Internet-Draft R. Li Intended status: Informational China Mobile Expires: 7 January 2027 6 July 2026 Transparent Ordered Bonding for High-Capacity WAN Links draft-pan-transparent-ordered-bonding-00 Abstract This document describes a Transparent Ordered Bonding (TOB) mechanism for high-capacity wide-area links that are physically composed of multiple lower-rate member links. Traditional flow-based load balancing mechanisms can preserve packet ordering, but they cannot fully utilize all member links for a single large traffic stream. TOB introduces a bonding layer between two adjacent network nodes. The bonding layer fragments ingress packets into cells, distributes them across member links according to link status, and reassembles them in strict order at the remote node. This document specifies the applicability, architecture, packet format, scheduling considerations, and resequencing behavior of TOB. Note The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 7 January 2027. Pan & Li Expires 7 January 2027 [Page 1] Internet-Draft Ordered Bonding July 2026 Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Deployment Scenarios . . . . . . . . . . . . . . . . . . . . 4 3.1. Deployment over Bundled WAN Links . . . . . . . . . . . . 4 3.2. Deployment between Data Center Gateways . . . . . . . . . 4 4. Overview of TOB Framework . . . . . . . . . . . . . . . . . . 4 5. TOB Sender Behavior . . . . . . . . . . . . . . . . . . . . . 5 6. TOB Receiver Behavior . . . . . . . . . . . . . . . . . . . . 6 7. TOB Control Plane . . . . . . . . . . . . . . . . . . . . . . 7 8. TOB Packet Format . . . . . . . . . . . . . . . . . . . . . . 7 8.1. TOB Encapsulation . . . . . . . . . . . . . . . . . . . . 7 8.2. TOB Header Fields . . . . . . . . . . . . . . . . . . . . 8 9. Loss Recovery . . . . . . . . . . . . . . . . . . . . . . . . 9 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 11. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 9 12. Informative References . . . . . . . . . . . . . . . . . . . 9 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10 1. Introduction High-capacity wide-area network links are often implemented by bonding multiple lower-rate physical or logical member links. For example, a 400 Gbit/s logical connection between two network nodes may be constructed from eight 50 Gbit/s member links. In order to fully utilize the aggregate capacity, traffic needs to be distributed across all member links. Existing link aggregation and equal-cost multipath mechanisms commonly rely on hash-based traffic distribution. IEEE 802.1AX defines link aggregation for multiple point-to-point links [IEEE8021AX]. ECMP and link aggregation deployments typically use flow-based hashing in order to avoid packet reordering within a flow Pan & Li Expires 7 January 2027 [Page 2] Internet-Draft Ordered Bonding July 2026 [RFC2992] [RFC7424]. This behavior is appropriate for many network scenarios, but it also means that a single large flow may be pinned to one member link and cannot consume the full capacity of the logical bundle. Packet-level or cell-level striping across member links can improve utilization of the aggregate capacity. However, different member links may have different propagation delay, queuing delay, loss rate, or operational state. If packets are simply sprayed across member links, the receiver may observe packet reordering. Since the bonding layer cannot assume that the end-to-end transport protocol has a strong in-order delivery or reordering capability, the bonding mechanism needs to provide in-order delivery before packets are released to the upper layer. This document defines a prototype mechanism named Transparent Ordered Bonding (TOB). TOB is inspired by the multilink fragmentation and resequencing model of Multilink PPP [RFC1990] and the connection- level sequencing concept used by Multipath TCP [RFC8684]. TOB is intended to be deployed between two adjacent network nodes, such as WAN edge routers, data center gateways, or transport network adaptation devices. The endpoints and upper-layer protocols are not required to be aware of the member-link structure. 2. Terminology The following terms are used in this document: * Transparent Ordered Bonding (TOB): A bonding mechanism that distributes traffic over multiple member links and delivers reassembled packets in their original order. * Bundle: A logical link composed of two or more member links. * Member Link: A physical link, logical tunnel, wavelength, timeslot, or other transport resource that carries TOB cells. * TOB Cell: A fragment generated by the TOB sender. A TOB cell contains a TOB header and a portion of an original packet. * Frame Sequence Number: A monotonically increasing sequence number assigned to each original ingress packet before fragmentation. * Fragment Sequence Number: A monotonically increasing sequence number assigned to each TOB cell. Pan & Li Expires 7 January 2027 [Page 3] Internet-Draft Ordered Bonding July 2026 * Resequencing Buffer: A receiver-side buffer that stores out-of- order cells and completed packets until all earlier packets have been delivered. 3. Deployment Scenarios 3.1. Deployment over Bundled WAN Links In wide-area network deployments, operators may provide a high- capacity logical link by combining multiple lower-rate links. The member links may be parallel optical channels, Ethernet services, packet tunnels, or other transport resources. If flow-based distribution is used, a small number of large flows may lead to uneven member-link utilization. TOB enables the two edge nodes of the bundle to stripe cells across all available member links while preserving packet order toward the customer-facing interface. In this scenario, TOB sender and TOB receiver functions are implemented on the two WAN edge nodes. The WAN edge nodes monitor member-link rate, delay, loss, and queue depth. The sender schedules TOB cells across member links, and the receiver performs verification, resequencing, reassembly, and ordered delivery. 3.2. Deployment between Data Center Gateways Distributed computing, disaster recovery, and cross-data-center storage replication often require high-throughput data transfer between data centers. The traffic carried over the WAN may include TCP, QUIC, UDP, RDMA-based encapsulations, or proprietary transport protocols. A network operator may not be able to rely on any specific endpoint transport behavior for packet reordering recovery. TOB can be deployed on data center gateways to provide a single high- capacity ordered logical link to the upper network layers. The gateways hide member-link differences and avoid exposing reordering to endpoint protocols. This is especially useful when member links have similar capacity but non-identical delay or when member links are added and removed dynamically according to operational state. 4. Overview of TOB Framework This section describes the overall architecture of the TOB framework. TOB introduces a bonding layer between two adjacent network nodes, as shown in Figure 1. Pan & Li Expires 7 January 2027 [Page 4] Internet-Draft Ordered Bonding July 2026 +-----------+ +-----------+ | | member 0 ======================> | | | TOB | member 1 ======================> | TOB | | Sender | ... | Receiver | | | member N ======================> | | +-----^-----+ +------v----+ | | | ordered packet stream | ordered packet stream +-----------------------------------------------+ Figure 1: Transparent Ordered Bonding Architecture The TOB sender receives packets from an upper interface, assigns a Frame Sequence Number, fragments the packet into TOB cells, assigns Fragment Sequence Numbers, and transmits the cells over selected member links. The TOB receiver validates cells, stores out-of-order cells in a resequencing buffer, reassembles original packets, and releases only the next expected packet to the upper interface. The TOB layer is transparent to endpoints. The endpoints do not need to negotiate multipath capability, change congestion control behavior, or understand the number of member links in the bundle. TOB operates hop-by-hop between the two nodes that terminate the bundle. 5. TOB Sender Behavior The TOB sender performs the following functions: * Assign a monotonically increasing Frame Sequence Number to each ingress packet. * Fragment the ingress packet into one or more TOB cells according to the configured cell size and member-link MTU. * Assign a monotonically increasing Fragment Sequence Number to each TOB cell. * Select an output member link for each TOB cell according to the scheduling algorithm. * Maintain a transmission buffer until cells have been acknowledged or otherwise considered recoverable by a configured loss-recovery mechanism. The scheduler SHOULD consider member-link rate, queue occupancy, measured delay, loss, and administrative weight. For member links with the same capacity and similar delay, a weighted deficit round- Pan & Li Expires 7 January 2027 [Page 5] Internet-Draft Ordered Bonding July 2026 robin scheduler is sufficient. When member links have non-negligible differential delay, the scheduler SHOULD reduce the amount of traffic sent to paths that would significantly increase resequencing depth. An implementation MAY use the following scheduling score for each member link: score[i] = queue_bytes[i] / rate[i] + measured_delay[i] The sender selects the member link with the lowest score among eligible member links. Other algorithms are possible, as long as they preserve the receiver's ability to reassemble packets and deliver them in order. 6. TOB Receiver Behavior The TOB receiver performs cell validation, missing-cell detection, packet reassembly, and ordered delivery. The receiver MUST NOT deliver a reassembled packet with a Frame Sequence Number greater than the next expected Frame Sequence Number unless all lower Frame Sequence Numbers have already been delivered or declared unrecoverable according to a configured policy. The basic receiver procedure is as follows: on_cell(cell): verify(cell) store cell by Frame Sequence Number and Fragment Offset while packet[next_frame_seq] is complete: reassemble packet[next_frame_seq] deliver packet[next_frame_seq] release buffer for packet[next_frame_seq] next_frame_seq = next_frame_seq + 1 If a missing Fragment Sequence Number or incomplete packet is detected, the receiver MAY request retransmission by sending a negative acknowledgment to the sender. Alternatively, if forward error correction is enabled, the receiver MAY recover missing cells from parity cells before requesting retransmission. The receiver needs to provision the resequencing buffer according to the aggregate bundle rate and the maximum expected differential delay among member links. A minimum buffer estimate is: reorder_buffer >= aggregate_rate * max_differential_delay Pan & Li Expires 7 January 2027 [Page 6] Internet-Draft Ordered Bonding July 2026 For example, a 400 Gbit/s bundle with 2 ms maximum differential delay requires approximately 100 MBytes of resequencing buffer before considering burst absorption, retransmission, or implementation overhead. A practical implementation SHOULD apply a safety factor based on operator policy and measured traffic burstiness. 7. TOB Control Plane TOB requires a control function between the two bundle endpoints. This document does not mandate a specific control protocol, but the control function SHOULD support the following capabilities: * Discovery and authentication of member links belonging to the same bundle. * Negotiation of TOB version, cell size, maximum frame size, maximum resequencing window, loss-recovery mode, and integrity protection mode. * Member-link liveness detection and removal of failed member links from the scheduler. * Generation number synchronization when the bundle membership or critical operating parameters change. * Telemetry exchange for member-link rate, delay, loss, jitter, and queue depth. The Generation Number is used to distinguish cells sent under different bundle configurations. When bundle membership changes, the sender and receiver SHOULD complete or drain cells from the old generation before fully switching to the new generation, unless an implementation-specific recovery policy is used. 8. TOB Packet Format This section defines a prototype TOB cell format. TOB may be carried directly over an Ethernet service using a dedicated EtherType, or it may be carried inside an IP tunnel such as UDP or GRE. The exact encapsulation is deployment-specific. 8.1. TOB Encapsulation When TOB is carried over Ethernet, a TOB cell is encapsulated as follows: [ETH + TOB Header + TOB Payload + Integrity Check] Pan & Li Expires 7 January 2027 [Page 7] Internet-Draft Ordered Bonding July 2026 When TOB is carried over UDP/IP, a TOB cell is encapsulated as follows: [ETH + IP + UDP + TOB Header + TOB Payload + Integrity Check] The UDP destination port or EtherType value is outside the scope of this prototype document and is expected to be assigned or configured according to deployment requirements. 8.2. TOB Header Fields TOB Header { Version (8), Header Length (8), Flags (16), Bundle ID (32), Generation Number (32), Member ID (16), Traffic Class (16), Frame Sequence Number (64), Fragment Sequence Number (64), Fragment Offset (32), Original Frame Length (32), Header CRC (32) } * Version (8 bits): TOB protocol version. The initial version defined in this document is 0x01. * Header Length (8 bits): Length of the TOB header in 4-octet units. * Flags (16 bits): Control flags. This document defines BEGIN, END, SINGLE, RETX, FEC, ACK, and NACK flags. Additional flags are reserved for future use. * Bundle ID (32 bits): Identifier of the logical bundle. * Generation Number (32 bits): Identifier of the active bundle configuration. * Member ID (16 bits): Identifier of the member link selected by the sender. * Traffic Class (16 bits): Traffic class or quality-of-service indication copied from the ingress packet or assigned by local policy. Pan & Li Expires 7 January 2027 [Page 8] Internet-Draft Ordered Bonding July 2026 * Frame Sequence Number (64 bits): Sequence number of the original ingress packet. * Fragment Sequence Number (64 bits): Sequence number of the TOB cell. * Fragment Offset (32 bits): Offset of the payload carried in this TOB cell within the original ingress packet. * Original Frame Length (32 bits): Length of the original ingress packet before fragmentation. * Header CRC (32 bits): Integrity check for the TOB header. 9. Loss Recovery TOB deployments may operate over member links with different loss characteristics. This document defines three prototype operating modes: * Resequencing-only mode: The receiver only compensates for differential delay and reordering. Loss recovery is left to lower or upper layers. * Selective retransmission mode: The receiver sends NACK messages for missing cells. The sender retransmits missing cells on any healthy member link. * FEC-assisted mode: The sender periodically transmits parity cells for a block of data cells. The receiver uses the parity cells to recover from limited loss before requesting retransmission. The configured loss-recovery mode needs to balance latency, bandwidth overhead, and head-of-line blocking. In all modes, the receiver MUST have a policy for declaring a missing packet unrecoverable. Otherwise, a single lost cell can block delivery of all subsequent packets. 10. IANA Considerations This document does not require any IANA actions. Future versions may request assignment of a UDP port, EtherType, or protocol registry values if a common encapsulation is standardized. 11. Contributors 12. Informative References Pan & Li Expires 7 January 2027 [Page 9] Internet-Draft Ordered Bonding July 2026 [IEEE8021AX] IEEE, "IEEE Standard for Local and Metropolitan Area Networks--Link Aggregation", IEEE 802.1AX-2020, 2020. [RFC1990] Sklower, K., Lloyd, B., McGregor, G., Carr, D., and T. Coradetti, "The PPP Multilink Protocol (MP)", RFC 1990, August 1996, . [RFC2992] Hopps, C., "Analysis of an Equal-Cost Multi-Path Algorithm", RFC 2992, November 2000, . [RFC7424] Krishnan, Y., Yong, A., Haleplidis, E., Pentikousis, K., and J. Halpern, "Mechanisms for Optimizing Link Aggregation Group (LAG) and Equal-Cost Multipath (ECMP) Component Link Utilization in Networks", RFC 7424, January 2015, . [RFC8684] Ford, A., Raiciu, C., Handley, M., Bonaventure, O., and C. Paasch, "TCP Extensions for Multipath Operation with Multiple Addresses", RFC 8684, March 2020, . Authors' Addresses Haoyu Pan China Mobile Beijing China Email: panhaoyu@chinamobile.com Ruifeng Li China Mobile Beijing China Email: liruifengyjy@chinamobile.com Pan & Li Expires 7 January 2027 [Page 10]