Transparent Ordered Bonding for High-Capacity WAN Links

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 7 January 2027.¶

Copyright Notice

Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

High-capacity wide-area network links are often implemented by bonding multiple lower-rate physical or logical member links. For example, a 400 Gbit/s logical connection between two network nodes may be constructed from eight 50 Gbit/s member links. In order to fully utilize the aggregate capacity, traffic needs to be distributed across all member links.¶

Existing link aggregation and equal-cost multipath mechanisms commonly rely on hash-based traffic distribution. IEEE 802.1AX defines link aggregation for multiple point-to-point links [IEEE8021AX]. ECMP and link aggregation deployments typically use flow-based hashing in order to avoid packet reordering within a flow [RFC2992] [RFC7424]. This behavior is appropriate for many network scenarios, but it also means that a single large flow may be pinned to one member link and cannot consume the full capacity of the logical bundle.¶

Packet-level or cell-level striping across member links can improve utilization of the aggregate capacity. However, different member links may have different propagation delay, queuing delay, loss rate, or operational state. If packets are simply sprayed across member links, the receiver may observe packet reordering. Since the bonding layer cannot assume that the end-to-end transport protocol has a strong in-order delivery or reordering capability, the bonding mechanism needs to provide in-order delivery before packets are released to the upper layer.¶

This document defines a prototype mechanism named Transparent Ordered Bonding (TOB). TOB is inspired by the multilink fragmentation and resequencing model of Multilink PPP [RFC1990] and the connection-level sequencing concept used by Multipath TCP [RFC8684]. TOB is intended to be deployed between two adjacent network nodes, such as WAN edge routers, data center gateways, or transport network adaptation devices. The endpoints and upper-layer protocols are not required to be aware of the member-link structure.¶

2. Terminology

The following terms are used in this document:¶

• Transparent Ordered Bonding (TOB): A bonding mechanism that distributes traffic over multiple member links and delivers reassembled packets in their original order.¶

• Bundle: A logical link composed of two or more member links.¶

• Member Link: A physical link, logical tunnel, wavelength, timeslot, or other transport resource that carries TOB cells.¶

• TOB Cell: A fragment generated by the TOB sender. A TOB cell contains a TOB header and a portion of an original packet.¶

• Frame Sequence Number: A monotonically increasing sequence number assigned to each original ingress packet before fragmentation.¶

• Fragment Sequence Number: A monotonically increasing sequence number assigned to each TOB cell.¶

• Resequencing Buffer: A receiver-side buffer that stores out-of-order cells and completed packets until all earlier packets have been delivered.¶

3. Deployment Scenarios

4. Overview of TOB Framework

This section describes the overall architecture of the TOB framework.¶

TOB introduces a bonding layer between two adjacent network nodes, as shown in Figure 1.¶

+-----------+                                  +-----------+
|           |   member 0  ======================>  |           |
|  TOB      |   member 1  ======================>  |  TOB      |
|  Sender   |   ...                               |  Receiver |
|           |   member N  ======================>  |           |
+-----^-----+                                  +------v----+
      |                                               |
      | ordered packet stream                         | ordered packet stream
      +-----------------------------------------------+

The TOB sender receives packets from an upper interface, assigns a Frame Sequence Number, fragments the packet into TOB cells, assigns Fragment Sequence Numbers, and transmits the cells over selected member links. The TOB receiver validates cells, stores out-of-order cells in a resequencing buffer, reassembles original packets, and releases only the next expected packet to the upper interface.¶

The TOB layer is transparent to endpoints. The endpoints do not need to negotiate multipath capability, change congestion control behavior, or understand the number of member links in the bundle. TOB operates hop-by-hop between the two nodes that terminate the bundle.¶

5. TOB Sender Behavior

The TOB sender performs the following functions:¶

• Assign a monotonically increasing Frame Sequence Number to each ingress packet.¶

• Fragment the ingress packet into one or more TOB cells according to the configured cell size and member-link MTU.¶

• Assign a monotonically increasing Fragment Sequence Number to each TOB cell.¶

• Select an output member link for each TOB cell according to the scheduling algorithm.¶

• Maintain a transmission buffer until cells have been acknowledged or otherwise considered recoverable by a configured loss-recovery mechanism.¶

The scheduler SHOULD consider member-link rate, queue occupancy, measured delay, loss, and administrative weight. For member links with the same capacity and similar delay, a weighted deficit round-robin scheduler is sufficient. When member links have non-negligible differential delay, the scheduler SHOULD reduce the amount of traffic sent to paths that would significantly increase resequencing depth.¶

An implementation MAY use the following scheduling score for each member link:¶

score[i] = queue_bytes[i] / rate[i] + measured_delay[i]

The sender selects the member link with the lowest score among eligible member links. Other algorithms are possible, as long as they preserve the receiver's ability to reassemble packets and deliver them in order.¶

6. TOB Receiver Behavior

The TOB receiver performs cell validation, missing-cell detection, packet reassembly, and ordered delivery. The receiver MUST NOT deliver a reassembled packet with a Frame Sequence Number greater than the next expected Frame Sequence Number unless all lower Frame Sequence Numbers have already been delivered or declared unrecoverable according to a configured policy.¶

The basic receiver procedure is as follows:¶

on_cell(cell):
    verify(cell)
    store cell by Frame Sequence Number and Fragment Offset

    while packet[next_frame_seq] is complete:
        reassemble packet[next_frame_seq]
        deliver packet[next_frame_seq]
        release buffer for packet[next_frame_seq]
        next_frame_seq = next_frame_seq + 1

If a missing Fragment Sequence Number or incomplete packet is detected, the receiver MAY request retransmission by sending a negative acknowledgment to the sender. Alternatively, if forward error correction is enabled, the receiver MAY recover missing cells from parity cells before requesting retransmission.¶

The receiver needs to provision the resequencing buffer according to the aggregate bundle rate and the maximum expected differential delay among member links. A minimum buffer estimate is:¶

reorder_buffer >= aggregate_rate * max_differential_delay

For example, a 400 Gbit/s bundle with 2 ms maximum differential delay requires approximately 100 MBytes of resequencing buffer before considering burst absorption, retransmission, or implementation overhead. A practical implementation SHOULD apply a safety factor based on operator policy and measured traffic burstiness.¶

7. TOB Control Plane

TOB requires a control function between the two bundle endpoints. This document does not mandate a specific control protocol, but the control function SHOULD support the following capabilities:¶

• Discovery and authentication of member links belonging to the same bundle.¶

• Negotiation of TOB version, cell size, maximum frame size, maximum resequencing window, loss-recovery mode, and integrity protection mode.¶

• Member-link liveness detection and removal of failed member links from the scheduler.¶

• Generation number synchronization when the bundle membership or critical operating parameters change.¶

• Telemetry exchange for member-link rate, delay, loss, jitter, and queue depth.¶

The Generation Number is used to distinguish cells sent under different bundle configurations. When bundle membership changes, the sender and receiver SHOULD complete or drain cells from the old generation before fully switching to the new generation, unless an implementation-specific recovery policy is used.¶

8. TOB Packet Format

This section defines a prototype TOB cell format. TOB may be carried directly over an Ethernet service using a dedicated EtherType, or it may be carried inside an IP tunnel such as UDP or GRE. The exact encapsulation is deployment-specific.¶

[ETH + TOB Header + TOB Payload + Integrity Check]

[ETH + IP + UDP + TOB Header + TOB Payload + Integrity Check]

TOB Header {
    Version (8),
    Header Length (8),
    Flags (16),
    Bundle ID (32),
    Generation Number (32),
    Member ID (16),
    Traffic Class (16),
    Frame Sequence Number (64),
    Fragment Sequence Number (64),
    Fragment Offset (32),
    Original Frame Length (32),
    Header CRC (32)
}

9. Loss Recovery

TOB deployments may operate over member links with different loss characteristics. This document defines three prototype operating modes:¶

• Resequencing-only mode: The receiver only compensates for differential delay and reordering. Loss recovery is left to lower or upper layers.¶

• Selective retransmission mode: The receiver sends NACK messages for missing cells. The sender retransmits missing cells on any healthy member link.¶

• FEC-assisted mode: The sender periodically transmits parity cells for a block of data cells. The receiver uses the parity cells to recover from limited loss before requesting retransmission.¶

The configured loss-recovery mode needs to balance latency, bandwidth overhead, and head-of-line blocking. In all modes, the receiver MUST have a policy for declaring a missing packet unrecoverable. Otherwise, a single lost cell can block delivery of all subsequent packets.¶

10. IANA Considerations

This document does not require any IANA actions. Future versions may request assignment of a UDP port, EtherType, or protocol registry values if a common encapsulation is standardized.¶

11. Contributors

Authors' Addresses