Transparent Ordered Bonding for High-Capacity WAN Links

Introduction High-capacity wide-area network links are often implemented by bonding multiple lower-rate physical or logical member links. For example, a 400 Gbit/s logical connection between two network nodes may be constructed from eight 50 Gbit/s member links. In order to fully utilize the aggregate capacity, traffic needs to be distributed across all member links. Existing link aggregation and equal-cost multipath mechanisms commonly rely on hash-based traffic distribution. IEEE 802.1AX defines link aggregation for multiple point-to-point links . ECMP and link aggregation deployments typically use flow-based hashing in order to avoid packet reordering within a flow . This behavior is appropriate for many network scenarios, but it also means that a single large flow may be pinned to one member link and cannot consume the full capacity of the logical bundle. Packet-level or cell-level striping across member links can improve utilization of the aggregate capacity. However, different member links may have different propagation delay, queuing delay, loss rate, or operational state. If packets are simply sprayed across member links, the receiver may observe packet reordering. Since the bonding layer cannot assume that the end-to-end transport protocol has a strong in-order delivery or reordering capability, the bonding mechanism needs to provide in-order delivery before packets are released to the upper layer. This document defines a prototype mechanism named Transparent Ordered Bonding (TOB). TOB is inspired by the multilink fragmentation and resequencing model of Multilink PPP and the connection-level sequencing concept used by Multipath TCP . TOB is intended to be deployed between two adjacent network nodes, such as WAN edge routers, data center gateways, or transport network adaptation devices. The endpoints and upper-layer protocols are not required to be aware of the member-link structure.

Terminology The following terms are used in this document:

Transparent Ordered Bonding (TOB): A bonding mechanism that distributes traffic over multiple member links and delivers reassembled packets in their original order.
Bundle: A logical link composed of two or more member links.
Member Link: A physical link, logical tunnel, wavelength, timeslot, or other transport resource that carries TOB cells.
TOB Cell: A fragment generated by the TOB sender. A TOB cell contains a TOB header and a portion of an original packet.
Frame Sequence Number: A monotonically increasing sequence number assigned to each original ingress packet before fragmentation.
Fragment Sequence Number: A monotonically increasing sequence number assigned to each TOB cell.
Resequencing Buffer: A receiver-side buffer that stores out-of-order cells and completed packets until all earlier packets have been delivered.

Deployment Scenarios

Deployment over Bundled WAN Links In wide-area network deployments, operators may provide a high-capacity logical link by combining multiple lower-rate links. The member links may be parallel optical channels, Ethernet services, packet tunnels, or other transport resources. If flow-based distribution is used, a small number of large flows may lead to uneven member-link utilization. TOB enables the two edge nodes of the bundle to stripe cells across all available member links while preserving packet order toward the customer-facing interface. In this scenario, TOB sender and TOB receiver functions are implemented on the two WAN edge nodes. The WAN edge nodes monitor member-link rate, delay, loss, and queue depth. The sender schedules TOB cells across member links, and the receiver performs verification, resequencing, reassembly, and ordered delivery.

Deployment between Data Center Gateways Distributed computing, disaster recovery, and cross-data-center storage replication often require high-throughput data transfer between data centers. The traffic carried over the WAN may include TCP, QUIC, UDP, RDMA-based encapsulations, or proprietary transport protocols. A network operator may not be able to rely on any specific endpoint transport behavior for packet reordering recovery. TOB can be deployed on data center gateways to provide a single high-capacity ordered logical link to the upper network layers. The gateways hide member-link differences and avoid exposing reordering to endpoint protocols. This is especially useful when member links have similar capacity but non-identical delay or when member links are added and removed dynamically according to operational state.

Overview of TOB Framework This section describes the overall architecture of the TOB framework. TOB introduces a bonding layer between two adjacent network nodes, as shown in Figure 1.

Transparent Ordered Bonding Architecture +-----------+ +-----------+ | | member 0 ======================> | | | TOB | member 1 ======================> | TOB | | Sender | ... | Receiver | | | member N ======================> | | +-----^-----+ +------v----+ | | | ordered packet stream | ordered packet stream +-----------------------------------------------+ The TOB sender receives packets from an upper interface, assigns a Frame Sequence Number, fragments the packet into TOB cells, assigns Fragment Sequence Numbers, and transmits the cells over selected member links. The TOB receiver validates cells, stores out-of-order cells in a resequencing buffer, reassembles original packets, and releases only the next expected packet to the upper interface. The TOB layer is transparent to endpoints. The endpoints do not need to negotiate multipath capability, change congestion control behavior, or understand the number of member links in the bundle. TOB operates hop-by-hop between the two nodes that terminate the bundle.

TOB Sender Behavior The TOB sender performs the following functions:

Assign a monotonically increasing Frame Sequence Number to each ingress packet.
Fragment the ingress packet into one or more TOB cells according to the configured cell size and member-link MTU.
Assign a monotonically increasing Fragment Sequence Number to each TOB cell.
Select an output member link for each TOB cell according to the scheduling algorithm.
Maintain a transmission buffer until cells have been acknowledged or otherwise considered recoverable by a configured loss-recovery mechanism.

The scheduler SHOULD consider member-link rate, queue occupancy, measured delay, loss, and administrative weight. For member links with the same capacity and similar delay, a weighted deficit round-robin scheduler is sufficient. When member links have non-negligible differential delay, the scheduler SHOULD reduce the amount of traffic sent to paths that would significantly increase resequencing depth. An implementation MAY use the following scheduling score for each member link: score[i] = queue_bytes[i] / rate[i] + measured_delay[i] The sender selects the member link with the lowest score among eligible member links. Other algorithms are possible, as long as they preserve the receiver's ability to reassemble packets and deliver them in order.

TOB Receiver Behavior The TOB receiver performs cell validation, missing-cell detection, packet reassembly, and ordered delivery. The receiver MUST NOT deliver a reassembled packet with a Frame Sequence Number greater than the next expected Frame Sequence Number unless all lower Frame Sequence Numbers have already been delivered or declared unrecoverable according to a configured policy. The basic receiver procedure is as follows: on_cell(cell): verify(cell) store cell by Frame Sequence Number and Fragment Offset while packet[next_frame_seq] is complete: reassemble packet[next_frame_seq] deliver packet[next_frame_seq] release buffer for packet[next_frame_seq] next_frame_seq = next_frame_seq + 1 If a missing Fragment Sequence Number or incomplete packet is detected, the receiver MAY request retransmission by sending a negative acknowledgment to the sender. Alternatively, if forward error correction is enabled, the receiver MAY recover missing cells from parity cells before requesting retransmission. The receiver needs to provision the resequencing buffer according to the aggregate bundle rate and the maximum expected differential delay among member links. A minimum buffer estimate is: reorder_buffer >= aggregate_rate * max_differential_delay For example, a 400 Gbit/s bundle with 2 ms maximum differential delay requires approximately 100 MBytes of resequencing buffer before considering burst absorption, retransmission, or implementation overhead. A practical implementation SHOULD apply a safety factor based on operator policy and measured traffic burstiness.

TOB Control Plane TOB requires a control function between the two bundle endpoints. This document does not mandate a specific control protocol, but the control function SHOULD support the following capabilities:

Discovery and authentication of member links belonging to the same bundle.
Negotiation of TOB version, cell size, maximum frame size, maximum resequencing window, loss-recovery mode, and integrity protection mode.
Member-link liveness detection and removal of failed member links from the scheduler.
Generation number synchronization when the bundle membership or critical operating parameters change.
Telemetry exchange for member-link rate, delay, loss, jitter, and queue depth.

The Generation Number is used to distinguish cells sent under different bundle configurations. When bundle membership changes, the sender and receiver SHOULD complete or drain cells from the old generation before fully switching to the new generation, unless an implementation-specific recovery policy is used.

TOB Packet Format This section defines a prototype TOB cell format. TOB may be carried directly over an Ethernet service using a dedicated EtherType, or it may be carried inside an IP tunnel such as UDP or GRE. The exact encapsulation is deployment-specific.

TOB Encapsulation When TOB is carried over Ethernet, a TOB cell is encapsulated as follows: [ETH + TOB Header + TOB Payload + Integrity Check] When TOB is carried over UDP/IP, a TOB cell is encapsulated as follows: [ETH + IP + UDP + TOB Header + TOB Payload + Integrity Check] The UDP destination port or EtherType value is outside the scope of this prototype document and is expected to be assigned or configured according to deployment requirements.

TOB Header Fields TOB Header { Version (8), Header Length (8), Flags (16), Bundle ID (32), Generation Number (32), Member ID (16), Traffic Class (16), Frame Sequence Number (64), Fragment Sequence Number (64), Fragment Offset (32), Original Frame Length (32), Header CRC (32) }

Version (8 bits): TOB protocol version. The initial version defined in this document is 0x01.
Header Length (8 bits): Length of the TOB header in 4-octet units.
Flags (16 bits): Control flags. This document defines BEGIN, END, SINGLE, RETX, FEC, ACK, and NACK flags. Additional flags are reserved for future use.
Bundle ID (32 bits): Identifier of the logical bundle.
Generation Number (32 bits): Identifier of the active bundle configuration.
Member ID (16 bits): Identifier of the member link selected by the sender.
Traffic Class (16 bits): Traffic class or quality-of-service indication copied from the ingress packet or assigned by local policy.
Frame Sequence Number (64 bits): Sequence number of the original ingress packet.
Fragment Sequence Number (64 bits): Sequence number of the TOB cell.
Fragment Offset (32 bits): Offset of the payload carried in this TOB cell within the original ingress packet.
Original Frame Length (32 bits): Length of the original ingress packet before fragmentation.
Header CRC (32 bits): Integrity check for the TOB header.

Loss Recovery TOB deployments may operate over member links with different loss characteristics. This document defines three prototype operating modes:

Resequencing-only mode: The receiver only compensates for differential delay and reordering. Loss recovery is left to lower or upper layers.
Selective retransmission mode: The receiver sends NACK messages for missing cells. The sender retransmits missing cells on any healthy member link.
FEC-assisted mode: The sender periodically transmits parity cells for a block of data cells. The receiver uses the parity cells to recover from limited loss before requesting retransmission.

The configured loss-recovery mode needs to balance latency, bandwidth overhead, and head-of-line blocking. In all modes, the receiver MUST have a policy for declaring a missing packet unrecoverable. Otherwise, a single lost cell can block delivery of all subsequent packets.

IANA Considerations This document does not require any IANA actions. Future versions may request assignment of a UDP port, EtherType, or protocol registry values if a common encapsulation is standardized.

Contributors