Elastic Bandwidth-aware Routing Framework

1.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

1.2. Terminology

Load-balancing Alternate path:

Alternate TE routing paths used for traffic load balancing when the primary path is congested.¶

Congestion Threshold:

A configured value, when the bandwidth utilization of a local link exceeds this value, then the traffic distribution is initiated among the primary path and load-balancing alternate paths.¶

Restore Threshold:

A configured value, when the bandwidth utilization of a local link falls below this value, then the traffic distribution among the primary path and load-balancing alternate paths is canceled for the local link, and the primary path forwarding is restored.¶

Local Load Balancing Node (LLBN):

An EBR-enabled network node, which performs traffic load balancing upon detecting congestion on one of its local links.¶

2.1. Congestion Caused by Link Degradation

Existing IGP protocols calculate the shortest paths for traffic forwarding, it usually does not consider the actual link bandwidth and the traffic rates on the links. As a consequence, congestion may occur when network failure results in capacity reduction on a network link. An example network topology is shown in Figure 1.¶

+-----+     cost=1       +-----+     cost=2        +-----+
|  A  |==================|  B  |===================|  C  |
+-----+                  +-----+                   +-----+
   ||               cost=2  ||                       ||
   ||                    +-----+                     ||
   ++====================|  D  |=====================++
         cost=3          +-----+           cost=2

There are four nodes A, B, C, and D in the network, between each pair of the adjacent nodes, two fibers are bound as one bundle link. The shortest path from A to C is A->B->C. However, one of the fibers between A and B is broken due to unexpected events, but IGP protocol does not perceive this change because the link state connectivity does not change. Then the shortest path from A to C is still A->B->C, and the traffic from A to C is still forwarded along A->B->C. As a consequence, congestion may occur between node A and B, since the link bandwidth has been reduced by half. However, there are alternate paths from A to C (A->D->C and A->D->B->C), which provide plenty of available bandwidth. A mechanism is needed to distribute the traffic among the primary path and the alternate paths to accommodate the traffic during link failure and avoid congestion on the shortest path.¶

2.2. Congestion Caused by Burst Traffic

Another example is the congestion caused by bursts traffic. Considering the same network topology as described in Figure 1. The shortest path from A to C is A->B->C. Generally, the bandwidth of the link from A to B is capable of carrying the traffic from A to B. However, there may be some burst traffic from A to C due to some unexpected events (such as a concert or a football match), and the traffic exceeds the available bandwidth of link A-B. As a consequence, congestion may occur on the link from A to B. However, there are alternate paths from A to C (A->D->C and A->D->B->C), which provide plenty of available bandwidth to accommodate the burst traffic and avoid congestion. A mechanism is needed to distribute the traffic among the primary path and the alternate paths to accommodate the traffic burst and avoid congestion on the shortest path.¶

4.1. Monitoring and Advertisement of Link Bandwidth Information

LLBN needs to continuously monitor the bandwidth utilization and available bandwidth of its outbound links. The bandwidth utilization is the key information for determining whether a link is congested. The determination of a link congestion depends on a configurable Congestion Threshold (such as 80%, 90%, etc.), which can be configured by network operator according to the network conditions.¶

The mechanism used to obtain the bandwidth utilization information of a local link is out of scope of this document.¶

Upon local link congestion, distributing traffic blindly to alternate paths may lead to new congestion occurring on other links. Therefore, it is crucial to obtain the information about the available bandwidth and bandwidth utilization of each link in the network. With such information, the headend can distribute traffic based on the available bandwidth and bandwidth utilization of the links on each path, thereby avoiding the occurrence of new congestion.¶

The advertisement of bandwidth information can be achieved through IGP advertisement. Existing TE metric extensions to IGPs already allow a node to advertise the bandwidth related information (e.g. maximum link bandwidth, available bandwidth, and utilized bandwidth, etc.) of its links (IS-IS[RFC8570], OSPF[RFC7471]). The considerations for bandwidth information advertisement is introduced in Section 5.3.¶

4.2. Load-balancing Alternate Path Calculation

The load-balancing alternate paths in EBR are used for load balancing when the primary path is congested. Although the algorithm used for load-balancing alternate calculation is implementation-specific, it should meet the following requirements:¶

1. The load-balancing alternate paths should be calculated by LLBN in advance to allow quick triggering of load balancing when local link congestion is detected.¶

2. The calculation should make that the load-balancing alternate paths and the primary path are disjoint as much as possible.¶

3. The bandwidth utilization of each link may be considered during the calculation, to avoid using links with high bandwidth utilization for congestion traffic offloading.¶

4. Depending on service flow needs, other metrics and constraints may be considered during calculation.¶

The number of alternate paths depends on the configuration and network topology. The algorithm for calculating load-balancing alternate paths is out of scope of this document.¶

4.3. Traffic Distribution Upon Congestion

Once a LLBN detects the bandwidth utilization of one of its outbound links exceeds the Congestion Threshold, the load balancing mechanism will be triggered to distribute traffic among the primary path and load-balancing alternate paths.¶

UCMP is recommended to distribute flows among the paths, the weight of each path is determined according to the available bandwidth and bandwidth utilization of the path. The available bandwidth of a path is the available bandwidth of the link with the smallest available bandwidth on the entire path. The bandwidth utilization of each path is the utilization of the link with highest utilization rate on the entire path.¶

Once the weight of each path is determined, it will not change unless new congestions are detected on the links of load-balancing alternate paths.¶

Traffic distributed to load-balancing alternate paths will be forwarded based on SR-TE mechanism, which ensures the traffic be forwarded without loops.¶

• For SR-MPLS network, packets will be encapsulated with an ordered list of MPLS labels which represent the load-balancing alternate path.¶

• For SRv6 network, packets will be encapsulated with an outer IPv6 header, together with an SRH which contains the SID list representing the load-balancing alternate path.¶

Policies may be used for determining which groups of flows (e.g., according to traffic class, IP prefixes, etc.) should be migrated from the primary path to the load-balancing alternate paths.¶

4.4. Traffic Fallback

Traffic fallback means a LLBN migrate the traffic on load-balancing alternate paths back to the primary path. This process restores the network to the original state. There are several methods for triggering traffic fallback:¶

1. Threshold-based traffic fallback: When a LLBN detects that the bandwidth utilization of a congested link falls below the Restore Threshold, then the traffic fallback is triggered.¶

2. Dynamic traffic fallback: In this method, there is no static threshold for traffic fallback, the fallback is triggered dynamically upon the local link of the LLBN can bear all the traffic without congestion.¶

In both of the above methods, the traffic fallback should be triggered only when the conditions have been met for a configurable period of time.¶

Method 2 is recommended as it can avoid oscillations in traffic distribution and traffic fallback.¶

5.1. Oscillation Suppression

Micro-burst traffic or flapping bundle member link may cause frequent change of the link utilization and congestion state, and may result in oscillation in traffic distribution. The following oscillation suppression measures should be taken:¶

• The determination of congestion and restoration should consider the statistical characteristics of bandwidth utilization over a period of time, rather than only bandwidth utilization in a short interval.¶

• If threshold-based traffic fallback is used, then the Congestion Threshold should be sufficiently far from the Restore Threshold to avoid oscillation in the link's congestion status caused by small traffic fluctuation.¶

5.2. Alleviation of Possible New Congestions

The available bandwidth and utilization of load-balancing alternate paths are considered in traffic distribution, which effectively reduces the possibility of secondary congestion on the alternate paths. While in some cases it is possible that distributing traffic from primary path to load-balancing alternate paths may cause new congestion for the following reasons:¶

• The available bandwidth of an alternate path does not match the rate of assigned flows. Although UCMP is used in the distribution of flows to alternate paths, due to different size of flows, a big flow may cause new congestion on some links of an alternate path.¶

• Simultaneous traffic distribution initiated by different nodes. Since network nodes which support EBR act independently, simultaneous traffic distribution is possible, which may cause the total diverted traffic rate exceeds the available bandwidth of some links of an alternative path.¶

There are two mechanisms to alleviate the new congestions.¶

• When a LLBN which initiated the traffic distribution perceives that the bandwidth utilization of an in-use load-balancing alternate path exceeds the Congestion Threshold, it can adjust the traffic distribution weight on different alternate paths to reduce the flows on the congested paths to relieve the congestion.¶

• The LLBN which is adjacent to the newly congested link can initiate traffic distribution and divert a portion of traffic to its load-balancing alternate paths to alleviate the congestion.¶

5.3. Considerations on Bandwidth Information Advertisement

5.4. Compatibility

EBR can be deployed incrementally in the network. Network nodes which support EBR can calculate the load-balancing alternate paths and initiate UCMP load-balancing upon local link congestion. Network nodes which do not support EBR do not calculate the load-balancing alternate paths, and will not initiate UCMP load-balancing, while they can forward the offloaded traffic according to the SR SID list in the packets.¶

Author's note: more operational considerations will be added in future.¶

8.1. Normative References

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/rfc/rfc2119>.

[RFC8174]

Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

8.2. Informative References

[RFC8570]

Ginsberg, L., Ed., Previdi, S., Ed., Giacalone, S., Ward, D., Drake, J., and Q. Wu, "IS-IS Traffic Engineering (TE) Metric Extensions", RFC 8570, DOI 10.17487/RFC8570, March 2019, <https://www.rfc-editor.org/rfc/rfc8570>.

[RFC7471]

Giacalone, S., Ward, D., Drake, J., Atlas, A., and S. Previdi, "OSPF Traffic Engineering (TE) Metric Extensions", RFC 7471, DOI 10.17487/RFC7471, March 2015, <https://www.rfc-editor.org/rfc/rfc7471>.

[RFC8919]

Ginsberg, L., Psenak, P., Previdi, S., Henderickx, W., and J. Drake, "IS-IS Application-Specific Link Attributes", RFC 8919, DOI 10.17487/RFC8919, October 2020, <https://www.rfc-editor.org/rfc/rfc8919>.