Guidance to Avoid Carrying RPKI Validation States in Transitive BGP Path Attributes

The Resource Public Key Infrastructure (RPKI) allows for validation of received Border Gateway Protocol (BGP) routes. By means of a validation process, e.g., BGP Prefix Origin Validation (), routers attain locally significant validation states. In the past, some operators and vendors suggested to use BGP Communities ( and ) to annotate received routes with validation states local to a router. Some claim that the practice of signaling validation states could be useful, for example to Internal BGP (IBGP) speakers, in order to avoid each IBGP speaker having to perform their own route origin validation. However, such a practice does not only conflict with core concepts of RPKI, but also threatens the stability of networks. Annotating a route with a transitive attribute (based on validation states) means that BGP UPDATE messages have to be sent to every neighbor when the aforementioned validation states changes. This means that when, for example, Route Origin Authorizations (ROAs) are issued, or are revoked, or RPKI-To-Router (RTR) sessions are terminated, new BGP UPDATE messages will have to be send for all routes that were previously annotated with a BGP Community associated with a different validation state. Furthermore, given that BGP Communities are a transitive attribute, such a BGP UPDATE might end up propagating through large portions of the Default-Free Zone (DFZ). Hence, this document provides guidance to avoid carrying RPKI-derived validation states in transitive Border Gateway Protocol (BGP) Path Attributes (). Specifically, Operators SHOULD NOT associate BGP Prefix Origin Validation results with BGP routes using transitive BGP Communities (, , ). Furthermore, this document contains data on the measured current impact of BGP Communities used to signal validation states. Avoiding the use of BGP Communities to signal RPKI-derived validation states prevents BGP UPDATE messages from being flooded into the global Internet routing system.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

This document discusses signaling locally significant RPKI validation states to EBGP neighbors through transitive BGP attributes. This includes operator-specific BGP Communities to signal validation states, as well as any current or future standardized well-known BGP Communities denoting validation states. As BGP message churn is associated with internal signaling changes, it is RECOMMENDED that operators also consider the guidance () within their network, i.e., between their IBGP speakers. The guidance in this document applies to all current and future transitive BGP attributes that may be implicitly or explicitly used to signal validation states to neighbors. Similarly, this guidance also applies to non-ROA validation mechanics based on RPKI, e.g., Autonomous System Provider Authorization (ASPA) , BGPsec , and any other future validation mechanic built upon the RPKI.

This section outlines the risks of signaling RPKI-derived validation states using BGP Communities. While the current description is specific to BGP communities, the observations hold similar for all transitive attributes that may be added to BGP routes.

This section describes examples on how a large amount of RPKI ROA changes may occur in a short time, leading to generation of a large amounts of BGP UPDATE messages.

Large-Scale ROA issuance events should be a comparatively rare event for individual networks. However, several cases exist where issuance by individual operators or (malicious) coordinated issuance of ROAs by multiple operators may lead to a high route churn, triggering a continuous flow of BGP UPDATE messages caused by operators using transitive BGP attributes to signal validation states. Specifically:

When an operator issues new ROAs for their netblocks, possibly by issuing one ROA with a non-minimal maxLength () covering a large number of prefixes. This may also occur when incorrectly migrating to minimally covering ROAs (), i.e., when the previous ROA is first revoked (see ) and the new ROAs are only issued after this revocation has been propagated, e.g., due to an operational error or a bug in the issuance pipeline used by the operator.
When multiple Certification Authorities (CAs) coordinate to issue new ROAs at the same time.
When a CA has been unavailable or unable to publish for some time, but then publishes all updates at once, or - as unlikely as it is - if a key-rollover encounters issues.

Large-Scale ROA revocation should be a comparatively rare event for individual networks. However, several cases exist where revocations by individual operators or (malicious) coordinated revocation of ROAs by multiple operators may lead to a high route churn triggering a continuous flow of BGP Update messages caused by operators using transitive BGP attributes to signal validation states. Specifically:

When one large operator revokes all ROAs for their netblocks at once, for example, when migrating to minimally covering ROAs , or when revoking one ROA with a long maxLength covering a large number of prefixes.
When multiple CAs coordinate to revoke ROAs at the same time.
When a CA becomes unavailable or unable to publish for some time, e.g., due to the CA expiring (, , , ).

Similar to the issuance and revocation of ROAs, the validation pipeline of a Relying Party (RP) may encounter issues. Issues may occur on the cache side () or on the router side () with network connectivity issues having specific impact on either of the two. While, in general, implementations should not have bugs, operators should not make mistakes, and the network should be reliable, this is usually not the case in practice. Instead, the worst-case of sudden and unexpected, yet unintentional, loss of cached validated payloads is an event that, however unlikely in a specific system, may and will happen. Hence, systems should be resilient in case of unexpected issues, and should not exacerbate issues by inadvertently contributing to BGP UPDATE floods. Below, examples are provided of events for both categories that may lead to the validation state of routes in one or multiple routers of an operator changing from Valid to NotFound (). These lists are provided for illustrative purposes and not exhaustive.

The following events may impact a cache's ability to provide validated payload information to routers:

The RPKI-To-Router (RTR) service may have to temporarily be taken offline by the RP operator for maintenance. While operators should, in general, take care to provision sufficient redundancy, critical vulnerabilities may necessitate the immediate simultaneous shutdown of all RTR instances.
A cache may crash due to bugs when ingesting unexpected data from the RPKI, or run into performance issues due to insufficient available memory or limited I/O performance on the host. In the worst case, especially memory issues, can lead to a flapping cache, e.g., when the system runs out of memory after a few minutes of transmitting validated payloads to routers.
Validation state may seemingly lapse due to issues with time synchronization if, e.g., the clock of the cache diverts significantly, starting to consider CA's certificates invalid.
The cache may lose its network connectivity in general, or to specific CAs. While, in general, the cache should be able to serve from cache, an operator may have to shutdown the cache in such a case, to prevent dropping prefixes as invalid due to stale data.

Examples of encountered issues are:

An RTR client, especially when implemented as a dedicated daemon, may fail to start, or terminate when receiving unexpected data. Especially when this leads to a flapping client, e.g., due to a bug in the handling of incremental updates leading to a crash, while the initial retrieval is successful, this will lead to flapping between routes being Valid and NotFound.
A misconfiguration may impact a router's ability to communicate with the RTR service. For example, an RTR client may lose its credentials or may not receive updated credentials in time when these are changed, or the address of the RTR service changes and is not updated on the router in time.
An RTR client may lose network connectivity to the RTR service. While, in general, caches should prevent this from having immediate impact, an RTR client's behavior in case of a flapping network connection with frequent interruptions may lead to unexpected behavior and cache invalidation. Similarly, after cache expiry, routes will change from Valid to NotFound.
As an extension of the previous point, multiple operators might be using one central RTR service hosted by an external party, or depend on a similar cache, which becomes unavailable, e.g., due to maintenance or an outage. If local instances are not able to handle loss of this external service without changing validation states, i.e., do not serve from cache or the outage extends beyond the expiry of cached information, routes will change local validation states from Valid to NotFound. Naturally, the negative impact in such a case is significantly larger in comparison to each operator running their own cache.

The above non-exhaustive listing suggests that issues in general operations, CA operations, and RPKI cache implementations simply are unavoidable. Hence, Operators have to plan and design accordingly.

For each change in the validation state of a route, an Autonomous System (AS) in which the local routing policy sets a BGP Community based on the BGP Prefix Origin Validation result, routers would need to send BGP UPDATE messages for more than half of the global Internet routing table if the validation state changes to NotFound. The same, reversed case, would be true for every new ROA created by the address space holders, whereas a new BGP UPDATE would be generated, as the validation state would change to Valid. Furthermore, adding additional attributes to routes increases their size and memory consumption in the Routing Information Base of BGP routers. Given the continuous growth of the global routing table, in general, operators should be conservative regarding the additional information they add to routes.

The aforementioned scaling issues are not confined to singular UPDATE events. Instead, changes in validation state may lead to floods and/or cascades of BGP UPDATE messages throughout the Internet.

Flooding events are caused by an individual operator losing validation states. If that operator annotates validation states using BGP communities, the AS will send updates for all routes that changed from Valid to NotFound to its customers, as well as updates for routes received from customers to its providers. Following an RPKI service affecting outage, given that, at the time of writing, more than half the global Internet routing table nowadays is covered by RPKI ROAs , such convergence events represent a significant burden. See for an elaboration on this phenomenon.

For events that are not specific to one operator, e.g., a malicious withdrawal of a ROA, loss of a major CA, or an unexpected downtime of a major centralized RTR service, events can also cascade for ASes annotating validation states using BGP communities. Given that routers' view of the RPKI with RTR are only loosely consistent, BGP UPDATE messages may cascade, i.e., one event affecting validation states may actually trigger multiple subsequent BGP UPDATE floods. Assume, for example, that AS65536 is a customer of AS65537 (both annotating validation states with BGP Communities and using a 300 second RTR cycle), and a centralized RTR service fails. In the example, AS65536 has their routers updated from that cache a second before the service went down, while AS65537 was due for a refresh a second thereafter. This means that a second after the RTR service went down, AS65537 will trigger a BGP UPDATE flood down its cone. AS65536 will ingest and propagate these BGP UPDATE messages down its own cone as well. When, roughly 300 seconds later, AS65536 fails to disseminate validated payload information as well, the community set by AS65536 will again change for ROA covered routes, and it will again trigger a BGP UPDATE flood and propagate further on. Even if either or both of AS65536 and AS65537 use a cache after RTR expiry, the underlying issue would not change, assuming the RTR service downtime spans beyond the cache TTL. Assuming a 30 minute cache TTL, both ASes using a cache would only move the cascading event 30 minutes later. If only one of the two uses a cache, the two flood events get moved further apart. However, the overall issue of two independent floods due to one event remains.

In February 2024, a data-gathering initiative reported that between 8% and 10% of BGP UPDATE messages seen on the Routing Information Service (RIS), contained publicly-known communities from large ISPs signaling either NotFound or Valid BGP Origin Validation results. The study also demonstrated that the creation or removal of a ROA object triggered a chain of updates in a period of circa 1 hour following the change. Such a high percentage of unnecessary BGP UPDATE messages constitutes a considerable level of noise, impacting the capacity of the global routing system while generating load on router CPUs and occupying more RAM than necessary. Containing this information within the administrative boundaries of a single AS helps reduce the burden on the rest of the global routing platform, reducing workload and noise.

RTR has been developed to communicate validated payload information to routers. BGP Attributes are not signed, and provide no assurance against third parties adding them, apart from BGP Communities--ideally--being filtered at a network edge. So, even in IBGP scenarios, their benefit in comparison to using RTR on all BGP speakers is limited. For EBGP, given they are not signed, they provide even less information to other parties except introspection into an ASes internal validation mechanics. Crucially, they provide no actionable information for BGP neighbors. If an AS validates and enforces based on RPKI, Invalid routes should never be imported and, hence, never be sent to neighbors. Hence, the argument that exposing validation states through communities enables, for example, customers to filter RPKI Invalid routes is moot, as the only routes a customer should see are NotFound and Valid.

As outlined in , signaling validation states through transitive attributes represents risks for the stability of the global routing ecosystem. Not signaling validation states, hence, has tangible benefits, specifically:

Reduction of memory consumption on customer/peer facing PE routers (less BGP communities == less memory pressure).
No effect on the age of a BGP route when a ROA or ASPA is issued or revoked.
Avoids having to resend, e.g., more than 500,000 BGP routes towards BGP neighbors (for the own cone to peers and providers, for the full table towards customers) if the RPKI cache crashes and RTR sessions are terminated, or if flaps in validation are caused by other events.

Guidance Operators SHOULD NOT signal RPKI-derived validation states using transitive BGP attributes. The document acknowledges that specific operational requirements, such as a BGP implementation used by an operator still being dependent on annotating RPKI-derived validation states using BGP attributes, may necessitate the use of BGP path attributes to signal validation states between BGP speakers. If this is the case, the dependent operator SHOULD ensure that these attributes are removed before announcing NLRI to EBGP neighbors. Depending on the supported functionality of the BGP implementations in use in a given AS, removal of the aforementioned attributes may not be consistently possible across the AS, leading to the conditions this document is intended to discuss.

This document provides guidance to prevent using BGP UPDATE messages to carry RPKI Validate State. Following this guidance allows operators to enhance the stability of local routing tables and the global Internet routing system, in particular.

BGP is not guaranteed to converge, and the view on the RPKI within an individual administrative domain is only loosely consistent. External validation states annotated in a received NLRI may either depend on a different view on the RPKI than the one in the local administrative domain, or the NLRI may be several hours old itself. Hence, any validation states contained in a received route announcement only have significance in the sender's own local scope. Additionally, the use of transitive attributes to signal RPKI-derived validation states may enable attackers to cause notable route churn. This can be accomplished by an attacker issuing and withdrawing, e.g., ROAs for their prefixes, or by the attacker continuously altering transitive attributes used to signal RPKI-derived validation states for NLRI they re-advertise. The latter is possible as NLRI carry no information allowing an ingesting party to validate the integrity of transitive BGP attributes. DFZ routers may not be equipped to handle BGP message churn in all directions at global scale, especially if aforementioned routing churn cascades, persists, or repeats periodically. Adhering to the helps prevent such churn.

This document does not make any request to IANA.

The authors would like to thank , , , , , , , , and for their helpful review of this document.