Happy Eyeballs Version 3: Better Connectivity Using Concurrency

Happy Eyeballs Version 3: Better Connectivity Using Concurrency Apple Inc

tpauly@apple.com

Google LLC

dschinazi.ietf@gmail.com

Google LLC

nidhijaju@google.com

Google LLC

bashi@google.com

WIT HAPPY Working Group ipv6 ipv4 racing tcp quic svcb ech Many communication protocols operating over the modern Internet use hostnames. These often resolve to multiple IP addresses, each of which may have different performance and connectivity characteristics. Since specific addresses or address families (IPv4 or IPv6) may be blocked, broken, or sub-optimal on a network, clients that attempt multiple connections in parallel have a chance of establishing a connection more quickly. This document defines the algorithm for "Happy Eyeballs", a technique for reducing user-visible delays on dual-stack hosts. This document updates the algorithm in RFC 8305. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the HAPPY Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction Many communication protocols operating over the modern Internet use hostnames. These often resolve to multiple IP addresses, each of which may have different performance and connectivity characteristics. Since specific addresses or address families (IPv4 or IPv6) may be blocked, broken, or sub-optimal on a network, clients that attempt multiple connections in parallel have a chance of establishing a connection more quickly. This document defines the algorithm for "Happy Eyeballs", a technique for reducing user-visible delays on dual-stack hosts. This definition updates the description in , which itself obsoleted . The Happy Eyeballs algorithm of racing connections to resolved endpoints has several stages to avoid delays to the user whenever possible, while respecting client priorities, such as preferring the use of IPv6 or the availability of protocols like HTTP/3 and TLS Encrypted Client Hello . This document discusses how to initiate DNS queries when starting a connection, how to sort the list of destination addresses received from DNS answers, and how to race the connection attempts. The major difference between the algorithm defined in this document and is the addition of support for SVCB / HTTPS resource records . SVCB records provide alternative endpoints and information about application protocol support, Encrypted Client Hello keys, address hints, and other relevant details about the services being accessed. Discovering protocol support during resolution, such as for HTTP/3 over QUIC , allows upgrading between protocols on the current connection attempts, instead of waiting for subsequent attempts to use information from other discovery mechanisms such as HTTP Alternative Services . These records can be queried along with A and AAAA records, and the updated algorithm defines how to handle SVCB responses to improve connection establishment.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Overview This document defines a method of connection establishment, named the "Happy Eyeballs Connection Setup". This approach has several distinct phases:

Asynchronous resolution of a hostname into destination endpoints ()
Sorting of the resolved destination endpoints ()
Initiation of asynchronous connection attempts ()
Successful establishment of one connection and cancellation of other attempts ()

Note that this document assumes that the preference policy for the destination address family favors IPv6 over IPv4. IPv6 has many desirable properties designed to be improvements over IPv4 . This document also assumes that the preference policy favors QUIC over TCP. QUIC only requires one packet to establish a secure connection, making it quicker compared to TCP . If the host is configured to have different preferences, the recommendations in this document can be easily adapted.

Hostname Resolution When a client is trying to establish a connection to a named host, it needs to determine which destination endpoint it can use to establish a connection. The client resolves the endpoints by sending DNS queries and collecting the answers. This section describes how a client initiates DNS queries and asynchronously handles the answers.

Sending DNS Queries Clients first need to determine which DNS resource records they will include in queries for a named host. This decision is based on if client has "connectivity" using IPv4 and IPv6. In this case, "connectivity" for an address family is defined as having at least one local address of the family from which to send packets, and at least one non-link local route for the address family. When a client has both IPv4 and IPv6 connectivity, it needs to send out queries for both AAAA and A records. On a network with only IPv4 connectivity, it will send a query for A records. On a network with only IPv6 connectivity, the client will either send out queries for both AAAA and A records, or only a query for AAAA records, depending on the network configuration. See for more discussion of handling IPv6-mostly and IPv6-only networks. In addition to requesting AAAA and A records, depending on which application is establishing the connection, clients can request either SVCB or HTTPS records . For applications using HTTP or HTTPS (including applications using WebSockets), the client SHOULD send a query for HTTPS records. All of the DNS queries SHOULD be made as soon after one another as possible. The order in which the queries are sent SHOULD be as follows (omitting any query that doesn't apply based on the logic described above):

SVCB or HTTPS query
AAAA query
A query

Handling DNS Answers Asynchronously Once the client receives sufficient answers to its DNS queries, it can move onto the phases of sorting endpoints () and establishing connections (). Implementations SHOULD NOT wait for all answers to return before starting the next steps of connection establishment. If one query fails or takes significantly longer to return, waiting for those answers can significantly delay connection establishment that could otherwise proceed with already received answers. Therefore, the client SHOULD treat DNS resolution as asynchronous, processing different record types independently. Note that if the platform does not offer an asynchronous DNS API, this behavior can be simulated by making separate synchronous queries for each record type in parallel. The client moves onto sorting endpoints and establishing connections once one of the following condition sets is met: Either:

Some positive (non-empty) address answers have been received AND
A postive (non-empty) or negative (empty) answer has been received for the preferred address family that was queried AND
SVCB/HTTPS service information has been received (or has received a negative response)

Or:

Some positive (non-empty) address answers have been received AND
A resolution time delay has passed after which other answers have not been received

Positive answers can be IP addresses received either from AAAA or A records, or IP address hints received directly in SVCB/HTTPS records. Negative answers are exclusively responses to AAAA or A records that contain no IP addresses (with or without an error like NXDOMAIN). If all answers come back with negative answers, the connection establishment will fail or need to wait until other answers are received. On networks that have both default routes for IPv6 and IPv4, IPv6 is assumed to be the preferred address family. If only one of IPv6 or IPv4 has a default route, that address family should be considered the preferred address family for progressing the algorithm. The resolution time delay is a short time that provides a chance to receive preferred addresses (via AAAA records) along with service information (via SVCB/HTTPS records). This accounts for the case where the AAAA or SVCB/HTTPS records follow the A records by a few milliseconds. This delay is referred to as the "Resolution Delay". The RECOMMENDED value for the Resolution Delay is 50 milliseconds.

Resolving SVCB/HTTPS Aliases and Targets SVCB and HTTPS records describe information for network services. Individual records are either AliasMode or ServiceMode records, where AliasMode requires another SVCB/HTTPS query for the alias name. ServiceMode records either are associated with the original name being queried, in which case their TargetName is ".", or are associated with another service name (see ). The algorithm in this document does not consider service information to be received until ServiceMode records are available. ServiceMode records can contain IP address hints via ipv6hint and ipv4hint parameters. When these are received, they SHOULD be considered as positive non-empty answers for the purpose of the algorithm when A and AAAA records corresponding to the TargetName are not available yet. Note that clients are still required to issue A and AAAA queries for those TargetNames if they haven't yet received those records. When those records are received, they replace the hints and update the available set of responses as new answers (see ).

Handling New Answers If new records arrive while connection attempts are in progress, but before any connection has been established, then any newly received endpoints are incorporated into the list of available candidate endpoints (see ) and the process of connection attempts will continue with the new endpoints added, until one connection is established.

Handling Expired Answers Previously received DNS answers might expire (exceed their TTL) while connection attempts are still in progress. A Happy Eyeballs client will not cancel those ongoing attempts when the answers expire. However, the client MAY trigger new DNS queries to fetch non-expired answers after the original records have expired, in the case in which the original answers have not yet led to a successful connection. Addition of any new answers is handled as described in and ; that is, ongoing connection attempts will not be cancelled, but any endpoint for which a connection attempt had not yet been started that is no longer in the resolved endpoint list will be removed. Additionally, the fact that Happy Eyeballs enables clients to handle updates to the resolved endpoint list asynchronously enables the use of expired answers "optimistically" while waiting for non-expired answers. This approach is called "optimistic DNS", and is described in .

Handling Multiple DNS Server Addresses If multiple DNS server addresses are configured for the current network, the client may have the option of sending its DNS queries over IPv4 or IPv6. In keeping with the Happy Eyeballs approach, queries SHOULD be sent over IPv6 first (note that this is not referring to the sending of AAAA or A queries, but rather the address of the DNS server itself and IP version used to transport DNS messages). If DNS queries sent to the IPv6 address do not receive responses, that address may be marked as penalized and queries can be sent to other DNS server addresses. As native IPv6 deployments become more prevalent and IPv4 addresses are exhausted, it is expected that IPv6 connectivity will have preferential treatment within networks. If a DNS server is configured to be accessible over IPv6, IPv6 should be assumed to be the preferred address family. Client systems SHOULD NOT have an explicit limit to the number of DNS servers that can be configured, either manually or by the network. If such a limit is required by hardware limitations, the client SHOULD use at least one address from each address family from the available list.

Grouping and Sorting Endpoints Before attempting to connect to any of the resolved destination endpoints, the client defines the order in which to start the attempts. Once the order has been defined, the client can use a simple algorithm for racing each option after a short delay (see ). It is important that the ordered list involve all endpoints from both address families and all protocols that have been received by this point, as this allows the client to get the racing effect of Happy Eyeballs for the entire list, not just the first IPv4 and first IPv6 addresses. The client performs three levels of grouping and sorting of endpoints based on the DNS answers received. Each subsequent level of sorting only changes orders and preferences within the previously defined groups.

Grouping and sorting by application protocol and security requirements ()
Grouping and sorting by service priorities ()
Sorting by destination endpoint preferences ()

Grouping By Application Protocols and Security Requirements Clients first group based on which application protocols the destination endpoints support and which security features those endpoints offer. These are based on information from SVCB/HTTPS records about application-layer protocols ("alpn" values) and other parameters like TLS Encrypted Client Hello configuration ("ech" values, see ). For cases where the answers do not include any SVCB/HTTPS information, or if all of the answers are associated with the same SVCB/HTTPS record, this step is trivial: all answers belong to one group, and the client assumes they support the same protocols and security properties. However, the client is aware of different sets of destination endpoints that advertise different capabilities when it receives multiple distinct SVCB/HTTPS records. The client SHOULD separate these endpoints into different groups, such that all endpoints in a group share the same application protocols and relevant security properties. The specific parameters that are relevant to the client depend on the client implementation and application. Note that some destination endpoints might need to be added to multiple groups at this stage. For example, consider the following HTTPS records: In this case, 2001:db8::2 can be used with HTTP/3 and HTTP/2, but 2001:db8::4 can only be used with HTTP/2. If the client creates a grouping for HTTP/3-capable endpoints and HTTP/2-capable endpoints, 2001:db8::2 would exist in both groups (assuming that all other security properties are the same). Connection racing as described in applies to different destination endpoint options within one of these groups. The logic for prioritizing and falling back between groups of endpoints with different security properties and protocol properties is implementation-defined.

When to Apply Application Preferences Whether or not specific application protocols or security features are grouped separately is a client application decision. Clients SHOULD avoid grouping and sorting separately in cases where their use of an application protocol or feature is non-critical. For example, an HTTP client loading a simple webpage may not see a large difference between using HTTP/3 or HTTP/2, and thus can group the ALPNs together to respect service-determined priorities where HTTP/3 might be prioritized behind HTTP/2. However, another client might see significant performance improvements by using HTTP/3's ability to send unreliable frames for its application use-case and will group HTTP/3 before HTTP/2. Similarly, a particular application might require or strongly prefer the use of TLS ECH for privacy-sensitive traffic, while others might support ECH opportunistically. recommends against SVCB record sets that contain some answers that include ECH configuration and some that don't, but notes that such cases are possible. It is possible that services only include ECH configurations on SVCB answers that are prioritized behind others that don't include ECH configurations; for example, this might be used as an experimenation or roll-out strategy. Due to such cases, clients ought to not arbitrarily group ECH-containing answers and sort them first if they won't use the ECH information, or if the connection would not benefit from the use of ECH. However, for cases where there is a reason for an application preference for ECH, the client MAY group and prioritize those answers separately. Even though this might conflict with the published service record priorities, any answers published by the service are eligible to be used by clients, and clients can choose to use them.

Grouping By Service Priority The next step of grouping and sorting is to group across different services (as defined by SVCB/HTTPS records), and sort these groups by priority. This step allows server-published priorities to be reflected in the client connection establishment algorithm. SVCB records indicate a priority for each ServiceMode response. This priority applies to any IPv4 or IPv6 address hints in the record itself, as well as any addresses received on A or AAAA queries for the name in the ServiceMode record. The priority in a SVCB ServiceMode record is always greater than 0. SVCB answers with the lowest numerical value (such as 1) are sorted first, and answers with higher numerical values are sorted later. Note that a SVCB record with the TargetName "." applies to the owner name in the record, and the priority of that SVCB record applies to any A or AAAA records for the same owner name. These answers are sorted according to that SVCB record's priority. All endpoints received from a particular SVCB service (within a group as defined in ), either by an associated AAAA or A record or address hints, SHOULD be separated into a group by the client. These service-based groups SHOULD then be sorted using the service priority. For cases where the answers do not include any SVCB/HTTPS information, or if all of the answers are associated with the same SVCB/HTTPS record, this step is trivial: all answers belong to one group that has the same priority. When there are multiple services, and thus multiple groups, with the same priority, the client SHOULD shuffle these groups randomly. If there are some SVCB/HTTPS services received, but there are AAAA or A records that do not have an associated service (for example, if no SVCB/HTTPS record is received for the original name using the "." TargetName), the unassociated addresses SHOULD be put in a group of endpoints that is prioritized at the end of the list.

Sorting Destination Endpoints Within Groups Within each group of endpoints, after grouping based on the logic in and , the client sorts the endpoints based on preference and historical data. First, the client MUST sort the endpoints using Destination Address Selection (). If the client is stateful and has a history of expected round-trip times (RTTs) for the routes to access each endpoint's IP address, it SHOULD add a Destination Address Selection rule between rules 8 and 9 that prefers addresses with lower RTTs. If the client keeps track of which addresses it used in the past, it SHOULD add another Destination Address Selection rule between the RTT rule and rule 9, which prefers used addresses over unused ones. This helps servers that use the client's IP address during authentication, as is the case for TCP Fast Open and some Hypertext Transport Protocol (HTTP) cookies. This historical data MUST be partitioned using the same boundaries used for privacy-sensitive information specific to that endpoint, and MUST NOT be used across different Provisioning Domains (PvDs) . The data SHOULD be flushed whenever a device changes the PvD to which it is attached. However, if a client can reliably identify a previously attached PvD, it MAY retain and resume using historical data associated with that PvD upon reconnection. Clients that use historical data MUST ensure that clients with different historical data will eventually converge toward the same behaviors. For example, clients can periodically ignore historical data to ensure that fresh addresses are attempted. Next, the client SHOULD modify the ordered list to interleave address families. Whichever address family is first in the list should be followed by an endpoint of the other address family. For example, if the first endpoint in the sorted list has an IPv6 address, then the first IPv4 endpoint should be moved up in the list to be second in the list. An implementation MAY choose to favor one address family more by allowing multiple addresses of that family to be attempted before trying the next. The number of contiguous endpoints of the first address family of properties will be referred to as the "Preferred Address Family Count" and can be a configurable value. This avoids waiting through a long list of endpoints for a given address family if connectivity over that address family is impaired. Note that the endpoint selection described in this section only applies to destination addresses; Source Address Selection () is performed once per destination address and is out of scope of this document.

Connection Attempts Once the list of endpoints received up to this point has been constructed, the client will attempt to make connections. In order to avoid unreasonable network load, connection attempts SHOULD NOT be made simultaneously. Instead, one connection attempt to a single endpoint is started first, followed by the others, one at a time. Starting a new connection attempt does not affect previous attempts, as multiple connection attempts may occur in parallel. Once one of the connection attempts succeeds (), all other connections attempts that have not yet succeeded SHOULD be canceled. Any endpoint that was not yet attempted as a connection SHOULD be ignored. At that time, any asynchronous DNS queries MAY be canceled as new endpoints will not be used for this connection. However, the DNS client resolver SHOULD still process DNS replies from the network for a short period of time (recommended to be 1 second), as they will populate the DNS cache and can be used for subsequent connections. If grouping endpoints by application or security requirements () produced multiple groups, the application SHOULD start with connection attempts to the most preferred option. The policy for attempting any endpoints outside of the most preferred group is up to the client implementation and out of scope for this document. If grouping endpoints by service () produced multiple groups, all of the endpoints of the first group SHOULD be started before starting attempts using the next group. Attempts across service groups SHOULD be allowed to continue in parallel; in effect, the groups are flattened into a single list. The intent of these groups is to strictly prioritize the most preferred service. Therefore, if a higher priority group has many endpoints, there is a potential for long delays before falling back to lower priority groups. A simple implementation can have a fixed delay for how long to wait before starting the next connection attempt. This delay is referred to as the "Connection Attempt Delay". One recommended value for a default delay is 250 milliseconds. A more nuanced implementation's delay should correspond to the time when the previous attempt is retrying its handshake (such as sending a second TCP SYN or a second QUIC Initial), based on the retransmission timer (, ). If the client has historical RTT data gathered from other connections to the same host or prefix, it can use this information to influence its delay. Note that this algorithm should only try to approximate the time of the first handshake packet retransmission, and not any further retransmissions that may be influenced by exponential timer back off. Clients that support session resumption mechanisms (such as TLS session tickets or QUIC 0-RTT ) SHOULD use the same Connection Attempt Delay for resumed connections as for new connections. The Connection Attempt Delay MUST have a lower bound, especially if it is computed using historical data. More specifically, a subsequent connection MUST NOT be started within 10 milliseconds of the previous attempt. The recommended minimum value is 100 milliseconds, which is referred to as the "Minimum Connection Attempt Delay". This minimum value is required to avoid congestion collapse in the presence of high packet-loss rates. The Connection Attempt Delay SHOULD have an upper bound, referred to as the "Maximum Connection Attempt Delay". The current recommended value is 2 seconds. The Connection Attempt Delay is used to set a timer, referred to as the "Next Connection Attempt Timer". Whenever this timer fires and a connection has not been successfully established, the next connection attempt starts, and the timer either is reset to a new delay value or, in the case of the end of the list being reached, is cancelled. Note that the delay value can be different for each connection attempt (depending on the protocol being used and the estimated RTT).

Determining successful connection establishment The determination of when a connection attempt has successfully completed (and other attempts can be cancelled) ultimately depends on the client application's interpretation of the connection state being ready to use. This will generally include at least the transport-level handshake with the remote endpoint (such as the TCP or QUIC handshake), but can involve other higher-level handshakes or state checks as well. Client connections that use TCP only (without TLS or another protocol on top, such as for unencrypted HTTP connections) will determine successful establishment based on completing the TCP handshake only. When TLS is used on top of of TCP (such as for encrypted HTTP connections), clients SHOULD wait for the TLS handshake to successfully complete before cancelling other connection attempts. This is particularly useful for networks in which a TCP-terminating proxy might be causing TCP handshakes to succeed quickly, even though end-to-end connectivity with the TLS-terminating server will fail. QUIC connections inherently include a secure handshake in their main handshakes, and thus usually only need to wait for a single handshake to complete. Beyond TCP, TLS, and/or QUIC handshakes, clients may also wait for other requirements to be met before determining that the connection establishment was successful. For example, clients generally validate that the server's certificate provided via TLS is trusted, and that operation can be asynchronous. In cases where the connection establishment determination goes beyond the initial transport handshake, the Next Connection Attempt Timer ought to be adjusted after the initial transport handshake is completed. When the connection establishment makes progress, but has not completed, the timer SHOULD be extended to a new value that represents an estimated time for the full connection establishment to complete. For example, consider a case where connection establishment involves both a TCP handshake and a TLS handshake. If the timer is initially set to be roughly at the time when a TCP SYN packet would be retransmitted, and the TCP handshake completes before the timer fires, the timer should be adjusted to allow for the time in which the TLS handshake could complete. While transport layer handshakes generally do not have restrictions on attempts to establish a connection, some cryptographic handshakes may be dependent on SVCB ServiceMode records and could impose limitations on establishing a connection. For instance, ECH-capable clients may become SVCB-reliant clients () when SVCB records contain the "ech" SvcParamKey . If the client is either an SVCB-reliant client or a SVCB-optional client that might switch to SVCB-reliant connection establishment during the process, the client MUST wait for SVCB records before proceeding with the cryptographic handshake.

Handling Application Layer Protocol Negotiation (ALPN) The alpn and no-default-alpn SvcParamKeys in SVCB records indicate the "SVCB ALPN set," which specifies the underlying transport protocols supported by the associated service endpoint. When the client requests SVCB records, it SHOULD perform the procedure specified in to determine the underlying transport protocols that both the client and the service endpoint support. The client SHOULD NOT attempt to make a connection to a service endpoint whose SVCB ALPN set does not contain any protocols that the client supports. For example, suppose the client is an HTTP client that only supports TCP-based versions such as HTTP/1.1 and HTTP/2, and it receives the following HTTPS record: In this case, attempting a connection to 2001:db8::2 or any other address resolved for svc1.example.com would be incorrect because the record indicates that svc1.example.com only supports HTTP/3, based on the ALPN value of "h3". If the client is an HTTP client that supports both Alt-Svc and SVCB (HTTPS) records, the client SHOULD ensure that connection attempts are consistent with both the Alt-Svc parameters and the SVCB ALPN set, as specified in .

Dropping or Pending Connection Attempts Some situations related to handling SVCB responses can require connection attempts to be dropped, or pended until SVCB responses return. describes client behavior for handling resolution failures when responses are "cryptographically protected" using DNSSEC or encrypted DNS (, , or , for example). If SVCB resolution fails when using cryptographic protection, clients SHOULD abandon connection attempts altogether to avoid downgrade attacks. Use of cryptographic protection in DNS can influence other parts of Happy Eyeballs connection establishment, as well. Situations in which DNS is not protected allow for any records to be blocked or modified, so security properties derived from SVCB records are opportunistic only. However, when DNS is cryptographically protected, clients can be stricter about relying on the properties from SVCB records. explains that clients "MUST NOT transmit any information that could be altered by the SVCB response until it arrives", and specifically mentions properties that affect the TLS ClientHello. This restriction specifically applies when a client's behavior will be altered by the SVCB response, which depends both on the client implementation's ability to support a particular feature, as well as the client implementation's willingness to rely on the SVCB response to enable a particular feature. Based on this, clients in some scenarios MUST pend starting a TLS handshake (either after TCP or as part of QUIC) until SVCB responses have been received, even after the "Resolution Delay" defined in has been reached. Specifically, clients MUST pend starting handshakes if all of the following are true:

DNS responses are cryptographically protected with DNSSEC or encrypted DNS. Note that, if unencrypted and unsigned DNS is used, SVCB information is opportunistic; clients MAY wait for SVCB responses but do not need to.
The client implementation supports parsing and using a particular security-related SVCB parameter, such as the "ech" SvcParamKey . (In contrast, implementations that do not support actively using ECH do not need to wait for SVCB resolution if that is the only reason to do so).
The client relies on the presence of the particular SVCB-related parameter to enable the relevant protocol feature. For example, if a connection attempt would normally be using cleartext HTTP unless an HTTPS DNS record would cause the client to upgrade, the client needs to wait for the record; however, if the client already would be using HTTP over TLS, then it is not relying on that signal from SVCB. As another example, some SVCB properties can affect the TLS ClientHello in ways that optimize performance (like tls-supported-groups ) but only aim to save round trips. The other TLS groups can be discovered through the TLS handshake itself, instead of SVCB, and thus do not require waiting for SVCB responses.

DNS Answer Changes During Happy Eyeballs Connection Setup If, during the course of connection establishment, the DNS answers change by either adding resolved endpoints (for example due to DNS push notifications ) or removing previously resolved endpoints (for example, due to expiry of the TTL on that DNS record), the client should react based on its current progress. Additionally, endpoints from SVCB IP hints SHOULD be removed from the list once A and AAAA records are received for the corresponding name, if the endpoints from the hints are absent from the received records . If an endpoint is removed from the list that already had a connection attempt started, the connection attempt SHOULD NOT be canceled, but rather be allowed to continue. If the removed endpoint had not yet had a connection attempt started, it SHOULD be removed from the list of endpoints to try. If an endpoint is added to the list, its position SHOULD be determined by applying the sorting rules (see ) to the complete list of endpoints, including those previously received. This ensures that sorting rules, such as address family interleaving, are maintained correctly regardless of when endpoints arrive. For example, consider a connection attempt in which only IPv6 addresses are available initially, and an attempt to one IPv6 address endpoint is already in progress. Then, when IPv4 addresses are later received, an IPv4 address endpoint should be placed next in the list of endpoints to attempt (to account for interleaving address families) ahead of any remaining IPv6 address endpoints, as if it had been available initially. When DNS answers change while a Connection Attempt Delay timer is running, the client MAY reschedule the timer. While a DNS update alone does not justify rescheduling the timer, if at least one connection attempt started before the DNS update has not made progress within an expected time window, the client MAY treat this as a signal that the previous addresses may no longer be viable and fire the timer sooner to start a connection attempt to the next address in the updated list, subject to the Minimum Connection Attempt Delay.

Supporting IPv6-Mostly and IPv6-Only Networks While many IPv6 transition protocols have been standardized and deployed, most are transparent to client devices. Supporting IPv6-only networks often requires specific client-side changes, especially when interacting with IPv4-only services. Two primary mechanisms for this are the combined use of NAT64 with DNS64 , or leveraging NAT64 with a discovered PREF64 prefix . One possible way to handle these networks is for the client device networking stack to implement 464XLAT . 464XLAT has the advantage of not requiring changes to user space software; however, it requires per-packet translation if the application is using IPv4 literals and does not encourage client application software to support native IPv6. On platforms that do not support 464XLAT, the Happy Eyeballs engine SHOULD follow the recommendations in this section to properly support IPv6-mostly () and IPv6-only networks. The features described in this section SHOULD only be enabled when the host detects an IPv6-mostly or IPv6-only network. A simple heuristic to detect one of these networks is to check if the network offers routable IPv6 addressing, does not offer routable IPv4 addressing, and offers a DNS resolver address.

IPv4 Address Literals If client applications or users wish to connect to IPv4 address literals, the Happy Eyeballs engine will need to perform NAT64 address synthesis for them. The solution is similar to "Bump-in-the-Host" but is implemented inside the Happy Eyeballs client. Note that some IPv4 prefixes are scoped to a given host or network, such as 0.0.0.0/8, 127.0.0.0/8, 169.254.0.0/16, and 255.255.255.255/32, and therefore do not require NAT64 address synthesis.

Discovering and Utilizing PREF64 When an IPv4 address is passed into the Happy Eyeballs implementation instead of a hostname, it SHOULD use PREF64s received from Router Advertisements . With PREF64 available, networks might choose to not deploy DNS64, as the latter has a number of disadvantages (see ). To ensure compatibility with such networks, if PREF64 is available, clients SHOULD send an A query in addition to an AAAA query for a given hostname. This allows the client to receive any existing IPv4 A records and perform local NAT64 address synthesis, eliminating the network's need to run DNS64. If the network does not provide PREF64s, the implementation SHOULD query the network for the NAT64 prefix using "Discovery of the IPv6 Prefix Used for IPv6 Address Synthesis" . It then synthesizes an appropriate IPv6 address (or several) using the encoding described in "IPv6 Addressing of IPv4/ IPv6 Translators" . The synthesized addresses are then inserted into the list of addresses as if they were results from DNS A queries; connection attempts follow the algorithm described above (see ). Such translation also applies to any IPv4 addresses received in A records and IPv4 address hints received in SVCB records.

Supporting DNS64 If PREF64 is not available and the NAT64 prefix cannot be discovered, clients SHOULD assume the network is relying on DNS64 for IPv4-to-IPv6 address synthesis. In this scenario, clients will typically only receive AAAA records from DNS queries, as DNS64 servers synthese these records for IPv4-only domains.

Hostnames with Broken AAAA Records At the time of writing, there exist a small but non-negligible number of hostnames that resolve to valid A records and broken AAAA records, which we define as AAAA records that contain seemingly valid IPv6 addresses but those addresses never reply when contacted on the usual ports. These can be, for example, caused by:

Mistyping of the IPv6 address in the DNS zone configuration
Routing black holes
Service outages

While an algorithm complying with the other sections of this document would correctly handle such hostnames on a dual-stack network, they will not necessarily function correctly on IPv6-only networks with NAT64 and DNS64. Since DNS64 recursive resolvers rely on the authoritative name servers sending negative (no error, no data) responses for AAAA records in order to synthesize, they will not synthesize records for these particular hostnames and will instead pass through the broken AAAA record. In order to support these scenarios, the client device needs to query the DNS for the A record and then perform local synthesis. Since these types of hostnames are rare and, in order to minimize load on DNS servers, this A query should only be performed when the client has given up on the AAAA records it initially received. This can be achieved by using a longer timeout, referred to as the "Last Resort Local Synthesis Delay"; the delay is recommended to be 2 seconds. The timer is started when the last connection attempt is fired. If no connection attempt has succeeded when this timer fires, the device queries the DNS for the IPv4 address and, on reception of a valid A record, treats it as if it were provided by the application (see ).

Virtual Private Networks Some Virtual Private Networks (VPNs) may be configured to handle DNS queries from the device. The configuration could encompass all queries or a subset such as "*.internal.example.com". These VPNs can also be configured to only route part of the IPv4 address space, such as 192.0.2.0/24. However, if an internal hostname resolves to an external IPv4 address, these can cause issues if the underlying network is IPv6-only. As an example, let's assume that "www.internal.example.com" has exactly one A record, 198.51.100.42, and no AAAA records. The client will send the DNS query to the company's recursive resolver and that resolver will reply with these records. The device now only has an IPv4 address to connect to and no route to that address. Since the company's resolver does not know the NAT64 prefix of the underlying network, it cannot synthesize the address. Similarly, the underlying network's DNS64 recursive resolver does not know the company's internal addresses, so it cannot resolve the hostname. Because of this, the client device needs to resolve the A record using the company's resolver and then locally synthesize an IPv6 address, as if the resolved IPv4 address were provided by the application ().

Summary of Configurable Values The values that may be configured as defaults on a client for use in Happy Eyeballs are as follows:

Resolution Delay (): The time to wait for AAAA and/or SVCB/HTTPS records after receiving an A record. Recommended to be 50 milliseconds.
Preferred Address Family Count (): The number of endpoints belonging to the preferred address family (such as IPv6) that should be attempted before attempting the next address family. Recommended to be 1; 2 may be used to more aggressively favor a particular combination of address family and protocol.
Connection Attempt Delay (): The time to wait between connection attempts in the absence of RTT data. Recommended to be 250 milliseconds.
Minimum Connection Attempt Delay (): The minimum time to wait between connection attempts. Recommended to be 100 milliseconds. MUST NOT be less than 10 milliseconds.
Maximum Connection Attempt Delay (): The maximum time to wait between connection attempts. Recommended to be 2 seconds.
Last Resort Local Synthesis Delay (): The time to wait after starting the last IPv6 attempt and before sending the A query. Recommended to be 2 seconds.

The delay values described in this section were determined empirically by measuring the timing of connections on a very wide set of production devices. They were picked to reduce wait times noticed by users while minimizing load on the network. As time passes, it is expected that the properties of networks will evolve. For that reason, it is expected that the values chosen by implementations will change over time. Implementors MAY use values different from the recommended values listed above, without changing this specification, as long as they do not violate any of the normative requirements (such as minimum delays).

Limitations Happy Eyeballs will handle initial connection failures at the transport layer (such as TCP or QUIC); however, other failures or performance issues may still affect the chosen connection.

Path Maximum Transmission Unit Discovery Connections that use TCP only (without TLS or another protocol on top) perform connection racing only during the TCP handshake. Such connections can encounter issues where the handshake (using small packets) completes but then large packets are dropped or incorrectly handled by the network due to a small MTU being configured on the network. Such issues might be specific to IPv6 traffic due to the IPv6 header being larger than the IPv4 header. This problem can also occur when TLS is used, but the larger messages used in many TLS handshakes are often sufficient to ensure maximum-size TCP segments are deliverable. For QUIC connections, a minimum MTU of at least 1200 bytes is guaranteed, but there is a chance that larger values may not be available. Solving these issues is out of scope of this document. One suggested approach is to use "Packetization Layer Path MTU Discovery" .

Application Layer If the DNS returns multiple endpoints for different application servers, the application itself may not be operational and functional on all of them. Common examples include Transport Layer Security (TLS) and HTTP.

Hiding Operational Issues It has been observed in practice that Happy Eyeballs can hide issues in networks. For example, if a misconfiguration causes IPv6 to consistently fail on a given network while IPv4 is still functional, Happy Eyeballs may impair the operator's ability to notice the issue. It is recommended that network operators deploy external means of monitoring to ensure functionality of all address families.

Security Considerations Note that applications should not rely upon a stable hostname-to-address mapping to ensure any security properties, since DNS results may change between queries. Happy Eyeballs may make it more likely that subsequent connections to a single hostname use different IP addresses. When using HTTP, HTTPS resource records indicate that clients should require HTTPS when connecting to an origin (see ), so an active attacker can attempt a downgrade attack by interfering with the successful delivery of HTTPS resource records. When clients use insecure DNS mechanisms, any on-path attacker can simply drop HTTPS resource records, so clients cannot tell the difference between an attack and a resolver that fails to respond to HTTPS queries. However, when using cryptographically protected DNS mechanisms, as described in , both SVCB-reliant and SVCB-optional clients MUST NOT send any unencrypted data after the TCP handshake completes unless they have received a valid HTTPS response. Those clients need to complete a TLS handshake before proceeding if that response is non-negative.

IANA Considerations This document does not require any IANA actions.

References Normative References TLS Encrypted Client Hello This document describes a mechanism in Transport Layer Security (TLS) for encrypting a message under a server public key. Service Binding and Parameter Specification via the DNS (SVCB and HTTPS Resource Records) This document specifies the "SVCB" ("Service Binding") and "HTTPS" DNS resource record (RR) types to facilitate the lookup of information needed to make connections to network services, such as for HTTP origins. SVCB records allow a service to be provided from multiple alternative endpoints, each with associated parameters (such as transport protocol configuration), and are extensible to support future uses (such as keys for encrypting the TLS ClientHello). They also enable aliasing of apex domains, which is not possible with CNAME. The HTTPS RR is a variation of SVCB for use with HTTP (see RFC 9110, "HTTP Semantics"). By providing more information to the client before it attempts to establish a connection, these records offer potential benefits to both performance and privacy. Key words for use in RFCs to Indicate Requirement Levels In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Bootstrapping TLS Encrypted ClientHello with DNS Service Bindings To use TLS Encrypted ClientHello (ECH), the client needs to learn the ECH configuration for a server before it attempts a connection to the server. This specification provides a mechanism for conveying the ECH configuration information via DNS, using a SVCB or HTTPS resource record (RR). Default Address Selection for Internet Protocol Version 6 (IPv6) This document describes two algorithms, one for source address selection and one for destination address selection. The algorithms specify default behavior for all Internet Protocol version 6 (IPv6) implementations. They do not override choices made by applications or upper-layer protocols, nor do they preclude the development of more advanced mechanisms for address selection. The two algorithms share a common context, including an optional mechanism for allowing administrators to provide policy that can override the default behavior. In dual-stack implementations, the destination address selection algorithm can consider both IPv4 and IPv6 addresses -- depending on the available source addresses, the algorithm might prefer IPv6 addresses over IPv4 addresses, or vice versa. Default address selection as defined in this specification applies to all IPv6 nodes, including both hosts and routers. This document obsoletes RFC 3484. [STANDARDS-TRACK] Prioritizing known-local IPv6 ULAs through address selection policy Energy Sciences Network Jisc Tachyon Dynamics This document updates the default address selection algorithm for Internet Protocol Version 6 (IPv6), originally specified in RFC 6724, based on accumulated operational experience. It introduces the concept of "known-local" Unique Local Address (ULA) prefixes within the fd00::/8 block and specifies that ULA-to-ULA communications using such prefixes should be preferred over both IPv4-to-IPv4 and GUA-to- GUA (Global Unicast Address) communications in local use scenarios. The document defines mechanisms for nodes to identify and incorporate known-local prefixes into their address selection policy tables. It introduces a requirement to implement Rule 5.5 of RFC 6724 and reduces the default precedence for 6to4 addresses. These updates enhance the supportability of typical deployment environments, including automatic and unmanaged configurations, and promote consistent IPv6-over-IPv4 precedence behavior for both ULA and GUA within local networks. The document acknowledges that certain atypical deployment models may require explicit configuration to achieve intended operational outcomes. Computing TCP's Retransmission Timer This document defines the standard algorithm that Transmission Control Protocol (TCP) senders are required to use to compute and manage their retransmission timer. It expands on the discussion in Section 4.2.3.1 of RFC 1122 and upgrades the requirement of supporting the algorithm from a SHOULD to a MUST. This document obsoletes RFC 2988. [STANDARDS-TRACK] Stateful NAT64: Network Address and Protocol Translation from IPv6 Clients to IPv4 Servers This document describes stateful NAT64 translation, which allows IPv6-only clients to contact IPv4 servers using unicast UDP, TCP, or ICMP. One or more public IPv4 addresses assigned to a NAT64 translator are shared among several IPv6-only clients. When stateful NAT64 is used in conjunction with DNS64, no changes are usually required in the IPv6 client or the IPv4 server. DNS64: DNS Extensions for Network Address Translation from IPv6 Clients to IPv4 Servers DNS64 is a mechanism for synthesizing AAAA records from A records. DNS64 is used with an IPv6/IPv4 translator to enable client-server communication between an IPv6-only client and an IPv4-only server, without requiring any changes to either the IPv6 or the IPv4 node, for the class of applications that work through NATs. This document specifies DNS64, and provides suggestions on how it should be deployed in conjunction with IPv6/IPv4 translators. [STANDARDS-TRACK] Discovering PREF64 in Router Advertisements This document specifies a Neighbor Discovery option to be used in Router Advertisements (RAs) to communicate prefixes of Network Address and Protocol Translation from IPv6 clients to IPv4 servers (NAT64) to hosts. Dual-Stack Hosts Using "Bump-in-the-Host" (BIH) Bump-in-the-Host (BIH) is a host-based IPv4 to IPv6 protocol translation mechanism that allows a class of IPv4-only applications that work through NATs to communicate with IPv6-only peers. The host on which applications are running may be connected to IPv6-only or dual-stack access networks. BIH hides IPv6 and makes the IPv4-only applications think they are talking with IPv4 peers by local synthesis of IPv4 addresses. This document obsoletes RFC 2767 and RFC 3338. [STANDARDS-TRACK] Discovery of the IPv6 Prefix Used for IPv6 Address Synthesis This document describes a method for detecting the presence of DNS64 and for learning the IPv6 prefix used for protocol translation on an access network. The method depends on the existence of a well-known IPv4-only fully qualified domain name "ipv4only.arpa.". The information learned enables nodes to perform local IPv6 address synthesis and to potentially avoid NAT64 on dual-stack and multi-interface deployments. IPv6 Addressing of IPv4/IPv6 Translators This document discusses the algorithmic translation of an IPv6 address to a corresponding IPv4 address, and vice versa, using only statically configured information. It defines a well-known prefix for use in algorithmic translations, while allowing organizations to also use network-specific prefixes when appropriate. Algorithmic translation is used in IPv4/IPv6 translators, as well as other types of proxies and gateways (e.g., for DNS) used in IPv4/IPv6 scenarios. [STANDARDS-TRACK] QUIC: A UDP-Based Multiplexed and Secure Transport This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm. Packetization Layer Path MTU Discovery This document describes a robust method for Path MTU Discovery (PMTUD) that relies on TCP or some other Packetization Layer to probe an Internet path with progressively larger packets. This method is described as an extension to RFC 1191 and RFC 1981, which specify ICMP-based Path MTU Discovery for IP versions 4 and 6, respectively. [STANDARDS-TRACK] Service Binding and Parameter Specification via the DNS (SVCB and HTTPS Resource Records) This document specifies the "SVCB" ("Service Binding") and "HTTPS" DNS resource record (RR) types to facilitate the lookup of information needed to make connections to network services, such as for HTTP origins. SVCB records allow a service to be provided from multiple alternative endpoints, each with associated parameters (such as transport protocol configuration), and are extensible to support future uses (such as keys for encrypting the TLS ClientHello). They also enable aliasing of apex domains, which is not possible with CNAME. The HTTPS RR is a variation of SVCB for use with HTTP (see RFC 9110, "HTTP Semantics"). By providing more information to the client before it attempts to establish a connection, these records offer potential benefits to both performance and privacy. Informative References Happy Eyeballs Version 2: Better Connectivity Using Concurrency Many communication protocols operating over the modern Internet use hostnames. These often resolve to multiple IP addresses, each of which may have different performance and connectivity characteristics. Since specific addresses or address families (IPv4 or IPv6) may be blocked, broken, or sub-optimal on a network, clients that attempt multiple connections in parallel have a chance of establishing a connection more quickly. This document specifies requirements for algorithms that reduce this user-visible delay and provides an example algorithm, referred to as "Happy Eyeballs". This document obsoletes the original algorithm description in RFC 6555. Happy Eyeballs: Success with Dual-Stack Hosts When a server's IPv4 path and protocol are working, but the server's IPv6 path and protocol are not working, a dual-stack client application experiences significant connection delay compared to an IPv4-only client. This is undesirable because it causes the dual- stack client to have a worse user experience. This document specifies requirements for algorithms that reduce this user-visible delay and provides an algorithm. [STANDARDS-TRACK] HTTP/3 The QUIC transport protocol has several features that are desirable in a transport for HTTP, such as stream multiplexing, per-stream flow control, and low-latency connection establishment. This document describes a mapping of HTTP semantics over QUIC. This document also identifies HTTP/2 features that are subsumed by QUIC and describes how HTTP/2 extensions can be ported to HTTP/3. HTTP Alternative Services This document specifies "Alternative Services" for HTTP, which allow an origin's resources to be authoritatively available at a separate network location, possibly accessed with a different protocol configuration. Internet Protocol, Version 6 (IPv6) Specification This document specifies version 6 of the Internet Protocol (IPv6). It obsoletes RFC 2460. QUIC: A UDP-Based Multiplexed and Secure Transport This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm. Optimistic DNS Apple Inc. Apple Inc. Apple Inc. DNS lookups introduce user-visible delay, particularly when cached records have expired and must be refreshed from the network. This document describes Optimistic DNS, a client-side stub resolver mechanism that immediately returns expired cached DNS records to applications while simultaneously refreshing them with a network query. The application receives an answer in microseconds rather than milliseconds, and if the data has changed receives an updated answer shortly thereafter. Optimistic DNS is complementary to RFC 8767, which addresses serving stale data at recursive resolvers. This document focuses exclusively on client-side stub resolver behavior, including explicit signaling from the application to inform the stub resolver that the application is able to handle old and possibly incorrect information. TCP Fast Open This document describes an experimental TCP mechanism called TCP Fast Open (TFO). TFO allows data to be carried in the SYN and SYN-ACK packets and consumed by the receiving end during the initial connection handshake, and saves up to one full round-trip time (RTT) compared to the standard TCP, which requires a three-way handshake (3WHS) to complete before data can be exchanged. However, TFO deviates from the standard TCP semantics, since the data in the SYN could be replayed to an application in some rare circumstances. Applications should not use TFO unless they can tolerate this issue, as detailed in the Applicability section. Multiple Provisioning Domain Architecture This document is a product of the work of the Multiple Interfaces Architecture Design team. It outlines a solution framework for some of the issues experienced by nodes that can be attached to multiple networks simultaneously. The framework defines the concept of a Provisioning Domain (PvD), which is a consistent set of network configuration information. PvD-aware nodes learn PvD-specific information from the networks they are attached to and/or other sources. PvDs are used to enable separation and configuration consistency in the presence of multiple concurrent connections. QUIC Loss Detection and Congestion Control This document describes loss detection and congestion control mechanisms for QUIC. The Transport Layer Security (TLS) Protocol Version 1.3 This document specifies version 1.3 of the Transport Layer Security (TLS) protocol. TLS allows client/server applications to communicate over the Internet in a way that is designed to prevent eavesdropping, tampering, and message forgery. This document updates RFCs 5705 and 6066, and obsoletes RFCs 5077, 5246, and 6961. This document also specifies new requirements for TLS 1.2 implementations. Using TLS to Secure QUIC This document describes how Transport Layer Security (TLS) is used to secure QUIC. DNS Security Extensions (DNSSEC) This document describes the DNS Security Extensions (commonly called "DNSSEC") that are specified in RFCs 4033, 4034, and 4035, as well as a handful of others. One purpose is to introduce all of the RFCs in one place so that the reader can understand the many aspects of DNSSEC. This document does not update any of those RFCs. A second purpose is to state that using DNSSEC for origin authentication of DNS data is the best current practice. A third purpose is to provide a single reference for other documents that want to refer to DNSSEC. Specification for DNS over Transport Layer Security (TLS) This document describes the use of Transport Layer Security (TLS) to provide privacy for DNS. Encryption provided by TLS eliminates opportunities for eavesdropping and on-path tampering with DNS queries in the network, such as discussed in RFC 7626. In addition, this document specifies two usage profiles for DNS over TLS and provides advice on performance considerations to minimize overhead from using TCP and TLS with DNS. This document focuses on securing stub-to-recursive traffic, as per the charter of the DPRIVE Working Group. It does not prevent future applications of the protocol to recursive-to-authoritative traffic. DNS Queries over HTTPS (DoH) This document defines a protocol for sending DNS queries and getting DNS responses over HTTPS. Each DNS query-response pair is mapped into an HTTP exchange. DNS over Dedicated QUIC Connections This document describes the use of QUIC to provide transport confidentiality for DNS. The encryption provided by QUIC has similar properties to those provided by TLS, while QUIC transport eliminates the head-of-line blocking issues inherent with TCP and provides more efficient packet-loss recovery than UDP. DNS over QUIC (DoQ) has privacy properties similar to DNS over TLS (DoT) specified in RFC 7858, and latency characteristics similar to classic DNS over UDP. This specification describes the use of DoQ as a general-purpose transport for DNS and includes the use of DoQ for stub to recursive, recursive to authoritative, and zone transfer scenarios. TLS Key Share Prediction Google LLC This document defines a mechanism for servers to communicate supported key share algorithms in DNS. Clients may use this information to reduce TLS handshake round-trips. DNS Push Notifications The Domain Name System (DNS) was designed to return matching records efficiently for queries for data that are relatively static. When those records change frequently, DNS is still efficient at returning the updated results when polled, as long as the polling rate is not too high. But, there exists no mechanism for a client to be asynchronously notified when these changes occur. This document defines a mechanism for a client to be notified of such changes to DNS records, called DNS Push Notifications. 464XLAT: Combination of Stateful and Stateless Translation This document describes an architecture (464XLAT) for providing limited IPv4 connectivity across an IPv6-only network by combining existing and well-known stateful protocol translation (as described in RFC 6146) in the core and stateless protocol translation (as described in RFC 6145) at the edge. 464XLAT is a simple and scalable technique to quickly deploy limited IPv4 access service to IPv6-only edge networks without encapsulation. IPv6-mostly Networks: Deployment and Operations Considerations Energy Sciences Network RIPE NCC Google This document discusses a deployment scenario called "an IPv6-mostly network", when IPv6-only and IPv4-enabled endpoints coexist on the same network (network segment, VLAN, SSID etc). The proposed approach enables smooth and incremental transition from dual-stack to IPv6-only network by allowing IPv6-capable devices to remain IPv6-only while the network is seamlessly supplying IPv4 to those that require it.

Acknowledgments The authors thank Dan Wing, Andrew Yourtchenko, and everyone else who worked on the original Happy Eyeballs design , Josh Graessley, Stuart Cheshire, and the rest of team at Apple that helped implement and instrument this algorithm, and Jason Fesler and Paul Saab who helped measure and refine this algorithm. The authors would also like to thank Fred Baker, Nick Chettle, Lorenzo Colitti, Igor Gashinsky, Geoff Huston, Jen Linkova, Paul Hoffman, Philip Homburg, Warren Kumari, Erik Nygren, Jordi Palet Martinez, Rui Paulo, Stephen Strowes, Jinmei Tatuya, Dave Thaler, Joe Touch, and James Woodyatt for their input and contributions.