]> Google LLC

ianswett@google.com

Meta Platforms, Inc.

jbeshay@meta.com

Transport QUIC Internet-Draft This document defines an extension to the QUIC transport protocol which supports reporting multiple packet receive timestamps for post-handshake packets.

Introduction The QUIC Transport Protocol provides a secure, multiplexed connection for transmitting reliable streams of application data. This document defines an extension to the QUIC transport protocol which supports reporting multiple packet receive timestamps.

Motivation QUIC congestion control () supports sampling round-trip time (RTT) by measuring the time from when a packet was sent to when it is acknowledged. However, more precise delay signals measured via packet receive timestamps have the potential to improve the accuracy of network bandwidth measurements and the effectiveness of congestion control, especially for latency-critical applications such as real-time video conferencing or game streaming. Numerous existing algorithms and techniques leverage receive timestamps to improve transport performance. Examples include:

• The WebRTC congestion control algorithm described in uses the difference between packet inter-departure and packet inter-arrival times as the input to its delay-based controller.
• The pathChirp () technique estimates available bandwidth by measuring inter-arrival time of multiple packets.

Notably, these techniques require receive timestamps for more than one packet per round-trip in order to best measure the network. Additionally, receive timestamps can provide valuable network telemetry, even if they are not used by the congestion controller.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

New ACK_RECEIVE_TIMESTAMPS Frame Wire Format Once the receive timestamps extension is negotiated (see ), an endpoint MAY use the ACK_RECEIVE_TIMESTAMPS frame defined below to report receive timestamps to its peer. An endpoint MAY continue to use the existing ACK frames as specified in if it does not have any receive timestamps or does not want to report them. Endpoints send ACK_RECEIVE_TIMESTAMPS frames in 1-RTT packets, with 0 or more receive timestamps following the Ack Ranges and optional ECN Counts. Similar to to the ACK frame types (x02..0x03), the ACK_RECEIVE_TIMESTAMPS frame defines two frame types (0x03178307..0x03178308) to indicate whether the frame includes ECN counts. ACK frames are never sent in 0-RTT packets, so the same applies to ACK_RECEIVE_TIMESTAMPS frames.

ACK Frame Format

The fields Largest Acknowledged, ACK Delay, ACK Range Count, First ACK Range, ACK Range and ECN Counts are the same as for ACK (type=0x02..0x03) frames specified in . The format of the Receive Timestamps field is shown in .

Receive Timestamps Fields

Timestamp Range Count:

A variable-length integer specifying the number of Timestamp Range fields in the frame.

Timestamp Ranges:

Ranges of receive timestamps for contiguous packets in descending packet number order; see .

Timestamp Ranges Each Timestamp Range describes a series of contiguous packet receive timestamps in descending sequential packet number (and descending timestamp) order. Timestamp Ranges consist of a Delta Largest Acknowledged indicating the largest packet number in the range, followed by a list of Timestamp Deltas describing the relative receive timestamps for each contiguous packet in the Timestamp Range (descending). Packets within a range are in descending packet number and timestamp order. Ranges are in descending timestamp order but do not have to be in descending packet number order. Each packet in a range MUST be an acknowledged packet, i.e., the packet number MUST have been included in an ACK Range in the current or a previously sent ACK, ACK_RECEIVE_TIMESTAMPS, PATH_ACK, or PATH_ACK_RECEIVE_TIMESTAMPS frame. Timestamp Ranges are structured as shown in .

Timestamp Range Format

The fields that form each Timestamp Range are:

Delta Largest Acknowledged:

A variable-length integer indicating the largest packet number in the Timestamp Range as a delta to subtract from the Largest Acknowledged in the ACK frame. For example, 0 indicates the range starts with the Largest Acknowledged.

Timestamp Delta Count:

A variable-length integer indicating the number of Timestamp Deltas in the current Timestamp Range. The sum of Timestamp Delta Counts for all Timestamp Ranges in the frame MUST NOT exceed max_receive_timestamps_per_ack as specified in .

Timestamp Deltas:

Variable-length integers encoding the receive timestamp for contiguous packets in the Timestamp Range in descending packet number order as follows: For the first Timestamp Delta of the first Timestamp Range in the frame: the value is the difference between (a) the receive timestamp of the largest packet in the Timestamp Range (indicated by Gap) and (b) the session receive_timestamp_basis (see ), decoded as described below. For all other Timestamp Deltas: the value is the difference between (a) the receive timestamp specified by the previous Timestamp Delta and (b) the receive timestamp of the current packet in the Timestamp Range, decoded as described below. All Timestamp Delta values are decoded by mulitplying the value in the field by 2 to the power of the receive_timestamps_exponent transport parameter received by the sender of the frame (see ):

When the receiver receives packets out-of-order, it SHOULD report them with other packets in a single ACK_RECEIVE_TIMESTAMPS or PATH_ACK_RECEIVE_TIMESTAMPS frame, starting with the most recently received packet regardless of the packet number order. See for examples of reporting timestamps of out-of-order packets.

PATH_ACK_RECEIVE_TIMESTAMPS Frame Wire Format When both the receive timestamps extension and the multipath extension are negotiated, an endpoint MAY use the PATH_ACK_RECEIVE_TIMESTAMPS frame defined below to report receive timestamps for packets received on a specific path. An endpoint MAY continue to use the existing PATH_ACK frames as specified in if it does not have any receive timestamps or does not want to report them. Endpoints send PATH_ACK_RECEIVE_TIMESTAMPS frames in 1-RTT packets, with 0 or more receive timestamps following the Ack Ranges and optional ECN Counts. Similar to the PATH_ACK frame types (0x3e..0x3f), the PATH_ACK_RECEIVE_TIMESTAMPS frame defines two frame types (0x03178309..0x0317830a) to indicate whether the frame includes ECN counts.

PATH_ACK_RECEIVE_TIMESTAMPS Frame Format

Compared to the ACK_RECEIVE_TIMESTAMPS frame defined in , the following field is added:

Path Identifier:

The path ID associated with the packet number space of the 1-RTT packets which are acknowledged by this frame, as specified in .

All other fields are the same as for the ACK_RECEIVE_TIMESTAMPS frame (). The Receive Timestamps field follows the same format described in . When truncation is necessary to fit within the max_receive_timestamps_per_ack limit or reduce the size of the frame, the receiver SHOULD retain timestamps for the most recently received packets and omit timestamps for older packets.

Extension Negotiation

max_receive_timestamps_per_ack (0x4ac07 temporary value for draft use):

A variable-length integer indicating that the maximum number of receive timestamps the sending endpoint would like to receive in an ACK_RECEIVE_TIMESTAMPS or PATH_ACK_RECEIVE_TIMESTAMPS frame. If this transport parameter is absent, this extension is not supported, which is equivalent to a value of 0. Each ACK_RECEIVE_TIMESTAMPS or PATH_ACK_RECEIVE_TIMESTAMPS frame sent MUST NOT contain more than the peer's maximum number of receive timestamps. An endpoint MUST NOT send ACK_RECEIVE_TIMESTAMPS or PATH_ACK_RECEIVE_TIMESTAMPS frames to a peer whose max_receive_timestamps_per_ack is not present or 0. A value of 0 does not permit sending these frames with zero timestamps; it prohibits sending them entirely.

receive_timestamps_exponent (0x4ac26 temporary value for draft use):

A variable-length integer indicating the exponent to be used when encoding and decoding timestamp delta fields in ACK_RECEIVE_TIMESTAMPS and PATH_ACK_RECEIVE_TIMESTAMPS frames sent by the peer (see ). If this value is absent, a default value of 0 is assumed (indicating microsecond precision). Values above 20 are invalid. If the receive_timestamps_exponent transport parameter is present and max_receive_timestamps_per_ack is absent or has a value of 0, receive timestamps are not supported and receive_timestamps_exponent MUST be ignored.

Receive Timestamp Basis Endpoints which negotiate the extension need to determine a value, receive_timestamp_basis, relative to which all receive timestamps for the session will be reported (see ). The value of receive_timestamp_basis MUST be less than the smallest receive timestamp reported, and MUST remain constant for the entire duration of the session. The receive_timestamp_basis is a local value that is not communicated to the peer. Receive timestamps are reported relative to the basis, rather than in absolute time to avoid requiring clock synchronization between endpoints and to make the frame more compact.

Discussion

Best-Effort Behavior Receive timestamps are sent on a best-effort basis. Endpoints MUST gracefully handle scenarios where the receiver does not communicate receive timestamps for acknowledged packets. Examples of such scenarios are:

• A packet containing an ACK_RECEIVE_TIMESTAMPS or PATH_ACK_RECEIVE_TIMESTAMPS frame is lost.
• The receiver truncates the number of timestamps sent in order to (a) avoid sending more than max_receive_timestamps_per_ack (); or (b) fit the ACK_RECEIVE_TIMESTAMPS or PATH_ACK_RECEIVE_TIMESTAMPS frame into a packet.
• The receiver is unable to measure the arrival timestamp of a packet with sufficient accuracy, for example due to a scheduling delay in a userspace implementation, and omits the packet from the Timestamp Ranges while still acknowledging it in the ACK Ranges.

Frame Size The addition of receive timestamps increases the size of ACK frames. Receivers SHOULD use receive timestamps to fill available space in packets that would already be sent, rather than sending additional packets solely to report timestamps. In such cases, the receiver would send fewer timestamps than the maximum allowed by max_receive_timestamps_per_ack.

Examples To illustrate the usage of the Receive Timestamps fields, consider a peer that sent 14 packets with numbers 87 to 100. Assume the receiver receives packets 87 to 91 and 96 to 100 at the following timestamps relative to the basis:

Packet Number	Relative Timestamp
87	300
88	305
89	310
90	320
91	330
96	350
97	355
98	360
99	370
100	380

When it's time to acknowledge these packets, the receiver will send an ACK frame with two ranges, as follows: After that assume that the receiver receives packets 92 to 95 out-of-order at the following timestamps relative to the basis:

Packet Number	Relative Timestamp
92	390
93	392
94	394
95	395

The receiver can send a new ACK frame with all of the timestamps, as follows: In this particular scenario, the receiver can also choose to report the first timestamp range only since the timestamps for the other two ranges have already been reported.

Security Considerations The security properties described in apply, as this extension only defines frames sent in 1-RTT packets, which are authenticated and integrity-protected. As noted in , congestion control relies on signals from unauthenticated entities. Receive timestamps are integrity-protected but remain under the peer's control and could be crafted to influence the sender's congestion control or bandwidth estimation. A malicious peer could report incorrect timestamps to cause the sender to over- or under-estimate available bandwidth. Implementations that use receive timestamps SHOULD perform plausibility checks and bound their impact using other signals (loss, ECN, RTT); implausible timestamps MAY be ignored. Processing and storing timestamps requires additional state. Endpoints SHOULD bound the resources dedicated to this extension, advertise a max_receive_timestamps_per_ack they are willing to handle, and be prepared to truncate or ignore excess timestamps as described in . The size of ACK_RECEIVE_TIMESTAMPS and PATH_ACK_RECEIVE_TIMESTAMPS frames is bounded by the peer's advertised limit and the packet size, limiting amplification as discussed in .

IANA Considerations This document uses temporary values for the transport parameters max_receive_timestamps_per_ack (0x4ac07) and receive_timestamps_exponent (0x4ac26), and for the frame types ACK_RECEIVE_TIMESTAMPS (0x03178307..0x03178308) and PATH_ACK_RECEIVE_TIMESTAMPS (0x03178309..0x0317830a). Prior to publication, these will be replaced with permanent registrations in the "QUIC Transport Parameters" () and "QUIC Frame Types" () registries.

References Normative References Alibaba Inc. Uber Technologies Inc. University of Mons (UMONS) UCLouvain and WELRI Private Octopus Inc. Ericsson This document specifies a multipath extension for the QUIC protocol to enable the simultaneous usage of multiple paths for a single connection. It introduces explicit path identifiers to create, delete, and manage multiple paths. This document does not specify address discovery or management, nor how applications using QUIC schedule traffic over multiple paths. This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References This document describes loss detection and congestion control mechanisms for QUIC. Google Google Politecnico di Bari Politecnico di Bari Politecnico di Bari This document describes two methods of congestion control when using real-time communications on the World Wide Web (RTCWEB); one delay- based and one loss-based. It is published as an input document to the RMCAT working group on congestion control for media streams. The mailing list of that working group is rmcat@ietf.org.

Acknowledgments The editors would like to thank Connor Smith for writing the initial draft. The editors would also like to thank Sharad Jaiswal, Ilango Purushothaman, and Brandon Schlinker for their contributions to the design of this QUIC extension.