]> Hammerspace

United States of America bcodding@hammerspace.com

Red Hat

United States of America smayhew@redhat.com

United States of America cel-ietf@chucklever.net

Transport Network File System Version 4 This document specifies a protocol for conveying Remote Procedure (RPC) messages via QUIC version 1 connections. It requires no revision to application RPC protocols or the RPC protocol itself. Note This note is to be removed before publishing as an RFC. Discussion of this draft occurs on the NFSv4 working group mailing list, archived at . Working Group information is available at . Submit suggestions and changes as pull requests at . Instructions are on that page.

Introduction The QUIC version 1 protocol is a secure, reliable connection-oriented network transport described in . Its features include integrated transport layer security, multiple independent streams over each connection, fast reconnecting, and advanced packet loss recovery and congestion avoidance mechanisms. Open Network Computing Remote Procedure Call (often shortened to "RPC") is a Remote Procedure Call protocol that runs over a variety of network transports . RPC implementations so far use UDP , TCP , or RDMA . This document specifies how to transport RPC messages over QUIC version 1.

Motivation For a New RPC Transport Viewed at a moderate distance, RPC over QUIC provides a similar feature set as RPC over TCP with TLS (as described in ). However, a closer look reveals some essential benefits of using QUIC transports:

• Even though the QUIC protocol utilizes the same set of encryption algorithms as TLSv1.3, the QUIC record protocol encrypts nearly the entire transport layer header and authenticates each IP packet. Advanced traffic analysis which was possible with TLS on TCP is no longer possible. QUIC protects against transport packet spoofing and downgrade attacks better than TLS on TCP.
• Because many real IP networks are oversubscribed, packet loss due to momentary link or switch saturation continues to be likely even on well-maintained data center-quality network fabrics. The QUIC protocol utilizes packet loss recovery and congestion avoidance features that are lacking in TCP. Because TCP protocol design has ossified, it is unlikely to gain these improvements. QUIC is more extensible than TCP, meaning future improvements in this area can be designed and deployed without application disruption.
• Further, because QUIC handles packet loss on a per-stream rather than a per-connection basis, spreading RPC traffic across multiple streams enables workloads to continue largely unperturbed while packet recovery proceeds.
• The QUIC protocol is designed to facilitate secure and automatic transit of firewalls. Firewall transparency is a foundational feature of NFSv4 (which is built on RPC).

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

RPC-over-QUIC Framework RPC is first and foremost a message-passing protocol. This section covers the implementation details of exchanging RPC messages over QUIC. Readers should already be familiar with the fundamentals of ONC RPC . RPC-over-QUIC relies on QUIC version 1 as the underlying transport . The use of other QUIC transport versions with RPC MAY be defined by future specifications.

Establishing a Connection When a network host wishes to send RPC requests to a remote service via QUICv1, it must first find an established QUICv1 connection, or establish a new one. For the purpose of explanation, this document refers to the peer that initiates QUICv1 connection establishment as an "RPC client" peer. This document refers to the peer that passively accepts an incoming connection request as an "RPC server" peer. QUICv1 connections are not completely defined by the classic 5-tuple (IP proto, source address, source port, destination address, and destination port). Each connection is also defined by its QUIC connection ID. For instance, if the IP address of either peer should change, or a NAT/PAT binding and the source UDP port changes, the receiver can still recognize an ingress QUICv1 packet as belonging to an established connection. As a result, due to network conditions or administrative actions, an RPC-over-QUIC connection can be replaced (a reconnect event) or migrated (a failover event) without interrupting the operation of an upper layer protocol such as RPC-over-QUIC. A more complete discussion can be found in .

Connection Transport Parameters When establishing a connection, peers exchange transport parameters, as described in .

Initial Flow Control Limits These limits control the amount of data that each peer may send on a newly-created stream. The limits are used for flow control and cap the amount of memory needed by both peers to keep data flowing on the connection. The value of these limits are typically based on the bandwidth-delay of the physical link between the peers, and are not exposed to RPC applications.

Number of Streams Per Connection Each QUICv1 peer can limit the number of streams per connection; see . Given the definition of RPC message framing in , it is possible for an RPC client to create a stream, send one RPC Call, receive one RPC Reply, then destroy the stream. That usage might be common with simple RPC-based protocols like rpcbind. For protocols that carry a more intensive workload, this style of stream allocation generates needless overhead. Moreover, stream identifiers cannot be re-used on a single QUICv1 connection, so eventually a QUICv1 connection can no longer create a new stream for each RPC XID. Finally, a connection peer can advertise a max_streams value that is significantly lower than 2 ^ 60. Instead, RPC clients can create enough streams to maximize workload parallelism, and ought to avoid sending only a few RPCs on each stream before creating a new one. For example, an RPC client could allocate a handful of streams per CPU core to reduce contention for the streams and their associated data structures. Or, an RPC client could create a set of streams whose count is the same as the number of slots in an NFSv4.1 session. Even so, to provide a framework that makes implementing RPC-over-QUIC as fast and simple as possible, this specification needs to focus on enabling the use of as few as a single stream per connection. Servers that implement RPC-over-QUIC need to recognize that each additional stream amounts to incremental overhead. RPC servers MAY deny the creation of new streams if an RPC client already has many active streams. RPC clients need to be prepared for that behavior.

Maximum Frame Size This size is the largest QUIC frame that can appear in any stream on this connection. The QUIC framing protocol is not visible to the RPC application. The RPC client and server can therefore negotiate a frame size that enables efficient transit of RPC traffic with minimal internal memory fragmentation.

RPC Service Discovery For RPC, the destination port is special. RPC services may use a standardized destination port that is bound to an RPC program number. Such ports are assigned in the IANA Service Name and Transport Protocol Port Number registry . For example, the rpcbind program, which is RPC program 100000, listens on port 111. This is done so that RPC clients can always contact the rpcbind service and discover the other RPC services that are operating on that network peer. In other cases, an RPC service might use any available port. The RPC server registers its port number with the local rpcbind service so that RPC clients can contact that service. This mechanism is no different for RPC-over-QUIC than it is for RPC on other network transports. rpcbind clients specify an RPC program number and either the "quic" or "quic6" netid when requesting information about a QUIC-based RPC service. More detail is available in .

Transport Layer Security During connection establishment, the client peer indicates RPC-over-QUIC support by presenting the ALPN token "sunrpc" in the TLS handshake. Support for other application-layer protocols MAY be offered in the same handshake. As part of establishing a QUICv1 connection, the two connecting peers authenticate to each other and choose encryption parameters to establish a confidential channel of communication. All traffic on one QUICv1 connection is thus bound to the authenticated identities that were presented during connection establishment. These peer identities apply to the streams and RPC messages carried by that connection. RPC-over-QUIC provides peer authentication and encryption services using a framework based on Transport Layer Security (TLS). Ergo, RPC-over-QUIC inherently fulfills many of the requirements of . The details of QUIC's use of TLS are specified in . In particular:

• With QUICv1, security at the transport layer is always enabled. Therefore, the discussion in about the opportunistic use of TLS does not apply to RPC-over-QUIC, and the STARTTLS mechanism described in MUST NOT be used on RPC-over-QUIC connections.
• The peer authentication requirements in apply to RPC-over-QUIC.
• The PKIX Extended Key Usage values defined in are valid for use with RPC-over-QUIC.
• The ALPN defined in is also used for RPC-over-QUIC.

QUIC Streams RPC-over-QUIC connections are mediated entirely by each peer's RPC layer and, aside from authentication and connection transport parameters, are not otherwise visible to RPC applications. An RPC client establishes an RPC-over-QUIC connection whenever there are application RPC transactions to be executed. QUICv1 provides a "stream" abstraction, described in . A QUICv1 connection carries one or more streams. Once a QUICv1 connection has been established, either connection peer may create a stream. Typically, the RPC client peer creates the first stream on a connection. Unless explicitly specified, when RPC upper-layer protocol specifications refer to a "connection", for RPC-over-QUIC, this is a QUIC stream. For instance, when NFSv4.1 layered on RPC-over-QUIC uses BIND_CONN_TO_SESSION to associate a "connection" with a session, the bound entity is a QUICv1 stream rather than the encompassing QUICv1 connection. A peer that closes a QUICv1 stream signals that any RPC request still in flight on that stream is to be treated as lost. Closing a stream does not affect the encompassing QUICv1 connection or any other stream within it. In terms of TI-RPC semantic labels, a QUICv1 stream behaves as a "tpi_cots_ord" transport: connection-oriented and in order.

RPC Message Framing RPC-over-QUIC uses only bidirectional streams. When a connection peer creates a QUICv1 stream, that peer's stream endpoint is referred to as a "Requester", and MUST emit only RPC Call messages on that stream. The other endpoint is referred to as a "Responder", and MUST emit only RPC Reply messages on that stream. Receivers MUST silently discard RPC messages whose direction field does not match their Requester or Responder role. A receiver MAY also signal the violation to the peer by closing the offending stream with a RESET_STREAM frame () carrying an application error code; specific error code values and an extension process for defining additional codes are left to a future revision of this specification. Requesters and Responders match RPC Calls to RPC Replies using the XID carried in each RPC message. Responders MUST send RPC Replies on the same stream on which they received the matching RPC Call. Each QUICv1 stream provides reliable in-order delivery of bytes. However, each stream makes no guarantees about delivery order with regard to bytes on other streams on the same connection. The stream data containing RPC records is carried by QUIC STREAM frames, but this framing is invisible to the RPC layer. The transport layer buffers and orders received stream data, exposing only a reliable byte stream to the RPC layer. Although QUIC permits out-of-order delivery within a stream, RPC-over-QUIC does not make use of this feature. Because each QUICv1 stream is an ordered-byte stream, an RPC-over-QUIC stream carries only a sequence of complete RPC messages. Although data from multiple streams can be interleaved on a single QUICv1 connection, RPC messages MUST NOT be interleaved on one stream. Just as with RPC on a TCP socket, each RPC message is an ordered sequence of one or more records on a single stream. Such RPC records bear no relationship to QUIC stream frames; in fact, stream frames as defined in are not visible to RPC endpoints. Each RPC record begins with a four-octet record marker. A record marker contains the count of octets in the record in its lower 31 bits, and a flag that indicates whether the record is the last record in the RPC message in the highest order bit. See for a comparison with TCP record markers.

Receiver Data Placement Assistance One recurring weakness with RPC on TCP is that large payloads (for instance, in NFS WRITEs) can land at arbitrary offsets in receive buffers, limiting the ability for receivers to handle the payloads with zero-touch tactics such as direct I/O. It remains an open question whether RPC-over-QUIC should implement RDMA-like features or features that simply provide help with data placement on receivers. Possibilities include:

• A single additional integer giving the offset of a payload, serving only as a hint;
• Include references to separate streams in the same connection that contain opaque payloads, similar to RDMA chunks; this would presume that it is valid for some streams on a QUIC connection to carry traffic that is not in the form of an RPC message sequence.

Long-term there could be interest in supporting RDMA over QUIC. Direct data placement over TCP can already be accomplished today using MPA/DDP protocols (formerly known as iWARP; see ). Using a software iWARP implementation means no special hardware is required. If the MPA/DDP protocols themselves can be made to operate directly on QUIC transports, much of the need for a separate RPC-over-QUIC becomes moot. It would bring transport layer security to other RDMA-enabled protocols, such as RPC-over-RDMA .

Application Error Codes Receivers signal certain stream- or connection-level conditions by closing the affected stream or connection with a QUICv1 application error code, conveyed in a RESET_STREAM () or CONNECTION_CLOSE () frame. RPC-over-QUIC defines a set of named application error codes for this purpose; numeric values are TBD.

PROTOCOL_VIOLATION

Signaled by a peer that has detected a violation of this specification, such as an RPC message whose direction field does not match the receiver's Requester or Responder role.

SERVER_BUSY

Signaled by a Responder that is unable to accept additional RPC traffic at this time. Used as the backpressure indication for RPC-over-QUIC.

REQUEST_DROPPED

Signaled by a Responder that has discarded an RPC Call before generating an RPC Reply.

Additional code points are allocated through the registry described in .

QUIC Load Balancing The discussion in this section is preliminary. A complete specification of how load balancing applies to RPC-over-QUICv1 is left to a future revision of this document. The QUIC Load Balancing specification defines methods for encoding routing information in QUIC connection IDs, so that a load balancer can route packets to a particular backend server with minimal per-connection state, even when a client's network address changes due to NAT rebinding or migration. Because RPC-over-QUIC may carry multiple streams over one connection (see ), routing at the connection-ID layer keeps all streams of a given connection on the same backend. The connection ID length self-encoding feature of QUIC-LB assists hardware cryptographic offload devices that look up connection-specific keys. RPC-over-QUIC implementations MAY use QUIC-LB. A full specification of its use with RPC-over-QUIC is out of scope for this document. QUIC-LB requires no support from QUIC clients beyond conformance to .

RPC Authentication Flavors Streams in a QUIC connection may use different RPC authentication flavors. One stream might use RPC_AUTH_UNIX, while at the same time, another might use RPCSEC_GSS. GSS mutual (peer) authentication occurs only after a QUIC connection has been established. It is a separate process, and is unchanged by the use of QUIC. Additionally, authentication of RPCSEC_GSS users is unchanged by the use of QUIC. RPCSEC_GSS can optionally perform per-RPC integrity or confidentiality protection. When operating within a QUIC connection, these GSS services become largely redundant. An RPC implementation capable of concurrently using QUIC and RPCSEC_GSS MUST use Generic Security Service Application Program Interface (GSS-API) channel binding, as defined in , to determine when an underlying transport already provides a sufficient degree of confidentiality. RPC-over-QUIC implementations MUST provide the "tls-exporter" channel binding type, as defined in .

Implementation Status This section is to be removed before publishing as an RFC. This section records the status of known implementations of the protocol defined by this specification at the time of posting of this Internet-Draft, and is based on a proposal described in . The description of implementations in this section is intended to assist the IETF in its decision processes in progressing drafts to RFCs. Please note that the listing of any individual implementation here does not imply endorsement by the IETF. Furthermore, no effort has been spent to verify the information presented here that was supplied by IETF contributors. This is not intended as, and must not be construed to be, a catalog of available implementations or their features. Readers are advised to note that other implementations may exist. There are no known implementations of RPC-over-QUIC as described in this document.

Security Considerations RPC-over-QUIC inherits the transport-layer security properties of QUIC version 1, including always-on encryption, authentication, and integrity protection of QUIC packets, and the connection migration protections discussed in and . Implementers should be familiar with that material. Because QUIC integrates TLS into the transport handshake, the peer identities established during connection establishment apply to every stream and RPC message carried on that connection, as described in . The peer authentication, PKIX Extended Key Usage, and certificate-handling considerations in apply equally to RPC-over-QUIC. When RPCSEC_GSS is used concurrently with RPC-over-QUIC, the GSS confidentiality and integrity services are largely redundant with the protection supplied by the QUIC connection. To detect this condition, implementations use the GSS-API channel-binding mechanism , as required by . The "tls-exporter" binding type ties the GSS context to the specific TLS handshake of the underlying QUIC connection; a successful binding check confirms that the GSS context and the RPC traffic share the same protected channel. QUIC permits the use of 0-RTT data, which is replayable as described in . RPC procedures are generally not idempotent, so the same replay hazard that motivates the 0-RTT prohibition in applies to RPC-over-QUIC. RPC-over-QUIC implementations MUST NOT use 0-RTT data.

IANA Considerations RFC Editor: In the following subsections, please replace RFC-TBD with the RFC number assigned to this document. Furthermore, please remove this Editor's Note before this document is published.

Netids for RPC-over-QUIC Each new RPC transport is assigned one or more RPC "netid" strings. These strings are an rpcbind string naming the underlying transport protocol, appropriate message framing, and the format of service addresses and ports, among other things. This document requests that IANA allocate the following "Netid" registry strings in the "ONC RPC Netid" registry, as defined in : These netids MUST be used for any transport satisfying the requirements described in this document. The "quic" netid is to be used when IPv4 addressing is employed by the underlying transport, and "quic6" for IPv6 addressing. IANA should use this document (RFC-TBD) as the reference for the new entries.

ALPN Identifier for SunRPC on QUIC RPC-over-QUIC utilizes the same ALPN string as RPC-with-TLS does, as defined in : This document requests that a reference to (RFC-TBD) be added to the SunRPC protocol entry in the "TLS Application-Layer Protocol Negotiation (ALPN) Protocol IDs" registry.

RPC-over-QUIC Application Error Codes Registry This document requests that IANA establish a new registry titled "RPC-over-QUIC Application Error Codes". The allocation policy for this registry is to be specified by a future revision of this document. Each registry entry comprises a numeric code point, a symbolic name, and a reference to a stable specification defining the semantics of the code. Initial entries are:

Code	Name	Reference
TBD	PROTOCOL_VIOLATION	(RFC-TBD)
TBD	SERVER_BUSY	(RFC-TBD)
TBD	REQUEST_DROPPED	(RFC-TBD)

References Normative References This document describes the binding protocols used in conjunction with the ONC Remote Procedure Call (ONC RPC Version 2) protocols. [STANDARDS-TRACK] The concept of channel binding allows applications to establish that the two end-points of a secure channel at one network layer are the same as at a higher layer by binding authentication at the higher layer to the channel at the lower layer. This allows applications to delegate session protection to lower layers, which has various performance benefits. This document discusses and formalizes the concept of channel binding to secure channels. [STANDARDS-TRACK] This document describes the Open Network Computing (ONC) Remote Procedure Call (RPC) version 2 protocol as it is currently deployed and accepted. This document obsoletes RFC 1831. [STANDARDS-TRACK] This document lists IANA Considerations for Remote Procedure Call (RPC) Network Identifiers (netids) and RPC Universal Network Addresses (uaddrs). This document updates, but does not replace, RFC 1833. [STANDARDS-TRACK] This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm. This document describes how Transport Layer Security (TLS) is used to secure QUIC. This document defines a channel binding type, tls-exporter, that is compatible with TLS 1.3 in accordance with RFC 5056, "On the Use of Channel Bindings to Secure Channels". Furthermore, it updates the default channel binding to the new binding for versions of TLS greater than 1.2. This document updates RFCs 5801, 5802, 5929, and 7677. This document describes a mechanism that, through the use of opportunistic Transport Layer Security (TLS), enables encryption of Remote Procedure Call (RPC) transactions while they are in transit. The proposed mechanism interoperates with Open Network Computing (ONC) RPC implementations that do not support it. This document updates RFC 5531. n.d. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References This document specifies the Transmission Control Protocol (TCP). TCP is an important transport-layer protocol in the Internet protocol stack, and it has continuously evolved over decades of use and growth of the Internet. Over this time, a number of changes have been made to TCP as it was specified in RFC 793, though these have only been documented in a piecemeal fashion. This document collects and brings those changes together with the protocol specification from RFC 793. This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093, 6429, 6528, and 6691 that updated parts of RFC 793. It updates RFCs 1011 and 1122, and it should be considered as a replacement for the portions of those documents dealing with TCP requirements. It also updates RFC 5961 by adding a small clarification in reset handling while in the SYN-RECEIVED state. The TCP header control bits from RFC 793 have also been updated based on RFC 3168. This document defines a Remote Direct Memory Access Protocol (RDMAP) that operates over the Direct Data Placement Protocol (DDP protocol). RDMAP provides read and write services directly to applications and enables data to be transferred directly into Upper Layer Protocol (ULP) Buffers without intermediate data copies. It also enables a kernel bypass implementation. [STANDARDS-TRACK] This document describes a simple process that allows authors of Internet-Drafts to record the status of known implementations by including an Implementation Status section. This will allow reviewers and working groups to assign due consideration to documents that have the benefit of running code, which may serve as evidence of valuable experimentation and feedback that have made the implemented protocols more mature. This process is not mandatory. Authors of Internet-Drafts are encouraged to consider using the process for their documents, and working groups are invited to think about applying the process to all of their protocol specifications. This document obsoletes RFC 6982, advancing it to a Best Current Practice. This document specifies a protocol for conveying Remote Procedure Call (RPC) messages on physical transports capable of Remote Direct Memory Access (RDMA). This protocol is referred to as the RPC-over- RDMA version 1 protocol in this document. It requires no revision to application RPC protocols or the RPC protocol itself. This document obsoletes RFC 5666. This document describes the Network File System (NFS) version 4 minor version 1, including features retained from the base protocol (NFS version 4 minor version 0, which is specified in RFC 7530) and protocol extensions made subsequently. The later minor version has no dependencies on NFS version 4 minor version 0, and is considered a separate protocol. This document obsoletes RFC 5661. It substantially revises the treatment of features relating to multi-server namespace, superseding the description of those features appearing in RFC 5661. Google Microsoft Private Octopus Inc. QUIC address migration allows clients to change their IP address while maintaining connection state. To reduce the ability of an observer to link two IP addresses, clients and servers use new connection IDs when they communicate via different client addresses. This poses a problem for traditional "layer-4" load balancers that route packets via the IP address and port 4-tuple. This specification provides a standardized means of securely encoding routing information in the server's connection IDs so that a properly configured load balancer can route packets with migrated addresses correctly. As it proposes a structured connection ID format, it also provides a means of connection IDs self-encoding their length to aid some hardware offloads.

Acknowledgments The authors express their deepest appreciation for the contributions of J. Bruce Fields who was an original author of this document. In addition, we are indebted to Lars Eggert and the QUIC working group for the creation of the QUIC transport protocol. The editor is grateful to Bill Baker, Greg Marsden, Richard Scheffenegger, Martin Thomson, and Long Xin for their input and support. Special thanks to Area Director Gorry Fairhurst, NFSV4 Working Group Chairs Brian Pawlowski and Christopher Inacio, and NFSV4 Working Group Secretary Thomas Haynes for their guidance and oversight.

Open Issues This appendix is to be removed before publishing as an RFC. Open design questions identified during the development of this document are tracked as issues in . Each item below links the affected section to the GitHub issue that captures the question.

• : #7 - Stream as a control plane for RPC-over-QUIC
• : #8 - Should each stream carry only one RPC program/version combination?
• : #9 - Stream lifecycle: reconnection, resend semantics, and stream reuse
• : #10 - Server backpressure mechanism for RPC-over-QUIC
• : #11 - Why register a netid per IP address family?
• : #12 - Versioning of the RPC-over-QUIC ALPN identifier