]> Hammerspace

loghyr@gmail.com

Transport Network File System Version 4 Internet-Draft The Parallel Network File System (pNFS) allows for the metadata server to use CB_LAYOUTRECALL to recall a layout from a client by file id or file system id or all. It also allows the server to use CB_NOTIFY_DEVICEID to delete a deviceid. It does not provide a mechanism for the metadata server to recall all layouts that have a data file on a specific deviceid. This document presents an extension to RFC7862 to allow the server to recall layouts from clients based on deviceid. Note to Readers Discussion of this draft takes place on the NFSv4 working group mailing list (nfsv4@ietf.org), which is archived at . Working Group information can be found at .

Introduction In the Network File System version 4 (NFSv4) with a Parallel NFS (pNFS) metadata server (), there is no mechanism for the metadata server to recall layouts from the client when a particular deviceid (see Section 3.3.14 of ) either temporarily or permanently is no longer available. The Flex Files layout type () is a primary motivating use case. A Flex Files layout describes data spread across multiple data servers, each identified by a distinct deviceid, which may reside in separate power fault domains. One use case is when the deviceids in a layout are separated by power fault domains. Each layout might describe 3 different devices, each contained in a different power fault domain. In such a scenario, a single fault domain can have the power removed and not cause the loss of access to the data. However, client I/O will be impacted as the client still has to perform WRITEs (see Section 18.32 of ) to the unavailable device, send LAYOUTERRORs (see Section 15.6 of ) to inform the metadata server of NFS4ERR_NXIO (see Section 15.1.16.3 of ). If the metadata server had the means to recall layouts by deviceid, a lot of this unnecessary traffic could be eliminated. While the metadata server could use LAYOUTRECALL4_ALL to recall all of a client's layouts, that would evict layouts pointing at healthy devices unnecessarily. The deviceid-specific recall is surgical: only the layouts referencing the unavailable device are returned, leaving unaffected layouts in place. Similarly, while the metadata server could recall affected layouts one by one via LAYOUTRECALL4_FILE, that generates one RPC per file and the work can instead be offloaded to the client with a single recall. Besides the use case above, consider if the metadata server wants to set the NOTIFY4_DEVICEID_DELETE in the CB_NOTIFY_DEVICEID callback (see Section 20.12 of ). This flag cannot be set if a layout is outstanding for a deviceid. While the metadata server can revoke all such layouts, there is no way to know that the client has acknowledged that revocation and hence is still not doing I/O to other data files in the layout. The metadata server could fence those layouts as well (see Section 12.5.5 of ), but that can be an expensive operation. Using the process detailed in , the revisions in this document become an extension of NFSv4.2 . They are built on top of the external data representation (XDR) generated from .

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Client Capability Advertisement Before the server may send a CB_LAYOUTRECALL with LAYOUTRECALL4_DEVICEID, it MUST know that the client supports the new union arm. Per Section 6, a server MUST NOT send a new callback operation or new union arm to a client that has not indicated support for it. A client that supports LAYOUTRECALL4_DEVICEID signals this by setting a new flag in the eia_flags field of the EXCHANGE_ID operation (see Section 18.35 of ): The specific bit value 0x02000000 is a placeholder. EXCHGID4 flag bit values are assigned from the reserved flag space defined in Section 18.35 of and are coordinated across in-flight NFSv4 Working Group extensions. The final value will be confirmed during Working Group Last Call. A client that sets EXCHGID4_FLAG_SUPP_RECALL_DEVICEID in its EXCHANGE_ID request advertises that it supports handling CB_LAYOUTRECALL with a LAYOUTRECALL4_DEVICEID recall type. A server MUST NOT send a CB_LAYOUTRECALL with LAYOUTRECALL4_DEVICEID to a client that did not set EXCHGID4_FLAG_SUPP_RECALL_DEVICEID in its EXCHANGE_ID request. A server that does not wish to support this capability MAY ignore the flag. If the client does not set EXCHGID4_FLAG_SUPP_RECALL_DEVICEID, the server MAY fall back to recalling individual layouts via LAYOUTRECALL4_FILE (one CB_LAYOUTRECALL per layout file that references the unavailable deviceid), as described in Section 20.3 of . This is less efficient but correct, and preserves interoperability with clients that predate this extension.

Extension to CB_LAYOUTRECALL - Recall Layout from Client The original union layoutrecall4 (see Section 20.3.1 of ) is: The proposed extension is: Note that LAYOUT4RET_REC* constants are shared between layoutrecall_type4 (used in CB_LAYOUTRECALL) and layoutreturn_type4 (used in LAYOUTRETURN, see Section 18.44.1 of ). Adding LAYOUT4_RET_REC_DEVICEID = 4 therefore also extends layoutreturn_type4 with a LAYOUTRETURN4_DEVICEID value. A client that receives a LAYOUTRETURN with lor_recalltype set to LAYOUTRETURN4_DEVICEID and does not recognise it MUST return NFS4ERR_UNION_NOTSUPP per . The server MUST NOT send CB_LAYOUTRECALL with LAYOUTRECALL4_DEVICEID to a client that has not set EXCHGID4_FLAG_SUPP_RECALL_DEVICEID (see ). This satisfies the requirement in Section 6 of that a server establish client awareness before sending new callback extensions. The existing clora_iomode field in CB_LAYOUTRECALL4args (see Section 20.3.1 of ) applies normally: the client MUST return all layouts matching both the given deviceid and the given iomode. The server can determine that the client no longer has any layouts with the given deviceid and iomode once the client replies with NFS4ERR_NOMATCHING_LAYOUT.

Extraction of XDR This document contains the external data representation (XDR) description of the extension to CB_LAYOUTRECALL. The XDR description is presented in a manner that facilitates easy extraction into a ready-to-compile format. To extract the machine-readable XDR description, use the following shell script: #!/bin/sh grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??' ]]> For example, if the script is named 'extract.sh' and this document is named 'spec.txt', execute the following command: sh extract.sh < spec.txt > recalldevice.x ]]> This script removes leading blank spaces and the sentinel sequence '///' from each line. XDR descriptions with the sentinel sequence are embedded throughout the document. Note that the XDR code contained in this document depends on types from the NFSv4.2 nfs4_prot.x file (generated from ). This includes both nfs types that end with a 4, such as offset4, length4, etc., as well as more generic types such as uint32_t and uint64_t. The extracted XDR extends types that are already defined in rather than introducing new types. An implementer MUST integrate the extracted block into the base XDR by replacing the existing layoutrecall_type4 enum declaration and layoutrecall4 union declaration with the extended versions in this document, and by adding the LAYOUT4RET_REC_DEVICEID constant in the same file area as the existing LAYOUT4_RET_REC* constants. Appending the extracted block verbatim to the RFC 7863 XDR produces duplicate- type errors at compile time. The EXCHGID4_FLAG_SUPP_RECALL_DEVICEID constant is purely additive and is added to the EXCHGID4 flag constant block without replacing anything.

Security Considerations The extension introduced by this document inherits the Security Considerations of CB_LAYOUTRECALL in Section 20.3 of and of NFSv4.2 in . Three aspects of the deviceid-scoped recall warrant explicit mention.

Blast radius:

A single CB_LAYOUTRECALL with LAYOUTRECALL4_DEVICEID can cause a client to return many layouts in one callback exchange. This is not a new risk category -- LAYOUTRECALL4_FSID and LAYOUTRECALL4_ALL already allow wider-scoped recalls -- but administrators sizing callback-channel timeouts and client-side recall-handler resources SHOULD account for the possibility that a deviceid-scoped recall triggers per-layout cleanup work proportional to the number of layouts that reference the deviceid.

Cross-client scope:

In deployments where a single deviceid is shared across clients serving different tenants or applications, a deviceid-scoped recall is cross-client by construction: every client that holds a layout referencing the deviceid is recalled. This is the same property as LAYOUTRECALL4_FSID and does not introduce a new information-exposure surface. Deployments that prefer per-tenant failure-domain isolation MAY treat this extension as an operational benefit: by assigning distinct deviceids per tenant, deviceid-scoped recall becomes a per-tenant operation with no cross-tenant effect.

Capability negotiation as defence in depth:

The capability flag defined in ensures that a server cannot silently change the callback wire format for a client that did not opt in. A client that does not advertise EXCHGID4_FLAG_SUPP_RECALL_DEVICEID continues to receive only the layoutrecall_type4 values defined in . A server that sends LAYOUTRECALL4_DEVICEID to a client that did not advertise support is in violation of Section 6 of ; clients that encounter such a violation MUST return NFS4ERR_UNION_NOTSUPP and SHOULD log the server for operator attention.

IANA Considerations This document has no IANA actions.

References Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. This document describes the External Data Representation Standard (XDR) protocol as it is currently deployed and accepted. This document obsoletes RFC 1832. [STANDARDS-TRACK] This document describes NFS version 4 minor version 2; it describes the protocol extensions made from NFS version 4 minor version 1. Major extensions introduced in NFS version 4 minor version 2 include the following: Server-Side Copy, Application Input/Output (I/O) Advise, Space Reservations, Sparse Files, Application Data Blocks, and Labeled NFS. This document provides the External Data Representation (XDR) description for NFS version 4 minor version 2. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This document describes the rules relating to the extension of the NFSv4 family of protocols. It covers the creation of minor versions, the addition of optional features to existing minor versions, and the correction of flaws in features already published as Proposed Standards. The rules relating to the construction of minor versions and the interaction of minor version implementations that appear in this document supersede the minor versioning rules in RFC 5661 and other RFCs defining minor versions. This document describes the Network File System (NFS) version 4 minor version 1, including features retained from the base protocol (NFS version 4 minor version 0, which is specified in RFC 7530) and protocol extensions made subsequently. The later minor version has no dependencies on NFS version 4 minor version 0, and is considered a separate protocol. This document obsoletes RFC 5661. It substantially revises the treatment of features relating to multi-server namespace, superseding the description of those features appearing in RFC 5661. Informative References This paper describes the NFS version 3 protocol. This paper is provided so that people can write compatible implementations. This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Parallel NFS (pNFS) allows a separation between the metadata (onto a metadata server) and data (onto a storage device) for a file. The flexible file layout type is defined in this document as an extension to pNFS that allows the use of storage devices that require only a limited degree of interaction with the metadata server and use already-existing protocols. Client-side mirroring is also added to provide replication of files.

Acknowledgments Trond Myklebust and Paul Saab were involved in the initial requirements for this functionality. Brian Pawlowski and Gorry Fairhurst helped guide this process.