]> TCP RST Diagnostic Payload Orange
mohamed.boucadair@orange.com
Nokia
India kondtir@gmail.com
Tencent
kerneljasonxing@gmail.com
TCP Maintenance and Minor Extensions service diagnostic service function troubelshooting diagnostic serviceability root cause anomaly This document specifies an experimental diagnostic payload format returned in TCP RST segments. Such payloads are used to share with an endpoint the reasons for which a TCP connection has been reset. Sharing this information is meant to ease diagnostic and troubleshooting. This specification builds on provisions that are already present in RFC 9293 "Transmission Control Protocol (TCP)". As such, this document does not require any change to RFC 9293. Discussion Venues Discussion of this document takes place on the TCP Maintenance and Minor Extensions mailing list (tcpm@ietf.org), which is archived at . Source for this draft and an issue tracker can be found at .
Introduction A TCP connection can be reset by a peer for various reasons, e.g., received data does not correspond to an active connection. Also, a TCP connection can be reset by an on-path service function (e.g., Carrier Grade NAT (CGN) , NAT64 , or firewall) for several reasons. Typically, a Network Address Translator (NAT) function can generate an RST segment to notify an endpoint upon the expiry of the lifetime of the corresponding mapping entry or because an RST segment was received from a peer (). A TCP connection can also be closed by a user or an application at any time. However, the peer that receives an RST segment does not have any hint about the reason that led to terminating the connection. Likewise, the application that relies upon such a TCP connection may not easily identify the reason for the connection reset. Troubleshooting such events at the remote side of the connection that receives the RST segment may not be trivial. This document fills this void by specifying an experimental format of the diagnostic payload returned in an RST segment. This design is backward compatible with TCP as further clarified in . The generic procedure for processing an RST segment is specified in . Only the deviations from that procedure to insert and validate a diagnostic payload is provided in . This document specifies the format and the overall approach to ease maintaining the list of codes while allowing for adding new codes as needed in the future and accommodating any existing vendor-specific codes. An initial version of error codes is available in . However, the authoritative source to retrieve the full list of error codes is the IANA-maintained registry (). provides a set of examples to illustrate the use of TCP RST diagnostic payloads. provides an informative discussion of socket API considerations. Implementation and experimental validation are detailed in . Experiment goals are listed in .
Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document makes use of the terms defined in . SEG.LEN is defined in . SHLD-2 is defined in while MUST-12 is defined in . This document uses the following terms:
RST diagnostic payload:
The payload of an RST message that conveys diagnostic data.
RST with diagnostic payload:
An RST segment that includes diagnostic payload.
Reset reason information:
Refers to a reset reason code and a Private Enterprise Number (PEN). The PEN can be omitted if it is set to 0.
Experiment Description & Goals The main objective of this experiment is to have a common format of RST diagnostic payload that would be used as basis for consistent testing and evaluation in a variety of deployment contexts (Internet, data centers, application clients/servers provided and managed by the same entity, clients and servers software owned by distinct entities, etc.). Experiments reports are encouraged to share the main lessons learned in these experimentations. Specifically, the following items are of interest:
Delivery and on-path device interference:
Identify and share issues (or lack thereof) related to the delivery of RST with diagnostic payload.
CPU/load impact:
Assess CPU/load impact (or lack thereof) of handling RSTs, including when a mix of RSTs with and without diagnostic payload are sent by an endpoint.
Standard reset reasons:
Assess whether the list of code reasons () reflects observed reset cases.
Integration of socket API extensions:
Exercise the socket API extensions and identify any required adjustement ().
Operational guidance:
Strengthen the operational guidance for deploying RSTs with diagnostic payload.
RST Diagnostic Payload
Payload Format This section defines the message format to convey RST diagnostic payload. This format is designed to minimize the length of the payload. The format of the RST diagnostic payload is shown in .
Structure of the RST Diagnostic Payload
The RST diagnostic payload comprises a magic number that is used to unambiguously identify an RST payload that follows this specification. The magic number MUST be set to 0x33AA. The descriptions of other fields shown in are as follows:
reason-code:
This field takes a value from an available registry (IANA or vendor-specific).
Value 0 is reserved and MUST NOT be used.
The reason code is taken from the "TCP Failure Causes" registry if "pen" is set to 0.
It is RECOMMENDED that implementations support all codes defined in provided in .
If the "pen" is not set to 0, then the reason code refers to a registry of the entity identified by the "pen" parameter.
pen:
Includes a Private Enterprise Number (PEN) .
The reserved PEN value "0" is used to indicate that the reason code refers to the IANA-maintained registry ().
SEG.LEN MUST be 8 for an RST with diagnostic payload.
Behavior Malformed RST diagnostic payloads that include the magic numbers 0x33AA MUST be silently ignored by the receiver. RSTs that carry such malformed diagnostic payloads are handled like an RST without payload. A peer that receives a valid diagnostic payload may pass the reset reason information to the local application in addition to the information (MUST-12) described in . The reset reason information may also be logged locally, unless a local policy specifies otherwise. How the reset reason information is passed to an application and how it is stored locally is implementation-specific. Because new codes may be supported by a sender, receivers SHOULD NOT discard received RST diagnostic payloads with an unknown reason code unless configured otherwise. Vendor-specific registries may be maintained by applications. It is out of scope to describe how an implementation associates a PEN with a vendor-specific registry.
Some Examples depicts an example of an RST diagnostic payload that is generated to inform the peer that the TCP connection is reset because an ACK was received from that peer while the connection is still in the LISTEN state ().
Example of an RST Diagnostic Payload with Reason Code
An RST diagnostic payload may also be sent by an on-path service function. For example, the following diagnostic payload is returned by a NAT function upon expiry of the mapping entry to which the TCP connection is bound ().
Example of an RST Diagnostic Payload to Report Connection Timeout
illustrates an RST diagnostic payload that is returned by a peer that resets a TCP connection for a reason code 1234 (0x4D2) defined by a vendor with the private enterprise number 32473 (0x7ED9).
Example of an RST Diagnostic Payload to Report Vendor-Specific Reason Code
uses the Enterprise Number 32473 defined for documentation use .
Operational Considerations
Backward Compatibility Returning diagnostic data in an RST segment is consistent with the provision in for RST segments, especially:
TCP implementations SHOULD allow a received RST segment to include data (SHLD-2).
Also, this document does not change the conditions under which an RST segment is generated ().
Multiple RSTs Per , one or more RST segments can be sent to reset a connection. Sending more RST segments to reset a connection can be used to mitigate deployment contexts where some on-path devices may discard RSTs with payload data. Whether a TCP endpoint elects to send more than one RST with only a subset of them that include the diagnostic payload is implementation-specific for clients and policy-based for servers (see ).
Manageability TCP server implementations should support the following parameters:
  • A parameter to control the activation of RSTs with diagnostic payload.
  • A parameter to expose the set of reason codes that are supported by an implementation.
  • A parameter to control whether "empty" RSTs are also sent together with an RST with diagnostic payload.
  • A rate-limit of RSTs with diagnostic payload.
  • Counters to track sent/received RSTs with diagnostic payload.
  • Counters to track received invalid RSTs with diagnostic payload.
Maintenance As new reason codes may be added to the IANA registry, there is a risk that the codes that are supported by an implementation do not match the latest version in the registry. Deviations can be detected using the exposure parameter in . It is RECOMMENDED to proceed with regular software updates to align with the latest version in the registry.
Socket API Considerations (Informative) This section describes how the socket API can be extended to provide a way for an application to use the functionality described in this document. This section is informational only. The API described in this section can change in a non-backwards compatible way during the evolution of this document due to changed functionality or gained experience during the implementation.
Socket Options provides an overview of the IPPROTO_TCP-level socket options defined in this section. Socket Options
Option Name Data Type Set Get
TCP_RST_REASON_ENABLE uint32_t X  
TCP_RST_REASON_CODE struct tcp_rst_reason X X
Enable the Sending of the Diagnostic Payload (TCP_RST_REASON_ENABLE) Using setsockopt() with the IPPROTO_TCP-level socket option with the name TCP_RST_REASON_ENABLE enables or disables the sending of the diagnostic payload using a reason-code and pen. The option_value of type uint32_t specifies the pen in host byte order to use. When 0 is used (), the reason-codes from the registry specified in are used. When 0xffffffff is used (), the sending is disabled. The default is that the sending of a diagnostic payload is disabled. An implementation might not support the use of PENs different from zero and 0xffffffff.
Get or Set the Diagnostic Payload as Code (TCP_RST_REASON_CODE) Using getsockopt() with the IPPROTO_TCP-level socket option with the name TCP_RST_REASON_CODE allows the caller to retrieve the reason-code and the pen of the diagnostic payload in the received RST segment, which terminated the corresponding TCP connection. Using setsockopt() with this socket option allows the caller to provide reason-code and pen to be sent as part of the diagnostic payload when the application triggers the sending of a RST segment by using close(). In addition to using close() in combination with the SOL_SOCKET-level socket option with name SO_LINGER, the application can just provide the TCP_RR_RST_ON_CLOSE flag in trr_flags. This way the application can trigger the sending of a RST segment by calling setsockopt() once followed by close(). For accepted sockets, this socket option is inherited from the listening socket. The following structure is used as the option_value:
trr_flags:
This field is reported as 0 for getsockopt() calls. For setsockopt()
calls, the following flag can be used:
TCP_RR_RST_ON_CLOSE:
When this flag is set, calling close() triggers the sending of a RST segment similar to case, where the SOL_SOCKET-level socket option with name SO_LINGER is used to enable lingering with the linger time of 0. When this flag is cleared, the corresponding functionality is disabled.
trr_code:
The reason-code in host byte order to be interpreted in combination with the PEN provided in trr_pen. In case of trr_pen being zero, trr_code refers to a value in the registry defined in .
trr_pen:
The PEN in host byte order to is used in combination with the reason-code specified in trr_code. When this socket option is used with setsockopt(), it is an error to use zero as a value for trr_pen as long as trr_code is not zero.
When getsockopt() with this socket option is performed on a socket, which has not received a RST with a diagnostic payload containing a reason-code and pen, zero is provided as the trr_code and trr_pen. When setsockopt() with a trr_code and trr_pen of zero is performed, the special handling of RST segments sent during the ungraceful termination of the TCP connection is disabled.
Security Considerations discusses TCP-related security considerations. In particular, RST-specific attacks and their mitigations are discussed in . The presence of vendor-specific reason codes may be used to fingerprint hosts. Such a concern does not apply if the reason codes are taken from the IANA-maintained registry. Implementers are, thus, encouraged to register new codes within IANA instead of maintaining specific registries.
IANA Considerations
New Registry for TCP Failure Causes This document requests IANA to create a new registry entitled "TCP Failure Causes" under the "Transmission Control Protocol (TCP) Parameters" registry group . Values are taken from the 1-65535 range. The assignment policy for this registry is "Expert Review" (). See more guidance at . The registry is initially populated with the values listed in . Initial TCP Failure Causes
Value Description Specification (if available)
0 Reserved ThisDocument
1 Illegal option length
2 Desynchronized state
3 New data is received after CLOSE is called Sections 3.6.1 and 3.10.7.1 of
4 ABORT process
5 Unexpected ACK received by non-synchronized state connection
6 Unexpected SYN in the window
7 Unexpected security compartment
8 Malformed message ThisDocument
9 Not authorized ThisDocument
10 Resource exceeded ThisDocument
11 Network failure ThisDocument
12 Reset received from the peer ThisDocument
13 Destination unreachable ThisDocument
14 Connection timeout ThisDocument
15 Too much outstanding data
16 Unacceptable performance
17 Middlebox interference
Note that codes in the 8-14 range can be used by service functions (CGN, firewall, proxy, etc.).
  • Note to the RFC Editor: Please replace ThisDocument with the RFC number assigned to this document.
Guidelines for the Designated Experts Criteria that should be applied by the designated experts include determining whether the proposed registration duplicates existing entries and whether the registration description is clear and fits the purpose of this registry. The designated experts may approve registration once they checked that the new requested code is not covered by an existing code and if the provided reasoning to register the new code is acceptable. A registration request may supply a pointer to a specification where that code is defined. However, a registration may be accepted even if no permanent and readily available public specification is available. Registration requests are to be sent to rst-diag-review@ietf.org and are evaluated within a three-week review period on the advice of one or more designated experts. Within the review period, the designated experts will either approve or deny the registration request, communicating this decision to the review list and IANA. Denials should include an explanation and, if applicable, suggestions as to how to make the request successful.
References Normative References Transmission Control Protocol (TCP) Parameters Private Enterprise Numbers Transmission Control Protocol (TCP) This document specifies the Transmission Control Protocol (TCP). TCP is an important transport-layer protocol in the Internet protocol stack, and it has continuously evolved over decades of use and growth of the Internet. Over this time, a number of changes have been made to TCP as it was specified in RFC 793, though these have only been documented in a piecemeal fashion. This document collects and brings those changes together with the protocol specification from RFC 793. This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093, 6429, 6528, and 6691 that updated parts of RFC 793. It updates RFCs 1011 and 1122, and it should be considered as a replacement for the portions of those documents dealing with TCP requirements. It also updates RFC 5961 by adding a small clarification in reset handling while in the SYN-RECEIVED state. The TCP header control bits from RFC 793 have also been updated based on RFC 3168. Key words for use in RFCs to Indicate Requirement Levels In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Guidelines for Writing an IANA Considerations Section in RFCs Many protocols make use of points of extensibility that use constants to identify various protocol parameters. To ensure that the values in these fields do not have conflicting uses and to promote interoperability, their allocations are often coordinated by a central record keeper. For IETF protocols, that role is filled by the Internet Assigned Numbers Authority (IANA). To make assignments in a given registry prudently, guidance describing the conditions under which new values should be assigned, as well as when and how modifications to existing values can be made, is needed. This document defines a framework for the documentation of these guidelines by specification authors, in order to assure that the provided guidance for the IANA Considerations is clear and addresses the various issues that are likely in the operation of a registry. This is the third edition of this document; it obsoletes RFC 5226. TCP Extensions for Multipath Operation with Multiple Addresses TCP/IP communication is currently restricted to a single path per connection, yet multiple paths often exist between peers. The simultaneous use of these multiple paths for a TCP/IP session would improve resource usage within the network and thus improve user experience through higher throughput and improved resilience to network failure. Multipath TCP provides the ability to simultaneously use multiple paths between peers. This document presents a set of extensions to traditional TCP to support multipath operation. The protocol offers the same type of service to applications as TCP (i.e., a reliable bytestream), and it provides the components necessary to establish and use multiple TCP flows across potentially disjoint paths. This document specifies v1 of Multipath TCP, obsoleting v0 as specified in RFC 6824, through clarifications and modifications primarily driven by deployment experience. Informative References Common Requirements for Carrier-Grade NATs (CGNs) This document defines common requirements for Carrier-Grade NATs (CGNs). It updates RFC 4787. Stateful NAT64: Network Address and Protocol Translation from IPv6 Clients to IPv4 Servers This document describes stateful NAT64 translation, which allows IPv6-only clients to contact IPv4 servers using unicast UDP, TCP, or ICMP. One or more public IPv4 addresses assigned to a NAT64 translator are shared among several IPv6-only clients. When stateful NAT64 is used in conjunction with DNS64, no changes are usually required in the IPv6 client or the IPv4 server. Updates to Network Address Translation (NAT) Behavioral Requirements This document clarifies and updates several requirements of RFCs 4787, 5382, and 5508 based on operational and development experience. The focus of this document is Network Address Translation from IPv4 to IPv4 (NAT44). This document updates RFCs 4787, 5382, and 5508. Enterprise Number for Documentation Use This document describes an Enterprise Number (also known as SMI Network Management Private Enterprise Code) for use in documentation. This memo provides information for the Internet community. Registration Procedures for Private Enterprise Numbers (PENs) This document describes how Private Enterprise Numbers (PENs) are registered by IANA. It shows how to request a new PEN and how to modify a current PEN. It also gives a brief overview of PEN uses. The Constrained Application Protocol (CoAP) The Constrained Application Protocol (CoAP) is a specialized web transfer protocol for use with constrained nodes and constrained (e.g., low-power, lossy) networks. The nodes often have 8-bit microcontrollers with small amounts of ROM and RAM, while constrained networks such as IPv6 over Low-Power Wireless Personal Area Networks (6LoWPANs) often have high packet error rates and a typical throughput of 10s of kbit/s. The protocol is designed for machine- to-machine (M2M) applications such as smart energy and building automation. CoAP provides a request/response interaction model between application endpoints, supports built-in discovery of services and resources, and includes key concepts of the Web such as URIs and Internet media types. CoAP is designed to easily interface with HTTP for integration with the Web while meeting specialized requirements such as multicast support, very low overhead, and simplicity for constrained environments.
Implementation and Experimental Validation in Linux Questions and concerns have been raised regarding whether RST with payload affects the normal termination of flows across different software platforms, operating systems, middleboxes, etc. Even though explicitly allows this behavior, a full implementation is needed to widely verify if unexpected cases can happen in the real world. The overall design in Linux is to pre-allocate a large enough zeroed buffer, put a reset reason code in the first byte and sent it out to verify whether the RST with payload can be possibly declined by any equipment in between two sides and the other side successfully parses the RST with payload.
Implementation The following implementation is accomplished on top of Linux 6.16:
Payload Attachment:
Allocate a 1000-byte data payload attached to all generated RST packets.
Reason Code Encoding:
The first byte of the payload is used to store a predefined reset reason code that is listed in include/net/rstreason.h file, while the remainder of the payload is zero-padded. The reason code is generated by the existing mechanism called TCP reset reasons.
Handling of Reset Types:
The implementation distinguishes between the two primary reset scenarios in tcp_send_active_reset() and tcp_v4_send_reset() respectively:
  • For an Active Reset, initiated proactively by the local system, the payload is placed in the linear area of the socket buffer (sk_buff).
  • For a Passive Reset, sent in response to an unexpected or invalid incoming packet, the payload is stored in the non-linear (paged) area of the sk_buff.
Complete patch is shown in .
Complete Patch sk_mark = 0; ctl_sk->sk_priority = 0; } + memcpy(buffer, (char *)&rep, arg.iov[0].iov_len); + /* put rst reason into the first byte in payload */ + buffer[arg.iov[0].iov_len] = reason; + arg.iov[0].iov_len += PAYLOAD_LEN; ip_send_unicast_reply(ctl_sk, sk, skb, &TCP_SKB_CB(skb)->header.h4.opt, ip_hdr(skb)->saddr, ip_hdr(skb)->daddr, diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index b616776e3354..c07dd009a0de 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -3628,12 +3628,14 @@ void tcp_send_fin(struct sock *sk) void tcp_send_active_reset(struct sock *sk, gfp_t priority, enum sk_rst_reason reason) { + u32 len = MAX_TCP_HEADER + PAYLOAD_LEN; + char payload[PAYLOAD_LEN]; struct sk_buff *skb; TCP_INC_STATS(sock_net(sk), TCP_MIB_OUTRSTS); /* NOTE: No TCP options attached and we never retransmit this. */ - skb = alloc_skb(MAX_TCP_HEADER, priority); + skb = alloc_skb(len, priority); if (!skb) { NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTFAILED); return; @@ -3641,8 +3643,13 @@ void tcp_send_active_reset(struct sock *sk, gfp_t priority, /* Reserve space for headers and prepare control bits. */ skb_reserve(skb, MAX_TCP_HEADER); + skb_put(skb, PAYLOAD_LEN); tcp_init_nondata_skb(skb, tcp_acceptable_seq(sk), TCPHDR_ACK | TCPHDR_RST); + memset(payload, 0, PAYLOAD_LEN); + payload[0] = reason; + skb_store_bits(skb, 0, payload, PAYLOAD_LEN); + TCP_SKB_CB(skb)->end_seq += PAYLOAD_LEN; tcp_mstamp_refresh(tcp_sk(sk)); /* Send it off. */ if (tcp_transmit_skb(sk, skb, 0, priority)) ]]>
Experimental Validation To ensure a thorough evaluation, a multi-layered experimental methodology was designed, progressing from basic functional checks to complex, real-world compatibility and stability tests. The whole implementation has been deployed in Tencent's production environment for almost six months.
Functional Verification The basic functionality test is using iperf or iperf3 to construct a normal termination senario. The tcpdump tool with -X option effectively helps to show the [RST+] flag and the 1000-byte payload, confirming that the kernel correctly generated and transmitted the augmented RST packets. Two servers, designated as Client A and Server B. The test is conducted as following:
  1. Start the iperf3 server on Server B (iperf3 -s).
  2. Initiate a connection from Client A to Server B (iperf3 -c [IP_of_B]).
  3. After the connection is established, one of the iperf3 processes is terminated using the kill command, triggering the kernel to send an RST packet.
  4. Simultaneously, tcpdump is run on either host to capture the reset packet using the filter: 'tcp[tcpflags] & tcp-rst != 0' -X -nn -vv -S.
Compatibility Verification
Hardwares and Kernels:
Tests were conducted on various Linux distributions (e.g., Ubuntu or CentOS) with different kernel versions. The physical hosts were equipped with a range of network interface cards (NICs), including Intel i40e, ixgbe, and Mellanox mlx5.
Virtualization:
The mechanism was tested in a virtualized environment where the VM used a virtio_net driver and the host employed DPDK to redirect packets in the host.
Middleboxes:
Tests were performed with Layer 4 (L4) and Layer 7 (L7) gateways placed between the client and server to verify correct packet parsing and forwarding.
Wide Area Network (WAN):
The setup was tested over long-haul international links to simulate complex conditions, including China-to-Singapore (RTT > 30ms) and China-to-Germany (RTT > 200ms).
In conclusion, across all complex environment tests, the RST packets with payloads were successfully received by the peer. No instances of packets being dropped or mishandled by intermediate middleboxes, gateways, or diverse hardware and software configurations were observed.
Acknowledgments The "diagnostic payload" name is inspired by that was cited by Carsten Bormann in the tcpm mailing list. Thanks to Jon Shallow, Neal Cardwell, Lars Eggert, Eric Dumazet, Rick Jones, Yoshifumi Nishida, Li Jinghui, and Gleb Smirnoff for the review and comments.
Contributors
tuexen@fh-muenster.de