]> PipeStream AI

kristian.rickert@pipestream.ai

Applications and Real-Time Individual Submission quic streaming recursive distributed-processing scatter-gather consistency This document specifies PipeStream, a recursive scatter-gather streaming protocol for hierarchical task decomposition and distributed processing over QUIC transport. PipeStream enables the decomposition (scattering) of complex, arbitrary workloads into constituent sub-tasks, their transmission across distributed processing nodes, and subsequent rehydration (gathering) at destination endpoints. While application-layer protocols like gRPC provide stream multiplexing, PipeStream embeds a hierarchical state machine directly into the protocol. It employs a dual-stream architecture consisting of a data stream for payload transmission and a control stream for tracking completion status and maintaining distributed consistency. PipeStream defines a generic 2-bit Data Layer field for entity representation, leaving the concrete payload semantics to application profiles. To ensure consistency across parallel processing pipelines, the protocol implements checkpoint blocking, guaranteeing that all constituent parts of a decomposed workload are successfully processed before rehydration operations commence. Status information for this document may be found at . Discussion of this document takes place on the Individual Group mailing list ().

Introduction

Problem Statement Distributed processing pipelines face significant challenges when handling large, complex inputs that require multiple stages of transformation, analysis, and enrichment. Traditional batch processing approaches require entire inputs to be loaded into memory, processed sequentially, and transmitted in their entirety between processing stages. This methodology introduces substantial latency, excessive memory consumption, and poor utilization of distributed computing resources. Modern distributed processing workflows increasingly demand the ability to: Process inputs incrementally as data becomes available Distribute processing load across heterogeneous worker nodes Maintain consistency guarantees across parallel processing paths Handle inputs of arbitrary size without memory constraints Support recursive decomposition where constituent parts may themselves be decomposed Scale from single inputs to collections of millions of entries Current approaches based on batch processing and store-and-forward architectures are inefficient for large inputs and fail to exploit the inherent parallelism available in distributed processing environments. Furthermore, existing streaming protocols do not provide the consistency semantics required for hierarchical processing where the integrity of the rehydrated output depends on the successful processing of all constituent parts. These pipelines also increasingly span organizational and vendor boundaries: a processing stage operated by one party must interoperate with stages operated by others. Today that interoperability is achieved through per-vendor SDKs and bespoke coordination logic, because no standardized wire protocol expresses the decomposition, tracking, and reassembly semantics these systems share. PipeStream specifies that wire protocol.

The Limits of Existing Transport and RPC Mechanisms Existing multiplexed RPC frameworks (for example, gRPC over HTTP/2 or HTTP/3) and raw transport APIs (for example, WebTransport) excel at point-to-point data streaming. However, they lack native mechanisms for distributed consistency when workloads are hierarchically decomposed. In a standard streaming RPC model, an application must manually track state if a primary task spawns tens of thousands of distributed sub-tasks. PipeStream solves this by pushing the scatter-gather state machine down to the protocol level. Through the use of cryptographic Scope Digests (Section 6.3) and Barrier Frames (Section 6.4), PipeStream provides native barrier synchronization semantics, ensuring a parent stream cannot logically terminate until all scattered child entities across distributed nodes have reached a terminal status. Broker-based pipeline architectures (distributed commit logs and message queues) address a related problem by decoupling stages through intermediary persistence, but they externalize exactly the semantics PipeStream makes native: determining whether a hierarchically decomposed job has fully completed requires an external coordination store, and end-to-end integrity verification is left to the application. Appendix B compares PipeStream to these and other existing protocols in detail.

Applicability PipeStream is a domain-neutral recursive streaming protocol. Its design has been validated against distributed document processing, but it is equally applicable to any workload that can be modeled as hierarchical decomposition of a root entity into sub-entities, parallel processing of those sub-entities, and deterministic reassembly of results. Example domains include: Distributed document processing and content enrichment pipelines Hierarchical video transcoding and rendering (scene decomposition) Federated computation pipelines such as distributed machine learning Genomic sequencing and assembly workflows PipeStream uniquely bridges true streaming and request-response processing models. Layer 0 streaming is optimal when processing nodes have low-latency interconnects and can process entities incrementally. Layer 2 yield/resume and claim checks handle workflows that require external service calls, rate limiting, human approval gates, or cross-session resumption. This dual nature allows a single protocol to serve both real-time streaming pipelines and complex multi-stage workflows with heterogeneous latency characteristics.

PipeStream Overview PipeStream addresses these challenges by defining a streaming protocol that enables incremental processing with strong consistency guarantees. The protocol is built upon QUIC transport, leveraging its native support for multiplexed streams, low-latency connection establishment, and reliable delivery semantics. The fundamental innovation of PipeStream is its treatment of inputs as recursive compositions of entities. A root entity MAY be decomposed into multiple sub-entities, each of which MAY itself be further decomposed, creating a tree structure of processing tasks. This recursive decomposition enables fine-grained parallelism while the protocol's control stream mechanism ensures that all branches of the decomposition tree are tracked and synchronized. PipeStream employs a dual-stream design: Data Stream: Carries entity payloads through the processing pipeline. Entities flow through this stream with minimal buffering, enabling low-latency incremental processing. Control Stream: Carries control information tracking the status of entity decomposition and rehydration. The control stream ensures that all parts of a dehydrated entity are accounted for before rehydration proceeds.

Design Philosophy PipeStream implements a recursive scatter-gather pattern over QUIC streams. An input entity is "dehydrated" (scattered) at the source into constituent sub-entities, these sub-entities are transmitted and processed in parallel across distributed pipeline stages, and finally the results are "rehydrated" (gathered) at the destination to reconstitute the complete processed output. The checkpoint blocking mechanism (Section 9.3) provides barrier synchronization semantics analogous to the barrier pattern in parallel computing. This approach provides several advantages: Incremental Processing: Processing nodes MAY begin work on early entities before the complete input has been transmitted. Parallelism: Independent entities MAY be processed concurrently across multiple worker nodes. Memory Efficiency: No single node is required to hold the complete input in memory. Fault Isolation: Failures in processing individual entities can be detected, reported, and potentially retried without affecting other entities. Consistency: The checkpoint blocking mechanism ensures that rehydration operations proceed only when all constituent parts have been successfully processed.

Protocol Layering PipeStream is organized into three protocol layers to accommodate varying deployment requirements: Protocol Layer Name Description Layer 0 Core Basic streaming, dehydrate/rehydrate, checkpoint Layer 1 Recursive Hierarchical scopes, digest propagation, barriers Layer 2 Resilience Yield/resume, claim checks, completion policies Implementations MUST support Layer 0. Support for Layers 1 and 2 is OPTIONAL and negotiated during connection establishment.

Scope This document specifies the PipeStream protocol including message formats, state machines, error handling, and the interaction between data and control streams. The document defines the transport-level layer field and recursive processing semantics but leaves concrete payload meaning to application profile specifications.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Protocol Entities

Entity

The fundamental unit of data flowing through a PipeStream pipeline. An Entity represents either a complete input or a constituent part of a decomposed input. Each Entity possesses a unique identifier within its processing scope and carries payload data tagged with a transport-level layer value. Entities are immutable once created; transformations produce new Entities rather than modifying existing ones. An input enters the pipeline as a single root Entity and MAY be decomposed into multiple Entities during processing; it is considered complete when its root Entity (or the rehydrated result of its decomposition) exits the pipeline.

Scope

A hierarchical namespace for Entity IDs. Each scope maintains its own Entity ID space, cursor, and Assembly Manifest. Scopes enable root entities to contain parts, and parts to contain sub-tasks, each with independent ID management. (Protocol Layer 1)

Dehydration and Rehydration

Scatter-Gather

The distributed processing pattern implemented by PipeStream. A single input is "scattered" (dehydrated) into multiple parts for parallel processing, and the results are "gathered" (rehydrated) back into a single output. PipeStream extends classical scatter-gather with recursive nesting: any scattered part may itself be scattered further.

Dehydrate (Scatter)

The operation of decomposing an Entity into multiple constituent Entities for parallel or distributed processing. When an Entity is dehydrated, the originating node MUST create an Assembly Manifest entry recording the identifiers of all resulting sub-entities. The dehydration operation is recursive; a sub-entity produced by dehydration MAY itself be dehydrated, creating a tree of decomposition. Dehydration transitions data from a solid state (a single complete unit) to a fluid state (multiple in-flight entities).

Rehydrate (Gather)

The operation of reassembling multiple Entities back into a single composite Entity. A rehydrate operation MUST NOT proceed until all constituent Entities listed in the corresponding Assembly Manifest entry have been received and processed (or handled according to the Completion Policy). Rehydration transitions data from a fluid state back to a solid state.

Solid State

An Entity that exists as a complete, stored unit or as a single root Entity entering or exiting a pipeline. Contrast with "fluid state".

Fluid State

An input that has been decomposed into multiple in-flight Entities being processed in parallel across distributed nodes. An input is in the fluid state between dehydration and rehydration. Contrast with "solid state".

Consistency Mechanisms

Checkpoint

A synchronization point in the processing pipeline where all in-flight Entities MUST reach a consistent state before processing may continue. A checkpoint is considered "satisfied" when all Assembly Manifest entries created before the checkpoint have been resolved.

Barrier

A synchronization point scoped to a specific subtree. Unlike checkpoints which are global, barriers block only entities dependent on a specific parent's descendants. (Protocol Layer 1)

Control Stream

The control stream that tracks Entity completion status throughout the processing pipeline. The Control Stream is transmitted on a dedicated QUIC stream parallel to the data streams.

Assembly Manifest

A data structure within the Control Stream that tracks the relationship between a composite Entity and its constituent sub-entities produced by dehydration.

Cursor

A pointer to the lowest unresolved Entity ID within a scope. Entity IDs behind the cursor are considered resolved and MAY be recycled. The cursor enables efficient ID space management without global coordination.

Resilience Mechanisms (Protocol Layer 2)

Yield

A temporary pause in Entity processing, typically due to external dependencies (API calls, rate limiting, human approval). A yielded Entity carries a continuation token enabling resumption without reprocessing.

Claim Check

A detached reference to a deferred Entity that can be queried or resumed independently, potentially in a different session. Claim checks enable asynchronous processing patterns and retry queues.

Completion Policy

A configuration specifying how to handle partial failures during dehydration. Policies include STRICT (all must succeed), LENIENT (continue with partial results), BEST_EFFORT (complete with whatever succeeds), and QUORUM (require minimum success ratio).

Data Representation

Data Layer

A transport-level 2-bit field carried in the Entity Header that identifies the payload representation. The concrete meaning of each layer value is defined by an application profile rather than by the core protocol.

Additional Terms

Pipeline

A configured sequence of processing stages through which Entities flow.

Processor

A node in the mesh that performs operations on entities (e.g., transformation, dehydration, or rehydration).

Sink

A terminal stage in a pipeline where rehydrated entities or terminal entities are persisted or delivered to an external system.

Stage

A single processing step within a Pipeline.

Scope Digest

A cryptographic summary (Merkle root) of all Entity statuses within a completed scope, propagated to parent scopes for efficient verification. (Protocol Layer 1)

Protocol Layers PipeStream defines three protocol layers that build upon each other. This layered approach allows simple deployments to use only the core protocol while complex deployments can leverage advanced features.

Layer 0: Core Protocol Layer 0 provides the fundamental streaming capabilities: Unified Control Frame (UCF) header (1-octet type + 4-octet length prefix) Status frame (16-octet base bit-packed frame) Entity frame (header + payload) Status codes: PENDING, PROCESSING, COMPLETE, FAILED, CHECKPOINT Assembly Manifest for parent-child tracking Cursor-based Entity ID recycling Single-level dehydrate/rehydrate Checkpoint blocking All implementations MUST support Layer 0.

Layer 1: Recursive Extension Layer 1 adds hierarchical processing capabilities: Scoped Entity ID namespaces (root -> component -> sub-task -> leaf) Explicit Depth tracking in status frames SCOPE_DIGEST for Merkle-based subtree completion BARRIER for subtree-scoped synchronization Nested dehydration with depth tracking Layer 1 is OPTIONAL. Implementations advertise Layer 1 support during capability negotiation.

Layer 2: Resilience Extension Layer 2 adds fault tolerance and async processing: YIELDED status with continuation tokens DEFERRED status with claim checks RETRYING, SKIPPED, ABANDONED statuses Completion policies (STRICT, LENIENT, BEST_EFFORT, QUORUM) Claim check extensions and deferred processing tokens Stopping point validation Layer 2 is OPTIONAL and requires Layer 1. Implementations advertise Layer 2 support during capability negotiation.

Capability Negotiation PipeStream uses a two-tier negotiation model. The ALPN identifier (Section 11.1) identifies the base PipeStream transport mapping, while the capabilities structure handles dynamic resource limits and optional layer support that may vary based on endpoint configuration or real-time load. During CONNECT, endpoints exchange supported capabilities using the capabilities structure. This message MUST be encoded using the default CBOR format for both the client's initiation and the server's response (Section 3.4.2).

Peers negotiate down to common capabilities. If Layer 2 is requested but Layer 1 is not supported, Layer 2 MUST be disabled. The layer0-core field MUST be true; an endpoint that receives a Capabilities message with layer0-core set to false MUST close the connection with PIPESTREAM_LAYER_UNSUPPORTED (0x0C). The max-entities-per-scope and max-window-size values are bounded by the Entity ID space constraints defined in Section 9.1; an endpoint MUST NOT advertise values exceeding those bounds.

Version Negotiation PipeStream protocol versioning is carried in two places: the ALPN identifier and the Ver field in the STATUS frame (Section 6.2.1). The ALPN identifier pipestream/1 identifies the major protocol version and the QUIC transport mapping defined in this document. The 4-bit Ver field in STATUS frames carries the value 0x1 for this specification. A future major version of PipeStream (e.g., pipestream/2) would register a new ALPN identifier. QUIC's native ALPN negotiation during the TLS handshake provides version selection: if a client offers both pipestream/2 and pipestream/1 and the server supports only pipestream/1, the TLS handshake selects pipestream/1 without additional round trips. This mechanism is consistent with the versioning approach used by HTTP/3 and DNS over QUIC . Minor, backward-compatible extensions (such as new optional capability fields or new status codes within the reserved ranges) do not require a new ALPN identifier. Such extensions are negotiated through the capabilities structure or the IANA registries defined in Section 11.

Serialization Format Negotiation The serialization_format field determines the encoding used for all variable-length control messages (frame types 0x80-0xFF) and entity headers. This document defines a single serialization format: CBOR , value 0 in the PipeStream Serialization Formats registry (Section 11.5). All PipeStream implementations MUST support CBOR. The serialization-format capability field and its registry exist to permit future specifications to define additional formats; any such specification MUST normatively define the encoding of every serialized message in this document. Negotiation proceeds as follows: CBOR is the default and MUST be supported by all endpoints. Each peer advertises its preferred serialization-format in its Capabilities message. If both peers advertise the same format and both support it, that format is used. If preferences differ, if either peer omits the field, or if a peer advertises a value the other does not recognize, the peers MUST fall back to CBOR . The initial Capabilities exchange on a new connection MUST use the default CBOR format for both the client's initiation and the server's response. The negotiated serialization format and resource limits take effect immediately following the successful completion of this initial Capabilities exchange (one request and one response). If a peer cannot decode the initial Capabilities exchange, it MUST close the connection with PIPESTREAM_FRAME_ERROR (0x0D).

Protocol Overview This section provides a high-level overview of the PipeStream protocol architecture, design principles, and operational model.

Design Goals This section is descriptive; the normative requirements that realize these goals appear in Sections 5 through 9.

True Streaming Processing PipeStream enables entities to be transmitted and processed incrementally as they become available. The framing and status mechanisms operate on individual entities rather than complete inputs, so a sender never needs to buffer a complete input before initiating transmission.

Recursive Decomposition The protocol supports recursive decomposition of entities, wherein a single input entity may produce zero, one, or many output entities.

Checkpoint Consistency PipeStream provides checkpoint blocking semantics (Section 9.3) to maintain processing consistency across distributed workers.

Control and Data Plane Separation The protocol maintains strict separation between the control plane (the Control Stream) and the data plane (Entity Streams).

QUIC Foundation PipeStream is defined directly over QUIC to leverage: Native stream multiplexing without head-of-line blocking Built-in flow control at both connection and stream levels TLS 1.3 security by default Connection migration capabilities

Multi-Layer Data Representation The protocol carries a 2-bit data layer field supporting four distinct data representation layers whose concrete semantics are defined by application profiles: Layer Conventional Role Description 0 Raw input Binary or originating payload bytes 1 Enriched intermediate Annotated or partially processed payload 2 Structured result Normalized or extracted output 3 Extension Application-specific payload semantics

Architecture Summary PipeStream uses a dual-plane architecture within a single QUIC connection. The endpoint that initiates the QUIC connection is the client. Either endpoint MAY originate entities (Section 5.2); in many deployments the client acts as producer and the server as consumer, but the protocol does not require this asymmetry after connection establishment. Stream Type Plane Content Stream 0 Bidirectional (client-initiated) Control STATUS, SCOPE_DIGEST, BARRIER, GOAWAY, CAPABILITIES, CHECKPOINT Entity Streams Unidirectional (either endpoint) Data Entity frames (Header + Payload)

Connection Lifecycle A PipeStream connection follows this lifecycle: Establishment: Client initiates QUIC connection with ALPN identifier "pipestream/1" Capability Exchange: Client and server exchange supported protocol layers and limits Control Stream Initialization: Client opens Stream 0 as bidirectional Control Stream Entity Streaming: Entities are transmitted per Sections 5 and 6 Termination: Connection closes via GOAWAY-initiated graceful shutdown (Section 6.5) or QUIC CONNECTION_CLOSE

QUIC Stream Mapping PipeStream leverages the native multiplexing capabilities of QUIC to provide a clean separation between control coordination and data transmission.

Control Stream (Stream 0) The Control Stream provides the control plane for PipeStream operations.

Stream Identification The Control Stream MUST use QUIC Stream ID 0, which per RFC 9000 is a bidirectional, client-initiated stream.

Usage Rules The Control Stream MUST be opened immediately upon connection establishment. Capability negotiation (Section 3.4) MUST occur on Stream 0 before any Entity Streams are opened. Stream 0 MUST NOT carry entity payload data. Implementations SHOULD assign the Control Stream a high priority to ensure timely delivery of status updates. An implementation MAY choose a different priority policy when operating in constrained environments where QUIC stream scheduling overhead must be minimized.

Flow Control Considerations The Control Stream carries small, bit-packed control frames. STATUS frame payloads are 16 octets base (21 octets on wire including the 5-octet UCF header). Implementations MUST ensure adequate flow control credits: The initial MAX_STREAM_DATA for Stream 0 SHOULD be at least 8192 octets. A lower value is permissible for extremely constrained devices but risks stalling status delivery. Implementations SHOULD NOT block Entity Stream transmission due to Control Stream flow control exhaustion. In rare cases where strict ordering between control and data planes is required by the application, an implementation MAY temporarily pause entity transmission until control stream credits are replenished.

Heartbeat Mechanism QUIC already provides native transport liveness signals (for example, PING and idle timeout handling). Implementations SHOULD rely on those transport mechanisms for connection liveness. PipeStream heartbeat frames are OPTIONAL and are intended for application-level responsiveness checks (for example, detecting stalled processing logic even when the transport remains healthy). When used, an endpoint sends a STATUS frame with all fields set to their heartbeat values: Field Value Description Type 0x50 (STATUS)   Length 16 STATUS payload length with no cursor or extension Stat 0x0 (UNSPECIFIED) Heartbeat signal Entity ID 0xFFFFFFFF CONNECTION_LEVEL Scope ID 0x00000000 Root scope Reserved 0x00000000 MUST be zero The KEEPALIVE_TIMEOUT defaults to 30 seconds. Endpoints MAY negotiate a different value by including a keepalive-timeout-ms field (in milliseconds) in the capabilities exchange (Section 3.4); the effective timeout is the minimum of the two peers' advertised values. When no status updates have been transmitted for KEEPALIVE_TIMEOUT, an endpoint MAY send a heartbeat frame. If no data is received on Stream 0 for 3 * KEEPALIVE_TIMEOUT, the endpoint SHOULD first apply transport-native liveness policy (e.g., QUIC PING); it MAY close the connection with PIPESTREAM_IDLE_TIMEOUT (0x02) when application-level inactivity policy requires it.

Transport Session vs. Application Session Context The session-id segment of the pipestream URI scheme (Section 11.6) identifies application context for detached or resumable resources (for example, Layer 2 yield/claim-check flows). PipeStream Layer 0 streaming semantics do not depend on this URI scheme.

Interaction with QUIC Flow Control and Congestion Control PipeStream relies on the flow control and congestion control mechanisms provided by QUIC and does not define its own transport-layer congestion control. QUIC provides flow control at both the stream level (MAX_STREAM_DATA) and the connection level (MAX_DATA). PipeStream's cursor-based backpressure (Section 9.1) operates at the application layer and is complementary to QUIC flow control: QUIC flow control limits the number of bytes in flight on any given stream or connection. PipeStream backpressure limits the number of entities in flight (i.e., the number of Entity IDs between cursor and last_assigned). When the QUIC connection-level flow control window is exhausted, new Entity Streams cannot transmit data regardless of whether PipeStream's entity window has capacity. Conversely, when PipeStream's entity window is full, no new Entity Streams are opened even if QUIC flow control credits are available. Implementations MUST respect both limits. An implementation SHOULD monitor QUIC-level flow control credit availability and avoid opening new Entity Streams when connection-level credits are below the expected entity size, to prevent head-of-line blocking across streams sharing the connection budget.

Entity Streams Entity Streams carry entity payload data using the Entity Frame format defined in Section 6.8.

Stream Identification and Direction Each Entity Stream is a QUIC unidirectional stream. Either endpoint MAY open Entity Streams once the initial Capabilities exchange (Section 3.4) has completed. Per , client-initiated unidirectional streams carry Stream IDs 2, 6, 10, and so on, and server-initiated unidirectional streams carry Stream IDs 3, 7, 11, and so on. Endpoints MUST NOT open bidirectional streams other than Stream 0. An endpoint that receives a bidirectional stream other than Stream 0 MUST treat this as a connection error of type PIPESTREAM_FRAME_ERROR (0x0D).

One Entity per Stream Each Entity Stream carries exactly one Entity Frame (Section 6.8): a Header Length prefix, a serialized EntityHeader, and the payload octets. The following rules apply: The sender MUST close the stream (QUIC FIN) immediately after the final payload octet. The end of the stream delimits the payload. When the EntityHeader includes a payload-length field, the receiver MUST verify that the number of payload octets received before the FIN equals that value. A mismatch MUST be treated as a stream error of type PIPESTREAM_ENTITY_INVALID (0x05). A sender that abandons transmission of an entity SHOULD abort the stream with RESET_STREAM carrying an appropriate PipeStream error code; the receiver MUST discard any partial payload data (see Section 5.5). A receiver that no longer requires an entity MAY send STOP_SENDING on the corresponding Entity Stream. Additional data received on an Entity Stream after a complete Entity Frame MUST be treated as a stream error of type PIPESTREAM_FRAME_ERROR (0x0D).

Prohibition of 0-RTT Early Data QUIC permits applications to send data in 0-RTT early data before the TLS handshake completes; such data is replayable by an attacker. PipeStream capability negotiation establishes per-connection state whose replay could alter negotiated limits or serialization formats. Endpoints MUST NOT send PipeStream frames in 0-RTT early data, and a server MUST NOT process PipeStream frames received in early data before the QUIC handshake is confirmed. A future extension may define 0-RTT semantics along with the required replay protections.

Performance Considerations

Entity Granularity and Stream Overhead PipeStream's "one entity per stream" model (Section 5.2) provides clean multiplexing and flow control but introduces per-stream overhead (QUIC STREAM frame headers and internal stack state). Small Payloads: For workloads consisting of many small entities (e.g., <1 KiB), the per-stream overhead may become significant. Implementations SHOULD avoid excessive fragmentation and prefer coarser entity granularity where possible. Aggregation: Application profiles MAY define mechanisms for bundling multiple small logical units into a single PipeStream entity to reduce transport-layer overhead. Stream Limits: Senders MUST respect the peer's QUIC initial_max_streams_uni and MAX_STREAMS limits. High-frequency entity producers SHOULD monitor stream credit availability and adjust dehydration rates accordingly to avoid stalling the pipeline.

Control Stream Priority Implementations SHOULD assign the Control Stream (Stream 0) a higher priority than Entity Streams. Timely delivery of STATUS and SCOPE_DIGEST frames is critical for advancing the cursor and releasing backpressure (Section 9.1).

Transport Error Mapping PipeStream error signaling on Stream 0 and QUIC transport signals are complementary. Endpoints SHOULD bridge them so peers receive both transport-level and protocol-level context. An implementation MAY omit bridging when operating as a simple pass-through proxy that does not inspect entity status. If an Entity Stream is aborted with RESET_STREAM or STOP_SENDING, the endpoint SHOULD emit a corresponding status (FAILED when terminal, ABANDONED, or policy-driven equivalent) for that entity on Stream 0. The endpoint MAY omit the status frame only if the connection itself is being closed immediately. If PipeStream determines a terminal entity error first (for example, checksum failure or invalid frame), the endpoint SHOULD abort the affected Entity Stream with an appropriate QUIC error and emit the corresponding PipeStream status/error context on Stream 0. Aborting the stream MAY be deferred if the entity payload is still needed for diagnostic purposes. If Stream 0 is reset or becomes unusable, endpoints SHOULD treat this as a control-plane failure and close the connection with PIPESTREAM_CONTROL_RESET (0x03). An endpoint that can recover control-plane state through an application-layer mechanism MAY attempt reconnection before closing. On QUIC connection termination (CONNECTION_CLOSE), entities without a previously observed terminal status MUST be handled by local failure policy, typically by marking them FAILED or ABANDONED.

Frame Formats This section defines the wire formats for PipeStream frames. All multi-octet integer fields are encoded in network byte order (big-endian). Forward Compatibility: All Reserved and Flags fields defined in this section MUST be set to zero when sent and MUST be ignored by receivers. This convention enables future specifications to assign meaning to currently-reserved bits without breaking deployed implementations. Receivers that encounter non-zero values in reserved fields MUST NOT treat this as an error.

Control Stream Framing (Stream 0) To ensure forward compatibility and consistent parsing, all messages on the Control Stream use a Unified Control Frame (UCF) structure with an explicit length prefix.

UCF Header Every message on Stream 0 MUST begin with a 1-octet Frame Type and a 4-octet Length field. Field Type Description Type 1 octet Frame Type identifier Length 4 octets Total length of the frame payload (excluding Type/Length) Payload variable Frame-specific data This structure allows any implementation to skip an unknown frame type by reading its length and discarding the payload. For frame types with a fixed or computable payload layout defined by this specification, the Length field MUST equal the length implied by that layout (for STATUS frames, the base payload plus any cursor update and extension; see Section 6.2.2). A receiver that detects a Length value inconsistent with the frame's type-specific layout MUST close the connection with PIPESTREAM_FRAME_ERROR (0x0D). Although the Length field can express values up to 2^32 - 1, implementations SHOULD enforce a locally configured maximum control frame size and MAY treat frames exceeding it as a connection error of type PIPESTREAM_FRAME_ERROR (0x0D). See also the incremental allocation requirement in Section 10.3. For compactness, the fixed-frame diagrams below show the frame payload that follows the common UCF header. On the wire, each Stream 0 frame is encoded as Type (1 octet) | Length (4 octets) | Payload.

Frame Types The following frame types are defined by this document. All frames use the common UCF header: Type Name Layer Payload Format 0x50 STATUS 0 Fixed 16-octet payload, with optional cursor and extension payloads 0x54 SCOPE_DIGEST 1 Fixed 72-octet payload 0x55 BARRIER 1 Fixed 12-octet payload 0x56 GOAWAY 0 Fixed 8-octet payload 0x80 CAPABILITIES 0 Serialized message payload (negotiated format) 0x81 CHECKPOINT 0 Serialized message payload (negotiated format)

Status Frames (Layer 0)

Status Frame Format (0x50) The Status Frame payload reports lifecycle transitions for entities. The payload is 128-bit aligned for efficient parsing on 64-bit architectures.

Ver (4 bits):

Protocol version. MUST be set to 0x1 for this specification. Receivers that encounter an unsupported version MUST close the connection with PIPESTREAM_LAYER_UNSUPPORTED (0x0C).

Stat (4 bits):

Status code (see Section 6.2.2).

E (1 bit):

Extended frame flag. If set, an Extension Header (Section 6.6.1) MUST follow the base frame (and any cursor update).

C (1 bit):

Cursor update flag. A 4-octet cursor value follows (Section 6.2.4).

D (3 bits):

Explicit scope nesting depth (0-7). 0=Root. Layer 1.

Flags (19 bits):

Reserved for future use. MUST be zero when sent and MUST be ignored by receivers.

Entity ID (32 bits):

Unsigned integer identifying the entity.

Scope ID (32 bits):

Identifier for the scope to which this entity belongs. Expanding to 32 bits ensures uniqueness across high-frequency workloads.

Reserved (32 bits):

Reserved for future use. MUST be zero when sent and MUST be ignored by receivers.

Status Codes Value Name Layer Description 0x0 UNSPECIFIED - Default / heartbeat signal 0x1 PENDING 0 Entity announced, not yet transmitting 0x2 PROCESSING 0 Entity transmission in progress 0x3 COMPLETE 0 Entity successfully processed 0x4 FAILED 0 Entity processing failed 0x5 CHECKPOINT 0 Synchronization barrier 0x6 DEHYDRATING 0 Dehydrating into children 0x7 REHYDRATING 0 Rehydrating children 0x8 YIELDED 2 Paused with continuation token 0x9 DEFERRED 2 Detached with claim check 0xA RETRYING 2 Retry in progress 0xB SKIPPED 2 Intentionally skipped 0xC ABANDONED 2 Timed out The base STATUS payload is 16 octets (21 octets on the wire including the UCF header). When C=1, a 4-octet cursor value follows (20-octet payload, 25 octets on the wire). When E=1, an Extension Header follows all other STATUS fields.

Entity Status State Machine The following table defines the complete set of valid status transitions. A receiver that observes a transition not listed in this table MUST treat the status frame as a protocol error and close the connection with PIPESTREAM_ENTITY_INVALID (0x05) as the QUIC Application Error Code. From State Valid Transitions (To) PENDING PROCESSING, DEHYDRATING, FAILED, SKIPPED, ABANDONED PROCESSING COMPLETE, FAILED, DEHYDRATING, CHECKPOINT, YIELDED, DEFERRED, ABANDONED DEHYDRATING REHYDRATING, FAILED, ABANDONED REHYDRATING COMPLETE, FAILED, ABANDONED CHECKPOINT PROCESSING YIELDED PROCESSING, FAILED, DEFERRED, ABANDONED DEFERRED PROCESSING, FAILED, SKIPPED, ABANDONED FAILED RETRYING, ABANDONED RETRYING PROCESSING, FAILED, ABANDONED COMPLETE (terminal -- no transitions) SKIPPED (terminal -- no transitions) ABANDONED (terminal -- no transitions) Notes: PENDING is the implicit initial state for every entity upon ID assignment. The FAILED -> RETRYING transition is valid only when the entity's completion policy permits retries (max-retries > 0) and the retry count has not been exhausted. If retries are not permitted or are exhausted, FAILED MUST be treated as a terminal state. Layer 2 states (YIELDED, DEFERRED, RETRYING, SKIPPED, ABANDONED) MUST NOT appear when Layer 2 has not been negotiated. A receiver operating at Layer 0 or Layer 1 that observes a Layer 2 status code MUST treat it as PIPESTREAM_LAYER_UNSUPPORTED (0x0C). The UNSPECIFIED (0x0) status is used only for heartbeat frames (Section 5.1.4) and connection-level signals. It is not a valid entity lifecycle state and MUST NOT appear in transitions for entity IDs other than 0xFFFFFFFF.

Cursor Update Extension When C=1, a 4-octet cursor update follows the status frame:

New Cursor Value (32 bits):

The numeric value of the new cursor. Entities with IDs lower than this value (modulo circular ID rules) are considered resolved and their IDs MAY be recycled.

Status Reporting Direction STATUS frames flow in both directions on Stream 0. For each entity, responsibility for status reporting is divided between the originating endpoint (the endpoint that assigned the Entity ID and transmits the entity) and the processing endpoint (the endpoint that receives the corresponding Entity Stream): The originating endpoint MAY announce an entity with a PENDING status frame before opening its Entity Stream. Once the Entity Stream has been opened, the processing endpoint is authoritative for the entity's lifecycle and emits the STATUS frames for subsequent transitions (PROCESSING, DEHYDRATING, REHYDRATING, COMPLETE, FAILED, and the Layer 2 statuses). The originating endpoint applies terminal statuses (typically FAILED or ABANDONED) on its own authority only as part of transport error handling (Section 5.5) or local failure policy, for example when the Entity Stream is reset or the connection is lost before a terminal report arrives. The originating endpoint advances its cursor based on observed terminal statuses and announces cursor advancement using the C flag (Section 6.2.4). If an endpoint receives a STATUS frame for an entity from a peer that is not authoritative for that transition under these rules, the frame MUST be validated against the state machine of Section 6.2.3; conflicting reports are resolved in favor of the processing endpoint.

Scope Digest Frame (0x54) When Protocol Layer 1 is negotiated, a scope completion is summarized:

Flags (8 bits):

Reserved for future use. MUST be zero when sent and MUST be ignored by receivers.

Scope ID (32 bits):

Identifier of the scope being summarized.

Entities Processed (64 bits):

The total number of entities that were processed within the scope.

Entities Succeeded (64 bits):

The number of entities that reached a terminal success state.

Entities Failed (64 bits):

The number of entities that reached a terminal failure state.

Entities Deferred (64 bits):

The number of entities that were deferred via claim checks.

Merkle Root (256 bits):

The SHA-256 Merkle root covering all entity statuses in the scope (see Section 9.5).

The SCOPE_DIGEST payload is 72 octets (77 octets on the wire including the UCF header). The Scope ID MUST match the 32-bit identifier defined in Section 6.2.1.

Barrier Frame (0x55)

S (1 bit):

Status (0 = waiting, 1 = released).

Reserved (31 bits):

Reserved for future use. MUST be zero when sent and MUST be ignored by receivers.

Scope ID (32 bits):

Identifier for the scope to which this barrier applies.

Parent Entity ID (32 bits):

The identifier of the parent entity whose sub-tree is blocked by this barrier.

The BARRIER payload is 12 octets (17 octets on the wire including the UCF header).

GOAWAY Frame (0x56) The GOAWAY frame signals that the sender will not accept new entities beyond a specified Entity ID. It enables graceful shutdown: in-flight entities with IDs at or below the Last Entity ID are processed to completion, while the peer refrains from opening new Entity Streams.

Reserved (32 bits):

Reserved for future use. MUST be zero when sent and MUST be ignored by receivers.

Last Entity ID (32 bits):

The highest Entity ID that the sender will process. Entities with IDs greater than this value (per the circular ordering defined in Section 9.1) MUST NOT be sent by the peer after receiving this frame.

The GOAWAY payload is 8 octets (13 octets on the wire including the UCF header).

Graceful Shutdown Procedure An endpoint wishing to shut down sends a GOAWAY frame on Stream 0 with Last Entity ID set to the highest entity it is willing to process. Upon receiving GOAWAY, the peer MUST NOT open new Entity Streams for entities with IDs above Last Entity ID. Entity Streams already open for IDs at or below Last Entity ID continue to completion. Both peers continue processing status updates on Stream 0 until all in-flight entities reach terminal state. Once all entities are resolved, either peer MAY close the QUIC connection with PIPESTREAM_NO_ERROR (0x00). If an endpoint receives an entity with an ID above the Last Entity ID after sending GOAWAY, it MUST reject it with PIPESTREAM_ENTITY_INVALID (0x05). An endpoint MAY send multiple GOAWAY frames to progressively lower the Last Entity ID. The Last Entity ID MUST NOT increase across successive GOAWAY frames within the same connection, where "increase" is evaluated using the circular ordering function is_before defined in Section 9.3.

Yield and Claim Check Extensions (Layer 2) When E=1 in a status frame, an Extension Header MUST follow.

Extension Header

Extension Length (32 bits):

The total length of the extension data that follows this header, in octets. This allows receivers that do not support specific status extensions to skip the extension data and continue parsing the control stream.

If E=1 is set for a Status code that does not define an extension layout in this specification (or a negotiated extension), the receiver MUST use the Extension Length to skip the data. If the Extension Length is zero, extends beyond the end of the frame as delimited by the UCF Length field, or is missing, the receiver MUST treat the frame as malformed and close the connection with PIPESTREAM_FRAME_ERROR (0x0D).

Yield Extension (Stat = 0x8)

Yield Reason (8 bits):

The reason for yielding (see Section 6.6.2.1).

Token Length (24 bits):

The length of the Yield Token in bytes (maximum 16,777,215).

Yield Token (variable):

The opaque continuation state.

For a Yield Extension, the Extension Length (Section 6.6.1) MUST equal 4 + Token Length. A mismatch MUST be treated as PIPESTREAM_FRAME_ERROR (0x0D).

Yield Reason Codes Value Name Description 0x1 EXTERNAL_CALL Waiting on external service 0x2 RATE_LIMITED Voluntary throttle 0x3 AWAITING_SIBLING Waiting for specific sibling 0x4 AWAITING_APPROVAL Human/workflow gate 0x5 RESOURCE_BUSY Semaphore/lock 0x0, 0x06-0xFF Reserved Reserved for future use

Claim Check Extension (Stat = 0x9)

Claim Check ID (64 bits):

A cryptographically secure random identifier for the claim.

Expiry Timestamp (64 bits):

Unix epoch timestamp in microseconds when the claim expires.

Serialized Message Frames (0x80-0xFF) These frame types use the common UCF header defined in Section 6.1. The payload body is encoded using the serialization format negotiated during capability exchange (Section 3.4.2). If no format was negotiated, CBOR is the default. Type Message Name Reference 0x80 Capabilities Section 3.4 0x81 Checkpoint Section 9.3

Entity Frames Entity frames carry the actual entity payload data on Entity Streams.

Entity Frame Structure

Header Length (4 octets):

The length of the serialized EntityHeader in bytes.

Header (serialized):

The EntityHeader message encoded in the negotiated serialization format (see Section 6.8.2).

Payload (variable):

The raw entity data. The payload extends to the end of the Entity Stream (QUIC FIN); see Section 5.2. When the payload-length field is present in the EntityHeader, it MUST equal the number of payload octets.

Message Schema (CDDL) Normative definitions for serialized PipeStream messages use CDDL notation; Appendix C consolidates the complete schema.

Entity Header

tstr }, ? chunk-info: chunk-info, ? completion-policy: completion-policy, ; Layer 2 } ]]>

Chunk Info

Yield and Deferral

Support Types

Checksum Algorithm PipeStream uses SHA-256 for payload integrity verification. The checksum MUST be exactly 32 octets.

Chunked Transfer An entity whose payload is too large to transmit conveniently as a single frame MAY be split into chunks. Each chunk is transmitted as its own Entity Frame on its own Entity Stream, subject to the following rules: Every chunk of an entity MUST carry the same entity-id, parent-id (if present), scope-id (if present), and layer values, and MUST include a chunk-info structure. The payload-length and checksum fields in a chunk's EntityHeader, when present, cover only that chunk's payload octets (see Section 10.2 for whole-entity integrity requirements). chunk-offset is the octet offset of the chunk's first payload octet within the complete entity payload. total-chunks MUST be identical across all chunks of an entity, and the chunk-index values MUST cover the range 0 to total-chunks - 1 with no duplicates. Chunks MAY be transmitted concurrently on separate Entity Streams and MAY arrive in any order. The receiver reassembles the payload by chunk-offset. The entity payload is complete when all total-chunks chunks have been received and the reassembled ranges are contiguous and non-overlapping. Overlapping or duplicated ranges MUST be treated as PIPESTREAM_ENTITY_INVALID (0x05). Lifecycle status for a chunked entity is reported once for the entity as a whole (Section 6.2), not per chunk.

Entity Payload Model This section describes the payload model visible to the core protocol. PipeStream distinguishes between the wire-level Entity Header carried on Entity Streams and any application-level envelope that may be embedded within the payload bytes.

Entity Header The normative wire-level Entity Header is defined in Section 6.8.2. It provides transport-level identification and routing metadata, including entity-id, parent-id, scope-id, layer, payload-length, and optional integrity metadata. The core protocol interprets these fields for stream processing, flow control, and recursive coordination, but it does not define application-specific payload structure.

Data Layers PipeStream defines a 2-bit Data Layer field (values 0-3) in the Entity Header that identifies the payload encoding class. The semantics of each layer value are defined by Application Profile specifications. The core protocol does not assign meaning to specific layer values; it ensures that the layer field is faithfully transmitted and that receivers can route entities to appropriate handlers based on layer. Application Profiles SHOULD define at most four data layers. Common patterns include: Layer Conventional Meaning Purpose 0 Raw or binary payload Unprocessed input 1 Annotated or enriched payload Intermediate processing artifacts 2 Structured or normalized payload Final extracted or transformed output 3 Application-specific extension Domain extension point This layering enables pipeline stages to progressively transform entities from raw input to structured output while maintaining type distinction through the layer field.

Application-Level Envelope The Entity Header (Section 6.8.2) provides transport-level identification (entity-id, scope-id, and layer). Applications that require additional identification, such as a stable input identifier that persists across pipeline runs, multi-tenancy context, or domain-specific metadata, SHOULD define an application-level envelope message carried within the entity payload. If an application-level envelope carries its own entity identifier, that identifier MUST match the entity-id in the enclosing Entity Header. The definition of application-level envelopes is outside the scope of this specification. See for an example companion profile that defines such an envelope for document-processing workloads.

Common Storage References PipeStream defines a common storage reference type for entities that refer to externally stored data. Application Profiles are not required to use this type but SHOULD prefer it over ad hoc storage reference formats to improve interoperability between pipeline stages from different implementations.

tstr }, ; Provider-specific attributes ? encryption: encryption-metadata, } encryption-metadata = { algorithm: tstr, ; "AES-256-GCM", "AES-256-CBC" ? key-provider: tstr, ; "aws-kms", "azure-keyvault", ; "gcp-kms", "vault" ? key-id: tstr, ; Key ARN/URI/ID ? wrapped-key: bstr, ; Client-side encrypted DEK ? iv: bstr, ; Initialization vector ? context: { * tstr => tstr }, ; Encryption context } ]]>

Protocol Operations This section defines the protocol-level operations that PipeStream endpoints perform during a session. These operations describe the phases of a PipeStream session lifecycle, from connection establishment through entity processing to terminal consumption.

Overview A PipeStream session proceeds through four sequential actions:

Phase Action Cardinality Description 1 CONNECT 1:1 Session establishment and capability negotiation 2 PARSE 1:N Decomposition: dehydrate a root entity into sub-entities 3 PROCESS 1:1 or N:1 Transform, rehydrate, aggregate, or pass through entities (parallel) 4 SINK N:1 Terminal consumption by an application-defined sink

CONNECT Action The CONNECT action establishes the session with capability negotiation.

ALPN Identifier ALPN Protocol ID: pipestream/1

Capability Exchange Immediately after QUIC handshake, peers exchange Capabilities messages on Stream 0. The Capabilities exchange includes serialization format negotiation (Section 3.4.2). The agreed-upon format applies to all subsequent variable-length serialized messages on Stream 0 and to all entity headers on Entity Streams.

PARSE Action The PARSE action performs decomposition by dehydrating an input entity into one or more sub-entities. When Layer 2 is negotiated, the sender MAY attach a completion policy that governs how partial success, timeouts, or retries are handled during recursive processing.

PROCESS Action Mode Description TRANSFORM 1:1 entity transformation REHYDRATE N:1 merge of siblings from dehydration AGGREGATE N:1 with reduction function PASSTHROUGH Metadata-only modification

SINK Action The SINK action represents terminal consumption of processed entities. Sink implementations are application-specific and are defined by Application Profile specifications. Common sink patterns include persistent storage, search engine indexing, event notification, and downstream service delivery.

Rehydration Semantics

Entity ID Lifecycle and Cursor Entity IDs are managed using a cursor-based circular recycling scheme within the 32-bit ID space. All circular arithmetic in this document is performed modulo 0xFFFFFFFD; this modulus excludes the reserved values at the top of the 32-bit space from circulation. The following Entity ID values are reserved and MUST NOT be assigned to entities: Entity ID Name Purpose 0x00000000 NULL_ENTITY Reserved; skipped during assignment 0xFFFFFFFD-0xFFFFFFFE Reserved Outside the circular ID space 0xFFFFFFFF CONNECTION_LEVEL Connection-scoped signals such as heartbeats (Section 5.1.4) Assignable Entity IDs therefore range from 0x00000001 to 0xFFFFFFFC, yielding 4,294,967,292 usable identifiers per scope. The ID space is divided into three logical regions relative to the current cursor and last_assigned pointers: Region ID Range Description Recyclable IDs behind cursor Resolved entities; IDs may be reused In-flight cursor to last_assigned Active entities (PENDING, PROCESSING, etc.) Free Beyond last_assigned Available for new entity assignment The window size is computed as (last_assigned - cursor) mod 0xFFFFFFFD. If window_size >= max_window, the sender MUST apply backpressure and stop assigning new IDs until the cursor advances. The max_window value MUST NOT exceed 2,147,483,646 (the largest value strictly less than 0xFFFFFFFD / 2). This bound guarantees that the circular comparison function is_before (Section 9.3) is unambiguous for any pair of in-flight Entity IDs. The default max-window-size advertised in the capabilities exchange (Section 3.4) is 2,147,483,646. Rules: new_id = (last_assigned + 1) % 0xFFFFFFFD If new_id == 0, new_id = 1 (skip reserved NULL_ENTITY) If (new_id - cursor) % 0xFFFFFFFD >= max_window -> STOP, apply backpressure On reaching a terminal state (COMPLETE, SKIPPED, ABANDONED, or FAILED with no remaining retries): mark resolved; if entity_id == cursor, advance cursor past all contiguous resolved IDs IDs behind cursor are implicitly recyclable An entity in the FAILED state that may still transition to RETRYING (see the state transition table in Section 6.2) MUST NOT be marked resolved. Only when retries are exhausted or no completion policy permits retries does FAILED become terminal for cursor purposes.

Assembly Manifest The Assembly Manifest is a local data structure maintained by each endpoint to track the parent-child relationships created during dehydration. It is not transmitted on the wire; rather, each endpoint constructs its own manifest from the parent-id fields in received EntityHeaders and from status updates observed on the Control Stream. The CDDL below defines the logical structure of each entry; implementations MAY use any internal representation that preserves the required semantics. Each Assembly Manifest entry tracks:

Checkpoint Blocking A checkpoint is satisfied when: All entities in the checkpoint scope with IDs less than checkpoint_entity_id (considering circular wrap) have reached terminal state. All Assembly Manifest entries within the checkpoint scope have been resolved. All nested checkpoints within the checkpoint scope have been satisfied. CheckpointFrame (Section 6.7 / Appendix C) carries both:

checkpoint_id: an opaque identifier for logging and correlation. checkpoint_entity_id: the numeric ordering key used for barrier evaluation. Implementations MUST use checkpoint_entity_id (not checkpoint_id) when evaluating Condition 1. For circular comparison in Condition 1, implementations MUST use the same modulo ordering as cursor management. Define MAX = 0xFFFFFFFD and: is_before(a, b) = ((b - a + MAX) % MAX) < (MAX / 2) An entity ID a is considered "less than checkpoint_entity_id b" iff is_before(a, b) is true.

Scope ID Allocation (Layer 1) When Layer 1 is negotiated, Scope IDs are 32-bit unsigned integers assigned by the endpoint that initiates the dehydration. The allocation scheme is as follows: Scope ID 0 is the root scope and MUST NOT be used for child scopes. The dehydrating endpoint assigns a unique Scope ID to each new child scope created during dehydration. The Scope ID MUST be unique within the connection for the lifetime of that scope (i.e., until the scope's SCOPE_DIGEST frame has been sent and all entities in the scope have reached terminal state). Scope IDs MAY be allocated sequentially or randomly; the protocol does not require any particular ordering. Sequential allocation is RECOMMENDED for simplicity and debuggability. Once a scope has been closed (its SCOPE_DIGEST has been sent and all entities within the scope have reached terminal state), the Scope ID MAY be reused for a new scope. Implementations MUST ensure that no in-flight status frames reference a recycled Scope ID before reuse.

Scope Digest Propagation (Layer 1) When a scope completes, the endpoint MUST compute a Scope Digest and propagate it to the parent scope via a SCOPE_DIGEST frame (Section 6.3). The Merkle root in the Scope Digest is computed as follows: For each entity in the scope, ordered by ascending numeric Entity ID value, construct a 5-octet leaf value by concatenating: The 4-octet big-endian Entity ID. A 1-octet status field where the lower 4 bits contain the entity's terminal Stat code (Section 6.2.2) and the upper 4 bits are zero. Compute each leaf hash as SHA-256(0x00 || leaf), where leaf is the 5-octet value from step 1. Build a binary Merkle tree by repeatedly hashing pairs of sibling nodes: SHA-256(0x01 || left || right). If the number of nodes at any level is odd, the last node is promoted to the next level without hashing. The root of this tree is the merkle_root value in the SCOPE_DIGEST frame. The 0x00 and 0x01 domain-separation prefixes distinguish leaf hashes from interior-node hashes, preventing second-preimage attacks in which an interior node is presented as a leaf (or vice versa). This construction follows the approach used for Merkle Tree Hashes in Certificate Transparency. This construction is deterministic: any two implementations processing the same set of entity statuses MUST produce the same Merkle root. Because each Entity ID contributes exactly one leaf, an implementation MUST NOT recycle an Entity ID within a scope whose SCOPE_DIGEST has not yet been computed. Cursor-based recycling (Section 9.1) already guarantees this when scopes complete before the ID space wraps; implementations whose scopes approach the ID-space capacity MUST close and digest the scope before reusing any of its Entity IDs.

Rehydration Readiness Tracking Implementations MUST track Assembly Manifest resolution order using a local data structure that can efficiently identify the next parent entity eligible for rehydration as child statuses arrive out of order. The specific algorithm and internal representation are implementation choices and are outside the scope of this specification.

Stopping Point Validation (Layer 2) When yielding or deferring, include validation:

Security Considerations

Transport Security PipeStream inherits security from QUIC and TLS 1.3 . All connections MUST use TLS 1.3 or later. Implementations MUST NOT provide mechanisms to disable encryption. PipeStream frames MUST NOT be sent or processed in 0-RTT early data (Section 5.3), which removes the replay exposure that early data would otherwise introduce for capability negotiation and status frames.

Entity Payload Integrity Each Entity SHOULD include a SHA-256 checksum in its EntityHeader (the checksum field defined in Section 6.8.2). The checksum is OPTIONAL in the wire format to accommodate zero-length entities, streamed entities whose final length is unknown at header-emission time, and scenarios where application-layer integrity mechanisms provide equivalent guarantees. When a checksum is present, it MUST be exactly 32 octets containing the SHA-256 digest computed over the raw payload bytes (the octet sequence following the EntityHeader on the Entity Stream). The checksum does not cover the EntityHeader itself. For chunked entities (where chunk-info is present in the EntityHeader), each chunk MAY carry its own per-chunk checksum. The checksum in the first chunk's EntityHeader, if present, MUST cover only that chunk's payload bytes. An implementation that requires whole-entity integrity verification MUST either compute a rolling digest across all chunks or require the sender to transmit a final summary entity containing the whole-payload checksum. To support true streaming of large entities, implementations MAY begin processing an entity payload before the complete payload has been received and verified. However, the final rehydration or terminal SINK operation MUST NOT be committed until the complete payload checksum has been verified. If a checksum verification fails, the implementation MUST: Reject the entity with PIPESTREAM_INTEGRITY_ERROR (0x04). Discard any partial results or temporary state associated with the entity. Propagate the failure according to the Completion Policy (Section 8.3). Implementations that require immediate consistency SHOULD buffer the entire entity and verify the checksum before initiating processing.

Algorithm Agility This specification mandates SHA-256 as the sole checksum algorithm for both payload integrity (this section) and Merkle tree construction (Section 9.5). SHA-256 is well-studied and widely deployed; however, future developments may necessitate migration to a different algorithm. PipeStream supports algorithm migration through the capability negotiation mechanism (Section 3.4). A future specification MAY define additional fields in the capabilities structure to advertise supported checksum algorithms, following the general principles outlined in . Until such negotiation is defined, all implementations MUST use SHA-256 when producing or verifying checksums. An implementation that receives a checksum of a length other than 32 octets MUST reject the entity with PIPESTREAM_INTEGRITY_ERROR (0x04). The checksum field in the EntityHeader is typed as bstr .size 32 in the CDDL schema (Appendix C). A future algorithm negotiation extension would need to update this constraint, the SCOPE_DIGEST Merkle root size, and the corresponding IANA registry entries.

Resource Exhaustion Limit Default Description Max scope depth 7 Prevents recursive bombs (8 levels: 0-7) Max entities per scope 4,294,967,292 Memory bounds (Section 9.1) Max window size 2,147,483,646 Max in-flight entities (Section 9.1) Checkpoint timeout 30s Prevents stuck state Claim check expiry 86400s Garbage collection Implementations MUST enforce all resource limits listed above. Exceeding any limit MUST result in the corresponding error code (see Section 11.4). Implementations SHOULD allow operators to configure stricter limits than the defaults shown here. To prevent memory-exhaustion attacks, implementations MUST NOT pre-allocate memory for variable-length payloads based solely on the 32-bit Length field in the UCF header (Section 6.1.1). Memory MUST be allocated incrementally as octets are received, or capped at a smaller initial buffer until the message type and context are verified.

Amplification Attacks A single dehydration operation can produce an arbitrary number of child entities from a small input, creating a potential amplification vector. To mitigate this: Implementations MUST enforce the max_entities_per_scope limit negotiated during capability exchange (Section 3.4). Any dehydration that would exceed this limit MUST be rejected. Implementations MUST enforce the max_scope_depth limit. A dehydration chain deeper than this limit MUST be rejected with PIPESTREAM_DEPTH_EXCEEDED (0x07). Implementations SHOULD enforce a configurable ratio between input entity size and total child entity count. A recommended default is no more than 1,000 children per megabyte of parent payload. The backpressure mechanism (Section 9.1) provides a natural throttle: when the in-flight window fills, no new Entity IDs can be assigned until existing entities complete and the cursor advances. Implementations MUST NOT bypass backpressure for dehydration-generated entities.

Privacy Considerations PipeStream entity headers and control stream frames carry metadata that may reveal information about the entities being processed, even when payloads are encrypted at the application layer: Entity structure leakage: The number of child entities produced by dehydration, the scope depth, and the Entity ID assignment pattern may reveal the structure of the input being processed (e.g., an entity that dehydrates into 50 children is likely a multi-part input). Implementations that require structural privacy SHOULD pad dehydration counts or use fixed decomposition granularity. Deployments that do not handle privacy-sensitive data MAY omit this padding. Metadata in headers: The content_type, metadata map, and payload_length fields in EntityHeader (Section 6.8.2) are transmitted in cleartext within the QUIC-encrypted stream. Implementations that require metadata confidentiality beyond transport encryption SHOULD encrypt EntityHeader fields at the application layer and use an opaque content_type such as application/octet-stream. This overhead is unnecessary when the deployment operates within a trusted network. Traffic analysis: The timing and size of status frames on the Control Stream may correlate with processing patterns. Implementations operating in privacy-sensitive environments SHOULD send status frames at fixed intervals with padding to obscure processing timing. Deployments in trusted environments MAY omit traffic padding to reduce bandwidth overhead. Identifiers: Application-level input identifiers and filenames carried inside profile-defined payload envelopes are not interpreted by PipeStream Core but may still be logged by intermediate processing nodes. Implementations SHOULD provide mechanisms to redact or pseudonymize such identifiers at pipeline boundaries. This recommendation may be relaxed when all nodes in the pipeline are operated by the same administrative entity.

Replay and Token Reuse

Yield Token Replay Yield tokens (Section 6.6.2) contain opaque continuation state that enables resumption of paused entity processing. A replayed yield token could cause an entity to be processed multiple times or to resume from a stale state. To prevent this: Implementations MUST associate each yield token with a stable application context identifier (for example, a session identifier) and Entity ID. In Layer 0-only operation, this context MAY be implicit in the active transport connection. For Layer 2 resumptions that can occur across reconnects or different nodes, the context identifier MUST remain stable across transport connections. A yield token MUST be rejected if presented in a different context than the one that issued it, unless the token was explicitly transferred via a claim check. Implementations MUST invalidate a yield token after it has been consumed for resumption. A second resumption attempt with the same token MUST be rejected. The StoppingPointValidation (Section 9.7) provides integrity checking at resume time. Implementations MUST verify the state_checksum field before accepting a resumed entity. If the checksum does not match the current state, the resumption MUST be rejected and the entity MUST be reprocessed from the beginning.

Claim Check Replay Claim checks (Section 6.6.3) are long-lived references that can be redeemed in different sessions. To prevent misuse: Each claim check carries an expiry_timestamp (Unix epoch microseconds). Implementations MUST reject expired claim checks. Implementations MUST track redeemed claim check IDs and reject duplicate redemptions. The tracking state MUST persist for at least the claim check expiry duration. Claim check IDs MUST be generated using a cryptographically secure random number generator to prevent guessing. Because the claim check identifier space is 64 bits, an online attacker with sufficient query volume could attempt to enumerate valid identifiers. Implementations SHOULD rate-limit claim redemption attempts per peer and SHOULD treat repeated redemption failures as a signal of probing.

Encryption Key Management When using FileStorageReference with encryption: Key IDs MUST reference keys in approved providers. Wrapped keys MUST use approved envelope encryption. Key rotation MUST be supported via key_id versioning. Implementations MUST NOT log key material. Implementations MUST NOT include unwrapped data encryption keys in EntityHeader metadata or Control Stream frames.

IANA Considerations This document requests the creation of several new registries and one ALPN identifier registration. All registries defined in this section use the "Expert Review" policy for new assignments unless otherwise stated; guidance for the designated experts appears in Section 11.7.

ALPN Identifier Registration This document registers the following ALPN protocol identifier:

Protocol:

PipeStream Version 1

Identification Sequence:

0x70 0x69 0x70 0x65 0x73 0x74 0x72 0x65 0x61 0x6D 0x2F 0x31 ("pipestream/1")

Specification:

This document

PipeStream Frame Type Registry IANA is requested to create the "PipeStream Frame Types" registry. All frames on Stream 0 MUST use a 4-octet length prefix following the 1-octet Type. The registry is partitioned into three ranges that determine both the registration policy and the framing class of the payload: Range Framing Class Registration Policy 0x00-0x7F Fixed bit-packed payload (Section 6) Expert Review 0x80-0xBF Serialized payload in the negotiated format (Section 6.7) Expert Review 0xC0-0xFF Private use Not applicable Each registration consists of a value, a frame type name, the minimum protocol layer, a short description, and a reference to a specification. The initial contents are: Value Frame Type Name Layer Description Reference 0x50 STATUS 0 Entity lifecycle status Section 6.2 0x54 SCOPE_DIGEST 1 Merkle completion summary Section 6.3 0x55 BARRIER 1 Subtree synchronization Section 6.4 0x56 GOAWAY 0 Graceful shutdown signal Section 6.5 0x80 CAPABILITIES 0 Negotiated limits/layers Section 3.4 0x81 CHECKPOINT 0 Global synchronization Section 9.3

Unknown Frame Handling Receivers that encounter a frame type that they do not recognize MUST skip the frame by reading and discarding the number of octets indicated by the 4-octet length prefix. This mechanism ensures that future protocol extensions can be introduced without breaking backward compatibility for older implementations.

PipeStream Status Code Registry IANA is requested to create the "PipeStream Status Codes" registry. Status codes are 4-bit values (0x0-0xF). Values 0xD-0xE are reserved for Expert Review. Value 0xF is reserved for private use. Value Name Layer Description 0x0 UNSPECIFIED - Default / heartbeat 0x1 PENDING 0 Entity announced 0x2 PROCESSING 0 In progress 0x3 COMPLETE 0 Success 0x4 FAILED 0 Failed 0x5 CHECKPOINT 0 Barrier 0x6 DEHYDRATING 0 Dehydrating into children 0x7 REHYDRATING 0 Rehydrating children 0x8 YIELDED 2 Paused 0x9 DEFERRED 2 Claim check issued 0xA RETRYING 2 Retry in progress 0xB SKIPPED 2 Intentionally skipped 0xC ABANDONED 2 Timed out

PipeStream Error Code Registry IANA is requested to create the "PipeStream Error Codes" registry. Values in the range 0x00-0x3F are assigned by Expert Review. Values in the range 0x40-0xFF are reserved for private use. PipeStream error codes are used as QUIC application error codes in CONNECTION_CLOSE and RESET_STREAM frames. When terminating a connection or aborting a stream due to a protocol-level error, the endpoint MUST use the corresponding PipeStream error code value as the QUIC Application Error Code. Value Name Description 0x00 PIPESTREAM_NO_ERROR Graceful shutdown 0x01 PIPESTREAM_INTERNAL_ERROR Implementation error 0x02 PIPESTREAM_IDLE_TIMEOUT Idle timeout 0x03 PIPESTREAM_CONTROL_RESET Control stream must reset 0x04 PIPESTREAM_INTEGRITY_ERROR Checksum failed 0x05 PIPESTREAM_ENTITY_INVALID Invalid format or state 0x06 PIPESTREAM_ENTITY_TOO_LARGE Size exceeded 0x07 PIPESTREAM_DEPTH_EXCEEDED Scope depth exceeded 0x08 PIPESTREAM_WINDOW_EXCEEDED Window full 0x09 PIPESTREAM_SCOPE_INVALID Invalid scope 0x0A PIPESTREAM_CLAIM_EXPIRED Claim check expired 0x0B PIPESTREAM_CLAIM_NOT_FOUND Claim check not found 0x0C PIPESTREAM_LAYER_UNSUPPORTED Protocol layer not supported 0x0D PIPESTREAM_FRAME_ERROR Malformed frame or improper stream usage

PipeStream Serialization Format Registry IANA is requested to create the "PipeStream Serialization Formats" registry. New entries require Expert Review . A registration request MUST include a reference to a specification that normatively defines the encoding of every serialized PipeStream message (Section 3.4.2). Value Name Description Reference 0 CBOR Concise Binary Object Representation RFC 8949, this document 1-255 Unassigned Available via Expert Review  

URI Scheme Registration This section registers the "pipestream" URI scheme per . The URI scheme identifies application context for detached or resumable resources (for example, Layer 2 yield/claim-check flows). PipeStream Layer 0 streaming semantics do not depend on this URI scheme.

Scheme name:

pipestream

Status:

Permanent

Applications/protocols that use this scheme:

PipeStream protocol (this document)

Scheme syntax:

See Section 11.6.1.

Scheme semantics:

A pipestream URI identifies an application-level session context within a PipeStream deployment and, optionally, a scope path and an entity reference within that context. It is used to locate detached or resumable resources, such as Layer 2 claim checks, potentially across transport connections. The authority component identifies the endpoint that issued the resource.

Encoding considerations:

All components other than the authority are restricted to a subset of unreserved ASCII characters by the ABNF in Section 11.6.1. Percent-encoding is neither required nor permitted in those components. The authority component follows the syntax and encoding rules of , Section 3.2.

Interoperability considerations:

None beyond those described in this document.

Security considerations:

See Section 10 of this document. A pipestream URI may embed session and claim identifiers that function as bearer capabilities; such URIs SHOULD be treated as sensitive and MUST NOT be exposed in logs or referral contexts where they could enable claim redemption by unauthorized parties.

Contact:

Kristian Rickert (kristian.rickert@pipestream.ai)

Change controller:

IETF

References:

This document

URI Syntax The URI syntax is defined using ABNF :

]]>

Examples: pipestream://processor.example.com/a1b2c3d4 pipestream://processor.example.com:8443/a1b2c3d4/1.42/e5f6

Guidance for Designated Experts For all registries defined in this section, the designated experts are advised to apply the following criteria when evaluating requests: A permanent, readily available public specification of the proposed value is required and needs to define the value's semantics precisely enough for independent interoperable implementations. The proposed value must not conflict with, duplicate, or create ambiguity with existing registrations, including the framing class of frame types (fixed bit-packed versus serialized payload; Section 11.2). Requests should be evaluated for their effect on existing deployments; extensions that require behavior changes from unmodified endpoints are inappropriate for these registries and instead require a new protocol version (Section 3.4.1). Assignments from scarce code spaces (in particular the 4-bit status code space) should be granted conservatively; the expert may require demonstrated deployment interest before assigning values from a scarce space.

In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm. This document specifies version 1.3 of the Transport Layer Security (TLS) protocol. TLS allows client/server applications to communicate over the Internet in a way that is designed to prevent eavesdropping, tampering, and message forgery. This document updates RFCs 5705 and 6066, and obsoletes RFCs 5077, 5246, and 6961. This document also specifies new requirements for TLS 1.2 implementations. Many protocols make use of points of extensibility that use constants to identify various protocol parameters. To ensure that the values in these fields do not have conflicting uses and to promote interoperability, their allocations are often coordinated by a central record keeper. For IETF protocols, that role is filled by the Internet Assigned Numbers Authority (IANA). To make assignments in a given registry prudently, guidance describing the conditions under which new values should be assigned, as well as when and how modifications to existing values can be made, is needed. This document defines a framework for the documentation of these guidelines by specification authors, in order to assure that the provided guidance for the IANA Considerations is clear and addresses the various issues that are likely in the operation of a registry. This is the third edition of this document; it obsoletes RFC 5226. The Concise Binary Object Representation (CBOR) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation. These design goals make it different from earlier binary serializations such as ASN.1 and MessagePack. This document obsoletes RFC 7049, providing editorial improvements, new details, and errata fixes while keeping full compatibility with the interchange format of RFC 7049. It does not create a new version of the format. This document proposes a notational convention to express Concise Binary Object Representation (CBOR) data structures (RFC 7049). Its main goal is to provide an easy and unambiguous way to express structures for protocol messages and data formats that use CBOR or JSON. Internet technical specifications often need to define a formal syntax. Over the years, a modified version of Backus-Naur Form (BNF), called Augmented BNF (ABNF), has been popular among many Internet specifications. The current specification documents ABNF. It balances compactness and simplicity with reasonable representational power. The differences between standard BNF and ABNF involve naming rules, repetition, alternatives, order-independence, and value ranges. This specification also supplies additional rule definitions and encoding for a core lexical analyzer of the type common to several Internet specifications. [STANDARDS-TRACK] A Uniform Resource Identifier (URI) is a compact sequence of characters that identifies an abstract or physical resource. This specification defines the generic URI syntax and a process for resolving URI references that might be in relative form, along with guidelines and security considerations for the use of URIs on the Internet. The URI syntax defines a grammar that is a superset of all valid URIs, allowing an implementation to parse the common components of a URI reference without knowing the scheme-specific requirements of every possible identifier. This specification does not define a generative grammar for URIs; that task is performed by the individual specifications of each URI scheme. [STANDARDS-TRACK] This document updates the guidelines and recommendations, as well as the IANA registration processes, for the definition of Uniform Resource Identifier (URI) schemes. It obsoletes RFC 4395. This document describes a Transport Layer Security (TLS) extension for application-layer protocol negotiation within the TLS handshake. For instances in which multiple application protocols are supported on the same TCP or UDP port, this extension allows the application layer to negotiate which protocol will be used within the TLS connection. National Institute of Standards and Technology This document describes a simple process that allows authors of Internet-Drafts to record the status of known implementations by including an Implementation Status section. This will allow reviewers and working groups to assign due consideration to documents that have the benefit of running code, which may serve as evidence of valuable experimentation and feedback that have made the implemented protocols more mature. This process is not mandatory. Authors of Internet-Drafts are encouraged to consider using the process for their documents, and working groups are invited to think about applying the process to all of their protocol specifications. This document obsoletes RFC 6982, advancing it to a Best Current Practice. The QUIC transport protocol has several features that are desirable in a transport for HTTP, such as stream multiplexing, per-stream flow control, and low-latency connection establishment. This document describes a mapping of HTTP semantics over QUIC. This document also identifies HTTP/2 features that are subsumed by QUIC and describes how HTTP/2 extensions can be ported to HTTP/3. This document describes HTTP Datagrams, a convention for conveying multiplexed, potentially unreliable datagrams inside an HTTP connection. In HTTP/3, HTTP Datagrams can be sent unreliably using the QUIC DATAGRAM extension. When the QUIC DATAGRAM frame is unavailable or undesirable, HTTP Datagrams can be sent using the Capsule Protocol, which is a more general convention for conveying data in HTTP connections. HTTP Datagrams and the Capsule Protocol are intended for use by HTTP extensions, not applications. This document describes the use of QUIC to provide transport confidentiality for DNS. The encryption provided by QUIC has similar properties to those provided by TLS, while QUIC transport eliminates the head-of-line blocking issues inherent with TCP and provides more efficient packet-loss recovery than UDP. DNS over QUIC (DoQ) has privacy properties similar to DNS over TLS (DoT) specified in RFC 7858, and latency characteristics similar to classic DNS over UDP. This specification describes the use of DoQ as a general-purpose transport for DNS and includes the use of DoQ for stub to recursive, recursive to authoritative, and zone transfer scenarios. The Peer-to-Peer Streaming Peer Protocol (PPSPP) is a protocol for disseminating the same content to a group of interested parties in a streaming fashion. PPSPP supports streaming of both prerecorded (on- demand) and live audio/video content. It is based on the peer-to- peer paradigm, where clients consuming the content are put on equal footing with the servers initially providing the content, to create a system where everyone can potentially provide upload bandwidth. It has been designed to provide short time-till-playback for the end user and to prevent disruption of the streams by malicious peers. PPSPP has also been designed to be flexible and extensible. It can use different mechanisms to optimize peer uploading, prevent freeriding, and work with different peer discovery schemes (centralized trackers or Distributed Hash Tables). It supports multiple methods for content integrity protection and chunk addressing. Designed as a generic protocol that can run on top of various transport protocols, it currently runs on top of UDP using Low Extra Delay Background Transport (LEDBAT) for congestion control. Many IETF protocols use cryptographic algorithms to provide confidentiality, integrity, authentication, or digital signature. Communicating peers must support a common set of cryptographic algorithms for these mechanisms to work properly. This memo provides guidelines to ensure that protocols have the ability to migrate from one mandatory-to-implement algorithm suite to another over time. Cisco Google Google Meta This document defines Media over QUIC Transport (MOQT), a publish/ subscribe protocol that runs over QUIC and WebTransport. MOQT leverages the features of these transports, such as streams, datagrams, priorities, and partial reliability. MOQT operates both point-to-point and through intermediate relays, enabling scalable low-latency delivery. Despite its name, MOQT is media agnostic and can be used for a wide range of use cases.

Protocol Layer Capability Matrix Feature Layer 0 Layer 1 Layer 2 Unified status frame (128-bit base) X X X Entity streaming X X X PENDING/PROCESSING/COMPLETE/FAILED X X X DEHYDRATING/REHYDRATING X X X Checkpoint blocking X X X Assembly Manifest X X X Cursor-based ID recycling X X X Scoped status fields (Scope ID, depth)   X X Hierarchical scopes   X X Scope digest (Merkle)   X X Barrier (subtree sync)   X X YIELDED status     X DEFERRED status     X Claim checks     X Completion policies     X SKIPPED/ABANDONED statuses     X

Relationship to Existing Protocols This appendix discusses the relationship between PipeStream and existing transport and application protocols. The intent is to clarify the design rationale for specifying a new application protocol directly over QUIC rather than layering on HTTP/3 , gRPC, or WebTransport .

QUIC as Application Transport RFC 9308 provides guidance for designers of application protocol mappings to QUIC. PipeStream follows this guidance: stream semantics are mapped to the protocol's data model (Section 4), flow control operates at both the stream and connection level, and an ALPN token is registered for protocol identification (Section 11). The precedent for application protocols that bypass HTTP and map directly onto QUIC is well established. DNS over Dedicated QUIC Connections adopts a direct mapping on the grounds that HTTP framing introduces unnecessary overhead when the application has its own message semantics. The Media over QUIC Transport protocol similarly defines its own framing and control messages over QUIC streams, with HTTP/3 as an optional encapsulation rather than a requirement. PipeStream's requirements align with these precedents. The protocol's dual-stream architecture (Section 4), bit-packed control frames (Section 6), and recursive entity lifecycle (Section 9) have no counterpart in HTTP semantics, and mapping them onto request-response pairs would add complexity without benefit.

Efficiency Considerations The decision to implement PipeStream as a standalone protocol over QUIC is driven by the following efficiency requirements:

Framing Overhead HTTP/3 and gRPC introduce multiple layers of framing. A gRPC message over HTTP/3 incurs QUIC stream overhead, HTTP/3 DATA frame overhead (minimum 2 octets), and gRPC envelope overhead (5 octets). For small status updates like PipeStream's 21-octet on-wire STATUS frame, this encapsulation would still materially increase bandwidth requirements. PipeStream's bit-packed frames minimize coordination overhead for high-frequency control traffic.

Stateless Control Plane HTTP/3 requires maintaining state for QPACK header compression. In high-concurrency environments where a single node manages thousands of sessions, the memory overhead of these contexts is significant. PipeStream status frames are stateless and self-describing; a receiver can parse any frame without reference to previous frames, enabling low-latency processing and easier load balancing across redundant nodes.

Stream Independence gRPC bidirectional streaming is constrained by the lifecycle of a single RPC call. While multiple entities can be multiplexed into one RPC, this reintroduces head-of-line blocking at the application layer if one entity requires retransmission. Alternatively, opening a new RPC for every entity (which could number in the millions for large input sets) incurs substantial setup and teardown overhead. PipeStream treats each entity as an independent QUIC stream, ensuring maximum parallelism and optimal use of QUIC's loss recovery mechanisms.

HTTP/3 HTTP/3 provides multiplexed request-response exchanges over QUIC. Its stream model binds each client-initiated request to a server response on the same stream. PipeStream requires bidirectional, peer-initiated entity streams where either endpoint may open new streams to transmit sub-entities arising from recursive decomposition. The request-response constraint precludes this. PipeStream also requires a persistent control stream carrying compact, fixed-size status frames at high frequency. HTTP/3 does define unidirectional control streams, but their framing is specific to HTTP semantics (SETTINGS, GOAWAY, MAX_PUSH_ID) and cannot be repurposed for application-level status coordination without introducing a parallel signaling mechanism that duplicates much of what QUIC already provides.

gRPC gRPC defines a remote procedure call framework over HTTP/2, with experimental support for HTTP/3. Bidirectional streaming in gRPC is scoped to a single RPC method: one request stream and one response stream per call. PipeStream requires an arbitrary number of concurrent entity streams with independent flow control, plus a dedicated control stream, all within a single connection. Achieving this over gRPC would require either multiplexing all entities onto a single bidirectional RPC (sacrificing per-stream flow control and head-of-line independence) or opening a separate RPC per entity (sacrificing session-level coordination and incurring per-call overhead). Beyond the stream model mismatch, gRPC does not provide protocol-level primitives for recursive entity decomposition, checkpoint-based consistency across parallel processing paths, or hierarchical scope management with digest propagation. Building equivalent semantics on top of gRPC requires application-layer coordination logic that PipeStream provides natively as part of the protocol specification. gRPC further mandates a 5-octet length-prefixed framing envelope for every message. PipeStream's fixed-size control frames are bit-packed at the wire level with zero serialization overhead, which is material at the status update frequencies the protocol is designed to sustain. PipeStream complements rather than replaces gRPC: a PipeStream pipeline stage MAY internally use gRPC to communicate with processing services while relying on PipeStream for inter-stage coordination and consistency.

WebTransport WebTransport provides bidirectional streams and unreliable datagrams over HTTP/3, and is the closest existing protocol to the transport abstraction PipeStream requires. However, several properties make it unsuitable as a substrate: WebTransport sessions are established via an HTTP/3 CONNECT request, inheriting the client-server asymmetry of HTTP. In PipeStream, both endpoints participate symmetrically in capability negotiation and may initiate entity streams; the protocol does not distinguish a "client" role from a "server" role after the handshake. WebTransport is designed for environments constrained by the web security model (origin-based isolation, CORS). PipeStream targets server-to-server processing pipelines where these constraints are inapplicable. WebTransport provides raw byte streams with no built-in coordination semantics. PipeStream would need to implement its own framing, status state machine, checkpoint barriers, and scope hierarchy on top of WebTransport streams. At that point, the HTTP/3 session layer introduces an additional round trip during establishment and per-stream framing overhead, with no corresponding benefit.

Media over QUIC Transport The Media over QUIC Transport protocol is architecturally the closest existing QUIC-native protocol to PipeStream: both use a bidirectional control stream for session coordination alongside unidirectional streams carrying independent data objects, following the application-mapping guidance of . The two protocols nevertheless solve different problems. MOQT is a publish/subscribe media delivery protocol optimized for fan-out through relays: objects within a track are delivered to a dynamic set of subscribers, delivery of any individual object may be abandoned when it ceases to be useful (for example, after a latency deadline passes), and no participant tracks whether the totality of published objects reached any subscriber. PipeStream is a scatter-gather processing protocol with the opposite requirements: the set of child entities produced by a decomposition is determinate, every entity MUST be driven to a terminal status, and reassembly is gated on cryptographically verifiable completion of the entire set (Section 9). MOQT has no counterpart to hierarchical scopes, scope digests, barrier synchronization, or completion policies, and PipeStream has no counterpart to tracks, subscriptions, or relay-based fan-out. Extending MOQT with completion tracking would require adding precisely the control-plane machinery this document defines, while inheriting subscription semantics that have no role in pipeline processing.

Broker-Based Pipeline Architectures Many production processing pipelines are built not on RPC frameworks but on message brokers and distributed commit logs, with stages communicating through persistent intermediary queues or topics. These architectures provide durable decoupling and replay, and PipeStream is not a replacement for them where store-and-forward durability between organizationally separate stages is the primary requirement. However, broker-based architectures externalize the semantics that PipeStream provides natively. Completion of a hierarchically decomposed job must be tracked in a coordination store outside the message path; end-to-end integrity of a reassembled output must be verified by application logic; and backpressure operates per queue rather than end-to-end across a pipeline. Each stage boundary additionally imposes a full persist-and-forward cycle, which defeats incremental streaming of large entities. The broker itself is also an interoperability boundary: no standardized wire protocol expresses scatter-gather semantics across brokers from different vendors, so multi-organization pipelines are coupled through vendor-specific client libraries. A PipeStream deployment MAY use a broker behind an individual processing stage (as it may use gRPC; see above) while relying on PipeStream for inter-stage transport, completion tracking, and integrity verification.

Summary PipeStream occupies a design point not addressed by existing protocols: a QUIC-native application protocol combining multiplexed entity streaming, recursive decomposition with hierarchical scopes, Merkle-based integrity propagation, and barrier-synchronized reassembly. While existing protocols like SCTP and PPSPP address subsets of these requirements, none provide the integrated lifecycle and coordination semantics that PipeStream defines.

Schema Reference (CDDL) This appendix consolidates the normative CDDL schema definitions for PipeStream Core messages that use negotiated serialization. These definitions are authoritative for the wire format when CBOR is the negotiated serialization format (the default). Fixed control frames on Stream 0, such as STATUS and SCOPE_DIGEST, use the bit-packed wire formats defined in Section 6 and are not serialized as CDDL messages. Application-specific payload envelopes and profile-specific schemas are outside the scope of this appendix; see for an example companion profile that defines such messages.

tstr }, ? chunk-info: chunk-info, ? completion-policy: completion-policy, ; Layer 2 } chunk-info = { total-chunks: uint, chunk-index: uint, chunk-offset: uint, } ; ----------------------------------------------------------- ; Completion policy (Layer 2) ; ----------------------------------------------------------- completion-policy = { ? mode: completion-mode, ? max-retries: uint, ? retry-delay-ms: uint, ? timeout-ms: uint, ? min-success-ratio: float32, ? on-timeout: failure-action, ? on-failure: failure-action, } completion-mode = &( unspecified: 0, strict: 1, lenient: 2, best-effort: 3, quorum: 4, ) failure-action = &( unspecified: 0, fail: 1, skip: 2, retry: 3, defer: 4, ) ; ----------------------------------------------------------- ; Checkpoint frame (Type 0x81) ; ----------------------------------------------------------- checkpoint-frame = { checkpoint-id: tstr, sequence-number: uint, checkpoint-entity-id: uint, ? scope-id: uint, ? flags: uint, ? timeout-ms: uint, } ; ----------------------------------------------------------- ; Entity status codes ; ----------------------------------------------------------- entity-status = &( unspecified: 0, pending: 1, processing: 2, complete: 3, failed: 4, checkpoint: 5, dehydrating: 6, rehydrating: 7, yielded: 8, deferred: 9, retrying: 10, skipped: 11, abandoned: 12, ) ; ----------------------------------------------------------- ; Assembly Manifest entry ; ----------------------------------------------------------- assembly-manifest-entry = { parent-id: uint, ? scope-id: uint, children-ids: [* uint], ? children-status: [* entity-status], ? policy: completion-policy, ? created-at: uint, ? state: resolution-state, } resolution-state = &( unspecified: 0, active: 1, resolved: 2, partial: 3, failed: 4, ) ; ----------------------------------------------------------- ; Yield token (Layer 2) ; ----------------------------------------------------------- yield-token = { reason: yield-reason, ? continuation-state: bstr, ? validation: stopping-point-validation, } yield-reason = &( unspecified: 0, external-call: 1, rate-limited: 2, awaiting-sibling: 3, awaiting-approval: 4, resource-busy: 5, ) ; ----------------------------------------------------------- ; Claim check (Layer 2) ; ----------------------------------------------------------- claim-check = { claim-id: uint, entity-id: uint, ? scope-id: uint, expiry-timestamp: uint, ? validation: stopping-point-validation, } ; ----------------------------------------------------------- ; Stopping point validation (Layer 2) ; ----------------------------------------------------------- stopping-point-validation = { ? state-checksum: bstr, ? bytes-processed: uint, ? children-complete: uint, ? children-total: uint, ? is-resumable: bool, ? checkpoint-ref: tstr, } ; ----------------------------------------------------------- ; File storage reference ; ----------------------------------------------------------- file-storage-reference = { provider: tstr, bucket: tstr, key: tstr, ? region: tstr, ? attrs: { * tstr => tstr }, ? encryption: encryption-metadata, } encryption-metadata = { algorithm: tstr, ? key-provider: tstr, ? key-id: tstr, ? wrapped-key: bstr, ? iv: bstr, ? context: { * tstr => tstr }, } ]]>

Implementation Status [[RFC Editor: please remove this entire appendix, and the reference to , before publication.]] This appendix records the status of known implementations of the protocol defined by this specification at the time of posting of this Internet-Draft, following the process described in . The description of implementations in this appendix is intended to assist the IETF in its decision processes in progressing drafts to RFCs. Please note that the listing of any individual implementation here does not imply endorsement by the IETF. Furthermore, no effort has been spent to verify the information presented here that was supplied by IETF contributors. This is not intended as, and must not be construed to be, a catalog of available implementations or their features. Readers are advised to note that other implementations may exist.

Reference Implementation

Organization:

PipeStream AI

Description:

Reference implementation of the PipeStream protocol.

Maturity:

Under development; not yet publicly released.

Coverage:

To be updated as development progresses.

Licensing:

To be determined.

Contact:

Kristian Rickert (kristian.rickert@pipestream.ai)

The authors welcome reports of additional implementations for inclusion in future revisions of this appendix.