| Internet-Draft | YANG Message Keys | September 2025 |
| Graf, et al. | Expires 24 March 2026 | [Page] |
This document describes a mechanism to define a unique message key for a YANG to message broker integration and a topic addressing scheme based on YANG-Push subscription type and a YANG index defined in this document. This enables a Message Broker to serve from a single partition, compress to current state in case of YANG state metrics and YANG data consumer to consume for a specific YANG node identifier of a network node YANG datastore.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 24 March 2026.¶
Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Nowadays network operators are using machine and human readable YANG [RFC7950] to model their configurations and obtain YANG modelled operational data from their networks according to [Mar24].¶
YANG data can be used for several network analytic use cases. Depending on use case, only a subset of the subscribed data might be necessary. This subset could be the current network state instead of state changes over a period of time, or instead of consuming data for all network nodes maybe only for a particular network node or network node component of a YANG subscription.¶
Most network analytic use cases require real-time data and deliver near real-time analytical, actionable insights. This requires high scalability, resilience and low overhead in the data processing pipeline. Accessing the right data for the right use case with minimal overhead and in the shortest period of time is therefore crucial.¶
Network operators organize their data in a Data Mesh [Deh22] according to [Bod24] where a Message Broker such as Apache Kafka [Kaf11] or Apache Pulsar [Pul16] facilitates the exchange of messages among data processing components in topics and subjects. Typically, data is being stored in Message Broker topics for several hours or days to facilitate resilience in the data processing chain and addressed in subjects depending on schema. Enabling a data consumer to address and re-consume previously consumed data again if previously lost.¶
An Architecture for YANG-Push to Message Broker Integration [I-D.ietf-nmop-yang-message-broker-integration] defined an architecture for integrating YANG-Push with Message Brokers for a Data Mesh architecture. How the notification messages at a YANG-Push Receiver is being transformed to the Message Broker is being described in Section 4.5 of [I-D.ietf-nmop-yang-message-broker-integration] and to which message schema in Section 3 of [I-D.ietf-nmop-message-broker-telemetry-message], however how messages should be indexed best for dimensional YANG data is left unspecified.¶
Due to the missing dimensional indexing for Message Broker stored YANG data, all YANG data is stored in one single Topic, distributed round robin across multiple Partitions and each YANG schema id is a subject within that topic. Therefore, the entire Topic from all Partitions needs to be consumed first before data selection can be applied. This leads to avoidable data processing overhead which in turn impairs scalability and real-time capabilities which are requirements for certain Network Analytics use cases.¶
Dimensional data is structured information in a data warehouse. It uses a model of dimension tables to organize business metrics and their descriptive context. This model, developed by Ralph Kimball [Kim96], simplifies data analysis and reporting by creating denormalized, easy-to-understand structures for quick querying. It is optimized for online analytical processing (OLAP) and data warehouses. YANG [RFC7950] as a data modelling language facilitates the modelling of dimensional data.¶
This document defines how YANG messages should be indexed and organized in Message Broker topics by leveraging the network node hostname, YANG datastore name and YANG item identifier for indexing and YANG-Push subscription type and YANG schema name for a Message Broker topic naming scheme.¶
Network node hostname, YANG datastore name and subtree and xpath filters are part of "ietf-yang-push-telemetry-message" structured YANG data defined in Section 3 of [I-D.ietf-nmop-message-broker-telemetry-message]. YANG item identifier are derived from subtree and xpath filters respectively from their YANG schema tree.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
The following terms are used as defined in [I-D.ietf-nmop-terminology]:¶
The following terms are used as defined in [I-D.ietf-nmop-yang-message-broker-integration]:¶
The following terms are used as defined in Apache Kafka [Kaf11] and Apache Pulsar [Pul16] Message Broker:¶
The following terms are used as defined in Confluent Schema Registry Documentation [ConDoc18]:¶
The following terms are used as defined in [RFC8641]:¶
The following terms are used as defined in [I-D.ietf-netconf-notif-envelope]:¶
The following terms are used as defined in [RFC8342]:¶
The following terms are used as defined in [RFC7950]:¶
The following terms are used as defined in [RFC9254]:¶
This document defines the following term:¶
In order to identify which YANG node identifier of a network node YANG datastore is produced in which Message broker Topic and Partition for which Subject, YANG Message Keys and Indexes (Section 3.1) are being introduced.¶
In order to facilitate Message Broker Topic Compaction, a YANG-Push subscription type based topic nameing scheme (Section 3.2) is proposed. This segregates statistical (Value), State and State change YANG metrics and facilitates a YANG Message Broker Consumer to use the Topic wild card consumption method to select based on YANG-Push subscription type.¶
A Message Broker uses a Message Key to index the message and a value to carry the Message content. If no Message Key is defined then the Messages are distributed in a round robin fashion across partitions. If a Message Key is defined, then the value of the Message Key is being used as input for the Message Broker Producer hash function to distribute across Partitions. Therefore, Message Keys facilitate Message ordering.¶
The Message Key not only used for Message indexing at the Message Producer but also at the Message Broker for topic compaction.¶
For YANG, the network node hostname, from which YANG datastore the YANG metrics are published from and the YANG index is used to generate the Message Key.¶
YANG data nodes are uniquely identifiable. Section 6.5 of [RFC7950] defines with "absolute-schema-nodeid" how absolute YANG schema node identifiers are being crafted locally unique to the YANG module.¶
Section 3.3 of [RFC9254] defines how globally unique YANG item identifiers are defined as text strings.¶
Section 3.6 of [RFC8641] defines how YANG data nodes can be subscribed with subtree and xpath selection filters. A YANG-Push publisher publishes with "subscription-started" state notifications for each subscription which filter and filter type is being used to the YANG-Push receiver.¶
To calculate the YANG Index of the Message Key, the YANG item identifier needs to be extracted from the used YANG-Push subtree or xpath subscription filter. If the YANG item identifier is a YANG list as defined in Section 7.8 of [RFC7950] the YANG list key defined in Section 7.8.2 of [RFC7950] statement is suffixed with a "/" to the YANG item identifier.¶
For example, if the following xpath filter is being used, the YANG item identifier is "ietf-interface:interfaces/interface". Interface is a YANG list with name as key. Therefore, the YANG Index of the Message Key is "ietf-interface:interfaces/interface/name".¶
ietf-interface:interfaces/interface[type='ianaift:ethernetCsmacd']
For example, if the following subtree filter is being used, the YANG item identifier is "ietf-hardware:hardware/component/state". Therefore, the YANG Index of the Message Key is "ietf-hardware:hardware/component/state".¶
<get>
<filter type="subtree">
<hardware xmlns="urn:ietf:params:xml:ns:yang:ietf-hardware">
<component>
<state/>
</component>
</hardware>
</filter>
</get>
When the Message is being produced to the Message Broker, the Network node hostname and YANG datastore name is used from the structured YANG data defined in "ietf-yang-push-telemetry-message" Section 3 of [I-D.ietf-nmop-message-broker-telemetry-message] where the YANG Index is derived from subtree and xpath filters, respectively from their YANG schema tree.¶
The consumer hashes the Message Key and applies modulo with the number of partitions to determine the partition it needs to consume from to obtain Messages with desired Message Key.¶
YANG can be subscribed periodically, on-change or on-change with sync-on-start. Periodical subscriptions are used for obtaining statistical metrics. On-Change subscriptions are used for obtaining State Changes and on-change with sync-on-start for obtaining States.¶
Message Brokers topics are addressed with a unique name. Usually topics are named hierarchically similar to the DNS namespace where "." deliminates hierarchies.¶
This document proposes to include "statistics", "states" and "state-changes" in the topic name as the first part to denote the types of data. Followed by "yang" to denote YANG data. Followed by the YANG module names subscribed, and followed by the YANG Schema Node Identifier where "/" is substituted by "_".¶
For example, if the "ietf-interface:interfaces/interface" xpath filter is being used, the Message Broker topic name would be as following. In the example the project name and environment (prod, dev, test etc.) is prefixed.¶
project.enviroment.statistics.yang.ietf-interfaces.interfaces_interface
For the Message Broker topic creation, the "periodic", "on-change" and "sync-on-start" contained data in "update-trigger" from "ietf-subscribed-notifications", YANG module defined in Section 4.1 of [RFC8641], subscription state notifications are being used to derive wherever subscribed YANG data is "statistics", "states" or "state-changes". The YANG Index is derived from subtree and xpath filter data of subscription state notifications, respectively from their YANG schema tree.¶
The consumer has the ability to consume with a wildcard denoted with "*" in the topic name to consume from more than one topic.¶
For example, if YANG states should be consumed and indexed in Time Series database or stream processor than below Topic Name could be used, and the YANG data could be ingested into tables according to topic names and indexed per Message Key. If Topic Compaction is enabled, only current state is consumed.¶
project.enviroment.states.yang.*
Topic, Partitioning and Message Keying are generic concepts of Message Brokers. There are two known Message Broker implementations supporting all features described in this document.¶
Apache Kafka supports Message Keying, Partitioning and Log Compaction. The topic names are constrained to 249 character length and the following characters: "a-z", "A-Z", "0-9", ".", "_" and "-".¶
Apache Pulsar supports Message Keying, Partitioning and Topic Compaction. The topic names allow all characters except: "/".¶
This document includes no request to IANA.¶
This document should not affect the security of the Internet.¶
The YANG Message Broker Producer of a YANG-Push receiver SHOULD have three config knobs facilitate the features described in this document as optional:¶
To accommodate for potential date loss throughout the data processing pipeline, periodical update of the current State for State metrics is RECOMMENDED. This can be accommodated with YANG-Push as defined in [RFC8641] by complementing "on-change sync on start" subscriptions with periodical subscriptions. Alternatively, in YANG-Push Lite defined in Section 7.6 of [I-D.wilton-netconf-yang-push-lite] this simplified in one subscription.¶
Thanks to¶
We like to thank Victor Lopez for the initial idea on the network controller use case. Ashley Woods, Sivakumar Sundaravadivel and Rafael Julio for the idea of grouping topics by YANG-Push subscription type and insisting that Topic Compaction is a key enabler for inventory metrics and YANG data consumer integration and should be supported day 1. And Nigel Davis for confirming that Topic Compaction simplifies indeed data processing system architecture.¶