]> National Institute of Advanced Industrial Science and Technology (AIST)

y.oiwa@aist.go.jp

GMO CONNECT Inc.

kanno@gmo-connect.jp

GMO CONNECT Inc.

sakemi-yumi@gmo-connect.jp

General Internet-Draft Hybrid cloud Multi-cloud Security monitoring Telemetry This document presents a problem statement and gap analysis for ensuring and monitoring the security status of networks operating in complex environments, such as hybrid and multi-cloud systems. It identifies a missing capability: verifying the path and security properties of communications across multiple administrative domains while preserving each provider's confidentiality and local policy boundaries. The document also outlines, non-normatively, potential solution directions.

Introduction Virtualized resources such as cloud computing infrastructure are increasingly replacing traditional network and computing environments such as local servers and on-premises clusters. In such infrastructure, the details of physical resources -- servers, local networks, routers, and so on -- are hidden from users in exchange for flexibility, service redundancy, and lower cost. Cryptographic protocols such as TLS and IPsec are typically used to protect communication into and out of these systems against eavesdropping and tampering. However, there are many use cases where service still depends on the security nature of underlying physical resources, instead of just encrypting the communication: Traffic analysis on encrypted communication may reveal partial information of the payload; Legal or jurisdictional requirements (such as personal data protection) demand that specific properties (such as governing laws, geographic locations, or operators) be checked; Denial-of-service and several other attacks may not be prevented by encryption only. Such high-security applications need a technical means to continuously check the properties and status of the underlying network and its intermediate nodes. In a non-virtualized, self-managed setting, several existing technologies (e.g., NETCONF and path validation) can obtain this status. However, they are not sufficient for the virtualized, multi-stakeholder setting of modern cloud infrastructure. This document presents a first-stage problem analysis for ensuring and monitoring the security status of networks operating in complex environments such as hybrid and multi-cloud deployments. This document provides (1) a problem statement and gap analysis, and (2) a non‑normative outline of potential solution directions. It does not define protocol requirements. A companion document, the Path Characteristics Service , elaborates one such solution direction in detail.

Background

Multi-cloud and Hybrid Cloud Systems The concepts of multi-cloud and hybrid cloud are defined in . In short, a multi-cloud system implements a single service using two or more independently operated cloud services. A hybrid cloud system combines two or more computing environments that differ in their operation, security level, or other aspects, at least one of which is typically a public cloud service. Often, subsystems on a privately operated cloud, on-premises, or edge networks are connected to public cloud infrastructure over a network to form a single hybrid cloud system. A hybrid cloud is generally adopted when the security or other provisions of a public cloud are not sufficient for some part of the data or for a subsystem component (otherwise, a simple public or multi-cloud environment would suffice). At the same time, the benefits of public cloud systems -- scalability, cost, resilience, maintainability, and so on -- are often still desired (otherwise, a simple on-premises deployment would be enough). These mixed and seemingly conflicting requirements make it difficult to ensure and monitor the security of hybrid cloud systems.

Security Implications of Hybrid Clouds Multi-cloud and hybrid cloud systems require internal communications that flow beyond the boundary of a single cloud. In the simplest case, this can be implemented using authenticated TLS or HTTPS over the public Internet. For high-security systems, it is often implemented using dedicated communication channels, such as VPNs, private peering, or even dedicated optical fiber. To maintain the security of the whole system, monitoring the integrity of these dedicated channels is essential. Furthermore, IP-based software systems depend on many additional components to keep such communication secure, which also widens the attack surface. For example, if a DNS record is tampered with or misconfigured, traffic intended for a secure channel might instead be routed over public channels; a routing misconfiguration can have the same effect. Standardized network telemetry provides building blocks for collecting such operational status, but systematically enumerating these dependencies and assessing their security impact across administrative boundaries remains difficult today.

Problem Statement Several existing technologies address related aspects of this problem. SAVNET (Source Address Validation in Intra-domain and Inter-domain Networks) develops mechanisms for source address validation, helping detect and discard packets that carry spoofed source addresses. SRv6 network programming lets a source encode an intended packet-processing path as a sequence of segments, enabling per-packet steering within SRv6-enabled domains. RPKI provides trust anchors and a public-key infrastructure for validating the authorization of route originations in inter-domain routing. These technologies each address part of the picture, but none provides what hybrid cloud security most needs: a common, privacy-preserving way to verify the path and security properties of a communication across multiple administrative domains -- confirming, for example, the jurisdictions traversed, the operators involved, and the protection applied -- without forcing any provider to disclose its internal topology or violate its local policy boundaries. A privacy-preserving path-characteristics verification service (see ) is one solution direction for this gap. The following aspects, in particular, remain insufficiently addressed.

The Nature of Multiple Operators/Stakeholders Hybrid cloud systems depend on many resources that are not under the control of the application operator. Public clouds (both IaaS and SaaS) are run by external service providers, each with its own operational policies and its own decisions about maintaining or replacing the hardware and software it provides, as long as its service-level agreements (SLAs) are met. This makes it undesirable to expose information about every intermediate network node to the application operator. First, detailed design and implementation information about the cloud infrastructure is confidential and a competitive asset of the cloud provider. Moreover, a degree of independence between application operators and cloud service providers is important for the cost-effectiveness, maintainability, and security of the cloud services.

Determination of the "Correct" States In a small, hand-crafted network, determining whether the current running state is the intended one is relatively straightforward. In a complex multi-cloud system, however, this is very hard or even impossible -- even if one could observe every detail of the running state of the entire global network. Making that determination also requires knowing the design principles and implicit assumptions behind the operation of each constituent network.

Shared Infrastructure and Information Leakage Cloud infrastructure is deeply shared among many clients. Although some information about the cloud-side operational status is needed to assess the reliability of a client's applications, exposing raw operational parameters to one client may reveal security-critical information about others. Before any cloud-side status is exposed, it must be processed and filtered so that only the information relevant to that specific client is included.

Virtualized Infrastructure Many cloud resources -- not only compute nodes but also routers, switches, and VPN endpoints -- are virtualized and provisioned through infrastructure-as-code (IaC) systems. Unlike physical routers and switches, identifying a virtual intermediate node on a traffic path reveals nothing about its physical location, physical properties, or security characteristics. (Consider, for example, how little can be inferred from a traceroute or ICMP ping that traverses a virtual private network.) Where virtual nodes are present, the physical properties of the underlying infrastructure may need to be traced and verified to ensure security and integrity. This requires cooperation from the virtual-resource or cloud providers and integration with their infrastructure management systems.

Risks Beyond Network Layers Today, many networks are operated through complex management systems. As a result, any compromise of the IT-side assets of those management systems poses serious risks to the network layer. Such assets include, but are not limited to, software asset management, software vulnerability management, and identity management. To correctly evaluate the risks to overall network operation, the risks associated with these management systems must also be taken into account.

Security Risks in Hybrid Cloud Environments Because there is no common, privacy-preserving way to verify the path and security properties of communications across administrative domains, hybrid cloud environments are exposed to the classes of risk below. Each is, at its core, a consequence of insufficient cross-domain visibility and verification:

Invisible Attack Vectors Current hybrid cloud deployments create numerous blind spots where malicious activities can occur undetected: Traffic Misdirection: DNS tampering or routing misconfigurations can redirect secure traffic through compromised or unintended paths without detection Virtual Infrastructure Exploitation: Attacks targeting hypervisors or virtual network components remain invisible to traditional network monitoring Cross-Tenant Information Leakage: Shared infrastructure may enable side-channel attacks or resource-based information disclosure between different cloud tenants

Compromised Trust Assumptions The multi-stakeholder nature of hybrid clouds breaks traditional security perimeters: Fragmented Visibility: No single entity has complete visibility into the security posture, creating gaps that attackers can exploit Unclear Responsibility Boundaries: Security incidents may go undetected when responsibilities are unclear between cloud providers and users Unverifiable Provider Dependencies: Dependencies on multiple cloud providers expand the attack surface, yet there is no common way to verify that each provider continues to meet the security properties an application relies on

Persistent Security Degradation Without proper monitoring capabilities, security posture deteriorates over time: Configuration Drift: Gradual misconfigurations accumulate, creating vulnerabilities that remain undetected Stale Security Policies: Security rules become outdated as infrastructure evolves, but changes go unnoticed Delayed Incident Response: Without a way to verify path and security properties on demand, security incidents can remain undetected for extended periods, allowing attackers to establish persistence

Amplified Attack Impact The interconnected nature of hybrid clouds amplifies the impact of successful attacks, and the absence of cross-domain verification makes that impact harder to bound: Undetected Lateral Movement: Without cross-domain visibility into path and operator properties, compromised components can serve as stepping stones into other parts of the hybrid infrastructure unnoticed Unattributed Propagation: When responsibility for each path segment cannot be verified, an incident in one provider can propagate to other components of the hybrid system without clear attribution Undetected Exfiltration: Sensitive data may traverse multiple untrusted networks because the path and its security properties cannot be independently verified Any solution addressing these problems must carefully balance security monitoring requirements with the protection of sensitive infrastructure information and the preservation of multi-stakeholder operational independence.

Security Considerations This document is a problem statement and does not define a protocol; the considerations below apply to the class of monitoring and verification mechanisms it motivates (see for one such mechanism). Information disclosure: A monitoring or verification capability necessarily handles sensitive operational data. It needs to avoid becoming a new disclosure channel for provider topology, tenant identities, or other confidential details, and is expected to return only the minimum information needed (for example, a conformance result rather than raw state). Probing and reconnaissance: An interface that answers questions about path or security properties can be abused to map otherwise-hidden infrastructure. Such a mechanism ought to constrain its responses to a pre-agreed scope and to resist systematic probing. False assurance: A positive verification result that is stale, incomplete, or derived from untrustworthy inputs can give operators unwarranted confidence. Carrying provenance and freshness metadata helps consumers judge how far a result can be relied upon. Stale telemetry: Underlying telemetry may lag the actual network state. The trustworthiness of any assurance depends on the timeliness of its inputs, which needs to be made explicit. Accountability boundaries: In a multi-stakeholder setting, responsibility for each assertion needs to be attributable to a specific operator within its own scope, so that an incorrect or compromised assertion can be localized. Over-reliance on opaque provider assertions: When a provider asserts a property without exposing the underlying evidence, consumers depend on that provider's trustworthiness. Representing the degree of verifiability of each assertion explicitly, rather than assuming it, reduces this risk.

IANA Considerations This document has no IANA actions.

This document describes an architecture for an infrastructure to support improved security of Internet routing. The foundation of this architecture is a Resource Public Key Infrastructure (RPKI) that represents the allocation hierarchy of IP address space and Autonomous System (AS) numbers; and a distributed repository system for storing and disseminating the data objects that comprise the RPKI, as well as other signed objects necessary for improved routing security. As an initial application of this architecture, the document describes how a legitimate holder of IP address space can explicitly and verifiably authorize one or more ASes to originate routes to that address space. Such verifiable authorizations could be used, for example, to more securely construct BGP route filters. This document is not an Internet Standards Track specification; it is published for informational purposes. Network telemetry is a technology for gaining network insight and facilitating efficient and automated network management. It encompasses various techniques for remote data generation, collection, correlation, and consumption. This document describes an architectural framework for network telemetry, motivated by challenges that are encountered as part of the operation of networks and by the requirements that ensue. This document clarifies the terminology and classifies the modules and components of a network telemetry system from different perspectives. The framework and taxonomy help to set a common ground for the collection of related work and provide guidance for related technique and standard developments. The Segment Routing over IPv6 (SRv6) Network Programming framework enables a network operator or an application to specify a packet processing program by encoding a sequence of instructions in the IPv6 packet header. Each instruction is implemented on one or several nodes in the network and identified by an SRv6 Segment Identifier in the packet. This document defines the SRv6 Network Programming concept and specifies the base set of SRv6 behaviors that enables the creation of interoperable overlays with underlay optimization. National Institute of Advanced Industrial Science and Technology (AIST) GMO CONNECT Inc. GMO CONNECT Inc. "Secure hybrid network monitoring - Problem statement" identifies challenges in securing and monitoring networks deployed across hybrid and mixed cloud environments. This document introduces the Path Characteristics Service (PCS), a framework that enables applications and operators to obtain and evaluate verifiable information about the characteristics of the network paths they use. It outlines a non- normative architecture and interfaces for PCS and explains how PCS can help address the identified challenges; it does not define protocol requirements. ISO/IEC

Acknowledgments This work is based on results obtained from a project JPNP23013 commissioned by the New Energy and Industrial Technology Development Organization (NEDO).