Internet-Draft Customer Experience Index for Evaluating March 2024
Liu, et al. Expires 2 September 2024 [Page]
Workgroup:
IP Performance Measurement
Internet-Draft:
draft-hz-ippm-cei-00
Published:
Intended Status:
Standards Track
Expires:
Authors:
S. Liu
Huawei Cloud
Y. Wang
Huawei
W. Sun
Huawei Cloud
X. Huang
Huawei Cloud
S. Zhou
Huawei Cloud
H. Huang, Ed.
Huawei
T. Zhou, Ed.
Huawei

Customer Experience Index for Evaluating Network Quality for Cloud Applications

Abstract

This document outlines a unified Customer Experience Index (CEI) designed to assist cloud vendors in assessing network quality, reflecting the customer experience with cloud applications when accessed via the public network.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 2 September 2024.

Table of Contents

1. Introduction

This document introduces a unified Customer Experience Index (CEI) designed to assist cloud vendors in assessing the network quality that mirrors the customer experience of cloud applications when accessed via the public network. The CEI, once quantified, empowers cloud vendors to proactively enhance network services, aiding in network planning and construction. Furthermore, it enables cloud customers to distinguish the service quality of various cloud vendors, allowing them to select cost-effective services tailored to their applications.

Cloud vendors and cloud enterprises focus on different network indicators (Key Performance Index) used to anticipate the quality of customer experience regarding various applications(e.g., gaming, audio and video, online stores). However, KPIs only provide implicit information and cannot directly reflect the customers' perceived experience. Moreover, there is no unified evaluation method of customer experience based on common network KPIs in the industry. On the other hand, it is difficult for cloud vendors to directly access application-level Key Quality Index (KQI) data though it may explicitly imply customer experience. As the number of enterprises who deploy the service in the cloud gradually increases, there is growing demand for deriving authentic customer experience from basic network metrics to facilitate network optimizations.

A significant gap persists between network KPIs and customer experience. The primary network KPIs accessible to most cloud vendors—network latency, packet-loss rate, and jitter—encompass three categories. Considering multiple dimensions of network quality proves beneficial for end-users. [I-D.teigen-ippm-app-quality-metric-reqs] Customers' demands for experience quality vary across different cloud services and are linked to specific KPIs. For instance, those accessing real-time interactive games prioritize network latency; those utilizing video-on-demand services are more concerned with packet-loss rate than latency; and those engaging with cloud storage services consider both latency and packet-loss rate. No single KPI can provide an accurate reflection of the experience for diverse services. Both cloud vendors and customers seek unified evaluation standards for experience quality when accessing cloud services.

This document accounts for a range of key network-observable indicators, offering a unified, objective, and comprehensive CEI to help enterprises evaluate customer experience through measurable network KPIs in a reasonable and fair manner. Predominantly based on three network KPIs—network latency, packet-loss rate, and jitter—the CEI aims to thoroughly assess network quality. The allocation of weights to these KPIs within the CEI can be customized to suit different application scenarios.

1.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

1.2. Terminology

CEI: Customer Experience Index

KPI: Key Performance Indicator

KQI: Key Quality Indicator

2. Motivation

Cloud service providers aim to precisely evaluate the network quality, crucial to the customer experience of their cloud services, and implement targeted improvements to their network infrastructure. Similarly, cloud customers seek a unified and fair scoring standard to guide their selection of superior cloud services. But they currently face some challenges:

3. Customer Experience Index

This document introduces the Customer Experience Index (CEI), a measure reflecting customer experience with cloud services. It enables cloud service providers to swiftly evaluate their service quality through a synthesis of key network metrics.

3.1. Observation

Customer experience often exhibits distinct zones—sensitive and smooth—based on their response to changes in specific indicators. For instance, in scenarios sensitive to latency, such as cloud gaming, customer satisfaction remains high within an acceptable latency range (smooth zone). However, exceeding a certain latency threshold leads to a sharp decline in experience (sensitive zone).

Accordingly, the CEI employs an S-Curve for its calculation, a method prevalent in biostatistics and sociology for modeling ecosystems and urbanization trends. The S-Curve, particularly through the Sigmoid function, effectively maps values to a (0,1) interval, delineating two smooth zones and a sensitive zone, mirroring the nuanced nature of customer experience.

     ^
   1-|  '''---...           ┓
     |           ''--..     ┃- smooth zone
     |                 '.   ┛
     |                   -       ┓
     |                    .      ┃- sensitive zone
     |                     .     ┃
     |                      -    ┛
     |                   ┏   '.
     |      smooth zone -┃     ''--..
   0-|                   ┗           '''---...
     +-------------------------------------------->
Figure 1: S-curve Example

Therefore, this document designs the following formula to evaluate customer experience for each network KPI:

f(x) = (1+e^b)/(1+e^(a*x+b))

  • x is the value measured by network KPI, a and b are tunable parameters, and f(x) represents S-curve for certain KPI.

  • Parameter a represents the overall slope of the curve, mainly affecting the range of the central sensitive area.

  • Parameter b represents the offset and scaling of the curve. The initial smoothing area can be shielded via tuning b, which can express KPIs that immediately enter the sensitive area from the very beginning.

3.2. Unified Index

Each KPI is represented by a distinct S-curve to ensure independence among the indicators. Specifically, unique S-curves for network latency, packet-loss rate, and jitter are created by assigning specific parameters (a and b), offering tailored indexes for applications sensitive to these different metrics. The comprehensive CEI score is then derived by aggregating these three S-curves, each weighted appropriately:

CEI(x, y, z) = w1 * f1(x) + w2 * f2(y) + w3 * f3(z)

  • x, y, and z respectively indicate values of the three major network KPIs: network latency, packet-loss rate, and jitter.

  • f1, f2, f3 represent the three individual S-curves.

  • w1, w2, w3 represent the empirical weights.

3.3. Parameter Tuning

3.3.1. Weight Proportion

The CEI's flexibility allows for fine-tuning to meet specific application needs by adjusting its weight values (w1, w2, w3), enabling precise adaptation for various application categories. Typically, cloud customers engage in scenarios that are either sensitive to latency—like gaming applications—or to packet loss, such as audio and video streaming. For instance, in latency-sensitive scenarios, the weights for latency, packet-loss rate, and jitter could be adjusted to a ratio of 7:2:1 (w1:w2:w3); whereas for packet-loss-sensitive scenarios, a ratio of 2:7:1 (w1:w2:w3) might be more appropriate. This tailored approach allows the CEI to accurately assess network quality for different types of applications from a specific viewpoint (e.g., a fixed test point) across various cloud vendors.

3.3.2. Parameter a, b

The parameters a and b of the CEI formula can be fine-tuned via:

  • Determine the initial values of parameters a and b by fitting each KPI CEI curve based on a large amount of operational data.

  • Parameters a and b can be further tuned based on preferences of certain application class. For example, when the packet-loss rate is no higher than 𝑘, it is desired that CEI goes up as the network latency lowers. CEI can set tuning goals according to such preferences and fine-tune parameters a and b.

4. Security Considerations

TBD.

5. IANA Considerations

This document has no IANA actions.

6. References

6.1. Normative References

[RFC2119]
Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, , <https://www.rfc-editor.org/rfc/rfc2119>.
[RFC8174]
Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, , <https://www.rfc-editor.org/rfc/rfc8174>.

6.2. Informative References

[I-D.teigen-ippm-app-quality-metric-reqs]
Teigen, B. I. and M. Olden, "Requirements for a Network Quality Framework Useful for Applications, Users, and Operators", Work in Progress, Internet-Draft, draft-teigen-ippm-app-quality-metric-reqs-02, , <https://datatracker.ietf.org/doc/html/draft-teigen-ippm-app-quality-metric-reqs-02>.

Authors' Addresses

Sifa Liu
Huawei Cloud
China
Yaojing Wang
Huawei
China
Wei Sun
Huawei Cloud
China
Xiang Huang
Huawei Cloud
China
Shuai Zhou
Huawei Cloud
China
Hongyi Huang (editor)
Huawei
China
Tianran Zhou (editor)
Huawei
China