Integration of Speech Codec Enhancement Algorithms into the Opus Codec

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 2 January 2027.¶

Copyright Notice

Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

2. An Illustrative Example

We use the linear-adaptive coding enhancer (LACE) [lace-paper] as an illustrative example to highlight the specific challenges of integrating a speech codec enhancement algorithm into the Opus decoder. LACE is trained to enhance the output signal of the SILK decoder, the speech coding mode of Opus, and Figure 1 depicts a high-level overview of the Opus decoder with LACE added as an enhancement algorithm.¶

The first requirement for a speech coding enhancement algorithm concerns the performance of the algorithm itself. In this example it relates to the question of how the SILK decoder output compares to the LACE output. In [lace-paper] this has been evaluated on clean speech samples using a P.808 listening test [p.808] as well as the objective method PESQ, which showed consistent improvement for all tested bitrates. For a general enhancement algorithm it will be necessary to specify testing material and performance criteria to prevent unintended quality degradation of the Opus codec.¶

The second requirement concerns performance of the Opus decoder as a whole. Depending on the bitstream the decoder may have to perform mode switching, e.g. between SILK and CELT, or it may combine the SILK and CELT outputs when the codec operates in hybrid mode. Changes to the SILK output signal by an enhancement algorithm, such as added delay, phase shifts, or level alterations can therefore negatively impact the performance of the Opus decoder even if the first requirement is met. LACE solves this problem by adding no delay and by being approximately phase and level preserving. However, since many enhancement algorithms are non-causal and non-phase-preserving, these requirements may be too strict for a general enhancement algorithm.¶

The third requirement concerns interoperability. The Opus specification provides significant freedom for tuning the encoder and the presence of an enhancement algorithm in the decoder may change the optimal encoding choices significantly. In the present example encoding e.g. wideband content at 6 kb/s still leads to fair-to-good quality when using the LACE-enhanced decoder while the quality of a legacy decoder is significantly worse. To make full use of these new enhancement algorithms, such encoder tunings should be allowed but basic interoperability with legacy decoders or other enhanced decoders needs to be ensured.¶

                 ┌──────────────────────────────┐
                 │           Bitstream          │
                 └─────┬──────────────────┬─────┘
                       │                  │
                       ▼                  ▼
                 ┌───────────┐      ┌───────────┐
                 │   CELT    │      │   SILK    │
                 │  decoder  │      │  decoder  │
                 └─────┬─────┘      └─────┬─────┘
                       │                  │
                       │                  ▼
                       │            ┌───────────┐
                       │            │   LACE    │
                       │            └─────┬─────┘
                       │                  │
                       │                  ▼
                       │            ┌───────────┐
                       │            │ Resampler │
                       │            └─────┬─────┘
                       │                  │
                       ▼                  ▼
                 ┌──────────────────────────────┐
                 │        Mode Handling         │
                 └──────────────┬───────────────┘
                                │
                                ▼
                         decoded  signal

LACE has meanwhile been superseded by the non-linear adaptive coding enhancer (NoLACE) [nolace-paper] which shares all basic properties of LACE outlined above but provides higher quality. This stresses the advantage of specifying requirements for an enhancement algorithm over specifying the algorithm itself.¶

3. Definition of a SILK Enhancement Algorithm

A SILK enhancement algorithm denotes any algorithm that replaces the native SILK decoded output with an enhanced output signal, an example of which is depicted in Figure 1. Here, the native SILK decoded output refers to the signal that an unmodified SILK decoder would produce. A SILK enhancement algorithm is not restricted to operating on the native SILK decoded output alone; it may equally draw on internal quantities of the SILK decoder (such as gains, LSFs, pitch or excitation), modify such internal values, or replace parts of the SILK decoder altogether. The requirements in this document therefore constrain only the relation between the native SILK decoded output and the output of the enhancement algorithm. If the encoded content is wideband and the decoder sampling rate allows for a higher bandwidth than wideband, a SILK enhancement algorithm may also produce an output signal of higher bandwidth than the native SILK decoded output, replacing the resampler in Figure 1, or it may modify the combination of a SILK-decoded wideband signal with a CELT decoded highband signal in hybrid mode. Such bandwidth extension is only permitted for wideband speech; narrowband and mediumband content MUST NOT be extended. In any case, a SILK enhancement algorithm MUST NOT modify the output of pure CELT frames. A SILK enhancement algorithm whose output has a higher bandwidth than the native SILK decoded output will be referred to as extending, whereas a SILK enhancement algorithm whose output preserves the bandwidth of the native SILK decoded output will be referred to as non-extending. Furthermore, an Opus decoder including a SILK enhancement algorithm will be referred to as enhanced decoder. Note however, that simply resampling the signal to a higher sampling rate is neither considered enhancing nor extending.¶

5. IANA Considerations

This document updates the media type registration for the audio/opus media subtype defined in [RFC7587] by adding the two optional parameters below to its list of optional parameters. Both are informative parameters by which a receiver signals its decoder's intent to enhance (and, for extendedbandwidth, to extend) the received signal, so that the far-end encoder MAY select an operating point (for example, a lower bitrate or a wideband-only mode) that the enhancement can compensate for. The decoder is not bound by this signaling and may at runtime reduce or disable enhancement, for example when CPU load is high. The parameters do not change which bitstreams a decoder accepts, and by the requirement of Section 4.1.3 any stream that relies on them MUST remain intelligible when decoded by a legacy Opus decoder.¶

speechenhancement:

an integer from 0 to 40 indicating the effectiveness of the receiver's speech enhancement (see Section 3). It is calibrated at 6 kb/s: if the enhanced decoder at 6 kb/s achieves the quality of an unenhanced decoder at X kb/s, the value is round(10 * (X - 6) / 6). A value of 0 (the default when the parameter is absent) indicates no speech enhancement; 10 indicates that enhancement at 6 kb/s reaches the quality of an unenhanced decoder at 12 kb/s, and 40 (the maximum) corresponds to the quality of an unenhanced decoder at 30 kb/s or better.¶

extendedbandwidth:

an integer giving the upper limit, in kHz, of the audio bandwidth the receiver's decoder produces after blind bandwidth extension. It applies only to content coded in the speech coding mode; content coded in the general (music) coding mode is not affected by this parameter and is reproduced at the bandwidth selected by the encoder. Typical values follow the Opus bandwidth tiers: 8 (wideband, i.e. no extension and the default when the parameter is absent), 12 (super-wideband) and 20 (fullband). For example, an extending algorithm such as BBWENet [bbwenet-paper] that extends wideband speech to fullband would signal extendedbandwidth=20.¶

An offerer or answerer MAY include either parameter; a value that is absent, zero (speechenhancement) or 8 (extendedbandwidth) is equivalent to the corresponding baseline capability. An implementation that does not recognize these parameters ignores them, as required by [RFC7587].¶

6. Security Considerations

A SILK enhancement algorithm runs after the SILK decoder and post-processes (or replaces) its output. It introduces no new bitstream elements and does not change which packets a decoder accepts or rejects, so it does not expand the packet-parser attack surface and the security considerations of [RFC6716] apply unchanged.¶

The following additional considerations apply to the enhancement itself:¶

• Memory safety on attacker-influenced input: the enhancement operates on the decoder output and on internal parameters (such as gains, LSFs, pitch and excitation) that are ultimately derived from untrusted bitstreams. An implementation MUST handle degenerate or extreme values, including out-of-range or non-finite intermediates, without out-of-bounds access, and MUST run in bounded memory.¶
• Bounded, data-independent complexity: the example enhancement models (LACE, NoLACE and BBWENet) are of fixed size, so their per-frame cost is essentially constant and independent of the packet content. Future enhancement models SHOULD likewise avoid any data-dependent unbounded work, so that crafted streams cannot force excessive CPU usage; because the enhancement is gated by decoder complexity, a decoder can also bound or disable it.¶
• Model integrity: the model weights are part of the implementation and are not signalled in the bitstream, so they are not attacker-controlled through Opus packets. When the weights are loaded from an external artifact, their integrity should be ensured (for example through a verified hash). A corrupted model can only degrade audio quality, not compromise memory safety, provided inference is bounds-checked.¶

Authors' Addresses