This document defines a set of ECMAScript APIs in WebIDL to extend the WebRTC 1.0 API to enable user agents to support scalable video coding (SVC).

The API is based on preliminary work done in the W3C ORTC Community Group.

Introduction

This specification extends the WebRTC specification [[WEBRTC]] to enable configuration of encoding parameters for scalable video coding (SVC). Since SVC bitstreams are self-describing and SVC-capable codecs implemented in browsers require that compliant decoders be capable of decoding any legal encoding sent by an encoder, this specification does not support decoder configuration. However, it is possible for decoders that cannot decode any legal bitstream to describe the supported scalability modes.

This specification defines conformance criteria that apply to a single product: the user agent that implements the interfaces that it contains.

Conformance requirements phrased as algorithms or specific steps may be implemented in any manner, so long as the end result is equivalent. (In particular, the algorithms defined in this specification are intended to be easy to follow, and not intended to be performant.)

Implementations that use ECMAScript to implement the APIs defined in this specification MUST implement them in a manner consistent with the ECMAScript Bindings defined in the Web IDL specification [[WEBIDL]], as this specification uses that specification and terminology.

Terminology

The term simulcast envelope refers to the maximum number of simulcast streams and the order of the encoding parameters.

This specification references objects, methods, internal slots and dictionaries defined in [[!WEBRTC]]

For Scalable Video Coding (SVC), the terms single-session transmission (SST) and multi-session transmission (MST) are defined in [[RFC6190]]. This specification only supports SST but not MST.

The term Single Real-time Transport Protocol (RTP) stream Single Transport (SRST), defined in [[RFC7656]] Section 3.7, refers to SVC implementations that transmit all layers within a single transport, using a single RTP stream and synchronization source (SSRC). The term Multiple RTP stream Single Transport (MRST), also defined in [[RFC7656]] Section 3.7, refers to implementations that transmit all layers within a single transport, using multiple RTP streams with a distinct SSRC for each layer. This specification only supports SRST transport, not MRST. Codecs with RTP payload specifications supporting SRST transport include VP8 [[RFC7741]], VP9 [[VP9-PAYLOAD]], AV1 [[AV1-RTP]] and H.264/SVC [[RFC6190]].

Operational model

This specification extends [[!WEBRTC]] to enable configuration of encoding parameters for Scalable Video Coding (SVC), as well as discovery of the SVC capabilities of both an encoder and decoder, by extending the {{RTCRtpEncodingParameters}} and {{RTCRtpCodecCapability}} dictionaries.

Since this specification does not change the behavior of WebRTC objects and methods, restrictions relating to Offer/Answer negotiation and encoding parameters remain, as described in [[!WEBRTC]] Section 5.2: "{{RTCRtpSender/setParameters()}} does not cause SDP renegotiation and can only be used to change what the media stack is sending or receiving within the envelope negotiated by Offer/Answer."

The configuration of SVC-capable codecs implemented in browsers fits within this restriction. Codecs such as VP8 [[RFC6386]], VP9 [[VP9]] and AV1 [[AV1]] mandate support for SVC and require a compliant decoder to be able to decode any compliant encoding that an encoder can send. Therefore, for these codecs there is no need to configure the decoder or to negotiate SVC support within Offer/Answer, enabling encoding parameters to be used for SVC configuration.

Error handling

[[!WEBRTC]] Section 5.2 describes error handling in {{RTCRtpSender/setParameters()}}, including use of {{RTCError}} to indicate a {{RTCErrorDetailType/"hardware-encoder-error"}} due to an unsupported encoding parameter, as well as {{OperationError}} for other errors. Implementations of this specification utilize {{RTCError}} and {{OperationError}} in the prescribed manner when an invalid {{RTCRtpEncodingParameters/scalabilityMode}} value is provided to {{RTCRtpSender/setParameters()}} or {{RTCPeerConnection/addTransceiver()}}.

When the {{RTCPeerConnection/addTransceiver()}} and {{RTCRtpTransceiver/setCodecPreferences()}} methods are called prior to conclusion of the Offer/Answer negotiation, the negotiated codec and its capabilities may not be known. In this situation the {{RTCRtpEncodingParameters/scalabilityMode}} values configured in {{RTCRtpTransceiverInit/sendEncodings}} may not be supported by the eventually negotiated codec. However, an error will result only if the requested {{RTCRtpEncodingParameters/scalabilityMode}} value is invalid for any supported codec. To determine whether the requested {{RTCRtpEncodingParameters/scalabilityMode}} values have been applied, an application can call the {{RTCRtpSender.getParameters()}} method after negotiation has completed and the sending codec has been determined. If the configuration is not satisfactory, the {{RTCRtpSender/setParameters()}} method can be used to change it.

Note that where SVC support is negotiated in SDP Offer/Answer, {{RTCRtpSender/setParameters()}} can only change {{RTCRtpEncodingParameters/scalabilityMode}} values within the envelope negotiated by Offer/Answer, resulting in an error if the requested {{RTCRtpEncodingParameters/scalabilityMode}} value is outside this envelope.

Negotiation

So as to ensure that the desired {{RTCRtpEncodingParameters/scalabilityMode}} values can be applied, {{RTCRtpTransceiver/setCodecPreferences()}} can be used to limit the negotiated codecs to those supporting the desired configuration. For example, if temporal scalability is desired along with spatial simulcast, when {{RTCPeerConnection/addTransceiver()}} is called, {{RTCRtpTransceiverInit/sendEncodings}} can be configured to send multiple simulcast streams with different resolutions, with each stream utilizing temporal scalability. If only the VP8, VP9 and AV1 codec implementations support temporal scalability, {{RTCRtpTransceiver/setCodecPreferences()}} can be used to remove the H.264/AVC codec from the Offer, guaranteeing that a codec supporting temporal scalability is negotiated.

There are situations where a peer may only support reception of a subset of codecs and scalability modes. For example, an SFU that parses codec payloads may only support temporal scalability modes with the VP8 and VP9 codecs. As another example, a browser that can decode any VP8 scalability mode may not support VP9 or AV1 at all, and may only support decoding of H.264/SVC temporal scalability modes. In these situations, the {{RTCRtpReceiver}}'s getCapabilities method can be used to determine the scalability modes supported by the {{RTCRtpReceiver}}, and the {{RTCRtpSender}}'s getCapabilities method can be used to determine the scalability modes supported by the {{RTCRtpSender}}. After exchanging capabilities and computing their intersection, the application can compute the intersection of the {{RTCRtpEncodingParameters/scalabilityMode}} values supported by both the local and remote peers. These values can then be provided to the {{RTCPeerConnection/addTransceiver()}} and {{RTCRtpSender/setParameters()}} methods.

Dictionary extensions

RTCRtpEncodingParameters Dictionary Extensions

partial dictionary RTCRtpEncodingParameters {
             DOMString           scalabilityMode;
};

Dictionary RTCRtpEncodingParameters Members

scalabilityMode of type {{DOMString}}

A case-sensitive identifier of the scalability mode to be used for this stream. The {{RTCRtpEncodingParameters/scalabilityMode}} selected MUST be one of the scalability modes supported for the codec, as indicated in {{RTCRtpCodecCapability}}. Scalability modes are defined in Section 6.

{{RTCRtpCodecCapability}} Dictionary Extensions

partial dictionary RTCRtpCodecCapability {
             sequence<DOMString>           scalabilityModes;
};

Dictionary {{RTCRtpCodecCapability}} Members

scalabilityModes of type sequence<{{DOMString}}>

An sequence of the scalability modes (defined in Section 6) supported by the encoder implementation.

In response to a call to {{RTCRtpSender}}.getCapabilities(kind), conformant implementations of this specification MUST return a sequence of scalability modes supported by each codec of that kind. If a codec does not support encoding of any scalability modes, then the {{scalabilityModes}} member is not provided.

In response to a call to {{RTCRtpReceiver}}.getCapabilities(kind), decoders that do not support decoding of scalability modes (e.g. an H.264/AVC decoder) or that are required to decode any scalability mode (such as compliant VP8, VP9 and AV1 decoders) omit the {{scalabilityModes}} member. However, decoders that only support decoding of a subset of scalability modes MUST return a sequence of the scalability modes supported by that codec.

The {{scalabilityModes}} sequence represents the scalability modes supported by a user agent. While the user agent's SVC capabilities are assumed to be static, this may not be the case for a Selective Forwarding Unit (SFU), whose ability to forward SVC modes may depend on the negotiated header extensions. For example, if the SFU cannot parse codec payloads (either because it is not designed to do so, or because the payloads are confidential), then negotiation of an RTP header extension (such as the AV1 Descriptor defined in Appendix A of [[AV1-RTP]] or frame marking [[FRAME-MARKING]]) may be required to enable the SFU to forward scalability modes. As a result, an application may choose to exchange {{scalabilityModes}} with an SFU after the initial offer/answer negotiation, so that the SFU's supported {{scalabilityModes}} can be determined, allowing the application to compute the intersection of supported {{scalabilityModes}}.

Scalability modes

The scalability modes supported in this specification, as well as their associated identifiers and characteristics, are provided in the table below. The syntax for naming scalability modes is case sensitive and is taken from [[AV1]] Section 6.7.5, which also provides additional information on the modes, including dependency diagrams.

While [[AV1]] and VP9 [[VP9-PAYLOAD]] implementations can support all the modes defined in the table, other codecs cannot. For example, VP8 [[RFC7741]] only supports temporal scalability (e.g. "L1T2", "L1T3"). H.264/SVC [[RFC6190]], which supports both temporal and spatial scalability, only permits transport of simulcast on distinct SSRCs, so that it does not support the "S" modes (where multiple encodings are transported on a single SSRC).

Scalability Mode Identifier Spatial Layers Resolution Ratio Temporal Layers Inter-layer dependency
"L1T2" 1 2
"L1T3" 1 3
"L2T1" 2 2:1 1 Yes
"L2T2" 2 2:1 2 Yes
"L2T3" 2 2:1 3 Yes
"L3T1" 3 2:1 1 Yes
"L3T2" 3 2:1 2 Yes
"L3T3" 3 2:1 3 Yes
"L2T1h" 2 1.5:1 1 Yes
"L2T2h" 2 1.5:1 2 Yes
"L2T3h" 2 1.5:1 3 Yes
"S2T1" 2 2:1 1 No
"S2T2" 2 2:1 2 No
"S2T3" 2 2:1 3 No
"S2T1h" 2 1.5:1 1 No
"S2T2h" 2 1.5:1 2 No
"S2T3h" 2 1.5:1 3 No
"S3T1" 3 2:1 1 No
"S3T2" 3 2:1 2 No
"S3T3" 3 2:1 3 No
"S3T1h" 3 1.5:1 1 No
"S3T2h" 3 1.5:1 2 No
"S3T3h" 3 1.5:1 3 No
"L3T2_KEY" 3 2:1 2 Yes
"L3T3_KEY" 3 2:1 3 Yes
"L4T5_KEY" 4 2:1 5 Yes
"L4T7_KEY" 4 2:1 7 Yes
"L3T2_KEY_SHIFT" 3 2:1 2 Yes
"L3T3_KEY_SHIFT" 3 2:1 3 Yes
"L4T5_KEY_SHIFT" 4 2:1 5 Yes
"L4T7_KEY_SHIFT" 4 2:1 7 Yes

Guidelines for addition of {{RTCRtpEncodingParameters/scalabilityMode}} values

When proposing a {{RTCRtpEncodingParameters/scalabilityMode}} value, the following principles should be followed:

  1. The proposed {{RTCRtpEncodingParameters/scalabilityMode}} MUST define entries to the table in Section 6, including values for the Scalabilty Mode Identifier, spatial and temporal layers, Resolution Ratio and Inter-layer dependency.
  2. The Scalability Mode Identifier SHOULD be consistent with naming schemes used in existing specifications such as [[AV1]] Section 6.7.5. The AV1 naming scheme utilizes LxTy to denote a {{RTCRtpEncodingParameters/scalabilityMode}} with x spatial layers using a 2:1 resolution ratio and y temporal layers. LxTyh denotes x spatial layers with a 1.5:1 resolution ratio and y temporal layers. SxTy denotes a {{RTCRtpEncodingParameters/scalabilityMode}} with x simulcast encodings with a 2:1 resolution ratio, with each simulcast encoding containing y temporal layers. SxTy denotes a 1.5:1 resolution ratio. In addition, [[AV1]] Section 6.7.5 defines LxTy_KEY and LxTy_KEY_SHIFT modes.
  3. If the new {{RTCRtpEncodingParameters/scalabilityMode}} does not reference a dependency diagram defined within an existing specification, then a dependency diagram MUST be supplied.

Examples

Spatial Simulcast and Temporal Scalability

const signaling = new SignalingChannel(); // handles JSON.stringify/parse
const constraints = {audio: true, video: true};
const configuration = {'iceServers': [{'urls': 'stun:stun.example.org'}]};
let pc;

// call start() to initiate
async function start() {
  pc = new RTCPeerConnection(configuration);

  // let the "negotiationneeded" event trigger offer generation
  pc.onnegotiationneeded = async () => {
    try {
      await pc.setLocalDescription();
      // send the offer to the other peer
      signaling.send({description: pc.localDescription});
    } catch (err) {
      console.error(err);
    }
  };

  try {
    // get a local stream, show it in a self-view and add it to be sent
    const stream = await navigator.mediaDevices.getUserMedia(constraints);
    selfView.srcObject = stream;
    pc.addTransceiver(stream.getAudioTracks()[0], {direction: 'sendonly'});
    pc.addTransceiver(stream.getVideoTracks()[0], {
      direction: 'sendonly',
   // Example of 3 spatial simulcast layers + 3 temporal layers with
   // an SSRC and RID for each simulcast layer
      sendEncodings: [
        {rid: 'q', scaleResolutionDownBy: 4.0, scalabilityMode: 'L1T3'}
        {rid: 'h', scaleResolutionDownBy: 2.0, scalabilityMode: 'L1T3'},
        {rid: 'f', scaleResolutionDownBy: 1.0, scalabilityMode: 'L1T3'},
      ]
   // Example of 3 spatial simulcast layers + 3 temporal layers on a single SSRC
   //    sendEncodings: [
   //       {scalabilityMode: 'S3T3'}
   //    ]             
    });
  } catch (err) {
    console.error(err);
  }
}

signaling.onmessage = async ({data: {description, candidate}}) => {
  try {
    if (description) {
      await pc.setRemoteDescription(description);
      // if we got an offer, we need to reply with an answer
      if (description.type == 'offer') {
        await pc.setLocalDescription();
        signaling.send({description: pc.localDescription});
      }
    } else if (candidate) {
      await pc.addIceCandidate(candidate);
    }
  } catch (err) {
    console.error(err);
  }
};       

Spatial and Temporal Scalability

// Example of 1 spatial layer + 2 temporal layers
var sendEncodings = [
  {scalabilityMode: 'L1T2'}
];

// Example of 2 spatial layers (with 2:1 ratio) + 3 temporal layers
var sendEncodings = [
  {scalabilityMode: 'L2T3'}
];

SVC Encoder Capabilities

// Capabilities returned by RTCRtpSender.getCapabilities('video').codecs[]
  "codecs": [
    {
      "clockRate": 90000,
      "mimeType": "video/VP8",
      "scalabilityModes": [
        "L1T2",
        "L1T3"
      ]
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=96"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/VP9",
      "scalabilityModes": [
        "L1T2",
        "L1T3",
        "L2T1",
        "L2T2",
        "L2T3",
        "L3T1",
        "L3T2",
        "L3T3",
        "L1T2h",
        "L1T3h",
        "L2T1h",
        "L2T2h",
        "L2T3h"
      ]
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=98"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/H264",
      "sdpFmtpLine": "packetization-mode=1;profile-level-id=42001f;level-asymmetry-allowed=1"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=100"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/H264",
      "sdpFmtpLine": "packetization-mode=0;profile-level-id=42001f;level-asymmetry-allowed=1"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=102"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/H264",
      "sdpFmtpLine": "level-asymmetry-allowed=1;profile-level-id=42e01f;packetization-mode=1"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=104"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/H264",
      "sdpFmtpLine": "level-asymmetry-allowed=1;profile-level-id=42e01f;packetization-mode=0"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=106"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/H264",
      "sdpFmtpLine": "level-asymmetry-allowed=1;profile-level-id=4d0032;packetization-mode=1"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=108"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/H264",
      "sdpFmtpLine": "level-asymmetry-allowed=1;profile-level-id=640032;packetization-mode=1"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=110"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/red"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=112"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/ulpfec"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/AV1",
      "scalabilityModes": [
        "L1T2",
        "L1T3",
        "L2T1",
        "L2T2",
        "L2T3",
        "L3T1",
        "L3T2",
        "L3T3",
        "L1T2h",
        "L1T3h",
        "L2T1h",
        "L2T2h",
        "L2T3h",
        "S2T1",
        "S2T2",
        "S2T3",
        "S3T1",
        "S3T2",
        "S3T3",
        "S2T1h",
        "S2T2h",
        "S2T3h",
        "S3T1h",
        "S3T2h",
        "S3T3h"
      ]
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=113"
    }
]

SFU Capabilities

// RTCRtpReceiver.getCapabilities('video').codecs[] returned by 
// SFU that only supports forwarding of VP8, VP9 and AV1 temporal scalability modes
 "codecs": [
    {
      "clockRate": 90000,
      "mimeType": "video/VP8",
      "scalabilityModes": [
        "L1T2",
        "L1T3"
      ]
    },
    {
      "clockRate": 90000,
      "mimeType": "video/VP9",
      "scalabilityModes": [
        "L1T2",
        "L1T3",
        "L1T2h",
        "L1T3h"
      ]
    },
    {
      "clockRate": 90000,
      "mimeType": "video/AV1",
      "scalabilityModes": [
        "L1T2",
        "L1T3",
        "L1T2h",
        "L1T3h"
      ]
    }
]

SVC Decoder Capabilities

// RTCRtpReceiver.getCapabilities('video').codecs[] returned by a browser that can
// support all scalability modes within the VP8 and VP9 codecs.
  "codecs": [
    { 
      "clockRate": 90000,
      "mimeType": "video/VP8"
    },
    { 
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=96"
    },
    { 
      "clockRate": 90000,
      "mimeType": "video/VP9"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=98"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/H264",
      "sdpFmtpLine": "packetization-mode=1;profile-level-id=42001f;level-asymmetry-allowed=1"
    },

    ...
]

Privacy and Security Considerations

This section is non-normative; it specifies no new behaviour, but instead summarizes information already present in other parts of the specification. WebRTC protocol security considerations are described in [[RTCWEB-SECURITY-ARCH]] and the security and privacy considerations for the WebRTC APIs are described in [[WEBRTC]] Section 13.

Persistent information

The WebRTC API exposes information about the underlying media system via the {{RTCRtpSender}}.getCapabilities and {{RTCRtpReceiver}}.getCapabilities methods, including detailed and ordered information about the codecs that the system is able to produce and consume. The WebRTC-SVC extension adds the {{RTCRtpCodecCapability/scalabilityModes}} supported by the {{RTCRtpSender}} to that information, which is persistent across time, therefore increasing the fingerprint surface. Since for SVC codecs implemented in WebRTC compliant decoders are required to be able to decode all scalability modes, additional information is not provided relating to the {{RTCRtpReceiver}}.

Since for SVC codecs implemented in WebRTC the use of scalable coding tools is not negotiated and is independent of the supported profiles, the {{RTCRtpCodecCapability/scalabilityModes}} supported by the {{RTCRtpSender}} does not provide additional information on the underlying hardware. Since browsers may differ in their support for SVC modes, the supported {{RTCRtpCodecCapability/scalabilityModes}} may initially permit differentiation between browsers. This additional fingerprint surface is expected to decrease over time as this specification is more widely implemented.