Scalable Video Coding (SVC) Extension for WebRTC

This document defines a set of ECMAScript APIs in WebIDL to extend the WebRTC specification to enable configuration of encoding parameters for Scalable Video Coding (SVC). Discovery of SVC encoder and decoder capabilities is handled by the Media Capabilities specification.

Configuration

This specification enables the configuration of encoding parameters for SVC by extending the {{RTCRtpEncodingParameters}} dictionary.

{{RTCRtpEncodingParameters}} Dictionary Extensions

partial dictionary RTCRtpEncodingParameters {
             DOMString scalabilityMode;
};

Dictionary {{RTCRtpEncodingParameters}} Members

scalabilityMode of type {{DOMString}}: A case-sensitive identifier of the scalability mode to be used for this stream. Scalability modes are defined in Section 5.

Behavior

[[!WEBRTC]] describes error handling in {{RTCPeerConnection/addTransceiver()}} (Section 5.1) and {{RTCRtpSender/setParameters()}} (Section 5.2), including use of {{RTCError}} to indicate a {{RTCErrorDetailType/"hardware-encoder-error"}} due to an unsupported encoding parameter, as well as other errors. Implementations utilize {{RTCError}} and other errors in the prescribed manner when an invalid {{RTCRtpEncodingParameters/scalabilityMode}} value is provided to {{RTCRtpSender/setParameters()}} or {{RTCPeerConnection/addTransceiver()}}.

[[!WEBRTC]] Section 5.1 describes validation of {{RTCRtpTransceiverInit/sendEncodings}} within {{RTCPeerConnection/addTransceiver()}}. To validate {{RTCRtpEncodingParameters/scalabilityMode}}, add the following steps after step 3 of [=RTCPeerConnection/addTransceiver sendEncodings validation steps=]:

If sendEncodings contains any encoding with a {{RTCRtpEncodingParameters}}.codec value codec [=map/exists=] and where the same encoding's {{RTCRtpEncodingParameters/scalabilityMode}} value is not supported by codec, [= exception/throw =] an {{OperationError}}.
Else if sendEncodings contains any encoding whose {{RTCRtpEncodingParameters/scalabilityMode}} value is not supported by any codec in the [=RTCRtpSender/list of implemented send codecs=] for kind, [= exception/throw =] an {{OperationError}}.
If {{RTCRtpEncodingParameters}} stored in sendEncodings contains more than 1 encoding with an {{RTCRtpEncodingParameters/active}} member with a value of true and sendEncodings contains any encoding whose {{RTCRtpEncodingParameters/scalabilityMode}} value represents an "S mode" and whose {{RTCRtpEncodingParameters/active}} member has a value of true, [= exception/throw =] an {{OperationError}}.

When the {{RTCPeerConnection/addTransceiver()}} and {{RTCRtpTransceiver/setCodecPreferences()}} methods are called prior to conclusion of the Offer/Answer negotiation, the negotiated codec and its capabilities may not be known. In this situation the {{RTCRtpEncodingParameters/scalabilityMode}} values configured in {{RTCRtpTransceiverInit/sendEncodings}} may not be supported by the eventually negotiated codec. However, an error will only result if the requested {{RTCRtpEncodingParameters/scalabilityMode}} value is invalid for any supported codec, or if mixed simulcast transport is requested.

So as to ensure that the desired {{RTCRtpEncodingParameters/scalabilityMode}} values can be applied, {{RTCRtpTransceiver/setCodecPreferences()}} can be used to prefer or only include codecs supporting the desired configuration. For example, if temporal scalability is desired along with spatial simulcast, when {{RTCPeerConnection/addTransceiver()}} is called, {{RTCRtpTransceiverInit/sendEncodings}} can be configured to send multiple simulcast streams with different resolutions, with each stream utilizing temporal scalability. If only the VP8, VP9 and AV1 codec implementations support temporal scalability, {{RTCRtpTransceiver/setCodecPreferences()}} can be used to remove the H.264/AVC codec from the Offer, improving the chances that a codec supporting temporal scalability is negotiated.

When {{RTCRtpTransceiverInit/sendEncodings}} is used to request the sending of multiple simulcast streams using {{RTCPeerConnection/addTransceiver()}}, an "S mode" cannot be requested. The browser may only be configured to send simulcast encodings with multiple SSRCs and RIDs, or alternatively, to send all simulcast encodings on a single RTP stream. Simultaneously using both simulcast transport techniques is not permitted.

[[!WEBRTC]] Section 5.2 describes validation of parameters within {{RTCRtpSender/setParameters()}}. Insert the following conditions under which the operation causes a promise rejected with an {{InvalidModificationError}} (step 4) of [=RTCRtpSender/setParameters validation steps=]:

If encodings contains any encoding with an existng {{RTCRtpEncodingParameters}}.codec value codec, where the same encoding's {{RTCRtpEncodingParameters/scalabilityMode}} value is not supported by codec.
Else if sender.{{RTCRtpSender/[[SendCodecs]]}} [=map/is empty=] and encodings contains any encoding whose {{RTCRtpEncodingParameters/scalabilityMode}} value is not supported by any codec in the [=RTCRtpSender/list of implemented send codecs=] for kind.
Else if sender.{{RTCRtpSender/[[SendCodecs]]}} [=map/is empty | is not empty=] and encodings contains an encoding whose {{RTCRtpEncodingParameters/scalabilityMode}} value is not supported by the codec used for the encoding's RTP stream.
If {{RTCRtpEncodingParameters}} stored in encodings contains more than one encoding with an {{RTCRtpEncodingParameters/active}} member with a value of true and encodings contains any encoding whose {{RTCRtpEncodingParameters/scalabilityMode}} value represents an "S mode" and whose {{RTCRtpEncodingParameters/active}} member has a value of true.

The [="L1T1"=] scalability mode enables SVC encoding to be turned off using {{RTCRtpSender/setParameters()}}. If [="L1T1"=] is set using {{RTCRtpSender/setParameters()}} then it will be returned in response to {{RTCRtpSender/getParameters()}}.

Before the initial negotiation has completed, {{RTCRtpSender/getParameters()}} returns the {{RTCRtpEncodingParameters/scalabilityMode}} value for each encoding in encodings, as last set by {{RTCPeerConnection/addTransceiver()}} or {{RTCRtpSender/setParameters()}}. If no {{RTCRtpEncodingParameters/scalabilityMode}} value was provided for an encoding in encodings, or if a value was not successfully set, then {{RTCRtpSender/getParameters()}} will not return a {{RTCRtpEncodingParameters/scalabilityMode}} value for that encoding.

After the initial negotiation has completed, {{RTCRtpSender/getParameters()}} returns the currently configured {{RTCRtpEncodingParameters/scalabilityMode}} value for each encoding in encodings which had a value before the initial negotiation. This MAY be different from the values requested in {{RTCPeerConnection/addTransceiver()}} or {{RTCRtpSender/setParameters()}}. For example, if the codecs selected during negotiation do not include an encoder supporting the desired {{RTCRtpEncodingParameters/scalabilityMode}} value, the user agent MAY select another value. If the configuration is not satisfactory, {{RTCRtpSender/setParameters()}} can be used to change it.

If {{RTCPeerConnection/addTransceiver()}} or {{RTCRtpSender/setParameters()}} did not provide a {{RTCRtpEncodingParameters/scalabilityMode}} value for an encoding in encodings, then after the initial negotiation has completed, {{RTCRtpSender/getParameters()}} will not return a {{RTCRtpEncodingParameters/scalabilityMode}} value and the encoder will use the default {{RTCRtpEncodingParameters/scalabilityMode}} of the codec for that encoding's RTP stream. The default {{RTCRtpEncodingParameters/scalabilityMode}} for each codec is implementation dependent. The default {{RTCRtpEncodingParameters/scalabilityMode}} SHOULD be one of the temporal scalability modes (e.g. [="L1T1"=],[="L1T2"=],[="L1T3"=], etc.).

Negotiation issues

SVC is most often employed in video conferencing, where a conferencing server such as a SFM selectively forwards layers to participants based on their device characteristics and available bandwidth. In such an environment, the application will negotiate codecs to be sent and received with the conferencing server. However, since scalability modes are not negotiated in Offer/Answer, the application needs to determine which scalability modes the browser and conference server support by other means.

The [[?Media-Capabilities]] API enables discovery of encoder and decoder support for scalable video coding. {{VideoConfiguration//scalabilityMode}} is used to query whether an encoder supports a {{RTCRtpEncodingParameters/scalabilityMode}} value, indicating whether it is "supported", "smooth" and "power efficient".

The [[?Media-Capabilities]] API also provides information on decoder support for spatial scalablity modes. {{VideoConfiguration/spatialScalability}} indicates whether a decoder has the ability to support spatial prediction, which requires the ability to use frames of a resolution different than the current resolution as a dependency. If {{VideoConfiguration/spatialScalability}} is set to true, the decoder can decode any {{RTCRtpEncodingParameters/scalabilityMode}} value supported by the encoder. If {{VideoConfiguration/spatialScalability}} is set to false or is absent, the decoder cannot decode spatial scalability modes, but can can decode all other {{RTCRtpEncodingParameters/scalabilityMode}} values supported by the encoder.

Once an application has determined which combination of codecs and {{RTCRtpEncodingParameters/scalabilityMode}} values it has available to use, it needs to determine which of these combinations are supported by the SFM. One way to handle this is for the SFM to indicate the combination of codecs and scalability modes it can forward in the form of receiver capabilities After receiving the SFM's capabilities, the application can compute the intersection of codec and {{RTCRtpEncodingParameters/scalabilityMode}} values supported by the browser's {{RTCRtpSender}} and the SFM's receiver. This can be used to determine the arguments passed to the browser's {{RTCPeerConnection/addTransceiver()}} and {{RTCRtpSender/setParameters()}} methods.

Here are some examples:

An SFM that parses codec payloads may only support the H.264/AVC codec without scalability and the VP8 codec with temporal scalability. On the other hand, the browser may be able to encode VP8 with temporal scalability, VP9 and AV1 with temporal and spatial scalability and H.264/AVC with temporal scalability. In this example, an application desiring to use SVC would only be able to encode VP8 with temporal scalability.
An SFM may only support single stream simulcast using the [="S2T1"=] and [="S2T1h"=] scalability modes with the AV1 codec, while the browser may support encoding single stream simulcast using the [="S2T1"=], [="S2T1h"=], [="S3T1"=] and [="S3T1h"=] modes with both the VP9 and AV1 codecs. In this example, an application would only be able to use single-stream simulcast with a maximum of two layers with the AV1 codec. If the application prefers to utilize three layers, then it may decide to forgo use of single stream simulcast and negotiate multi-stream simulcast instead.

There are situations in which computing the intersection of the browser and SFM capabilities needs to take RTP header extensions into account. If the SFM cannot parse codec payloads (either because it is not designed to do so, or because the payloads are encrypted), then negotiation of an RTP header extension (such as the AV1 Dependency Descriptor defined in Appendix A of [[?AV1-RTP-SPEC]]) could be required for the SFM to forward a particular codec. To take this into account, the RTP header extensions required for forwarding of a codec could be added to the SFM's receiver capabilities. The application could then compute the intersection of codec, header extension and {{RTCRtpEncodingParameters/scalabilityMode}} values supported by the browser's {{RTCRtpSender}} and the SFM's receiver.

Scalability modes

The {{RTCRtpEncodingParameters/scalabilityMode}} values supported in this specification, as well as their associated identifiers and characteristics, are provided in the table below. The names of the {{RTCRtpEncodingParameters/scalabilityMode}} values (which are case sensitive) are provided, along with the scalability mode identifiers assigned in [[?AV1]] Section 6.7.5, and links to dependency diagrams provided in Section 9.

While the [[?AV1]] and VP9 [[?VP9]] specifications support all the {{RTCRtpEncodingParameters/scalabilityMode}} values defined in the table, other codec specifications do not. For example, VP8 [[?RFC6386]], H.264 [[?RFC6184]] and H.265 [[?RFC7798]] only support temporal scalability (e.g. [="L1T2"=], [="L1T3"=]). Also VP8 [[?RFC6386]], H.264 [[?RFC6184]] and H.265 [[?RFC7798]] only permit transport of simulcast on distinct SSRCs, so that "S" modes (where multiple encodings are transported on a single RTP stream) are not supported.

Scalability Mode Identifier	Spatial Layers	Resolution Ratio	Temporal Layers	Inter-layer dependency	AV1 scalability_mode_idc
"L1T1"	1		1		N/A
"L1T2"	1		2		SCALABILITY_L1T2
"L1T3"	1		3		SCALABILITY_L1T3
"L2T1"	2	2:1	1	Yes	SCALABILITY_L2T1
"L2T2"	2	2:1	2	Yes	SCALABILITY_L2T2
"L2T3"	2	2:1	3	Yes	SCALABILITY_L2T3
"L3T1"	3	2:1	1	Yes	SCALABILITY_L3T1
"L3T2"	3	2:1	2	Yes	SCALABILITY_L3T2
"L3T3"	3	2:1	3	Yes	SCALABILITY_L3T3
"L2T1h"	2	1.5:1	1	Yes	SCALABILITY_L2T1h
"L2T2h"	2	1.5:1	2	Yes	SCALABILITY_L2T2h
"L2T3h"	2	1.5:1	3	Yes	SCALABILITY_L2T3h
"L3T1h"	3	1.5:1	1	Yes
"L3T2h"	3	1.5:1	2	Yes
"L3T3h"	3	1.5:1	3	Yes
"S2T1"	2	2:1	1	No	SCALABILITY_S2T1
"S2T2"	2	2:1	2	No	SCALABILITY_S2T2
"S2T3"	2	2:1	3	No	SCALABILITY_S2T3
"S2T1h"	2	1.5:1	1	No	SCALABILITY_S2T1h
"S2T2h"	2	1.5:1	2	No	SCALABILITY_S2T2h
"S2T3h"	2	1.5:1	3	No	SCALABILITY_S2T3h
"S3T1"	3	2:1	1	No	SCALABILITY_S3T1
"S3T2"	3	2:1	2	No	SCALABILITY_S3T2
"S3T3"	3	2:1	3	No	SCALABILITY_S3T3
"S3T1h"	3	1.5:1	1	No	SCALABILITY_S3T1h
"S3T2h"	3	1.5:1	2	No	SCALABILITY_S3T2h
"S3T3h"	3	1.5:1	3	No	SCALABILITY_S3T3h
"L2T2_KEY"	2	2:1	2	Yes	SCALABILITY_L3T2_KEY
"L2T2_KEY_SHIFT"	2	2:1	2	Yes	SCALABILITY_L3T2_KEY_SHIFT
"L2T3_KEY"	2	2:1	3	Yes	SCALABILITY_L3T3_KEY
"L2T3_KEY_SHIFT"	2	2:1	3	Yes	SCALABILITY_L3T3_KEY_SHIFT
"L3T1_KEY"	3	2:1	1	Yes
"L3T2_KEY"	3	2:1	2	Yes	SCALABILITY_L4T5_KEY
"L3T2_KEY_SHIFT"	3	2:1	2	Yes	SCALABILITY_L4T5_KEY_SHIFT
"L3T3_KEY"	3	2:1	3	Yes	SCALABILITY_L4T7_KEY
"L3T3_KEY_SHIFT"	3	2:1	3	Yes	SCALABILITY_L4T7_KEY_SHIFT

Guidelines for addition of {{RTCRtpEncodingParameters/scalabilityMode}} values

When proposing a {{RTCRtpEncodingParameters/scalabilityMode}} value, the following principles should be followed:

The proposed {{RTCRtpEncodingParameters/scalabilityMode}} MUST define entries to the table in Section 5, including values for the Scalabilty Mode Identifier, spatial and temporal layers, Resolution Ratio, Inter-layer dependency and the corresponding AV1 scalability_mode_idc value (if assigned).
The Scalability Mode Identifier SHOULD be consistent with the existing naming scheme, which utilizes LxTy to denote a {{RTCRtpEncodingParameters/scalabilityMode}} with x spatial layers using a 2:1 resolution ratio and y temporal layers. LxTyh denotes x spatial layers with a 1.5:1 resolution ratio and y temporal layers. SxTy denotes a {{RTCRtpEncodingParameters/scalabilityMode}} with x simulcast encodings with a 2:1 resolution ratio, with each simulcast encoding containing y temporal layers. SxTyh denotes a 1.5:1 resolution ratio. LxTy_KEY denotes a {{RTCRtpEncodingParameters/scalabilityMode}} with x spatial layers using a 2:1 resolution ratio and y temporal layers in which spatial layers only depend on lower spatial layers at a key frame. LxTy_KEY_SHIFT modes denotes a {{RTCRtpEncodingParameters/scalabilityMode}} with x spatial layers using a 2:1 resolution ratio and y temporal layers in which spatial layers only depend on lower spatial layers at a key frame and subsequent frames have their temporal identifier shifted upward.
A dependency diagram MUST be supplied, in the format provided in Section 9.

Examples

Spatial Simulcast and Temporal Scalability

This example extends [[WEBRTC]] Section 7.1 (Example 1) to demonstrate sending three spatial simulcast layers each with three temporal layers, using an SSRC and RID for each simulcast layer. Only the "sendEncodings" attribute is changed from the original example.

const signaling = new SignalingChannel(); // handles JSON.stringify/parse
const constraints = {audio: true, video: true};
const configuration = {'iceServers': [{'urls': 'stun:stun.example.org'}]};
let pc;

// call start() to initiate
async function start() {
  pc = new RTCPeerConnection(configuration);

  // let the "negotiationneeded" event trigger offer generation
  pc.onnegotiationneeded = async () => {
    try {
      await pc.setLocalDescription();
      // send the offer to the other peer
      signaling.send({description: pc.localDescription});
    } catch (err) {
      console.error(err);
    }
  };

  try {
    // get a local stream, show it in a self-view and add it to be sent
    const stream = await navigator.mediaDevices.getUserMedia(constraints);
    selfView.srcObject = stream;
    pc.addTransceiver(stream.getAudioTracks()[0], {direction: 'sendonly'});
    pc.addTransceiver(stream.getVideoTracks()[0], {
      direction: 'sendonly',
      sendEncodings: [
        {rid: 'q', scaleResolutionDownBy: 4.0, scalabilityMode: 'L1T3'},
        {rid: 'h', scaleResolutionDownBy: 2.0, scalabilityMode: 'L1T3'},
        {rid: 'f', scalabilityMode: 'L1T3'}
      ]
    });
  } catch (err) {
    console.error(err);
  }
}

signaling.onmessage = async ({data: {description, candidate}}) => {
  try {
    if (description) {
      await pc.setRemoteDescription(description);
      // if we got an offer, we need to reply with an answer
      if (description.type == 'offer') {
        await pc.setLocalDescription();
        signaling.send({description: pc.localDescription});
      }
    } else if (candidate) {
      await pc.addIceCandidate(candidate);
    }
  } catch (err) {
    console.error(err);
  }
};

This is an example with two spatial layers (with a 2:1 ratio) and three temporal layers.

let sendEncodings = [
  {scalabilityMode: 'L2T3'}
];

This is an example of mixed codec simulcast, with each simulcast layer having 3 temporal layers.

let sendEncodings = [
  {rid: 'q', codec: {clockRate: 90000, mimeType: 'video/AV1'}, scaleResolutionDownBy: 4.0, scalabilityMode: 'L1T3'},
  {rid: 'h', codec: {clockRate: 90000, mimeType: 'video/VP8'}, scaleResolutionDownBy: 2.0, scalabilityMode: 'L1T3'},
  {rid: 'f', codec: {clockRate: 90000, mimeType: 'video/VP8'}, scalabilityMode: 'L1T3'}
];

This is an example with three spatial simulcast layers each with three temporal layers on a single SSRC.

let sendEncodings = [
  {scalabilityMode: 'S3T3'}
]

SVC Encoder Capabilities

This is an example of {{MediaCapabilities/encodingInfo(configuration)}} returned by a browser implementing [[WEBRTC]] and [[?Media-Capabilities]].

const contentType = 'video/VP9';

const configuration = {
  type: 'webrtc',
  video: {
    contentType,
    width: 640,
    height: 480,
    bitrate: 10000,
    framerate: 29.97,
    scalabilityMode: 'L3T3_KEY'
  }
};

try {
  const info = await navigator.mediaCapabilities.encodingInfo(configuration);

  if (!info.supported) {
    console.log(`${contentType} is unsupported.`);
    return;
  }
  console.log(`${contentType} is ${info.smooth || 'NOT '}smooth, and ` +
              `${info.powerEfficient || 'NOT '}power efficient`);
} catch (err) {
  console.error(err, ' caused encodingInfo to fail');
}

SFM Capabilities

This is an example of receiver capabilities returned by an SFM that only supports forwarding of VP8, VP9 and AV1 temporal scalability modes.

 "codecs": [
    {
      "clockRate": 90000,
      "mimeType": "video/VP8",
      "scalabilityModes": [
        "L1T1",
        "L1T2",
        "L1T3"
      ]
    },
    {
      "clockRate": 90000,
      "mimeType": "video/VP9",
      "scalabilityModes": [
        "L1T1",
        "L1T2",
        "L1T3"
      ]
    },
    {
      "clockRate": 90000,
      "mimeType": "video/AV1",
      "scalabilityModes": [
        "L1T1",
        "L1T2",
        "L1T3"
      ]
    }
]