This document defines a set of ECMAScript APIs in WebIDL to extend the WebRTC 1.0 API to enable user agents to support scalable video coding (SVC).
The API is based on preliminary work done in the W3C ORTC Community Group.
This specification extends the WebRTC specification [[WEBRTC]] to enable configuration of encoding parameters, as well as the discovery of Scalable Video Coding (SVC) encoder capabilities. The discovery of decoder capabilities and configuration of decoding parameters is not supported.
Since this specification does not change the behavior of WebRTC objects and methods, restrictions relating to Offer/Answer negotiation and encoding parameters remain, as described in [[WEBRTC]] Section 5.2: "{{RTCRtpSender/setParameters()}} does not cause SDP renegotiation and can only be used to change what the media stack is sending or receiving within the envelope negotiated by Offer/Answer."
The configuration of SVC-capable codecs implemented in browsers fits within this restriction. Codecs such as VP8 [[?RFC6386]], VP9 [[?VP9]] and AV1 [[?AV1]] do not negotiate SVC support within Offer/Answer, enabling encoding parameters to be used for SVC configuration.
This specification defines conformance criteria that apply to a single product: the user agent that implements the interfaces that it contains.
Conformance requirements phrased as algorithms or specific steps may be implemented in any manner, so long as the end result is equivalent. In particular, the algorithms defined in this specification are intended to be easy to follow, and not intended to be performant.
Implementations that use ECMAScript to implement the APIs defined in this specification MUST implement them in a manner consistent with the ECMAScript Bindings defined in the Web IDL specification [[WEBIDL]], as this specification uses that specification and terminology.
The term "simulcast envelope" is defined in [[!WEBRTC]] Section 5.4.1.
This specification references objects, methods, internal slots and dictionaries defined in [[!WEBRTC]].
For Scalable Video Coding (SVC), the terms "single-session transmission" (SST) and "multi-session transmission" (MST) are defined in [[?RFC6190]]. This specification only supports SST but not MST.
The term "Single Real-time Transport Protocol stream Single Transport" (SRST), defined in [[RFC7656]] Section 3.7, refers to SVC implementations that transmit all layers within a single transport, using a single Real-time Transport Protocol (RTP) stream and synchronization source (SSRC). The term "Multiple RTP stream Single Transport" (MRST), also defined in [[RFC7656]] Section 3.7, refers to implementations that transmit all layers within a single transport, using multiple RTP streams with a distinct SSRC for each layer. This specification only supports SRST, not MRST. Codecs with RTP payload specifications supporting SRST include VP8 [[?RFC7741]], VP9 [[?VP9-PAYLOAD]], AV1 [[?AV1-RTP]] and H.264/SVC [[?RFC6190]].
The term "S mode" refers to a scalability mode in which multiple encodings are sent on the same SSRC. This includes the [="S2T1"=], [="S2T1h"=], [="S2T2"=], [="S2T2h"=], [="S2T3"=], [="S2T3h"=], [="S3T1"=], [="S3T1h"=], [="S3T2"=], [="S3T2h"=], [="S3T3"=] and [="S3T3h"=] {{RTCRtpEncodingParameters/scalabilityMode}} values.
The term Selective Forwarding Middlebox (SFM) is defined in Section 3.7 of [[RFC7667]].
spatialScalability is an addition to {{VideoConfiguration}} in the Media Capabilities API.
This specification enables the configuration of encoding parameters for SVC by extending the {{RTCRtpEncodingParameters}} dictionary.
partial dictionary RTCRtpEncodingParameters { DOMString scalabilityMode; };
A case-sensitive identifier of the scalability mode to be used for this stream. Scalability modes are defined in Section 6.
[[!WEBRTC]] describes error handling in {{RTCPeerConnection/addTransceiver()}} (Section 5.1) and {{RTCRtpSender/setParameters()}} (Section 5.2), including use of {{RTCError}} to indicate a {{RTCErrorDetailType/"hardware-encoder-error"}} due to an unsupported encoding parameter, as well as other errors. Implementations utilize {{RTCError}} and other errors in the prescribed manner when an invalid {{RTCRtpEncodingParameters/scalabilityMode}} value is provided to {{RTCRtpSender/setParameters()}} or {{RTCPeerConnection/addTransceiver()}}.
[[!WEBRTC]] Section 5.1 describes validation of {{RTCRtpTransceiverInit/sendEncodings}} within {{RTCPeerConnection/addTransceiver()}}. To validate {{RTCRtpEncodingParameters/scalabilityMode}}, add the following steps after step 3 of [=RTCPeerConnection/addTransceiver sendEncodings validation steps=]:
codec
value codec [=map/exists=] and where the same encoding's {{RTCRtpEncodingParameters/scalabilityMode}}
value is not supported by codec, [= exception/throw =] an {{OperationError}}.
true
and
sendEncodings contains any encoding whose
{{RTCRtpEncodingParameters/scalabilityMode}} value
represents an "S mode" and whose
{{RTCRtpEncodingParameters/active}} member has a
value of true
, [= exception/throw =] an
{{OperationError}}.
When the {{RTCPeerConnection/addTransceiver()}} and {{RTCRtpTransceiver/setCodecPreferences()}} methods are called prior to conclusion of the Offer/Answer negotiation, the negotiated codec and its capabilities may not be known. In this situation the {{RTCRtpEncodingParameters/scalabilityMode}} values configured in {{RTCRtpTransceiverInit/sendEncodings}} may not be supported by the eventually negotiated codec. However, an error will only result if the requested {{RTCRtpEncodingParameters/scalabilityMode}} value is invalid for any supported codec, or if mixed simulcast transport is requested.
So as to ensure that the desired {{RTCRtpEncodingParameters/scalabilityMode}} values can be applied, {{RTCRtpTransceiver/setCodecPreferences()}} can be used to prefer or only include codecs supporting the desired configuration. For example, if temporal scalability is desired along with spatial simulcast, when {{RTCPeerConnection/addTransceiver()}} is called, {{RTCRtpTransceiverInit/sendEncodings}} can be configured to send multiple simulcast streams with different resolutions, with each stream utilizing temporal scalability. If only the VP8, VP9 and AV1 codec implementations support temporal scalability, {{RTCRtpTransceiver/setCodecPreferences()}} can be used to remove the H.264/AVC codec from the Offer, improving the chances that a codec supporting temporal scalability is negotiated.
When {{RTCRtpTransceiverInit/sendEncodings}} is used to request the sending of multiple simulcast streams using {{RTCPeerConnection/addTransceiver()}}, an "S mode" cannot be requested. The browser may only be configured to send simulcast encodings with multiple SSRCs and RIDs, or alternatively, to send all simulcast encodings on a single RTP stream. Simultaneously using both simulcast transport techniques is not permitted.
[[!WEBRTC]] Section 5.2 describes validation of parameters within {{RTCRtpSender/setParameters()}}. Insert the following conditions under which the operation causes a promise rejected with an {{InvalidModificationError}} (step 4) of [=RTCRtpSender/setParameters validation steps=]:
true
and
encodings contains any encoding whose
{{RTCRtpEncodingParameters/scalabilityMode}} value
represents an "S mode" and whose
{{RTCRtpEncodingParameters/active}} member has a
value of true
.
The [="L1T1"=] scalability mode enables SVC encoding to be turned off using {{RTCRtpSender/setParameters()}}. If [="L1T1"=] is set using {{RTCRtpSender/setParameters()}} then it will be returned in response to {{RTCRtpSender/getParameters()}}.
Before the initial negotiation has completed, {{RTCRtpSender/getParameters()}} returns the {{RTCRtpEncodingParameters/scalabilityMode}} value for each encoding in encodings, as last set by {{RTCPeerConnection/addTransceiver()}} or {{RTCRtpSender/setParameters()}}. If no {{RTCRtpEncodingParameters/scalabilityMode}} value was provided for an encoding in encodings, or if a value was not successfully set, then {{RTCRtpSender/getParameters()}} will not return a {{RTCRtpEncodingParameters/scalabilityMode}} value for that encoding.
After the initial negotiation has completed, {{RTCRtpSender/getParameters()}} returns the currently configured {{RTCRtpEncodingParameters/scalabilityMode}} value for each encoding in encodings which had a value before the initial negotiation. This MAY be different from the values requested in {{RTCPeerConnection/addTransceiver()}} or {{RTCRtpSender/setParameters()}}. For example, if the codecs selected during negotiation do not include an encoder supporting the desired {{RTCRtpEncodingParameters/scalabilityMode}} value, the user agent MAY select another value. If the configuration is not satisfactory, {{RTCRtpSender/setParameters()}} can be used to change it.
If {{RTCPeerConnection/addTransceiver()}} or {{RTCRtpSender/setParameters()}} did not provide a {{RTCRtpEncodingParameters/scalabilityMode}} value for an encoding in encodings, then after the initial negotiation has completed, {{RTCRtpSender/getParameters()}} will not return a {{RTCRtpEncodingParameters/scalabilityMode}} value and the encoder will use the default {{RTCRtpEncodingParameters/scalabilityMode}} of the codec for that encoding's RTP stream. The default {{RTCRtpEncodingParameters/scalabilityMode}} for each codec is implementation dependent. The default {{RTCRtpEncodingParameters/scalabilityMode}} SHOULD be one of the temporal scalability modes (e.g. [="L1T1"=],[="L1T2"=],[="L1T3"=], etc.).
The [[?Media-Capabilities]] API provides information on encoder support for spatial scalability modes. {{VideoConfiguration//scalabilityMode}} is used to query whether an encoder has the ability to support this mode. The SFM can provide information on the codecs and scalability modes it can decode by providing its receiver capabilities. After exchanging capabilities, the application can compute the intersection of codecs and {{RTCRtpEncodingParameters/scalabilityMode}} values supported by the browser's {{RTCRtpSender}} and the SFM's receiver. This can be used to determine the arguments passed to the browser's {{RTCPeerConnection/addTransceiver()}} and {{RTCRtpSender/setParameters()}} methods.
The [[?Media-Capabilities]] API also provides information on decoder support
for spatial scalablity modes. {{VideoConfiguration/spatialScalability}}
indicates whether a decoder has the ability to support spatial prediction,
which requires the ability to use frames of a resolution different than
the current resolution as a dependency. If {{VideoConfiguration/spatialScalability}}
is set to true
, the decoder can decode any
{{RTCRtpEncodingParameters/scalabilityMode}} value supported by the encoder.
If {{VideoConfiguration/spatialScalability}} is set to false
or is absent, the decoder cannot decode spatial scalability modes, but can
can decode all other {{RTCRtpEncodingParameters/scalabilityMode}} values
supported by the encoder.
There are situations where an SFM may only support reception of a subset of codecs and scalability modes. For example, an SFM that parses codec payloads may only support the H.264/AVC codec without scalability and the VP8 codec with temporal scalability. On the other hand, the browser may be able to encode VP8 with temporal scalability, VP9 with temporal and spatial scalability and or H.264/AVC with temporal scalability. In such a situation, an application desiring to use SVC would only be able to encode VP8 with temporal scalability.
Since sending simulcast encodings on a single stream is not negotiated within Offer/Answer, an application using SDP signaling needs to determine whether single stream simulcast transport is supported prior to the Offer/Answer negotiation. This can be handled by having the SFM send it's receiver capabilities to the application prior to Offer/Answer. This allows the application to determine whether single stream simulcast is supported, and if so, what scalability modes the SFM can handle. For example, an SFM that can only support reception of a maximum of 2 simulcast encodings on a single SSRC with the AV1 codec would only indicate support for the [="S2T1"=] and [="S2T1h"=] scalability modes in its receiver capabilities.
For an SFM the supported {{RTCRtpEncodingParameters/scalabilityMode}} values may depend on the negotiated RTP header extensions. For example, if the SFM cannot parse codec payloads (either because it is not designed to do so, or because the payloads are encrypted), then negotiation of an RTP header extension (such as the AV1 Dependency Descriptor defined in Appendix A of [[?AV1-RTP]]) could be a prerequisite for the SFM to forward a {{RTCRtpEncodingParameters/scalabilityMode}} value. As a result, the {{RTCRtpEncodingParameters/scalabilityMode}} values supported by an SFM may not be determined until completion of the Offer/Answer negotiation.
The {{RTCRtpEncodingParameters/scalabilityMode}} values supported in this specification, as well as their associated identifiers and characteristics, are provided in the table below. The names of the {{RTCRtpEncodingParameters/scalabilityMode}} values (which are case sensitive) are provided, along with the scalability mode identifiers assigned in [[?AV1]] Section 6.7.5, and links to dependency diagrams provided in Section 10.
While the [[?AV1]] and VP9 [[?VP9]] specifications support all the {{RTCRtpEncodingParameters/scalabilityMode}} values defined in the table, other codec specifications do not. For example, VP8 [[?RFC6386]] only supports temporal scalability (e.g. [="L1T2"=], [="L1T3"=]); H.264/SVC [[?RFC6190]], which supports both temporal and spatial scalability, only permits transport of simulcast on distinct SSRCs, so that it does not support "S" modes, where multiple encodings are transported on a single RTP stream.
Scalability Mode Identifier | Spatial Layers | Resolution Ratio | Temporal Layers | Inter-layer dependency | AV1 scalability_mode_idc |
---|---|---|---|---|---|
"L1T1" | 1 | 1 | N/A | ||
"L1T2" | 1 | 2 | SCALABILITY_L1T2 | ||
"L1T3" | 1 | 3 | SCALABILITY_L1T3 | ||
"L2T1" | 2 | 2:1 | 1 | Yes | SCALABILITY_L2T1 |
"L2T2" | 2 | 2:1 | 2 | Yes | SCALABILITY_L2T2 |
"L2T3" | 2 | 2:1 | 3 | Yes | SCALABILITY_L2T3 |
"L3T1" | 3 | 2:1 | 1 | Yes | SCALABILITY_L3T1 |
"L3T2" | 3 | 2:1 | 2 | Yes | SCALABILITY_L3T2 |
"L3T3" | 3 | 2:1 | 3 | Yes | SCALABILITY_L3T3 |
"L2T1h" | 2 | 1.5:1 | 1 | Yes | SCALABILITY_L2T1h |
"L2T2h" | 2 | 1.5:1 | 2 | Yes | SCALABILITY_L2T2h |
"L2T3h" | 2 | 1.5:1 | 3 | Yes | SCALABILITY_L2T3h |
"L3T1h" | 3 | 1.5:1 | 1 | Yes | |
"L3T2h" | 3 | 1.5:1 | 2 | Yes | |
"L3T3h" | 3 | 1.5:1 | 3 | Yes | |
"S2T1" | 2 | 2:1 | 1 | No | SCALABILITY_S2T1 |
"S2T2" | 2 | 2:1 | 2 | No | SCALABILITY_S2T2 |
"S2T3" | 2 | 2:1 | 3 | No | SCALABILITY_S2T3 |
"S2T1h" | 2 | 1.5:1 | 1 | No | SCALABILITY_S2T1h |
"S2T2h" | 2 | 1.5:1 | 2 | No | SCALABILITY_S2T2h |
"S2T3h" | 2 | 1.5:1 | 3 | No | SCALABILITY_S2T3h |
"S3T1" | 3 | 2:1 | 1 | No | SCALABILITY_S3T1 |
"S3T2" | 3 | 2:1 | 2 | No | SCALABILITY_S3T2 |
"S3T3" | 3 | 2:1 | 3 | No | SCALABILITY_S3T3 |
"S3T1h" | 3 | 1.5:1 | 1 | No | SCALABILITY_S3T1h |
"S3T2h" | 3 | 1.5:1 | 2 | No | SCALABILITY_S3T2h |
"S3T3h" | 3 | 1.5:1 | 3 | No | SCALABILITY_S3T3h |
"L2T2_KEY" | 2 | 2:1 | 2 | Yes | SCALABILITY_L3T2_KEY |
"L2T2_KEY_SHIFT" | 2 | 2:1 | 2 | Yes | SCALABILITY_L3T2_KEY_SHIFT |
"L2T3_KEY" | 2 | 2:1 | 3 | Yes | SCALABILITY_L3T3_KEY |
"L2T3_KEY_SHIFT" | 2 | 2:1 | 3 | Yes | SCALABILITY_L3T3_KEY_SHIFT |
"L3T1_KEY" | 3 | 2:1 | 1 | Yes | |
"L3T2_KEY" | 3 | 2:1 | 2 | Yes | SCALABILITY_L4T5_KEY |
"L3T2_KEY_SHIFT" | 3 | 2:1 | 2 | Yes | SCALABILITY_L4T5_KEY_SHIFT |
"L3T3_KEY" | 3 | 2:1 | 3 | Yes | SCALABILITY_L4T7_KEY |
"L3T3_KEY_SHIFT" | 3 | 2:1 | 3 | Yes | SCALABILITY_L4T7_KEY_SHIFT |
When proposing a {{RTCRtpEncodingParameters/scalabilityMode}} value, the following principles should be followed:
LxTy
to denote a {{RTCRtpEncodingParameters/scalabilityMode}} with x
spatial layers using a 2:1 resolution ratio and y temporal layers.
LxTyh
denotes x spatial layers with a 1.5:1 resolution ratio and
y temporal layers. SxTy
denotes a {{RTCRtpEncodingParameters/scalabilityMode}}
with x simulcast encodings with a 2:1 resolution ratio, with each
simulcast encoding containing y temporal layers. SxTyh
denotes
a 1.5:1 resolution ratio. LxTy_KEY
denotes a {{RTCRtpEncodingParameters/scalabilityMode}}
with x spatial layers using a 2:1 resolution ratio and y temporal layers in which spatial layers only
depend on lower spatial layers at a key frame. LxTy_KEY_SHIFT
modes denotes a
{{RTCRtpEncodingParameters/scalabilityMode}} with x spatial layers using a 2:1 resolution ratio and
y temporal layers in which spatial layers only depend on lower spatial layers at a key frame and subsequent
frames have their temporal identifier shifted upward.
This example extends [[WEBRTC]] Section 7.1 (Example 1) to demonstrate sending three spatial simulcast layers each with three temporal layers, using an SSRC and RID for each simulcast layer. Only the "sendEncodings" attribute is changed from the original example.
const signaling = new SignalingChannel(); // handles JSON.stringify/parse const constraints = {audio: true, video: true}; const configuration = {'iceServers': [{'urls': 'stun:stun.example.org'}]}; let pc; // call start() to initiate async function start() { pc = new RTCPeerConnection(configuration); // let the "negotiationneeded" event trigger offer generation pc.onnegotiationneeded = async () => { try { await pc.setLocalDescription(); // send the offer to the other peer signaling.send({description: pc.localDescription}); } catch (err) { console.error(err); } }; try { // get a local stream, show it in a self-view and add it to be sent const stream = await navigator.mediaDevices.getUserMedia(constraints); selfView.srcObject = stream; pc.addTransceiver(stream.getAudioTracks()[0], {direction: 'sendonly'}); pc.addTransceiver(stream.getVideoTracks()[0], { direction: 'sendonly', sendEncodings: [ {rid: 'q', scaleResolutionDownBy: 4.0, scalabilityMode: 'L1T3'}, {rid: 'h', scaleResolutionDownBy: 2.0, scalabilityMode: 'L1T3'}, {rid: 'f', scalabilityMode: 'L1T3'} ] }); } catch (err) { console.error(err); } } signaling.onmessage = async ({data: {description, candidate}}) => { try { if (description) { await pc.setRemoteDescription(description); // if we got an offer, we need to reply with an answer if (description.type == 'offer') { await pc.setLocalDescription(); signaling.send({description: pc.localDescription}); } } else if (candidate) { await pc.addIceCandidate(candidate); } } catch (err) { console.error(err); } };
This is an example with two spatial layers (with a 2:1 ratio) and three temporal layers.
let sendEncodings = [ {scalabilityMode: 'L2T3'} ];
This is an example of mixed codec simulcast, with each simulcast layer having 3 temporal layers.
let sendEncodings = [ {rid: 'q', codec: {clockRate: 90000, mimeType: 'video/AV1'}, scaleResolutionDownBy: 4.0, scalabilityMode: 'L1T3'}, {rid: 'h', codec: {clockRate: 90000, mimeType: 'video/VP8'}, scaleResolutionDownBy: 2.0, scalabilityMode: 'L1T3'}, {rid: 'f', codec: {clockRate: 90000, mimeType: 'video/VP8'}, scalabilityMode: 'L1T3'} ];
This is an example with three spatial simulcast layers each with three temporal layers on a single SSRC.
let sendEncodings = [ {scalabilityMode: 'S3T3'} ]
This is an example of {{MediaCapabilities/encodingInfo(configuration)}} returned by a browser implementing [[WEBRTC]] and [[Media-Capabilities]].
const contentType = 'video/VP9'; const configuration = { type: 'webrtc', video: { contentType, width: 640, height: 480, bitrate: 10000, framerate: 29.97, scalabilityMode: 'L3T3_KEY' } }; try { const info = await navigator.mediaCapabilities.encodingInfo(configuration); if (!info.supported) { console.log(`${contentType} is unsupported.`); return; } console.log(`${contentType} is ${info.smooth || 'NOT '}smooth, and ` + `${info.powerEfficient || 'NOT '}power efficient`); } catch (err) { console.error(err, ' caused encodingInfo to fail'); }
This is an example of receiver capabilities returned by an SFM that only supports forwarding of VP8, VP9 and AV1 temporal scalability modes.
"codecs": [ { "clockRate": 90000, "mimeType": "video/VP8", "scalabilityModes": [ "L1T1", "L1T2", "L1T3" ] }, { "clockRate": 90000, "mimeType": "video/VP9", "scalabilityModes": [ "L1T1", "L1T2", "L1T3" ] }, { "clockRate": 90000, "mimeType": "video/AV1", "scalabilityModes": [ "L1T1", "L1T2", "L1T3" ] } ]
This section is non-normative; it specifies no new behaviour, but instead summarizes information already present in other parts of the specification. The privacy considerations for the WebRTC APIs are described in [[WEBRTC]] Section 13.
Since for SVC codecs implemented in WebRTC the use of scalable coding tools is not negotiated and is independent of the supported profiles, and since SVC is rarely supported in hardware encoders, knowledge of the {{RTCRtpEncodingParameters/scalabilityMode}} supported by the {{RTCRtpSender}} does not provide additional information on the underlying hardware. However, since browsers may differ in their support for SVC modes, the supported {{RTCRtpEncodingParameters/scalabilityMode}} may permit differentiation between browsers. This additional fingerprint surface is expected to decrease over time as this specification is more widely implemented.
This section is non-normative; it specifies no new behaviour, but instead summarizes information already present in other parts of the specification. WebRTC protocol security considerations are described in [[RFC8827]] and the security and privacy considerations for the WebRTC APIs are described in [[WEBRTC]] Section 13.
Dependency diagrams for the scability modes defined in this specification are provided below.
The editors wish to thank Robin Raymond, Michael Horowitz, Harald Alvestrand, Chris Cunningham, Danil Chapovalov, Florent Castelli and Henrik Boström for their contributions to this specification, which evolved from the ORTC API developed in the W3C ORTC CG.