This specification extends {{MediaStreamTrack}} to provide an optional hint about the user's preference on how the media should be treated when insufficient resources for perfect reproduction are available.

This optional hint permits {{MediaStreamTrack}} sinks such as RTCPeerConnection (defined in [[webrtc]]) or MediaRecorder (defined in [[mediastream-recording]]) that process a track's audio or video content to choose processing parameters that are appropriate to the user's preferences.

Introduction

Algorithms used for processing speech and music differ greatly. Echo-cancellation algorithms developed for speech-type content might not work well on music, and noise-suppression algorithms might remove drum snares or other "noisy" content. While this makes speech more intelligible it is less appropriate for music signals.

For video, webcam content often require denoising and is often intelligible even when downscaled or with high quantization levels. Screencast content of presentations or webpages with a lot of text content is completely unintelligible if the quantization levels are too high or if the content is downscaled or otherwise blurry.

Without automatic detection of media content, a {{MediaStreamTrack}} consumer can only make an educated guess. This guess may be based on assuming that screencast content, such as chrome.desktopCapture, contains text content and must use low quantization levels, and drop frames extensively to meet bitrate requirements. Another assumption is that regular USB video devices provide webcam video, and higher quantization levels and downscaling are acceptible.

While usually appropriate this educated guess leads to sub-optimal settings when incorrect. This manifests as high framedropping when screencasting high-motion content such as a movie or streaming a video game and treating it as text. Treating highly-detailed content as regular webcam video on the other hand leads to too-blurry content when being either quantized or downscaled beyond readability to meet bitrate requirements. This mismatch may also happen when HDMI video-capture cards are seen as USB webcams but actually screencast webpage text.

In some cases the web application can make a more-educated guess or take user input to inform consumers of what kind of content is being encoded. A web application that streams video-game content would be able to preserve motion from desktop capture at the cost of individual frame detail. A music-studio application would be able to prevent noise suppression from removing snares from a music track.

These settings are not intended to replace encoder-level settings completely but rather complement them with a simpler hint that does not require broad knowledge of video encoders, audio-processing steps or more extensive tuning.

A separate section of this specification describes expected behavior for specific components that process a MediaStreamTrack.

Terminology

The terms RTCRtpSender and RTCRtpSendParameters are defined in [[!WEBRTC]].

Extension to MediaStreamTrack

    partial interface MediaStreamTrack {
      attribute DOMString contentHint;
    };
    

This specification extends {{MediaStreamTrack}} and makes use of its {{MediaStreamTrack/kind}} attribute as defined in [[GETUSERMEDIA]].

Each {{MediaStreamTrack}} has an associated application-set content hint, which is initially "", signifying unset. This application-set content hint corresponds to the contentHint attribute of {{MediaStreamTrack}} which may be used by the web application to provide a hint of what type of content is contained within the track, to guide how it should be treated by {{MediaStreamTrack}} consumers.

Valid values for the application-set content hint are dependent on the {{MediaStreamTrack/kind}} of {{MediaStreamTrack}} contained. On setting contentHint to value,

1. If this {{MediaStreamTrack}}'s {{MediaStreamTrack/kind}} attribute is "audio", and value is not one of "", "speech", "speech-recognition", or "music", abort these steps.
2. If this {{MediaStreamTrack}}'s {{MediaStreamTrack/kind}} attribute is "video", and value is not one of "", "motion", "detail" or "text", abort these steps.
3. Set this {{MediaStreamTrack}}'s application-set content hint to value.
4. The implementation should adapt its decision on how to handle the content of this {{MediaStreamTrack}} according to the new value of its application-set content hint. This adaptation should happen as quickly as reasonable, e.g. within the next couple of captured video frames or audio buffers.

On getting {{MediaStreamTrack/contentHint}},

1. Return this {{MediaStreamTrack}}'s [=application-set content hint=].

Note that the initial value of application-set content hints is "", corresponding to that no hint has been provided. It does not default to the implementation's best guess of contained type of content.

Behaviors of other components based on content-hint

enum RTCDegradationPreference {
  "maintain-framerate",
  "maintain-resolution",
  "balanced"
};

partial dictionary RTCRtpSendParameters {
        RTCDegradationPreference degradationPreference;
       };

Security and Privacy Considerations

This specification adds an API where the user can send information to the user agent. This information is not communicated elsewhere, and is not stored in any permanent location.

By probing this API, the client may get some information on how the user agent implements this specification, which may give some insight into its configuration. Apart from this, no user agent information or data is exposed to the client through this API, and the API does not give any access to browser UI, influence over the user agent's security characteristics or the generation of temporary identifiers.

This API does not distinguish between first-party, third-party or incognito contexts; it behaves identically in all these cases.

No personally identifiable or otherwise sensitive information is handled via the use of this API.