This specification defines an API extending the
{{HTMLMediaElement}} that enables controlling remote
playback of media from a web page.
This document builds on the group's experience on presenting web content on external presentation-type displays, and re-uses patterns and design considerations from the Presentation API specification whenever appropriate [[PRESENTATION-API]].
Although this document is still a work in progress and is subject to change, the Working Group believes that the API surface is stable. Most of the remaining issues listed on the issue tracker are considered minor at this stage except for Issue #41.
Issue #41 discusses the set of media playback features that remote playback devices are expected to support. The group will seek further developer feedback and implementation experience to identify any interoperability issues around these features when used during remote playback, and will further clarify the specification based on feedback received.
For other issues or concerns, it is possible to file a bug or send an email to the mailing list. For small editorial changes like typos, sending a pull request is appreciated.
The Working Group invites everyone to review this document, and will work with relevant groups at W3C to conduct horizontal reviews on accessibility, internationalization, privacy, security and technical architecture principles.
No feature has been identified as being at risk.
The Second Screen Working Group will develop a test suite for the Remote Playback API during the Candidate Recommendation period and prepare an implementation report. For this specification to advance to Proposed Recommendation, two independent, interoperable implementations of each feature must be demonstrated, as detailed in the Candidate Recommendation exit criteria section.
This specification defines conformance criteria that apply to a single product: the user agent that implements the interfaces that it contains.
Implementations that use ECMAScript to expose the APIs defined in this specification MUST implement them in a manner consistent with the ECMAScript Bindings defined in the Web IDL specification [[!WEBIDL]].
This specification aims to make remote playback devices such as connected TVs, projectors or audio-only speakers, available to the Web and takes into account playback devices that are attached using wired (HDMI, DVI, or similar) and wireless technologies (Miracast, Chromecast, DLNA, AirPlay, or similar).
Devices with limited screen size or quiet speakers lack the ability to playback media content to a larger audience, for example, a group of colleagues in a conference room, or friends and family at home. Playing media content on an external larger and/or louder remote playback device helps to improve the perceived quality and impact of the played media.
At its core, this specification enables a page that acts as the browsing context to initiate and control remote playback of a particular media element on a selected remote playback device. How the remoting is initiated and controlled is left to the UA in order to allow the use of remote playback devices that can be attached in a wide variety of ways. For example, when a remote playback device is attached using HDMI or Miracast, the same UA that acts as the browsing context renders the remote media. Instead of playing the media on the same device, however, it can use whatever means the operating system provides for using the external remote playback device. In such a case, both the browsing context and the media player run on the same UA and the operating system is used to route the player output to the remote playback device. This is commonly referred to as the media mirroring case. This specification imposes no requirements on the remote playback devices connected in such a manner.
If the remote playback device is able to play the media and communicate with the browsing context but is unable to fetch the media, the browsing context needs to fetch the media data and pass it on to the remote playback device for rendering. This is commonly referred to as media remoting case.
If the remote playback device is able to fetch and play the media and communicate with the browsing context, the browsing context does not need to fetch or render the remoted media. In this case, the UA acts as a proxy that requests the remote playback device to play the media itself by passing the necessary data like the media source. This is commonly referred to as the media flinging case. This way of attaching to displays could be enhanced in the future by defining a standard protocol for delivering these types of messages that remote playback devices could choose to implement.
The API defined here is intended to be used with UAs that attach to remote playback device devices through any of the above means.
The use cases and requirements of this specification are captured in a separate document available here.
This section shows code examples that highlight the usage of the main features of the Remote Playback API. In these examples, `player.html` implements the player page controlling the remote playback and `media.ext` is the media file to be played remotely. Both the page and the media are served from the domain `https://example.org`. Please refer to the comments in the code examples for further details.
<!-- player.html -->
<!-- The video element with custom controls that supports remote playback. -->
<video id="videoElement" src="https://example.org/media.ext" />
<button id="deviceBtn" style="display: none;">Pick device</button>
<script>
// The "Pick device" button is visible if at least one remote playback device is available.
const deviceBtn = document.getElementById("deviceBtn");
const videoElem = document.getElementById("videoElement");
function availabilityCallback(available) {
// Show or hide the device picker button depending on device availability.
deviceBtn.style.display = available ? "inline" : "none";
}
videoElem.remote.watchAvailability(availabilityCallback).catch(() => {
// Availability monitoring is not supported by the platform, so discovery of
// remote playback devices will happen only after remote.prompt() is called.
// Pretend the devices are available for simplicity; or, one could implement
// a third state for the button.
deviceBtn.style.display = "inline";
});
</script>
<!-- player.html -->
<script>
devicesBtn.onclick = () => {
// Request the user to select a remote playback device.
videoElem.remote.prompt()
// Update the UI and monitor the connected state.
.then(updateRemotePlaybackState);
// Otherwise, the user cancelled the selection UI or no screens were found.
};
<script>
<!-- player.html -->
<script>
// The remote playback may be initiated by the user agent,
// so check the initial state to sync the UI with it.
if (videoElem.remote.state == "disconnected")
switchToLocalUI();
else
switchToRemoteUI();
videoElem.remote.onconnecting = switchToRemoteUI;
videoElem.remote.onconnect = swithToRemoteUI;
videoElem.remote.ondisconnect = switchToLocalUI;
// Handles both 'connecting' and 'connected' state. Calling more than once
// is a no-op.
function switchToRemoteUI() {
// Indicate that the state is 'connecting' or 'connected' to the user.
// For example, hide the video element as only controls are needed.
videoElem.style.display = "none";
// Stop monitoring the availability of remote playback devices.
videoElem.remote.cancelWatchAvailability();
};
function switchToLocalUI() {
// Show the video element.
videoElem.style.display = "inline";
// Start watching the device availability again.
videoElem.remote.watchAvailability(availabilityCallback);
};
<script>
A local playback device is the device the browsing context is running on along with the default video/audio outputs the device has.
A remote playback device is any other device but the local playback device that the browsing context can use to play media on.
A media element state is the set of all single media element properties observable by the page and/or the user via the user agent implementation. The new properties introduced by this spec are not considered part of the media element state for convenience.
A local playback state is the user agent implementation of media element state for the particular media element for playback on the local playback device.
A remote playback state is the user agent implementation of media element state for the particular media element for playback on the certain remote playback device.
For a good user experience it is important that the media element state doesn't change unexpectedly when the `state` changes. It is also important that remote playback state is in sync with the media element state so when the media is paused on the remote playback device it looks paused to both the user and the page.
The task source for the tasks mentioned in this specification is the media element event task source.
[Exposed=Window]
interface RemotePlayback : EventTarget {
Promise<long> watchAvailability(RemotePlaybackAvailabilityCallback callback);
Promise<undefined> cancelWatchAvailability(optional long id);
readonly attribute RemotePlaybackState state;
attribute EventHandler onconnecting;
attribute EventHandler onconnect;
attribute EventHandler ondisconnect;
Promise<undefined> prompt();
};
enum RemotePlaybackState {
"connecting",
"connected",
"disconnected"
};
callback RemotePlaybackAvailabilityCallback = undefined(boolean available);
A RemotePlayback object is an interface allowing the page to detect availability of, connect to and control playback on remote playback devices.
A RemotePlaybackState enumeration represents the state of connection to some remote playback device.
A RemotePlaybackAvailabilityCallback represents the remote playback device availability information.
A `RemotePlaybackAvailabilityCallback` is the way for the page to obtain the remote playback device availability for the corresponding media element. If the user agent can monitor the list of available remote playback devices in the background (without a pending request to `prompt``()`), the `RemotePlaybackAvailabilityCallback` behavior defined below MUST be implemented by the user agent. Otherwise, the promise returned by `watchAvailability``()` MUST be rejected with {{NotSupportedError}}.
The user agent MUST keep track of the set of availability callbacks registered with each media element through the `watchAvailability``()` method. The set of availability callbacks for each RemotePlayback object is represented as a set of tuples (|callbackId:long|, |callback:RemotePlaybackAvailabilityCallback|), initially empty, where:
Since there's one and only one RemotePlayback object per each media element, set of availability callbacks of a media element is the same set as the set of availability callbacks of the RemotePlayback object referred to by the element's remote property.
The combined set of all sets of availability callbacks of all RemotePlayback objects known to the browsing context is referred to as global set of availability callbacks.
The user agent MUST keep a list of available remote playback devices. This list contains remote playback devices and is populated based on an implementation specific discovery mechanism. It is set to the most recent result of the algorithm to monitor the list of available remote playback devices or an empty list if the algorithm hasn't been run yet.
The user agent MAY not support running the algorithm to monitor the list of available remote playback devices continuously, for example, because of platform or power consumption restrictions. In this case the promise returned by `watchAvailability``()` MUST be rejected with {{NotSupportedError}}, the global set of availability callbacks will be empty and the algorithm to monitor the list of available remote playback devices will only run as part of the initiate remote playback algorithm.
When the global set of availability callbacks is not empty, the user agent MUST monitor the list of available remote playback devices continuously, so that pages can keep track of the last value received via the registered callbacks to offer remote playback only when there are available devices.
The user agent is expected not to monitor the list of available remote playback devices when possible, to satisfy the power saving non-functional requirement. For example, the user agent might choose not to run the monitoring algorithm when the global set of availability callbacks is empty, when every page that has media elements with non-empty set of availability callbacks is in the background.
Some remote playback devices may only be able to play a subset of media resources because of functional, security or hardware limitations. Examples are set-top boxes, smart TVs or networked speakers capable of rendering only certain formats of video and/or audio. We say that such a device is a compatible remote playback device for a media resource if the user agent can reasonably guarantee that the remote playback of the media specified by the resource will succeed on that device.
The user agent can use the {{HTMLTrackElement/srclang}}
attribute of the [^track^] element as a hint of
the language of the text track data to help identify a
compatible remote playback device.
The media resources of a media element, that were considered by the user agent to find a compatible remote playback device, are called the availability sources set.
The media resource of a media element that is used to initiate remote playback on the selected remote playback device is called the remote playback source. A remote playback source MUST belong to the media element's availability sources set.
The mechanism to choose the remote playback source from the availability sources set is implementation-specific, but the user agent SHOULD consider every resource in the availability sources set as a potential remote playback source.
If the user agent cannot determine a remote playback source appropriate for the remote playback device, it is RECOMMENDED that the user agent send metadata (for example, the extended MIME type) about all resources in the availability sources set to the remote playback device so it can run its own resource selection algorithm and choose the remote playback source.
Remote playback is said to be unavailable for the media element if the list of available remote playback devices is empty or none of them is compatible with any source from availability sources set for the media element. The remote playback is said to be available otherwise. A `boolean` set to `false` if the remote playback is unavailable for the media element or `true` if it is available is called availability for the media element.
If the user agent stops monitoring the list of available remote playback devices (for example by a user control or for power saving), it SHOULD invoke all callbacks in the global set of availability callbacks with `false` so that pages can update their user experience appropriately. It SHOULD also set the availability value for all media elements to `false` so that availability information can be propagated correctly if the user agent later resumes monitoring the list of available remote playback devices.
When the `watchAvailability``()` method is called, the user agent MUST run the following steps:
A simple algorithm for assigning |callbackId| values is to keep a counter for each browsing context and incrementing it in step 6.
If the set of availability callbacks is non-empty, or there is a pending request to initiate remote playback, the user agent MUST monitor the list of available remote playback devices by running the following steps:
When a `cancelWatchAvailability` `()` method is called, the user agent MUST run the following steps:
When the `prompt``()` method is called, the user agent MUST run the following steps:
By picking a remote playback device the user grants permission to use the device.
The `state` attribute represents the RemotePlayback connection's current state. It can take one of the values of RemotePlaybackState depending on the connection state:
When the user agent is to establish a connection with the remote playback device, it MUST run the following steps:
The user agent SHOULD pause local audio and video output of the media element while the remote playback state is connected.
If the user agent is exposing a user interface to the user for the media element (i.e., using default controls), the user agent SHOULD convey the fact that the remote playback state is connected through an icon or other means.
A user agent MAY support connecting to a remote playback device from the browser. This can be done by adding appropriate media controls to the user interface that is exposed to the user, or when the user activates system-wide mirroring of the display. This feature is known as browser initiated remote playback. A user agent that supports browser initiated remote playback SHOULD initiate the remote playback only when the user has expressed an intention to do so via a user gesture, for example by clicking a button in the browser.
If the user agent supports browser initiated remote playback, the state attribute MUST reflect the current state of the connection to the remote playback device. When the browser initiates or terminates remote playback, it MUST fire the corresponding events by following the algorithms to establish a connection with the remote playback device and disconnect from a remote playback device.
If the browser will initiate remote playback on a newly created media element, it SHOULD initialize the value of its state attribute to connecting and then follow the steps to establish a connection with the remote playback device.
The {{HTMLMediaElement}} interface interacts with the remotely played media as soon as the connection with the remote playback device is established.
When the state of a RemotePlayback object is connected, the following conditions relate the local playback state, the media element state, and the remote playback state:
If sending any command fails, the user agent MAY disconnect from a remote playback device.
The remote playback device can implement a subset of the capabilities of the playback engine of the user agent, and some {{HTMLMediaElement}} APIs do not always make sense to use during remote playback. In this case, the local playback state is expected to reflect as closely as possible the actual remote playback state after a media command that is not supported during remote playback.
For example, after calling {{HTMLMediaElement/fastSeek()}}
while connected to a remote playback device that does not
support it, the {{HTMLMediaElement/seeking}} attribute of the
{{HTMLMediaElement}} is expected to remain `false` and no
`seeking` event is to be fired.
When the user agent is to disconnect from a remote playback device, it MUST do the following:
If the remote playback device is abruptly disconnected during playback (for example, by power loss or a network disconnection), the user agent SHOULD run the steps to monitor the list of available remote playback devices before the steps to disconnect from a remote playback device. This allows callbacks in the set of availability callbacks to be invoked before the disconnect event is fired, so the page can update itself to show resumption of playback is not possible.
The following are the event handlers (and their corresponding event handler event types) that must be supported, as event handler IDL attributes, by objects implementing the RemotePlayback interface:
| Event handler | Event handler event type |
|---|---|
| `onconnecting` | `connecting` |
| `onconnect` | `connect` |
| `ondisconnect` | `disconnect` |
{{HTMLMediaElement}}
partial interface HTMLMediaElement {
[SameObject] readonly attribute RemotePlayback remote;
[CEReactions] attribute boolean disableRemotePlayback;
};
The remote attribute MUST return the RemotePlayback instance associated with the media element.
Some pages may wish to disable remote playback of a media element; for example, they may prefer to use a `PresentationRequest` to present a complete document on a presentation screen. To support this use case, a new `disableRemotePlayback` attribute is added to the list of content attributes for `audio` and `video` elements.
A corresponding disableRemotePlayback IDL attribute which reflects the value of each element's `disableRemotePlayback` content attribute is added to the `HTMLMediaElement` interface. The disableRemotePlayback IDL attribute MUST reflect the content attribute of the same name.
If the disableRemotePlayback attribute is present on the media element, the user agent MUST NOT play the media element remotely or present any UI to do so.
When the disableRemotePlayback attribute is added to the media element, the user agent MUST run the steps to disable remote playback:
Firing the `callback` provided via the `watchAvailability``()` method reveals one bit of information about the presence (or non-presence) of a remote playback device typically discovered through the local area network. This could be used in conjunction with other information for fingerprinting the user. However, this information is also dependent on the user's local network context, so the risk is minimized. Also, by design, the human readable name of a remote playback device is not revealed to the page.
The API enables monitoring the list of available remote playback devices. How the user agent determines the compatibility and availability of a remote playback device with a media element's resource is an implementation detail. If a user agent matches a media resource to a particular type of device to determine its availability, this feature can be used to probe information about which remote playback device the user has without user consent.
The user agent should not monitor the list of available remote playback devices if the user disables background monitoring through a browser setting.
When the user is asked permission to use a remote playback device during the steps to change remote playback state, the user agent should make it clear what origin the request is coming from.
Display of the origin requesting remote playback will help the user understand what content is making the request, especially when the request is initiated from a nested browsing context. For example, embedded content may try to convince the user to click to trigger a request to start an unwanted remote playback.
Showing the origin that will be presented will help the user know if that content is from a potentially trustworthy origin (e.g., `https:`), and corresponds to a known or expected site.
The Remote Playback API abstracts away what "local" means for displays, meaning that it exposes network-accessible displays as though they were local displays. The Remote Playback API requires user permission for a page to access any display to mitigate issues that could arise, such as showing unwanted content on a display viewable by others.
This specification will not mandate communication protocols between the local playback device and the remote playback device, but the user agent should set some guarantees of message confidentiality and authenticity between them.
The Remote Playback API is not limited to [[SECURE-CONTEXTS]] because it exposes a feature to web applications that user agents usually offer natively on all media regardless of the browsing context. A user agent can limit the API to [[SECURE-CONTEXTS]] by always returning an empty list as part of the monitor the list of available remote playback devices algorithm in contexts that are not secure.
For this specification to be advanced to Proposed Recommendation, there must be at least two independent, interoperable implementations of each feature. Each feature may be implemented by a different set of products, there is no requirement that all features be implemented by a single product. Additionally, implementations must demonstrate support for the media remoting and media flinging cases, either within the same product or within different products.
For the purposes of these criteria, we define the following terms: