Consider a Web application |capturer| which has used {{MediaDevices/getDisplayMedia()}} to start capturing another [=display surface=], |capturee|. This specification introduces a set of APIs that allow |capturer| the following new capabilities:
Nearly all video-conferencing Web applications offer their users the ability to share [=display surfaces=] - typically a browser tab ([=display surface/browser=]), a native app's window ([=display surface/window=]), or an entire screen ([=display surface/monitor=]).
Many of these applications also show the local user a "preview tile" with a video of the captured [=display surface=].
All these applications suffer from one key drawback - if the user wishes to interact with a captured [=display surface=], the user must first switch to that surface, taking them away from the video-conferencing application. This presents a few issues:
It bears mentioning that Document Picture-in-Picture goes a long way towards addressing some of these issues. However, it not always a suitable solution, as not all use cases are adequately addressed by a floating window which will often be small, which obscures arbitrary other content on the screen, and whose size and positioning must be manually controlled by the user.
This specification defines a [=policy-controlled feature=] identified by the string `"captured-surface-control"`. Its [=policy-controlled feature/default allowlist=] is `"self"`.
The API surfaces introduced by this specification can be categorized as either read-access or write-access. Note that only the write-access APIs ({{CaptureController/forwardWheel}}, {{CaptureController/increaseZoomLevel}}, {{CaptureController/decreaseZoomLevel}} and {{CaptureController/resetZoomLevel}}) are gated by the "captured-surface-control" permissions policy.
We define a concept of an integer "zoom level" that can be applied to [=display surfaces=] of any type, and which is independent of the user agent and the platform. It is expected that in the case of [=display surface/browser=] [=display surfaces=], this concept will match the concept of zoom level that user agents typically exposed to the user.
For a given [=display surface=] of type |surfaceType|, we define the user agent's set of supported zoom levels for |surfaceType| as a non-empty set of integers including at least the [=default zoom level=] (100), and not including any integers lesser than 1.
We define the permitted event types for zoom-setting as a set composed of the following event types:
partial interface CaptureController { sequence<long> getSupportedZoomLevels(); long getZoomLevel(); Promise<undefined> increaseZoomLevel(); Promise<undefined> decreaseZoomLevel(); Promise<undefined> resetZoomLevel(); attribute EventHandler onzoomlevelchange; };
This method allows applications to discover the set of [=zoom levels=] supported by the user agent.
When invoked, the user agent MUST run the following steps:
This method allows applications to discover the captured [=display surface=]'s [=zoom level=].
When invoked, the user agent MUST run the following steps:
This method allows applications to set the captured [=display surface=]'s [=zoom level=] one step higher than its current value.
When this method is invoked, the user agent MUST run the [=set zoom level algorithm=] with [=this=] as the |controller| and `"increase"` as the |zoomAction|.
This method allows applications to set the captured [=display surface=]'s [=zoom level=] one step lower than its current value.
When this method is invoked, the user agent MUST run the [=set zoom level algorithm=] with [=this=] as the |controller| and `"decrease"` as the |zoomAction|.
This method allows applications to set the captured [=display surface=]'s [=zoom level=] to 100.
When this method is invoked, the user agent MUST run the [=set zoom level algorithm=] with [=this=] as the |controller| and `"reset"` as the |zoomAction|.
An [=event handler IDL attribute=] whose [=event handler event type=] is `zoomlevelchange`.
Given a {{CaptureController}} |controller|, the user agent MUST [=fire an event=] named `zoomlevelchange` at |controller| whenever |controller|.{{CaptureController/[[Source]]}}'s [=zoom level=] changes.
Examples of causes include:
partial interface CaptureController { constructor(); Promise<undefined> forwardWheel(HTMLElement? element); };
{{CaptureController}}'s constructor is extended to also define and initialize the following internal slots:
Internal Slot | Initial value |
---|---|
[[\ForwardWheelElement]] | `null` |
[[\ForwardWheelEventListener]] | `null` |
This method allows applications to automatically forward wheel events from an {{HTMLElement}} to the viewport of a captured [=display surface=].
When invoked, the user agent MUST run the following steps:
This step ensures that on the one hand, permission prompts are not be shown without [=transient activation=], while on the one hand, if the permission is already {{PermissionState/"granted"}}, {{CaptureController/forwardWheel()}} may be called immediately after {{MediaDevices/getDisplayMedia()}} resolves, even if the [=transient activation=] that permitted the call to {{CaptureController/forwardWheel()}} has since expired.
To determine if a {{CaptureController}} |controller| is actively capturing, run the following steps:
To determine if a {{CaptureController}} |controller| is is self-capturing, run the following steps:
To determine if a [=display surface=] |surfaceType| is supported display surface type, run the following steps:
Whether [=display surface/window=] should be supported is under discussion.
The set zoom level algorithm, given a |controller:CaptureController| of type {{CaptureController}} and a |zoomAction:DOMString| of type {{DOMString}} as arguments, consists of running the following steps:
Ensure that the code is running from within the context of an event handler which was triggered by the browser agent firing a trusted event, triggered by the user interacting with the user agent. To do so, run the following steps:
It follows from these steps that {{CaptureController/increaseZoomLevel()}}, {{CaptureController/decreaseZoomLevel()}} and {{CaptureController/resetZoomLevel()}} are only callable with [=transient activation=], because [=permitted event types for zoom-setting=] only contains event types that confer this activation.
In fact, our API shape implies a stronger guarantee - whereas [=transient activation=] persists for several seconds after the user action, the API shape here limits zoom-setting to immediately after the user's action.
If |zoomAction| is `"decrease"` then:
Else, if |zoomAction| is `"increase"` then:
Else:
Run the following steps [=in parallel=]:
The forward wheel event algorithm takes a {{CaptureController}} |controller| and a {{WheelEvent}} |event|, and runs the following steps:
The scale element coordinates algorithm takes {{double}} coordinates [|x|, |y|] and a {{CaptureController}} |controller|, and run the following steps:
(|x| /
|controller|.{{CaptureController/[[ForwardWheelElement]]}}.{{Element/getBoundingClientRect()}}.{{DOMRect/width}})
.
(|x| /
|controller|.{{CaptureController/[[ForwardWheelElement]]}}.{{Element/getBoundingClientRect()}}.{{DOMRect/height}})
.
This subroutine assumes that |controller| is [=actively capturing=].
The API surfaces introduced in this specification allow a capturing application limited control over a captured application. These APIs allow the capturing application to gain access to additional pixels in the captured application. This specification employs multiple means to ensure that new capabilities are used in accordance with the user's intentions. Among these means:
{{CaptureController/increaseZoomLevel()}}, {{CaptureController/decreaseZoomLevel()}} and {{CaptureController/resetZoomLevel()}} are only callable from event handlers of specific event types - the [=permitted event types for zoom-setting=]. These are events dispatched directly by the user agent, triggered by user interaction. This specification intentionally excludes from this set such events as "mousemove", which users are liable to trigger inadvertently.
The shape of {{CaptureController/forwardWheel()}} is intentionally chosen to limit the capturing application's control. The application designates a specific element which, when the user scrolls over it, the corresponding wheel events are forwarded to the captured application.
This specification does not limit the type of {{Element}} for which either {{CaptureController/increaseZoomLevel()}}, {{CaptureController/decreaseZoomLevel()}}, {{CaptureController/resetZoomLevel()}} or {{CaptureController/forwardWheel()}} work. Such a limitation would accomplish nothing, because malicious applications could always overlay transparent permitted {{Element}} types on top of visible non-permitted {{Element}}s, thereby bypassing this restriction.
The limitation of interaction types is sufficient. This is accomplished by {{CaptureController/forwardWheel()}} through its shape, and by {{CaptureController/increaseZoomLevel()}}, {{CaptureController/decreaseZoomLevel()}} and {{CaptureController/resetZoomLevel()}} through their gating on event types.