Well-deployed technologies
Measuring the Quality of Experience
The Web Performance Working Group developed a number of specifications that expose timing hooks to Web applications, to analyze the time spent doing various tasks. Although not specifically targeted at media scenarios, these specifications make it possible to monitor the performance of running applications. High-Resolution Time exposes a monotonic sub-millisecond resolution clock to Web applications so that they can precisely measure time elapsed between two events. Performance Timeline defines a unified interface to store and retrieve performance metric data. Individual performance metric interfaces are enumerated in the Timing Entry Names Registry and defined in separate specifications. Well-deployed interfaces include:
- Navigation Timing exposes timing information related to navigation and elements;
- Resource Timing exposes timing information for resources in a document;
- User Timing help applications measure the performance of their applications using high precision timestamps.
Technologies in progress
Detecting Capabilities
To improve the user experience and take advantage of advanced device capabilities when they are available, media providers need to know the decoding (and encoding) capabilities of the user's device. Can the device decode a particular codec at a given resolution, bitrate and framerate? Will the playback be smooth and power efficient? Can the display render high dynamic range (HDR) and wide color gamut content? The Media Capabilities specification defines an API that helps application developers determine device capabilities. The API aims at replacing the more basic and vague canPlayType() and isTypeSupported() functions defined in HTML and Media Source Extensions (MSE).
On top of getting answers to "Can" questions, media providers also want answers to "Should" questions, such as "Should I send HDR content if I have both HDR and SDR variants?". The Media Capabilities specification does not answer these questions, which rather fall within the realm of CSS extensions describing the capability of the display. Notably, CSS Media Queries Level 4 includes means to detect wide-gamut displays and adapt the rendering of the application to them. CSS Media Queries Level 5, to be published after Level 4, will introduce a video- prefix to some features to detect differences between the video plane and the graphics plane on devices that render video separately from the rest of an HTML document, and that e.g. may support High Dynamic Range (HDR) for video and only Standard Dynamic Range (SDR) for other types of content.
Measuring the Quality of Experience
The Media Playback Quality exposes metrics on the number of frames that have been displayed or dropped, giving some mechanisms to application developers to assess the user's perceived playback quality and alter the quality of content transmitted using adaptive streaming accordingly.
The Web Performance Working Group develops additional generic timing hooks for Web applications, to analyze the time spent doing various tasks:
- Server Timing enables a server to communicate performance metrics about the request-response cycle to the user agent, and allows applications to act on these metrics to optimize application delivery.
- The Long Tasks API exposes a mechanism to detect long running tasks that monopolize the user interface's main thread for extended periods of time.
- Paint Timing allows the application to capture a series of key moments such as first paint and first contentful paint during page load.
Exploratory work
Common baseline of the Web platform
The Web Media API Community Group, created under the aegis of the CTA WAVE project, develops the Web Media APIs document, that defines a common baseline of Web technologies that should be included in device implementations to support media Web applications. This document provides an annual snapshot of the Web platform that developers can rely on being supported across devices when they develop media applications.
Media Web applications Best Practices
The Community Group also develops the Web Media Application Developer Guidelines document, a companion guide to the Web Media APIs specification that outlines best practices and developer guidance for implementing Web media applications.
Media production on the Web
Professional media assets, including audio-visual masters for television and motion pictures, are increasingly being stored in the cloud. There is a corresponding growing interest in building web applications that allow end-users to manipulate these assets, e.g., quality checking, versioning, timed text authoring, etc. While the web platform has evolved to support consumer media applications, professional applications require additional capabilities, including precise timing, wider color gamut and high-dynamic range, high-fidelity timed text, etc. The Media and Entertainment Interest Group has started to explore the problem space, notably to identify possible technical gaps.
Measuring the Quality of Experience
Some specifications are under incubation in the Web Platform Incubator Community Group to expose additional timing hooks for Web applications:
- The Frame Timing API aims at providing detailed information on the frame-per-second obtained when an application is running on the user device.
- The Event Timing API exposes a mechanism to measure the latency of some events triggered by user interaction.
- The Element Timing API enables monitoring when large or developer-specified image elements and text nodes are displayed on screen.
- The Layout Instability API provides web page authors with insights into the stability of their pages based on movements of the elements on the page that detract from the user's experience.
| Feature | Specification / Group | Implementation intentsSelect browsers… |
|---|---|---|
| Common baseline of the Web platform | Web Media API Snapshot 2019 Web Media API Community Group | |
| Media Web applications Best Practices | Web Media Application Developer Guidelines Web Media API Community Group | |
| Measuring the Quality of Experience | Frame Timing Web Performance Working Group | |
| Event Timing API W3C | ||
| Element Timing API Web Platform Incubator Community Group | ||
| Layout Instability Web Platform Incubator Community Group |
Features not covered by ongoing work
- Guidelines on integration of Web media APIs with hardware-based media decoders
- A number of practical questions arise when a browser codebase needs to be ported to the specific hardware architecture of a media device, notably for integration of Web media APIs with hardware video and audio decoders. This includes questions such as: At which steps does the latency of hardware setup impact the various media algorithms? How should implementations handle cases where the application attempts to render a video while decoders are already in use? The Web Media API Community Group discusses the development of guidelines for integrators in that space.
- Performance requirements in existing technologies
- Web technologies are designed with performance in mind, but they typically do not mandate measurable metrics on that front. For instance, it is common to talk in terms of frame rate in the video world but there is no requirement to playback video content at 30-60 frames per second, or to process events at any given rate on the Web. There are good reasons not to restrict implementations in that domain (e.g. so that they can reduce battery consumption when needed). There may be good reasons to specify minimum levels of support to guarantee user experiences too.