Well-deployed technologies

The Canvas API enables pixel-level manipulation of images and, by extension, video frames. One of its limitations is that it operates using the CPU instead of the more optimized operations enabled by modern GPUs.

The Media Source Extensions API (MSE) allows applications to insert chunks of media (e.g. a video clip) into an existing media stream, and implement adaptive streaming algorithms at the application level.

For the distribution of media whose content needs specific protection from being copied, the Encrypted Media Extensions specification enables the decryption and rendering of encrypted media streams based on Content Decryption Modules (CDM).

The Web Audio API provides a full-fledged audio processing and synthesis API with low-latency guarantees and hardware-accelerated operations when possible.

FeatureSpecification / GroupMaturityCurrent implementations
Select browsers…
Image and Video ProcessingHTML Canvas 2D Context
HTML Media Extensions Working Group
Adaptive streamingMedia Source Extensions™
HTML Media Extensions Working Group
Protected MediaEncrypted Media Extensions
HTML Media Extensions Working Group
Audio ProcessingWeb Audio API
Audio Working Group
Candidate Recommendation

Exploratory work

Video processing using the Canvas API is very CPU-intensive. Beyond traditional video processing, modern GPUs often provide advanced vision processing capabilities (e.g. face and objects recognition) that would have direct applicability e.g. in augmented reality applications. The Shape Detection API is exploring this space.

Some scenarios require the ability to switch to a different codec during media playback. For instance, this need may arise on program boundaries when watching linear content. Also, typical ads are of a very different nature than the media stream that they need to be inserted into. The first version of the Media Source Extensions (MSE) specification does not address such heterogeneous scenarios, forcing content providers to use complex workarounds, e.g. using two <video> elements overlaid one top of the other with no guarantees that transition between streams will be smooth, or transcoding and re-packaging the ad content to be consistent with the program content. The Web Platform Incubator Working Group is incubating a codec switching feature for MSE, to be integrated in a second version of MSE.

Features not covered by ongoing work

JavaScript-based Codecs
The algorithms used to compress and decompress bandwidth-intensive media content are required to be provided by the browsers at this time; a system enabling these algorithms to be written and distributed in JavaScript or Web Assembly and get them integrated in the overall media flow of user agents would provide much greater freedom to specialize and innovate in this space (see related discussion on W3C’s discourse forum).
Content Decryption Module API
The capabilities offered by the Encrypted Media Extensions rely on the integration with undefined interfaces for the Content Decryption Modules. Providing a uniform interface for that integration would simplify the addition of new CDMs in the market.
Conditional Access System
Broadcasters use a different approach to protect the content they distribute, known as Conditional Access System (CAS). As broadcast streams are coming to Web browsers, providing integration with these systems may be needed.
Hardware-accelerated video processing
The Canvas API provides capabilities to do image and video processing, but these capabilities are limited by their reliance on the CPU for execution; modern GPUs provide hardware-acceleration for a wide range of operations, but browsers don't provide hooks to these. The GPU for the Web Community Group is discussing solutions to expose GPU computation functionality to Web applications, which could eventually allow web applications to process video streams efficiently, taking advantage of the GPU power.

Discontinued features

Media Capture Stream with Worker
Video processing using the Canvas API is very CPU intensive, and as such, can benefit from executing separately from the rest of a Web application. The Media Capture Stream with Worker specification was an early approach to process video streams in a dedicated worker thread. That processing would still be done on the CPU though, and work on this document has been discontinued. Work in the GPU for the Web Community Group could eventually allow Web applications to process video streams directly using the power of the GPU.