The Canvas API enables pixel-level manipulation of images and, by extension, video frames. One of its limitations is that it operates using the CPU instead of the more optimized operations enabled by modern GPUs.
The Media Source Extensions API (MSE) allows applications to insert chunks of media (e.g. a video clip) into an existing media stream, and implement adaptive streaming algorithms at the application level.
For the distribution of media whose content needs specific protection from being copied, the Encrypted Media Extensions specification enables the decryption and rendering of encrypted media streams based on Content Decryption Modules (CDM).
The Web Audio API provides a full-fledged audio processing and synthesis API with low-latency guarantees and hardware-accelerated operations when possible.
Video processing using the Canvas API is very CPU-intensive. Beyond traditional video processing, modern GPUs often provide advanced vision processing capabilities (e.g. face and objects recognition) that would have direct applicability e.g. in augmented reality applications. The Shape Detection API is exploring this space.
Some scenarios require the ability to switch to a different codec during media playback. For instance, this need may arise on program boundaries when watching linear content. Also, typical ads are of a very different nature than the media stream that they need to be inserted into. The first version of the Media Source Extensions (MSE) specification does not address such heterogeneous scenarios, forcing content providers to use complex workarounds, e.g. using two
<video> elements overlaid one top of the other with no guarantees that transition between streams will be smooth, or transcoding and re-packaging the ad content to be consistent with the program content. The Web Platform Incubator Working Group is incubating a codec switching feature for MSE, to be integrated in a second version of MSE.
|Feature||Specification / Group||Implementation intents|
|Video processing||Accelerated Shape Detection in Images|
Web Platform Incubator Community Group
|Adaptive streaming||Codec switching in Media Source Extensions™|
Web Platform Incubator Community Group
Features not covered by ongoing work
- Content Decryption Module API
- The capabilities offered by the Encrypted Media Extensions rely on the integration with undefined interfaces for the Content Decryption Modules. Providing a uniform interface for that integration would simplify the addition of new CDMs in the market.
- Conditional Access System
- Broadcasters use a different approach to protect the content they distribute, known as Conditional Access System (CAS). As broadcast streams are coming to Web browsers, providing integration with these systems may be needed.
- Hardware-accelerated video processing
- The Canvas API provides capabilities to do image and video processing, but these capabilities are limited by their reliance on the CPU for execution; modern GPUs provide hardware-acceleration for a wide range of operations, but browsers don't provide hooks to these. The GPU for the Web Community Group is discussing solutions to expose GPU computation functionality to Web applications, which could eventually allow web applications to process video streams efficiently, taking advantage of the GPU power.
- Media Capture Stream with Worker
- Video processing using the Canvas API is very CPU intensive, and as such, can benefit from executing separately from the rest of a Web application. The Media Capture Stream with Worker specification was an early approach to process video streams in a dedicated worker thread. That processing would still be done on the CPU though, and work on this document has been discontinued. Work in the GPU for the Web Community Group could eventually allow Web applications to process video streams directly using the power of the GPU.