Video event firing

To fire a video event named e, the user agent must run the following steps:

1. If e is videomonitorchange, create a VideoMonitorEvent, initialize it to have a given name e, to not bubble, to not be cancelable.
2. If e is videoprocessorchange, create a VideoProcessorEvent, initialize it to have a given name e, to not bubble, to not be cancelable.
3. Initialize the trackId attribute to the value of the id attribute of the input media stream track.
4. Initialize the playbackTime attribute to the value of the currentTime attribute of the MediaStream that contains the input media stream track.
5. Initialize the inputImageBitmap attribute to the bitmap data of the input media stream track's video frame at the current stream position at playbackTime.
6. If e is videomonitorchange, dispatch the newly created VideoMonitorEvent object at each video monitor that is monitoring.
7. If e is videoprocessorchange, initialize the outputImageBitmap attribute to null, and dispatch the newly created VideoProcessorEvent object at each target.

MediaStreamTrack interface

          partial interface MediaStreamTrack {
              void              addVideoMonitor(VideoMonitor monitor);
              void              removeVideoMonitor(VideoMonitor monitor);
              MediaStreamTrack  addVideoProcessor(VideoProcessor processor);
              void              removeVideoProcessor();
          };
        

Image format

An image or a video frame is conceptually a two-dimentional array of data and each element in the array is called a pixel. The pixels are usually stored in a one-dimensional array and could be arranged in a variety of image formats. Developers need to know how the pixels are formatted so that they are able to process them.

The image format describes how pixels in an image are arranged. A single pixel has at least one, but usually multiple pixel values. The range of a pixel value varies, which means different image formats use different data types to store a single pixel value.

The most frequently used data type is 8-bit unsigned integer whose range is from 0 to 255, others could be 16-bit integer or 32-bit floating points and so forth. The number of pixel values of a single pixel is called the number of channels of the image format. Multiple pixel values of a pixel are used together to describe the captured property which could be color or depth information. For example, if the data is a color image in RGB color space, then it is a three-channel image format and a pixel is described by R, G and B three pixel values with range from 0 to 255. As another example, if the data is a gray image, then it is a single-channel image format with 8-bit unsigned integer data type and the pixel value describes the gray scale. For depth data, it is a single channel image format too, but the data type is 16-bit unsigned integer and the pixel value is the depth level.

For those image formats whose pixels contain multiple pixel values, the pixel values might be arranged in one of the following ways:

1. Planar pixel layout: each channel has its pixel values stored consecutively in separated buffers (a.k.a. planes) and then all channel buffers are stored consecutively in memory. (Ex: RRRRRR......GGGGGG......BBBBBB......)
2. Interleaving pixel layout: each pixel has its pixel values from all channels stored together and interleaves all channels. (Ex: RGBRGBRGBRGBRGB......)

            enum ImageFormat {
                "RGBA32",
                "BGRA32",
                "RGB24",
                "BGR24",
                "GRAY8",
                "YUV444P",
                "YUV422P",
                "YUV420P",
                "YUV420SP_NV12",
                "YUV420SP_NV21",
                "HSV",
                "Lab",
                "DEPTH",
                /* empty string */ ""
            };
          

            enum ChannelPixelLayoutDataType {
                "uint8",
                "int8",
                "uint16",
                "int16",
                "uint32",
                "int32",
                "float32",
                "float64"
            };
          

Pixel layout

For generalizing the variety of pixel layouts among image formats, here we propose the dictionary ChannelPixelLayout and the ImagePixelLayout which is a sequence of ChannelPixelLayout.

The ImagePixelLayout represents the pixel layout of a certain image format. Since an image format is composed of at least one channel, an ImagePixelLayout object contains at least one ChannelPixelLayout object.

Although an image or a video frame is a two-dimensional structure, its data is usually stored in a one-dimensional array in the row-major way and each channel describes how pixel values are arranged in the one dimensional array buffer.

A channel has an associated offset that denotes the beginning position of the channel's data relative to the given BufferSource parameter of the mapDataInto() method.

A channel has an associated width and height that denote the width and height of the channel respectively. Each channel in an image format may have different height and width.

A channel has an associated data type used to store one single pixel value.

A channel has an associated stride that is the number of bytes between the beginning two consecutive rows in memory. (The total bytes of each row plus the padding bytes of each raw.)

A channel has an associated skip value. The value is zero for the planar pixel layout, and a positive integer for the interleaving pixel layout. (Describes how many bytes there are between two adjacent pixel values in this channel.)

          Example1: RGBA image, width = 620, height = 480, stride = 2560

          chanel_r: offset = 0, width = 620, height = 480, data type = uint8, stride = 2560, skip = 3
          chanel_g: offset = 1, width = 620, height = 480, data type = uint8, stride = 2560, skip = 3
          chanel_b: offset = 2, width = 620, height = 480, data type = uint8, stride = 2560, skip = 3
          chanel_a: offset = 3, width = 620, height = 480, data type = uint8, stride = 2560, skip = 3

                  <---------------------------- stride ---------------------------->
                  <---------------------- width x 4 ---------------------->
          [index] 01234   8   12  16  20  24  28                           2479    2559
                  |||||---|---|---|---|---|---|----------------------------|-------|
          [data]  RGBARGBARGBARGBARGBAR___R___R...                         A%%%%%%%%
          [data]  RGBARGBARGBARGBARGBAR___R___R...                         A%%%%%%%%
          [data]  RGBARGBARGBARGBARGBAR___R___R...                         A%%%%%%%%
                       ^^^
                       r-skip
        

          Example2: YUV420P image, width = 620, height = 480, stride = 640

          chanel_y: offset = 0, width = 620, height = 480, stride = 640, skip = 0
          chanel_u: offset = 307200, width = 310, height = 240, data type = uint8, stride = 320, skip = 0
          chanel_v: offset = 384000, width = 310, height = 240, data type = uint8, stride = 320, skip = 0

                  <--------------------------- y-stride --------------------------->
                  <----------------------- y-width ----------------------->
          [index] 012345                                                  619      639
                  ||||||--------------------------------------------------|--------|
          [data]  YYYYYYYYYYYYYYYYYYYYYYYYYYYYY...                        Y%%%%%%%%%
          [data]  YYYYYYYYYYYYYYYYYYYYYYYYYYYYY...                        Y%%%%%%%%%
          [data]  YYYYYYYYYYYYYYYYYYYYYYYYYYYYY...                        Y%%%%%%%%%
          [data]  ......
                  <-------- u-stride ---------->
                  <----- u-width ----->
          [index] 307200              307509   307519
                  |-------------------|--------|
          [data]  UUUUUUUUUU...       U%%%%%%%%%
          [data]  UUUUUUUUUU...       U%%%%%%%%%
          [data]  UUUUUUUUUU...       U%%%%%%%%%
          [data]  ......
                  <-------- v-stride ---------->
                  <- --- v-width ----->
          [index] 384000              384309   384319
                  |-------------------|--------|
          [data]  VVVVVVVVVV...       V%%%%%%%%%
          [data]  VVVVVVVVVV...       V%%%%%%%%%
          [data]  VVVVVVVVVV...       V%%%%%%%%%
          [data]  ......
        

          Example3: YUV420SP_NV12 image, width = 620, height = 480, stride = 640

          chanel_y: offset = 0, width = 620, height = 480, stride = 640, skip = 0
          chanel_u: offset = 307200, width = 310, height = 240, data type = uint8, stride = 640, skip = 1
          chanel_v: offset = 307201, width = 310, height = 240, data type = uint8, stride = 640, skip = 1

                  <--------------------------- y-stride -------------------------->
                  <----------------------- y-width ---------------------->
          [index] 012345                                                 619      639
                  ||||||-------------------------------------------------|--------|
          [data]  YYYYYYYYYYYYYYYYYYYYYYYYYYYYY...                       Y%%%%%%%%%
          [data]  YYYYYYYYYYYYYYYYYYYYYYYYYYYYY...                       Y%%%%%%%%%
          [data]  YYYYYYYYYYYYYYYYYYYYYYYYYYYYY...                       Y%%%%%%%%%
          [data]  ......
                  <--------------------- u-stride / v-stride -------------------->
                  <------------------ u-width + v-width ----------------->
          [index] 307200(u-offset)                                       307819  307839
                  |------------------------------------------------------|-------|
          [index] |307201(v-offset)                                      |307820 |
                  ||-----------------------------------------------------||------|
          [data]  UVUVUVUVUVUVUVUVUVUVUVUVUVUVUV...                      UV%%%%%%%
          [data]  UVUVUVUVUVUVUVUVUVUVUVUVUVUVUV...                      UV%%%%%%%
          [data]  UVUVUVUVUVUVUVUVUVUVUVUVUVUVUV...                      UV%%%%%%%
                   ^            ^
                  u-skip        v-skip
        

          Example4: DEPTH image, width = 640, height = 480, stride = 1280

          chanel_d: offset = 0, width = 640, height = 480, data type = uint16, stride = 1280, skip = 0

                  <----------------------- d-stride ---------------------->
                  <----------------------- d-width ----------------------->
          [index] 012345                                                  1280
                  ||||||--------------------------------------------------|
          [data]  DDDDDDDDDDDDDDDDDDDDDDDDDDDDD...                        D
          [data]  DDDDDDDDDDDDDDDDDDDDDDDDDDDDD...                        D
          [data]  DDDDDDDDDDDDDDDDDDDDDDDDDDDDD...                        D
          [data]  ......
        

            dictionary ChannelPixelLayout {
                required unsigned long offset;
                required unsigned long width;
                required unsigned long height;
                required ChannelPixelLayoutDataType dataType;
                required unsigned long stride;
                required unsigned long skip;
            };
          

            typedef sequence<ChannelPixelLayout> ImagePixelLayout;
          

ImageBitmap interface

          [Exposed=(Window,Worker)]
          partial interface ImageBitmap {
              ImageFormat                     findOptimalFormat (optional sequence<ImageFormat> possibleFormats);
              long                            mappedDataLength (ImageFormat format);
              Promise<ImagePixelLayout> mapDataInto (ImageFormat format, BufferSource buffer, long offset, long length);
          };
        

ImageBitmapFactories interface

          [NoInterfaceObject, Exposed=(Window,Worker)]
          partial interface ImageBitmapFactories {
              Promise<ImageBitmap> createImageBitmap (BufferSource buffer, long offset, long length, ImageFormat format, ImagePixelLayout layout);
          };
        

          <script>
          var processor = new VideoProcessor();
          var inputTrack = mediaStream.getVideoTracks()[0];
          var outputTrack = inputTrack.addVideoProcessor(processor);
          newMediaStream.addTrack(outputTrack);
          var worker = new Worker("processing.js");
          worker.postMessage({aCommand : 'pass_processor', aProcessor: processor}, [processor]);
          </script>
        

The worker executes the following processing.js script:

          self.onmessage = function(msg) {
            switch (msg.data.aCommand) {
                  case 'pass_processor':
                          bindProcessor(msg.data.aProcessor);
                      break;
                  default:
                      throw 'no aTopic on incoming message to Worker';
              }
          };

          function bindProcessor(processor) {
            processor.onvideoprocessorchange = function(event) {
              // Check if the browser supports YUV format.
              var bitmap = event.inputImageBitmap;
              var yuvFormats = ["YUV444P", "YUV422P", "YUV420P", "YUV420SP_NV12", "YUV420SP_NV21"];
              var bitmapFormat = bitmap.findOptimalFormat(yuvFormats);
              if (bitmapFormat == "") {
                console.log("The browser does not support YUV formats.");
                return;
              }
              // Get the need buffer size to read the image data in YUV format.
              var bitmapBufferLength = bitmap.mappedDataLength(bitmapFormat);

              // Create the buffer for mapping data out.
              var bitmapBuffer = new ArrayBuffer(bitmapBufferLength);
              var bitmapBufferView = new Uint8ClampedArray(bitmapBuffer, 0, bitmapBufferLength);

              // Map the bitmap's data into the buffer created in the previous step.
              var promise = bitmap.mapDataInto(bitmapFormat, bitmapBuffer, 0, bitmapBufferLength);
              promise.then(function(bitmapPixelLayout) {
                // Read out the y-channel properties.
                var ywidth  = bitmapPixelLayout.channels[0].width;
                var yheight = bitmapPixelLayout.channels[0].height;
                var yoffset = bitmapPixelLayout.channels[0].offset;
                var ystride = bitmapPixelLayout.channels[0].stride;
                var yskip   = bitmapPixelLayout.channels[0].skip;   // This should be 0.

                // Initialize the buffer for the result gray image.
                var rgbaBufferLength = ywidth * yheight * 4;
                var rgbaBuffer = new ArrayBuffer(rgbaBufferLength);
                var rgbaBufferView = new Uint8ClampedArray(rgbaBuffer, 0, rgbaBufferLength);

                // Convert YUV to Gray.
                for (var i = 0; i < yheight; ++i) {
                  for (var j = 0; j < ywidth; ++j) {
                    var index = ystride * i + j;
                    var y = parseFloat(bitmapBufferView[yoffset + index]);
                    rgbaBufferView[index * 4 + 0] = y;
                    rgbaBufferView[index * 4 + 1] = y;
                    rgbaBufferView[index * 4 + 2] = y;
                    rgbaBufferView[index * 4 + 3] = 255;
                  }
                }

                // Create a new ImageBitmap from the processed rgbaBuffer and assign to the
                // event.outputImageBitmap.
                var channelR = {offset:0, width:ywidth, height:yheight, dataType:"uint8", stride:ywith * 4, skip:3);
                var channelG = {offset:1, width:ywidth, height:yheight, dataType:"uint8", stride:ywith * 4, skip:3);
                var channelB = {offset:2, width:ywidth, height:yheight, dataType:"uint8", stride:ywith * 4, skip:3);
                var channelA = {offset:3, width:ywidth, height:yheight, dataType:"uint8", stride:ywith * 4, skip:3);
                var layout = [channelR, channelG, channelB, channelA];
                var p = createImageBitmap(rgbaBuffer, 0, rgbaBufferLength, "RGBA32", layout);
                p.then(function(bitmap) {
                  event.outputImageBitmap = bitmap;
                }).catch(function(ex) {
                  console.log("Call createImageBitmap() failed. Error: " + ex);
                });
              },
              function(ex) {
                console.log("Call mapDataInto() failed. Error: " + ex);
              });
            };
          }