Processing Video Streams

One of the really "fun" things that you can do with WebRTC and media capture in general, is to do some post-processing on the captured video. This is a reasonably simple process whereby, you draw the video frames to a <canvas> element and then get the pixel data, manipulate the data and then push it back to the canvas.

As with other parts of the rtc.io suite, we've tried to make this as simple as possible, and much of the magic is achieved by using the rtc-canvas helper module.

The rtc-canvas module allows you to render a captured rtc-media stream to a virtual <video> element which redraws it's content on a canvas. This virtual video element also has a processing pipeline that you can add manipulation methods to using the .pipeline.add(handler) method.

For example, here is what you will commonly include if you wish to do some video manipulation using the rtc-canvas module:

var crel = require('crel');
var media = require('rtc-media');
var videoproc = require('rtc-videoproc');
var video = crel('video');
var canvas = crel('canvas');

// set up the video processing
videoproc(video, canvas, {
  filter: require('rtc-filter-grayscale')
});

// capture and render the video
media().render(video);

// add the canvas to the document body
document.body.appendChild(canvas);

In the code above, you can see that we are using a node style require statement to include a grayscale filter into our processing code. Let's have a look at the implementation of that small module now:

module.exports = function(imageData) {
  var channels = imageData.data;
  var channelCount = channels.length;

  // iterate through the data
  for (var ii = 0; ii < channelCount; ii += 4) {
    // update the values to the rgb average
    channels[ii] =       // update R
      channels[ii + 1] = // update G
      channels[ii + 2] = // update B
      (channels[ii] + channels[ii + 1] + channels[ii + 2] ) / 3;
  }

  // return true to flag that we want to write our pixel data
  // back to the canvas
  return true;
};

This code takes the image data that has already been extracted via the canvas 2d context getImageData method and applies some processing to it (in this case a simple grayscale filter).

By returning true at the end of the function, we are flagging to the rtc-canvas module that modifications have been made to the reference pixel data and that a putImageData call should be made to replace the contents of the canvas.

Run Sample

Adjusting the Capture Rate

By default, the canvas is set to capture at 25 fps, but this can be adjusted if you feel like giving your CPU a bit of a rest:

var crel = require('crel');
var media = require('rtc-media');
var videoproc = require('rtc-videoproc');
var video = crel('video');
var canvas = crel('canvas');

// set up the video processing
videoproc(video, canvas, {
  fps: 5,
  filter: require('rtc-filter-grayscale')
});

// capture and render the video
media().render(video);

// add the canvas to the document body
document.body.appendChild(canvas);

Run Sample