At Fullstack London 2019 conference, Forbes Lindesay, a software developer at Threads Styling, shared his experience of creating Video Renderer, a high-performance video editor that was written for a mobile web application.
To edit a video inside the browser, we use the Canvas API. While Canvas is usually used for editing still images, it's surprisingly simple to render video frames, and even entire videos, inside a canvas.
The following example is taken from video + canvas = magic
function draw(v,c,w,h) {
if(v.paused || v.ended) return false;
c.drawImage(v,0,0,w,h);
setTimeout(draw,20,v,c,w,h);
}
It accepts an HTML video element, a Canvas element, and the width/height of our frame and uses drawImage to copy the current frame in the video into the Canvas element. Using a setTimeout with 20ms gives us a smooth 50fps video.
Once we copy the image into the Canvas we can read and edit individual frames. At this point, we can treat each frame as a still image and take advantage of the many libraries available in Github for Canvas photo manipulation.
The next capability we wish to add to our editor is layering. While it is quite simply to stack one video on top of the other, it's quite challenging to add drawings and animated gifs as they often include transparency, which does not work out of the box.
The simplest way to handle this requires an extra step on the server. Once a user uploads the image, we create a black and white copy of the image, where white represents visible parts of the image, and black represents the transparent areas. We then use the transparency 'guide' to determine which sections of the image should be drawn, and render the background video on the rest.
Creating the 'transparency' guide on the server can be done by using the Filter Complex method of the FFmpeg library, which allows us to create a graph of filters
There's one limitation that we need to discuss before you can start building a video editor in the browser. While editing the video inside the browser works quite well, compressing the video, especially on mobile phones, is not recommended. To overcome that, we replicate the video editing on the server and use FFmpeg to generate and compress the video. This is done by sending the server the original video along with a set of instructions that reproduce the editing done by the user.
This is a temporary solution. In the future, we could use WebAssembly to compile and run FFmpeg in the browser, in fact, there are experiments on Github but they are not quite ready for production.
These methods were used to create the Threads Styling mobile video editor which is available on Github under the MIT license.