BT

InfoQ Homepage News High Performance Video Editing for the Mobile Web

High Performance Video Editing for the Mobile Web

Bookmarks

At Fullstack London 2019 conference, Forbes Lindesay, a software developer at Threads Styling, shared his experience of creating Video Renderer, a high-performance video editor that was written for a mobile web application.

To edit a video inside the browser, we use the Canvas API. While Canvas is usually used for editing still images, it's surprisingly simple to render video frames, and even entire videos, inside a canvas.

The following example is taken from video + canvas = magic

function draw(v,c,w,h) {
    if(v.paused || v.ended) return false;
    c.drawImage(v,0,0,w,h);
    setTimeout(draw,20,v,c,w,h);
}

It accepts an HTML video element, a Canvas element, and the width/height of our frame and uses drawImage to copy the current frame in the video into the Canvas element. Using a setTimeout with 20ms gives us a smooth 50fps video.

Once we copy the image into the Canvas we can read and edit individual frames. At this point, we can treat each frame as a still image and take advantage of the many libraries available in Github for Canvas photo manipulation.

The next capability we wish to add to our editor is layering. While it is quite simply to stack one video on top of the other, it's quite challenging to add drawings and animated gifs as they often include transparency, which does not work out of the box.

The simplest way to handle this requires an extra step on the server. Once a user uploads the image, we create a black and white copy of the image, where white represents visible parts of the image, and black represents the transparent areas. We then use the transparency 'guide' to determine which sections of the image should be drawn, and render the background video on the rest.

Creating the 'transparency' guide on the server can be done by using the Filter Complex method of the FFmpeg library, which allows us to create a graph of filters

There's one limitation that we need to discuss before you can start building a video editor in the browser. While editing the video inside the browser works quite well, compressing the video, especially on mobile phones, is not recommended. To overcome that, we replicate the video editing on the server and use FFmpeg to generate and compress the video. This is done by sending the server the original video along with a set of instructions that reproduce the editing done by the user.

This is a temporary solution. In the future, we could use WebAssembly to compile and run FFmpeg in the browser, in fact, there are experiments on Github but they are not quite ready for production.

These methods were used to create the Threads Styling mobile video editor which is available on Github under the MIT license.

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT

Is your profile up-to-date? Please take a moment to review and update.

Note: If updating/changing your email, a validation request will be sent

Company name:
Company role:
Company size:
Country/Zone:
State/Province/Region:
You will be sent an email to validate the new email address. This pop-up will close itself in a few moments.