Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Meta Open-Sourced AI Tool to Animate Child and Amateur Drawings of Human Figure

Meta Open-Sourced AI Tool to Animate Child and Amateur Drawings of Human Figure

Based on a joint research by Meta AI Research, Tencent America, MIT CSAIL, and Carnegie Mellon, Meta released Animated Drawings, an AI-based tool to create animations from hand drawn human-like characters.

Animated Drawings combines different techniques to reach its goal, including object detection to predict a tight bounding box encompassing the figure; image segmentation to separate pixels belonging to the human figure from those belonging to the background; and pose estimation to identify a series of skeletal joints. This makes it possible to create a digital version of a drawing which can then be animated using computer graphics techniques by applying motion data from preselected clips obtained from human performers.

According to the researchers, this project is a first step towards a drawing-to-animation storytelling system:

We demonstrate the value of our method in several ways. First, we explore the accuracy and success from each stage of our system as a function of training dataset size. Second, we perform a perceptual study to show the
appeal of twisted-perspective retargeting when animating these characters.

As the researchers explain, after an initial unsuccessful attempt to build a dataset from synthetic data, they set up a Web-based tool called Animated Drawings Demo to collect data from creators. Their initial objective was to collect about 10,000 drawings, but the overwhelming user response brought in over 3 million drawings, of which about 200,000 made it into the final dataset.

The results of the research, which is presented in a paper just accepted for publication on ACM Transactions on Graphics, are now available for experimental use thanks to Meta open-sourcing an implementation of the algorithm described in the paper.

(image courtesy of Meta)

Published on GitHub, Animated Drawings is written in Python and tested on macOS Ventura 13.2.1 and Ubuntu 18.04. The tool allows for easily creating an animation for any of the existing annotated drawings by using a command like:

from animated_drawings import render

In order to use additional drawings, the user must annotate them first, which can be done manually or more easily using a set of scripts to run a humanoid figure detector and pose estimator. The scripts rely on TorchServe and can be easily deployed using a provided Dockerfile. It is possible that the annotation files could require some manual tweaking, Meta says, for which a Web-based tool is also provided in the container.

As a final step, characters can be animated using a motion clip in BVH format. Meta provides a number of BVH animations including jumping, waving hello, and others. Additional BVH animations can be created from an RGB video using a tool like rokoko.

As mentioned Animated Drawings is just a first attempt to AI-based drawing-to-animation system and there exist multiple possibilities of improvements. Specifically, the researchers mention improving segmentation by better handling lack of focus or distortion, lighting effects or shadows, as well as using pre-animation classification of the human figure to improve pose estimation and animation.

About the Author

Rate this Article