BT
x Your opinion matters! Please fill in the InfoQ Survey about your reading habits!

Using Amazon Web Services to Implement a Video File Conversion app

by Gavin Terrill on Jul 24, 2007 |
As covered on InfoQ in the past, Amazon's infrastructure services platform is enabling new levels of cost savings as well as capabilities for certain classes of applications that can map to its scalable compute and storage services.  One recent sample application demonstrates building a video file conversion service using three key Amazon web services: Simple Storage Service (S3), Simple Queue Service (SQS), and Elastic Compute Cloud (EC2). 

S3 is used to store the files for conversion:

"Amazon S3 is the perfect place to store the video files to be converted as well as any output files generated by our conversion service. In addition to being fast and reliable, we will never have to worry about our service running out of disk space."

To make the service scalable and highly available, the design of the service is message-driven, utilizing SQS's reliable message delivery. This ensures that execution of client requests happens in the order they are received.

The ConvertVideo service is written in Python and utilizes the boto library, which provides a set of classes for integrating with Amazon Web Services. To provision the service to EC2, an AMI (Amazon Machine Image) file needs to be created and registered so that instances may be created on demand.

On the client side, the boto library provides a command line interface that can upload a directory of files to an S3 bucket, posting a messages to an SQS queue for each file. Once the files have been uploaded, a service instance can be started to process the messages in the queue.

To test scalability, an initial conversion run is performed on 50 videos by 1 instance:

  • Average Processing Time: 17.820000
  • Elapsed Time: 896
  • Throughput: 3.348214 transactions / minute

The next conversion run is based on 500 videos, and 10 instances:

  • Average Processing Time: 17.794000
  • Elapsed Time: 928
  • Throughput: 32.327586 transactions / minute

The additional service instances have increased throughput in a linear and predictable manner:

Sure enough, the average processing time and elapsed time are almost exactly the same but our overall throughput is roughly 10 times higher than in our previous example which is exactly the sort of behavior we would expect and hope for.

The tutorial breaks down the cost of converting the 500 videos:

Storage 2.5 GBytes $0.38/Month
Transfer 2.5 GBytes $0.50
Messages 1000 $0.10
Compute Resources 8 Instances for ~ 20 minutes $0.80
  Total: $1.78

A total of about $1.78 for converting 500 videos means a per/video cost of less than $0.004.

Compute services such as file conversion seem a good fit for the AWS infrastructure, however questions have been raised on the utility of the platform without an a database. Dare Obasanjo, in his blog posting "Amazon EC2 + S3 doesn't cut it", laments the lack of a database while experimenting with a Facebook application:

"it seems supporting this fairly straightforward application is beyond the current capabilities of EC2 + S3. S3 is primarily geared towards file storage so although it makes a good choice for cheaply hosting images and CSS stylesheets, it's a not a good choice for storing relational or structured data."

Of course, Amazon has deep experience in scaling out services. In his summary of the Google Seattle Scalability Conference, Robin Harris remarks on Amazon's CTO Verner Wogels memorable line: "Databases are Dinosaurs". Perhaps Dynamo, Amazon's scalable data store and due to be presented at SOSP 2007, is the remaining missing piece of the AWS puzzle.

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT