Measuring and Comparing the Performance of 5 Cloud Platforms

Posted by Abel Avram on Jul 23, 2010

Sections

Architecture & Design,

Operations & Infrastructure

Topics

Performance & Scalability ,

Operations ,

Cloud Computing

Tags

Amazon Web Services ,

SaaS ,

Salesforce.com ,

PaaS ,

IaaS ,

Google AppEngine

Bitcurrent and Webmetrics have run a number of tests for a month on 5 different cloud platforms - Amazon, Google, Rackspace, Salesforce.com, and Terremark -, attempting to measure the performance of each platform. One of their conclusions is that each platform works better for different application types.

In order to test cloud performance, the authors of the report have selected 4 types of tests and benchmarking 5 native applications running on 5 different cloud platforms:

Tests

1. Requesting a Small Object – 1x1 pixel GIF
2. Requesting a Large Object – 2 MB image
3. Performing a CPU Intensive Task – 1,000,000 sine and sum operations. For Salesforce.com a 100,000 ops load was used because of a platform limitation.
4. Performing an IO Intensive Task – Querying a 500,000 rows table using a MySQL database with cleared cache for Amazon, Rackspace and Terremark, a data store for Salesforce.com, and a BigTable for Google.

Native Apps

To benchmark native applications, the authors chose 5 real-world websites developed for the platform they ran on. The websites were written in Apex for Salesforce.com, in Java and Python for GAE, websites running on Linux servers on Xen server for Amazon and Rackspace, and websites running on VMware VMs for Terremark. The names of the applications were not disclosed.

To perform the measurements, the authors used WebMetrics’ services, sending requests at various time intervals from multiple worldwide locations for a month.

Two of the cloud platforms, namely Salesforce.com and GAE, provide PaaS services, while the other three,
Amazon, Rackspace and Terremark, provide IaaS services.

Results

The latency of the 4 tests for all 5 cloud platforms were as following:

All platforms performed well for small objects, while PaaS platforms performed better than IaaS for larger objects. Salesforce.com performed poorly for CPU intensive tasks although the test included only 10% of the number of operations used on the other platforms. Google and Rackspace were best at the IO test.

The testing results for native applications look as following:

The PaaS clouds tested performed better, followed by Amazon and Rackspace, while Terremark came last due to a large percentage of responses at 12 seconds.

Conclusions

The authors of the performance measuring report drew up a number of conclusions summarizing the lessons learned during the tests:

• Watch your neighbors. We’ve seen good evidence that several cloud applications slow down at once, so you’ll definitely be affected by others using the same cloud as you.
• Understand the profile of your cloud. The histograms shown here demonstrate that different clouds are good at different tasks. You’ll need to choose the size of your virtual machines—in terms of CPU, memory, and so on—in order to deliver good performance.
• You need an agent on the inside. When you plan a monitoring strategy, you need custom code that exercised back-­‐end functions so you can triage the problem quickly.
• Choosing between PaaS or IaaS depends on your intended workload. If you’re willing to re-­‐code your application to take advantage of “big data” systems like Bigtable, you can scale well by choosing a PaaS cloud. On the other hand, if you need individual machines, you’ll have to build elasticity into your IaaS configuration.
• Big data doesn’t come for free. Using large, sharded data stores might seem nice; but it takes time to put the data in there (37 hours, in Google’s case!) which may not be appropriate for your application’s usage patterns.
• Monitor usage and governors. In PaaS, if you exceed your rate limits, your users will get errors.
• Troubleshooting gets harder. You need data on the Internet as a whole, your cloud provider as a whole, and your individual application’s various tiers, in order to properly triage the problem. When you were running on dedicated infrastructure, you didn’t have to spend as much time eliminating third-­‐party issues (such as contention for shared bandwidth or I/O blocking.)
• PaaS means you’re in the same basket. We noticed that if you’re using a PaaS, when the cloud gets slow, everyone gets slow. With IaaS, there’s more separation of the CPU and the server’s responsiveness—but you’re still contending for shared storage and network bandwidth.

The complete report contains detailed information collected during the performance benchmarks for each cloud platform.

Note: The authors advices to use the performance measurement results as a guideline only, considering that the results may vary depending on the type of workload, code, setup used, actual deployment environment, and other factors. The study is not intended to recommend a cloud platform over another.

1. Back to top

   Good report. But, should have compared PaaS and IaaS separately

   by mani doraisamy

   While datastore is part of PaaS architecture, IaaS is open about the choice of the datastore. Fixing MySQL as database for IaaS is inappropriate. (Performance Vs Scalability trade-off is the essence of SQL Vs NoSQL/distributed datastore architecture).
   
   PaaS vendors should be compared seperately from IaaS:
   manidoraisamy.blogspot.com/2009/01/who-are-comp...

   Reply

2. Back to top

   If you are redesigning your application for the cloud ...

   by Udayan Banerjee

   ... then here are some thoughts - setandbma.wordpress.com/2010/07/26/8-points-to-...

   Reply

3. Back to top

   Other comparisons

   by Vikas Hazrati

   Inphina ran a detailed analysis of various public cloud offerings and the results are present here.

   Reply

4. Back to top

   Cloud Platform Diagnostic

   by Rowan McCann

   Nice article. You might find this free cloud platform diagnostic interesting. It tells you which platform (azure, aws, appengine) is most suitable for your apps:
   
   www.whitestratus.com/cloud-platform-diagnostic-...

   Reply

• Older >