BT
x Your opinion matters! Please fill in the InfoQ Survey about your reading habits!

Comparing the Performance of Various Web Frameworks

by Abel Avram on May 02, 2014 |

TechEmpower has been running benchmarks for the last year, attempting to measure and compare the performance of web frameworks. For these benchmarks the term “framework” is used loosely including platforms and micro-frameworks.

The results of the first benchmark were published in March 2013, and the tests were conducted on two hardware configurations: 2 Amazon EC2 m1.large instances and 2 Intel Sandy Bridge Core i7-2600K with 8GB of RAM, both setups using Gbit Ethernet. The frameworks had to reply with a simple JSON response {"message" : "Hello, World!"}. Following is an excerpt of the results, showing 10 of the most common web frameworks out of 24 benchmarked along with their results measured in Requests per second (RPS) and percentage of the best result.

image

Over the year, the benchmark evolved to cover 119 frameworks, including multiple types of queries and responses and running on one additional hardware configuration: 2 x Dell R720xd Dual-Xeon E5 v2 with 32GB of RAM each and 10 Gbit Ethernet.

Following are excerpts from the recently published 9th round of benchmarking, this time performed on three different hardware configurations:

image

Several observations:

While high-end hardware certainly performs better than lower end in terms of RPS, there is a major change in hierarchy for the top frameworks, as it is the case for Xeon E5 vs. Intel I7. This can be correlated with the fact that there are many at over 90% for Intel I7, TechEmpower attributing it to a bottleneck generated by the Ethernet 1Gbit used in the respective configuration:

Since the first round, we have known that the highest-performance frameworks and platforms have been network-limited by our gigabit Ethernet…. With ample network bandwidth, the highest-performing frameworks are further differentiated.

Google’s Go is the first on Intel I7, but has a major drop in performance on Xeon E5 in spite of using all cores, emphasizing the importance of performance tuning for certain types of hardware architectures, as noticed by TechEmpower:

The Peak [Xeon E5] test environment uses servers with 40 HT cores across two Xeon E5 processors, and the results illustrate how well each framework scales to high parallelism on a single server node.

Of course, other variables are at play, such as significantly more HT processor cores--40 versus 8 for our i7 tests--and a different system architecture, NUMA.

Amazon EC2 M1.large is severely underperforming compared with other configurations, making one wonder if the price/performance ratio is still attractive for cloud computing instances.

-*) Most frameworks are based on Linux. No Windows framework was benchmarked on EC2 or I7. The best Windows framework (plain-windows) performed at 18.6% of the leader on E5. Also, Facebook’s HHVM does not appear on EC2. It is also likely that HHVM is tuned differently when it runs on Facebook’s hardware.

Some of the well known frameworks are severely underperforming on all hardware configurations used, such as Grails, Spring, Django and Rails.

The performance of the recently released Dart 1.3 has doubled and should be on par with Node.js’ but it does not show in the benchmark because this round used Dart 1.2. This may be applicable to other frameworks that have been improved lately and the improvements will show up in future benchmarks.

As it is with other performance benchmarks, the results are debatable since they depend on the hardware, network and system configurations. TechEmpower invites those interested in bumping up the results for their favorite framework to fork the code on GitHub.

The benchmark includes detailed explanations on how tests are conducted, the hardware configurations, and the components of each framework: the language, web server, OS, etc.

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

About Golang by 项 超

Maybe Golang1.3 is better. On the other hand, you can use pprof to analysis details.

Useless by Peter Pan

So all that was tested is how fast each framework responds with a JSON message. Did they have to respond to a parameterised request? Did they have to validate request parameters? Did they have to respond to POST or GET methods? Did they use data binding to populate the response message? Did they use a particular version of JSON parser? Etc, etc, etc. As such, the test is pretty useless, sorry you just wasted a year of your lives.

Re: Useless by Abel Avram

There are multiple types of tests performed, including reading and writing to a database. I mentioned the results for only one type of tests, the simple JSON one.
While it is obvious that no performance benchmark can satisfy all requirements, I would not dismiss this test so easily.
If a framework is in top 10, it does not automatically mean that it will do whatever I need it to do in production. One needs to perform some tests with his own configuration.
But, on the other hand, if a framework is at the bottom of the list with only 10% of the performance of the top one in a simple hello test, what are the chances it shines on a real test?
I would use this as a rough guidance not as a precise list of top 100 web frameworks.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

3 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT