BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News Comparing the Performance of Various Web Frameworks

Comparing the Performance of Various Web Frameworks

Leia em Português

This item in japanese

Bookmarks

TechEmpower has been running benchmarks for the last year, attempting to measure and compare the performance of web frameworks. For these benchmarks the term “framework” is used loosely including platforms and micro-frameworks.

The results of the first benchmark were published in March 2013, and the tests were conducted on two hardware configurations: 2 Amazon EC2 m1.large instances and 2 Intel Sandy Bridge Core i7-2600K with 8GB of RAM, both setups using Gbit Ethernet. The frameworks had to reply with a simple JSON response {"message" : "Hello, World!"}. Following is an excerpt of the results, showing 10 of the most common web frameworks out of 24 benchmarked along with their results measured in Requests per second (RPS) and percentage of the best result.

image

Over the year, the benchmark evolved to cover 119 frameworks, including multiple types of queries and responses and running on one additional hardware configuration: 2 x Dell R720xd Dual-Xeon E5 v2 with 32GB of RAM each and 10 Gbit Ethernet.

Following are excerpts from the recently published 9th round of benchmarking, this time performed on three different hardware configurations:

image

Several observations:

While high-end hardware certainly performs better than lower end in terms of RPS, there is a major change in hierarchy for the top frameworks, as it is the case for Xeon E5 vs. Intel I7. This can be correlated with the fact that there are many at over 90% for Intel I7, TechEmpower attributing it to a bottleneck generated by the Ethernet 1Gbit used in the respective configuration:

Since the first round, we have known that the highest-performance frameworks and platforms have been network-limited by our gigabit Ethernet…. With ample network bandwidth, the highest-performing frameworks are further differentiated.

Google’s Go is the first on Intel I7, but has a major drop in performance on Xeon E5 in spite of using all cores, emphasizing the importance of performance tuning for certain types of hardware architectures, as noticed by TechEmpower:

The Peak [Xeon E5] test environment uses servers with 40 HT cores across two Xeon E5 processors, and the results illustrate how well each framework scales to high parallelism on a single server node.

Of course, other variables are at play, such as significantly more HT processor cores--40 versus 8 for our i7 tests--and a different system architecture, NUMA.

Amazon EC2 M1.large is severely underperforming compared with other configurations, making one wonder if the price/performance ratio is still attractive for cloud computing instances.

-*) Most frameworks are based on Linux. No Windows framework was benchmarked on EC2 or I7. The best Windows framework (plain-windows) performed at 18.6% of the leader on E5. Also, Facebook’s HHVM does not appear on EC2. It is also likely that HHVM is tuned differently when it runs on Facebook’s hardware.

Some of the well known frameworks are severely underperforming on all hardware configurations used, such as Grails, Spring, Django and Rails.

The performance of the recently released Dart 1.3 has doubled and should be on par with Node.js’ but it does not show in the benchmark because this round used Dart 1.2. This may be applicable to other frameworks that have been improved lately and the improvements will show up in future benchmarks.

As it is with other performance benchmarks, the results are debatable since they depend on the hardware, network and system configurations. TechEmpower invites those interested in bumping up the results for their favorite framework to fork the code on GitHub.

The benchmark includes detailed explanations on how tests are conducted, the hardware configurations, and the components of each framework: the language, web server, OS, etc.

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • About Golang

    by 项 超,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Maybe Golang1.3 is better. On the other hand, you can use pprof to analysis details.

  • Useless

    by Peter Pan,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    So all that was tested is how fast each framework responds with a JSON message. Did they have to respond to a parameterised request? Did they have to validate request parameters? Did they have to respond to POST or GET methods? Did they use data binding to populate the response message? Did they use a particular version of JSON parser? Etc, etc, etc. As such, the test is pretty useless, sorry you just wasted a year of your lives.

  • Re: Useless

    by Abel Avram,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    There are multiple types of tests performed, including reading and writing to a database. I mentioned the results for only one type of tests, the simple JSON one.
    While it is obvious that no performance benchmark can satisfy all requirements, I would not dismiss this test so easily.
    If a framework is in top 10, it does not automatically mean that it will do whatever I need it to do in production. One needs to perform some tests with his own configuration.
    But, on the other hand, if a framework is at the bottom of the list with only 10% of the performance of the top one in a simple hello test, what are the chances it shines on a real test?
    I would use this as a rough guidance not as a precise list of top 100 web frameworks.

  • Dart vs Go

    by Richard Eng,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    So, basically, even assuming Dart 1.3, Go trounces Dart in both the JSON test and the Plaintext test. It will be interesting to see if the latest Dart 1.8 can close the gap significantly. I am highly doubtful (remember, they're continually optimizing Go, too). If you care about performance, you will always choose Go over Dart on the server side.

  • Re: Dart vs Go

    by Antonio Abella,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Dart is dead

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT