Considering Grails vs Rails Benchmarks

Posted by Scott Delap on Mar 23, 2007 09:57 AM

Community

Java,

Ruby

Topics

Web Frameworks

Tags

Rails,

Grails,

Mongrel

...I can't find a SINGLE benchmark comparing Groovy to competing technologies on the JVM (JRuby, Beanshell, Jython) or dynamic languages in general (Ruby, Python, Perl, and so on). Nor can I find any benchmarks concerning how Grails stacks up to JRuby on Rails, Ruby on Rails, vanilla Hibernate/Spring (something Grails uses under the covers), etc...

In response to this the grails community posted some rough unscientific numbers on Grails versus Rails. Among the benchmarks results:

[In respect to read performance]...Grails is able to handle 40 requests per second to Rails 32. Significantly you can notice Grails has a much poorer worst case response time, but a superior mean response time (The mean time is the average time taken per request over all 1000 requests). This can be put down the Java threading model and the Thread pool in Tomcat at some point running out of available threads and blocking for a moment. It just reflects the differences in architecture more than anything else...

[In respect to creating records] ... Grails is able to cope with 140 requests per second and has a mean average of 345ms per request compared to Rails' 35 (and a half) and 1.3 seconds per request. I believe, without having done any real profiling, the reasons for this can largely put down to Hibernate's session-based model. Each call to save() in ActiveRecord immediately persists the data whilst Hibernate batches updates up until the end...

[In respect to querying] ...the results here are much more equal as it is really reliant on database performance. Notably we can see Grails' worst case is poor down to blocking in the thread pool, but overall the mean time of 5.387ms is superior to Rails' 6.386, which illustrates why Grails' is able to deal with 8.92 requests per second as oppose to Rails' 7.65...

The article finishes by clearly stating that the authors are not Rails performance tuning wizards and are welcome to suggestions in optimizing Rails. The tests conducted were solely to inform Grails users asking for figures not as a definitive benchmark.

The Rails community has not gone quiet in regards to the benchmark data. Noted Rails developer and Mongrel user Jared Richardson commented in response:

...When your application creates a new instance or thread for every new request, it's easy for a server to get overwhelmed. Rails takes another approach. The Rails developers want you to make a conscious decision about how many resources you want an application to have ... So when you deploy a Rails application on Mongrel, you need to decide how many instances are available. This is completely different from the traditional Tomcat model, which looks something more like "spin up threads until the machine stops responding." ... If I've read the page properly, this benchmark is running against a single Mongrel instance, which is the equivalent of creating a Java singleton servlet and testing against it. Of course it's going to be quite slow ... For a real comparison, I'd try using the Apache load balancer plugin or my personal favorite Pound.

1. Back to top

   Benchmarks

   Mar 23, 2007 11:05 AM by Martin Gilday

   There has been plenty of talk recently on the Groovy mailing list about performance of the language in general, including comparisons with JRuby and c Ruby.

   Reply

2. Back to top

   Better App Server

   Mar 23, 2007 11:13 AM by Matt Giacomini

   Be interesting to see how grails performs under Jetty.

   Reply

3. Back to top

   Thread model vs. Process model

   Mar 23, 2007 2:52 PM by steven Vetzal

   You know, as long as you understand each model, it's not hard to work with either. Jared Richardson's comment doesn't show much knowledge of Tomcat. It hardly hands out new instances each request, you can configure it just as nicely as you can set up new mongrel instances - except you don't have the 20-50MB of process overhead with just an extra thread :)

   Reply

4. Back to top

   Not a language comparison

   Mar 23, 2007 3:13 PM by Tom Nichols

   This is a great comparison for the frameworks, but keep in mind this does not equate to relative language performance. Large portions of Grails are written in Java, while Rails is (AFAIK) all Ruby. All of this will likely change when YARV is complete and as JRuby improves as well. Nonetheless, from the POV of a user of either framework, the comparison is valid. Personally, as a Java user I would like to see Grails' performance compared to similar Java stacks... Say, Struts2 + Hibernate. Then you could see the trade-off of Grails' productivity for performance.

   Reply

5. Back to top

   Re: Thread model vs. Process model

   Mar 23, 2007 3:18 PM by jared richardson

   You know, as long as you understand each model, it's not hard to work with either.

   I agree completely. My intention wasn't to attack either point of view but to point out that the multi-Mongrel approach is required for a fair benchmark.

   Jared Richardson's comment doesn't show much knowledge of Tomcat. It hardly hands out new instances each request,

   My intended point was that Tomcat creates a new thread with each request, not a new instance. My blog post makes that a bit clearer, but I think the quoted bits make that pretty clear. Is my understanding incorrect?

   Reply

6. Back to top

   No mention of just-in-time compilation

   Mar 23, 2007 3:27 PM by James Tikalsky

   The article doesn't mention Just-in-Time compilation provided by the JVM they're running Grails on. For all we know, simply "warming up" the application with a few thousand requests before they took their measurements may have increased the speed significantly. Also, I believe that Groovy may support dynamic compilation of Groovy code (I may be wrong), which would subject Groovy code to JIT optimizations as well.

   Reply

7. Back to top

   Re: Thread model vs. Process model

   Mar 23, 2007 4:11 PM by steven Vetzal

   Nope, you don't get a thread unless there's one in the pool to accept the request. The pool has a min and max bounds, default max bounds I personally think is a little high (something like 150 threads). Depending upon your workload this may or may not be appropriate for you - if your workload involves a lot of waiting around for things, lots of threads isn't bad (helps balance wait time and keep the processor best utilized), but if they're all processing, you lose a lot in context switching. If memory serves, the connection hangs until there is a thread available or until a socket timeout occurs. You can always set the max threads down to 3-4 if you do a lot of processing in your requests. IMHO the big disadvantage of the thread model is synchronization across servers. If your app depends on internal state, replicating that state across a server farm is a different problem.

   Reply

8. Back to top

   New results with 10 mongrels...

   Mar 23, 2007 8:22 PM by John Wells

   The original test was, quite innocently I believe, invalid, as it only tested Tomcat against one mongrel process. I've updated my blog with the same tests run against 10 mongrels + nginx. http://thoughtmining.blogspot.com/2007/03/grails-versus-rails-comparison.html

   Reply

9. Back to top

   Re: New results with 10 mongrels...

   Mar 24, 2007 10:31 AM by Graeme Rocher

   I updated the benchmark and configured Rails with 10 Mongrels and Pound as a load balancer as per Jared's blog. I also re-did the Grails tests with a tomcat maxThreads of 10 and the results were the Grails performed better and Rails performed worse: http://docs.codehaus.org/display/GRAILS/Grails+vs+Rails+Benchmark Again, I appeal to the Rails community to suggest optimisation and improvements to the config of Rails. I will try out John's (above) suggestion of using nginx instead of Pound as a load balancer Cheers Graeme

   Reply

10. Back to top

    Re: New results with 10 mongrels...

    Feb 7, 2008 9:41 PM by Mohammad Ali

    Sorry for replying 8 months late, Redo the test on the same machine with 3 mongrels only + pound (or nginx which is better) since you have 2 cores, the best count would be 3 or 4, but since you have only 1 gig of ram then 3 should be your sweet spot. As for your previous test with 10 Mongrels, you were SWAPPING memory!! I believe that limiting threads for tomcat would cause them to queue more requests rather than spawning threads, which is a heavy process with a high cost of context switching (you only have 2 cores)

    Reply

• Older >