Lessons from building Oracle Mix on JRuby on Rails

Oracle Labs recently released Oracle Mix, a JRuby on Rails based social networking application. Rich Manalang now posted his experiences working on the project.  While the project is now running on JRuby, the initial work was done with MRI:

We built most of Mix using the standard Ruby interpreter (aka, MRI — Matz’ Ruby Interpreter) on Oracle DB XE using Ruby-OCI. Developing Rails apps is still much easier using MRI because jRuby takes a while to startup.

As it turns out, the Rails plugin system proved to be very useful in making Mix work with Oracle's internal systems - in this case the Single Sign On (SSO) policy:

One critical part of the project was to support Oracle’s SSO policy. Meaning every web app that we deploy has to support SSO using the Oracle SSO service. It took me a few days to figure out how all this worked but once we did, it wasn’t difficult to plug it into the acts_as_authenticated Rails plugin that we were using.

With these issues out of the way, the team encountered another problem when deploying to JRuby on Rails: Performance:

After I had the boxes primed, we started doing early deploys on jRuby. The performance was terrible. About 20-40 reqs/sec on a single app server. Turns out that I didn’t have some of the production settings configured properly (i.e., not caching ruby classes, etc.) Once I modified a few environment settings (standard rails prod settings), I got about 80 reqs/sec on a single app server… better but not enough.

Long story short, after a flurry of investigations into the performance problems, performance figures improved:

Ola and the jRuby team found some interesting bottlenecks in the jRuby code. Within in a day or two, Ola and team had them patched up and we were beginning to see around 150-200 reqs/sec. After the app server warmed up, things got real interesting… the numbers went way up (400-600 reqs/sec).

This is only the beginning, as the post states, as these numbers are taken on a system that doesn't do any caching. So what caused the increase in performance? The JRuby team investigated a list of problems, discovering old problems such as regular expression performance:

But the discovery I made was when I looked at the performance of the regular expressions used in Rails. There are exactly 50 of them for each request, so I did a script that checked the performance of each of them against MRI. And I found that there was one in particular that had really interesting performance when comparing MRI to JRuby. In fact, it was between 200 and a 1000 times slower. What's worse, the performance wasn't linear.

This problem actually occurred in the each_line implementation of JRuby and not in the Rails code. With performance problems as this, and some others, discovered and fixed, JRuby speed in general improved considerably..

Next to bottlenecks in the JRuby implementation, some other reasons can slow down a JRuby application. Nick Sieger points out a few tricks JRuby users can do to improve the speed of their applications. The most important steps to take seem to be:

• Switching off ObjectSpace.
  ObjectSpace is a feature that allows to iterate over all objects on the heap. Few libraries actually make use of this, so turning it off doesn't have consequences. JRuby 1.1, actually, turns this off by default. The increase in performance is considerable: because to provide the ObjectSpace functionality, every created object needs to be registered in a separate list structure, thus increasing costs for object creation.
• Ensure the "server" VM is used
  The "server" VM can be used with a simple command switch, and ensures more aggressive performance optimizations.
• Waiting for the JVM to warm up
  This is not really an optimization, but ensures that the right performance is measured. The way the JVM works, the bytecode is initially interpreted, and incrementally optimized and compiled to native code by a dynamic compiler. Only code that is actually used a lot is compiled and optimized, and this takes some time. This means, measuring performance, such as requests/second, on a freshly launched JVM will yield worse performance than on a JVM that has been running for some time.
• Switching from a stable JRuby (like JRuby 1.0.x) to the development version
  This works particularly for JRuby 1.1 (the current development version), since this is the first JRuby with a complete JIT. However, since the development version can change quite a lot, this also means that using the latest version can mean newly introduced, unreported bugs. These can break an application in unexpected ways. Of course a proper test suite can help with keeping the effects of this low. Whether the overhead of keeping up with JRuby development is worth the extra performance must be decided on a case to case basis.