Ruby on Rails case study: ChangingThePresent.org

Posted by Bruce Tate on Mar 26, 2007 10:28 AM

Community

Ruby

Topics

Ruby on Rails

Tags

Mongrel,

Casestudy,

LAMP,

MySQL,

Rake

A short five months ago, I was a busy man, flooded with more business than I could handle right after releasing two Ruby books in six months. Against this backdrop, I got one more call. The vendor, Arvato Systems, told me that I could:

• Lead a highly visible, important project
• Use the technology I love
• Work with people I could, at least in part, choose

Oh, and they also mentioned that I would change the world, in a substantive way. I've been oversold before, but this time, the pitch was true. Every word of it. In this article, I'd like to tell you about the technical details behind this extraordinary idea, but first, I need you to know the vision. ChangingThePresent.org has a simple but powerful premise. The site aims to capture just a fraction of the 250 billion dollars that we spend on gifts every year. Instead of giving another fruit cake or pair of fuzzy slippers, through ChangingThePresent, you can give a gift that has much more impact and meaning, like an hour of a cancer researcher's time or the preservation of an acre of rain forest. Building another donation portal may make it more convenient to give, and generate some extra money. But the opportunity to change the way that people think about giving got me excited.

An old problem

If you know the nonprofit world, full of technologies your grandmother wouldn't admire, you can already see where this vision is going. ChangingThePresent seeks to apply the same techniques that other industries have to use the Internet to bring people together. Only this time, the people are not buyers and sellers, but donors and nonprofits. With the heads of the Lance Armstrong foundation, Save the Children, First Book, and the Sierra Club heading up an impressive list of nearly 200 advisors and 380 nonprofits, the business development is well under way. But here, I want to tell you the technical side of the story.

In this article, I'll walk you through the way we're using Ruby on Rails to build the site. You'll see the core features we're using, as well as the primary plugins that we depend on every day. Most of our technology isn't really earth shattering, but I hope to give you a glimpse inside our day to day operations. My aim is to give you a broad overview of how the team works, the technology we trust in the production environment, the tools we use, and the Rails frameworks that are most important to us. I'll link to a resource rather than going into great detail in any single area, but if you want to know more about any part of it, leave a comment. I'll gladly write other articles to dive deeper, if there's enough interest.

The Basics

In the early stages, I joined the team as an independent contractor and lead programmer. The founder considered Java for scalability and stability, but the early bids came back and the project looked too expensive and complex. We settled on Ruby on Rails because it gave us the best combination of great productivity, a clean programming model, and reasonable performance. Still, the decision was not without concern. Based on existing portals and our business plan, we decided that we should be able to handle hits in the millions, daily. Though few Ruby sites have such high traffic requirements, we thought we could provide that kind of scalability through aggressive caching and simple shared-nothing clustering, though we couldn't find any evidence of such sites at that time.

September came and went. We were extremely productive, showing significant progress with each weekly demo, and the customer noticed, and I was sold on the vision, so in November, I joined WellGood, LLC as CTO. In December, we released the first beta version of the site. Our team built the initial version of the site at about one fifth of the projected cost of bids using other languages, and in about one sixth of the projected time. Rails has absolutely been critical to our productivity and success.

We're far from finished with the site--we've built roughly five percent of what we've planned--but we have a good enough foundation to begin to execute the next steps of our business plan. Though we've done some minor benchmarking and profiling to give us some assurance that we'll be able to handle the big loads when the time comes, we can't be sure that our current deployment will be able to handle the massive volumes we've projected for the long term, because few Rails sites ever have. But we've got high confidence that:

• We can build features, test them well, and deploy them quickly
• We will be able to scale to high volumes with minimal hardware
• We can add additional hardware transparently as our needs grow
• Our team of five developers can both develop and manage the site through the early growth curve
• We can solve any additional scalability and performance problems that come up

From the above list, the most important item for us is productivity. My team must be productive enough to satisfy our customers and investors, while staying ahead of our competition. Throughout the rest of this article, I'll dive into a little more detail on the details of our deployment. Where I can't provide a definitive description of what we're using, I will link you to a place to go for more information.

Topology

First, let's get the basics out of the way. Rails is fundamentally a LAMP architecture, We deploy our site on Sun hardware, behind BigIP load balancers. Like many newer Rails sites, we chose Mongrel as our application server, a lighttpd derivative for static content, and MySQL. Our database is deployed in a master/master/slave configuration for fail over, performance, and scalability.

We chose Text Drive to host our content because of their experience with high-volume Rails sites. Like most Internet sites, our traffic is dominated by read-only accesses, with the primary exceptions being several social networking personalization templates, and of course, the shopping experience. To get the performance characteristics we want, we've done some profiling, which showed us where to optimize our ActiveRecord models with eager associations to minimize the number of database requests. We also implement a caching strategy through the Acts As Cacheable plugin, backed by MemCache. Aside from the caching layer, our configuration is a pretty conventional Rails configuration.

When you're dealing with donations, you have to appeal to emotions. That usually means good pictures, and lots of them. Our nonprofits do a good job of picking good pictures that promote the causes that interest our visitors most. The pictures reinforce the theme of tangible donation opportunities, so our donors know their contributions will go directly to the people who need it most. But image downloads can stress the typical Rails setup. Since Mongrel does not deal with static data very well, we manage the images by URL rewriting, and dedicating a specific URL to static content delivery. We also are beginning to use Panther Express as an image accelerator. That service lets us serve cached images from around the world. We should have that service up some time in April.

Some people, mostly Java and .NET developers and vendors, have warned us that Ruby on Rails cannot give us the scalability we need, but I'm not overly concerned. I'm confident that the industry has amassed excellent experience with technologies that are similar to Rails. After all, the LAMP principles that form the foundation of Rails also form the backbone of many of the largest sites on the Web. To me, a far greater concern is the need to develop features fast enough to satisfy my investors, customers, and boss. These days, I spend much more time thinking about my team, tools, and implementation.

Process and Team

Rails says "be agile, be happy". We don't fight it. I love the agile approach. We firmly embrace these agile principles:

• We focus our process around short, sharp one-week iterations. We deploy at least weekly.
• Instead of forcing quality through the funnel of a formal test cycle, we rely heavily on our developers to build test cases as we code.
• We are not afraid to refactor when it suits our business purposes. We refactor to improve code quality, address requirements, or improve flexibility.
• We manage minor requirements weekly, and major requirements monthly.
• We maintain tight communication with our business users through weekly demos, even though our team is distributed.

We use Trac plus SVN to manage our source code, track basic requirements, and manage our basic release plan. The tight integration between Trac and SVN give me a convenient view to quickly understand what's happening with the source tree. We push slight changes directly into production after heavy automated testing and light manual testing, and use branches with a staging environment to work on major revisions. We prefer to operate off of the trunk, and will do so until our traffic and complexity prevent us from doing so. With the Rails platform, most revisions we need to make are relatively minor.

Our team is distributed across three time zones, and we're planning to spread the team out more broadly soon. We keep an open chat daily so we can quickly get questions answered. I would rather be able to go after the best Rails developers and communicate over long distances than limit my recruiting. So far, the tradeoff has paid off well. We do keep meetings to a minimum. My developers have one stand up meeting every morning, which we host on Skype chat. We also have a weekly technical meeting with the business people to keep our priorities in line, and plan weekly releases.

I've sometimes heard that Rails doesn't scale up to complex solutions very well. I disagree. Because it's so much more productive than alternative technologies, Rails has a dramatic impact on my development process, and subsequently way I can build and manage my team. The slower your technology, the longer you must go between iterations. The larger your team, the more you must spend on oversight and communications. After managing both Ruby and Java projects, I've now got even more tangible evidence to back that up. Though other organizations projected larger teams (from 10 to 15 developers), we've been able to keep our team small. At any given time, we have 5 Ruby developers, and up to two people working on pure HTML and design. We'll feel comfortable growing that team to seven. After that, we'll form a second development team to work on isolated elements of the code base that are more complex and expensive.

Tools

Our tool suite is extremely simple. We use the Rails testing stack, and throw in RCov for testing coverage. We use Selenium for integration testing. We drive all tests and coverage tools with Rake. We keep a target of 85% for test case coverage. We periodically fall below that target, but we've never gotten better than 90% or worse than 79% since we started seriously tracking coverage. We've had less success with Selenium because of the rapidly shifting user interface, but expect it to be more effective once our user interface settles down. We primarily use Rails unit and functional tests, but we're exploring some other testing alternatives as well.

Each of our developers can use whatever tool he wants to edit his code. So far, none of us use a full integrated development environment. Most of us use TextMate to edit source files. The Rails tools for benchmarking and debugging, combined with logging and testing, give us everything we need to debug the system. Some refactoring is a bit if a pain, especially when we have to change the database backed models, so we may well begin to use an IDE once we find one that is good enough. Still, from my experience, our maintenance cycle is easily three times as fast as a typical Java maintenance cycle, maybe more.

Core Architecture

Though we're part of a radical movement to change the way companies build major software projects, the call of the hour is conformity. Our core architecture adheres to the primary Rails architectures. We believe in migrations, through we see room for improvement. On the database end, we are DHH disciples. We don't try to force database constraints like referential integrity, and we don't use database views. And from a testing perspective, we rely primarily on the Rails framework, though we're starting to explore some alternatives as our test cases get larger. (I'm not ready to draw any conclusions yet, so I'll stick with our primary philosophy here.) But when you come right down to it, we're classic Rails MVC, with REST web services. I'll lay out the core enhancements below.

The Model

Like many social networking sites, we optimize for read-based performance first, but a significant portion of our content is relatively static, changing less than once per day. Based on watching the statistics for our competition, we expect high volumes, so we rely heavily on caching the stable content on the site, including nonprofits (such as UNICEF), gifts (such as making a blind person see or an hour of a cancer researcher's time), and causes (such as world peace and medical research.) Since these elements change infrequently, we can cache them aggressively. We use the Rails plugin ActsAsCacheable for that purpose. The back end for the cache is a distributed second level caching service called MemCache, but for all practical purposes, we use the ActiveRecord API directly, with few exceptions.

We also have some significant workflow features on the site. Nonprofits can submit content, and the admins at ChangingThePresent approve or edit each revision. We must be able to activate or deactivate any gift at any time, and we prefer to mark a record as deleted rather than remove it from the database. To support this customized work flow, we use a plugin called ActsAsStateMachine. That plugin lets us represent our state machine in a domain specific language, which you can read about at Crossing Borders: Rails Plugins.

I didn't realize how effective such a DSL could be until it was time to communicate with our business users. Ruth Ann Hacking manages our content acquisition and management. Though she has no programming experience, she is also a world cup fencer. Since she's perfectly capable of killing me with a sabre (or probably a ball-point pen), I try to keep her happy. On a whim, I sent her the code with the state machine in it. She was able to walk through it and find several critical bugs. Ruth Ann was happy, and I lived. But that's life with Rails. The unnecessary clutter stalks away and hides in the framework, leaving the fun stuff to me.

The User Interface

Our user interface uses classic Rails controllers, views, and layouts. We don't yet support the new Rails RESTful controllers, but new controllers will take advantage of them as our new web site features can use them. We do make limited use of AJAX in a few areas--our greeting card wizard, in-place editing--but in general, we prefer a cleaner, simpler interface. To make the site scale well, we do the following:

• We try to minimize images where possible. The site has many rounded corners, which we render with JavaScript libraries. We will probably revisit this strategy once Panther Express is in place.
• We do some limited caching of page content. We cache static pages, but not page fragments. There are very few static pages on the site, because we have some user customization on every page. We implement caching primarily at the ActiveRecord level, and image caching through PantherExperss.
• We initially showed our gifts in a paginated list, in alphabetical order. But we soon found that our users learned, and began all gift names with an A. Since our site can only use so many Aardvarks, we randomize the order of gifts, caching the results each day. Rather than do this work in the view, we just build a custom finder with the MySQL random function, seeding it with a formula with the current day, month, and year.

Going Forward

New features are going in quickly. Since early content was all customer facing, we've gone back to bolster our ability to manage content. We've also added a number of features since our launch in December. We've added a new console for our nonprofit outreach that allows them to review all changes that we make. We've also added community features to let people start their own donation drives. We've refined our home page, letting the gifts do more of the talking. And we'll be releasing blogs and improved wedding registries soon.

We'll continue to manage this project with small, geographically distributed teams. We'll continue to invest in Rails and LAMP. Now that we have a running, stable site, look for us to build features that will help us establish community and drive traffic. We do understand that building the technology is only one small part of the overall problem. Look for us to increase the number of nonprofits that actively participate in the site, and establish the ChangingThePresent brand.

Helping Out

The Ruby community has been fantastic, supporting us at every turn. We pass all donations through to the target nonprofit, taking only the credit card processing fee. But that means we're dependent on people like you for support. If you want to help, here are a number of things you can do:

• Log onto the site, and click on the link that says "Support Changing The Present."
• Put a banner on your blog supporting the site, your favorite cause, or a drive.
• Create a profile and tell five people that you know about it. Tell each of them to spread the word.
• If you want to code, we have a few opportunities for volunteers for designers, flash developers, and Ruby coders. Feel free to contact me through InfoQ.

It's not every day that you get a job that lets you do what you love, with people you enjoy, and know that you're making a difference in substantive ways. I hope you've enjoyed reading about this site as much as we've enjoyed building it.

About the Author

Bruce Tate is a kayaker, mountain biker, and father of two from Austin, Texas. He's the CTO of WellGood, LLC, the social entrepreneurs responsible for planning, running, and building ChangingThePresent.org. Bruce is the author of nine books, including From Java to Ruby, Rails Up and Running, and Beyond Java.

1. Back to top

   Similar Ruby on Rails project is going on in Canada

   Mar 27, 2007 12:51 AM by Tadatoshi Takahashi

   Christmas Future Donor Trust (http://donortrust.rubyforge.org/, http://www.christmasfuture.org/) is a Ruby on Rails application for relieving extreme poverty in the developing world by allowing users to use donation as a Christmas gift. It's on RubyForge (http://rubyforge.org/projects/donortrust/). It is gaining momentum these days. I myself have contributed to its development. Currently they are refining its RESTful architecture.

   Reply

2. Back to top

   A great business partner

   Mar 27, 2007 9:07 AM by Bruce Tate

   I did not mention, but should have, the involvement of EdgeCase. Chad Humphries is a great Rails developer, and has been absolutely instrumental in getting us pointed in the right direction and keeping us there. Chad helped us outline the caching strategy, our testing setup (including all of our experimental stuff), and much more.

   Reply

3. Back to top

   Re: Similar Ruby on Rails project is going on in Canada

   Mar 27, 2007 9:11 AM by Bruce Tate

   It sounds like a good idea. Good luck with it. Our primary strength from a business perspective is our strength of relationships, including our nonprofits and advisors. We're attacking poverty too, through donation giving. We hope that the gift model, combined with our customized greeting cards, set us apart from most other projects out there. In the end, every project like this will help to change the world in a structural way.

   Reply

4. Back to top

   More detail?

   Mar 29, 2007 7:11 AM by Bruce Tate

   Are there any areas where readers would like to see more details?

   Reply

5. Back to top

   Re: More detail?

   Mar 29, 2007 2:54 PM by Sammy Larbi

   Bruce, I thought it was a useful case study, and hit on all the points at the right level of detail. It gives a good starting point for people to search on some of the items if and when they need that level of detail to implement their own high-volume Rails site. With that said, if you wanted to catalog any links you found useful, or detail your setup yourself, I'm sure that would be much appreciated as well. I'm thinking of things like details on how you implemented the caching, the MySQL random sorting of query results, and your use of the plugins you mentioned. Of course, I realize those things are out of the scope of the article, but you asked =)...

   Reply

6. Back to top

   Re: More detail?

   Mar 30, 2007 5:18 PM by Matt Dragon

   I think at least another level down into the details would be beneficial to anyone evaluating RoR as a solution for such a project, as well as those looking to get started on RoR projects and making sure they're laying the right ground work. I'd be most interested in your experiences working with plugins vs. native Rails functionality, and the build vs. borrow decisions you were faced with

   Reply

7. Back to top

   Ruby vs JRuby

   Apr 3, 2007 1:46 PM by Rintcius Blok

   Good article. Just curious, are you considering or have you considered JRuby? It would be interesting to know how these compare to each other when used on this scale. Eg regarding performance, problems with libraries, platform etc.

   Reply

8. Back to top

   Re: More detail?

   Apr 11, 2007 5:49 PM by Bartek Urbanski

   Hi Bruce, I'd like to know more about decision-making process when it came to technology. I know you're a huge RoR enthusiast, but what were the major ChangingThePresent founders concerns and what where the case studies that you have based while convincing decision makers to Rails.

   Reply

9. Back to top

   Re: More detail?

   Oct 21, 2007 4:29 AM by Oleksandr Bondar

   Thanks for good article, I would like to know more about caching dynamic images that are stored in database. Working on current project we store images in postgresql database and it take time to load them. Also all images are reloading all time I open the page, it looks like browser do not cache them. Thank you.

   Reply

• Older >