So this is the first Rails conference?
Yes it's the first and it has been interesting to see the number of announcements that have been popping up. I think we're going to have a big number of Rail Conferences throughout the year. It's going to be the Rails Conf in Chicago in June, Rails Conf is also coming to London in September, and then there's going to be Italy On Rails in October and then RubyConf at the end of October, and I've heard rumblings about Germany on Rails too. So there's a lot of interest from Railers to meet up.
What do you get out of attending these conferences yourself?
It's just fun to meet people who use something you've made and to see some of the stuff that other people are developing on top of it. So, even though I do keep a fair eye on things myself, I thought it was great, for example, to get that small demo of RadRails. I haven't really dug into it, but it's just great to hear from the people in person about the stuff they are working on.
A little while ago people from Canada were asked to raise their hands and most of the room raised their hands. There are a lot of people here from a fairly limited area. To me it seems that Rails is moving towards mainstream acceptance. What does that mean for us old old-time Rails developers and what does that mean for someone who's just getting into Rails? Where do you see us going?
I think that is mostly a positive thing, definitely. I think that having a critical mass, having a large number of people who depend on Rails is a big part of ensuring that Rails is going to be around in two or five years from now.
And the bad parts about mainstream that I usually perceive is people trying to conform themselves to be mainstream, trying to change whatever they have. Trying to dumb it down and make it easier for people who are not really interested in what they are doing to pick it up. I don't see that's what we are doing.
We are creating some aspects, making Rails more hardcore; a lot of the new features we are adding are in some respects "hardcore" features from just grasping more concepts. We're introducing more (hardcore) concepts into the world of Rails, which is not necessarily the best thing you could do if you want to make it as mainstream as possible.
If you want to make it as mainstream as possible you would take many of the concepts, dumb them down, try to shield people from shooting themselves in the foot and all that stuff, that in my mind led to Java. Having that consciousness that you are deciding for somebody else, I think that's perhaps my worst fear of mainstream.
If I wanted to go mainstream, (I would act) like I was designing Rails for somebody else. And I'm not. I am designing Rails for me. And I think that that's a big part of what's going to keep us from the hell that can be trying to chase the mainstream, trying to fantasize what some lesser developer might need that's different from me. I'm never going to be a good person for doing that. I'm fairly decent at solving my own problems then sharing the solutions with the world.
Some would take that statement that you'd basically look to solve your problems, as a statement that "If I'm working on 37signal style apps, meaning Web 2.0, heavy Ajax, then I should use Rails, but otherwise I shouldn't"?
I don't think that's true, because when I'm solving my problems I'm solving them at an infrastructural level. I'm actually trying very hard to make sure that Rails is not optimized for any domain specific thing. It is not optimized for content management. It is not optimized for any Web 2.0 social tagging application. It is not optimized for any of these things, because I don't think that's a good fit for a framework.
I think that frameworks are all about infrastructure, all about what most people need the same way most of the time. So it's about abstracting things like, how do you handle filters, how do you handle proper use of HTTP, how do you handle all of these common concerns that everybody has to deal with, in a way that doesn't hurt.
I think that is universally applicable, whether you're working on a vertical app for insurance businesses, calculating some rates somewhere (and you need to have an interface for that), or if you're working on that social/mapping/mashup app. You have the same needs to use the core infrastructure; you have the same needs to use the relational database an object oriented system, have all these common concerns in a smooth way. So I don't see a lot of specialization in that area. It's definitely true that if you belong to the kind of people who are building apps like 37Signals do, what you usually do share are more cultural leanings than so much technical leanings.
Is Ruby on Rails the Mac of the development world?
I definitely like to call it that because we care about the whole experience. We care about putting a complete stack together. We care about sorting out all of those problems that usually fall between the cracks.
I think that the Apple and especially the IPod example are such perfect examples on how you can introduce something like a dumb USB stick, like the Shuffle and make it a huge success because it's part of the complete stack. It would be a complete failure if it weren't part of the Apple environment.
I think we can do the same things with a lot of tiny optimizations that seem inconsequential, but when they are seen across the whole stack, just like the stuff we are doing with Ajax now, it matters because we can control the whole level up. It's not just about shipping a JavaScript Library, it's about shipping a JavaScript Library that has hooks into the template language, that has hooks into how the controller operates, that has some general notions of how you are actually dealing with your model.
By creating each of the individual parts, knowing how the rest of the world looks you can usually do something that is a lot more targeted and seemingly coherent, than if you're just working with your shutters on, on one single thing.
You are brunt end of what I assume is a torrent of support requests, enhancement requests, praise, as well as complaints. What is it like to deal with that? What is your impression of the community?
I think the community is in great shape when you look at the number of different people who contribute to the Rails source, who were actually accepted. We get tons of patches all the time, from people all over the world who haven't necessarily supplied a patch before, but they supply a patch on something they feel pain with. We look at it and we accept it.
At the same time we also do have a fair number of requests to do this and that, and as long as they realize that we are not their vendor I think we're going to be fine. I'm not in this world to create Rails for you. I'm in this world to create Rails for me and if you happen to like that version of Rails that I'm creating for me, than you are going to have a great time.
How long until you are the vendor?
I don't think that is ever going to happen, simply for the fact that, at least currently, I have no direct commercial interest in it. I don't really care whether you use Rails or not. I think it's great that we have critical mass, and I care about that; but I don't care about you as a single individual customer.
So at what point do the commercial appeals, at what point does your capitalistic urge kick in and say: "Okay. There are 2 million people worldwide using Rails, I can do Rails Inc. I can leverage the trademark!" When does that happen?
I've already had a number of offers from various VCs and other parties that wanted to have exactly that happen (RailsInc) and shoot money into that. I'm not interested.
I get my capitalistic kick out of actually using Rails to produce something, so I think that's a great part of what's going to insure that Rails is going to be just fine, that I don't need a capitalistic kick on Rails itself directly, because I get that somewhere else. Rails is not my day-job, and I think that's a huge difference and distinction us and how a lot of other frameworks have been led astray.
Let's say that you don't indulge that capitalistic kick, someone could theoretically come along and fork Rails, right?
Sure! If they could actually do a better job than we are doing as a community right now, under an open-source license, to drive Rails, all the more power to them. I think that's highly, if not extremely, unlikely to happen.
What are the factors that make you say that it is unlikely that someone would come along and do a better job with Rails?
Look, what is the interest that Rails is generating? It was and it is mostly generating interest out of the work I did as one person over 4 or 5 months. So what are you actually going to offer up? Are you going to get smarter people? That's definitely possible, that you could find someone that's smarter than the core group, but what we are missing right now is not really resources.
If we had 50 programmers at our disposal, we would not really be making anything that much different! We would not be interested in having 50 programmers at our disposal. What difference choices would we make if we had 10 million dollars to blow on this? I don't know! I can't come up with a single good answer that would say "I'd love for Rails to have 10 million dollars to blow on stuff". We would probably just have tons of parties and distract ourselves blowing that money.
How do you feel about bug fixes and enhancements that are not fixed because no one feels enough pain, but actually we come across them, hit them and do a work around, because it's not worth enough time for us to go in and fix it. Do you take a pragmatic approach?
If they are not worth enough of your time to fix them, they are probably not worth enough for you to pay big bucks to have them fixed somewhere else, and especially not by some fork. It basically answers itself: If it didn't have enough value for you to investigate it, it's pretty unlikely that it has enough value to justify the creation of some commercial fork. You're not going to pay the amount of dollars that needs to be paid to some commercial entity to have something you don't really care about getting fixed.
What do you say to development managers, directors, and technical architects who are concerned about whether Rails is going to work for their particular needs?
I say "Try it out" It's free, it's fast, and you can pretty quickly get a feel for whether Rails is a good fit for the type of work you're doing.
I think in general you can just look at the cultural leanings of your product or project, if you already have people who subscribe to Agile or if they are already interested in working in an iterative fashion, and have people who are not mortally invested in static typing, then I think that you can do just fine with Rails.
On the other hand, if you have people who think of themselves as Java developers, then they are not developers in general. They are java developers, that's what they do.
Or if you have very long development cycles using waterfall-ish approaches, then Rails is probably not a very god fit for you. But the fact is that is so easy and fast to find out for yourself, that there's no reason for you to outsource that decision. Put some of your guys on a vertical slice, a "Spike" if you will, for a week. Anybody anywhere can take a week out of their time, or even just a few days.
What about the typical third of the IT community who are in it for just a job, they are not really interested in technical excellence? Let's talk about that mainstream. Let's talk about what happens if they feel compelled to start adopting Rails. Let's talk a little bit about what happens when their bosses watch this interview and say "Okay, I'm going to have my guys to do a vertical Spike", and they tell all these guys "Have at it! Do some Rails!" What happens then?
Hopefully bliss ensues and they are happy being more productive, because Rails was a good fit for their case. Or (!) Rails was not a good fit for their case and they'll go back to the dungeon.
Really, what are the risks? What are the risks of trying something out that doesn't require you to pay big bucks to get IDs, application servers, consultants or expensive pre-sales people into your organization? If something it's so easy to pick up that you can actually, as an individual, get value out of just playing with it for a few hours a day, or a week, then it totally changes the game.
This is not about comparing one huge enterprise stack to the next. And I think that's part of the allure of what we are offering. It's not just the same thing called open-source or just the same thing, but called Rails. We're trying to change the game on a fair number of issues.
Thanks to the Rails hype I think there are a lot of folks out there thinking mistakenly that we advocate Rails as a silver bullet! We don't advocate Rails as a silver bullet! Where wouldn't you use Rails? What applications are unsuitable for Rails?
First of all, I would actually say that the whole notion of Fred Brooks' "silver bullet" is overstated and a lot of people are using it to say that "I'm never going to be interested in new technology again because no technology is ever going to be a silver bullet, and only silver bullets interest me". That's a very common, easy and convenient way to sweep anything new out of the way by saying it's not a silver bullet.
I actually believe that, yes Rails could be a silver bullet for certain type of niches, compared against certain very onerous and laborous stacks. You can definitely find types of applications where you could do them an order of magnitude faster in Rails versus some aspect of another stack. But you can do that with most things so, of course, it is not true that any project anywhere, would be an order of magnitude faster; it is stupid to think that. And in any case Rails, in most stages is going to help you on the technical side, so if you have a corporate structure that does is such that only 5% of the cost of your product or project is actually doing development, well of course it is never going to be an order of magnitude faster. Rails can only help you on the development side of things.
Hopefully we can also influence thinking of complexity and how you should approach things, but that's like a second-order effect.
How does Rails represent getting real?
I think Rails represents getting real in the sense that we are calling bullshit on a lot of technical complexity and saying that it's simply not needed to think this hard about something this simple. There's a ton of problems out there where you're just killing it...
WS-* is that bullshit?
Yeah. I think there is a lot of bullshit around that and I think there is a lot of bullshit from analysts and consultants selling it, because it's a very convenient sale, as this is going to be...
I don't know if you remember Robocop. Robocop has this great scene where the archevil guy is up there talking about how the 8200. Who cares if it works! We have solid support contract with the military for 25 years to deliver spare parts. I think that's a lot of what is going on. People don't necessarily want things to be simple, because if you simplify things too much you might not need those analysts too much, to tell you which way to go and which horn to blow. And I think that the same thing is with the consultants. I think that there's just an industry built up around complexity. That complexity is putting food on the table and they don't want necessarily to rock that boat that too much.
So what I am saying is not that there's never any case where either J2EE or WS-* is appropriate, I'm just saying it's being over-sold, I'm just saying that most projects out there do not have the complexity in them to require these types of solutions. People are selling solutions for things that could be solved very much more simply, because of their own motivations or inclinations.
SOA and Rails, we had a presentation on that. Where does Rails fit into the question of SOA?
I think it is actually funny, because Rails is uniquely well-positioned to play a part in "SOA", simply because the whole notion of being service-oriented is something that makes it so much easier to say "I don't care what your implementation is. If it works, if it exposes the interface that's needed, I'm happy".
So on one level I appreciate it, because it's actually legitimizing Rails from the side of you can do whatever you want on the implementation side. On the other hand SOA is like this huge pedestal of bullshit and hype.
SOA is predicated on the existence of standards so you can have all these pieces inter-operating, right?
I don't think so. My mental picture of SOA, in the good sense, is just about wiring applications together and you can do that without the WS-*, the "death star".
Does 37 Signals have an SOA? Describe it.
Yes we definitely do.
So for example we have five applications. A good number of them actually interoperate. So Writeboards (Writeboard is a simple document sharing tool): we implemented Writeboard, but we put Writeboard into Basecamp, and into Backpack using web-services interface. So we treated Writeboard as a service from Basecamp and from Backpack.
How did we do that? By implementing WS-*? No. We have just used XML or HHTP and we are happy. That's exactly the kind of solution that a lot of problems could be totally fine using, they don't need these towers of complexity, and we already have the standards to solve a lot of these problems. HTTP and XML are great starting points. You don't need to go down that road of WS-* complexity for most problems.
You have five applications at 37signals that require sign-ups? That's a perfect scenario where someone would say "well of course they must have some sort of shared sign-up component". Do you have a shared signed-up component?
No, and I don't believe in components in that way. Component for me is just fancy-smancy copy and paste. So if I'm just going to reuse the same functionality over and over again, especially something like sign-up, I should be factoring that out as a service and that's actually how I subscribe to the SOA approach.
I think SOA is very different from components and in a lot of ways it's much better than the component approach, because you simply say "I'm going to have to service once, instead of duplicating component over component".
In the case of our applications, the reason we don't have single sign-up is that the sign-up doesn't match. Writeboards for example are app-less, there is no account. You just sing-up for a single writeboard and there's no sharing of that. Basecamp has namespaced sign-ups. Your sign-up is particular to a single company, so you can't reuse that.
Backpack on the other hand has a global sign-up. You can't be named the same as somebody else's Backpack. So here we have three different apps and we have three different needs in how we do sign-ups. So you can't really say "this is the same, because this is about sign-up" and that's the charge I have against component delivery in general.
People just gloss over it and say "Oh this is roughly the same and we can make a component out of it." No we can't! We can only make a component that actually works, if things are, if not completely identical, then as closely as possible they could be.
Not too long ago you guys introduced the Backpack API. That's going to allow outside developers to tie in their own applications into Basecamp. How is that going?
We have two APIs. We have Backpack and we have Basecamp. Backpack is our application mostly for individuals doing their own management of their lives. There has been reasonable uptake on that; people have made various applications like you can take your Backpack offline and send your data back and forth. It hasn't had that much uptake.
Basecamp on the other hand, has already had a huge uptake. Its API has really been used tons since it has been released three weeks ago. Because Basecamp is something that a lot of companies actually use "to run their company", that's the main help for their information sharing, that's the main help for task management and people are hooking into that in a big way.
Can you share some numbers in terms of how many people use Basecamp?
We have more than 500,000 people accessing our 5 different applications. When I say people, I have in mind that a single Basecamp account can have multiple people associated, so it doesn't map exactly to paying account. We are not sharing numbers on how many paying account we have; we aren't sharing financials of any kind, but we are doing quite well. We've been profitable since day one and we are definitely not complaining or looking for venture capital money.
Does Rails scale? Are people still asking if Rails has scalability issues. What do you say to them?
There are some, but I think the scale has tipped on that. I think it's no longer the same kind of legitimate question it was thought to be two years ago.
Why has the legitimacy of the question of Rails scaling tipped in your opinion? Why do you think there are less people still asking it?
a) Simply because there is a large number of applications out there that do scale on Rails, that are doing millions of requests per day and going along just fine; and b) because a lot of people who were asking, actually looked into Rails and figured out that's not a very interesting question to ask, because the approach of scaling using Rails is the same approach a ton of other stacks are using: the whole shared-nothing approach.
I think people are just realizing that it's not such an interesting argument anymore, that there are plenty of examples of shared-nothing architectures that scale very well and often they scale much easier than comparable stacks, like Java.
I think that that approach, the fact that we have a lot of cases out there now, people scaling millions of requests, and not only in the social tagging, but Rails application doing mortgage processing; insurance, and all of these so-called "enterprise environments" are proving that it scales simply by using it.
Let's dig a little deeper in terms of scaling, let's talk about using Rails in situations where, typically, Java would be called for or some other harder technology [...] Say it's going to do some sort of credit application processing and I have an SLA - a service level agreement - that specifies subsecond response (times) and I have legitimate concerns that Ruby is not going to cut it for that kind of service level agreement. Do you feel qualified to talk about that, do you feel that you need to defer, that they shouldn't use Rails? Where do you stand on that?
The hardest part of computer science to ever make general statements about, is performance. I think that there's nobody who can really answer your performance questions, except for you, in your particular situation, under your particular constraint, on that exact setup that you're going to use. Tons of people will claim that they can answer your scaling or performance needs and they will try to sell you a software to do that, and I think they are lying.
You've just run into a case where Ruby did not scale in that sense for your performance need haven't you?
Yes. We were doing campfire, we chose to go the poll route, where every single client polls every 3 seconds. You can add that up and see if you have a couple of thousand clients going, that's a lot of requests per second.
Can you tell us a little bit on how you went through the cycle of figuring out?
Sure. We knew from day one that if we were going to go with a poll approach and poll every 3 seconds, then we were going to have a performance issue and we knew that we were going to take steps for that. But we also knew it's a solvable problem, so when we started on day 1, not real symptoms we were worried about.
We built up the entire campfire system without really worrying about scalability. Then the day came when we were actually done with the system, when we had the system that we wanted to launch, and then we needed to care about scalability. So we started to do stress testing, we started running performance benchmarks and we realized that the initial setup we had was just not fast enough.
Step one was adding caching in Rails. We were generating a lot of JavaScript dynamically, so we started caching that JavaScript, shoving it in into a database and then serving it out. That made a huge jump and we actually launched on that. We launched on a complete Ruby system, just using a database cache and I think that's usually the appropriate thing to get you out of a lot of hard situations. If you can cache, your execution can be as slow as you want, as long as you can do it with cache and cache hits are high enough, you're totally fine.
But in our case we (eventually) did feel the need to make something faster, so we looked at the Campfire application. Where was the bottleneck? Where were we having issues or issues that were going too slow? We found that bottleneck and that bottleneck was a hundred lines of Ruby code. The responder to that poll "Did I get a new message, Did I get a new message" The guy who responded to that question was a bottleneck, a hundred lines of Ruby code.
Was it a collection of controller methods? Was it a vertical slice?
We had already had that as a separate FCGI, so the poll sequence was a separate FCGI but done entirely in Ruby, and we just realized, those hundred lines in Ruby code, that's the bottleneck! Ruby is not the fastest language in the world and sometimes you do hit a bottleneck. It is very rare. For none of the other applications we've done, have we ever hit the bottleneck where Ruby was not fast enough; and we do lots and lots of requests in those applications, but Campfire was special. We could easily do millions of requests per day, just because we have all these people asking "Did I get a new message?"
So we chose to rewrite those hundred lines of Ruby code, to see what it would take to write it in the fastest language we can rewrite it in; we didn't rewrite it in assember, but we rewrote it in C, a fantastic language for rewriting performance bottlenecks. So we rewrote a hundred lines of Ruby code into three hundred lines of C, in about 3 hours.
The great part was that we had already done all the locking, we already had a completely working system, we had already specified all of the behavior in Ruby, so we were not exploring anything in C, we were just translating lines pretty much, asking "How do we make this Ruby line into C and make it reasonably fast".
We still have the Ruby version so when you look at the Ruby version and you look at the C version, you see that they roughly do the same thing. Ruby does it a lot more eloquently, in a lot less code, and with a lot less mental overhead. The C version does roughly the same, it's just a lot faster. It's hitting the database: two requests per question, one request for permissions and one request to see if you got something new. But since we cache, those requests are stupidly cheap to make.
So even though we could have said: "Tthis is not even going to scale, we've got the poller in C, but we're still hitting the database so it's still too expensive" then the next option was right there for us. If the database were not fast enough, memcache would be the next obvious step; and that's going to scale definitely too. So we just haven't moved down the path yet, but it's there...
An early Java One had a panel about "Real-world adoption of J2EE containers". One of the panelists spoke about achieving massive scalability and performance. It was some telco and they said "we have to support this huge throughput". It was like, exaggerating a little for emphasis, like a billion request a second. It was for cellphone billing, running on Java and the whole crowd went "oooohhhh" when they heard that. Well I was sitting there wondering what vendor stack they were running on, and it turned out they had written their own J2EE container! I remembered that as you were describing how to fix the performance issue, because you guys [37signals] are the best. Of everyone doing Rails, you can do that particular performance optimization, just like the guys that wrote their own J2EE container, since obviously they were the best at that time. What do you say to people with legitimate concerns about Ruby performance, who want to push forward, but have concerns about the resources [talent] available?
First look at the specifics. What did we actually rewrite? Did we rewrite entire Rails app? Did we rewrite the entire Rails in C? No we didn't! We rewrote a hundred lines of Ruby code in three hours! Even if we didn't have anybody on staff that could do C, we could have hired a C programmer to do that in a day or two days! Or even if it took him a week to write those three hundred lines of C code, it would have been totally fine!
The fact is that most applications, most of the time, are just not performance critical. They have tiny performance bottlenecks, sometimes. Most of the applications do not even have that type of bottlenecks that can't be solved through caching and or by just being a little bit more clever in your Ruby code. It's very rare that you actually have to drop down to C, and when you do it, who cares! 99 percent of the app was in Ruby, so we got all the productivity benefits of that. By the time we actually hit that performance bottleneck, we had so much time left over versus implementing the whole thing in C, Java or something else, that it was no problem at all to solve that thing.
I think that you also need to be specific; if you have specific performance concerns then map them out. What are the numbers? What numbers do you need? Where is that performance bottleneck? It's most likely in a small part of the system, then make do a spike for that small part of the system. Can you do it in Rails, can you do it fast enough? If not, what would it take you to make that small part in C and just do the rest of the app in Rails. The fact is that most people who do very big numbers do all custom anyway. If you ask Google, Amazon, Yahoo, if you ask those huge sites that are doing amazing numbers... none of them are using off-the-shelf software, because off-the-shelf software it's just not fast enough for their bottlenecks.
Let's talk about that for a second. We said earlier in conversation would you do Amazon? would you do EBay? They seem like silly questions, but would you or are you are just saying that those sites are all custom?
I would definitely do a big part of any application in Rails. If you look at Amazon , they are actually doing a huge part of their application tying the front-end together with Perl and Mason, not exactly super fast technologies. Perl might be a little bit faster than Ruby and other things, and slower than other things, but it is in the same area of fast enough and they are using that
But their main CGI engine is C right?
Yes. They definitely have bottlenecks and some things, they just need to go custom. There's no framework in the world that's going to solve those problems, for Yahoo, Google or Amazon. Yes, all those need to be custom. What I wanted to communicate is that apps are not like that. Most people do not work on the performing bottleneck parts of Google, Amazon or Yahoo. Most people do not have to deal with that kind of scale most of the time; most of their apps are simply not under that kind of stress, most people would like to think that they have those special custom needs but they don't.
That is exactly why they are choosing to use frameworks. They are choosing to use pre-packaged software, because they don't have those kinds of needs and it's worth trading better performance to get productivity back. It's worth saying "I'm fine giving up some CPU cycles"
6 comments
Watch Thread Reply