James Adam on Rails Engines and Ruby in the Enterprise

1. We are here with James Adam and he just gave a talk about his claim to fame. Can you introduce yourself and let us know what you are working on?

Sure. My name is James Adam, I am a professional Rails developer working in London, in a small team, in a large media buying agency. I've been doing that for a year full-time, and most people know me because I'm the developer behind Rails Engines. That's something we use professionally and decided to make open source.

2. What are Rails Engines?

Essentially Rails Engines are Rails plug-ins, with a little bit more. For the kind of business that we have, we love the plug-in system but sometimes we need to share code that the plug-in systems from the Rails core isn't designed to share. We've created the Engines plug-in, just to add that tiny layer of extra stuff that we need. They are drop-in elements of code but instead of being just ActiveRecord extensions, the model part of MVC, we can include the full stack for a particular aspect. Login is a classic example.

3. What problems in Rails do Rails engines solve for you?

We found that in the industry that we are in we are solving the same problems over and over again. And this is not just from the Ruby code point of view but from the views of JavaScript, all kind of bundle that needs to be in place to solve a particular problem for media clients. That can be common stuff that crosses all web applications. A good example would be reporting: we need to perform the same kinds of SQL reporting on data for our clients, for every single business that we operate with. So rather than rewrite SQL code, every time that we want to solve that problem, we realized that it would be much more efficient if we could somehow abstract not only the SQL reporting side, but also the presentation of the reporting using JavaScript to build a nice interface to the reporting system, in a way that the media companies will find that they can interact with it intuitively.

4. DHH has gone on the record to say re-use is overrated and that he hates components. Are you causing a schism in the Rails community?

I hope not, that's certainly not the intention. As I said in my talk, I understand why David has said that reuse is overrated, why you try to bring to that to balance is that reuse can be overrated, or it can be essential. What it depends on is the context. And the only person who can determine whether or not some part of the application is right for reuse is you, the developer, because only you can understand the industry that you are working in, and only you know the problems you are going to be asked to solve over and over again, and how they might relate to software in the future with the client. While 37signals develops these small, targeted applications, as their target is very focused and they do one thing, which is great, there is this other side to development, enterprise development, where you are being asked to do the same general thing again and again.

5. What are some other examples? You mentioned reporting

The reporting system is very broad. We have a charting mechanism, loosely based on Geoffrey Grosenbach's Gruff graphing framework, which includes models for storing chart information, views for presenting the chart and also other helpers to render charts within a Rails view. We will be writing systems like auditing systems to track changes to data that users have made. We needed a distributed data store to send information around our network because we often have to process things like Microsoft Excel files; but we wanted to deploy to Linux because it's more reliable so we had to bridge this file system gap, so we have this distributed data store that we've written and shared between applications engines so we can monitor the store with views and controllers to view the status of files.

6. Are these hosted applications that you are talking about? Are you talking about developing multiple individual small hosted applications or are these hosted at your facility and you are re-using this functionality within one system?

We host the applications but we aren't building one grand application, one media industry application. They are individual applications.

7. Wouldn't that be the way to go to maximize the efforts and basically figure out the best way to do all the aspects of your business for all the clients? Why did you choose a different approach?

It could be, but you need to find the right level of granularity. At the moment I don't think anyone in our team understands enough about media industry and be able to say "this is how every media is going to act". To some extent you can't say "advertising works in this way in general across TV, across radio, print and the Internet", but the issue with generalizing is that you need to be sure; you learn what's good to generalize from experience. We start at the low level and as we grow more confident that this is the real solution to the problem then we can go from plug-ins up to engines that is a larger package, but we have to be sure that it is suitable. If we decided to just build one big media application, then there is no way that we could know it was going to be suitable for everything without having some omniscient knowledge of the industry. It's choosing the right level of granularity and for us is slightly higher than David's. But that depends on the context.

8. Let's get into the nitty-gritty for a second. Do you use Subversion?

Absolutely

9. Do you have extra repositories for your Engines or do you basically put everything in one big repository?

At the moment we have separate repositories for every project and that includes the Engines. The Engines we use are all in one repository but every application has its own repository, and we pull in the Engines from that central engine one.

10. If you are going with an engine approach to be able to do what you are saying in terms of using the same functionality, for many different clients, how do you handle the versioning and how do you handle upgrades of the engines and rolling those into your live applications that use them?

One thing that helps us do that is the fact that we are a small team, so we can always speak to each other. We try all to keep on top of the entire situation even though we all might be developing different applications. But typically what happens is that we talk about: "I've noticed there is a problem with our user authorization system" or "I didn't like this, I want to change it", and we discuss this as a team and someone will go away and implement the changes in the engine itself. And then as the opportunity arises for whoever is responsible for a particular project for a particular client, we roll out the changes in a controlled way. It's not something that we push out to everything, it's still on a client by client basis that we evaluate whether we have to do it now, or we should wait until the next release. But having the option is really powerful.

11. Do you handle responsibilities for fixes and upgrades to the engines with one developer owning an engine or is it shared team ownership of the engines?

In theory it's shared. Typically whoever originated it or whoever was working on the project, when we realized we're going to need this feature and we want to do it as an engine, will be responsible for it. But we try as much as possible to share the knowledge of how these subsystems work. Obviously I'm spending some time away from work, so some of the other developers can handle any problems with the stuff that I may have originally produced.

12. How did you go about determining the parts of these applications that you were going to need to abstract out into their own reusable engines?

Primarily our technical architect has been working in media for 25 years. Originally he had an outline of what he felt would be the most appropriate components that we would need. In all of our interviews we all had a sketch on a napkin that said these are the subsystems that he felt were appropriate. But then as we develop these applications and we deal with clients then other things might come along and say "well, I thought I was going to do a general charting component but this particular problem is actually specific to this client". We won't do a general solution yet because we don't think there's a need at the moment. It's quite organic although there is a grand plan at the same time.

13. Let's say you encounter a situation where something that you are using an engine for to provide a narrow size functionality in an application. It turns out that you actually need to do this in a more client specific way, or a more specialized way, do you have to stop using the engine at that point and roll it into the application functionality? Or do you extend the engine in some way?

It depends on the particular situation. But every option is available to you.

14. How do you go about determining where the boundary is?

I don't think there's a rule, this is a very organic thing and it is something that only you can decide. There aren't any clear boundaries were you can say "If I need to change 25% of this code then I will no longer use the generic".

15. Let's take an example: you said graphing of some sort is a reusable component.

Yes, originally it was going to be developed as a re-usable component. But we realized that for the specific problem that we needed to solve for the business at that particular time it wasn't necessary that we made it generic. So that was ruled into the application properly. It's still a library but we are not sharing it yet. But if we do that we need to address graphic again or charting, then we will look at what we've done and see how much of this solution can we apply again to this new problem. And if there's a good overlap then we'll extract and as opportunity arises we'll go back to the original application and see if we can replace the custom component that we had with the generic one and perhaps a few tweaks.

16. This sounds like a pretty serious enterprise development that you are doing, even though it's a small team. Can you share a little bit of insight as to how your team decided to go with Rails as their technical solution?

It's because it's written in Ruby and that's the fundamental reason. Our technical architect and certainly I have been using Ruby for a long time and we love the language. As soon as we realized that we could develop these tools using Ruby, through Ruby on Rails, then it seamed like there was no other choice. It has to make you happy when you develop. We could probably assemble these systems in Java.

17. And you have actually put Rails system into production?

Rails systems? Absolutely, yes!

18. What kind of reactions do you encounter from other members of your community? Local London Java people, other folks that you interact with that find out that you are doing real enterprise development in Rails?

I think the Java people are still curious about Ruby and there are always the Java people who are close to switching as well and they are really keen. And I hope that we started demonstrating that you don't need to be using Java to solve these "enterprise" problems. In the user trips that we go through the Java people are very curious and very enthusiastic and we don't encounter very much dismissal or skepticism at all.

19. You mentioned that you were a Java developer for some years.

Yes, Java used to be my primary language.

20. And you stayed away from the web?

When I was leaving University finishing my undergraduate degree was when Java started to pick up, and it was great for writing servers and I remember looking at that code even as I was choosing a project to do for the last year for the final exam. I couldn't see how you could enjoy programming for the web using Java because it seamed so heavy. The loops and iterators that it gives you, I just didn't see how it was going to map to throwing HTML up to a browser, it seemed painful. So I did my best to stay away from it and I comically swore that I would never do web development. I didn't hate Java, it is really good, compared to some other things it is great, and I really enjoyed writing the server side stuff so that's what I focused on.

21. You actually have some very advanced computer science knowledge. Have you been able to bring it to bear in any way?

Not in a practical sense. The PhD that I did is a very specialized topic and also quite abstract. It has to do with emergent behavior in multi-agent systems, which is basically when you have a collection of simple rule-based entities and you have enough of them in a shared environment. With certain rule sets you control the behavior that they have individually then you start to see global behavior. And I was fascinated by that, even in my undergraduate degree I knew this is a problem I will enjoy looking at. It's like a game where you have the little things moving around.

22. It sounds like you appreciate how little factors lead to the macro success levels?

Absolutely

23. Did Rail strike you as a sort of example of that?

Definitely. Rails through Ruby because it's so elegant from the bottom all the way up. Once you got the grips with the individual behaviors, the way that you deal with objects then everything becomes clear. And it seams natural, it seams like that's the way it should work. And if you have this flow that once you are comfortable with the basics then everything starts to make sense, and that's emergent programming.

24. It's been great talking to you!

Thanks!