Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Interviews Nathaniel Talbott on Experiment Driven Design

Nathaniel Talbott on Experiment Driven Design


1. I am here with Nathaniel Talbott who just wowed the Ruby community at RubyConf with a great new concept. Why don't you tell us all about it?

Yes, we talked about Experiment driven development, so a little bit of background about 9 months ago earlier this year, when I ran across this concept of Lean Startups, which is something that a guy named Eric Ries has been talking about a whole lot and he has basically taken the principles of Agile which we are all familiar with and put them together with these concepts from what's called "customer development" from a guy named Steve Blank, excellent book that everyone should read called "Four steps to Epiphany", just fantastic, it's like unedited Cafepress course notes from Steve Blank's course, the content is absolute gold.

And so I am reading through this stuff and I am learning about it and one of the things that Eric does when he is going through and talking about lean startups is he talks about like you should be a part of a process, part of what you need to do when you are starting a business is you need to be able to scientifically measure like instead of being opinions and conjectures around the table and just like my ego is bigger than yours in terms of what we should build and where the business should go; that development in particular can contribute to things by allowing the business to run experiments to say: "OK, you think that we need to collect all 5 of these fields, but I think we are going to double our signups if we only collect 2 fields."


2. Eric said they won't necessarily think of on their own, right?

Yes, absolutely. One of the ideas in the talk is that we need to start sort of reforming the conversation between the business side and the development side in terms of conversing around hypotheses. Instead of being business throwing features over the wall at development, instead the business needs to say: "Ok, here is what we think about what people are going to do.

Can you help us test that and figure out what people will do?" and so you can actually start to have fact based discussions around actual data you've collected, instead of being stuck in these never ending arguments about like: "Well, I read Jacob Nielsen said this thing one time and stop telling me this!" and the business going: "No, we have to collect this information or the business will collapse".


3. So how does the subject of developer driven experiments intersect with Ruby and RubyConf, why here, why now?

Why now is basically because I've gotten all fired up about this, just because as I went out and I was like: "Ok, Eric says I should run experiments" and then I am like: "How do I do it?" and you start looking at it and like Eric says "You should be able to write an experiment in one line of code" and I am like: "That sounds fantastic!" and then I go to actually try to do it and then you find out that there is this all this moving pieces to an experiment and it's fantastic that at the companies he's been at they have spent all this time building these internal frameworks, sort of allow them to write experiments in one line of code, but it doesn't help the rest of us.

And so it was just so frustrating to be like: "Ok, we really need to be doing this, but it's really hard and difficult" and when I looked back at the history of test driven development from a developer perspective and what it was like before we had frameworks like JUnit and TestUnit and SUnit, even SmalltalkUnit before that and NUnit and now we have like a whole plethora of different frameworks within the Ruby community, RSpec, etc.

It wasn't just a matter of us realizing that we should be writing test first that sort of caused the adoption of TDD. It was also the fact that we started to have these frameworks so you'd just be crazy not to do test driven development. It's like: "Why are we not doing this?" It's so easy to do. You go into a Rails project and you generate anything, and you generate stubs and you go and you run rake what's the default? It runs the test.

And so this infrastructure is really what helped get the momentum behind test driven development where it's really standard operating procedures especially in the Ruby community now. So what I want to see is the Ruby community just the way that we have this testing culture I want to see it having experimentation culture where the developer culture is constantly pushing back on the business going: "Do you have an experiment for that?" just the same way we say to each other: "Do you have a test for that?". So "Do you have an experiment for that? Why are you throwing an opinion at me? What's your hypothesis here and let me help you test it?

You say that this feature that is taking so much time is going to be really used, how many people are really using that? Let's actually test the usage. Oh, it turns out that you and one other user happen to use that feature. We are dropping it, it's ridiculous for us to keep spending money on this!" and having those kinds of conversations and make development an integral part of growing the business as opposed to just the separate part that kind of goes off and does their own thing and has tests that help them out, but it doesn't really feed back into the business.


4. How are you applying these concepts in your own business?

That is something that I am working on. I am really excited about A who does the Labnotes blog, I gave a preview of this talk a few weeks back and the day after I was at lunch with him and I was telling him about it and he got all fired up and put together a framework called Vanity, so that's at and I am really excited to take vanity now and put it in place because what I found is tried to apply this to my different businesses was that in client work it was like convincing a client that it was worth it to spend on the amount of infrastructure that we needed to produce to run the kind of experiments I thoughts would be valuable and sort of set up not just like one experiment, but sort of an ongoing culture of experimentation was prohibitive, like to take that to a client.

And the same was Spreedly as I looked and looked at doing in Spreedly it's like, well we are handling support requests we are trying to add these new features, how can we justify spending the time on experimentation even though we know it's really valuable it's really hard to justify. So it's something like Vanity now in place we can start and go in and it's getting to that point where it's like: "This is so easy. It's crazy not to." So I am really looking to start doing more and more and figuring out how to fit it into the businesses.


5. Final question, you brought Test Unit to the Ruby community years ago, right? And now you are bringing something that seems like you could be again like a land mark achievement. So where do you find inspiration to do these things?

Like I said a lot of the inspiration behind EDD specifically comes from Eric Ries and Steve Blank and what they are saying on the business side of things, but in general it's funny but a lot of my inspiration comes from trying to figure out what I should talk about at the next conference. Sometimes it like I want to go, I want to inspire people, I want to change behavior and so it's like what I am currently doing that I really feel like is a game changer and how can I bring that out in such a way as to really fire people up and get them change beahaviour.

Mar 19, 2010