Stephanie Kaiser on Metrics Driven Design

1. We’re here at GotoCon Copenhagen 2012. I am with Stephanie Kaiser. So Stephanie, who are you?

I am the product leader for Wooga. I’m currently responsible for four games. One of them is Monster World. That’s my baby. That’s what I started with at the company and three others are currently in development, so obviously I cannot talk about them, they are super-secret.

2. So Wooga is a game company?

Yes, Wooga is a social game company and we are based in Berlin. We have now 200 people coming from 35 nationalities; different countries. And we are building essentially Facebook games, so games for Facebook and for mobile, so for the iPhone and for the iPad.

3. So you’re building with Flash and HTML or iPhone Apps or?

We are building them with Flash for Facebook and iOS Native Apps for iPad and iPhone.

4. So you gave a talk on Metrics Driven Design. What’s that?

It’s essentially about how we design our products and how we are looking into a lot of numbers that describe the performance of the product. So we call them KPIs, Key Performance Indicators, and they are describing various user behaviors and that’s how we learn which kind of feature for example works better for the users that are out there. And, that’s how we basically enhance the games that we have live.

5. So how do you decide what is a metric?

Well, it really depends on what you are looking at. But for a game there are certain metrics that describe the success of the game. So obviously it’s about how many users are active on a daily basis, that’s the DAU, Daily Active User. So how many users actually play on a daily basis. And then, as well if you look into engagement, it’s about how many users that started newly on a certain day. How many are still there after one day, after three days, after seven days, so how engaging is the game for the new user that gets into the game.

And then, as well it’s about virality. So how well does the product spread on its own. On Facebook there are viral channels that you can use for example Feed posts and requests and you can send them out of the game and invite other people into the game. That’s how those games work very well. And then it’s obviously about monetization. So how well does the game monetize and then it’s about I don’t know how much revenue you make on a daily basis. How many users are actually paying on daily basis, etc.

6. So these metrics come in at what point of the design? At the beginning or when you have a product ready? Do you do internal beta tests?

Obviously, we do internal beta tests and we do a lot of testing beforehand, before the launch. But the numbers actually kick in once the game is live because then you have actual users using your product and you need a certain number of users to get a statistically relevant result and therefore, so it’s really after launch that’s where most of the work comes.

7. In a way you’re trying to adjust the course of your product in a way to optimize it, that’s what it’s for?

Exactly. So you obviously build a core game out of an idea that you have and you prototype that and you user-test it in the first place, so there are no numbers involved. You really put users in front of the product and see if they are having fun or not. And then, once you have this core game loop and you have a minimum viable product that is like the smallest version of your product that you can put out there and put in front of users to actually measure stuff, at this point the metrics kick in, but that’s really after launch.

8. Once you get these metrics, what do you actually do with them?

So let’s say, we look into the enhancement of the tutorial of a game, obviously it has to be really simple to enter a game through the tutorial and therefore we look into the metrics of the first user funnel, how many users do actually start a game on a specific day, and how many users drop off at each step.

Now, we see those percentages on each step and then if we see, “Oh, on this specific step, on this specific actions, users are actually not able to do that action because maybe we did a flawed design,” then we see that in the result in the number. And so then, we come up with, “Okay, at this step we need to enhance something,” and then we come up with variations of that specific step and then we A/B test it, that’s the result of looking into the metrics.

So we have let’s say two variations of a feature put that live to a subset of users and see how they perform, so which variation performs better. This happens at the same time that is very important because if you would do that in sequential order, then I don’t know, for example the weather could change and your numbers are messed up. But through this A/B testing for example we find the better version, so it’s really is a little tweaking of a specific feature and we find which version works better and put that live for 100% in the end.

9. So you say it’s mainly with A/B testing you tweak minor features, so it’s not like you’re forking off different products?

Exactly. So it’s really minor, minor, minor tweaks and that is a very important point that A/B testing and numbers never really help for the initial design of a product. Obviously, there needs to be a good initial design. But that is based on obviously gut feeling which makes it really difficult.

10. This also leads us into another topic you brought up and that’s usability studies. So how do you do it? Do you do it with a classic camera over the shoulder thing or what do you do?

We actually do it the human way, I would say. So we invite people and put them in front of a computer, in front of the product, in front of the game, in a hopefully very normal situation like that, they don’t feel it’s very different from their home computer. And then there is always a product manager besides them helping if there is help needed. And then there are three, I would say three people in the audience just watching what the tested person does. And we usually don’t give directions, we just look at what they are doing and we never answer questions. I think that’s a very important part because obviously those questions that arise throughout these usability tests, they are really helpful for us to understand where the user does not really understand how to use the product in the end.

So, we do that on average on a biweekly basis, I would say and it’s always one hour. And then right afterwards, we sit down like everybody who was included in this test, we sit down, right down the results, and discuss the findings. And then decide immediately afterwards what will be changed or not within let’s say the next couple of hours for the next test maybe or within the next week because otherwise you will never implement it, anyway.

11. So you do rapid changes? You say you do two tests in a day and you change in between?

Stephanie: Exactly. So sometimes we are planning to have a test in the morning and then we have some time to implement the changes, like we found something we had a learning and we implement the change and then we test it in the afternoon again.

Werner: That’s very Agile.

Stephanie: It is, yes. Agile and very rewarding, I have to say because you see immediately how the product got better over time just by looking at your users, thus really rewarding.

12. The developers of the product are they included in the usability test?

Yes, they are, sometimes. So the tests are always done by the people from that team, so we are organized in game teams I have to say that. That there are dedicated people on one game, so for example Flash developers, back-end developers, graphic designers, they are all dedicated to one product, so there is always one product person obviously in this test. But sometimes we take Flash developers as well, or graphic designers because I think it makes a very big difference for them to actually see users how they are using and failing on the product. It’s nothing better than to really look at it and to really see it live, so to speak.

13. How do you keep the developers from yelling at their test subjects "Just press this thing”?

We just tape their mouths.

14. So that helps? Okay. So you don’t have one of those two-way mirrors?

Stephanie: No, no, they are really in one room. Obviously you could have a setup with video cameras, but that even scares people more than having just human beings in the room and I really don’t want to make the situation artificial, so it has to like as usual as it gets, as homey as it gets, so to speak and therefore I keep it human.

Werner: Yes, cameras are scary.

Stephanie: Yes, they are. I can tell.

15. Finally, you already brought up the topic of Agile. So how Agile are you?

Well, we are as I said organized in small game teams, so it’s around 15 people. It highly depends on the kind of game whether it’s a mobile game or a Facebook game, whether it’s an arcade game or simulation game. So let’s say it’s 15 people and we are working, project management wise, we are using Scrum and Kanban, but not religiously. So we keep saying we use the best out of both worlds which means that we are working in weekly iterations where on Tuesday there is always a new version live on Facebook and then on Tuesday afternoon, the new development week starts until Friday, then you fresh up your mind over the weekend on Monday is testing day on Tuesday is a new release.

Now, Kanban, we are using in a way that whenever something comes up that is more important, we can reprioritize on a daily or whenever. You can reprioritize basically anytime and whenever there is a feature done already before Tuesday, let’s say, then we push it live as well. So we don’t really wait for the release day, but we keep the weekly iteration to have this frame for the developers.

16. Well, that’s great and we’ll all try to ruin our productivity by going to Wooga.

Yes, please. Go ahead. I’m aware of the fact that we are killing productivity on a lot of people.