Lean Start-Up, and How It Almost Killed Our Company
Since Eric Ries published his 2011 book – The Lean Start-Up – applying his experience with IMVU to business innovation, the idea has grown in popularity as a methodology. In fact, it has ceased to focus on the world of start-ups and has become the development methodology of choice for larger companies looking to improve their innovation success rate.
The MVP is almost universally accepted within software development as an aim, while ‘customer validation’ appears in every plan, presentation and project review.
As with many methodologies, as Lean Start-up has grown in acceptance and adoption, many of its subtleties have been lost. Rather than analysing the specific context in which it works best, ideas have hardened into orthodoxy as part of a solution intended to support a consultancy.
That consultancy exists because despite espousing the lean start-up method, very few companies practise it effectively. This article is not, however, a critique of companies doing lean innovation badly. Rather than calling for greater adherence to some original purity of method, I would say that the adoption of Lean Start-Up as a method (as distinct from the adoption of common-sense principles from within it), fails to take account of the limited conditions in which it is most appropriate. This holds true for the related approach of ‘Little Bets’, as described by Peter Sims in his 2013 book or Michael Schrage’s The Innovator’s Hypothesis.
The fault does not lie in the principles of Lean Start-Up, but in their application as a universal recipe to innovation success. Simple solutions are tempting – but they are rarely effective. I say this with more humility since it is a trap into which I, and my fellow founders, fell with our start-up, Gamevy. Let me tell you our story.
From Blackjack Attack to BornLucky Gameshows
The three founders of Gamevy had met researching and writing a series of books on Agile and Lean practices. Naturally, when it came to our own start-up, our mantra was to get the product out in front of customers. Find real data. Then adapt. Fail fast, fail cheap. From planning using the Lean Canvas to setting up a series of tests before we wrote a line of code, we tried to keep to a tight Build, Measure, Learn loop.
Gamevy had a clear vision. We wanted to add more fun into real money games, (real money, being the somewhat disingenuous way the gambling market likes to portray itself). In particular we wanted to build a game that felt like a TV gameshow, where combining a particular skill (answering trivia questions) with a luck/ chance mechanic gave you a shot at winning a big, jackpot prize.
The main business models were pleasingly simple – either a multiplayer version in which contestants wagered their stake against each other and we raked the winner, or a single-player version in which we acted as the house. Unfortunately, both were a form of gambling, a highly regulated industry in the UK. In either case we would need a gambling license. Within the strict rules of that regulation, we would not be able even to test a game with real money until we had a license in place.
We had created a series of playable paper prototypes. We looked at their possible value and profitability, how fun and playable they were, and how easy to build or market they might be. We took several of these to ‘play-testing’, including a memorable weekend at a fantasy convention, playtesting for 24 hours with hundreds of convention-goers all wearing corsets…
One game was the clear winner – a game we called Blackjack Attack.
At the time, our main options seemed rather like this:
Launch a fremium version
Go for the real money version
Faster to market – estimate 3 months
Slower to market – estimate 9-12 months
Cheaper, can be handled internally
Expensive license fees plus external costs
Small revenue stream via in-app purchases
Much bigger revenue stream
Easy / cheap to acquire customers via Facebook advertising on platform – industry average £1
Hard / Expensive to acquire customers willing to deposit – industry average £200 minimum
High Competition from small, innovative companies
Low innovation from large competitors focused on different sector
Payments/ accounts etc handled by platform
Need to build own platform or integrate with third party
Multi-player will be easier since free play means can use bots if necessary
Multi-player will be very hard since it will require high liquidity of simultaneous players
Not our end goal
What would you choose?
Be honest now, which sounds like the route the Lean Start-Up method would encourage you choose?
We decided to go fremium first.
The Increment Trap
We had two key questions – will people play the game and will people pay for it? Of course, we had some ideas about what we wanted the repeat play and retention metrics to look like and what we would deem a ‘success’ and what a ‘failure’.
Getting into the Facebook store took longer and was more costly than we had anticipated. In order to test the game with a statistically significant volume of customers and to charge real money, we were driving a bunch of requirements that we knew were not where we really wanted to be in the long term.
Once we were fully launched and we started getting feedback, it wasn’t perfectly clear.
Were people playing the game? Yes! Our repeat play figure was a relatively healthy 10%, although we suffered huge drop off during the first game.
Were people paying? Well, some people were. We were above the limit we’d set ourselves as ‘FAIL’, but it wasn’t enough people and not as much as we’d like.
That was all OK! We were getting feedback and we knew what we needed to do – improve!
So we began to see if we could up those numbers, refine the product, improve the drop-off rates, convert more people, see what referral and sharing metrics might look like, if the business model of gaming could be made to work…
Our changes worked. We managed to make improvements to all of our key metrics – but none of them were going fast enough. Looking forward we could see that it would be perfectly possible to spend the next year and all our resources improving the game. Would it ever do well enough?
Four months after we’d begun work on it and 1 month after launch, we bit the bullet. We needed to go back and take the other choice – the much bigger bet of regulation – and we needed to do it with a different product, one which accepted a different series of trade-offs.
With hindsight – that wonderful vision – we realised that the learning we had bought with such hard work was not that valuable to us after all. Rather than focusing on the product, we should have focused on the particular market we wanted to be in. The differing business models of the fremium and real money gaming markets turned out to be crucial. Interestingly, a traditional business plan might have helped us focus on that far better than Lean Start-Up.
We were extremely lucky that our costly experiment did not cost us too dear, that we stopped it before it was our only experiment.
In the last year, Gamevy has gone on to gain our gambling license and build two games (both very imperfect, still). In the last month, we have shifted our focus away from our eventual goal (be an operator direct to consumers) and towards being a supplier to a couple of specific partners. Doing this has meant delaying our launch, making the prospect of consumer feedback more distant as we redo work to integrate with a partner platform. We have, painfully, decided on a trade-off that lets us stay alive longer in the hope that it makes our eventual success more likely. This is not failing cheap and failing fast. Only time will tell if it’s the right move.
It has made us think hard about Lean Start-Up and some of the considerations that might have got us here faster if we’d thought about it differently.
When the MVP is not what you want
This is a popular diagram used by Spotify to explain the concept of an MVP. The problem is that like many simplifications, it offers as much of a block to real understanding as it does clarity. The bottom ‘correct’ How-to progression shows 5 ‘solutions’ to the problem of transport. Yet, of course, they are not really 5 different versions of the same thing – they are 5 separate products, each with specific benefits and requirements, each needing different learning validation.
If you want to build a car, then the learning you gain from dedicated skate-boarders is not going to help you very much. In fact, as you listen to your customers and implement features about carbon composite materials and epoxy fibre reinforcement to increase pop, you are going to get further away from your goal of creating a car, not closer.
It’s just an illustration! No-one’s meant to take it literally!
Yes, we know, but actually, we think the diagram rather neatly illustrates why Lean Start-Up doesn’t help you much where barriers to entry really exist. Where there is something that makes building your end-goal difficult (like the complexity of a car), then ‘steps’ along the way can often turn out to be not the right steps at all but rather choices that lead you in a different direction.
Let’s take Gamevy’s example. We knew that real-money gaming had formidable barriers to entry. Rather than facing those, we decided to try to validate the idea in an area with lower barriers to entry – fremium gaming. But in doing so we discovered several things:
- the learning was not that useful – these were not our eventual customers. They were essentially telling us about skateboard features when we wanted car enthusiasts. Even if some social gamers are also real money gamers, we were talking to them within the context of a social gaming market. It turned out that this really mattered when it came to learning.
- we discovered new barriers to entry existed that we had not foreseen. The costs of running the game live – technical, marketing and operations – were much higher than anticipated. Since this wasn’t the ‘real’ market we wanted, these were wasteful costs.
- The MVP was expanding. In order to try and get the validated learning – that is, would people PAY, for the game – we needed to build an entire fremium economy around it – referral and sharing mechanisms, sales, alerts and notifications…
The Barriers to Entry
Context matters – the higher the barrier to entry, in general, the less minimal an MVP will be. Trying to take smaller steps can often lead to errors like those Gamevy made.
As my co-founder Paul Dolman-Darrall once commented, ‘little bets’ or MVP steps work best in places that have lower barriers to entry:
- Disruptive markets where the barrier to entry will be attacked. We see this in examples such as Uber – attacking the traditional barrier to entry held by cab-drivers, or Air BnB, attacking the traditional hotelier market. They took a calculated risk that regulation either could or would not be applied to them. In many industries – gambling, healthcare etc – this risk would simply be too high.
- Industries with naturally low barriers to market. These are typically commodity or service billing industries, which offer a respectable living but are unlikely to lead to the asymmetric payoff associated with successful start-ups.
- Incremental improvements where the barrier to entry has already been paid. For large companies trying to innovate on an existing product or service – this is the most likely scenario and explains why small bets and the concept of the internal lean start-up works so well for larger companies.
- New industries where no barrier to market exists at all – arguably this is exactly where Eric Ries’s company IMVU was placed – 3D avatars were entirely new within an online social market that was itself in its infancy.
Markets with high regulation or incumbent competitors often require the opposite strategy – big bets. As Steve Jobs called it, moments in the life of a business when you ‘bet the farm’.
As for Gamevy and our decision to bite the bullet and gain our Gambling License, there is often no way around these big bets. If we had failed to get the license the company would have closed down. There remain dozens of failure points in our near future – but in our case the big bet was the only one that mattered.
Step-by-step versus ‘little bets’
Pixar is a frequently quoted example of little bets (including within Peter Sims’ book). There is an important differentiation between enforced step-by-step development and a deliberate ‘little bet’ or ‘MVP’ strategy. It is a difference that is frequently elided, but I would go so far as to claim that prior to their first success, Pixar’s actual strategy was to take the biggest bets that they could feasibly manage.
Pre Toy Story, for example, the team made several computer-animated sequences for adverts. Although such projects may well have provided valuable technical learning, they were a product of necessity – providing a small revenue stream to keep Pixar afloat. At no point did the team decide to pivot and chase such work; they were eager to be able to reject it. Similarly, the shorts that the team created (including Tin Toy, the genesis for Toy Story), were made to show off the company’s hardware capabilities and to gain a business customer. Pixar was not trying to build a consumer base, it was offering a shop window to gain a partner. As it successfully did when commissioned to make Toy Story for Disney.
Today, the available technology of YouTube, Vimeo and crowd-funding might encourage a young animation company to go direct to consumers. Who knows if Pixar would have made the same choices today? Who knows if they would have been more or less successful in doing so? Pixar accepted some painful trade-offs in order to survive.
I suggest that if in the 80s and 90s someone even richer than Steve Jobs had offered Pixar the opportunity to work ONLY on their own full-length animated feature film with no need to worry about revenue or a distributor, they would have jumped at the opportunity to turn down the adverts, the shorts and Disney’s heavy-handed oversight. It is only with hindsight that we perceive decisions that at the time appeared to be compromises as small steps along the way to animation dominance.
Characterising decisions made as ‘little bets’ or a series of MVPs, when in fact they were enforced by necessity (need for revenue, inability to acquire distribution or customers without a partner etc) offers a misleading picture both for start-ups and larger companies.
MVP vs MDP, The Great Start-Up Experiment
Start-ups work rather differently to how large companies run innovation projects. A big company with a portfolio of innovation products is the perfect place to implement the ‘little bets’ strategy – investing more in this seemingly-successful idea, killing off this poor one. For start-ups a poor innovation product is its only product. When it fails or delivers only a small revenue stream, there are a limited number of times that the start-up can pivot, or kill an idea and start again.
Each independent start-up is its own ‘little bet’ – the market gains the benefit of the few that succeed, but that’s not much consolation for the 80% of start-ups that close within the first 3 years. Those that succeed will have a mixture of good ideas, good management, good funding and luck. The Lean Start-Up method fails to say much about the equally important, latter two.
Since working in a start-up often means significant sacrifices, individuals would certainly rather fail fast and then move on to something more successful. But what happens when an MVP actually makes failure more likely?
In the IMVU story, there is little real cost to launching a buggy, poor product. Eric Ries humorously mentions the personal cost to his reputation as a technologist, but not to the product itself. This pre-supposes that there is a large pool of potential customers, that the costs of acquiring them will not outweigh the value of their feedback or revenue if acquired later, and that negative feedback will not have an impact on the product or company’s future.
In fact such circumstances are rare. In many industries, an MVP either delivers little learning or offers significant risk – launching a buggy drug or financial instrument might shut down your company. Instead, the company needs to focus on delivering the Minimum Desirable Product or MDP.
Teasing out the difference between viable and desirable can prove extremely difficult. Generally customers are not able to tell you what innovation they find desirable until they see it. And seeing it means that you have to do the work up front before validation – testing concepts and ideas will not always provide reliable feedback. As any marketer will tell you ‘would you pay for this?’ is a meaningless question compared to seeing what customers actually put their hands in their pockets for.
Take a non-technical example… Publishers will often say longingly that they are looking for ‘the next J. K. Rowling’. And if you ask children ‘what kind of book do you want to read next?’ they may tell you that they want something ‘just like Harry Potter’. Does that mean that publishers should be commissioning dozens of series about a boy wizard set in a boarding school? No. Although there are plenty of copy-cat books which seem to suggest publishers have not really figured this out. The disappointing sales of such series mean publishers are paradoxically LESS likely to take a chance on a new author with a kooky idea.
Novels are not MVPs. Attempts to make them so – outlines, single chapters and serialisation – are rarely successful. Few readers want to try a chapter knowing it might be months before they get the rest of the novel and any fiction author will tell you the first chapter will have many rewrites by the time the last chapter appears. For authors, it is hard work to get agents or publishers to read a manuscript and not much easier to get readers to your blog. There is also a cost to offering up first drafts: your limited group of readers will be unlikely to read your next draft and worse, they may leave a review telling everyone else how rubbish you are.
These risks hold equally true for most start-ups: a limited pool of customers who are potentially expensive or difficult to acquire and whose negative feedback is likely to be overheard and have a major impact. Eric Ries frequently stresses that early launches, ones expected to fail, should not have any press or marketing – but it’s a strangely out-dated idea to believe that only advertising or PR impacts on a brand’s reputation.
The Minimum Desirable Product – one which at least a sub-set of customers will love – is often not very minimal at all. Apple is the classic example – a company that focuses on ensuring the product is loved – even if that means completely redesigning something in order to achieve an aesthetic improvement (as occurred with the i-phone). Most start-ups have only one shot at such an idea.
At Gamevy today, we build game prototypes for internal testing – they help us refine the mechanics of the game, including how easy or hard it feels and what the win rate / sense of agency might be. But we do not put them live – partly because without official and expensive testing via an external party we would be violating our gambling license to do so and partly because we can’t risk lowering our repeat play and retention metrics by offering our (or our partner’s) expensively acquired customers an inferior game. Perhaps one day – in our successful future – we might implement a labs or beta environment where a few customers can play these prototypes for real and where we undertake the external testing early as part of our initial development cost.
At the moment, the cost of getting to the MDP is less than the risk of the MVP.
In such cases if all the Lean Start-Up method contributes is to say ‘make sure you don’t add in anything those customers won’t care about’, then it is not especially useful. You will still have to make a series of judgement calls on what is necessary, what is desirable, and what is a ‘nice-to-have’. The only validation you are likely to get are from informal focus groups or existing ‘friendly’ customers – neither of which offers a guaranteed guide to market performance.
In 2004, when Eric Ries co-founded IMVU, it was still relatively cheap to acquire customers online. Those costs have since increased significantly as the online environment matches other media in marketing budget requirements. This makes parts of the model around customer acquisition rather like turning for advice to the goldrush pioneers when nowadays the mining companies have moved in.
The insights of Lean Start-up are still valuable – but their best application is in larger companies looking for a more effective way to manage their innovation portfolio. Where customer acquisition is the ‘problem’ of a different department or asks for transition and conversion of existing customers, a single-minded focus on product development and validating learning may be exactly what is required. Similarly when the company already works in the space and has dealt with the barriers to entry, a single-minded pursuit of keeping the product as small as possible, is also excellent discipline. After all drugs companies and car manufacturers are both as determined to avoid ‘waste’ as any software company.
For start-ups, high barriers to entry can make the size of the MVP so large that there is little point in calling it a ‘little bet’. Instead, we should apply the common sense principles of avoiding waste and attempting to set up experiments to validate underlying business assumptions as soon as possible – although we should accept that the results are rarely binary and therefore there is little clarity on how to proceed. This is certainly incremental development with a focus on customer learning, but calling it a guaranteed process is overclaim by any standards.
In the end, I would sum up our caveats as these:
- Smaller is not necessarily better and viable is not always the right measure.
- Validated learning sounds great – but barriers to entry may force you to develop a product blind and without experiments. Blind progress may be better than open-eyed stasis.
- Pivot points do not only come from customer feedback – there are many other types of serendipity that may intervene and offer a choice. That choice is never easy because almost every trade-off hurts.
- All the metrics and hypotheses in the world will probably not help you when reality bites. Since luck plays such a major role in what occurs, try to keep this as light-weight as all your other planning.
- No process or discipline can guarantee success. There are always new, exciting and unforeseeable ways to fail. We’ll let you know what original ones Gamevy comes up with in the next year.
About the Author
Helen Walton is co-founder and Marketing Director of Gamevy, a tech start-up which recently won the PitchICE award and will launch its games soon via a partner. Her writing has appeared in diverse places, from the Daily Mail to the Tate Britain, and on topics from lipstick to organisational change. The three Gamevy founders met while writing the VFQ books which now form the BCS Agile Practitioner qualification. Three years of researching and debating methodologies, and interviewing hundreds of organisations led to the founders also setting up a community and conference - Spark the Change - for people who want to improve the whole organisation, not only IT or product development. Running in London 1-2 July 2015, Spark attracts people from around the business and from all industry sectors. Leading thinkers on the future of work and management will be speaking, along with case studies from innovative companies including WL Gore and Spotify, and practical workshops on skills and tools to implement real, lasting change.
The article presents a problem with people's perception of lean...
The issue I have with this article is that most folk have a misunderstanding of value and the scientific method. This leads to a misunderstanding of 'failing cheap and fast'. Indeed, having tried in high barrier to entry environments, I wouldn't say the way it's assessed here is correct. The crucial bit is that the barrier to entry is a cost. What you deliver is a benefit (and revenue). You have to make up that cost from the revenue. In classical VC you end up investing the barrier to entry cost as capital expenditure and hopefully operational revenue accumulates to pay it off. LS in this circumstance DOES work, as it raises your IRR & ROR to a point that you reach break-even faster.
The problem is that because folk don't see that on day-1 and indeed, may go out of business because they underestimated the barriers (you could argue that is OK, since the business didn't bankrupt everyone involved - mitigating risk), they assume LS doesn't apply. It does, it just requires a modicum of thinking instead of, as the article suggests, just reading the book. After all, all systems, even people oriented ones, require a process of some sort and an initial condition. Part of the latter is that barrier to entry. Not understanding that means your outcome will be different and hence, the process has to change to compensate for it, which in classical companies may include cutting costs.
The article also shows how the assumptions were badly worded. In scientific studies you actually assume you know nothing and that everything you do doesn't yield any benefit to anyone at all. That is the null hypothesis. Then when something you try does yield a significant difference, you chase and iterate on it! The use of a shell of a landing page is actually a good way to test these out. In addition, the two choices of fremium and premium content were just far too stark. You had a number of assumptions (each line) and tried to answer all of them at once. To big a junk, with too many covariates for you to come to any decent conclusion about to be honest.
However, as an experience report it's a good lesson for folk out there to understand exactly what LS is trying to do. If you are running a lean startup like the above article implies (note, I assume that it the ocmpany is not being run like this now) there is a whole host of holes you'll need to plug and it may still sink.
Re: The article presents a problem with people's perception of lean...
As usual, and as is typical of the consulting industry, you assume that the fault lies in the skills, or knowledge of the people in the story. With the background of the team, I am confident that we understood what value is, have a deep understanding of the application of the methods in the workplace, including our perception of lean and have applied a modicum of thinking. We made mistakes, who doesn't, but we started with a null hypothesis, carried over 3,000 experiments, built observations based on the experiments of others (which is cheaper, when it's your own money), and chased situations in which there was evidence of a significant difference.
Now you have to be pretty foolish to assume that we just summed up all of our experiments into a table of eight rows by two columns and made a big choice. It helps the reader understand in hindsight, quickly the major pieces of information on which we had to make a choice. Everything was far more nuanced than we can capture in a case study.
The other mistake that you make is to suggest that our choices were stark. One of choices was very stark. We weren't just talking about premium, we were talking about regulation that would prevent us from experimenting and cost over six figures. The other choice was not so stark, allowed lots of experimenting but became more costly than we expected to learn. This for many reasons is the obvious lean startup choice. The one you would have probably picked. And like us, you would have been probably wrong.
We told this story not to attack a method, but to provide an account of the use of ideas in practice. In the article we explained many of the trade offs that were in place. You cannot carry out an experiment on every decision (the costs of such insurance are too high).
In our case study we made clear that our choice was based on 'How can we learn more quickly what works, and discard what doesn't'.
In doing so we made a choice. A choice that turned out to be wrong. We placed learning quickly above the end goal. That trade off did not work.
We have skin the game. Something which clearly you do not. This is the story of a company who is not trying to sell a method, a solution to the problem, nor any consultancy on the subject. Instead, we want to encourage learning. To offer our results to help other people. Further to that, of course there were too many covariates, but choices have to be constantly made, and barriers to entry can increase the cost of decisions before the learning comes back. Lean startup may very well improve the IRR & ROR, but it does not necessarily lower the cost of the barrier, a quite crucial thing if I want to avoid going bankrupt. Skin in the game!
If anything this article is a criticism of the orthodoxy of solutions promoted by a consulting industry and them blocking learning. It's obsession with grand business theorising and applying sudo scientific approaches coached in theoretical language. Though we cannot hope to capture the complexity of our situation, we hope to capture something which is real, understandable and crucially something done in practice rather than a textbook.
Good example of why viable is the most imporant in MVP not minimum
My take on MVP is that you're primarily concerned about using one to determine viability, which comes in three forms:
1. Market: will a specific group of people buy what I want to sell?
2. Technical: can it be built?
3. Regulatory: will I be allowed to build it?
The first is easy enough to test with customer development feeding into Landing Page MVPs. The second can be somewhat bypassed with a concierge MVP, but there are certain product types that are just beyond the current state of tech (think text-to-voice that is indistinguishable from a human being). The third is the most complicated, and usually requires tough decisions. Either you are fully compliant (your approach) or you bet on massive growth which will help give you a negotiating power when you have to deal with regulators (Uber/AirBnB).
In your case, I'd argue that going after the minimum to receive regulatory approval would still be an MVP, even though it requires more money to build. You want the minimum to be able to run meaningful experiments. In other words, an MVP is not a beta or shoddy product. Good experiments are ones which de-risk your idea, and help you understand your market better, not necessarily ones which buggy.
Paid is a different audience. Just like someone who buys a cup of coffee at a cafe is a totally different buyer than a coffee wholesaler, insight from the smaller audience wouldn't necessarily translate over if your aim was to go after the "real money" crowd. You could do some prototyping on paper or talk to them, but at the moment you start building, you would need to do enough to be actually able to sell to them.
Minimum desirable product is what a successfully iterated MVP eventually becomes. The whole point of it is to learn--so that you can create something deeply desired by your audience. If you've run thousands of experiments, that's probably thousands more than most startups, and you are following the process. You'll get to an MDP faster with a series of MVPs, than you would by working silently behind closed doors.
As a side note, many novels were successfully serially published in the late 1800s (e.g. Charles Dickens), with market acceptance being a key criterion for continuing to write a serialized story for newspapers. Once complete, the most successful commerically serials were republished as novels.
In any case, I really found your insight valuable with respect to how Lean Startup might not necessarily apply to all industries and scenarios, as the enthusiasts (including Eric Reis) are keen to declare. Generalizing from Eric's IMVU experience is a bit dangerous, since IMVU is a very specific type of product and user.
Re: Good example of why viable is the most imporant in MVP not minimum
It does take guts to go contrarian, especially as we are still a live business operating in the startup community. A number of people told us they did not even want to comment on the article due to fear of being seen to question lean startup.
Our market is very specific, and it's regulation is far more stringent than markets that a Uber/AirBnB would operate in. There are many examples of people being sent to jail in our market. Betting on massive growth would be betting on going to jail. Not a bet I am sure most people would want to take, and nor one I doubt our partners and children would appreciate.
As such, a landing page is not allowed, a concierge MVP is not allowed, in fact no activity is allowed at all without independent approval by a test house and regulatory approval which in the end did take us about twelve months. Additionally, whilst we might be direct to customer, that is a very expensive approach. As such we are starting with business to business. No business we deal with will even remotely entertain such ideas, let alone do business with an unregulated company. You would get laughed out the room.
So you can see the choice, fully regulated or experiment in a similar market. Because we are fans of Lean Startup, we chose experiment in a similar market. That was a mistake in hindsight.
Now there is a nuance that I feel might not be covered in the article. And this goes back to the heart of where viability came from as a concept in Lean Startup. Viability for me is working out what can be financially viable? Desirability is what do people desire? There was a third which is feasibility, can it be done?
Now of course, in the real world of product development we focus on all of them. Learning all the time, and the magic is in the overlap. But Lean Startup has chosen to focus it's lens on MVP. I believe it should focus more on MDP. Our argument was that releasing an MVP can be damaging, and that Eric did not really take this properly into account. Thus, if you can, maybe you can carry out some closed MVP testing, but quite often you have to wait until MDP to go wider and public. I saw this captured in a quote from thegrid.io the other day.
"While "ship before you're ready" is a permanent part of startup vocabulary, it still needs to ship ready enough for a great experience." - Thegrid.io
But I agree with you, MVP, MDP are connected, and of course there are many things you can do to try and improve and learn from outside your office.
One of the reasons why we wrote this article was not to dig at lean startup, but instead to share a real experience of it's use, and to examine where experiential learning is difficult.
Interesting side note.
Re: Good example of why viable is the most imporant in MVP not minimum
There is no point having regret at making a mistake, as you did the best you could at the time, and you are out there executing and learning.
I'm not sure I understand how this follows: "Because we are fans of Lean Startup, we chose experiment in a similar market." The book LS clearly says that you should decide to pivot based only on gathered data. This sounds like you did a gut feeling decision, which is perfectly fine, but I'm not clear how that ties into Lean Startup.
As for viability, yes it could be clearer, particularly since viability's the important bit of the "value hypothesis".
To me your distinction of viability and desirability is useful, although they have a common denominator. You can prove both if are they willing to transact with you...give up something valuable (ideally money) in return for something they desire. I'm still thinking about that one.
It's admittedly a bit more complicated than that, as anybody would be willing to outright sell something of iPhone quality for £1/$1.
If you're building a business, though, desirability is only important from a business POV to the extent that it engages your prospects and gets them excited to buy (transact) with you. Any more than that is just premature optimization (certainly in a startup with limited funds). And definitely not Lean Startup. Which doesn't mean it's wrong or right, just not Lean Startup (and more risky). In the real world though, it's not that cut and dry as you pointed out.
Re: Good example of why viable is the most imporant in MVP not minimum
Giving money up isn't quite the same as desirability. I give money to all sorts of things, but I hate it. Given a chance I would happily switch supplier. Further to that, I almost never recommend anything that I pay for, but I do recommend many things that I love. I agree that desirability can be more than is necessary in some markets, but not all. Good to have the debate, I am sure we see eye to eye on many subjects.
Re: Good example of why viable is the most imporant in MVP not minimum
A quick note about the serial novels (my favourite subject!). While Dickens and many other writers published in serial format this was primarily a technological choice - not a design one. Books were expensive - well beyond the range of normal working people. Dickens and others worked out that through cheap magazines bought weekly or monthly they could hugely extend their readership. Dickens rarely responded to reader feedback or 'experimented' in any meaningful sense (except in one famous case where he made Oliver Twist less anti-semitic following complaints from Jewish readers). When publishing technology changed to allow the cheap paperback, the serialisation model effectively died. Its successor, the TV serial, is undergoing a similar revolution via the box-set and digital on demand.
My point is not that minimising things can't be useful (especially to us cheapskate start-ups or authors in garrets), but that the context of consumer demand, regulation, marketing etc it ceases to be a meaningful 'methodology' - especially in the case of start-ups.
Re: Good example of why viable is the most imporant in MVP not minimum
Yes, exactly--I would even say business (constraints imposed by technology) not design:
The form’s ubiquity was due to the economics of publishing in a day and age where manufacturing was far from streamlined. “The reason that serialized fiction was so popular back when it was, in the nineteenth century, was that it made really good economic sense for publishers and for writers,” explained Ms. Love. “Books were incredibly expensive to print, and so this was this wonderful way to test the commercial viability of a story and, hopefully, to build an audience before you laid out that expenditure.”
It's a similar business counterargument against BDUF (Big Design Up Front) that we have in software nowadays, certainly from the perspective of continuous delivery, cash flow patterns and audience feedback, which is where I see the similarities with Lean Startup and agile (as opposed to design thinking). If the audience didn't like the serialized story (which was both the product and the PR for the final product, then there was no need for a final product.
Of course, Dickens predates the Lean Startup's focus on structured experiments, so that analogy's a bit of a stretch. I would imagine, though, that if a particular serialized story didn't do well commercially, it hypothetically wouldn't have made it into a book format (since publishing a book cost a lot back then), regardless of the design.
My point is that lean startup MVPs as a methodology are primarily about focussing on being effective with viability testing (which includes the minimum of everything relevant like regulation and especially consumer demand), not about about being cheap/efficient and being forced to live with half-baked products as a result.
If you focus too much on "crappy products" or being cheap, you're missing the Lean Startup methodology's primary purpose. If anything, being disciplined about running Lean Startup tests costs more up front (in the same way TDD costs more than test-less development), for greater benefit down the line thanks to a deeper understanding of your customer's needs. It's just done in smaller batches than in design thinking....
Minimum viable product not minimal product
Another great case study would be the pharmaceutical industry and the concept of MVPs in the context of drug testing. The same principles apply here – but the context has to be taken into account. A good example is that completely different approach to an MVP in vaccine development when the Ebola outbreak tool hold in W.Africa recently – the context was very different to that of developing a flu vaccine. The biggest problem I have with Lean Startup, are the words Lean Startup. The principles apply to ANY product based based business, but the attributes used as well as the time span over which iterations take place will depend on context. Excellent article from the trenches guys!
This reminds me of Validating / Invalidating MVPs
"Lean Startup" is Not Only "MVP"
I think you are thinking that "lean startup" means "MVP." Have you read Osterwalder's "Business Model Generation?" In it, he specifically explains to model and test all parts of your business, the Riskiest Propositions First.
If you'd put forth an integrated model to test and laid out your hypotheses and tested those (and iteratively created new ones and new integrated models) you'd then test the riskiest parts first and should've certainly failed faster. Yes, most startups riskiest hypothesis is about the customer: it could also be about the regulatory environment.
You did not lay out a model. You picked the easier product to build of two and then called making the product better "lean startup."
Don't be too discouraged; the ego gets in the way of the best scientists sometimes. There's a natural tendency to want to justify something you've spent time on.
The business model canvas is free: just print and use from now on so you don't waste time.
Re: "Lean Startup" is Not Only "MVP"
It is a 3,500 word essay, it is hard to capture months of experimentation in so few words.
However, we discuss above that we started with "a canvas" - which is a minor variation on the Osterwalder model. Here's a copy from two years ago - i.imgur.com/9aw30ST.png (this itself a snapshot in time). All of our tests were designed to test as quickly as possible the biggest risks in this model.
We carried out months of testing, before reaching a point where we had to do something more substantial to find out more information about the biggest questions still remaining. Our tests were designed to that model, focusing on the riskiest bits.
The reason we did not fail faster is that it passed many of the earlier tests. In the article we talked about 'playtesting'. Also to put it into context. The game Blackjack Attack was played 600,000 times, 60 times per player, with some players playing over 10,000 game in less than two months. This was not a binary result (did we fail, did we succeed, in truth we did neither as a product). It wasn't good enough, nor was it a zero.
This is not a justification, but a critique of simplistic solutions to complex problems. And also a critique of those who often blame the implementation and the people rather than really questioning their tools. Going even further, this is a critique of the solution lead consulting industry which constantly sells solutions without having any skin in the game.