This article is part of the Agile Manifesto 10th Anniversary series that is being published on InfoQ.
It is human nature to look for patterns in solving problems and we have a tendency to want to reuse what we already know. Sometimes this is a useful tactic but often the problem we face is not the same as the previous problems so our previous way of doing things is not best suited for tackling the next problem. In the world of software development we often see the same desire to find the “one true way”, “the BEST method”.
Interestingly, most often, our better approaches are based on our knowledge and experience from the past. In other words, we rarely discard what we've already learned; we simply build on top of it. Its time to move on and create new approaches, if not new belief systems.
In the last few years, when I think about Agile, I've come to accept that I've moved on (building on top of it).
With my time-machine™, if I went back a decade, my journey to “nowhere” (a place beyond a prescribed set of rules), started with some core XP practices, then grew to broader Agile cultural foo, then Lean come along, and I also picked up stuff like AgileUX, DevOps, Product Discovery, etc.
In other words, over the years my understanding or perspective about Agile methods evolved as follows:
Back in the good old days, I believed clean code was everything and that software shops had to worship their code, because it was the only true asset they had. Then I worked with some really hep, successful software tycoons and realized how little attention they paid to clean code. In fact, they saw code as a liability and tried to get rid of it as quickly as possible.
This does not apply to every company out there. Some companies desperately need Refactoring and TDD mojo. For some, the technology and implementation solutions change so rapidly, that if they are not innovating and creating the latest, greatest stuff, no matter how clean their code is, they won't survive. Some companies struggle with lack of information to make decisions, while others are choked due to information overload. Some struggle due to poor management, while others lack execution.
I truly enjoy the irony of watching some folks trying the "one-size-fits-all" approach with Agile. There is a huge spectrum of companies all with different sets of pain points, each needing different solutions at different points in time. One size doesn’t fit all!
There is no doubt that over the last decade, Agile has helped me resolve many of the typical challenges we've faced during the delivery of software products. However for me over the last 4 years, the bottleneck has shifted from software delivery to discovering the right product fit, the set of challenges I'm facing cannot be fully solved by applying Agile and Lean thinking.
Over the last decade a lot has changed around us. From Nature of Business, Demand for Distributed Development, Tools and Technologies used, availability of powerful Web and Mobile Platforms to just about everything. Consider the amount of change we’ve seen, I think its very important for our principles and approach to evolve to accommodate this new context.
Couple of years ago, I wrote a blog titled "I’m not Attacking the Agile Manifesto Principles, I’m begging for Improvement." This post captured the essence of some challenges I was facing applying Agile principles at work.
I do think the Lean-Startup community has addressed many of those concerns. I haven’t seen much new come out of the Agile community in the last few years, but when I look at the Lean-Startup community, I feel a breeze of fresh perspective.
At Industrial Logic, we’ve gone ultra-lean and pretty much elimintaed much of the standard Agile practices like backlogs, user stories, iterations, planning meetings, retrospectives, daily standups and in some cases even TDD and Pair Programming. Instead we've been practicing Customer Development, Validated Learning (A/B Testing, Fake Features, …), Continuous Delivery and other Lean-Startup ideas for a couple of years now. So far we are loving it and we do see it working well in our current context. However, like always, there is a need to figure out things as we go along. And that's where it becomes fun.
Few years ago, Startup companies esp. in the Social web and Mobile development space had some interesting ideas around software develpoment built on top of Agile and Lean thinking. Over the last few years the Lean-Start-up community has packaged this knowledge really well. They’ve encapasses all the meaningful Agile and Lean thinking, but at the same time, they have given a whole new dimension to agility and lean.
I believe Lean-Startup will heavily influence the Software community over the next few years.
Following is how I view the Agile Ecosystem today:
The Lean-Startup community is great. However I’m also excited about things happening outside the software development space, that helps me understand the software ecosystem better.
For example, for many years, I’ve know the following about software development:
- Uncertainty is inherent and inevitable in software development processes and products. See this link.
- For a new software products, the software ecosystem will not be completely understood or known until after the system is in use (may be not even after that.)
- Predictability Paradox – The more time we spend planning and estimating software projects upfront does not mean our forecasts will be more accurate. See this link.
- It is impossible to completely specify nor test an interactive system. See this link.
- A problem is not fully understood until after you have developed a solution. See this link.
- If something hurts, doing it more frequently will alleviate the problem
- Throwing away code more frequently, keeps the overall cost down
- Slowing down more often and being aware of things around you (tighter feedback loops), makes you go faster
- Software evolves more rapidly as it approaches chaotic regions
- There is no such thing called Best Practices, its all contextual.
- Usually most successful product bloats up and dies under its own weight
- Kaizen (small, gradual, consistent improvement) can only take you so far. Kaikaku (disruptive change) can help you achieve quantum leaps of improvement.
I was able to kind-of explain why I noticed these behavior in software development but could never really explain the underlying principles. Agile and Lean methods did help me address a lot of these challenges. However I was still unable to explain the underlying principles. Until I stumbled across the following two concepts:
- Cognitive Complexity: I was lucky to attend a keynote from Dave Snowden at Limerick, Ireland in 2008. He got me hooked onto Complex Adaptive Systems, especially his Cynefin framework. He has some very neat ideas that the software community can benefit from. If you see many organizations know that they need to change, but are afraid to make the move. They are looking for fail-safe approaches to deal with change. Dave explains the core problems with this approach and how Safe Fail Experimentation can really help organizations. Many organizations struggle with measuring the wrong thing and forcing people to game the system. Dave’s ideas on Retrospective Coherence or Fundamental Attribution Error and Weak Signals Detection can be extremely helpful to address some of these concerns. Also I particularly like (his take on) Narrative.
- Affective Forecasting Psychology: I'm blown away by the anecdotes and studies presented in Daniel Gilbert's book Stumbling on Happiness. It has helped me understand the "filling-in" trick the mind plays while projecting ourselves into the future (or past) and its limitations. I've just started my journey into this subject and there is a long road ahead.
I’m going to leave you with an old Sanskrit sloka as some food for thought.
No Matter how fine you chop the Sandalwood, it does not give up its fragrance,
However old a businessman gets, he never gives up his profit,
In spite of crushing the sugarcane in a machine, it does not turn bitter,
However knowledgeable you become, don’t cease to learn.
About the Author
Naresh Jain is Director, Asia Operations & Agile Coach @ Industrial Logic. From Organizational Transformation to enhanced Developer productivity, Naresh helps organizations embrace, scale and sustain essential Agile and Lean thinking. Currently, Naresh spends a large portion of his time evangelizing and building Industrial Logic's eLearning platform. His current leadership role demands that he performs various critical roles including understanding market needs, prioritizing & designing solutions, collaborating on marketing & sales strategy, hands-on development (coding & testing) and finally deploying & managing the production servers.
Community comments
A preliminary comment
by Adam Nemeth,
A more elaborate comment
by Adam Nemeth,
You are not alone.
by Chris Matts,
Dealing with Change in an Evolving Contextual World
by Sandy Klausner,
Re: Dealing with Change in an Evolving Contextual World
by Naresh Jain,
Re: Dealing with Change in an Evolving Contextual World
by Adam Nemeth,
Agile & complexity
by Diana Larsen,
Re: Agile & complexity
by Naresh Jain,
A preliminary comment
by Adam Nemeth,
Your message is awaiting moderation. Thank you for participating in the discussion.
I feel I'll have to really read through all the articles here, and have a complete answer, as someone who cites references deserves answers which does such also, and currently I don't have my full library here (namely, I miss some Dijsktra and Boochs papers along with a few others), and I just skimmed through most of the linked articles.
But:
If I undestand well, Industrial Logic basically eliminated everything what is related to engineering, including clean code (you throw it away as fast as you can), any kind of supporting documentation (like, backlog) or conventions (like, standard meetings)
It's quite strange and against my prejudices and stereotypes to see this company outside bankruptcy, to be honest.
I quite agree that real-size software systems cannot be undestand fully, no problem with that.
Uncertainity is there in human thought systems, that's also agreed, period.
But what I'd like to challenge the author for: I believe a software system itself is fully specified. I mean, if it's running, it's running at a fully specified state, even if that specification is incorrect, it's not what you want, unclean, dirty, buggy, etc.
Every time a software system gets an input, given a context its output can be specified.
Now here comes the question: does software development have anything to do with going from the totally uncertain world of thought to the totally certain world of machines?
Also, since 1968, I think we can agree that some trillion lines of code was written and deployed, with increasing numbers. Also, there may have been never a point in history, where so much code was in use as it is today, as this is probably still on an expanding scale.
I think we can agree, that software is basically mass-produced, even if every software is a bit unique, there are a lot of softwares, and right now, thousands of software projects are running.
Do you claim, that there are no patterns in this mass?
Do you claim, that there are no solutions, which given a quite broad context are generally the best way to achieve the desired results?
Do you claim, there are no conceptual tools which would help humans to go from the uncertain world of thinking into the pretty much exact world of binary data?
I don't claim that software doesn't need creativity. But what software engineering is built on top of, I believe, is that you have to nail some parts of the problem, and you have to nail them down in a top-down, ever-detailing way, as in the end, the system will be fully detailed, wether you like it or not.
Also, although this was lost with the agile movement, but the old world still had it, we had ideas on what to nail down first, this was the long-lost use case analysis, where people tried to nail it down, what the system (or subsystem) will be used for, and saw everything as a consequence of this, or rather, some kind of support to solve a real-world problem. Even in early XP, they tried to deal with classes as units of problem solving (CRC cards)
This is all gone now, and we've gone from solvers of real-world problems to creative artists who create fuzzy things out of fuzzy things, namely, to convert crazy people's crazy ideas to crazy programs as fast-and-dirty as possible, with some kind of proud.
Do you think it's the way to do things?
(I really miss the old engineer world, where like every other engineers, architects and civil engineers included, we were trying to build structural support for communities of people; I mean, if you need tapwater, there's a huge, complex task to do in order to achieve this, which you don't want to know about; also, if you want to feel yourself really well in your home, or in your office, there are a miriad of questions to answer, but all this stems from the assumptions that there's actually a right set of questions to ask, a right way to do things, which people could be trained for, which could be looked up in textbooks to make sure you asked everything you really needed.
Do you think this isn't needed, or isn't possible for software? Do you think, that we shouldn't try to come up with some general questions which should be answered most of the time? Do you think we shouldn't try to find reassuring answers to questions, like, how long will it take? What will this cost for me about? Do you think we shouldn't make lists of what to consider? That we shouldn't have some thoposes, like, security or stability?
Of course, I agree, the actual set of questions at the end will always differ, but I hope there's a way to actually learn what to ask; it may need some creativity, but I hope there's still a reassuring way, that will help us lead those thousands of customers of those thousands of projects, and most importantly, their billions of end-users, without some kind of mystic "just do it from feeling" kind of way; not that I wouldn't listen to my gut normally, but depending solely on that while affecting billions of people's life is quite a frightening picture for me, if you ask.))
You are not alone.
by Chris Matts,
Your message is awaiting moderation. Thank you for participating in the discussion.
Naresh
I know a number of people who feel the same way. Most of them use Agile tools rather than sell them. After you are proficient with a tool there is no longer a need to use it all the while. You can use it when it is appropriate in context. The context is determined by the business investor and what they value the most.
I get the feeling that Agile is now a destination rather than a journey that generates new tools. Some new tools make old tools obsolete (even Agile tools).
Time to call time on Agile and get back to "Learning by Doing".
Chris
Dealing with Change in an Evolving Contextual World
by Sandy Klausner,
Your message is awaiting moderation. Thank you for participating in the discussion.
>Throwing away code more frequently, keeps the overall cost down.
But, what really needs to be thrown out is the basic notion of source code, either interpreted or compiled! We really have not advanced very far in managing complex software systems since Cobol and Fortran. Name even one programming language that inherently captures 'context.' But, you must be saying, this guy is nuts and besides what could we possibly replace this medium with? Well, this is where true innovation comes into play, something that gets most developers very nervous, since this questions the very foundation of one's professional expertise.
Agile & complexity
by Diana Larsen,
Your message is awaiting moderation. Thank you for participating in the discussion.
Hi Naresh,
Out on the leading edge, our focus does need to shift to how to develop, deliver, sustain, and use software in a complex world. I'm also applying many of the ideas from Lean Start-up and the field of complexity science as an extension of my toolbox of Agile mindsets and practices. In addition to Snowden's methods, another useful thread of models and methods comes from Human System Dynamics (www.hsdinstitute.org) and the work of Glenda Eoyang. Applying her work on chaos, complexity, complex adaptive systems, and types of change has given me new insights into the software "ecosystem."
Thanks for this perceptive article. :)
Diana
Re: Agile & complexity
by Naresh Jain,
Your message is awaiting moderation. Thank you for participating in the discussion.
Thanks for your feedback Diana. I've been following HSD and Glenda for some time. They certainly provide great perspective into CAS. IMHO Snowden is a good starting point.
Re: Dealing with Change in an Evolving Contextual World
by Naresh Jain,
Your message is awaiting moderation. Thank you for participating in the discussion.
Very true Sandy.
Re: Dealing with Change in an Evolving Contextual World
by Adam Nemeth,
Your message is awaiting moderation. Thank you for participating in the discussion.
Have you ever heard of smalltalk?
No sourcecode.
That's why all that stuff you see here were invented - XP was a smalltalk principle, MVC was a smalltalk principle. It was a right-click, edit source solution. It's not in use anymore.
A more elaborate comment
by Adam Nemeth,
Your message is awaiting moderation. Thank you for participating in the discussion.
After I've read all the documents, especially the critique of Agile you wrote, and looking at the IL website reassured me that the Earth is still on its edges, although some Agile craziness sometimes makes me feel otherwise.
In general: I feel a software is a very ordered piece of world; it may be that later we make them unreliable, but today, we need computers in order to trust them with jobs where we need some certainity and predictability; otherwise, we see it as a computational failure.
Software development currently is the way between the uncertain human thoughts and wishes (from customers) to predictable, certain computer thoughts (software). It might change, or might not: why would we employ a computer to do something uncertain, yet intelligent, when we have perfectly capable humans to do that job?
Also, I find the american model of problem-solving, namely that we take problems as parts perfectly inadequate: it's rather about resolution and sides for me. I feel that no problem is wicked to that sense: while the whole is always more than its parts, a whole shall be a whole on every resolution.
Hence, predictability also comes in place: as long as the side we've chosen is correct, and no other laws come in, scaling out works: it's not certain, yet if 10 people are hanging on a private beta product day-and-night, you have good bets that at least 100 will like it when it gets published, unless you mess up the part which was liked at the first place.
The interactive system principle vs predictability, while may be true in edge cases, in practice it's not the expectation against computers, hence I don't find it applicable at all (albeit Petri nets and other formal methods offer pretty good models at handling such in practical software engineering)
As for repetition, I don't think that what hurts shall be repeated: I don't think that software engineering has many places for repetition. Sometimes your body responds with becoming stronger (like that with weight-lifting or running), sometimes albeit it does alievate the symptoms the world becomes much worse (like permanent eyesight sharpness loss when watching monitor too much)
As for best practices, I hope there exists an induction-based proof that this is not true: once we find a basic context (namely, there's a problem to be solved with computers), we can always create contexts based on detailing the problem, and for that given context, there exists a best solution (these are the problem-and-solution patterns and pattern languages by Alexander et al)
So, in general I had problems with the assumptions which I see them as either false, or falsely applied.