Agile's Teenage Crisis?
This article is part of the Agile Manifesto 10th Anniversary series that is being published on InfoQ.
The “elephant in the room” is a metaphor for the behaviour of people who deliberately ignore an impending issue. They are fully aware of some major issue that really must be tackled or decided upon, but everyone keeps busy tackling other, often small items, ignoring the big issue, pretending it does not exist, hoping maybe that it will vanish by magic or that someone else will take care of it; that one day the elephant will have left the room.
Alistair Cockburn organized a celebration of the10 years of the agile manifesto in Snowbird, Utah on February 12, 2011, gathering some 30+ people who’ve been involved with that community at the original meeting and since. It was a very interesting discussion, and I am not purporting to report here the main results or findings of this great event. After covering the walls with a couple of hundred issues cards, David Anderson noted that there was “an elephant in the room”, a topic that few are willing to debate in the pen, namely the Agile Alliance (its role, mission, accomplishments, etc.). After the lunch, a small group gathered and identified a few other such elephants in the room, that is, other topics that the agile community is not really willing to tackle for a variety of reasons. We ended up with a long list of about 20 such “undiscussable” topics (or at least not discussable in the open).
A herd of elephants
Here’s the raw herd of elephants, with an additional sentence or two to explain what the title is about, based on the best of my recollection (see photograph at bottom):
1. Commercial interests censoring failures
The key players in this community have a direct financial interest and have fear that any negative info would be amplified, distorted, and turned against them
2. Pretending agile is not a business
see 1. above
3. Failure to dampen negative behaviours
We do not face, then analyse the failures and limitations of our assertions, claims, practices (see above)
4. Context and Contextual applicability (of practices)
We fail to describe the context in which some practice works, or does not work.
5. Context gets in the way of dogma
By evacuating context, we constantly return to dogmas, bigotry, and claims of universal applicability.
… is a bit at the root of “elephant in the room” (we actually know all this).
A more systematic and thorough acknowledgment that organizational politics plays a big role in the adoption of agile practices.
… in the agile and lean community itself, preventing a more systematic organization of the body of knowledge
… as a defense (against failure) we limit our message to the best… and blame failure on the “unenlightened” others…
10. Agile alliance
Divergence of views as to what it role has been, was supposed to be, will be…
11. Certification (the “zombie elephant”)
This massive elephant was reported dead a few times, but seems to reappear… Effective tool to assess maturity for individuals and organization, or commercial ploy from trainers and consultants?
12. Abdicating responsibility for product success to others
In particular, much of responsibility seems to be pushed to product owners, business analysts, users, etc.
13. Business value
The notion of value and in particular business value is mentioned everywhere, all the time, but never clearly defined, or confused with (development) cost, or pushed onto others (such as product owner) to resolve.
14. Managers and management are bad
… as an instance of pushing any failure to others?
There is often a tacit understanding that we (software developers) are all the same, and the agile movement has not fully integrated the concept of culture, both at the organization level and at the national level, and its subtle interaction with practices, an integral part of context.
16. Role of architecture and design
… depending on context; they are still often equated to BUFD, and rapidly brushed aside with claims of YAGNI, or “we’ll refactor it later”
17. Self-organizing teams
Maybe as a consequence of the axiom that management is bad, and also pushing aside the concept of software development governance.
18. Scaling naïveté (e.g., scrum of scrum)
With a bit of imagination all agile practices scale up, and if this does not work it is because you have not tried hard enough
19. Technical debt
While agile practices could help control technical debt, they are also often at the root, the cause of massive technical debt.
20. Effective ways of discovering information without writing source code
These last few elephants (#16-20) are practices where a large level of variability exists across software development organizations, and little in-depth analysis has occurred, in particular on how they tie to context.
Grouping the elephants
This is quite a large herd of elephants. So after a few months, here is my view on the elephants we have in the agile community room.
1. The agile alliance elephant
I’ll put this one aside; the first that entered the room in Snowbird, since I have nothing to say about it. I noted a small group of former Agile Alliance board members tackled this elephant late in the day. Maybe it is just one instance of the “Community leadership” elephant, which would include other organizations… Is there a scrum alliance elephant?
2. Failures and limitations of agile practices, & context
While the community has been good at amplifying what works, it often has not been good at dampening what did not work, analyzing why it did not work, or under which conditions it did not work (context!). There is in some cases a certain level of hypocrisy (many interested parties know this, some are willing to acknowledge it in private conversations, but then… pretend it does not exist or return to the official dogma in more visible venues).
This elephant has several causes, themselves listed above as elephants (i.e., “undiscussables”):
A) Commercial interests
Many of the key players in this community have a direct financial interest in selling something agile (tool, consulting, training, new idea), and there is this latent fear that potential buyers would be detracted by any hint of negativity, that any bad news would get unduly amplified, that it would in the end turn against them or against a whole community.
In most instances, when practices are described too little of the actual context is described, leaving an impression of very wide applicability; even if this wide applicability is not actually claimed. Sometimes this is just pure dogmatism, or bigotry.
C) No obvious way to make progress based on failure
Unlike other disciplines, like medicine for instance, there is very little reporting of failures or limitations in software, and no clear venue or even templates or exemplars of such reporting. And again the fear that this would reflect badly on the person reporting, or the “victim” or “culprit” organization. A good template for such report would include a detailed description of the context.
D) Limited objective evidence
There are still too few people gathering objective, significant, scientific evidence on our practices, except for a very few that are easy to instrument (pair programming, for example). This is probably exacerbated by some of the academic imperative to publish: doing extensive qualitative studies on development practices is not an easy route to a journal paper or master’s thesis.
E) Cognitive biases and Reasoning fallacies
Reasoning fallacies such as undue generalization (“it worked in two cases, therefore it will in all cases”), and cognitive biases: anchoring, golden hammer, cargo cult, etc. are rife in the agile world. Others reasoning fallacies I frequently encounter include: non sequitur (no logical implication), and post ergo propter ergo (correlation implies causation, or “guilt by association”), etc.
The two smaller elephants: elitism and hypocrisy are probably in the shadow of this main one, and the anarchism of our community does not help to organize a body of knowledge.
The practices in question for which there is little evidence or understanding of the role in various contexts are also some of the smaller elephants in the list above, often undiscussable (beyond loud rhetoric and posturing):
- thorough articulation of the concept of business value,
- the role of architecture and design, technical debt,
- self-organizing teams,
- writing code as a priority, etc.
3. Politics and culture elephants
These two elephants would require way more investigation. I do have some personal opinions on the impact of both national and organizational culture on agile practices, and some investigation has taken place in the Global Software Engineering community over the last 6-7 years. But I know nothing about politics, neither inside our community nor as an impediment in organizational change that could be addressed explicitly by our community. Room for more work.
The agile movement is in some ways a bit like a teenager: very self-conscious, checking constantly its appearance in a mirror, accepting few criticisms, only interested in being with its peers, rejecting en bloc all wisdom from the past, just because it is from the past, adopting fads and new jargon, at times cocky and arrogant. But I have no doubts that it will mature further, become more open to the outside world, more reflective, and also therefore more effective. I got a hint of what’s up for the future at the Snowbird meeting, which produced much more than our side meeting on elephants.
Note: this was originally published on February 13th, 2011 here.
About the Author
Philippe Kruchten is professor of software engineering in the department of Electrical and Computer Engineering of the University of British Columbia, in Vancouver Canada. He holds an NSERC chair in design engineering. He joined UBC in 2004 after a 30-year career in industry, where he worked mostly in with large software-intensive systems design, in the domains of telecommunication, defense, aerospace and transportation. Some of his experience is embodied in the Rational Unified Process (RUP) whose development he directed from 1996 until 2003, when Rational Software was bought by IBM. RUP includes an architectural design method, known as “RUP 4+1 views”. His current research interests still reside mostly with software architecture, and in particular architectural decisions and the decision process, as well as agile software engineering processes. He is a founding member of IFIP WG2.10 Software Architecture. Kruchten received his mechanical engineering diploma from Ecole Centrale de Lyon, and his doctorate degree in Information Systems from Ecole Nationale Supérieure des Télécommunications , Paris. He is a member of IEEE, ACM and AIS, and a Professional Engineer in British Columbia.
Is it Blasphemous to Criticize Agile?
Agile, like every thing, has its own flaws. It is important to acknowledge them. One of the point which needs to be debated is the agile recommendation of shunning upfront design. "...While agile practices could help control technical debt, they are also often at the root, the cause of massive technical debt..."
Using software development experiments to establish contextual applicabilit
(the links in the article to Jeffries blog are broken so here they are)
Norivg took the design driven approach while Jeffries tried to TDD his way to a Sudoku solver. Based on the results it looks like TDD is not a good approach for developing this type of algorithm.
This is the kind of experiment we need to have a lot more of in software engineering to understand which approaches work and which don't and in what context they work. If we had the software engineering equivalent of the Underwriters Laboratories that would go a long ways to solving the contextual applicability problem and put the software industry on more of an engineering footing than it is now. We would also need to develop some standard protocols for these experiments (i.e. analogous to the protocols for clinical trials in medicine).
The responses to the article above tended to focus on the individuals involved, but I believe this was a valuable experiment. We use a lot of metaphors in software development and I'd like to see the Underwriters Laboratory as a metaphor for how we should test new software develoment methods.
What do others think about this experimental approach to evaluating methodologies?
Agile's Scalability and Stubborn Adhesion to Dogma
Entertaining and Self-Serving
Yes, there are some issues, but I'm not sure I want to get lectures from the fathers of RUP.
Failures and limitations of agile practices, & context
Finding the right question - can we afford not to use Agile thinking?
When is the last time you've seen a customer happy to wait 12-24 months or more for a product increment? We simply cannot afford to work in large batches anymore. We cannot afford the massive waste of extended document-intensive hand-offs and the accumulated tide of defect injections they induce. We must shorten cycle times, reducing waste, absurdity and overburden - in short, we must establish an industry-wide culture of continuous improvement. That calls for more cross-functional collaboration, work in smaller batches and striving for flow.
Some of the elephants raised are impediments that have known and effective countermeasures already identified by Lean thinkers. For instance, elephants 14 (Manager and Management are bad), 15 (Culture) and 17 (Self-organizing teams) are closely related. Labelling people or management in general as "bad" is worse than unhelpful - it ignores the fact that the problem lies with the process, not the people. Managers can hardly ignore the (uninspired) incentive structures devised for them, and cannot "fight" the process alone. Continuous improvement cannot be sustained if people are treated as fungible "resources". Leadership support for a culture of respect for people and continuous improvement must underpin the practice of software-intensive systems engineering if it is to be performed with less waste and yield higher quality products.
Culture (the accumulated expectations and habits of organizational hierarchies) has a deep influence on the way in which processes are shaped. If managers do not take an interest in the work of their Teams, and instead attempt to manage them from afar through point "metrics", gaming of the process will inevitably ensue, with lots of efforts aimed at local optimization to the detriment of global efficiency and value flow. Queueing Theory, the Theory of Constraints and its combined practice with Lean thinking have lots to offer as inspiration.
Self-organizing teams are a natural response in such a harsh environment if any success is to be achieved. However, the practice of self-organization should not be construed to mean that leadership is useless. True, some folks do marvellous impressions of Dilbert's pointy-haired boss at times. On the other hand, declaring all management bad is no cleverer. There is no greater expression of respect for people than challenging people to improve their own work process. Leaders should consider taking a close interest in the work of their Teams. Asking tough questions, requiring analisys of root causes based on facts rather than subjective opinions, insisting on experiments to prove viable countermeasures. Behaving in this way, leaders effectively say to their people "since you're closest to the value-adding work, I trust you to be best placed to figure out what works best at this point in time and prove it". And the people doing the work can value the trust and responsibility they are given, and appreciate the insight that leaders can bring in the form of a wider organizational perspective, facilitating communities of practice that look after the welfare of "horizontal" value networks crossing "vertical" silos.
I suggest that inspiring such changes in mindset and habits to happen from Boards of Directors to Teams of practitioners and everywhere in between is our task as a community of Agilists. We have at least a few more decades of fun ahead.
Which Agile has the crisis?
- Because it's time to stop wasting our lives creating useless artifacts just because someone remote in space or time thought it was a good idea - instead, we can question and prove the value of our work, inspecting and adapting to suit.
- Because we can imagine better ways to streamline the flow of value through networks, with each work process optimally suited to the people involved at that point in time, with less waste, overburden and absurdity.
- Because it's time to expect deep respect and appreciation for mastery of engineering and other professional habits as well as inspired servant leadership.
- Because Agile is not a practice to be maximized for its own sake - instead, when done well, it is a practice that balances its benefits with the investment required to achieve them.
I suggest that it isn't Agile (the vision) that has the teenage crisis, but rather Agile (the community) that suffers from the symptoms. I suspect that may be due (at least in part) to the fact that in our current society it is still far simpler to take a zealous or dogmatic view than to practice the healthy skepticism of the scientific method. Consider how much intolerance and rudeness (teenage behavior hallmarks) are still evident in various business and social contexts.
Also, despite more than a decade of practice, as a community, we have relatively few examples of long-lived, masterful, tight-knit Teams that purposefully practice continuous improvement - for instance, most managers still disband teams when projects finish and then scramble to re-form teams when new projects kick-off, rather than keeping small cross-functional Teams intact and giving them new work as needed. Also, engineers are routinely not respected and valued enough, so natural career aspirations force them to abandon hands-on engineering work as soon as possible. Therefore, inertial cultural pressures force us to show up on the field of battle against complexity with mostly fledgling Teams, which will naturally behave "young", particularly without the steadying influence of a wise role model (think of a seasoned chief engineer or Agile Coach).
Once we start seeing more Teams and leaders behaving wisely, with good emotional intelligence and acuity in addition to engineering excellence, Agile (the community) will have overcome some of its natural growing pains. For the time being, let's focus our attention to perfecting habits that embody Agile (the vision), and do so with elegance and compassion for our fellows.
Re: Entertaining and Self-Serving
Anthony DaSilva Jr
A good list
As I see it, the community is divided. The majority of people who self-describe as members of this community are somewhat stuck on what we now call "first-generation" agile ideas and practices. The do indeed come across as somewhat dogmatic, and they do at times appear to be blind to context. There is certainly no doubt that "agile" is a business, and that many consultants, trainers, and software vendors have a vested interest in first-generation "agile" practices. This pattern is hardly unique to "agile," though. RUP is a commercial product, for example.
The other segment of the community has taken the lessons learned from the agile movement and is carrying them forward as they continue to pursue improvement in software development and delivery methods. These people are probably less susceptible to the problems Philippe mentions.
The only caution I might offer to readers who are eager to dispense with "agile" is this: Remember the old saying about throwing away the baby with the bathwater. A lot of useful improvements have come about because of the agile movement. If you discard them, you will be returning to the 1980s. If you are old enough to remember what it was like to work in IT in those days, you will not find the prospect a pleasant one.
Most of Philippe's observations are valid, or at least understandable from a certain point of view. There is one small thing, though: I would be interested to learn how "agile" practices, done properly, "cause" technical debt.
Now, in defense of "agile," I will say that there is a certain mindset associated with it. This constant demand that agile practitioners should focus on "failure" may actually reflect one of the very deep, fundamental problems with traditional IT thinking, and one of the cultural issues that the agile movement sought to correct. IMHO we should understand that the outcomes we achieve are a direct result of the choices we make. When outcomes are not as we had hoped, we need to think about how they came about so that we can change our choices the next time around. These two words, "success" and "failure," are just as Kipling describes them: Impostors. They are not reality. When we say "failure," we are finished. There is nothing more to do. Failure is final. An outcome that doesn't deliver all the benefits we had anticipated is quite a different matter. We can learn from it and improve our work. So, rather than insisting that people talk about "failure," I prefer to encourage people to keep thinking about positive outcomes, and keep working toward continuous improvement. Call that "agile" if you wish, or call it something else, if you're determined to bash "agile." The buzzword is pretty much used up by now anyway, so bash away...but forget the lessons at your own cost.
Re: A good list
I agree, before saying anything is a failure, observe everything that was accomplished and acknowledge what wasn't deliver and think about how to improve and learn from your mistakes.
Agile is more than just a software toolbox, guide, it's a culture that helps us bring our work industry into a better human focused process that will continue to adapt with our reality and context.
In all industry the new generation of workers want a new way of working, more flexibility, more human relation (at least here in Canada...) So we need to adopt a new paradigm to have a better way to address our working life.
Re: Agile's Scalability and Stubborn Adhesion to Dogma
So maybe some learn that Scrum shoulb be implemented like this but basically it's just a guide of what we consider a good way and we should be awar of where we are and put some small increment of new practices to bring ourselves more near of what we consider be the good way and this way will obviously vary depending on the context.