Why Agile Didn’t Work
As an agile coach, one of the questions I am often asked is: “we implemented agile, but why didn’t it work?”
The people who ask this question are truly puzzled. I’ve met teams who went through a lot of trouble to re-organize themselves, re-assign responsibilities, and re-define processes; and yet they were dismayed that their productivity and morale were even lower than before.
Each organization has its own unique dysfunctions. As a coach, I have to immerse myself deeply into teams, and observe them closely to understand how they work. But, despite their uniqueness, the same patterns frequently emerge. In almost every case, these teams decided to adopt Agile because they met real life issues, such as low productivity or low quality, and they had hoped that Agile would miraculously lift them up. What they didn’t realize is that Agile is a systematic approach to manage software. Agile upholds some lofty goals, such as “satisfying customers by satisfying their constant changing requirements”. Managers are typically elated to see these goals, and can’t wait to jump on the Agile bandwagon. The problem with Agile is that it comes with a lot of hassle: daily stand up meetings, planning meetings, and retrospective meetings. At first glance, the sight of a group of people hovering around a whiteboard, might make naïve managers think it has worked; but what they fail to realize is that without significant managerial and technical support, Agile won’t survive, and even worse, teams will secretly hate that hassle and hope that things will return to the way they were before.
One team I worked with this year fell into the trap described above. It was a big team, divided into 7 scrum teams. You can imagine how noisy the office was every morning, with scrum meetings happening roughly at the same time, and a scrum of scrum meeting happening afterwards. Management was thrilled; after all, they had paid good money to an Agile coach to implement Agile.
Things started to turn south pretty soon. I attended teams’ scrum meetings for weeks and noticed in particular that one team was continuously blocked by the same technical issues for several days in a row. The team couldn’t move forward, so they started to work on something else instead; which resulted in a lot of unfinished code that couldn’t be tested and demonstrated.
In Agile, the scrum master is supposed to remove obstacles. When asked, however, this team’s scrum master was pretty frustrated. He said there were only a couple of people in the team that understood the relevant technical details, but they were not in his scrum team. They were assigned to work on another feature by his manager.
These are typical problems in software teams:
- Agile advocates that everyone can pick up any task from the backlog, but in reality, some obscure technologies are understood by only a couple of people. It is hard to motivate others to familiarize themselves with these technologies, especially when they are outdated. Even if people are motivated, when facing a large backlog, it is hard to resist the urge to work on something else to make the backlog look smaller.
- The Agile scrum master is supposed to move mountains to keep the team going without distraction. In reality, the scrum master often lacks the power to do so. This power usually resides with managers, and as we all know, power is addictive and hard to relinquish. In practice, I do not urge organizations to change manager roles if it means resistance from managers. If a manager fits the bill for an agile scrum master, it is ok for this manager to take on some of the responsibilities of a scrum master: a manager usually is more experienced in acquiring resources and handling conflicts, especially conflicts with other parties.
But this team had bigger troubles. The team manager started a separate and private feature on the side – there was no scrum team assigned to that side feature, and I couldn’t establish its progress. After probing further, I discovered that the RnD team and the product manager team didn’t trust each other; the product manager team had “ordered” the RnD team to do one set of things, but the RnD team had reserved some resources to work on something else that they thought stood out more.
I sat down with the RnD management team, and drew a cause-effect diagram on the whiteboard:
(Click on the image to enlarge it)
The cause-effect diagram is a powerful tool. While the managers were complaining that Agile didn’t work for them, the diagram clearly showed the root causes lay somewhere deeper. The diagram also had loops, which showed that, if the root causes were not addressed, a vicious cycle would continue.
This team had the full support of management, but still not everything went well. In particular, continuous integration always failed. The team had set up a big screen displaying the integration status, right at the front of the office so everyone could see it when entering or leaving. The screen was green on average 1-2 hours a day. This, of course, set back the team’s progress a great deal. There was a rule that code could not be checked in unless the screen was green, but since it failed so often, some people just sneaked in their code.
We like to think that software development is very different from traditional manufacturing, because software development is more “brainy”, while workers in a traditional manufacturing environment just mindlessly screw on nuts (Modern Times by Charlie Chaplin comes to mind). In reality, software development also has an assembly line, except that this software assembly line incorporates a lot of iterations that serve to improve software quality. But, just like a factory assembly line, if one step slows down, things will pile up and the whole line will slow down.
For this team, the assembly line at the feature level looks like this:
(Click on the image to enlarge it)
This assembly line has many feedback loops. The shorter the feedback loop, the better, because the cost of correcting the mistake is smaller. The feedback loops from “demo” to the “requirement” and “coding” steps are too long; ideally the feedback loop should be from the “Dev coding and local test” step. However, bugs will always get missed, so we rely on automation and QA test to provide feedback. But in this team, the build-deploy-automation process was in serious trouble, and further complicated by the fact that some people were disregarding the rule and checking in code anyway, clogging the assembly line even more.
I made clear the concept of the assembly line to the team, and they quickly came up with solutions to smooth the assembly line, such as strengthening the code review and enforcing the no check-in rule. However, on the subject of test automation, opinions diverged. Some people believed that the current test automation was completely unmaintainable: the code was written by contractors who were long gone, and more time was spent on fixing bugs in the test code than fixing bugs caught by the test code. Others said even though the test automation was bad, it was better than nothing, and the cost to rewrite it would be too high for QA to take on.
I asked the team to perform a quick calculation to determine the merit of automation:
Bugs caught by automation/all regression bugs
It turned out that some automation was pretty good, especially some unit and component tests, but the result of UI end-to-end automation was miserable. The conclusion was clear: there was no point investing so much effort to work on something that had so little benefit.
I also pointed out that writing and maintaining automation tests was not just QA’s responsibility. UI end-to-end test code is completely dependent on production code. In a service oriented architecture, services have well-defined and often negotiated APIs, if one service wants to change its APIs, it will (or should) notify its dependent services so they can make the corresponding changes. UI test code is at the complete mercy of production code: if so much as the ID or name of a UI element changes in the production code, UI tests might fail.
To make test automation robust and flexible, the automation framework has to be well designed. Hiring temporary contractors is no way to ensure this, and having QA write automation architecture is probably not good either, depending on the technical skills and types of QA. For this team, it was obvious that most QA engineers were functional testers and did not possess the advanced architectural skills required to architect a framework.
It was therefore decided that Dev tech leaders would write the automation framework, and that Dev and QA would write and maintain the UI automation tests together. This agreement included rules such as Dev not changing the ID or name of UI elements at will.
You might think that all these happened in one single breezy meeting; in truth, it took a lot of managerial support to make happen, and it took nearly three months to write an automation framework that was fast, stable, and easy to write tests in – that is how real life issues are solved!
Agile is a systematic approach of managing software
We need to go back to the origin to find out the essence of Agile. In 2001, 17 software gurus brought up the Agile Manifesto, which itself doesn’t prescribe what to do:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
They also came up with 12 principles, a few of which do prescribe what to do (for example, convey information through face-to-face conversation, business people and developers work together), but most of which are still about values. And if we closely analyze these 12 principles, we can see they actually form a pyramid:
On the top is the lofty goal of “satisfying customers by satisfying their constant changing requirements”. We achieve this goal by “delivering working software frequently”. To deliver working software, though, requires significant technical and managerial support. Ensuring that changing requirements do not break the system and slow down development is foremost a technical issue: how to design the system in a way that is flexible and how to create automation that ensures changes do not break things. To foster advanced technical skills in teams, teams have to be motivated to learn from their mistakes and to develop themselves.
(By the way, I choose to interpret the simplicity referred to in principle 10 as Jobs did to emphasize the importance of usability: “Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple.”)
The transition to Agile is not an easy journey. If it is not working, take a closer look. It might have exposed some fundamental issues in your team. Go to the whiteboard, figure out what is slowing your team down, use lean tools such as cause-effect diagram and 5 whys to get to the root causes, fix them, and you will reap the full benefits of Agile.
About the Author
Chen Ping lives in Shanghai, China and graduated with a Masters Degree in Computer Science in 2005. Since then she has worked for Lucent and Morgan Stanley. Currently she is working for HP as a Development manager. Outside of work, she likes to study Chinese medicine. She blogs here.
No Agile here
No breakdown of most difficult/uncertain 10% of project to start with rest of the project in simple outline storyboarding form. This first step is critical because it sets the certainty of the inputs' (coders, domain, management, QA..) ability to complete the future iterative steps. These folks jumped in and starting Scrumming without a real deadline except for the silly green screen that most people simply ignored.
I love that the production code is being tested by QA at the end of project. This is the Niagara Falls of Waterfall. The coders should be testing their Contracts (Address, Bindings and Contracts - ABCs) while performing their coding (old school unit testing), the group should be testing the integration as the Scrum meetings and the total iterative sub-process shouldn't be lasting more than 3-6 weeks.
The full support of management is for the lofty concept of "...“satisfying customers by satisfying their constant changing requirements” but this goal is possible only "... by significant technical and managerial support. [that] design the system in a way that is flexible and [anticipates] change ..." This is what is missing in Agile as a buzzword for unplanned and un-architect driven software development.
A scrum master should never be left alone to remove impediments. Ideally he/she should be fully supported by a servant leader/manager.
In transformations or agile implementations, often the management will turn out to be the main impediment. As coaches we must spend time educating and engaging these managers and showing whats in it for them. Of course, if they decide that there is little or nothing in it for them, they will not support or engage. Your cause effect discussion was important in this regard.
Personally, I find giving managers an introduction to systems thinking and using the diagrammatic approach useful when investigating impediments with managers (who are engaged or interested ...).
Agile, the real manifesto = Job Security for developers
I am surprised the Quick and Dirty development has become a fad and it explains why many deployed software today are inefficient, incomplete and have so many bugs. This explains why Firefox is so inefficient and slow. This explains why IE is inefficient and virtually dead (discontinued). This explains why so many business software are very inefficient and riddled with bugs with the most simple bugs that are easy to fix but never fixed because these bug fixes are low on the product backlog priority.
All Agile does is create job security for the developers as firefighters to serve the impatient user who cannot make up his/her mind. Because the system is designed and developed in piecemeal, the number of bugs and incomplete features/functions escalate.
Take a survey of users and you will find out how many users are very frustrated with the massive waste of their time when a new or new release software system is deployed. They are having to work around the bugs, inefficiencies and incomplete features/functions of the system.
Gone are the days when a deployed system is bug-free, more efficient than the old system and have the complete all-working features/functions as originally required.
Re: Agile, the real manifesto = Job Security for developers
Thanks.Keep writing please.
I also like your other articles in Infoq, especially "The Original Sin of Software Metrics".
Re: Agile, the real manifesto = Job Security for developers
If you have ever read the Agile Manifesto and the founders' first writings, you'd notice that they are more talking about "quality" and "value" than "Quick & Dirty". Quick for sure, and Agile is about craft, not dirtiness and trying to put it everywhere without thinking.
This is the point indeed. In an Agile Transformation, a manager wants "one top success" so he can climb the hierarchical ladders or get a raise somewhere else. This way, he does not respect -if he ever read it- the first value of the Manifesto People and interactions over process and tools. If you want Agile to succeed (what for, I should ask?) you have to take care of the people. You can try whatever you want, but computer can't develop software all by itself. Matter 1 to solve: involve people at each level and make a clear alignment of what they want. This is the hard part.
So, Agile is more of a mindset, and the devs I have seen or coached in using or behaving in a Agile way, i.e. focusing on delivering Value to the customer, being proud of what they do, usually confronted with their boss wanting to control the costs.
And let it be clear, the boss genuinely like to control cost for their pleasure, but because it's how they are evaluated. This is common strategy by the way: you use a magic wand for six months, say Agile, for a very important project, and you ask in the same time to release one other constraint. You achieve a great deal, so the conclusion is that your magic wand is working. In a scientifically point of view, this is a fraudulent reasoning: as you have changed at least two factors in the experiment, which one brings the effect?
If I get back to devs, some of us could be happy to play the firefighters, and some are really, especially if they do that for a long time. They are used to it, it's exciting.
And if you are a developer, someone who at least likes to code, maybe loves it and aspire to craftsmanship, you can't say that Agile Manifesto is for job security. It aims at being proud of what you do, it's probably misused like a lot of "manifesto", but you can't blame the faithful believers because there are peddlers
good return of experience
Additionally, I think you should talk about prerequisite to agile. I did it in an article here:
I've heard this even today! "it's not meant to be perfect, is it? we're agile - iterative!".. eeh. It's hard to find - I think - a set of developers that will be both competent and well organized (read: open to doing real working agile). There is much more reasons why agile may not be working - few days back I read this story on some other causes: kanbantool.com/blog/agile-implementation-problems - see what u think.
I know my team just sucks and it's probably my fault (I'm too soft). Start again, I say!
Yousef Awad May 16, 2016
Jason McGee of IBM Talks about Open Source Projects and the Interactions at the Collaboration Summit
Jason McGee May 15, 2016
Srini Penchikala May 15, 2016