Tim Ottinger's blog post I want Agile back earlier this year led to discussions in the agile community about the way that organizations are adopting agile and the services that the industry provides to supports them.
Together with Ruud Wijnands he started "take back agile" which focuses on technical practices and craftsmanship in agile.
InfoQ did an interview with Tim Ottinger and Ruud Wijnands about their views on the problems that organizations are facing with agile adoption, how organizations and the industry are dealing with it and how we can find better ways for agile adoption.
InfoQ: Tim, in your blog post I want agile back you mentioned several problems that you see with agile adoptions, like a lack of technical practices, insufficient team autonomy and buzzword compliance. Can you elaborate on these problems?
Tim: Technical practices keep our code workable so we can release very frequently. We see companies using no technical practices at all
Their code gets messy from trying to release frequently without managing the code base, but they're complying and conforming to the new process. It's not okay with the programmers and testers, and nobody is happy to see the velocity drop, but managers won't give permission to adopt technical practices. They fear that pair programming or testing might not be efficient. They guess they may not be able to adopt technical practices because of micromanagement.
Autonomy is missing because the teams feel like they have to have permission to do pair programming and testing and refactoring. They think they need permission to clean up nasty code, and that they could lose their jobs by choosing how to do their own work efficiently. It's micromanagement and control, and it keeps teams from doing their best work.
For years, conference speakers have bragged on adoption rates and numbers of copies sold as a sign of agile success. But to get those numbers, you need low barriers to entry and a lot of compromises. The things making them successful are not the things that make their customers and clients successful.
In this way, Agile has suffered from its own commercial success.
You can do the same old things, the same old way, but if you add on a planning meeting, a retrospective, and a daily status meeting, you can call that agile now.
It's gone from lightweight methods that rely on technical skills and engagement to just another excuse to hold too many meetings.
Ruud: If I may…. the companies that I have seen that were successful in their agile adoption all valued and implemented technical practices. The companies less successful did not. What I have seen is that companies like to copy things that seem to work for others. Scrum is simple as a framework, but does not contain any technical practices. For many people scrum = agile, which to me means they don’t have the full picture.
There are also many success stories around Scrum and I noticed that success stories make things attractive. It triggers what I call the “Me too” syndrome. We want success too. This in combination with the perceived simplicity of the Scrum framework made Scrum even more attractive.
The founders of Scrum have always warned about the simplicity aspect of Scrum. The framework is simple, but very hard to implement. Somehow I think many have neglected these remarks in their hopes this would be an easy path to success. The Scrum framework gave them a perceived blueprint for success. Who would not want that?
As human beings we long for quick wins even though we know that if things sound to good to be true, that they probably are. There are no quick wins. None of the agile practices and frameworks are easy to implement. Even learning the basics of technical skills like TDD, BDD, refactoring, code smells take time. You don’t learn these things over one night of sleep. That’s different with Scrum. You can learn the basics in less than a couple of hours.
Maybe people thought: “Low investment, High impact”. Who know? I think many simply underestimated how much hard work it really takes. Why? Well Scrum is incomplete in that sense. It shows you really clearly what needs attention, but it does not help you to figure out what attention. If you run into issues, you need to fix them. Scrum doesn’t tell you the fix that is needed.
InfoQ: Ruud, in take back agile you talked about humanity in software development and craftsmanship. Why are they important in agile?
Ruud: I remember that in the early days of XP we used to call it “The Revenge Of The Nerds”. With XP and the values around it, it finally felt like we were allowed to not only properly communicate with customers, but our craftsmanship and opinions suddenly mattered.
I remember thinking many times before I started with XP that I like software development as a profession, just not the trouble around it: “If I only were allowed to just do my job". Others deciding for me what task I should do and how much time I was allowed to spend on it. Usually this was someone with no clue about what the work really involved. The recurring discussion why something so obvious and simple took so long to develop. The constant pressure to deliver functionality and hardly ever getting a chance to deliver the quality that we believed was necessary. And getting the blame when something went wrong even if we had tried to express the need to do (more) testing and quality. It just felt unfair and I was close to losing my love for software development.
Building in quality and being in charge of own work felt like we were moving away from being production machines to human beings again. I remember that “The Bill Of Rights for Developers and Customers” really resonated with me. It felt fair again. Those who do the work, decide how to do the work. What a magic idea! The fact that we needed to learn new skills to become multi-disciplinary also mattered a lot to me. Learning is fun, becoming better at what you love to do, is even more fun. XP cleared the way for this in my case.
InfoQ: What are your views on the deeper reasons why agile got in this situation described in your blog posts? Didn't we see it coming? Could we somehow have prevented it?
Ruud: I think we probably could not have prevented it. What I think is that we see the effect of market demand at work. When things become popular there will always be people that find smart ways to make money on it. People buy wants, they don’t buy needs. People want to buy the success they hear from the agile success stories. You can’t buy a new set of values for your company even though that is maybe what you need. You can buy products and services that help you implement practices, processes and frameworks that are claimed to help you achieve successes others had with them.
Companies also want to be able to measure things. I think that the need for standardization and the desire to be able to assess employees has created a demand for certification. We simply want to know if someone driving a car actually got their driving license.
At some point in time the certification became a goal by itself. Again, market demand? As a developer you need some proof that you at least got the basics of agile and apparently certification seems to be the answer. It is your proof that you’ve learned to drive and received your driving license. Consider it your diploma, just like the one from university. We all know how much we thought we would be capable of ones we had that diploma. Yet, most of us had to find out that the real learning just started ones we got our first job.
I think certification is much overrated. The certification programs out there are not based on real practice and are not measuring real skills. Consequently, they have become quite meaningless, because I think it requires more than just book knowledge to be able to become agile.
Tim: I'm afraid that it's not a fault of any person or person in particular. Nobody meant for it to go horribly wrong. Industry people were just excited about going mainstream and becoming a dominant form, as if that were a goal.
What is that Upton Sinclair quote? “It is difficult to get a man to understand something, when his salary depends upon his not understanding it!”
People made a lot of money and built companies on selling agile, and more sales meant more money. They didn't really need you to become truly agile. All you really had to do was buy their product or services. You didn't have to do a better job, just become a customer. If you did great, that's nice too. But really they wanted to tick up that % of market penetration.
Our commercial success came at a price.
Do I wish we were not successful? Do I want these companies to have failed? Do I feel that they need to be punished or stopped? No to all. I'm sorry that our industry sold the shell of agile with the soul scooped out, and I would like to return the soul to the shell if the world is willing to have it.
InfoQ: What have been the reactions from people in the agile communities on all of this? Do they recognize the problems? And see a need for action?
Tim: It ranges from enthusiasm to jaded suspicion.
There is a huge sampling error in my data. We haven't gotten organized and done any world-wide surveys. I only hear through social media and personal contacts.
So of course, it's overwhelmingly positive. A lot of people have been in the game a while, and they remember when XP was new and when people took on agile practices because they really wanted to try it, and see if they could take it to a new level.
A lot of people found that the agile methods really humanized the work, and opened the door to more lean and more continual development techniques. In the heyday of agile, people pushed the limit on how simple, how safe, how lean, and how human a software team could become.
It's fair that people would be suspicious: Are we trying to take ownership of the brand? Are we trying to set up an alternate, competing certification program? Are we trying to cast doubt on our competitors?
It might be good for folks to know that "Let's take back agile" has an "us" that includes everyone. We don't need a brand. We don't need yet another certification scheme, and it doesn't matter what we call it.
What we want is to be effective, safe, responsive, skilled, collaborative, ... all the things that we always wanted.
Ruud: Summarizing I think most reactions are related that becoming agile requires a culture change. But just that seems really hard and can immediately feel overwhelming. Just adopting practices will give you some results / improvements, but won’t make you more agile. Just adopting practices without paying attention to the human dynamics will easily get you into a mess. An expensive mess. You can spend a huge amount of money in change programs and get nowhere.
Just recently I saw an example of this. An organization that used all the agile vocabulary and called themselves agile, but there organization structure and processes were so complex that it took them seven months to get “user stories” to ready for development. Yet, they kept saying they were more agile than before.
They truly believe that. This is actually quite common. Why does this happen? Well, I think this is because there is no clear reference for what it means to be agile. The frameworks seem to play the function of that reference, but as I mentioned before they won’t make you more agile. For me true agility is being able to respond to change on all levels in an organization. Frameworks and best practices and the ability to respond to change is an oxymoron, isn’t it?
InfoQ: What about the organizations that are doing agile transitions, how do they feel about this?
Tim: I listed some suspicions above, all reasonable. I know that there are some companies who are not cooperating with consultants but who are instead demanding a formality-only kind of change. But there are plenty who are really looking to simplify and improve and make a more human-friendly workplace. Even some of the tool vendors, who you would suspect of making tool-only changes, are pretty good folks who want a better standard of treatment and performance.
Ruud: I don’t know. We just started talking about this. For those guiding agile transitions I just hope they really support the agile values and know how to help companies to adopt the values. The values are more important than the techniques, practices and frameworks. The latter are just implementations that help.
When an agile transition fails (and most do), I believe this is mainly because one is trying to implement the mechanics of practices and frameworks. You can be more successful than you were with that, but it can also work against you when you don’t have buy-in from your staff and they just do what they are told.
Tim: A lot of companies believe that a planning meeting, a daily status meeting, and the use of lifecycle management software are the defining aspects of agile. A lot of them believe that there is just one brand, and that "agile" and "scrum" are synonyms.
I'm not slamming scrum here. A scrum team can be agile as any other, or it can be as non-agile as it chooses. Scrum is really the box that agile methods come in. It's pretty neutral, but can be a good training ground. Agility of a scrum team hinges on this paragraph from The Scrum Guide (2013):
Development Teams are structured and empowered by the organization to organize and manage their own work. The resulting synergy optimizes the Development Team’s overall efficiency and effectiveness.
If they're not structured and empowered, and can't really organize and manage their own work, then they're not really doing scrum.
A team that takes on the minimal inconvenience of meetings without adopting any of the 12 principles or claiming autonomy will languish. I refer to that as "compliance scrum."
Compliance scrum represents a successful training/coaching/software sale that helps out the vendor, but produces only heartache for the customer.
"Agile" has become a dirty word where compliance scrum has been practiced. You can lose a training contract by using "the A word."
Do the companies know they're not really doing scrum, let alone doing "agile"? I'm not sure that they do. What I read on linked-in is that many people believe that compliance scrum is "the ultimate expression of agile" and not merely the most commercial face.
I see many companies declaring that their "agile transition" is complete because they have standups and planning meetings. That's compliance scrum.
If they knew how much more they could have, I think that would change their lives. Maybe that's why XP is making a comeback in Europe and in some places in the US. It didn't lose its soul in sales and marketing, and didn't compromise its values and principles in order to get more market penetration.
But I don't think we have to use XP either. If we look at the principles and values, it's all quite pragmatic. If we can focus on safety, we can get to the same place. Let people change their environments to protect themselves from project risks. Let them pair, test, experiment, grow. Let them retrospect. But hold them to continuous (or nearly continuous) delivery. Be sure you use real work to improve the team. Make it safer by making it simpler, easier, more clear. Make it safer by refusing to build draconian policies and refusing to work people to death. Instead, create agreements and autonomy and engagement.
People mostly need to hear that "compliance scrum" was never agile, and never will be.
InfoQ: What in your opinion are better ways to do agile adoption? What can we do to improve the way that we support organizations in their agile transition?
Ruud: I don’t think I have the answer. What I have seen work in some cases, did not work in others. Things that come to my mind: keep it simple, keep it small, small steps, small steps, small steps, ridiculously small steps. Pay attention, find out what works. Don’t be afraid to do things that are hard to do, they might worth it. Don’t be afraid to fail. Make it safe for the health of your company and its employees, the finances, your time, your reputation, your relationships and your information. Know what you want to achieve and why you are doing this. Make sure others know this too.
Tim: Kent Beck once said "Start with the values."
Ron Jeffries and others have said that we could start by just delivering every week or every two weeks, and do the work that makes it possible or easy.
Scrum (not even my favorite Agile method) suggests starting with transparency, inspection, and adaptation.
XP started with "what has worked well in the past?" and "how can we do those things more intensely?"
Either way, it is about delivering sooner (not faster) and more often using feedback and incremental, iterative, evolutionary design.
They didn't start by buying expensive tools and forcing the developers at the bottom of the V model to use them. The don't only happen at the bottom of an organization, and they don't happen only at the top.
Late adopters, especially those with strong command-and-control history, tend to demand that developers "go agile" but not management or contracts or executives or testing departments, nobody else at all.
They buy an agile life cycle management tool, get a day of training, and tell the developers to go for it -- expecting them to take 7-month-large, fully-specified features and get them done faster and more visibly and without defects. Good luck with that.
Agile transitions require a "thin vertical slice" of an organization. You need an executive, some product management, product owners, IT, support, testing, documentation, marketing... a full vertical slice. Otherwise you can't establish feedback and negotiation and working agreements you need to drive up trust and drive down ceremony and overhead.
Most agile transitions that work start with people in a thin vertical slice from executives to facilities, with the goal of learning together. Learning requires engagement and pre-permission and agreements to let people explore and grow (and sometimes purchase) in ways that seem new and sometimes scary to all the layers and departments in old-school organizations.
How you start that is largely up to you. You can hire someone who has actually done the work to advise you, or you can grow the expertise locally with a lot of research and a deep dive into the literature.
Ultimately, it is a transition of people not merely of tools or processes or schedules. Forget that, and you end up with the kind of mess we're ending up with now.
InfoQ: Can you give us an update on what has happened in the initiative "Take back Agile"? If people want to know more about this or join this initiative, what can they do?
Tim: Right now this "movement" is a conversation.
The best way people can become involved it to self-educate and join the conversation.
To self-educate, one can spend a very little bit of time at the agile manifesto web site, especially with the 12 principles (see link at bottom of first page). These are pretty abstract ideas, but they're the right place to go.
Following that, they should read up on XP. There are some great, thin, simple, clear books in the Addison-Wesley XP Series, especially "extreme programming explained" and "extreme programming installed."
Then they can follow Industrial Logic's blog, or Eighth Light's blog. There is a lot of very agile work going on, and some organizations are still pushing the envelope.
I highly recommend that they join the Google plus community, and visit some of the blogs we mention to see what's up.
Mostly, though, I want to see people who are practiced at agile development, preferably in at least two methods, taking to the web with their own blogs and forum posts. Linked-In is a mix of experienced agilists and two-day certified wonders, and there is as much horribly bad advice as sage advice. I'd like to see us win it back from the low-mileage experts.
Most companies suffer from a lack of good advice and a lack of good examples. We can fix that.
About the Interviewees
Ruud Wijnands is a senior coach and trainer who believes anzeneering brings a much-needed focus to human dynamics within software development. He has successfully helped transform many software teams to Agile, and helped organizations with change management towards a more lean way of working. His clients see him as a passionate and detailed coach who genuinely cares about each and every person, making sure they understand and implement their newly learned skills. Ruud's primary focus has been on software development for highly innovative research projects and large scale systems, mainly using programming languages ANSI-C, C++, Java and C#.
Tim Ottinger is an Industrial Logic Anzeneer, who has been writing software and helping organizations write software better since 1979. He is a husband, father, author, coauthor, blogger, software developer, coach, trainer, and twitter addict. He still loves it all.