Is the Agile Community Being Unreasonable?

Permit me to offend everyone on both sides of this debate.

I assert:

(1) Conventional assumptions and practices for IT management, governance, and so forth have always been flawed. They are the cause of nearly every problem companies have with IT. We have become so accustomed to a 16%-29% success rate (per the Chaos reports of 1994 and 2003 respectively) that we've merely redefined that level of performance as "success," so that we won't cry ourselves to sleep every night.

(2) "Agile" came from the trenches of software development. It's specifically designed for software development activities. It says so right there in the good old Manifesto, plain as day. "Agile" doesn't offer mechanisms to solve organizational or management problems. The fact agile works so very well for software development doesn't mean it works just as well for other things.

The two sides of this debate are only defending the point of view that each prefers. Neither appears to be looking for real answers to the real problems in IT.

Michael James quotes a few lines from the PMBOK. I don't see any conflict between these quotes and agile values or principles.

PMBOK says projects are not ongoing efforts. There's no error in that statement. It's quite right. The problem is that IT organizations plan their work as discrete projects. PMBOK treats this as a "norm" instead of as a problem. That's because the P in PMBOK stands for "project." If you want to manage the work in your company as a set of ongoing product development work streams or as value streams, go ahead and do so. Just do it without PMBOK. No conflict.

Recognition and rewards. PMBOK doesn't say how, it only says what. We can apply team-oriented rewards for team-oriented work styles like agile, and individual rewards for individually-oriented work styles. No conflict.

PMBOK allows for increasing the budget when the situation calls for it. As I read it, that means PMBOK doesn't lock us into an annual budget cycle (another problem posing as a "norm"). We could use a just-in-time budgeting method and still remain consistent with PMBOK. No conflict.

Wayne Mack says: "Before I can start a project or even have team members assigned, I need to present a schedule, budget, and scope for approval. Even with customer requested and approved changes to scope, I need to justify changes in schedule and budget."

So, we have: Annual budget cycle. Preapprovals far in advance. Fixed scope. Fixed schedule. Team members assigned on a per project basis (part-time basis, too?), teams disbanded after each project. Concept of an internal "customer," with the administrative separation the terminology implies.

Holy cow! Never mind "agile" or any other popular buzzwords: How many things can a company do wrong before it goes out of business?

"I am not finding information that is applicable to my day-to-day work activities."

Yeah? Go figure.

Brad Murphy says, "[An] industry leader advocates that In the event of a poorly performing contributor the team should counsel the team member together and if the team doesn't feel the colleague is correcting his/her behavior then the team is empowered to expel this person from the team. Really? In any larger organization this will result in legal wrangling, lawsuits and HR raining down like none other."

I can imagine how a person unfamiliar with agile team dynamics might make this sort of assumption. Experience makes certain things a bit more clear:

First, there's no question of a team "firing" anyone. Self-organizing teams don't have that level of authority. Technical people don't even want it. It isn't what they do.

Second, there has to be management support for team self-organization. Teams can't just wake up one fine morning and "be" self-organizing. The manager has to be savvy enough to allow the right degree of autonomy to each team, to let them learn from mistakes, to coach them, and to set clear boundaries; there's no PMBOK-style cookbook for this style of management.

Third, the process of counseling the team member is typically supported by the manager establishing the groundwork with HR, in case the counseling doesn't work out. HR is normally fully on board from the start. I've seen this sort of situation on a number of different teams. It hasn't been exactly the same in any two cases.

In the best example I've seen, a team had one member who didn't seem to be picking up on any technical skills or domain knowledge. After a few weeks, the team dedicated a retrospective to the matter. They all agreed (including the low performer) to a one-month program of mentoring, after which they would see how things were shaping up. They adjusted their iteration commitments downward so that they would be able to dedicate some time to helping their teammate. The manager agreed to support their plan. She talked to HR and let them know what was going on. The team's Product Owner understood the plan and accepted the lower delivery commitment for the one-month period.

The guy did his best to get his skills up to snuff. Ultimately, he wasn't able to rise to the challenge. He found another job in a less challenging environment where he did well. It sounds a bit trite, but it's actually quite true that everyone needs to find an environment where they can thrive. An agile team in a fast-paced, technically-challenging environment isn't the best place for everyone. One person might find it energizing and another might find it overwhelming. It's also true that no one in the organization is in a better position to see such a problem than the person's teammates; his pairing partners. "Expel" is probably too strong a word for this process, but the basic premise is sound.

This actually works better in a larger organization than a smaller one.
When an individual doesn't fit well in a team, a larger organization can offer more alternatives to him/her than a smaller organization can do.

Well, long story short: If "agile" doesn't solve all the world's problems, that in itself doesn't mean we need to scurry back to the 1980s as fast as our little legs will carry us.

LOL :)

Hi Dave,

I haven't laughed as much in ages. I thought I was going to split my sides at one point :) Thanks for brightening up my day.

Paul.

Dave,

Good comment,

One point though, I'd be careful in using the Standish Chaos report figures, and using this as an argument for driving through change. Although still sometimes referenced in literature, the 1994 report has largely been discredited as bad science, and more recently their method criticized for presenting an overly biased view of software project failure rates, by Robert Glass, Magne Jørgensen and others.

See e.g.
www.computer.org/portal/web/csdl/doi?doc=doi/10...
simula.no/research/se/publications/Jorgensen.20...
cacm.acm.org/magazines/2006/8/5855-the-standish...

Hi C Curl,

I'm aware of the criticisms of Standish's methodology, and your point is well taken. However, their results do seem to line up pretty well with my own experiences and observations in IT; and I don't think you'd have much trouble finding a lot of other people who have similar observations - especially those who have been "customers" of IT departments. I remember a comment by the Product Owner of one of the first agile projects I worked on: "I've been working with IT departments for 32 years, and this is the first methodology I've seen that actually works."

I wouldn't want to get bogged down trying to perfect a statistical analysis method to the extent that I forgot to look at what was actually happening around me! If someone is poking you in the eye with a stick, do you really need to wait for a methodologically-sound "study" to be published to prove that someone is poking you in the eye with a stick?

A Shaw quote comes to mind: "Reasonable people adapt themselves to the world. Unreasonable people attempt to adapt the world to themselves. All progress, therefore, depends on unreasonable people."

Cheers,
Dave

Dave,

Great follow up comments, well made. A very pragmatic look at what agile might offer over older styles, and equally on what it never said to offer within large organizations.

Ben

Well written David! I fully agree!

"It seems a bit brash if our recommendations to Project Managers is to not manage projects."

That sounds confusing but that's often the best advice you can give Project Managers. If it's not a Project but an ongoing effort, don't treat it as a project.

BTW: How is PMBOK pronounced? Humbug?

As a PMP who regularly teaches PMI Certification courses based on the PMBOK, a PMO executive and as the leader of my company's development organizations Agile transformation (over 1500 development associates), I agree with Dave's statements. I recognize that many of his comments are a little tongue-in-cheek and meant to call out the ridiculous stance that by the book PMs sometimes take when being resistant to the change that they should be leading, not impeding.

The PMI (and the PMBOK) are meant to be a toolkit that recommends that as a project manager you ask the right questions and focus on the right areas to ensure project success. It should not be viewed as an exact blueprint that has to be followed. The tools I would use for managing a construction project are different than what I would use for software development, but the general framework can ensure that we do the right thing in either case. Risks, stakeholders, change, scope, etc. all need to be managed in a corporate environment, but the rigor and process/methodology should adapt to the project needs. Factors like Regulatory compliance and government contractual requirements often add requirements and sometimes constraints to development teams

What Agile often exposes is inefficient or overly rigid corporate processes, often enforced by a PMO, that often were built around reporting rather than executing needs. The process and reporting become self-serving and extremely difficult to change because companies will often have invested significant time and money to create and protect them. I have found it very effective to educate senior leadership in agile and allow them to experience the culture change required to support the bold steps to eliminate these historical roadblocks.

While I believe that Agile and PMI do fit together, I know there are many on both sides that view them as polar opposites. My stance is that either, if taken strictly by the book and applied without thought or experience, will experience some failures when scaled. The key is to inspect and adapt. Failure is only a failure if you fail to learn from it and allow the failing behavior to persist.

I encourage our Project Managers and ScrumMasters to take the approach of being "gap fillers" and servant-leaders in that they work with the team and find the areas where the team needs the greatest support and fill those gaps. The methodologies they use, both from an Agile and a PMI perspective, should adapt to the business need. Just as whether the team determines it is best to use Scrum or Kanban, the project managers are expected to understand the level of rigor of change control, risk management, etc. required to meet the client's needs. In all cases, doing just enough process where they need to land.

Matt: Well said!

Obviously by the replies here and the discussion on the PMI group, this topic has created some real emotional reactions. That is GOOD. That means that there is something here to confront and learn from. What is it? I see a few things:

1) For those of us to claim to be part of the agile community, we are not in agreement. Name calling aside, this divergence is good for finding various solutions, but is also confusing to those trying to figure out what to do.

2) There was a question/answer pair given by Wayne above: a) Why does Agile want the ability to expel members off a team? b) This cannot possibly work and is naive. Are there large organizations that have tried this and failed? What were the results? Are there large organizations that have made this a rule? Sanjiv points to Whole foods, I've read about Semler's work in Brazil, are there others?

3) There is a clash of ideals and cultures between mainstream PMI and mainstream Agile. It sure would be great if we could be precise about them, then perhaps we can address them one by one.

Finally, anyone who has something to say about these topics and would like to write an article for us at infoQ, we'd be happy to publish it!

I do still find myself talkign to companies asking for 'scrum project managers' and other such things.
There are also massive problems in bringing scrum into any organisation - and these problems are all in the middle management - the top managers can see the massive lack of productivity in an organisation, the engineers are frustrated by the paper shuffling stifling atmosphere, but the middle managers don't want the change as they are scared about their jobs.

PMBOK and similar are about promoting paperwork and control, its all about 'planning' and 'controlling' not about 'doing'. Scrum is about doing, its about getting the people who are going to create the solution the time, space and enthusiasm for doing so.

Updates to a product are often looked at as individual projects, and I don't have a problem with that, scrum produces shippable product regularly, any one of those can be chosen as the end point of a project and let loose on the real world. Self organising teams, a direct link between the team and the person who is specifying (selling) the product, good communications are the cornerstones of the productivity scrum provides, these are the reasons scrum doesn't need a whole raft of middle managers, and why they are so scared.

Middle managers aren't dumb, they have a cushy, comfy position, well paid, a large empire, plenty of paper to shuffle and meetings to attend so look really busy, someone to blame if it doesn't work, the ability to grab the glory (bonus and payrise) when it does work, and above all no stress at all.

Amr,

I think one issue that is exposed by this discussion is at a level of abstraction higher than the specifics of PMBOK or agile or Scrum or what-have-you. It's the difference between choosing a particular named, branded package of ideas and defending it religiously versus looking at each organization's goals and needs and then determining a practical path for improvement.

Each of the approaches / frameworks / toolkits that people mention during discussions like this has its own strengths and weaknesses, and is designed to solve certain problems at a certain scope. When the debate is about whether "agile" works in large organizations or whether PMBOK is about "control", we miss the opportunity to identify real problems and discover practical solutions. We end up in a pointless, circular, quasi-religious debate. In that case, there's no reason not to be tongue-in-cheek or sarcastic about the debate, because it has no chance of resulting in anything useful anyway. It might as well serve as entertainment, if nothing else.

We're all fond of declaring that there's no such thing as a magic bullet, and then many of us revert to defending our favorite magic bullet. Real solutions may call for elements of agile, elements of PMBOK, elements of lean, elements of Six Sigma, elements of CMMI, elements of Systems Thinking, and many other disciplines or schools of thought, with each applied at the appropriate scope and to address the problems it was designed to solve.

"Agile" works in any size organization, not because it "scales" but because it supports software development activities regardless of whether the organization has one team or a thousand teams. "Agile" supports software development very well. It doesn't do anything else. That's not a flaw, because it's not supposed to do anything else. When I hear people say "agile" has failed because it didn't help them with portfolio management or strategic planning or choosing a tie to match their socks, I can only shake my head in bemusement. It's as if I bought a new electric nail gun and then complained about it because I couldn't use it to bake a cake. What a crappy nail gun. It won't even bake a cake!

By the same token, as others have pointed out, when managers use PMBOK as a cookbook or checklist in a rigid way, they end up pursuing the illusion of control with no view to the organization's goals or problems. When managers use PMBOK as a general framework to help them keep sight of important management functions while remaining flexible about management style, project planning and reporting, and other implementation details, then they can support a heterogeneous organization that uses different processes for different work flows.

There's no conflict except between quasi-religious adherents of a given approach or framework. This is not a genuine clash of ideals or cultures. It's just stubbornness and limited vision.

As a PMP and agile practitioner, I find that it's quite possible to observe all processes in PMBOK and still run a light-weight, agile project. Anybody remember dX? It was a flavor of RUP that was tailored so that it was actually XP (flip dX upside down and you'll see the pun). It's basically the same with PMBOK - you can adapt it to fit an agile project.

There are a lot of differences, but the only *conflict* I see between PMBOK and agile methodologies, is basically one of underlying philosophy. In PMBOK, the Project Manager is the project's hero and the Plan is sound (reality is wrong - when it doesn't conform, corrective action is taken). In agile, the project manager is a team member like any other, and reality always trumps the plan (so when they don't match, the plans are revised). This is a bit over-simplified, but it's just to illustrate the conflict.

However, it's not PMBOK or a methodology that decides the philosophy and the tone of the project - it's the people in and around the project.

Personally, I've learnt more useful stuff from the agile methodologies (for instance multi-level planning and useful estimating techniques) but I'd never want to run a project without the more disciplined risk management approach in PMBOK. But that's me, we each have our own path.

Trond Wingard posted:

There are a lot of differences, but the only *conflict* I see between PMBOK and agile methodologies, is basically one of underlying philosophy. In PMBOK, the Project Manager is the project's hero and the Plan is sound (reality is wrong - when it doesn't conform, corrective action is taken). In agile, the project manager is a team member like any other, and reality always trumps the plan (so when they don't match, the plans are revised).

Kudos to Matt and David!

One of my favorite questions from a PMI course I took was, "If the PMBOK is 'The Guide to the Project Management Body of Knowledge', where is the Body of Knowledge?"

The answer was/continues to be... "You are."

In that simple question and answer is the summary of what the Guide to the PMBOK is about: it is a merely a guide, a compendium of best practices (tools and techniques) for you to use in your daily struggle to get things done.

A related quote that I share from Alistair Cockburn from a thread where Agilists and PMPs were debating the approach to system testing on a software project: should it be a separate group or should it be part of the Agile team? Cockburn replied with the one liner: "Do what makes sense."

Isn't that what both the Guide to the PMBOK, the Agile principles, and your heart tells you to do? In this, there is harmony between the PMBOK and the Agile principles. In this, there is also harmony between the unholy triumvirate of the PMBOK, the Agile principles, and CMMI.

Maybe it's just me; however, I find myself often stopping meetings in mid-debate to ask the simple question: what makes sense? It cuts through team dysfunction, sacred cows, and brings the needle on my baloney meter back down into the tolerable range. Feel free to pick up your own Carl Sagan inspired "Baloney Detection Kit" here.

Your description of the process is much better than the original, and makes more sense the way you have put it.

I think to restate what Trond is saying:
1) PMI does champion the project manager as a leader who should use their formal and informal influence and authority to find resolutions to project issues.
2) Corrective action is taken when a project is not conforming to plan. Sometimes this is to change the plan, while other times it is to help get back on plan. PMI does not say what to do, only that you need to determine the variance and then take appropriate action. I don't see this in conflict with agile as it really boils down to resolving blockers that arise.

The biggest difference between Agile and the PMBOK is the amount of recommended planning up-front. PMI does encourage fixing Scope, Time, Cost and other constraints and "control" them within acceptable tolerance ranges. Agile is obviously looser in the control area, particularly in Scope (embracing change).

PMI does discuss "Rolling Wave Planning" as a means to avoid heavy big up front planning, but rather execute in phases. PRINCE2 (UK PM Methodology) has Stages and Stage Gates that roughly amount to the same thing. The key with any PM Methodology is to use the appropriate level of planning for the project requirements. Most of my planning for larger Agile projects is typically around creating ground rules and communication strategies for dealing with large project issues, not team software development where Agile is proving to be significantly more effective than traditional waterfall.

Dave,

I agree 100% with your point. It all boils down to taking a pragmatic approach for the project and environment you are working in.

Software Engineering is a vast domain ranging from safety-critical systems to simple HTML web-sites where these different systems can be implemented by teams from different cultures, organisational sizes, and skill levels.

The concepts presented in Barry Boehm's Balancing Agility and Scale can be used as a basis (bit.ly/8Xqfuf) to determine the best suited approach for a given environment.

To build on that, I believe, one should not discount the value of learning gained in Agile approaches such as taking an empirical approach and running self-organising teams.

Software development is not simple and one would be foolish to align yourself to one and only one approach to software development.

Trond Wingard posted (in a slightly different order):

In agile, the project manager is a team member like any other, and reality always trumps the plan (so when they don't match, the plans are revised).

In PMBOK, the Project Manager is the project's hero and the Plan is sound (reality is wrong - when it doesn't conform, corrective action is taken).