Adopting Agile Development Practices: Using Patterns to Share our Experiences

Abstract

The increasing popularity of Agile approaches for software development is forcing an increasing number of organizations to deal with issues of Agile adoption (and adaptation). In this article I introduce patterns that focus on the dynamics of adoption rather than the structure that results from adoption. These two patterns - Pair Programming, and Simple Design - are a result of an Open Space session at XPDay Montreal 2006 and are an example of a larger effort of documenting agile adoption experiences in pattern format. This effort is intended to help those who are new to Agile software development answer the questions: "Where do I start?", "What specific practices should I adopt?", "How can I adopt incrementally?" and "Where can I expect pitfalls?"

Introduction

Several times this year groups of software practitioners - I say practitioners because they were developers, analysts, customers, and managers who have a commonality of helping build software - came together and shared their experiences in adopting agile software practices. They then documented their successes and failures in adopting each of these practices, in pattern format. They agreed to use patterns because patterns are solutions within a context and are not prescriptive like 'best practices' tend to be. They used patterns to aggregate multiple experiences instead of 'war-stories' that usually don't transfer easily to specific projects because they are point experiences instead of aggregations. They used patterns because patterns are not prescriptive yet still give specific information on how to solve a particular problem. Patterns are an excellent vehicle to share expert information within context.

In this article I will report on the results of one of these meetings. Then I will tie this into a larger effort to document experiences of experts who have previously adopted and adapted agile. This larger effort is meant to help those starting their own agile adoption efforts. I hope you find this article useful and I encourage you to join us in our effort to document agile adoption experiences either as a reviewer or writer at the agile practice patterns wiki (Or contact me personally at http://elssamadisy.com/contact for more details). With that said, onwards!

XPDay Montreal

On a Saturday afternoon in September 2006 two Open Space sessions entitled "Agile Practice Patterns" and "Agile Anti-Patterns" were merged into one. We had a mix of attendees: a few consultants, some developers who were introducing XP practices in a grass-roots manner, and a group working together at the same company who were about to embark on adoption of some agile practices next Monday morning. This was a diverse group that had many points of view to offer. The consultants shared their experiences (both good and bad) in helping their clients adopt different practices. Developers who introduced practices in their own spheres of influence, each had insights into constraints unique to his/her own situation. The group embarking on adopting agile on their first project came to listen and learn and were invaluable in helping put together something useful for the ultimate consumers of such patterns.

In the limited timeframe available the group discussed and flushed out two agile practice patterns that included information needed for adoption. The discussion of the patterns led to "clusters" of practices that were called out but not documented in detail because of time constraints.

Pattern Format

The format we, the attendees of the Open Space session, agreed upon using is a modification of the Therefore..But format used on the original patterns wiki (Portland Pattern Repository http://c2.com/cgi/wiki?ThereforeBut).

• Name
• Summary:  A short sentence or two that briefly describe the pattern.
• Sketch:  A story/narrative that acts like a 'sketch' in design patterns. (idea taken from Fearless Change¹ )
• Context:  Who and in what circumstances this pattern is useful.
• {Forces:}  Used to elaborate context by giving specific issues. These issues are problems which are (partially) resolved by this pattern.
• Therefore:  The pattern description.
• {But:}  Negative consequences that can occur from applying this pattern.
• {How:}  Steps, ordering, guides to adopting this pattern. Sometimes interleaved with Smells: to indicate where the adoption can go wrong.
• {a.k.a.}  Similar published patterns.

Pair Programming

The first practice discussed was pair programming. This was one of the more controversial practices and did not have consensus among the group. The questions discussed were "how much slow-down - if any - is introduced by pairing?", "does design quality improve even if there is a mismatch in the experience of the pairs?", "is pairing appropriate in all cases - is there a context where you would not recommend pairing?", and finally "how can we go about adopting pair programming? Is it an all-or-nothing practice or are there intermediate steps that can be taken to give us a 'taste'?"

Pair Programming

Two developers work together side-by-side to solve a programming problem. This is a form of continuous code review that increases the quality of the software being built.

Sketch: Cindy Coder is having lunch with Dave Developer telling him about her experiences on her first Scrum/XP project - let's listen in:

"Cindy: ...and pair programming was one of the practices I was sure was going to be a waste of time! But looking back, all of my work done with pairing is of exceptional quality. This was true even when I coded with Natt the Newbie and Joey Jr. Developer. There is something about having to explain your code to your partner that just brings out the best quality. It is as if we are always performing code reviews. Also, now I know a much broader part of the code-base than I usually do. Because we're always coding in pairs at least two people know every part of the code - no more people bottlenecks.
Dave: Sounds almost too good to be true - and anything that sounds too good to be true usually is. What's the catch?
Cindy: First, it is a bit slower. But I figure the better code/design quality is going to help in the long run. That's just a hunch. Secondly, it is exhausting! Finally - it's not all rosy - there are some people on the team that are just not good at working closely with others. We've decided to leave them alone - but it does create a tension with those folks."

Context: You are on a development team where design quality is of primary importance.

Or, you are on a development team which is concerned with bottle-necked (specialized) resources, and needs to be able to move developers on and off the project or survive loss of team members.

Therefore: introduce a form of code review and inspection to increase quality. Train members of the team so that their skills overlap to remove resource bottle-necks.

Pair programming involves two people working on the same task. The person with the keyboard - the driver - is responsible to code the particular task at hand while the other person - the navigator - continuously looks ahead and thinks of the abstractions and design repercussions. Much of the communication is done via code - when the navigator has a suggestion she will take the keyboard to show what she means in code and tests.

Proper pair programming is a continuous form of Code Review, which is one of the few practices in our industry where there is experimental data ² . This indicates that pair programming greatly improves code quality.

Developers also transfer their expertise to their pairs. By working on the same issue the specialist implicitly trains their pair by solving the problem together.

But: Although it has been reported that pair programming does not affect development speed this is by no means a consensus. A significant number of experienced practitioners (me included) find that pair programming does slow down development.

Moreover, pair programming is much more than two people sitting at the keyboard. If developers are not careful one of the pairs can easily monopolize the keyboard and coding. Pair programming is not for everyone.

Finally, there is less flexibility in work-times because developers must work together at the same hours to pair. So options like 'flex-time' will need to be made less flexible for pairing.

How: There are several ways to introduce pair programming. By far, the best is to have one or more experienced pair programmers on the team in order to 'spread the virus' by pairing with those on the team who are new to the practice.

Ping-pong pair programming is a common way to introduce pairing to a development team. This involves one of the pairs writing the tests and handing them over to their partner to implement the code that will make the tests work. Then the roles are reversed. In this way, keyboard monopolization is reduced, and some structure is put into pairing until the team becomes accustomed to the practice.

Simple Design

The discussion on pair programming was useful but not of immediate use to all participants. One team attending the workshop was starting agile practice adoption but had decided not to pair program initially. They had, however, decided to start with Simple Design. They had specific questions concerning this particular practice: "are there any practices that are required for, or that enable, Simple Design?", "how simple is Simple Design?", "does Simple Design always make sense?", "should we ignore our 'look-ahead' instinct, always?", "are there times where it does not make sense to do Simple Design?" All of these are very good questions. The following pattern attempts to answer these questions:

Simple Design

The complexity of your design should support the current requirements at hand (being built in this Iteration) and no more. Do not design for future flexibility, let new requirements "pull" more complex designs and rely on Automated Tests to enable Refactoring later.

Sketch: When Waterfall Will joined Scott ScrumMaster's agile development team a year ago he had many misgivings about the project team and their coding techniques. Most importantly, he had a burning question, "Where is the Design?!" He could not fathom how starting with a Simple Design could ever work. In his experience a team must set the Architecture and design upfront. Otherwise, the frequent changes that will be required to change the design incrementally will incur exponential costs.
Will decided to suspend his disbelief for a few months and give this new development technique a chance. Grudgingly at first, Will had to admit that the simple designs were elegant in their own way. As he observed again and again the resilience of these designs and how they could be easily changed according to new requirements due to the of the safety net of Automated Developer Tests he started to enjoy this way of development. Overall, his designs were much leaner with Simple Design. He recognized that this was not really getting rid of the design cycle but making design part of every day's work.

Context:You are on a development team that is building a software system with requirements that change frequently. Therefore your team needs to build a system that is resilient to change. Competition from other companies mandates that minimizing the time-to-market be a high priority.

Forces:

• Feedback between customers and developers is infrequent.
• Customers may not know exactly what they want until they see working software.
• Integration is infrequent, costly, and often extremely painful.
• Function completions are out of synch so integration is usually pushed to the end of the release cycle.
• A "hardening" phase is frequently scheduled before a major release date to ensure that the product quality is acceptable. This phase includes no new feature additions and is focused on bug fixes only.
• Functionality is very complex and developers have a tendency to "go off track" and come back to the customer with an incorrect solution.

Therefore: In the agile community we believe that building in complexity in hopes of reducing the cost of change for the future is a false hope. Generalizations provide much more flexibility than what is strictly needed by the current requirements. In the agile community, this type of generalization is known, derogatively, as Big Design Up Front (BDUF).

We are not fortune-tellers and cannot foresee all of the changes. The upfront design is not for free - the extended generalizations made to allow for change are more complex and harder to understand and maintain than a Simple Design. The cost of carrying that design will far outweigh the benefits gained.

Your design should only be complex enough to meet the requirements of the current iteration. Your design should be a Simple Design that has no generalizations for needs that will come in the future for two reasons: (1) you really don't know what the requirements will be two years down the road. Putting those generalizations will incur a 'Cost of Design Carry' over those two years. (2) by enabling Refactoring via Automated Developer Tests you will have reduced the cost of change and will be able to cost-effectively make the changes when new requirements dictate them.

But:

• A team may drop Simple Design when some paths lead directly where a BDUF would have led them. They, thus, see the constant Refactoring as a waste. They don't realize that most of the BDUF still leads to over-generalization; conversely most of Simple Design leads to less complex designs.
• A team may interpret Simple Design as the design that takes the least time. Frequently that includes cut & paste solutions. This is NOT Simple Design. This is bad code.
• Teams frequently do not adopt Simple Design because it cannot possibly work in their opinion. It goes against all of their expertise and good sense. We highly recommend that teams suspend their disbelief as Waterfall Will did in the sketch of this pattern. Two to three months of practicing Simple Design diligently will make a believer out of a team!

How: So how do you apply Simple Design?

1. Determine non-ambiguously what the requirements are for the task at hand.
2. Determine what the solution will look like. This can be done by writing the tests first and letting them drive the solution, or more traditionally by coming up with a design before starting to code.
3. If the solution uses existing code that is not general enough for the new requirements, Refactor the code to make it amenable to adding the new functionality. Rely on existing tests to verify that you have only changed design and not behavior.
4. Add the new functionality with a solution that is only as complex as needed to meet the new requirements.

Simple Design should not be practiced without the ability to refactor and evolve the design. Refactoring, in turn, cannot be done effectively without a set of Automated Developer Tests. These are the necessary practices for simple design.

To effectively adopt the practice of Simple Design most developers must suspend their disbelief ³ for several iterations in order to observe it working effectively.

Variations: These are some common groupings and clusters of successful agile Iterations:

• Test-First Development, Simple Design, Refactoring
• Simple Design, Test-Last Development, Refactoring

Clusters of Practices

The Simple Design discussion at XPDay Montreal brought out the topic of clusters of practices. Simple Design cannot be effectively implemented without Refactoring which, in-turn, cannot be effectively implemented without Automated Developer Tests. Automated Developer Tests can be supported by either Test-First or Test-Last development. These are two very different ways to create the Automated Developer Tests.

We are well beyond the days of "thou shalt do all 12 practice of XP" but there is much truth and wisdom in clustering of practices together. Some practices, when used together, deliver more value than the sum of their parts. The individual practices within these clusters can be considered generative rules. By using practices within a cluster together they generate much more value than by doing each in isolation.

An example of a very common cluster is Test Driven Development (TDD) which involves many different practices. Unfortunately it is beyond the scope of this article to delve in deeply. This is, however, how TDD can be seen as a generative cluster of practices:

Aggregating Experiences

What I have described above is a slightly polished version of the two patterns that were discussed and documented at XPDay Montreal. Conferences are an excellent place to meet others in the field and exchange experiences and cross-pollinate ideas. They are also a great place for those new to agile practices to learn from more experienced practitioners. These patterns are not meant to replace the value gained by attending conferences or other ways of communicating, but they are meant to augment them. They are what many of us wished we'd had a few years ago when we were adopting agile practices for the first time. They are an aggregation without being an over-generalization of experiences and lessons learned.

Patterns

Although these patterns seem simple it is important to keep in mind that they are an aggregation of diverse experiences and not a single report or recommendation. Patterns are a succinct way of describing practices within context - they are, however, very hard to write. The patterns reported here are in very rough form. They need to be read, reviewed, rewritten, and polished until they are more accessible to the public.

Smells

One important topic we have not covered in this article is that of Smells and how they can be used to drive adoption of practices. When we adopt agile practices, we are constantly asking ourselves "which practice(s) should we adopt and in what order?" All we have described so far is each practice and in which context is it useful. We have jumped directly to the solution without asking ourselves: what is the problem we are solving?

A useful metaphor is that of going to a doctor. When I am not feeling well I head off to the doctor's office. The doctor takes my blood pressure, checks my temperature and asks me where it hurts. This is usually followed by an examination of some kind and possibly some blood-work if it is serious. After gathering this data she will usually prescribe one or more medicines for me to take and tell me to discontinue the medicine and come back immediately if I feel any adverse effects. If the medicine has any side effects she will usually warn me and it will be printed on the medicine bottles.

Now let's compare this with agile development practices: Why does someone adopt agile practices? Usually it is because something is wrong with the way their team is developing software. There are pains that they hope to alleviate by adopting these practices. What are these pains? Which practices alleviate which pains? These pains are the development Smells - the symptoms of an ill software development process. What is the remedy? The remedy can be one or more agile practices. Therefore, for this picture to be complete, we need to discuss software Smells and associate them to the appropriate patterns. That, unfortunately, is an involved story and cannot be told here.

The Dynamics of Adoption

"So, how does this help me adopt agile practices?" you ask.  With the building blocks at hand, that is: Smells, Practices, and Clusters of practices, it becomes possible to tailor your agile adoption in a step-wise and iterative manner.

1. Start with an evaluation of the status quo - what is being done well and what needs improvement?
2. Map the areas that need improvement to one or more Smells.
3. Prioritize the Smells and pull the first one off the list.
4. Check the Practice Patterns that address the Smell in (3). Read them and read their related clusters.
5. Depending on your environment, choose one or more of the patterns in (4) - maybe even an entire cluster. Use the advice in the patterns chosen to start adopting those practices.
6. Periodically evaluate that the Smell in (3) is indeed getting addressed by the practice being adopted. Adapt the practices for your particular environment.
7. Go back to (1) until perfection is reached ⁴.

Adoption should be iterative and always goal-oriented. The goal is to alleviate the Smells currently present by adopting and adapting the applicable agile practices. The patterns are starting points. Use them with a modicum of disrespect. As you gain experience, adapt the practices to fit your needs.

Where do we go from here?

The patterns written at XPDay Montreal are very rough. They still need to be reviewed by a larger group to refine the context, correct errors or misstatements, and expand upon and verify the adoption strategies suggested. Furthermore, a typical adopter will consider more than just Pair Programming and Simple Design.

Finally, there are still many more practices that could benefit from this type of documentation. They really need to be paired with common software development Smells. This will enable the adopter to start from the symptoms instead of adopting practices wholesale as we have done in the past (dare I say incorrectly?).

Critical Mass

The work on agile practice patterns has yet to gain critical mass. There are patterns that are in different states of completion. Let's be honest, this may seem (I hope) like a good idea to you, but unless these clusters of patterns are written out, they cannot be used by adopters effectively.

I've been jokingly asked several times "so where's the book that has all of these adoption patterns?" Unfortunately we are not there yet. Nevertheless, over the next few months those of us who are participating in this effort hope to have the first usable set of agile practice patterns ready that a reader can use in his/her adoption strategies.

I invite you to get involved by contacting me at http://elssamadisy.com/contact or directly posting to the wiki. All of the work in progress will be made available on the wiki and we will intermittently write similar articles introducing the results of our work to a wider audience. Look for more articles and possibly a mini-book documenting the Smells, Clusters of Practices, and more Practices of Agile Practice Adoption in the near future here at InfoQ.

Related Work

Similar sessions, of varying length and size, were run at ChiliPLoP 2006 and XP2006. Documentation of these patterns is available at agile practice patterns wiki. Some of the patterns from the ChiliPLoP sessions were polished and are to be presented at PLoP in October 2006 (http://hillside.net/plop/2006/Papers/Library/AdoptingAgilePractices.pdf and http://hillside.net/plop/2006/Papers/Library/functionaltesting.pdf) and the patterns that came out of XP2006 are reported upon in the fall issue of Agile Development from the Agile Alliance. There is also work by others that documents agile practices as patterns, most notably Joe Bergin's papers presented at PLoP and EuroPLoP and Jason Yip's work on Stand Up Meetings to be presented at this year's PLoP conference, but this work does not focus on process adoption and adaptation as we do here.

Glossary

Cost of Design Carry: The cost in time and effort a development team must exert to understand and maintain a piece of code. The more complex (generalized/flexible) a solution is the harder it is to understand that code correctly and modify that code when needed. The cost of design carry is a burden the development team carries every single time they must modify the code.

Development Practice: A development practice - a single action that is done by a development team. For example, writing Automated Unit Tests is a development practice.

Development Process: A collection of development practices performed together with constraints on their relationships and/or order. Examples of development processes are extreme Programming and Scrum.

Generative Practices (of software development): Practices are generative if they feed off of each other such that their value and effectiveness when used together is significantly greater than if they were used apart.

Pattern: "a proven solution to a problem in context." Patterns are a proven format for sharing expert information about complex problems. Patterns, including the ones presented here, are validated by their successful application in multiple instances. Documentation of a pattern generally includes: name, definition, context, problem - including identification of relevant forces, solution structure, implementation, and discussion of applicability and other usage issues.

Refactoring: Changing the design (structure) of an existing code base without changing the behavior.

Smell: A symptom of something going wrong. Smells are vague in nature and indicate that there is something wrong. Smells are not defined, they are recognized. A smell reflects a relatively vague sense of unease or discomfort. Reflection on a smell may result in the identification of a resolvable problem.

About the Author

Amr Elssamadisy is currently a Principal Consultant with Valtech Skill Development (www.valtech.com). He considers himself a programmer but has worked for consulting companies since 1999. So an outgoing, people-oriented, programmer is a better description. He has been working professionally as a software developer, architect, manager, consultant, etc... since the mid 1990s helping build software systems in C++, J2EE, and .NET. His first agile development project was a large project XP effort in 1999 where he had a chance to work and learn from some of the best in the field. Since then he has lead, participated, and mentored large and small development in both the .NET and J2EE worlds.