I have worked in Waterfall model for most of my career. Some time back I joined Xebia and started working in the Agile style. Specifically, we have been following Scrum and XP methodologies with TDD as an emphasized practice. The transition from Waterfall to Agile is like walking from one planet of the universe to some other planet. When you shift from one to the other there is a complete change in your world, everything from the way you think, work or collaborate, changes.
I joined a team of 8 professionals and contrary to my expectation 6 of us had no prior experience with the Agile approach. So we started under the guidance of our 2 experienced pros. Initially there were few ups and mostly downs. We found out Agile is not just about sprints and no documentation - there are more subtle aspects which you have to learn the hard way.
Here is an account of what went on:
Some Startup Hiccups:
- The Thinking Shift
Agile is all about thinking in the present, the current sprint, the current user story etc. We do not think about how things will work when some other story that we might encounter down the line comes to the board. In Waterfall we used to think about the whole system first, the HLDs and LLDs were the first steps and they had to be frozen before moving forward.
In contrast, Agile is all about current state and constant change, if our requirements change in the future, our design will evolve accordingly; but we do not concentrate much beyond the current sprint. It takes some time to absorb this fact.
- Changing Gears
Agile is a constantly changing environment. “Responding to Change” is one of its prime directives. In order to respond a developer must be in sync with market technologies else it will cost him a lot of pain.
We were using a search engine library in our project but only one of us had knowledge about it. Due to this handicap we had issues while estimating things, while pairing up, during stand-up meetings, and other day-to-day practices.
The other handicap that we faced was doing TDD appropriately. From a technology perspective, TDD has quite a few options. You can rely on various mocking and testing frameworks, which you do not use that often in Waterfall. A comprehensive knowledge of these is a must if you follow TDD else there will be a lot of hiccups.
The Sprint Slowdown
- Test driven development is all about writing tests first and then deriving business logic from the tests. This is one of the hardest things to adapt to, as it involves a complete change in your perspective.
In Waterfall there is code first and then some tests (usually, but not always); but TDD it is all about Red, Green and Re-factor. The approach asks us to think in abstract terms and then evolve concrete things from them. Since it is hard to adapt at first, we often resorted to the practice of writing the logic first and then making sure that there are tests to cover them. I would call this as Development Driven Tests (DDT) approach.
DDT does not add much sense. Why make a test if you know your code will work fine? Just to prove your point that the code works? There should be more to it than that. And there is. For example, writing tests does make your code better in terms of coupling and provides a huge safety net when making changes in the future.
- Thinking in abstract terms is a must for developers in Agile. In Agile we do not think of an upfront design, as in Waterfall, the design emerges in due time by creating abstractions and developing work flows. Developers in Agile must be well versed with at least the basics of deign patterns else they will litter the code a lot and only crap can emerge out of it.
TDD assists us a lot here. It asks us to think in abstract terms. Moreover as soon as we are done with fixing our tests we must think about and refactorings that can make the code better. DDT also helps identify right away, any quality issues so they can be addressed before it is too late. So we eventually learned that we must throw away the code first – test second approach, and then we re-started in the TDD manner.
- The code quality is team's responsibility and people in team will re-factor or possibly rewrite it time and time again, until it matches the standards prescribed. We must not attach any emotions to our code and should be prepared to toss or rewrite it again and again. Following advice from Venkat Subramaniam, on the practice of collective ownership: “we should strive to improve the code quality every time we performed a check-in.”
- In Waterfall requirements are signed, sort of, in blood and then you move forward but in Agile, the requirements evolve simultaneously with the solution. This helps to make things more clear as we go ahead, but they also result in demands for redesigns of the system at times and this will happen quite often in the initial sprints (1-4). Moreover the backlog can also change in the middle of the sprint, so the team should be geared to take up those changes, as delivering working software at the end of each sprint is the Agile motto.
- Pairing initially seems to be a waste of time and effort. Two people working together on the same story looks like a waste of 50 percent of both, time and effort. Moreover multiple people working on different tasks of the same story seems to be a cause of slowdown in velocity.
But teams in Agile are meant to work like partnerships and not like individuals working on the same project or in the same room. When teams work together they push things to their best and consequently you have a greater likelihood of finishing things consistently rather eventually failing on multiple fronts.
- Pairing is best done while following TDD. The driver and the navigator co-ordinate their efforts to develop a better system. But pairing while coding if you are following DDT approach is not the best way to go forward. In such a case neither of the partners will have any idea how to go forward e.g. what the system design will look like, so you will have lot of noise and then what emerges is a bunch of source files that must be rewritten.
The best approach is to follow TDD but if you cannot adopt to it you can still pair using a glass (white) board to first chalk out some design flow and then either of the partners can write the code while the other one can work on test cases.
- Delivering working software in the demo is the litmus test of the sprint. But, do you consider a sprint successful just because the software is build-able workable if it is not releasable?
Is a sprint successful when there are few stories completely done with some left out, or a sprint with all of the stories mostly done but with niggling issues?
Teams should focus on getting a full story done with all the issues sorted out and meeting the definition of done rather rushing over the backlog for finishing things. A poorly implemented story is nothing but a rework that must be done going forward. Often time is a constraint, so depending on the relative importance of stories teams should look for a qualitative implementation of the solution they propose. Stories on the top of the backlog must be developed with the best solution possible and as we move down the backlog there can be some compromise over quality of the solution but not on the definition of done. However it is better to leave a story rather to implement it quite poorly.
- Stand ups are sacrosanct. They are meant to be a shared platform, and not just your solo opportunity to tell others what you did and what's next to follow. People should try to listen to what's happening around rather checking their checklist for accomplished activities.
Sanctity of stand ups must also be maintained in terms of time-lines. We often are unable to resist the temptation of starting a discussion given a point but that must be avoided.
- Agile teams are quite small and in such a situation people can often be judged on the basis of their stand-up updates. So should you say we were banging brains against each other or I did nothing yesterday or we re-factored the code and now we do not know what is what ? These would create an awkward situation with the new ranger.
- Planning meetings are bound to be brain-whacking and exhausting. Sitting in the chair for 4 long hrs and deciding on the story points feels like playing roulette. The numbers will decide what will go in the sprint and what will not but how to decide the number when you had no experience at all? You are bound to lose.
But these numbers are meant to be ballpark figures which can go wrong, that does not mean you start playing roulette for estimations . The numbers will give you some order in few sprints down the line, giving an idea about the total amount of work that can be picked up.
- Assigning story points is a complex task and should be done in phases. Teams internally should first analyze the stories and have discussions with the product owner to get a clear view of what is required. After having a clear view, a broad level technical discussion should be done considering the best possible solution that can be implemented. Failure to do so will result in ambiguity in terms of how the story should be implemented and thus a flawed estimation. Let us say that if a story touches a new unexplored domain then it should to be quite complex, even if it is simple task. Even in the known domain depending on the technical solution the story can be quite complex.
Stories can be estimated best when they are crisp i.e. a clear view of what is required with some details of how should be done. The product owner cannot supply crisp stories. They can only be made by having discussions with the team. So teams must spend some amount of time in order to gear-up for the next sprint.
- Retrospectives look like speeches given at the Poll time. We should have done this or that, and in the next sprint we can do this or that, but nothing happens if some tasks are not taken in the sprint. People can express their comfort or discomfort with different aspects but teams should focus on getting some concrete tasks out of the retro points else the situation will always remain the same.
Here are some of the preparations that one can do before plunging into Agile:
- Get in sync with market technologies especially testing tools like JUnit, Fitnesse, EasyMock etc. Moreover people should constantly strive for better solutions and so look out for new tools and new ways of improving processes, and look for new frameworks or new design ideas and patterns for recurring problems.
- “Practices of an Agile Developer” by Venkat Subramaniam and Andy Hunt is a must read for every developer plunging into Agile.
- Read “Craftsman series” on objectmentor.com and “Clean Code” by Robert C Martin. You will not give them much priority at first but after you get a handful of pain you will know your priorities.
- Be open in stand-ups/planning/discussions, people value your suggestions. Say your opinion or ask for help if required, you are a stakeholder in the project.
- Tests are not just for code coverage or quality measure. They also provide a sort of documentation, that is always up-to-date . People can look into the tests and see how to use this piece of code.
- Automate everything in the build at the start of the project. If anything as small as Checkstyle gets left out then it will have the same fate as in any other model, mostly talk and very little action. When you realize this it is often too late for any damage repair techniques to help.
You have learned to speak A-to-Z so first unlearning it and then learning Z-to-A will not be easy. It will take its share of pain but after you are done you will know it is worth the effort.
So, Go team!! :)
Community comments
Great post!
by Martijn Verburg,
Some concern I have
by Sukant Hajra,
Re: Some concern I have
by Arun Batchu,
Re: Some concern I have
by Sukant Hajra,
The Agile Developer
by Mirko Novakovic,
How effective is Agile in a typical, maintenance/support project?
by kode ninja,
Re: How effective is Agile in a typical, maintenance/support project?
by Tiberiu Fustos,
Re: How effective is Agile in a typical, maintenance/support project?
by Michael Egner,
Re: So many companies seem to be jumping the gun on Agile
by Tien Do,
Re: So many companies seem to be jumping the gun on Agile
by Jeroen Kok,
Why must we following a stated process exactly?
by David Louis,
Pairing advantage
by David Poole,
Re: Pairing advantage
by kode ninja,
Great post!
by Martijn Verburg,
Your message is awaiting moderation. Thank you for participating in the discussion.
This is exactly the sort of in the trenches post that we all need to read and take on board, thanks for sharing your experiences, much food for thought there.
Cheers,
Martijn
Some concern I have
by Sukant Hajra,
Your message is awaiting moderation. Thank you for participating in the discussion.
The approach you're recommending is somewhat the canned way that surfaces popularly. I believe canned Agile methodologies are probably better than canned Waterfall methodologies. But there are aspects of your analysis that frustrate me. My worry is that they falsely support the canned methodology, and fail to highlight the underlying principles.
First off, I'm not sold on TDD given the way you've described it relative to DDT. You ask, "why make a test if you know your code will work fine?" My recommendation is to not write such tests. Tests that "lock down a system" without any added learning/teaching are a huge waste. If a BDD-style specification of a unit is more verbose, more state-based (less declarative), and most importantly less comprehensible than the tight 20 lines or so that it's testing. . . the test is just needless double-booking. If that unit needs to change, the test is only telling us what the unit does, not why it does what it does. So we end up just changing the test and code atomically and blindly. We need to test at the right level of learning. Which gets something that is unstated in your exposition, but that feels implicit in the discussion; I believe you are engaging religiously in unit testing with test doubles to drive your TDD. I'm in violent opposition of this strategy. I do a share of unit testing, but only for units that are actually interesting to test. But there's nothing stopping you from doing TDD with an integration test to cover underlying units. This gets us past the dilemma of having unproductive tests. But it doesn't help us understand why TDD is any better than DDT.
When people ask me what "Agile" is, I tend to focus heavily on a small set of core concepts: working iteratively in small batches to provide opportunities of increased feedback, customer focus, and reduced waste. When I look at Red-Green-Refactor (RGR), I get a process that kind of forces this upon us. But does it come at a cost? I believe it does. First off, I believe RGR complemented with pairing can lead us down a behavioral trap and designs that I claim suffer from TDD-incrementalism. The problem I have with RGR+pairing is that it only promotes refactoring in the small-scale. If two people have invested time in constructing a design with RGR-style TDD, then we have two people working hard on both tests and production code -- that's four times the work, relative to one person, just writing some production code. Sometimes, we go down a design trap. My assertion is that programming needs play to suss out good and elegant designs. With RGR+pairing, people often suffer from play-guilt. In many ways, the first design sustains, especially when bolstered loosely by abstract terms like "evolutionary design" and "YAGNI." Now this problem is not an absolute derivative of RGR+pairing. But in my experience, it's a steep uphill climb culturally to get people to deviate strongly from the current design within the context of something as heavy-weight as rigorous RGR+pairing. It's a social problem that the process just doesn't address well. So I'm concerned when you say "pairing is best done while following TDD."
Finally, I can't understand why your story estimations and planning meetings are so tedious. Routine four hour meetings should be needless, certainly for something like planning and estimation. At my current job, 1-week sprints/iterations have addressed a lot of the problems we might face with overloaded planning and estimation exercises. But shorter iterations don't solve everything. People just need to learn to relax and not get yanked around by a prescribed process. Also, you claim "assigning story points is a complex task." However you follow that claim with a discussion that I think conveys a different message, one I agree with a lot. I believe we should rather be saying, "estimation should not be a complex task; focus on the quality of decomposing your work and estimation will be a much easier task (trivial/easy/harder)."
Okay, so it's easy to expound deeply on disagreements. I wanted to also mention that I'm in agreement with a lot of what you said. In particular, I like what you wrote about stand-ups, retrospectives, and learning/unlearning.
Re: Some concern I have
by Arun Batchu,
Your message is awaiting moderation. Thank you for participating in the discussion.
Sukant, You seem to have passed the gate of 'big' Agile. Most people new to Agile will need to discover what agility is about through constraints before they can break the shackles and pass the gate. I really like your reply and thank you for it.
Re: Some concern I have
by Sukant Hajra,
Your message is awaiting moderation. Thank you for participating in the discussion.
I've heard this reply before in a couple of forms. A more colorful expression is "Shu Ha Ri." I first heard the term from Alistair Cockburn, who gets special attention as one of the original authors/signers of the Agile Manifesto. On top of that, we now have this association of something as new as Agile techniques with something as rich and refined as Japanese martial arts. Unfortunately, this can unevenly weight certain arguments in a discussion, since many of us get a bit star-struck by the mysticism of mastery.
So here we are, people that have "passed through the Agile gate." And we're in a position of great responsibility to help those that haven't. But do we merely run those newer practitioners through the same route gauntlet that we learned from? I don't think Shu Ha Ri is a useless analogy, and maybe I'm in a Ha phase of my own career (I don't want to commit to that assertion). But how long should Scrum+XP+RGR be the first-tier process that we ask everyone to "obey" (in the spirit of Shu). As educators, can we do better?
I also feel strongly about this line of reasoning because of my own learning trajectory. I moved from a Waterfall environment rather quickly to an environment that was doing far less process and overhead from what I hear from others. But was my learning impeded by not having seen long 4-hour planning meetings? Should I really have seen two-week or three-week Scrum iterations before doing something "progressive" like a one-week iteration? Furthermore, I was encouraged to study TDD, but not forced to practice it. Those that had more experience made themselves available, but ultimately, they wanted software that satisfied fundamental criteria, regardless of methodology. I'm not even saying that the process I learned from was the best, but I feel I got good exposure to Agile ideas without any obligatory pain.
This same argument applies to almost every variant of education. How many times do we rail at academia for things like an underemphasis on design and test (and that's just a small sample of typical complaints). As another example, just because we all started with procedural or even object-oriented code doesn't mean that the next generation should too. Maybe Scheme or Haskell is a more proper introduction, before young minds are corrupted with mutable-state paradigms and needless coupling of data with functions.
Even martial arts katas change shape and form through the decades as different masters tune the exercises (or create new ones). Anyone that's past Shu is obligated to help augment the strategies with which we teach, or at least promote discussion.
The Agile Developer
by Mirko Novakovic,
Your message is awaiting moderation. Thank you for participating in the discussion.
Goog article!
We have written our thoughts a while ago in two blog entries
The Agile Developer: blog.codecentric.de/en/2010/03/the-agile-develo...
The fairy tale of the agile developer: blog.codecentric.de/en/2010/03/the-fairy-tale-o...
How effective is Agile in a typical, maintenance/support project?
by kode ninja,
Your message is awaiting moderation. Thank you for participating in the discussion.
So many companies seem to be jumping the gun on Agile. I've seen "Agile" being followed in a recent Java/J2EE support project that a major offshore IT Service Provider is doing for a big Bank and from the time Agile has been introduced, the client has literally screwed the offshore team's happiness. I mean how many requirement changes can you have with an enhancement coming in at the start of a Sprint, in a maintenance/support scenario where most of the application is pretty much there? Before you know it, you're nearing the end of the Sprint and the Functional Spec is still going thru revisions, well past the Sprint planning stage. Finally, if there are problems delivering *on time*, the vendor gets lashed again for not meeting desired SLAs, Quality etc etc.
Most clients are simply abusing Agile as a means to get more work done for the same amount of dough within the same amount of time, coupled with air-tight monitoring about what time the offshore team is spending where.
I'm not even sure something like Agile makes sense in this kind of scenario.
Re: How effective is Agile in a typical, maintenance/support project?
by Tiberiu Fustos,
Your message is awaiting moderation. Thank you for participating in the discussion.
I wonder how many readers of your post are smiling while reading this as I did...
It's funny: as "agile" goes mainstream in enterprise IT projects, there is the tendency for each stakeholder to take only the parts they find more convenient for their cause.
For example, business stakeholders are really happy, since they believe they can now change the requirements on every iteration, especially where the organization of a project is still like in the waterfall times - often there is no "product owner", just a bunch of guys writing up requirements into a huge list and throwing it to the IT team...this is what usually is left after people attended some agile conferences or events or read some stories in magazines.
This was just one example. Another thing that comes to my mind are larger projects where system and organisational boundaries are being crossed. In big organisations one does not "develop" interfaces or service contracts between systems/domains in iterations, but rather a "negotiation" process takes place and all this requires quite some up-front analysis/design effort across multiple stakeholders before a single line of code is written.
On the positive side, some organizations reporting successful agile rollouts are radically changing the organization and project delivery itself, even working on alternative career paths for project and line managers (usually converting them in product owners or scrum masters for example). A long way is still ahead - not to forget the entire budgeting and financing processes in large organizations, which are also essentially waterfall.
Re: How effective is Agile in a typical, maintenance/support project?
by Michael Egner,
Your message is awaiting moderation. Thank you for participating in the discussion.
You might have a look at the Kanban approach of software development. A good starting point is Hendrik Knibergs comparison of Scrum and Kanban: www.infoq.com/minibooks/kanban-scrum-minibook. I can also recommend the Kanban book by David Anderson: www.amazon.com/Kanban-David-J-Anderson/dp/09845...
Cheers
Michael
Why must we following a stated process exactly?
by David Louis,
Your message is awaiting moderation. Thank you for participating in the discussion.
I am the sole senior developer at my company and have been making use of some Agile methodologies. I make use of whatever helps me build better applications more quickly. I keep techniques which help and discard techniques which wouldn't make sense on my team of one.
Agile has fundamentally changed how I code. My first large project fully making use of Agile (CMMI Measurement Repository) ended up as an amazingly flexible solution to a project that already failed in the hands of our BI expert. The business users didn’t really know what they wanted. The main reason it turned out so well were the 2-3 week sprints of continuous improvement. In the old days with waterfall I would have done an excellent job documenting everyone’s shared delusion, I mean requirements, on how the application should work. Then I would work diligently on different parts of the application and database design. Usually starting with the easier tasks to built momentum. The end result is an application that usually misses the mark by some margin and since a lot of code was already written changes are more difficult.
I enjoy weekly refactoring my code and webpage design to best reflect what the users want based on demo’s. When the users aren't available it is pretty easy to give myself a demo. After I enter potions of data I quickly notice what doesn't work well and add it to my list of changes. I fully test each iteration myself so the application is always in a workable state. The application turned out quite elegant. I allowed users to define all measurement fields themselves instead of tying them to the database design. Users can quickly modify all fields in the entire application themselves and then use those fields to store CMMI level 3 measurement that have been collected within a number of different categories.
I focus on understanding Agile instead of memorizing it and then I make use of it, when it makes sense. I make use of the idea behind Agile. I'm not concerned with Agile terminology and accurately following the ideal process. Generally accepting parts of Agile has already turned me into a better and faster programmer.
David C.
Pairing advantage
by David Poole,
Your message is awaiting moderation. Thank you for participating in the discussion.
One of our scrum masters has pointed out that in any team you have people with individuals with expertise in certain areas. If you always leave a particular task to someone with that expertise then you create a bottleneck and dependency on that person.
Pairing is a means to enhance knowledge sharing amongst the team and implicit in that is that you rotate the pairs at regular intervals. An expert will always be an expert but over time pairing increases the likelihood that the team will gain a more rounded skillset and to a good average level of knowledge.
By the way, is anyone aware of any good agile DB books/articles apart from Scott Ambler's work?
Re: So many companies seem to be jumping the gun on Agile
by Tien Do,
Your message is awaiting moderation. Thank you for participating in the discussion.
I totally agree, I was also a member of a Agile team last year, in fact I was the team lead and we was in oursourcing business. My customer is a crazy fan of Agile and they asked us use Scrum. Could we do it well? No, we didn't, and both our customer and us were very frustrated, deadlines were missed, and endless overtime. Why could we not do Scrum well? Yes, we couldn't just because none of us had worked with Scrub before, and except I, most of team members were junior and just recruited, I mean they were not working together before. Frankly, we tried our best and achieved a reasonable result, but we also spent about half of time to rewrited our code (but I named it refactoring in our timesheet :)).
Right before our project was started I had read the great book "Scrum and XP from the trenches" and I didn't wait too long to know that we started without any intrinsic requirement for a successful Scrum team. We were new to each other, we wrote buggy code, most of us never did unit testing before, didn't know what TDD is, even someone were new to source control.
Re: Pairing advantage
by kode ninja,
Your message is awaiting moderation. Thank you for participating in the discussion.
I agree. Agile or no Agile, this approach has tremendous benefits. It acts as a cushion, a sort of Plan B. Like, when someone from the team suddenly has to take an unplanned leave, the process we've followed is that everyone in the team has another person assigned as a backup. Those 2 people, at least, make sure they know in detail what module/ticket/bug etc they're each working on and can more easily take up each other's work if and when required. It's not fool-proof but it generally works out quite well. This, coupled with a good design/code review process generally helps avoid last minute problems.
Another thing is that everyone in the team should know at a high level what everyone else's working on, because it's all too easy to slip into your own work silos. We've used quick 15-20 mins meetings on what each person is doing so that everyone's on the same page. It's generally not the technology, but the business that represents the most challenge and that's where the real dependency comes when some people who've been around longer in the team have more domain/business knowledge and you've the newer folks running up to them every now and again for clarifications. This I think is the bigger dependency and a more serious problem to tackle. And, that's where things get messy when your Specs are constantly changing, causing a lot of PITA, thanks to a half-baked Agile process!
Re: So many companies seem to be jumping the gun on Agile
by Jeroen Kok,
Your message is awaiting moderation. Thank you for participating in the discussion.
In my experience Scrum has never been successful when it was imposed on the team. On the other hand if it's the teams own initiative to adopt scrum, the changes of success become higher. But it still can fail, because a team is not an island. It still has to interact with its stakeholders or other teams. This is where the friction becomes appearent. Even if your customer is enthousiastic about Agile and Scrum they can have different expectations as kode ninja already mentioned.