Agile, Architecture and the 5am Production Problem

Posted by Deborah Hartmann on Jun 25, 2007 01:06 AM

Community

Architecture,

Agile

Topics

Delivering Quality

Tags

Refactoring ,

Complementary Practices ,

Criticism ,

Testing ,

TDD

RelatedVendorContent

Agile Development: A Manager's Roadmap for Success

The Agile Project Manager

Effective Management of Static Analysis Vulnerabilities and Defects

Free $40 SOA Demystified Book Offer

Ensuring Code Quality in Multi-threaded Applications

Related Sponsor

VersionOne is recognized by Agile practitioners as the leader in Agile project management tools. Companies such as Adobe, BBC, CNN, Dow, HP, IBM, Sony and 3M have turned to VersionOne to help deliver greater value to their customers.

1. Back to top

   Just enough design up front

   Jun 25, 2007 2:36 AM by Arnon Rotem-Gal-Oz

   I agree. On decent sized project bottom-up design techniques such as TDD are not enough. You also need to do some some up front work and think about quality attributes. I think that architectural retirements expressed as user stories should (so that implementing them would also bring business value) be emphasized in the fist few iterations.

   Reply

2. Back to top

   Happend to me too

   Jun 25, 2007 7:18 AM by Erik Pischel

   Hi, nice article. The same once happened to me, too. See http://www.epischel.de/wordpress/?p=45. I agree with you that most failure points in the field are integration issues - in particular when practicing TDD. In most cases you would try to mock up external systems. And even if you use the real one, you probably won't test in your production environment. In an other, much bigger project, the first failure during load-testing was (against all odds) a network switch that failed under heavy network load only. Should we take that into account when developing software? When human lifes depend on it - yes. But otherwise?

   Reply

3. Back to top

   Re: Just enough design up front

   Jun 25, 2007 7:25 AM by Julian Browne

   Me too. I've never believed (or found) that architecture and agile are mutually exclusive. Overblown and over-engineered architectures, sure, but then these should have no place in the waterfall world either. I do have some sympathy for the "let the architecture emerge" school of thought, and certainly no architecture should be so inflexible so as not to change, but properly skilled architects (who are grounded in something more technically relevant than just powerpoint..) ought to be able to lay down some appropriate guidelines to meet NFRs and shifting requirements that save buckets of time later. The trick is to get 'just enough' in place, and not to get carried away with gold-plating the design when you can't be 100% sure it won't all change in the next iteration.

   Reply

4. Back to top

   Is agile Silver bullet?

   Jun 25, 2007 8:19 AM by Renat Zubairov

   I tend to think that author expects to solve all problems with Agile/TDD/XP methods which is obviously not correct. I would agree that integration problems are tend to be very complex once and not possible to handle in "Unit" testing environments, but as usual very important aspect of agile development is forgotten - and this aspect is "evolution". We can't fix/test/predict integration problems with our unit test wherever methodology we were using. It is indeed very complex/impossible to create a fake interfaces that will match 100% interfaces of the integration point, but what we can do with Agile/TDD/XP/(put your agile method here) is to make system evolution simple! I would never assume that TDD will replace normal functional testing and will solve 100% of problems system will have in the production environment, but I strongly beleve that high unit test coverage, big number of automated end2end functional tests will help us to deliver new functionality faster without worring about broken existing functionality. We also have allot of discussions in our company about value of automated unit/functional testing and only one conclusion is feasible for me - all automation tests are by nature regression tests.

   Reply

5. Back to top

   Re: Just enough design up front

   Jun 25, 2007 8:25 AM by Arnon Rotem-Gal-Oz

   Yup. As I posted in my blog I found you need to take a mixed approach, try to come up with a basic a architecture and evolve it with the project: 1. Set the first one or two iterations as architectural ones. Some of the work in these iterations is to spike technological and architectural risk. Nevertheless most of architectural iterations are still about delivering business value and user stories. The difference is that the prioritization of the requirements is also done based on technical risks and not just business ones. By the way, when you write quality attribute requirements as scenarios makes them usable as user stories helps customers understand their business value. 2. Try to think about prior experience to produce the baseline architecture 3. One of the quality attributes that you should bring into the table is flexibility - but be weary of putting too much effort into building this flexibility in 4. Don't try to implement architectural components thoroughly - it is enough to run a thin thread through them and expand then when the need arise. Sometimes it is even enough just to identify them as possible future extensions. 5. Try to postpone architectural decisions to the last responsible moment. However, when that moment comes -make the decision. try to validate the architectural decisions by spiking them out before you introduce them into the project PS Here is the correct link to the post I made on quality attributes - the link my first message is broken.

   Reply

6. Back to top

   Re: Is agile Silver bullet?

   Jun 25, 2007 8:44 AM by Michael Nygard

   It's not so much that I expect TDD or XP to solve everything, but I am a strong proponent of agile methods and the agile values. So, this isn't so much meant to complain that there was a problem we didn't find through unit testing as it is meant to draw a parallel. We (using the "royal we" for a moment) invented and adopted unit testing to solve our own problem of producing buggy code. Here, I see a similar problem. I often here XPers say there should be no architecture up front, that it should all emerge through the practices. On the opposite end of the spectrum, there are the Zachman framework types that want to define the world before any projects can begin. Even on the most pragmatic of agile teams, there's still a kind of connotation that some of amount of up-front architecture is probably necessary, but it's a compromise---a necessary evil. That leaves us wide open to this kind of problem, and myriad others that I've seen. Failures in the white space. Cracks originate in the gaps between boxes. Is there something analogous we could invent to address architecture issues while remaining consistent with agile values?

   Reply

7. Back to top

   Re: Is agile Silver bullet?

   Jun 25, 2007 9:14 AM by Arnon Rotem-Gal-Oz

   I Is there something analogous we could invent to address architecture issues while remaining consistent with agile values?

   As I said above - I think this can be handled within the practices of agile development. if you express architectural constrains as user stories - by demonstrating how the concern is manifested in the application. You can then prioritize and handle it like other user stories (you can look at an architect as a type of a technical product owner).

   Reply

8. Back to top

   Re: Is agile Silver bullet?

   Jun 25, 2007 9:28 AM by Amr Elssamadisy

   I'm not sure I agree that there is any conflict at all. Nobody, to my knowledge, has ever said that Agile or TDD is a silver bullet. But, with respects to this particular example (in the article), I'm not sure it has anything to do with Agile or TDD at all. Production problems happen - Agile or not. The fair question to ask would be 'could I have avoided this problem?' And if your answer is yes, that you could have foretold this problem, the next question would be 'with what accuracy?' By and large, we in the technical community are TERRIBLE fortune-tellers. You will miss things. But if you try to foresee all, you will over-design, over-complicate, and increase your code debt. I think the question should be stated differently - 'What kinds of architectures evolve?' If TDD (via refactoring) is a local improvement - isn't this analogous to a steepest-decent algorithm for design? We know that steepest-decent gets stuck in local minima. Does TDD really give us a good architecture? Does it give us good-enough architecture? Or does the local nature of TDD preclude evolving towards an acceptable architecture?

   Reply

9. Back to top

   Architecture & Agile

   Jun 25, 2007 9:32 AM by Stefan Tilkov

   I'm puzzled -- is there really an Agile school of thought that says "architecture" is something bad?

   Reply

10. Back to top

    Re: Architecture & Agile

    Jun 25, 2007 9:42 AM by Arnon Rotem-Gal-Oz

    I'm puzzled -- is there really an Agile school of thought that says "architecture" is something bad?

    I think that for many YAGNI is just that

    Reply

11. Back to top

    Re: Architecture & Agile

    Jun 25, 2007 10:00 AM by Amr Elssamadisy

    That is a misread/misunderstanding of YAGNI. Actually, YAGNI says no fortune-telling because we tend to be wrong more than we tend to be right. You are within YAGNI if you have an architecture in mind and then wait to evolve it in that direction when the requirements ask for it. This is very similar to Real Options (http://www.infoq.com/articles/real-options-enhance-agility )or the interview with Erich Gamma last year (http://www.artima.com/lejava/articles/designprinciples.html ) where he described how the eclipse team refactors to patterns. Which, of course, brings up Joshua Kerievsky's Refactoring to Patterns work ( http://www.industriallogic.com/xp/refactoring/).

    Reply

12. Back to top

    Production mindset

    Jun 25, 2007 10:09 AM by Robert Merrill

    A few random thoughts. When I heard of refactoring, I remember thinking, "Now I know what architecture is. It's the stuff that's hard to refactor!" I guess that's the art of "just-enough" or delaying decisions--knowing when you have to make them. There are some decisions that do have to be made earlier. One agile value is "don't throw stuff over the wall." I've almost always had to support what I wrote, and that forces a production mindset. I don't want the phone to ring at 5 AM, and if it does, I want the problem to be obvious. So I build in monitoring and logging functionality from the start. I guess I could cover proper behavior of logs and monitors with unit tests. Find a copy of "Writing Solid Code." It's 10 years old, and C-centric, but I learned a ton from that book. Another agile value is "test early and often," and I guess that can include load testing. I like to try and build the simplest-possible feature that spanned all of the components in the architecture, and load test that. If you log and graph CPU, memory, network, and disk I/O on all components, you will begin to see patterns. As you test, monitor various system components and graph the output. You will start to see patterns long before flames start shooting out. If you have underpowered hardware, all the better. You're trying to see where and how the software breaks. Robert Merrill www.ufunctional.com

    Reply

13. Back to top

    Re: Architecture & Agile

    Jun 25, 2007 4:50 PM by Stefan Tilkov

    I consider YAGNI to be a good guideline for a good architecture - i.e., one that is only as complicated as it needs to be. Having no architecture at all, and no-one explicitly or implicitly responsible for it, is a sure recipe for failure.

    Reply

14. Back to top

    Re: Architecture & Agile

    Jun 25, 2007 5:05 PM by Arnon Rotem-Gal-Oz

    I agree with both you and Amr - However, what I see is that a lot of people look at YAGNI as an excuse not to do any forward thinking activity like architecture or design.

    Reply

15. Back to top

    No test environment?

    Jun 25, 2007 10:27 PM by Angus McDonald

    I am the only one reading this who is amazed that you didn't have a test environment that exactly replicated (down to the firewalls used) the production environment? Saying that you CAN'T test your production architecture because you don't bother to is not a good enough answer. There are some things you cannot test effectively, but firewall rules should definitely be the same. Where do Agile Development methodologies recommend that functional testing be done in an environment that does not mimic production? Having said that, I admire your skill at finding the problem, and this is a good write-up of how to do this sort of low-level packet sniffing.

    Reply

16. Back to top

    Re: No test environment?

    Jun 26, 2007 10:08 AM by Michael Nygard

    Angus, You make a great point, and one that I address in the book. One of my major themes is getting grounded and connecting with the actual deployment environment. It's the only way to have true confidence in what you deliver. Most companies will not build exact replicas for their test environments, though. They choose to save a bit of money by eliminating expensive network components like firewalls and hardware load balancers. This is a penny-wise, pound-foolish decision. Whatever money they save on network equipment will surely be lost in production outages. Nevertheless, budgetecture happens, particularly in QA. Sometimes, it's not as much a budget issue as it is a knowledge gap. Development may not know what the enterprise network will be, particularly if development is outsourced. Other times, the network architecture changes late in the game. I've heard, "We can't disrupt the QA environment now! We're too busy getting ready for release to lose a day while you change the network." Of course, what happens then when it does hit the real network? Anyway, I always fight to have the QA network match the production network. About 50% of the time, I win that fight. Cheers, -Michael Nygard

    Reply

17. Back to top

    Re: No test environment?

    Jun 26, 2007 11:54 AM by Amr Elssamadisy

    Excellent. So what does this have to do with Agile? Forgive me, but it is a good article with a misleading title. It seems to me you are only perpetuating the misunderstanding of Agile methods.

    Reply

18. Back to top

    Problem vs. Architecture, Agile

    Jun 26, 2007 1:45 PM by Wayne Mack

    I guess I don't see the mapping between the problem encountered and either Architecture or Agile. I am not sure what the authors proposes to do differently. An underlying concept in Agile is that not everything can be forseen. I am not sure what could have predicted this problem nor discovered it quicker than actually fielding the software. It is only by fielding quickly that we can discover what we don't know.

    Reply

19. Back to top

    Re: Problem vs. Architecture, Agile

    Jun 26, 2007 5:11 PM by Jim Standley

    There were several references to "no architecture". That's simply not possible. Everything has one whether you gave it any thought or not. What Agile doesn't do is try to design and build it before doing any other coding, which often involves trying to predict every darned thing the application(s) will ever need. Just build enough to support today's needs, be mentally and technically prepared to add or change or remove bits as the app evolves. To the original post - this was a fascinating story. That detective work would be beyond me and the project teams I know in my company. We do some "unplug the network cable" testing for failures, but this situation would have been way hard to predict and test for.

    Reply

20. Back to top

    Re: No test environment?

    Jun 30, 2007 11:58 AM by Oso Rojo

    Have to agree with this one. There's no evidence here that a more "architectural heavy" approach would have inexorably discovered this flaw, nor is there any definitive proof that "Agile" and not "gross developer error" was the root culprit. It's a great case study right up until you start Agile Bashing. At that point, this article becomes FUD, pure and simple. Michael, please stop spreading it.

    Reply

21. Back to top

    Re: No test environment?

    Jul 1, 2007 9:45 AM by Michael Nygard

    "Agile Bashing" and "FUD" are both very incendiary terms... not conducive to conversation at all. Neither is asking people to be silent. I will attempt to respond to the substance of your comment rather than the terms you've put it in. My purpose here is certainly not to bash Agile. I've been a proponent and practitioner since before the moniker existed. I was doing unit testing, pairing, refactoring, and short iterations back when it was all just called "XP" or, more generally, lightweight methods. Several years back, I even quit my job to start a company explicitly built on agile methods. More recently, I spent an intense year in a fully agile Scrum/XP project. In the first 8 months, we delivered what the client had failed to deliver over the previous 2 1/2 years. In the next 4 months of my time on that project, we did six additional releases. I'm speaking from within the Agile community, not from outside of it. I can see that several people have misread my intention. I blame myself, as the author, for not being clear enough. I will try to make myself more clear here in the comments. I don't attribute the failure here to a "failure of agile". Nor do I expect that agile methods, as formulated today, should have prevented this problem. What I am presenting is a problem that has two very difficult characteristics:

    • It's virtually impossible to predict.
    • It only occurs in the actual production environment.

    I'm drawing an analogy to unit testing. In days past, people thought it was impossible to test software within the development environment. Testing was done in a test lab, by testers, using testing tools. We have rewritten those rules. We now understand that unit testing won't catch every bug, but it sure catches a lot of them. (And, yes, unit testing also motivated changes in the way we design the code itself. We don't mind that much, since the design changes needed for unit testing are all "virtues" that we endorse anyway: decoupling, isolation, single-responsibility, and so on.) Furthermore, we use automation to solve problems once and keep them solved. So, once a bug is discovered, we write a test to verify the bug. Once we fix the bug, the test acts as a barrier to keep the bug from re-emerging. We use our suite of automated tests to "nail down" the functionality. (And, they allow us to retain existing value while incrementally adding more value.) As a practice, automated unit testing supports many positive virtues. We don't expect it to prevent or solve every problem. There are known challenges---areas that work, sort of, but not very well---mostly around databases and GUIs. Despite those challenges, I would never give up unit testing. My point in this piece is to ask a question, not to bash anyone or anything. Can we think of a practice, consistent with agile values, that would advance architecture work the same way that unit testing has advanced coding? I am asking this question by using a specific example of a general class of problems to illustrate a difficult, costly situation that I would like to have avoided rather than solved. I'm asking this question because I see a need for more connection to the actual deployment environment: filesystems, servers, networks, databases, etc. There are times and places for isolation, but we cannot always be isolated from the deployment environment. By the way, this "disconnectedness" is not unique to agile developers. I suspect that the best solution to disconnectedness will come from the agile community. The Ivory Tower architects have already had their whack at it---and they responded with even larger diagrams that got further disconnected from the real environment. I very much want to avoid problems like this one, but I've got a hundred stories like this. Some come from agile projects, most come from non-agile projects. Some come from projects with heavy "big architecture up front", some come from projects with incrementally developed architecture. I'm certainly not blaming "agile" for these problems. I'm looking for a solution to them, and for that solution, I'm asking the agile community if we can find a practice that fits our values: incremental, automated, expressive, "just-in-time", self-describing, executable documentation, and enabling. Like software testing, traditional (heavy) approaches to architecture have not moved the needle on the quality gauge. Let's see if there's a way to do for architecture what unit testing did for software quality. Hope this helps clarify my intentions.

    Reply

22. Back to top

    Re: No test environment? (Preventing Failures)

    Jul 2, 2007 11:19 AM by Wayne Mack

    I think the issue is that there will always be unknowns in software development and any test environment will always be just an approximation of the actual operating environment. We can only make our bets guesses at what parts of the physical and operational environments are significant, what needs to be accurately reflected, what needs to be simulated or approximated, and what can be ignored. Yes, we should do the best we can do in testing given real world constraints. The value, though, we get from agile development is risk reduction through reduction in costs sunk in an absolute failure. By having multiple, iterative releases, a failed iteration can be rolled back with only the sunk costs of 2-4 weeks of effort, however, we must plan for and be prepared to rollback. Iterative development moves us past the risks of all-or-nothing one-shot project development. The way to address the risks of unknowns is to push them to the front of the queue and force them to arise as soon as possible. There is no way to plan to avoid the unknown, one can only force it to arise as early as possible allowing time to recover from it.

    Reply

23. Back to top

    How would upfront architectural design have prevented this problem

    Jul 3, 2007 10:46 AM by Steven Gordon

    Really nice article! One thing that was nicely illustrated was that discovering and fixing this problem was a team effort, not just a heroic effort by one "architect". What I do not see is how doing upfront architectural design would have prevented this from occurring (except armed with 20/20 hindsight). It seems to me that the same problem could easily have occurred in a waterfall project. The lack of unit tests and functional tests (and likely code bloat to handle dozens of other potential problems that never do happen in production) in most waterfall projects would have made the set of possible causes orders of magnitude larger. Being less sure that each unit was working correctly and that the system works correctly under normal conditions, discovering the root cause would have been much more difficult. Furthermore, once a fix was determined, establishing that the fix did not break anything else would have also been much more difficult without all those automated unit and functional tests. I do not even see how doing a few iterations of architectural spikes at the beginning of the project would have prevented this anomoly. In my experience, making sure the first story really goes end-to-end forces a slice of each architectural component to be implemented. This gives us the best of both worlds - validating everyone's understanding of the requirements as well as laying out and testing the architectural approach. Steven Gordon

    Reply

24. Back to top

    Re: How would upfront architectural design have prevented this problem

    Jul 18, 2007 9:57 AM by li xiao

    Agree! It's like talk about that hammer is not good because it can't clinch screw, but someone just forget even stone also can't clinch screw.

    Reply

• Older >