Mixing Agile with Waterfall for Code Quality
The 2014 CAST Research on Application Software Health (CRASH) report states that enterprise software built using a mixture of agile and waterfall methods will result in more robust and secure applications than those built using either agile or waterfall methods alone.
Data from CAST’s Appmarq benchmarking repository was analyzed to discover global trends in the structural quality of business application software. The report explores the impact of factors such as development method, CMMI maturity level, outsourcing, and other practices on software quality characteristics that are based upon good architectural and coding practices.
You can download a copy of the executive summary of the 2014 CRASH report (registration required) from the CAST website.
InfoQ interviewed Bill Curtis, Senior Vice President and Chief Scientist at CAST, about the research done by CAST, structural quality factors, and mixing agile and waterfall methods.
InfoQ: For readers that do not know CAST, can you describe what CAST is?
Bill: CAST (www.castsoftware.com) has developed a technology called the Application Intelligence Platform (AIP) for analyzing the structural quality of software in large multi-layer, multi-language business systems. Structural quality consists of the non-functional, internal attributes of a system such as Robustness, Performance Efficiency, Security, Changeability, etc. CAST AIP detects over 1200 violations of good architectural and coding practice at both the system and code unit levels in 28 different languages such as Java, .NET, C, C++, COBOL, PHP, and ABAP. The information and measures resulting from the analysis provides developers with the location and severity of weaknesses, as well as providing executives with visibility into the risks and costs of their application portfolio.
InfoQ: Why is structural quality important?
Bill: Structural quality represents the engineering of an application—not whether it computes an answer correctly, but whether the architecture and code is constructed in a manner that avoids crashes, unauthorized access, data corruption, inability to scale, excessive complexity, and similar problems. Damages when these problems strike business critical applications can run into the millions, and in the worst cases they have made software quality a boardroom issue.
As Diomedis Spinellis states in his excellent book Code Quality, structural problems are very difficult to detect through traditional testing, and the majority of test cases are designed to check functional correctness. The most damaging structural defects frequently result from interactions among components in different architectural layers where assumptions about the interactions prove incorrect. These architecturally complex defects are rarely detected with unit level static analysis. Research has shown that although they rarely account for more than 10% of the structural problems in a system, they consume over half of the rework effort because of the number of files involved.
InfoQ: Which of the structural quality factors pose the biggest problems in organizations?
Bill: While I can make a case for several structural quality factors, Security and Reliability are getting the most attention since breaches and critical outages seem to be reported daily. Embarrassingly, we have known about many of the most frequently exploited Security weaknesses such as SQL injection for years. With current advances in detection capabilities, why haven’t these weaknesses been removed? For many applications, especially if they are customer facing, Reliability is critical for avoiding outages during peak hours and recovering quickly when they occur.
InfoQ: Why did CAST do this research on application software health?
Bill: Many of our customers submit the results of their application system analyses to our Appmarq Repository so that they can benchmark their structural quality scores. The anonymized data provide a rich source for research on structural quality and the factors that affect it. In addition to the vast scientific value of these data, we want to show customers how they can use their internal data to answer questions that drive improvement. We are trying to get IT to understand that analyzing the structural quality of their products is just as important as it would be if they were in manufacturing. Continual improvement must involve the product as much as it has involved the process and method.
InfoQ: How was the research done?
Bill: We drew a sample from Appmarq of all applications containing at least 10,000 lines of code (LOC). The 2014 sample consisted of 1,316 applications from 212 organizations in 12 industry sectors primarily in North America, Europe, and Asia. The median size was 128,000 LOC, but the mean was 470,000 LOC since 143 applications were over 1 million LOC. Many of the application owners provided descriptive information on location, development method, CMMI Level, and other factors. In some cases it is difficult to compare analyses across languages and technologies because of differences in the types of structural violations that can occur. Just over 40% of the applications were written in Java-EE, and we had enough descriptive information on them to support the analyses in the CRASH Report.
InfoQ: One of the conclusions is that "Across all [structural quality characteristics], a mix of Agile and Waterfall methods produced higher scores than either Agile or Waterfall methods alone". Can you elaborate?
Bill: Consistently across all the structural quality characteristics—Robustness, Performance Efficiency, Security, Changeability, etc.—the mix of Agile and Waterfall methods had significantly higher scores. In fact, for Robustness and Changeability ¾ of the scores for the hybrid methods were higher than the median for Agile or Waterfall methods used alone. In addition, there was less variance in the structural quality scores on the hybrid methods. Remember, these are primarily large business critical applications, so late corrections to the architecture are very expensive and often constrained. Believing that a satisfactory architecture for such systems will emerge across numerous Agile sprints can be risky. However, not evaluating architectural decisions until system testing in Waterfall projects is also risky. Good structural quality was most often observed when the early architectural analysis characteristic of Waterfall methods was combined with continual testing of the emerging software in Agile sprints. We did not see significant differences between Agile and Waterfall methods in structural quality, it was really their combination into hybrid methods that made a difference.
InfoQ: I also noticed some findings on sourcing and process maturity?
Bill: Consistent with previous CRASH Reports, we did not find any differences between applications that had been developed in-house or outsourced. The major factors affecting structural quality appear to be something other than the sourcing of development.
We did find that the structural quality of software built in CMMI Level 1 organizations was significantly lower than that developed in more mature environments. When developers are ensnared in a Level 1 death march, they are working overtime against unachievable schedules and consequently produce code with many more structural weaknesses that they do not have time to detect or refactor. Getting control of commitments and baselines allows developers to work at a more sustainable pace and produce . I have even heard developers on Agile projects complain about adding numerous stories during a sprint under pressure from the business thus overcommitting the team’s work. Sprints are best served frozen, not thawed.
The objective of CMMI Level 2 is to establish control over commitments and baselines on each project or application so that developers are protected from perpetual firefighting and chaos, enabling them to work at an orderly and professional pace. Embracing change does not imply embracing chaos. We did not see differences between applications in CMMI Level 2 and Level 3 organizations because the transition between Levels 2 and 3 is primarily an organizational initiative to establish common practices that can be adapted to different projects in order to develop consistent understanding, organizational learning, and an economy of scale.
InfoQ: Is there any advice that you want to give to enterprises that have implemented or are implementing agile software development?
Bill: These results confirm with hard data what many Agile thought leaders such as Alistair Cockburn, Scott Ambler, Dean Leffingwell, and Todd Little said for years. Agile projects should adjust their practices to the size, complexity, and uncertainty of the application challenges they face. The larger more complex the challenge, the more removing some of the architectural uncertainty early helps avoid rework and technical limitations in later sprints. Agile projects should also be evaluating system level structural quality after several of the builds in each sprint in order to detect problems between different subsystems or layers of the application.
Some uncertainty is functional while some is structural. Agile Methods have focused on managing the functional uncertainty arising from the changing needs of customers. Structural uncertainty is often relegated to ‘emergent design’. However, refactoring architectures in large systems can be excessively expensive and slow the pace of delivering valuable software. So look at development as a collection of practices you can adjust and balance to manage the levels of structural and functional uncertainty in each system. Be agile with your methods.
'Found' Architectures and Performance
While such 'design-last' risks exist, a persistent and pernicious problem with agile/waterfall hybrids is QA is invariably waterfall, still heavily manual in many cases, and gates production releases. As a consequence, you typically see little real incentive to up maturity levels of test automation and test data management.
And even though continuous deployment isn't coming to [most] large enterprises in our lifetimes, the CD/DevOps movement does exhibit a number of traits large enterprises shouldn't dismiss out of hand along with it. Foremost among those are discipline, walking the talk, and highly integrated QA functions - i.e. they've invested the time, money, and sweat to have it all lined-up and wired end-to-end without gaps, without intra-IT culture wars, and without a myriad of excuses as to why the whole end-to-end process is so long, halting and painful.
Valid data, Wrong Conclusions
Firstly CMMI Level 2 is defined as having a repeatable process at the team level. It makes no mention of waterfall. An understanding of the objectives behind the KPA's that represent this level of maturity points to good engineering practices and team discipline. Any team with a disciplined, methodic approach will satisfy the goals of the CMMI Level 2 KPA's.
Secondly quoting Alistair Cockburn in support of the conclusions reached is a misnomer. What Alistair advocated is changing process in response to team size, acknowledging that the communication overhead grows in a nonlinear fashion as the number of team members increase. This normally means moving beyond informal osmotic communication mechanisms which work best within a small intimate collocated team, to something more formal, say a wiki and written documentation for a large distributed team. This has nothing to do with software quality. Designs communicated via documents can be just as poor as designs communicated less formally, either through code or face to face at a white board.
Interestingly it has been the Agile community that has focused the most on overcoming communication hazards within a project. With Agile teams using techniques like virtual presence when not collocated, videoing design presentations, photographing white boards etc. The waterfall community still largely rely on documents, which are inherently low bandwidth and struggle to capture the ephemeral nature of software design.
The last misconception expressed is that structural code quality is somehow better under waterfall. How did they come to that conclusion? Did they measure it? The ultimate measure of what I describe as internal code quality is just how maintainable the code is. Namely how easy it is to change. In a waterfall context there is no way of knowing this, whilst a team doing iterative development will find out how easy their code is to change in the next iteration. Many Agile teams make use of their iteration velocity as a signal that perhaps code quality is less then it should be. Also retrospectives are an ideal opportunity for a good team to reflect on just how much technical debt there is and whether they are delivering sustainably.
So what can we reliably conclude? Well that disciplined teams produce higher quality software then undisciplined teams. Also that the CMMI Level 2 KPA's can be used as an indicator of how disciplined a team is when it comes to applying good engineering practices, like version control, unit testing, peer review, requirements management, planning, etc. These things are important whether you do Waterfall or Agile.
What you will find is that for good Agile or Waterfall teams, that the basic engineering practices will differ, but the goals being addressed are the same.
Developers need to know more in order to assume responsibility for sec & reliab.
Author, High-Assurance Design
Waterfall does not get the architecture early (enough)
Valentin Tudor Mocanu
If it is needed a process approach that is build to be architecture-driven (more then Waterfall and more than Agile), that it is Unified Process. UP does not wait to gather BUFR with details, but it is focus to gather just enough requirements for define the architecture.
The problem with UP is that pure architecture-driven approach is not so pragmatic for most of the software projects. A balance between this approach and more Agile methods could be more useful.
The real useful "hybrid" it is between UP and Agile methods, but in most of the cases that does not mean to change the Agile life-cycle, but rather making architectural concerns more explicit. All Agile methods recommend an early focus on architecture, but unfortunately, in most of the cases it is not specified in an explicit manner.