BT

New Early adopter or innovator? InfoQ has been working on some new features for you. Learn more

Quantitative Analysis of Agile Methods Study (2017): Twelve Major Findings

| Posted by Donald J. Reifer Follow 1 Followers , reviewed by Shane Hastie Follow 9 Followers on Aug 10, 2017. Estimated reading time: 40 minutes |

Key Takeaways

  • Agile is firmly entrenched as the most commonly used approach to software development
  • Agile methods have reached a steady state in most firms embracing them and productivity improvements are stabilizing
  • Productivity increases are not being realized by those pursuing either large agile projects or enterprise-wide utilization
  • Agile methods improve the ability to meet schedule deadlines
  • Agile quality is better than traditional norms
  • Agile methods continue to provide their users with a competitive advantage

In 2015, Reifer Consultants published a study1 that looked at how widely agile methods were being used and whether they were yielding superior benefits when compared to similar efforts completed using traditional methods (waterfall, spiral, etc.).  Since then, we have analyzed data from 3,000 completed projects generated by 150 organizations located around the world.  The results of this analysis are reported in this technical note whose results are based on 1,500 completed projects that used agile methods and another 1,500 which used traditional approaches.  As expected, projects vary widely in size and complexity.  However, none of the data was over ten years old. This white paper summarizes the twelve most important findings from this analysis.

Finding “1” - Agile Methods Continues to be the Predominate Approach used for Software Development

As detailed in the in Figure 1 below, agile methods continue as the dominant software development approach used throughout the world.  Using Moore’s technology adoption model2, the majority of the 150 organizations polled have adopted agile methods enterprise-wide.  Which of the many agile practices they selected is the topic of the next finding.  Based on the data in the Figure, more than 70 percent of the organizations reporting have been using agile methods operationally for a period of between three and thirteen years.   

Figure 1:  Worldwide Agile Introduction by Number of Organizations by Technology Adoption Phase

Legend

  • EA:  Early Adopters (41) – introduced agile methods during period from 2004 to 2009.
  • EM:  Early Majority (66) – introduced agile methods during period from 2009 to 2014.
  • LM:  Late Majority (33) – introducing agile methods presently.
  • Laggards (10) – considering, but not currently using agile methods.

Finding “2” – Scrum is the Most Popular Agile Methodology 

As shown in Figure 2, the most popular agile method used today continues to be Scrum3 or a Scrum-based derivative like Large Scale Scrum (LeSS)4.  Other agile methods employed, but in much smaller percentages of our survey population, include the Agile Unified Process (AUP)5, Extreme Programming (XP)6, and Feature-Driven Development (FDD)7.  As amplified in subsequent findings, methods used for agile-at-scale (AS) like the Scaled Agile Framework (SAFe)8 and hybrid (HYB) methods have increased in popularity as agile has both been used in larger projects and adopted enterprise-wide.  These findings are not surprising as they are consistent with the trends that we observed in the previous versions of this and our other reports9,10, 11.

The most popular agile-at-scale method is SAFe.  Its utilization exceeds its competitors by a factor of three.  The most popular hybrid methods reported include those that embrace traditional (incremental, waterfall, etc.) and/or Kanban12 and lean13 concepts.  These hybrids are used by organizations striving to reduce waste and achieve consistency enterprise-wide.

Figure 2 - Methodology Usage by No. of Organizations by Size of Project

Notes

  • Small Project (567):  project that delivers a product can be developed by a single agile team.
  • Medium Projects (581): agile project that uses 2 to 5 teams at same locations to develop products.
  • Large Projects (352):  large project that uses 5 or more teams often at different locations to develop products.

Legend

  • AUP – Agile Unified Process            XP – Extreme Programming              FDD – Feature-Driven Development
  • Scrum – Scrum and derivatives         A-Scale – Agile at scale methods        Hybrid – Mix of methods

Finding “3” – Usage of Hybrid Methods and “Agile at Scale” Methods on Large Projects are Nearly Equal

Scaling of agile methods for large projects is a difficult task14.  Teams are large, efforts difficult, and deadlines tight.  Multiple releases often have to be worked in parallel across geographical distances in distributed environments.  The outputs of these releases must be planned, coordinated and synchronized in such a manner that development. integration and test flows naturally.  Based on the recent data shown in Figure 2, the organizations pursuing such large developments are using either an agile-at-scale (48%) or a hybrid (52%) methodology. This represents a dramatic change from two years ago when agile-at-scale usage was about half of what it is today. Such growth has been propelled primarily by the fan-out of agile methods enterprise-wide.   However, use of hybrid agile methods should not be discounted especially when they are adapted for use by large interdisciplinary organizations. Many of those who have succeeded with hybrid methods use them to embrace project management concepts to (1) plan, coordinate and synchronize team deliveries, (2) facilitate communications flows up, down and across distributed groups, (3) enable tracking and reporting of progress and performance, and (4) insulate the development teams from interference by outside groups (customers, senior management, etc.).  Because many organizational roles and responsibilities change when agile methods are used across the enterprise, the organizational infrastructure, management discipline and processes that these project management methods provide make it easier to manage agile-at-scale projects. 

Finding “4” – Agile Reversal Rate Has Decreased

As shown in Figure 3, the reversal rate for those trying to introduce agile methods into the mainstream has dropped considerably.  During the early part of the last decade, the reversal rate averaged about 25 percent.  As agile methods gained wider acceptance and matured, the reversal rate improved and converged at about 15 percent.  This trend seems reasonable when you investigate the approaches that are being used for replacement.  Based on our data, only 10 percent of those dropping agile methods reverted to traditional methods.  The others moved to the use of another agile, hybrid, or home-grown approaches to fill the void.

Figure 3 - Agile Method Reversal Rate by Calendar Year

  • : based on our analysis of the data, the top five current concerns relative to agile reversal rates include:
  • Recognize that reversals occur when adopting agile methods.  Primary reasons for reversals include poor preparation, mismatches, and a lack of senior management support.
  • Build skills and competencies along with a pace as the project unfolds.
  • Keep track of performance during short-term so that you can make adjustments should issues arise that impede long-term success.
  • Be prepared in case of problems with a Plan B, i.e., another agile approach to fill the void.
  • Be patient.  The old proverb that Rome was not built in a day holds true.

Exploitation Measures:  some suggestions for avoiding reversals include:

  • When introducing agile methods, be prepared.  Gather support up and down the organization.  Put those practices that match and mate to work with your existing process infrastructure.  Train your staff, update your processes and ready the organization for change.  Finally, be prepared to fan the methodology out once you are successful.
  • As you introduce, measure to see how you are doing and whether you will be successful with your end game. Be prepared to refine and adapt your approach should you find that you are not accomplishing your goals.
  • Understand that it is as easy to shift to another agile method as it is to revert to a traditional approach especially if you have paved the path to make the conversion.
  • Also realize that you are not finished once the initial project successes have been achieved.  You will need to fan the resulting methodology out first throughout your software organization.  Then, if desired, the methodology will need to be fanned out organization-wide.

Finding “5” - Agile Productivity Gains Have Stabilized

Based on our current data, agile productivity, as measured by outputs/units of input (equivalent source lines of code or function points/staff-month), has stabilized.  This is not unexpected as the methods have reached a steady state in most firms embracing them.  In other words, gains made during adoption tend to level out once the methods reach an operational status.  However, the average software productivity gains as compared to those being made by those using traditional methods still are exceptional.  For example, based on our data set of 1,500 traditional and 1,500 completed agile projects, the average productivity gains being achieved during development by groups using agile methods during the past three years were from 7 to 12 percent better per year than for those firms using traditional methods.  Such improvements are still notable especially in large organizations where these gains achieved using agile methods translate into savings of thousands of person-hours.

It is interesting to note that these software productivity increases are not being realized by those pursuing either large agile projects or enterprise-wide utilization.  These productivity gains, while still advantageous, drop to just 3 to 5 percent better than their previous norms.  This is not unexpected.  When you investigate the situation, you find that the firms implementing agile-at-scale and hybrid methods are using many of the same traditional management techniques that they used in the past to handle project and enterprise-wide management issues.  As a result, they replace instead of reduce management overhead and bureaucracy with a rebranded set of practices.  For example, they continue to use hierarchical organizations instead of flattened ones that employ self-organizing teams.  As another example, they use earned value, traditional metrics and progress reviews to track performance instead of emphasizing simplified reporting.  In contrast, those firms that use Kanban principles and lean approaches as part of their methodology tend to make better productivity gains because they focus their energy on getting rid of waste and inefficiency.

  • : based on our analysis of the data, the top five current concerns relative to improving productivity when using agile methods include:
  • Product versus project management conflicts that can raise questions as to who’s in charge.
  • Emphasis often placed on dogma instead of practicum often impedes the orderly fan-out of the technology.
  • Extra work often occurs when practitioners rework the code between releases instead of refactoring15 it.
  • Many agile teams, continue to have difficulty in building rhythm and maintaining momentum even when properly paced by a Scrum Master.  The situation worsens as projects get bigger and when agile and DevOps16 practices are mixed. We will release a separate report on the impact of DevOps on software development later this year.
  • Measures of value that drive agile developments continue to be vague and non-quantitative. 

Exploitation Measures:  some suggestions for exploiting agile productivity gains include:

  • Understand the variables that impact productivity (architecture stability, product complexity, staff skills and experience, etc.) improvement and address them in your practices.
  • Place attention on team building and approaches that allow you to set a realizable pace, build rhythm and maintain your momentum.
  • Gather productivity measures and metrics and use them to help make needed adjustments to your practices.
  • If your product involves more than just software, strive to develop a single engineering process by extending agile practices to encompass aligned disciplines like digital design, firmware and systems engineering. 
  • Focus agile on developing those applications where they can realize their value proposition.  Measure value whenever possible as quantitatively as possible using financial measures like return-on-investment and cost/benefits.

Finding “6” - Agile Cost Savings Have Also Stabilized

Based on our current data, agile costs, as measured in terms of $/unit of output, continue to be between 5 to 12 percent per year cheaper than the plan-driven or traditional project norms.  The data reveals that the cost reductions realized using agile methods during the past three years have stabilized as firms have fanned the use of agile methods out throughout the enterprise.  Cost reduction in this context often means that jobs of comparable scope can be done with less effort overall.  It does not mean reductions in staff.  As expected, gains made on agile-at-scale projects have been similar to those of smaller developments even though there were increased costs related to managing these larger jobs. Again, long-term cost reductions are greater when firms put best agile practices to work operationally across the enterprise.  As with software productivity, agile expenditures vary widely by application domain and those factors that influence costs (workforce composition, degree of geographical distribution, task difficulty, overhead rates, degree of bureaucracy, human resources, etc.) seem to be a function of the team’s experience and the manner in which agile methods are put to work.  However, realize that increasing productivity does not mean that related costs decrease.  For example, productivity and cost can increase at the same time when teams pursue implementing the wrong requirements.

  • : based on our current analysis of the data, the top five most current concerns relative to cost include:
  • Roles and responsibilities of team members need to be clear.  Else, costs will increase because confusion over who does what and when will reign.
  • Investments in infrastructure, facilities, equipment and software tools that support the use of the agile methods that you select including those for DevOps need to be identified and made.
  • Emphasis needs to be placed continuously on simplicity.  Else, costs will go up as your team flounders.  Key personnel have to be reminded and mentored on ways to realize this goal.
  • Mismatches that exist with supporting and DevOps processes (quality assurance, configuration management, human resources, program/project management, etc.) need to be eliminated.
  • While use of subcontracting/outsourcing can reduce labor rates, their use needs to be tempered.  The negatives associated with such personnel often outweighs the benefits especially when such resources must be supervised directly by your people.

Exploitation Measures: some suggestions for exploiting agile cost advantages are as follows:

  • Understand the variables that impact cost (they differ from productivity) and address them as you move to harvest available cost reductions.  For example, focus on reducing overhead rates because they have a direct impact on the costs that you will incur during development.
  • Time-box the project and then develop an overall estimate for the project before allocating time and labor budgets to individual sprints (see our newly rereleased Estimating Guide17 ).
  • Buy the right tools (agile metrics toolset, regression test suite, etc.) and setup the proper facilities (war room, task-board, etc.) as part of your agile effort.
  • Eliminate waste by combining agile with lean practices whenever it makes sense to do so (see our Guide18 on eliminating waste and rework which is available here).
  • Do things as simply as possible.  For example, refactor rather than redesign the product during development. As another example, develop a disciplined process.

Finding “7” – Agile Effort and Duration Distributions Differ from Traditional Norms

Again, based on our data, the manner in which time and effort are distributed during software development using agile methods differs from that experienced on plan-driven projects.  The primary reason for this dissimilarity is that those using agile methods focus their effort on creating working product in short spurts rather than on creating documentation.  Agile developers build and review working code with the user instead of putting as much as 50 percent of their effort and 40 percent of their time into generating requirements and design specifications before the start of coding.  Instead of trying to get the specifications right before they begin coding, agile developers spend as much as 80 percent of their time and effort developing working software that they can demonstrate and talk about instead of presenting viewgraphs about their progress at reviews. As shown in Figure 4, agile-at-scale developments differ from the agile norms in that they pay more attention to generating specifications while generating working software products.  They need to because they use documents as a means to coordinate work flow, track milestone performance and manage team activities.  The most interesting statistic this balance of activities results in agile developers spending less than 10 percent of their effort reworking code.  In contrast, those using traditional methods waste considerable effort spending as much as 25 percent of their effort modifying code. The reason for this sharp disparity is that agile developers design their products to be refactored instead of being reworked (i.e., refactoring is more adaptive and less error-prone).

Figure 4 – Distribution of Effort by Methodology

Notes

Requirements:

  • Agile: doing analysis, writing stories and doing architectural design work.
  • Traditional: specifying requirements and architectural design.
  • At Scale: developing requirements and architecture and then refining it release-by-release.

Development:

  • Agile: designing, coding and unit testing working products as they are integrated and tested continuously as releases.
    • Traditional: designing, coding and testing components as they are integrating and tested component by component into releases.
    • At Scale: designing, coding and unit testing releases that are then integrated with others into products and tested continuously.

Integration and Testing:

  • Agile: the release when readied is integrated and tested to ensure that the requirements are satisfied at the end of the cycle.
  • Traditional: done as part of the development.
  • At Scale: focuses on continuously integrating and testing the products of team efforts with one another and with   

   system elements.

Document:

  • Agile: the code serves as documentation along with stories and on-line user documentation.
  • Traditional: documents are generated to serve as products for each stage of development.
  • At Scale: documents are prepared to facilitate the integration and testing of team efforts and for the user.

Of course, we have not included DevOps and support tasks that agile developers perform as part of this analysis.  The reason for this omission is that these activities are typically accounted for as separate efforts and can confuse those trying to compare distribution statistics with heuristics19 that have become our and industry standards.

Issues: based on our analysis of the data, the top five current concerns relative to effort distribution statistics for agile include:

  • The amount of time and effort spent on a development should be treated as separable concerns.  For instance, it is not uncommon for requirements analysis and specification activities in a waterfall development to take 40 percent of the effort and 50 percent of the time to complete.
  • It is difficult to separate the integration and test effort from development activities especially when they are performed continuously as the sprint progresses.
  • It is also difficult to determine how much actual effort workers expend on each of their assigned tasks when they are tasked with supporting multiple developments in parallel.
  • As already noted, we account for DevOps and support tasks done by outside staff (technical writers, quality assurance, etc.) separately because their inclusion can cloud the findings.
  • Documentation that is generated for external uses (user’s manuals, build lists, etc.) is also accounted for separately.  That done as a normal part of a task (design notes, self-documenting code, etc.) is included when tabulating effort for the distributions.

Exploitation Measures: some suggestions for addressing effort allocations for agile developments are as follows:

  • You will be able to allocate budgets to tasks, teams, and projects more effectively by knowing where the effort went in the past and how it was distributed. For example, you might allocate more effort to team building than to documentation based on the results.
  • The documented distribution of effort reveals where investments need to be made to facilitate getting the work done more quickly and efficiently.  For example, the data might show that it makes more sense to spend money on collaboration rather than for project management tools because that is where the teams are expending most of their effort.
  • Knowing the distribution of effort also allows you to get rid of unnecessary tasks.  For example, you might want to reduce the number of documentation specialists you assign to projects when agile methods are used because the amount of paperwork is reduced with agile methods.
  • Distribution of effort also helps you identify where to spend your training dollars.  Investing in developing refactoring skills might seem beneficial when you find that your programmers are modifying rather than refactoring the code. 

Finding “8” - Agile Improves the Ability to Meet Schedule Deadlines

Based on our data, agile time-to-market, as measured by calendar time from start to software delivery, provides firms with an ability to realize tight schedule deadlines 75 to 90 percent of the time versus the 40 to 60 percent average performance that has been experienced on comparable traditional waterfall projects.  However, the typical content delivered on agile developments (i.e., the percent of the features that were promised that were actually delivered) ranges from 80 to 90% versus 95 to 100% for plan-driven or traditional developments.  In other words, not all features that were promised were delivered. This is expected because agile practitioners deliver those features that the customer deems to be the most important. When deadlines or changing requirements become a problem, the lower priority features are backlogged as the higher priority ones are emphasized.  In contrast, traditional developments adhere to an “all or nothing” philosophy where delays rather than timely deliveries are the norm. As noted, agile provides an 80 to 90 percent solution on time and budget, while traditional methods provide you with either a delay or overrun, but fully functional product. 

To address concerns relative to government contracting, agile developers often use a two-step process to address the customer’s desire for fixed-price or cost-plus contracting.  During the first step, they solidify requirements using a time and materials contract.  During the next step, they deliver full functionality per the terms and conditions of a fixed-price or cost-plus contract developed to share risk.  When this approach is used, our data shows a 25 to 40 percent greater chance that agile developers will be more successful.

  • : based on our analysis of the data, the top five current concerns relative to cost include:
  • You need to determine whether or not the schedule and budget are feasible at the project level in the first place because often they are not.
  • Time-box the agreed-to schedule to ensure that those features identified as important (i.e., those that contribute directly to the value proposition) can be delivered first.
  • To force timely decisions about release content, determine the number and priority of features that are considered essential for delivery.
  • Identify the number and priority of features that are considered acceptable in the backlog to ensure that it is not used as an excuse for poor performance.
  • Do the same for the number and type of defects in the backlog that are considered acceptable. Set a target so that quality will not be cut short in the name of budget or schedule.

Exploitation Measures: some suggestions for exploiting agile time-to-market advantages include:

  • Achieve an acceptance time-to-market with agile by trading off scope, budget and features and deferring the one variable that the customer deems is the least critical.
  • Develop realistic schedule as well as effort estimates for the project based on content and adjust sprint and release schedules so that you can deliver the promised features on schedule. 
  • Monitor and control release schedules rather than task milestones as you adjust deadlines and delivery expectations based on product owner inputs on features and priorities.
  • While managing release schedules at the project level, let the teams establish and control their sprint timelines.
  • In order to adjust timelines, manage content and then use the backlog to fine-tune results.
  • Hold demonstrations at the end of each release to show-off a working product and engage customer/users, marketing and management.

Finding “9” - Agile Quality is Better than Traditional Norms

Based on the results of our recent agile quality study20, expect agile software quality to exceed traditional method performance by a factor of from 6 to 12 percent in about three years.  Software quality improves both during development and post-release because it is engineered into the product by the development team as working software is generated.  When agile methods are used, quality is considered throughout the development process first via the use of test-first and then through continuous integration and testing approaches.  In addition, quality is addressed by the team and not left to third-party organizations to try to test it into the product as part of their assurance processes. As with agile software productivity and cost, quality varies by application domain and the factors that influence quality including the quality control/assurance practices that vary as a function of organizational culture and preference.

Issues: based on our analysis of the data, the top five current concerns relative to software product quality include the following:

  • Push for working software should be done with an emphasis on product quality.
  • Quality assurance should not be relegated and equated to testing in agile shops.  Along with emphasizing test-first principles, agile practitioners should focus on engineering quality into the product throughout its development.
  • Quality metrics (defect rates, densities, etc.) should be gathered and used during development to address quantitative goals and make process improvements. 
  • Quality control and assurance practices should be employed as separate by related disciplines. Recognize that quality control focuses on the product while assurance addresses process issues.
  • The product should be assessed to ensure quality expectations are realized before release.

It should be noted that confusion often occurs in a regulated environment over how to certify being developed using agile methods to ensure that it satisfies safety, security and other such governance requirements.  Quality personnel can tailor their processes to address such issues.

Exploitation Measures: some suggestions for maintaining high software quality standards follow:

  • Transform the culture to one that rewards product quality rather than timely delivery.
  • Task existing quality organizations to be the teachers rather than the inspectors.
  • Embrace quality and make it part of all of your work processes.  Hold peer reviews, discuss quality at your daily standups and make defect information visible on a daily basis on your task-boards and version status reports.
  • Task everyone on the team to check the quality of each other’s products.  Make building quality software fun by holding contests, giving prizes and providing recognition for a job well done.  Some firms use pair programming practices as part of implementing this recommendation.
  • Capture quality metrics and measures20 and use them to quantify product quality during both software development and maintenance.
  • Never release a product that is defect-prone as this act will sully your reputation and cause the customer to lose confidence and trust in your abilities.
  • Employ lean and Kanban principles which focus on engineering quality into the product as it is being developed.  Find and fix defects during sprints.  Else, put them in the backlog.

Finding “10” - Agile “Value” Continues to Remain Hard to Quantify

Agile methods place emphasis on providing its users/customers with value by delivering working software products continuously throughout the development life cycle.  Value is commonly measured in terms of the financial and/or competitive benefit that organizations receive for their expenditures. While there is some guidance available for value planning21, little help is offered on how to quantify the value derived from these efforts.  Instead, the term is sometimes used to convey morale and other important but not easily quantified measures of worth.  To address this lack of clarity, we suggest that you use classical financial measures22 such as cost/benefits, payback, or return-on-investment to quantify the returns derived from your agile alternatives.  We recommend this because managers understand and frequently use such measures to assess options and make financial decisions.   The survey data that we have been able to collect on “value quantification” validates the contention by customers that agile methods provide more worth than traditional approaches using the variety of measures used for this purpose.

  • : based on our data, the top five issues relative to portraying the value include the following:
  • Value propositions should be tied to current customer business goals, plans and investment strategies.
  • Value propositions should be formulated in monetary rather than technical terms.  Along with a technical justification, a good business case is needed when justifying a financial option.
  • Be sure to take both the customer’s current and future financial goals into account when developing the numbers.
  • Express the numbers in terms that the customers understand and whose accomplishment shows that they can be completed for the funds requested and within the timeline allocated.
  • Value propositions should hold up when scrutinized by outside examiners.

Exploitation Measures: some suggestions for quantifying value follow:

  • Whenever possible, let your financial numbers do the talking.
  • Always use money as the common denominator for your value propositions.
  • Make sure that you quantify both the tangible and intangible benefits.
  • Ensure that you consider both fixed and recurring costs and treat them separately.
  • Consider sunk costs irrelevant because they have already been incurred.
  • Be sure to take into account the cost of money when dealing with financial people.  This shows them that you have paid attention to the details.
  • Take existing financial folklore into account when expressing your numbers as these figures will be the ones that management will use to determine the reasonableness of your results. For example, use head counts rather than dollars to express cost when this is the measure the firm uses to express effort.  Use current overhead rates to load your costs.

Finding “11” - Agile Continues to Provide its Users with a Competitive Advantage

While we have anecdotal data to verify this conclusion, many organizations seem to believe that agile methods can represent a competitive advantage when organizations take full advantage of their strengths and compensate for their weaknesses.  Not only do agile methods provide software productivity, cost and time-to-market rewards, they give users the ability to attract and retain talent.  In order to tap these and other benefits, we recommend that you move to institutionalizing the use of methods enterprise-wide as quickly as possible.  Take the best of breed approaches that have worked for you on your successful projects and fan them out company-wide.  Train your people in the techniques that work and develop the leaders and subject matter experts you will need to solidify their use as part of the initiative.  If your products involve more than just software, consider moving other parts of the enterprise to agile methods.  The data that we have gathered shows that those firms making such moves reap the rewards more fully.  Finally, look to the future and keep a watch on on-going developments.  Because technology is changing, so should you.

  • : based on our analysis of the data, the top seven current concerns relative to tapping the method’s competitive advantages include:
  • Quicken the fan-out of agile methods throughout the organization by bringing professional coaches, tools and training.
  • Make necessary organizational changes to facilitate agile roles and responsibilities.
  • When making these changes, be sure to recognize the difference between product and project management.
  • Modify the existing enterprise infrastructure processes (Human Resources (HR), finance, contracts etc.) to support agile methods and minimize any clashes that might exist.
  • Capture current software productivity, cost and quality benchmarks for comparisons so that improvements can be quantified and best-in-class performance can be highlighted.
  • Place attention on building, increasing and maintaining an agile operational tempo once agile methods are in place and fan-out is proceeding.
  • Determine how to continue to reap the benefits of agile methods once stability has been reached and agile has become your way of doing business.

Exploitation Measures: some suggestions for exploiting agile competitive advantages follow:

  • Make those structural changes to your organization required to maintain an ever improving agile operational tempo.
  • Define agile job descriptions, roles and team concepts formally through position descriptions.  Then staff positions, hire agile coaches and educate and train the workforce.  Finally, spend time spinning up staff so that they can handle the transformation.
  • Make the cultural shift from a top-down, command-driven style of leadership to self-organizing, participatory teams at least within your software organization. 
  • Work towards achieving enterprise goals instead of focusing on projects that deliver products that benefit one customer or user at the expense of another.
  • Focus on keeping things simple.  This philosophy extends from the product architecture to the code itself. 
  • Always emphasize quality because it acts as the differentiator.  Along these lines, recognize that quality must be built into your products.  It cannot be tested or inspected in.
  • Transform organizational focus from achieving individual goals (perform assignment on schedule, etc.) to having teams provide value to the organization and its customers.
  • Create a culture where both successes and failures are celebrated and people are rewarded for making decisions, taking risks and trying to resolve problems.
  • Focus on creating a better way of doing business and a fun place to work for everyone involved in product development.  

Finding “12” – Agile Impacts on Government Acquisition Practices can be Major

The impact of agile methods is not confined to development practices.  When agile is used on contract, the method’s impact ripples through the way work is planned, schedules and budgets are allocated, requirements are captured and managed, progress is reviewed, and overall contract performance is determined23.  If the government and vendors set a collaborative management framework in place for the development, both can work according to agreed-upon plans towards the achievement of shared goals with a minimum of delay and friction.  When such a framework is not created, clashes occur due to the differences in the ways each conducts its business.  Such mismatches often result in bottlenecks, delays and distrust due to their misalignments.  To correct this situation, such acquisitions should set in place collaborative processes that enable both the government and vendor to form a partnership whose aim is to manage the timely and on-budget delivery of quality software that satisfies the requirements of the contract upon which its performance is based.

  • : based on our analysis of past performance on government contracts, the major concerns relative to tapping the benefits of agile methods in an acquisition environment include:
  • Structure contracts as needed to enable government and vendor personnel to work together to set in place a management infrastructure that provides visibility into and control over contract performance using agile, lean and Kanban concepts.
  • Formulate requirements to facilitate the development of agile work products.  This may mean that user stories might replace the initial specifications that form the basis of the contract.
  • In order to enable the vendors to do their job efficiently, bureaucracy and oversight need to be kept to a minimum.  That does not mean that the reviews, reports, documentation and other items that the government requires for contract administration/performance determination like metrics and Earned Value Management (EVM)24 will be totally eliminated.
  • Partnerships between the government and vendor personnel working on the project need to be built to reduce the distrust and enable both to focus on getting the job done.  This needs to include the third-party personnel that the government hire to help manage the acquisition.

Exploitation Measures: some suggestions for acquisition improvements follow:

  • Structure contracts so that they facilitate and reward the use of agile methods.  This requires some innovation because projects focus on deliver features rather than complete tasks when they use agile methods.
  • Do not superimpose your management structure on top of the vendors.  Instead, work with them to adapt both structures to facilitate getting the job done.
  • Clearly define roles and responsibilities for those organizations used to manage the acquisition at the performer, project/program and enterprise levels within both the government and vendor groupings.
  • Identify how results will flow up and down and across organizations.  For example, identify how results from self-organizing teams will be fed to the project/program level so that they can be synchronized, coordinated and managed.
  • Make sure that metrics used to measure progress and performance embrace agile principles and provide meaningful insights.
  • Make sure that agile concepts are used when EVM and other measurement systems are used to assess performance.  Else, use rate-of-progress or burn-down charts to take their place.

II.  Other Findings

Because agile methods represent the way most of the organizations surveyed currently do their software business, our findings should offer few surprises.  Many of the issues highlighted are being worked and solutions are being offered.  Once the transformation to the use of agile methods is completed, firms need to fan-out the technology and build an operational tempo that allows them to continue to tap the many benefits that use of agile methods offers.  The goal should be to build, stabilize and maintain momentum as the methodology is fanned-out throughout the enterprise. The techniques that seem to accomplish this transformation revolve around developing and putting best practices to work within a collaborative team environment that embraces both an interdisciplinary approach and a product management focus and discipline.

Being agile is a mindset.  As such, there is neither one definition nor one correct way.  Instead, we along with others define the state of being agile as conforming to the spirit and the twelve principles of the Agile Manifesto25.

Based on the analysis of hundreds of completed successful agile projects during the past decade, the following major findings continue to be pertinent for those trying to harvest the full benefits from agile methods:

  1. While there is a definite business case that can be made for the use of agile methods, there are serious barriers to adoption and fan-out that need to be overcome in order to institutionalize the use of the method.   The easiest way to overcome these barriers is bringing in professional coaching, mentoring, training and help to assist with the transformation.
  2. Change management principles26 dominate when fanning agile methods out throughout both the organization and the enterprise.  When making the move, strategies need to be developed, plans put into place, infrastructure revamped, staff trained, management educated, and effort expended to figure out how best to run agile projects within the context of your organization, its business goals, its processes and its past history. 
  3. You should recognize that some agile methods may work and others may not when put to use in different software development contexts.  For example, Scrum works well for small to medium-sized commercial developments.  As another example, firms often migrate to the use of the Agile Unified Method (AUP)27 when moving from traditional approaches to ease the transition especially when they have been using the Rational Unified Process (RUP)28 to develop their software products.  In contrast, scaled and hybrid methods including LeSS, SAFe and hybrids that embrace lean and traditional project management approaches may be used to deal with complex agile-at-scale projects.
  4. Once agile methods have been adopted throughout the software organization, firms need to focus on building an agile operational tempo.  As they switch gears, they do this as they try to fan the methodology out into widespread use throughout the organization and into aligned engineering and support groups.  While this may seem simple, it is not.  Groups like systems engineering and digital design tend to resist the change because they view agile as an undisciplined approach.  As expected, considerable time and effort and some executive encouragement is needed to nudge agile adoption ahead in all of these areas and arenas.   The major lessons learned are that these groups should be brought on-board as soon as possible and that a single engineering methodology should be embraced right from the start.
  5. Scaling of agile projects continues to be a problem because of organizational and process mismatches and the confusion over whether the emphasis should be either project or product management.  When conflicts between groups exist, little gets done.  To avoid these issues, we recommend moving to a single engineering discipline that embraces affected groups, exploits true interdisciplinary team concepts, revamps processes with agile concepts in mind and takes full advantage of project and product management techniques. In this case, a good management expert who possesses agile, process improvement, product management, and project management skills, knowledge and abilities can be of assistance especially if he/she has experience within your industry and applications domain in which you operate.
  6. From the competitive landscape, the move to agile methods often makes a lot of sense for three reasons.  First, you can use the methodology to catch up with your competitors who have most likely already made the commitment and made the move to use agile methods before you did.  Second, you can use agile methods as a means to attract talent.  The reason for this is that high performers want to work in an environment that uses the latest and greatest techniques.  Third and finally, you can use agile to show your customers that you are putting techniques into place to provide them with value; i.e., increase overall productivity, cut cost, reduce time-to-market, improve quality and provide a competitive advantage.
  7. If you are a laggard29; i.e., those just now starting the agile adoption process, there are ways to catch-up quickly.  For example, you can acquire rather than build your agile capabilities by purchasing a firm with the right credentials and capabilities.  As an alternative, you can hire a qualified firm to help to put agile methods to work as they train and mentor your staff to do so on the next round of developments.  However, whatever you do, don’t forget that the reason you are going agile is to improve your bottom line, be more competitive and deliver value to your customers.  You have a greater chance of succeeding if you keep these goals in mind as you strive to achieve them.

For those of you interested in learning more about agile methods, we refer you to our web site to view our blog and current reports which are summarized for your convenience in Appendix A of this report.     

Reifer Consultants also offered custom software productivity, cost and quality benchmarking services on a fee-for-service basis.  If interested, please contact the author at don@reifer.com.       

  1. Reifer Consultants LLC, Software Productivity, Cost and Quality Benchmarks, Vol. 7, No. 2, July 2017.
  2. Geoffrey Moore, Crossing the Chasm: Marketing and Selling High-Tech Products to Mainstream Customers, Harper, 1999.
  3. Kenneth Rubin, Essential Scrum: A Practical Guide to the Most Popular Agile Process, Addison Wesley, 2012.
  4. For more information on LeSS, see.
  5. Scott Ambler, Agile Modeling: Effective Practices for eXtreme Programming and the Unified Process, Wiley, 2002.           
  6. Kent Beck and Cynthia Anders, Extreme Programming Explained: Embrace Change, 2nd Edition, Addison-Wesley, 2004.
  7. Stephen Palmer and John Felsing, A Practical Guide to Feature-Driven Development, Prentice Hall, 2002.
  8. For more information on SAFe, see.
  9. Reifer Consultants LLC, Quantitative Analysis of Agile Methods, August 2014.
  10. 10 Reifer Consultants LLC, Quantitative Analysis of Agile Methods, August 2015.
  11. Reifer Consultants LLC, Agile Introduction: Are You a Laggard? June 2015.
  12. For more information on Kanban for agile teams, see.
  13. For a good discussion of lean versus agile concepts, see.
  14. Reifer Consultants LLC, Agile Scaling Findings and Productivity Statistics for Larger Enterprises, July 2016.
  15. William Wake, Refactoring Workbook, Addison-Wesley, 2003.
  16. For a good article on agile and DevOps considerations, see.
  17. Reifer Consultants LLC, Agile Estimating: Straightforward and Simple, October 2016.
  18. Reifer Consultants LLC, Agile Waste, Rework Reduction and Technical Debt, October 2014.
  19. Donald Reifer, Tutorial: Software Management, 7th Edition, John Wiley & Sons, 2006.
  20. Reifer Consultants LLC, Agile Software Quality: A Quantitative Assessment, February 2017.
  21. Tom Gilb, Value Planning, see.
  22. Donald Reifer, Making the Software Business Case: Improvement by the Numbers, Addison-Wesley, 2001.
  23. S. Miller, D. Ward, M. Lapham, R. Williams, C. Hammons, D. Burton, and A. Schenker, Update 2016: Considerations for Using Agile in DoD Acquisition, Software Engineering Institute, Report CMU/SEI-2016-TN-001, December 2016.
  24. See the following for a good discussion on Earned Value Management and agile methods.
  25. For the Manifesto, see.
  26. Donald Reifer, Software Change Management: Case Studies and Practical Advice, Microsoft Press, 2011.
  27. For info on the AUP, see.
  28. A good overview of the RUP is available.
  29. Reifer Consultants LLC, Agile Introduction: Are You a Laggard? July 2015.

The author acknowledges the helpful comments and critiques offered by those who took the time to review this document.  Their inputs greatly enhanced this white paper.  I would particularly like to call out the contributions of Mauricio Aguiar (TI Metricas - Brazil), Bob Epps (Lockheed Martin [retired] - USA), Dr. Ken Nidiffer (Software Engineering Institute – USA), and Kevin Wu (Singapore).  I would also like to thank my wife Carole for her technical editing efforts.

Appendix A - Available Reports

No.

Report Title

Date

Description

1

Agile Software Quality – A Quantitative Assessment (2017)

Feb 2017

This report summarizes an assessment of the quality of software developed using agile methods. The survey polled hundreds of participants in 26 countries and more than 200 organizations. It gathered results from over 2,000 agile software projects to develop its findings, recommendations and conclusions.

2

Size Estimating Approaches for Use with Agile Methods

Dec 2016

This report summarizes the results of an investigation to assess the different approaches that could be used to size software projects being developed using agile methods. The five software sizing methods reviewed include (1) analogy, (2) function points, (3) Halstead, (4) proxies and (5) user stories/story points.

3

Agile Estimating: Straightforward and Simple (Update)

Oct 2016

This white paper provides readers with an estimating procedure for use in sizing, time-boxing and estimating software cost for efforts being pursued using agile methods.  An innovative approach for normalizing size measures is included in the report.

4

Cognitive Computing and Software Development Automation: A Bright Future

Sep 2016

This white paper summarizes the findings of a study that investigated the use of cognitive computing technology to automate software development.  The paper looks at whether elements of the technology can be used to perform the intellectual tasks associated with the design, coding and testing of software now and in the future.

5

Agile Scaling Findings and Productivity Statistics for Larger Enterprises

Jul 2016

This technical memo summarizes the findings of a brief study looking into the productivity achieved using scaled agile methods as they are adopted, spread and institutionalized primarily within large enterprises, many of which use software as an enabling technology to sell more complex products.

6

Quantitative Analysis of Agile Methods - 2016

Jun 2016

This report compares productivity, cost and quality benchmarking results from agile projects against others that developed software using traditional plan-driven methods using “hard” data from 2,000 projects, 1,000 of which used agile methods.

7

CMMI and Agile: The Good, the Bad and the Ugly

Sep 2015

This white paper looks at both project and enterprise use of the CMMI® (Capability Maturity Model Integration) with agile methods to provide with an overarching infrastructure for process improvement. The paper highlights the pluses and minuses of using the framework along with the good, the bad and the ugly associated with it for agile shops.

8

Ten Major Findings: Quantitative Assessment of Agile Methods (2015)

Jul 2015

This report compares productivity, cost and quality benchmarking results from agile projects against others that developed software using traditional plan-driven methods using “hard” data from 1,500 projects, 500 of which used agile methods.

9

Agile Introduction: Are You a Laggard?

Jun 2015

This white paper concludes that agile methods are now the predominate approach to developing software in the world.  In addition, it provides insights into what to do after adoption and recommendations for improvement using Moore’s technology adoption framework.

10

Firmware Estimation1

Nov 2014

This white paper provides recommendations for estimating the cost of programs that reside in Read Only Memory (ROM), Programmable ROM (PROM), Erasable PROM (EPROM), and flash memories, and Field Programmable Gate Arrays (FPGAs).

11

Agile Metrics and Measurement, v1.3

Oct 2014

This white paper provides a shopping list of metrics that readers can use for gaining visibility into and control over agile projects.  It also identifies metrics that can be used in support agile methods like Scrum.

12

Agile Visibility and Control, v1.3

Oct

2014

This white paper provides guidance on how to maintain visibility into and control over agile developments.  It also identifies insight into how to use agile tools and measures to make such assessments.

13

Agile Rework, Waste Reduction and Technical Debt, v1.3

Oct 2014

This white paper provides an overview of lean, Kanban and Kaizen techniques and then discusses how they can be used in conjunction with agile methods to reduce rework, waste and technical debt.

14

Software Resources Data Report (SRDR) for Agile Projects: Some Serious Considerations1

Aug 2014

This white paper discusses issues and provides recommendations for the collection of cost data for major Department of Defense (DOD) software acquisitions using agile methods via the Software Resources Data Report (SRDR).

15

Special Report on the Use of UCC as a Standard Code Counter1

May 2014

This white paper provides a detailed analysis of the USC Uniform Code Counter (UCC) source line of code (SLOC) counter and provides insight into whether it should be adopted as a standard tool by subscribers.

16

Reifer/ISBSG/Namcook Analytics Joint Report: The Impact of Software Size on Productivity1

Sep 2013

This report done jointly with the International Software Benchmarking Standards Group (ISBSG) and Namcook Analytics looks at productivity as a function of size and shows that there are economies of scale available that firms can take advantage of when motivated.

Legend

1 These documents have release restrictions.  Please inquire about availability

About the Author

Reifer Consultants is a software consultancy that specializes in helping firms effectively use agile methods, benchmarks and analytics.  Since its founding in 1980, Reifer Consultants has provided its clients with a full range of services ranging from proposal preparation, product and program management, and recovery operations. The firm has over a decade of experience coaching and advising clients at the executive and mid-levels on how to take full advantage of agile methods. They have helped over a dozen Fortune 500 companies make an orderly transition to agile methods. Contact Information: Reifer Consultants, 79410 Azahar, La Quinta, CA 92253 Phone: 310-922-7043, Email: donald.reifer@gmail.com.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT