Making Agile Methods and Enterprise Architecture Play Nice

Posted by Amr Elssamadisy on Sep 05, 2007 06:00 AM

Community

Architecture,

Agile,

SOA

Topics

Agile in the Enterprise ,

Enterprise Architecture

Tags

Business Architecture ,

Best Practices ,

Business/IT Alignment

A report from the Cutter Consortium asks Are Agile Methods and Enterprise Architecture Compatible? and answers Yes, with Effort.  The authors recommended specific techniques to allow Agile Methods and Enterprise Architecture to be mutually beneficial. Moreover, their observations, analysis, and recommendations are directly applicable to the meshing of Agile Methods and Service Oriented Architectures.

Enterprise Architecture (EA) and Agile Methods(AM) share the same goal - delivering software that aligns to business needs and responding to the inevitable change of those needs. EA and AM, however, go about attaining this goal in extremely different ways. In the report EA and AM are defined as follows:

EA deals with enterprise level issues such as:

• Connecting business strategy to the IT system by providing an overall blueprint of the business process, which is mapped to architectural patterns, core services, and application capabilities
• Improving the consistency across the enterprise by maintaining an inventory of current data schemas, process flows, and service definitions
• Improving operational efficiencies by reducing redundancy between systems and identifying components and systems that can be consolidated
• Ensuring a flexible IT capability that can respond to changes in technology vendors and new/enhanced automation of business processes
• Supporting project costing and prioritization by maintaining an IT portfolio, current and target architectures, and a roadmap for the transition
• Providing a stable, reliable platform of infrastructure and common services for the ongoing operation and development of systems

Agile methods are concerned with the following ideas:

• Improved efficiency by focusing on near-term issues that are developed with just enough capability to meet the current needs, allowing the design to emerge over time
• Improved manageability (project-visibility) by focusing on short, iterative development cycles that allow task completion to be meaningfully assessed
• Improved quality by providing a gestalt of processes that focus on extensive automated testing, addressing integration issues early and often, allowing multiple developers (hence broader experience) to work on the same code, and continuous feedback from the end user
• Improved maintainability (internal quality) by integrating into the above a continuous refactoring process
• Improved ability to handle change - whether it is a change in requirements, a clarifications, or a new prioritization of features - by incorporating customer feedback and planning into each iteration
• Improved communications through the use of implicit knowledge, shared team space, and a focus on a small part of the problem

The gap between EA and AM was then analyzed by examining AM from an EA perspective and vice versa. From an EA perspective:

• Agile iterations, which are usually one to six week time boxes where a working subset of the system is built, is rarely adopted. When EA attempts at iterations occur, they are usually time-box decompositions - there is no working software at the end of the period.
• Evolutionary design, the organic, incremental way in which a systems design grows in a typical Agile project is risky, and can lead to redundancies and incompatibilities between different applications. EA groups want to lead the design and recommend common infrastructure components, database schema definitions, etc...
• Agile heavily relies on executable artifacts - i.e. the tests written (both unit and system). EA artifacts are not testable. They have limited impact on a project because they have no feedback loops - the designs do not give a red-bar when they are not followed.
• The Agile notion of a customer as a part of the team is unrecognizable. EA groups do not have one customer but have an indirect and large set of customers ranging from IT, to operations, to development teams, to end users.

From an AM perspective, EA doesn't quite make sense either:

• EA is concerned with aligning IT systems with business strategy. A plan is developed that maps from present to future systems which filters down to individual projects. Teams that use AM could probably use the information in these documents, but by the time they reach the teams they have lost context and are incomprehensible. Moreover, the documents are testable. This is one of the main reasons the EA teams are seen as "white tower" architects that have lost touch with reality.
• To reduce redundancy and improve consistency EA groups produce reference architectures, recommend frameworks, and release guidance on how applications should be built. AM teams see these decisions as the domain of the individual projects and should not be dictated by others who are not 'in the line of fire'.
• EA is also concerned with enabling integration among the different applications across the enterprise. Again, the EA group recommends specific solutions via reference architectures and frameworks. Many AM teams will see these decisions as premature and imposing an unwarranted technical debt.
• Finally, EA approaches accommodating change by designing for the future by creating abstractions that are easier to change. AM teams will see this as premature and typically see this as BDUF. Teams practicing AM would rather hold off on the abstractions and allow for change by relying on automated tests to enable refactoring.

The title of the report did say Yes, with Effort, so there is still hope, but both the EA group and the AM projects need to recognize each other's valuable contributions and make accommodations in the way their work. EA groups and AM teams can be of mutual benefit by:

• An AM team should realize that the reference architecture and frameworks may be BDUF at a project level, but are waste and rework for the enterprise. There is no reason to re-invent the wheel if it has already been built.
• EA teams should make information available at the right scope and in the right format. That is, an effort to create artifacts that are relevant to each project should be made.
• The EA group should be considered a customer and their guidance should be seen as a requirement.
• An architect for each AM project should be the liaison with the EA group.
• An effort should be made for refactoring done at the AM project scope to be incorporated at the enterprise level.
• Testable EA artifacts should be created for:
  • Standard infrastructure and platform configurations
  • Data schemas
  • Services
  • Reference architectures
  • Business process models
• EA should ensure that enterprise "context" can be phased in over time which will address the BDUF objection of AM teams.
• EA should be involved in project cycles.
• Flow of information should be a two way street, when AM teams find faults/inadequacies with reference architectures and frameworks, there needs to be a way for them to make the change and for that change to find its way back to the EA group. That is, the EA process must support the enhancement of enterprise services.
• AM teams can provide spikes for influencing the enterprise architecture.
• An enterprise test environment can increase consistency, reuse, and enable integration. AM teams are experts at writing tests, they should be leveraged. EA artifacts that can be made executable should be.

InfoQ spoke with two of the authors of the report, Michael Rosen and Jim Watson, about the content of the article and their client-experiences that led to the recommended solutions. Jim Watson described the most common scenario:

The experience of using both has largely been having a project that is using one and failing because of the lack of using the other. For example, an important document processing system can be developed with the best AM practices, but not be tuned into the EA needs of the system that span requirements, interfacing, operational issues, etc. Alternatively, a waterfall-ish project might have all of its EA ducks in a row, but not be able to show value to the customer early, or not be able to address risk issues through a means of iterations. So there the paper is mostly from the experiences of how did these projects get to the point where they ignored the other viable discipline, and what where the practical ways to fix it.

A more profound case would be starting a project with the balance of EA and AM….however, this turns out to be much harder and much less likely to occur, primarily because of the organizational issues and perspective of who is involved at which part of the process. You see this failure a lot when, for example, the architects write the requirements (with or, sadly, without the customers; but the development team is uninvolved), and then development team takes over with the architects AWOL.

Both Jim Watson and Michael Rosen told us that, with respect to the scope of this article, SOA can be seen as an instance of EA. Therefore, everything concerning the problems and solutions here are applicable to organizations adopting SOA that has existing AM teams. (Not surprisingly, there is considerable overlap between the InfoQ article SOA and Agile: Friends or Foes? )

The interaction of EA and AM does not depend on SOA, but it is worth noting that SOA provides some of the mutual interest and issues that allows for progress in using EA and AM together. For example, a project lead SOA initiative will probably have difficultly defining truly useful services at a business level; a EA lead SOA initiative without the benefit of AM development practices has produced lots of SOA shelfware because it is too hard to implement or only defines interfaces that are not truly needed.

One of the suggested solutions, that will stand out to an AM team is the inclusion of an Architect as a member of each team to act as a liaison to the EA group. When asked to clarify which type of architect was recommended, Architect Reloadus or Architect Oryzus (as defined by Martin Fowler in Who Needs an Architect? ), Michael Rosen suggested that neither is the case. In the largest organizations, those with significant EA groups, a typical IT group may have 20,000 employees with 500 architecturally significant projects and only 70 architects in the EA group. There are just not enough architects to go around so Architect Oryzus is not applicable. Architect Reloadus cannot work either because they cannot have enough context to be effective. An effective use of an architect is as a consultant to the individual AM teams. Therefore an architect, from the EA group, can be effective without being embedded in the teams.

So, in the end, for organizations that have both EA groups and AM teams it is not enough to live and let live. Although they have the same goals, their default mode of operations are at odds and can (and usually do) cause problems. Therefore these practices and others are useful to meet strategic enterprise goals and still deliver tactical software projects. 

The full twenty-five page report can be downloaded from the Cutter site. Be sure to use the promotion code at the bottom of the page.

1. Back to top

   Interesting viewpoint from Robert Martin

   Jan 18, 2008 2:29 PM by Javid Jamae

   http://www.artima.com/weblogs/viewpost.jsp?thread=17799

   Reply

• Older >