Architects: Chickens or Pigs in an Agile Development Process?
Can architects play a meaningful role in agile projects, or does their tendency to do “big design up front” make them a sideline resource? Nick Malik, an Enterprise Architect with Microsoft, recently explored this topic in a blog post and concluded that architects can absolutely play a key role in software projects that use Scrum.
Malik’s blog post was inspired by a project where he attempted to affix architecture practices to a Scrum process. His post began by immediately claiming that software architecture is not at odds with agile development processes. However, given the short delivery cycles associated with Scrum, Malik concedes that the traditional architecture practice of taking months to document and diagram a system is “pretty silly.” But any software project – including one delivered via Scrum development processes – has a fundamental software architecture that must be acknowledged.
The value of software architecture is that key decisions are made about the core infrastructure of the system itself: where will generalizations lie? Will a layered paradigm be used, and if so, what are the responsibilities of each layer? What modules will exist in each layer and why will they be created? How will the responsibilities of the system be divided up among the layers and components? How will the modules be deployed at scale? How will information flow among the modules and between the system and those around it?
The way these questions are answered will indicate what the architecture of the system is.
According to Malik, each of these choices is made by carefully balancing a set of system quality demands. These demands come into play as software architects work within the three different layers of software accountability.
The topmost layer, labeled Aligning Processes, occurs quarterly or semi-annually and addresses the organization-wide information and business strategy. The output of this layer is the “to be” software models for the organization. The second layer includes the Balancing Processes and is associated with a given software project and likely occurs alongside development of the first few Scrum sprints. Malik explains that this is where the logical system architecture is crafted.
These processes consider the needs of a single system but only from the level of deciding why the software will be divided up into modules, layers, and components, how that division of responsibility will occur, and what the resulting system will look like when deployed in specific technologies in a specific environment.
Finally, the bottom layer of this model is where Realization Processes reside. According to Malik, here is “where the architecture becomes software” and architects make the specific design decisions that the software developers use when building the system. Malik admits that developers may not adopt the design patterns chosen by the architect, although “the team will very likely implement the software architecture as described, but may choose to improve upon it.”
So how does this work in practice for a given Scrum software development process? Malik added a step immediately before the sprint planning session. Instead of proceeding from the prioritized product backlog directly to the sprint planning session and subsequent software sprint, teams should inject a “pre-sprint story review” where stories can be improved and architecturally assessed.
Malik suggests executing this new architecture-focused step one week before the sprint planning session.
In that week prior to sprint planning, those folks, working with the product owner, can improve the stories, add constraints, refine the description and acceptance criteria. And here’s where the architects get to play. Architects fulfilling the role of “Balancing” in the model above will have (or can create) an outline document describing the architecture of the software system, and can “link” that document to the SPECIFIC STORIES that are impacted by that design.
Malik concludes that an architect is a “chicken” or “pig” in an agile project based on which layer of architecture is being addressed. The architect is an involved member of the team when crafting the first two layers, and an invested participant when working in the third layer. This exercise by Malik continues his investigation into the role of “architecture” in agile practices. In a blog post earlier this year, Malik proposed that Enterprise Architects are inherently agile and that their focus on completing high value activities through incremental progress is directly in sync with the spirit of agile. The intersection of agile and architecture continues to be an area of great interest as the industry tries to improve the efficiency of software development while not incurring architectural debt.
Thank you for the write up
Architects in SAFe
Re: Thank you for the write up
Charles Bradley, Scrum Trainer...
1. Ivory tower architects. I don't really believe they are effective. To me, an Ivory Arch is someone who does not regularly participate as a Scrum Dev Team member. The way this is described, it's very unclear to me whether the architectural advice is 100% required, 100% optional, or what? When you "link to documents", that implies a contract, which causes problems no matter what the real truth is. Further, Ivory Tower archs also sometimes encourage teams to simply "outsource" their architecture to the ivories -- and that's usually not terribly optimal either. So, my suggestion, the Ivory Tower architects have to lead by example, eat their own dogfood, and code on Scrum Dev Teams for *at least* half of every year. What I'd really like is for them to be full time Scrum Dev Team members and also spend some of their time participating in an Architecture Community of Practice, along with any other Dev Team members who would like to participate. Like all other "scarce skills" (UX, DBA, etc), I also want that person to work very hard to spread their scarce skills across the Scrum Dev Team -- not so much that everyone is exactly equal in their scarce skills -- just so that the "bus count" is increased. See here for more from Larman's excellent Large Scale Scrum books:
2. Utilize the latest advancements in Scrum
Scrum has change significantly twice in the last two years. Utilizing the practice of Product Backlog Refinement (formerly called grooming) can help to flush out arch issues, with or without Ivory tower architects. We coach that teams should have their PBL refined at least 2 sprints ahead of the current sprint. Much of this "refinement time" can be used to flush out issues with arch, reqs, design, cross team coordination, etc
Further, the "chicken and pigs" metaphor was removed from the Scrum Guide because it wasn't helping us. As it turns out, archs, managers, etc, don't like being called "chickens", and even Dev Team members don't like being called "pigs" etc -- American language implications, etc. However, in your parlance, architects are chickens until they become Scrum Dev Team members.
In my opinion, almost no architects should be ivory tower, because I don't see it as effective in practice. I just see conflicts, hand offs, and misunderstandings.
Don't misunderstand me. I have the highest respect for the vast majority of people we call "architects". Some of them are downright brilliant. However, the ivory tower architect is a sub optimization from the waterfall mindset. The Agile mindset strongly encourages us to put them on the teams and make them help self organizing Scrum teams build software.
So, let's get right down to it. In the latest Scrum Guide this is very clear:
* "Cross-functional [Scrum] teams have all competencies needed to accomplish the work without depending on others not part of the [Scrum] team."
* "[Development Teams] are self-organizing. No one (not even the Scrum Master) tells the Development Team how to turn Product Backlog into Increments of potentially releasable functionality; Development Teams are cross-functional, with all of the skills as a team necessary to create a product Increment;"
Yes, yes, I realize that orgs cannot get to this level of practice overnight, but let's not lose sight of the end goal. IMO, whenever you have a "transitional strategy" in Agile/Scrum, you should be clear what the ideal situation is, and how you plan to get to the ideal eventually -- and make it clear what you are doing currently and why.
I will give full credit to the original blog author for respecting the self organizing empowerment of Development Teams. I could see he was clearing show deference to that.
Nick, I appreciate your article, and I look forward to collaborating with you further since you are clearly very talented!
Professional Scrum Trainer
Re: Architects in SAFe
Charles Bradley, Scrum Trainer...
"The best architectures, requirements, and designs emerge from self-organizing teams."
Unfortunately SAFe doesn't take that approach -- it too uses the Ivory Tower architect model. SAFe has other problems too, as many have opined about in recent months.
Anyway, see my other response regarding Ivory Tower architects.
Re: Thank you for the write up
Thank you for your detailed and excellent response. You are clearly a knowledgeable and experienced Scrum trainer and coach. We need more people like you!
You advocated "Product Backlog Refinement" which is the same process that I indicate as "Pre Sprint Story Review". Our timing is a bit different but not that far off. I indicated that the architectural issues are worked out during that period prior to the kickoff of the sprint -- and you take the same approach. It sounds like we reached the same conclusion from different directions. Great minds think alike.
As for the "chickens and pigs" thing -- We dropped it as well. I used the nomenclature in my blog post to be able to communicate effectively to folks who are familiar with that. In the actual training, we used the tigers and monkeys. (most of our developers were in India -- if you think Americans don't like being called pigs, try Indians :-). Tigers get the work done while monkeys watch from the trees. We even had a little fable that sold it.
There is no "ivory tower" anything. The architect is accountable to the system and owns its performance, security, reliability, and fit-for-purpose. Problems roll uphill to the architect in this organization. About as far from ivory tower as you get... but while they are adept at code, they are developing architecture for a dozen different concurrent projects at any one time. We pay architects about twice what we pay developers. If we use an architect as a senior dev, we have wasted money. If an architect is not able to be accountable through the development of excellent partitioning, interfaces, platform selection, data migration strategy, etc, then he or she doesn't deserve to be paid that much and they can go back to being a developer.
On the other hand, decisions about partitioning, interfaces, platform selection, data migration strategy, etc, that are NOT made in advance are likely to produce unanticipated consequences on OTHER PROJECTS, OTHER SYSTEMS, and OTHER CLIENTS (this particular organization hosted a multi-tenant SaaS portfolio). Those considerations had to be made often months or years prior to the project and had to be followed through on, or consistency would suffer, costs would skyrocket, and service would suffer. That's simply bad business.
I think we agree more than we disagree. I look forward to someday being able to work with you to demonstrate what I mean. I'm sure you and I would have a very productive working relationship if the opportunity arose.
Thanks again for your detailed response.
--- Nick Malik