The analytic hierarchy process uses pairwise comparisons and scoring for criteria between the alternatives to give insights into what the best option is and why. John Riviello spoke about applying the analytic hierarchy process to decide what JavaScript framework to use at QCon New York 2023.
When a decision was needed to pivot to a new JavaScript framework for one of their web apps, Riviello came across the Analytic Hierarchy Process (AHP). There were three groups, each with strong opinions on what that new framework should be, as Riviello explained:
The VP of engineering of my organization told me that he trusted the team to make the right decision, but wanted me to explore how I would know the process we would use to arrive at that decision was effective.
Like most decision-making frameworks, you start with some options and criteria to evaluate them, Riviello said. Where the AHP differs from some other more common process is that it involves doing pairwise comparisons (i.e. comparing only two items to each other at a time) for each option with respect to each criteria, and also pairwise comparisons with each criteria with respect to your overall goal:
There is a scale it uses from 1 to 9 to compare two items, with 9 being the case where one item is by far superior to the other, and 1 being the two are for all intents and purposes equal.
Once you have all those comparison numbers, it has a formula to give you insights into what the best option is and why, Riviello mentioned.
For their particular case, they used seven criteria to evaluate the three JavaScript frameworks they were considering: Community, Performance, Redux Compatibility, Web Components Support, Localization, Developer Productivity, and Hybrid App Support, Riviello explained:
After going through the exercise, it was clear Community and Performance were most important to us, with Developer Productivity being a close third, and the others not being as critical to our decision.
AHP is just a tool in the toolbox when it comes to making decisions and getting buy-in, Riviello said. Like any decision, how you go about soliciting feedback, getting approval for, and announcing the decision is critical to how well it will be received and acted upon. Riviello mentioned that the concept of nemawashi, which favors building consensus openly instead of trying to force consensus, goes along well with AHP:
The best way to leverage and commute decisions that use AHP is to be open about the inputs and the results.
The original JavaScript framework AHP exercise took almost six hours total, but it was six hours well spent to arrive at a common decision that we would be rallying around and living with for years to come, Riviello concluded.
InfoQ interviewed John Riviello about applying the analytic hierarchy process.
InfoQ: How did you adapt the Analytic Hierarchy Process to your needs?
John Riviello: The recommended process is to collect results on the pairwise comparisons via individual surveys and then use those to determine the results. The adjustment we made was we had participants come up with their pairwise comparison numbers in advance, but instead of blindly calculating at that point, we came together and discussed our numbers for each pairwise comparison until we had an agreement on what each pairwise comparison number should be. We adopted this from how we approach applying story points to user stories.
InfoQ: What JavaScript framework did you pick?
Riviello: One JS framework became the clear winner. But never share what that framework was because it is important to note that our criteria were specific to our team’s needs at that time, so I’d recommend others who are looking to follow the process to pick their own JS framework to come up with their own criteria and evaluate them.
InfoQ: What came out of the retrospective on using the Analytic Hierarchy Process?
Riviello: Many groups at Comcast have used AHP by this point, and the feedback has always been favorable. Many groups capture Architecture Decision Records (ADRs), and including the charts of AHP results in ADRs can be useful for those that go back to review them to understand why a decision was made.
We’ve also had groups mention that by going through the process, they’ve learned more about other teams (for example when teams that previously did not work together much in the past had to come together to build a new system or update an existing one).
InfoQ: What advice can you give to teams that want to experiment with the Analytic Hierarchy Process?
Riviello: Give it a try! The tool I opened sourced at Analytic Hierarchy Process (AHP) Tool seeks to make it easy to capture your AHP data and generate visual charts of your decision.
One important thing to note is when it comes to the number of options or criteria, you do not want to have more than eight, otherwise the number of pairwise comparisons you have to do will increase dramatically.