Brandon Carlson gave a talk at the Agile 2012 conference titled "Stop Listening to Your Customers" in which he made the case for using deep analytics and application instrumentation to discover customer behavior and use that to help guide product requirements. He spoke about the impact of wasted effort and unused features and how just listening to customer requests will result in unused, bloated products and how commonly used requirements techniques have built-in biases which lead towards building products that don't met customer expectations.
He started by stating that many of the technical problems ("building the product right") facing software development have been solved, teams today know how to build good software. The area where we still fall down is in deciding what features to build into our products ("building the product right"). He discussed the Standish Group survey results that indicate that 64% of features in a typical software product are seldom or never used. He maintains that this is because we listen to our customers.
He went on to tell the story of a mapping software product where there were many customer requests for a printed wall map feature - easy to build in the software but which would necisitate the company spending $150000 on a large format printer and a new logistics process to print and ship the printed wall maps. Instead of building the feature as requested and implementing the new business process they initially designed the featre to send an email with the map image file to Brandon to take to the local copy shop and have printed off site. Over the next six months the feature was used a total of three times. They then changed the feature to send the email with the image file back to the customer and give them the capability to print their maps at a local copy shop, saving the organisation thousands of dollars on a feature which was very seldom used.
He maintains that it is actually important to listen to what customers state they want, and even more important to understand what they actually do - combining listening with seeing in the field.
He went on to explain that the most common approaches (interviews, focus groups and surveys) used for listening to customers have bias and problems built in to them:
- Response Bias - when asked a question most people will respond with what they think you want to hear, rather than what they need. He used the example of a question such as "would you want this available on the iPad" to which most people will respond with yes, even if they do not own an iPad.
- Social Norms - people will often hide their true motivation behind polite facts For example when asked "why did you purchase xyz product over abc product" the reason given may list features which appear to be missing, but the actual reason may be as simple as "I didn't like the sales person". The list of missing features becomes an enhancement request, whereas it was actually just a convienent way of deflecting a difficult question.
- Stated & Revealed Preferences. "Actions speak louder than words". He discussed the question "do you recycle" to which most people will respond "yes", but observing their behavour it becomes clear that for many people the response is actually "yes, I think recycling is a good idea and I will do so if it is easy and convientent, but not if it requires much extra effort or inconvienence on my behalf". All interviews, focus groups and surveys provide information on stated preferences - revealed preferences can only be found by observing behavour.
He went on to discuss how to find revealed preferences as that will provide the best information to decide on features and capabilities of products. Specific ideas he suggested are:
- Within the current application - use log files and other analytic tools to see what actions users actually take and then explore what they were trying to achieve.
- Within other internal applications with similair characteristics - again using analytics and logs to understand what users do and then explore why they behave in certain ways
- From Beta softeare releases - provide new functionality to a small group and monitor their actual usage of features to see if they behave as expected
- Application instrumentation - buildnew logging and/or analytical capabilities into the product to answer specific questions about use and behavour. He strongly advices keeping this simple
He summed up his talk with two pieces of advice:
- Be skeptical - what is asked for is frequently not what is actually needed
- Use the data - tools and analytics can identify behavours and needs which must be examined further