Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Articles Testing the Internet of Things: The Human Experience

Testing the Internet of Things: The Human Experience

Testing in the IoT world means intimately understanding the human user

Mobile and embedded devices, more than any other technology, are an integral part of our lives and have the potential to become a part of us. This generation of mobile and embedded devices interacts with us, not just awaits our keystrokes. They operate in response to our voice, our touch, and the motion of our bodies.

Since all of these devices actually function with us, testing how the human experiences these devices becomes imperative. If we do not test the human interaction, our assessments and judgments of quality will be lacking some of the most important information needed to determine whether or not the device is ready to ship.

In this article, I will provide an understanding of what “human experience” testing is and is not , and using concepts from human computer interaction design theory, establish a framework for developing “human experience” test scenarios.

Why is “human experience” testing so important to mobile and embedded devices? What makes this different than what we have done in the past? Because when a mobile device is physically attached to us and works with us and through us, the more important the results of the interaction or collaboration becomes to us emotionally and physically. The user, a human being becomes a part of the “internet of things”.

For example, think about that feeling in the pit of your stomach when you are standing at the gate, ready to board your flight and your boarding pass doesn’t come up on your phone. Think about the aggravation you feel when your hands free device keeps falling out of your ear and after you finally get it in, it calls for a pizza delivery instead of calling your friend, Peter.

“Human experience” testing, or lack thereof, can lead to redesign of software, and sometimes, of the device itself. So what is testing the “human experience”? Although initially, usability comes to mind, human experience testing goes much deeper. Usability testing focuses the ways in which users accomplish tasks through the application under test.

Then the question becomes just how does “human experience” testing differs from usability testing? The answer lies in the scope, depth and approach. “Human experience” testing focuses on the actual interaction with the device. It involves not only the look and feel and ease of use, but also our emotional, physical and sensory reactions, our biases and our mindsets. It involves testing in the “real world” of the user; when, where and how the user and the device will function together.

“Human Experience” testing also focuses heavily on the value that the device provides to the user. The negative test scenarios are even more critical than the positive tests. This is because many of these involve failure on the part of the device. Non-functional testing, i.e., testing the “ilities” also has an incredibly important “Human Experience” testing component. For example, if even during one race, my watch fails to boot up or catch the satellite when I’m on the starting line, I no longer trust it. In the case, testing reliability is also about testing for the user’s trust.

“Human Experience” testing has the following components of human interaction with the device. We should test all things physical, including sizes, shapes and genders of the users. We should also include sensory reactions including sight, sound, and touch. Orientation or the interaction with human movement is an incredible crucial part of the test.

We must plan for testing in various geographical locations, different weather conditions and contexts. Finally we must consider value and most thoroughly test in terms of the users’ perceptions, mindsets, biases and emotions when interacting with the device.

The Importance of Human Experience Testing: An Example

To illustrate the importance of this testing, I’ll share a very personal example. It is a tale of two mobile devices attached to one woman, a marathon runner. Yes, me.

Join me on the starting line of the 115th running of the Boston Marathon, April 18th, 2011. I’m standing in my timing corral, excitedly anticipating the sound of the starting gun. Last year, I surprised myself by qualifying for Boston; only 10% of runners do, and I’m hoping for another qualifying run.

I have pinned my bib on the front of my shirt, carefully keeping it flat as it contains the RFID chip that will record my race for the Boston Athletic Association. The chip will record my time as I run over mats at various miles in the race. My current time, location on the course and my anticipated finish time will appear on the BAA website and will be texted to my friends’ and family’s smartphones so they can track my progress during the run.

But I also have a backup, my trusty Garmin watch. I click on my Garmin and anxiously await it’s catching the satellite to start the GPS. It’s ready and the gun goes off. I’m careful to click the start button at the exact moment I step over the starting line to ensure a correct timing. As I run along during the early miles, I check my watch for the pace, to validate that I’m running the speed I’ll need to qualify. As I push myself up Heartbreak Hill at mile 20, I check my heart rate monitor for feedback confirming that I can continue to run my current pace or that I can continue at all. It reassures me that as exhausted as I feel, I’m doing fine.

As I look at the elapsed time on my watch, I confirm that I’m on pace to reach my goal of another qualifying run. As I turn left on Boylston and the finish line is in sight, look at my watch to see that, not only a qualifying run, but also personal record, is in reach! I dig in and give it everything I have left. As I cross the finish line, physically totally spent but emotionally charged, I click my watch off and I see it… My qualifying time and my personal record! The feeling of accomplishment and elation is beyond description!

Now I’m in the car, riding home, just basking in my own glory. My cell phone rings and a friend asks my gently what happened. I hear concern in his voice and wonder why as I tell him about the best run of my entire life. And then he tells me, “Your run isn’t on the BAA website”. My elation immediately turns to grief. The chip, the timing device embedded in my bib, had failed to track my run. The only record of my qualifying run and my personal record is held within my watch. At that moment my watch becomes a part of me. As one runner once said, “the pain is temporary, but the time lasts forever”. And now my Garmin holds the only record of my accomplishment. What if it didn’t save?

Immediately upon arriving home, I go directly to my laptop and download my watch. My heart is literally in my mouth as I wait for the time to come up on the screen, documenting my time forever. And there it is, 3:51:58! My qualifying run and personal record are mine forever. And I will be on the starting line in Hopkinton for the 116th running of the Boston Marathon next year due to the collaboration among my body, my mind my emotions and my watch.

The lesson is that devices that interact intimately with the user require a different type of testing than other types of embedded systems. The closer a device is to the human user, the more it requires human testing; it requires testing the interaction between the device and users’ actions, senses and emotions.

Testing the Human Experience

So how do we begin to plan for testing the Human Experience? We need to understand the human requirements. We can gain this insight from many sources including media reviews for device class, CI/UX guidelines, and by searching the web for what people are doing with devices within that class. If we are testing a new device, we can observe participants using the device in prototype lab or field. In this setting, we can watch and listen to their reactions and possibly interview them regarding their feelings and impressions of the device.

Obviously, the test approach for Human Experience testing involves not only “field” testing, but testing in the real world of the user, but how do we design our test scenarios? The test scenarios need to be based not only on the characteristics of the users but also on the value that the interaction and collaboration with the device provides. The most effective way to design human experience tests is using human computer interaction design principles. By developing “Personas”, we delve into the users’ personalities and characteristics and we begin to understand their expectations of the device. Then we create “User Value Stories” to test the ways in which the human user will achieve value from the device.

For any particular mobile device or wearable, there may be several different “Personas” and there will be several “User Value Stories” for each “Persona”. If the device will be worn by an animal, for example the Whistle, a fitness tracker for dogs, there will be multiple personas for both the dogs and the dog “parents” and there will be several “User Value Stories” attached to each set of “Personas”.

Test Scenarios based on the “Personas” and “User Value Stories” include testing the physical, sensory, orientation, geographical and contextual expectations that the users have of the device or wearable. The test scenarios also involve the emotional, physical, sensory reactions, biases and mindsets as well as the social expectations and interactions of the user. A “Persona” is a model of a potential user. A “Persona” is developed by listing all of the characteristics of the user including age, family background, level of education, occupation, socio-economic status, physical size and condition, gender, hopes and desires, point of view, values and life expectations.

As an example, let’s develop a “Persona” to test my sports watch, based on my marathon story above. Let’s begin with my demographic characteristics. Since I am a marathoner, I am a distance runner. I run on the road; a trail runner would have a different persona. I am obviously conscious about health and physical fitness and running is likely a lifetime sport. Most marathoners are over thirty years of age, however; personas should be built for various age groups. I am a woman and small in stature. A persona should also be written about a male runner. Marathon runners will train and race in all types of weather conditions.

Now let’s complete my “Persona” with the psychosocial characteristics. I am a member of the running community and I train both alone and in groups. Running is a source of fulfillment; I thrive on both individual achievements through personal records as well as competition with my fellow runners.

From this “Persona” we can derive many of my expectations which become test scenarios for my sports watch. For example from my demographic characteristics, since I am small in stature, the watch must fit my wrist and as I’m older, I need the screen to be legible without my glasses while I’m moving. From my psychosocial characteristics, since I’m competitive, I require the watch to be accurate and reliable.

The “Personas” provide the basis of the test scenarios for testing the interaction between the user and the device, but for complete Human Experience testing, we must also test how the device provides value. This testing is designed from User Value Stories.

Like any story, a “User Value Story” has a beginning, middle, and an end. It has a main character, who is one of the personas that have been previously built. It discusses common practices when using the device and how the device might help in these situations. It includes realistic situations and field conditions like weather. The sensory perceptions including sights, sounds, tastes, smells, what the user may be touching are extremely important to include as are the user’s feeling and emotions. It is written in paragraphs as you would write a story.

Although there are many “User Value Stories” that are derived from my marathon story, let’s use just one as an example. Let’s build a user story for the end of the marathon. I am in mile 25, I turn right on Hereford Street, drag myself up the slight incline which at this point in the marathon, feels like an incredibly steep hill. I am sweaty, exhausted and still pushing hard for my personal record and qualifying time. I turn left of Boylston Street and the finish line is in sight. As I race to the end, listening to the crowds cheering, I step on the mat and click my watch at exactly the same time in order to record my exact time. I am thrilled to have completed my run in both a qualifying time and a personal best. I press the save button to save my time and reset the watch and then I turn the watch off.

In conclusion, by combining just this persona and user value story, you can see how many human experience test scenarios can be derived and you can see just how critical these points of interaction become to the user You can limit the number of scenarios you test by using risk-based testing. Most importantly, from building “Personas and User Value Stories” and developing the test scenarios that derive therefrom, we can understand that the closer mobile and embedded devices become physically to the user, the more important testing the “Human Experience” becomes.

About the Author

Gerie Owen is a Testing and Quality Assurance Consultant, Certified Scrum Master, Conference Presenter and Author on Testing and Test management topics. She has implemented the offshore model, developed and trained new teams. Gerie manages large, complex projects involving multiple applications, coordinates test teams across multiple time zones and delivers high quality projects on time and within budget. She enjoys mentoring new QA Leads and brings a cohesive team approach to testing. Gerie chooses her conference and writing topics based on her testing and test management experiences, what she has learned from them and what she would like to learn to improve them

Rate this Article