Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News State of Testing Report

State of Testing Report

The State of Testing 2013 report contains the results of a survey done by Joel Montvelisky from PractiTest together with Tea-Time with Testers. The survey, which has been filled in by people from testing and QA communities, provides insight in the adoption of test techniques and practices, test automation, and the challenges that testers are facing.

Joel will give a presentation at the the QA&Test conference about how to become a more professional tester. InfoQ will be covering this conference with live news and Q&As.

InfoQ did an interview with Joel about the state of testing survey, agile testing, the skills of testers and testing challenges.

InfoQ: What made you decide to do a survey on the state of testing?

Joel: It started with a blog post I wanted to write about the challenges of the average tester in today's testing world. 

I looked on the Internet for unbiased information and I was not able to find anything.  I was amazed to realize that no one had done a study except for specific niches (for example, there were things for functional automation or for mobile testing, but nothing in general).

As part of the process to look for information I contacted my friend Lalit Bhamare from Teatime with Testers, and we both agreed that it would be a great idea to create a survey that not only reviewed the current status of testing around the world, but also one that would be run once a year allowing us to benchmark the progress of testing over the years.

We launched the State of Testing survey towards the end of last year (2013), and with the support from many people in the Testing Community who helped us spread the word and create interest in our project, we were able to reach a large number of testers from many countries and organizations around the world. 

The State of Testing Report was released around February 2014, and we were very happy with the feedback we got from many people from the Testing Community.

Together with the good reviews, we also got many comments and ideas for questions we should improve and additional areas we should cover more in dept. 

InfoQ: The report mentions that a large number of companies are using agile practices. Looks like agile is being adopted more and more in testing, do you know why this is happening?

Joel: Having been in Software Development and Testing for over 15 years I've seen many "methodological revolutions" so far. 

I remember when all of us thought that UML would be the method/language that would solve all our issues.  Then we all started reading about XP and how to cut process in favor of a more simple development approach, and in the last years it has been all about Agile and SCRUM.

I think the difference with Agile is that it is relatively easy to adopt since there is no real threshold of what you need to do in order to proclaim you are working based on an Agile process. Interacting with hundreds and maybe thousands of team each year I get to see how no teams (even in the same company!) tend to define Agile each in their own different way.

In a sense you can catalog Agile as an easy religion to follow :-)

I do believe that the basics of Agile are also in tune with today's market needs.  For example, the need to release constant updates fits nicely with shorter release cycles.  Or the need to work with Global Teams gives a strong push to the emphasize on communication over process.

Specifically about Agile testing?  If the development world works agile, it is very hard for testing community to continue working Waterfall or V Model...  But here again, I am not sure I understand the definition that many give to the term Agile Testing.  Even if you look at one of my favorite testing books today, Agile Testing by Crispin and Gregory, most if not all the principles in that book can be applied to all testing models and not only to Agile.

Testing will still be testing, even if we are working Agile.  What will change is not the "how" you do testing, but maybe "when" and "who" does the testing.  Our jobs as testers in Agile is still the same, but maybe our responsibility and timelines have shifted in order to match the needs of our teams.

InfoQ: Which (agile) practices, techniques and tools are mostly used by testers? Do you know why they use these ones?

Joel: I am not sure what tools are mostly used by agile testers, I work at PractiTest and we see many of our users successfully working with our tool as part of their agile projects.  But I am sure there are many additional tools in use by agile teams as part of their testing work.

What we do see is that Agile teams have higher percentages of automation than "non-agile" testing teams.  We see many different automation tools being used, from Selenium to TestComplete and everything in between. 

Another thing we see a lot is that teams are running their test automation as an integral part of the development process and not as an isolated testing operation.  We see many automation testers working with the same development frameworks such as Jenkins and Bamboo to integrate their testing directly into the build process.

Regarding techniques, we see a lot of Exploratory Testing together with Scripted Testing.  This blend helps teams to cope with the dynamic demands of their project while still maintaining a level of security that comes from having a defined but maybe more limited formal testing suite.

InfoQ: Which skills do testers consider to be important? What do they do to develop those skills?

Joel: I think the realization has come that the 2 most important skills a professional tester should have are Communication Skills and Self Learning Skills.

Communication skills will help us both to gather the information we need to successfully plan and execute our test cases, and also to know when and how to transmit the information gathered to the relevant stakeholders, for them to take the correct actions in the project.

Self Learning skills are also essential because our challenges, products and requirements are constantly shifting and changing, and there is no longer the time and chance to formally learn all the technologies and processes we need for our work.

The third skill that I think is a must to every tester are the Technical Skills that will allow him or her to understand the technical aspects of his project, ask the correct questions, and to asses the risks by him or herself.  The days when a tester could be isolated from the technical aspects of the project are way passed.

InfoQ: What are the challenges that testers are facing? How do they deal with them?

Joel: The biggest challenge we are facing is the fast pace of change in our projects.  I think that a tester who is not versatile and who is not willing to accept change as a part of his daily tasks is doomed to fail in the long run.

How can they (or we) deal with this?  The answer is relatively simple:  Keep an open mind and try to learn as much as you can as part of your work.

There are also the "normal" challenges like lack of time, or lack of process, or lack of budget.  But these are more trivial, and I also think that they are not new nor will they ever go away in a satisfactory way (no matter what we do or where we work).

InfoQ: Your presentation at the QA&Test conference will explore how testers can become more professional. Can you give the InfoQ readers a sneak preview and maybe some tips what they can do in their daily work to increase professionalism?

Joel: With pleasure!

Some of the tips are about your personal assets and go around understanding what you are good as a tester and where do you need to improve your skills.

Other tips are about the technical side of our work, and deal with taking part on the technical discussion of the project even if we are not the technical experts of our team.

And there are tips about proper testing work that talk about things like the importance of being your customer's advocate and how to communicate your results to your stakeholders.

I guess that to get more information your readers will need to come to Bilbao :-)

InfoQ: Was the state of testing a one time only survey or are you planning to do it more often?

Joel: We plan to make the State of Testing a yearly project and to provide information not only about the specific state of testing for a given year but also after 2 or 3 of these surveys to be able to find and analyze trends in our professional reality.

Rate this Article