Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Articles 2018 State of Testing Report

2018 State of Testing Report

Key Takeaways

  • The testing profession is evolving, with testers expanding into tasks and areas previously far from their professional realms.
  • On the other hand, testers and their tasks are far from being irrelevant, and/or absorbed by the Dev Teams.
  • The flow of new testers into the profession is still strong, and we also see experienced testers staying for very long and fruitful careers.
  • The role of a tester does not remain around inspecting quality but it is also evolving to advocating it and accelerating it by educating other “non-tester” roles.
  • The State of Testing will continue reviewing the trends and giving visibility into the changing reality of our profession.

The State of Testing 2018 report provides insights into the adoption of test techniques, practices, and test automation, and the challenges that testers are facing. It shares results from this year’s testing survey organized by Joel Montvelisky from PractiTest and Lalit Bhamare from Tea-Time with Testers.

From the State of Testing 2018 report:

The State of Testing is the largest testing survey worldwide. With about 1,500 participants from more than 80 countries, the survey aims to provide the most accurate information of the testing profession and the global testing community. Held yearly, the survey also captures current and future trends.

This is fifth time that this survey has been done. Since most questions are similar to the ones in earlier surveys, it’s possible to see how trends have developed in the testing profession. There are also new questions which provide visibility into different angles of testing and technical aspects of testing.

InfoQ did an interview with the organizers of the State of Testing survey.

InfoQ: What are the major changes in the 2018 State of Testing report compared to previous reports?

Lalit Bhamare: The noticable change for me has been the realisation that testers are performing more and more activities which don’t traditionally identify as testing tasks. However, I see this as positive change and something that demonstrates multi-faceted calibre of tester role. 

This makes me even more curious as in where all it will lead testers and their contribution in product development in years to come. 

Joel Montvelisky: I think we are seeing an evolution in testing; we are expanding our responsibilities and the actions we take as part of our work (e.g. we are starting to work and monitor production environments as part of our comprehensive quality tasks).  We are also seeing changes to the technologies and types of products being tested.

I believe this testing evolution will eventually lead us to more technical testers, working hands-on and closer to developers, helping to achieve quality in their products in more ways than only by testing the system after it was written.

InfoQ: This year’s survey had a new question: the percentage of the job that goes into testing. What was the outcome?

Montvelisky: This question can be better understood in the context of additional things also asked as part of the survey, like what percentage of the testing is being done by non-testers, or what other tasks do you perform as part of your job…  With the idea of validating all the shift right, shift left, move up, etc trends that we are listening about all the time :-)

Still, as a stand-alone question we got some interesting data.  For me the highlight was to see close to 25% of respondents stating that for them testing is already less than 50% of their work.  Or in other words, we have ¼ of the respondents saying they are working in additional tasks in at least half their time.  And so it seems we are shifting (or I like to see this as expanding!) to other areas as well, such as monitoring production environments, coaching developers, interacting with customers, and in some cases even writing code for their applications.

Still, for me the most interesting aspect of this question will be the option to continue reviewing this data in future surveys and see how these numbers move one way or the other.

Bhamare: Joel has already covered the important aspect of this particular part of the survey. However, I feel that if we look at answers for this question together with the additional tasks testers are doing, it gives us a bigger picture of what is really going on.

I am happy to learn that 42% of respondents are taking up test coaching and consultation as part of their job, which is far different than typical 9-6 testing job. I can imagine the level of efforts and expertise it would take for an individual to make justice with that additional work. The future looks pretty bright to me if that trend continues to increase. 

InfoQ: How are testing salaries developing around the world?

Bhamare: The way I look at it, there has been an increase in salaries but a lot depends on the skills and experience testers have. Let’s not forget the geographical constraints that play important role too. 

For certain parts of the world, the increase has been marginal and for some it is steady but evident. 

Montvelisky: I think this is one of the questions with the most variance between the annual surveys we run, and this year is no difference.

There are some geographical areas where we are seeing increases, other areas where we see salaries staying more or less the same, and yet some areas where we see what can only be called corrections to salaries (towards lower levels of pay).

Overall, I think we are seeing mostly a slight increase in the salaries of testers in most geographical areas and experience levels.

InfoQ: How is the testing function embedded in organizations?

Montvelisky: The trends here show testing teams becoming smaller, spreaded within the organization (in many cases directly within Scrum or agile teams), reporting directly to functions other than a QA Director or VP, and overall providing their services closer to the development functions.  

It feels as if we are finally bringing down the organizational walls that used to separate us from our Dev brothers.

This does not seem to be universal, and the legitimacy of independent testing teams is still strong in some organizations that have not jumped the agile or lean wagons for any number of real and important reasons.

Bhamare: The way I look at it, there is clear indication that the decentralizing of QA/Testing departments and embedding dedicated testers to project teams is increasing. Of course, depending on the business model of organisations this picture may look different but if we are to talk about “trends” then “dedicated testers for project teams” seems a clear winner to me. 

InfoQ: The report mentions a significant increase in retrospective meetings. What causes this and what can be the result?

Bhamare: This is what I partly meant by multi-faceted role of testers and their contribution to project teams. 

By nature of their job-role and the abilities they posses thereby, I feel that testers are endowed with great observation skills that can benefit project teams to improve things that eventually add to product quality. These observations can be at the system level, application level, people level and what not, but the point is, there better be someone who “observes” things, analyses them and presents them in a form that enables team members to see things from a different perspective. 

Skilled testers, with their sharp observation skills can make retrospectives far more effective. As Jerry Weinberg explains in his ideas of System Collapse/Explosions and feedback loops to control them, “act early, act small” is the key and testers are naturally best candidates to make the feedback loop optimum for the controller. That’s how I look at this whole retrospective thing. 

Montvelisky: Again here we see another indication that testers are joining Scrum and agile teams, taking part in the different activities these teams do.  

And in parallel it also indicates that the teams, and the testers among them, are becoming more aware of the value of retrospectives, counting them as “static testing” activities where we have a chance to look not only at the bugs we are finding, but also at the incorrect processes generating these issues, in order to fix the process and not only the bug.

Testers have been striving to work in the quality of the process for ages, and in many ways agile practices such as retrospectives can give us a chance to finally achieve this goal.

The result of this type of processes will hopefully be a more integrated quality culture, where we celebrate and learn from our mistakes in order to improve our working methods and culture.

InfoQ: There was an open question on non-testing tasks that are done by testers. What came out of this?

Montvelisky: Open questions can sometimes be tricky but they are also incredibly interesting, as they provide an open platform for individual testers to express themselves and provide answers we could not foresee ahead of time.

Specifically in the question about non-testing tasks, we saw a number of recurring answers pointing towards testers working either closer to customers (organizing Beta Testing programs, or briefing customers directly on the functionality of the product), or representing these customers while serving as product owners within their teams.

We also saw a number of answers stating testers are now writing product code as part of their day-to-day tasks - aligned with the philosophy that teams are uniform and every member can and should be able to perform all actions.

Open questions are also an opportunity for respondents to release some of the tensions and frustrations they feel as part of their work...  Like the person who answered one of his non-testing tasks was to serve as a ZOO KEEPER, something I am sure many of us have felt one time or another in our testing careers.

Bhamare: A lot of interesting things among some obvious ones. I am personally happy to see that more and more testers are realising the value of monitoring production logs and using that information for improving quality. 

Testers contributing to “Customer support and training” is indeed an interesting activity that I would personally like to perform. If we are to dig it deep, this interaction with customers can be very well utilised for doing user tests and gather usability feedback. And with that knowledge, testers can contribute to product quality beyond their traditional area of expertise. 

However I am little disappointed too, because there is no mention for what testers are contributing to improve ‘testability’ of the products they are testing. I feel there is a lot more; testers can contribute to this part.  I have explained more of these ideas in my version of Session Based Testing.

InfoQ: What are the skills that testers need and how can they develop them?

Bhamare: Improving on technical efficiency has to be top priority, I would say, but this is also highly context dependent. 

In some context, domain knowledge combined with great communication skills, interactional expertise and of course great exploratory skills would be more important. 

The ability to automate is becoming important but at the same time, understanding test automation, when and how not to do it, how not to let it kill skilled brainual testing is an equally important skill that seems to be emerging.

How to develop these skills is the big question (giggles), but to keep it simple, I would say “practice”. The more aspects and skills needed for testing that are practiced, the more testers would develop themselves into a better version. 

Montvelisky: Over the years I’ve learned to see this question as two separate inquiries, one about soft skills and another about more technical skills and the technologies we are testing.

Among the soft skills, testers need to work on their communication skills, customer facing and business skills; as these are the ones we need to bring in the information key to achieving our visibility tasks and goals.

Having said that, we also see the need for more technical skills around the areas of scripting and code writing, API testing, and security testing.

How to develop these skills can be taken from another of our questions, where testers told us how they learn and expand their knowledge.  Here we saw many answers towards on-the-job-training, but also reading books, listening to podcasts, and taking part of courses and certification trainings.  

This shows that the paths to knowledge are many and different.

InfoQ: Any new developments in the challenges that test teams are facing?

Montvelisky: There are some slight changes in the challenges, but radical ones.  

Team size and training are more challenging than in previous years, maybe another sign that testers are being distributed among smaller agile teams and suffering the challenges from this change, where the testing responsibility is not always correctly distributed among all members of the team.

We can then turn to another question where we see what people are changing in their work in order to cope with these challenges.  Here we see people starting to coach developers to get additional testing resources. Some testers are switching from Functional Automation testing to more API automation, to make it faster and more robust.  We even see teams moving to more monitoring production environments in order to catch some of the bugs in production rather than trying to find everything in their testing and staging environments.

In short, we see some ingenious solutions to cope with our changing reality!

Bhamare: As Joel rightly pointed out, dealing with the changing time, coming out of comfort zone and finding ways to cope with changing surroundings are some of the key challenges testers are facing today. 

Quickly adapting to changing technological offerings and getting required training to cope with that seems like a challenge for some, actually many, testers. 

InfoQ: What will be important in the future of testing?

Bhamare: Let me take a deep breath first. 

Well, for me the important thing in the future of testing is going to be “having to share it” with different job-roles and managing it with authority at the same time, in such a way that quality of the end product is not compromised. 

As Brent Jenson has rightly said, the purpose of testing is going to be around accelerating the delivery of product with shippable quality. Testers would be required to master whatever skills it would need to make it happen. 

I see it as a challenging but extraordinary opportunity for testers. Our role seems to be going through an outstanding change and I am excited about it. 

Montvelisky: I think the report as a whole is showing that testing really is becoming a function distributed among the complete team, where the role of “the testing expert” becomes more to lead and coach the efforts, than to run all the testing activities.

I do not think this will happen overnight or in a handful of years, but there are enough indications this is taking place in the Industry and we need to be prepared for this change.

This means that our work will still be to represent the customer, but also to be more technical about our ways and the tools we use in order to perform these tasks.

Personally, I see this evolution as challenging but also reassuring, since we are finally starting to see a world where we focus on the quality of the process and the product, and start moving out of the world were we try desperately to test bugs out of products in an effort to inject quality after the code was already (sometimes) poorly written in the first place.

About the Interviewees

Lalitkumar Bhamareis currently working with XING SE as senior software test engineer. He is chief editor and co-founder of the famous testing magazine “Tea-time with Testers”, founder of and host of Techno-talks with Lalit. When not doing testing at his workplace, Bhamare teaches software testing as lead instructor with AST’s BBST Foundation class and regularly assists James Bach in his RSTA online class. He is a conference speaker and conducts public workshops on testing topics once in a while. Bhamare can be reached via his personal blog or via twitter @LalitBhamare

Joel Montveliskyis chief solution architect and QA manager at PractiTest. For the past 20 years he has worked as a QA manager, consultant and trainer for companies in Israel and in the US. You can check his thoughts and ideas as well as some of his webinars around testing related topics for PracitTest.

Rate this Article