Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Articles 2017 State of Testing Report

2017 State of Testing Report

Key Takeaways

  • Find out about the Annual State of Testing Report for 2017 which has just been published
  • See what is the world of software testing is up to
  • Read how the continued adoption of agile and DevOps impacts the tester's role
  • Learn about the latest trends and the techniques and tools that testers are using
  • Get insight into the challenges testers are facing and what they are doing to solve them

The State of Testing 2017 report provides insights in the adoption of test techniques, practices, and test automation, and the challenges that testers are facing. It shares results from this year’s testing survey organized by Joel Montvelisky from PractiTest and Lalit Bhamare from Tea-Time with Testers.

From the State of Testing 2017 report:

The State of Testing is the largest testing survey worldwide. With more than 1,600 participants from more than 60 countries, the survey aims to provide the most accurate information of the testing profession and the global testing community. Held yearly, the survey also captures current and future trends.

This is fourth time that this survey has been done. Most questions are similar to the ones in earlier surveys, which makes it possible to see how trends develop in the testing profession. There are new questions about professional development of testers, and a number of options of existing questions were added or modified in order to capture trends that have become more common in recent years.

InfoQ did an interview with the organizers of the State of Testing survey.

InfoQ: What are the major changes in the 2017 State of Testing report compared to previous reports?

Montvelisky: For me the most interesting thing is to see that testing changes as an evolution more than a revolution. We see a slow but bigger adoption of DevOps, we also see testers getting involved more and more in CI and CD, and overall playing a bigger part in the general trends that are affecting our development ecosystem as a whole.

Bhamare: In addition to what Joel has mentioned, I feel that the testing industry's openness and eagerness to embrace new challenges is more evident as compared to the past. In my opinion, it emphasizes that testers are flexible and open to embrace new ideas by finding their way to perform good under different contexts. It appears that testers are expected to be more technical as compared to the past but at the same time, they are expected to be skilled exploratory/thinking testers too with excellent "out of the box" thinking skills.

The 2017 State of Testing report sort of hints towards another big change in the testing space in the near future as we are standing right next to AI, machine learning, wearables, VR and other aspects that Joel mentioned already. 

InfoQ: How does the test profession keep up with new developments like testing AI systems and VR?

Bhamare: I feel that in order to test AI or VR effectively, testers would need to understand the technology itself in and out. The more concrete that understanding is the better they will be able to test it.

Testers’ ability to dissect products built on AI or VR would give them great benefit for designing more and more test ideas. I would personally try applying a variety of heuristics that I know (SFDIPOT, I SLICED UP FUN, FCC CUTS VIDS, Huh? Really? So? Etc.) in order to understand the product better. To keep up with such fascinating developments, I would recommend practicing critical thinking and develop a deeper understanding of the man-machine relationship.

For those who are serious about taking up this challenge, I would suggest to start with Weinberg’s “An Introduction to General Systems Thinking”. I can’t think of anything better as a perfect start in that direction..

Montvelisky: I agree that in addition to testing techniques testers need to learn about the matter to be tested.

Today, there are plenty of sources to get this knowledge, with the large amounts of data on the Internet, online trainings, and communities where people post and that are being updated all the time on the Internet.

A piece of advice that I think is extremely important for people, specially for testers starting to work on something new such as VR and AI, is to work as a user with these things. “Get your hands dirty” and experience first hand what your users will be doing with the products you are testing. This first hand experience will be second to none when approaching your projects and looking for those important bugs that represent the behavior of your users in the real world.

InfoQ: The report mentions that the testing function is reporting more and more to project management and less to a VP or director of QA. Do you know what causes this? What can be the implications?

Bhamare: I look at it as a combined result of change drivers like methodologies, cost/benefit models, technological changes and the way software gets built lately. Expectations from testers have been changing and project teams apparently count on them for things beyond pure quality assurance/advocacy. 

One benefit I see is that tester's can get significantly empowered if they get to work closely with project teams including programmers, POs, user care, users and marketing teams, etc. This facilitates communication and makes faster feedback possible, it also allows testers to contribute beyond their traditional job by collaborating with programmers (e.g. technical analysis of the code/defects, pair testing, doing deployments, production monitoring, etc).

The other implication could be that testers might feel alienated and compelled to follow the "project team's vision of quality" which can sometimes be different than what tester in team perceive it to be. If the project team is not well versed with "what it takes to do meaningful testing" then it might create a never ending struggle for the lone tester in the team. Anne-Marie Charrett has beautifully explained this kind of implications in her post leave the testing to the experts

It may also impact tester's career-path in general. What would a tester in a team mature into if there is no test team to lead or testing project to manage or central testing function to direct to? That's probably the question organizations will need to solve in the near future. 

Montvelisky: I think that what used to be concentrated testing teams that were hierarchically separated from the R&D are now being integrated into the functional teams and this is making a big change for the day-to-day work of the testers.

The question that I asked myself when I saw these numbers was less what happens to the testers and more "What is the future for the VP or Director of Quality?" and here I think my answer was to see this post as something that will have less direct responsibility for the testers and more methodological responsibility for the policies and practices followed in the organization.

InfoQ: How does the change in reporting impact tester's independence?

Montvelisky: I don’t think it impacts our independence, but it does require us to frame the information we are providing as part of the process in a different way.

If you are reporting to a VP of Testing that understands what you are saying and the issues you are describing, then this person can act as a proxy or a translator that will take the information you are providing, and present it to stakeholders within your organization in ways that make sense to them. Remember that each person will absorb information based on his knowledge and experience, so a non-tester such as a project manager of CTO will need some additional interpretation of the results for them to make sense.

When we take out this “middle man” then the tester needs to invest time in order to provide this information in ways that are actionable for the stakeholder they are communicating with.

In this case we are not really loosing our independence, but we do need to work harder in order to get our message through.

Bhamare: Hmm…that’s tricky. I would say it depends. If the teams or managers that testers are reporting to have hands-on testing experience or reasonably good understanding of how meaningful testing is done then it will definitely help testers and that can even empower them to contribute better in projects.

On the contrary, if testers end up reporting to someone who thinks “testing is all about executing checks that can be fully automated” then that is going to cause more bad than good for everyone. I have expressed my feelings and opinions on this topic in a dedicated blog post: No Testing, No Agile and No Software Delivery .

Of course the mediocre testing that has been happening for years does have a share in why some people think like this. But looking at the survey results and comments made by testers, I feel that things are much better than they used to be in past. Regardless of the reporting structure, testers are taking responsibility for their job and are keen on doing what it takes to do "meaningful testing" and I am quite happy about that.

InfoQ: What's the effect of the continued adoption of agile and DevOps on the tester's role?

Montvelisky: I think that the day-to-day life of testers working in Agile and DevOps teams is definitely more filled with adrenaline and excitement. We are being asked to work closer to the action, to provide faster feedback and to lead our functional teams where all of a sudden developers need to be testers and sometimes vice-versa.

The biggest challenge I see here is maintaining the level of testing professionalism required to deliver a good product. It is harder to apply breaks and ask the difficult questions when we are in the middle of the development action, but I think this is the biggest additional value we can provide in teams (together with our customer point of view of the product)

Bhamare: As mentioned earlier, I see "collaboration and faster feedback" as biggest advantages for testers working in agile/DevOps. This adoption requires testers to posses more technical skills that are indeed helpful. 

At the same time, it also warrants testers to be critical thinker and posses the ability to clearly communicate the risks/information without being a bottle-neck for product release. That said, adoption to agile and DevOps warrants testers to be "the jack of many skills and master of some that matter the most to project teams."

InfoQ: What are the main challenges that test teams are facing? How do they deal with them?

Bhamare: Dealing with high volume of testing in shorter periods of time continues to be the top challenge for test teams. Other key challenges can be summarized into one which is "dealing with changing expectations". High programmer to tester ratio, increased responsibilities beyond typical testing tasks, increased learning curve, and lack of clear direction and career path appear to be making things challenging for testers. 

Intelligent automation definitely helps. Knowing the right tools that can be used for expediting mundane tasks. Making project team understand that "Quality" is team's responsibility and not that of just a tester in team can greatly help. Implementing WIP limits in agile/Scrum teams can encourage non-tester team members to jump in and help in testing. Pairing with Programmers is the best way in my opinion to teach them how to test and at the same time gain more technical knowledge from them in exchange. This is likely to help testers with their learning requirements to a greater extent and can make them even more productive for project teams.  

Montvelisky: Our biggest challenge as testers is still coping with the tight timelines of our projects, that are getting tighter due to the adoption of agile and DevOps and the shorter calendar time we have to release products to the fields or deploy them to the cloud.

The only way to meet this challenge is to evolve and adapt! Automation is not a luxury anymore, we must have it in order to handle the repetitive tasks and to be able to focus our efforts on the more exploratory tests to be carried in the new functionality.

Another way teams are handling this is by making sure developers are more aware of the quality they need to deliver, and simply teaching them to test better themselves - yes this can be done if you really need to!

InfoQ: Which new technologies are being adopted by testers? How they impact the test profession?

Montvelisky: I think the technology that is helping testers the most nowadays is cloud computing and the ability to create testing environments on the fly to perform our tests.

There are teams using this technology directly, working in their own environments in AWS or Azure, but even those that are not doing this are still gaining benefits from working with tools that do this for them seamlessly. Regardless if you are talking about automation, load testing, or test management in general. Cloud computing and hosted systems are making the lives of testers better.

Bhamare: In addition to cloud computing as Joel rightly said, I think other technologies like Augmented Reality, Virtual Reality, wearables and mobile technologies of various forms are new cool things testers are working with today. The nature of these technologies challenges testers to think for different scenarios and for different elements of a product (see the Heuristic Test Strategy Model by James Bach) so that assessment of quality happens with broader coverage. Though this sounds challenging, I think it makes the tester's job equally exciting and gives them the opportunity to become the epicenter of new age software development.

InfoQ: What are the new or upcoming skills that testers need to be aware of? How can they acquire them?

Bhamare: Apart from more technical and technological awareness, I don't think testers need to be aware of any additional skills. It's just that they are likely to be required to implement their existing skills under different and demanding contexts. In order to do that effectively, testers must be aware of how their role is changing and how to apply their wisdom under different contexts. And that's the "key skill" in itself testers would need to master.

In order to acquire and master that, I would recommend to practice testing (skilled exploratory testing rather) and learn from the experiments/observations, be open minded and never stop learning. Fiona Charles has explained it beautifully in her recent interview testing challenges and essential skills for testers she did with InfoQ :). If anyone feels stuck, I would recommend them to go through it. 

Montvelisky: The survey showed that the most important skills for a testers are still the "old ones" like communication skills, functional testing, web technologies, etc. But we are also seeing more people naming new skills like testing in the cloud, IOT testing, and big data testing.

I have no doubt these will continue growing in importance in the years to come

About the Interviewees

Lalitkumar Bhamare is currently working with XING AG as Senior Software Test Engineer. He is Chief Editor and co-founder of famous testing magazine “Tea-time with Testers”, founder of and host of Techno-talks with Lalit. When not doing testing at his work place, Lalit teaches software testing as lead instructor with AST’s BBST Foundation class and regularly assists James Bach in his RSTA online class. He is a conference speaker and conducts public workshops on exploratory testing once in a while.

Joel Montvelisky is Chief Solution Architect and QA Manager at PractiTest.  For the past 20 years he has worked as a QA manager, consultant and trainer for companies in Israel and in the US. You can check his thoughts and ideas at as well as some of his webinars around Testing related topics for PracitTest -

Rate this Article