Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Articles 2021 State of Testing Report: Testing Careers, Covid-19, Agile, Coaching & Training, Automation

2021 State of Testing Report: Testing Careers, Covid-19, Agile, Coaching & Training, Automation

Key Takeaways

  • The vast majority (71%) of respondents mentioned that the Covid-19 crisis has not affected their income. But we do see an increase in the level of uncertainty of testers around their professional future.
  • The average experience of testers is growing - pointing at more professionals who see testing as their career and not only a stepping stone in their technical professional path.
  • Over 90% of testers are working Agile and above 40% DevOps, while Waterfall keeps shrinking (currently at 27%), showing that these have become the de-facto standards of the industry.
  • The role of testers in their organization is starting to encompass more and more tasks around coaching and training others on their testing tasks.
  • Around 75% of testers reported they are already working in automation and scripting-related tasks, cementing the fact that testing is a technically-oriented profession.

The 2021 State of Testing Report provides insights into the adoption of test techniques, practices, and test automation, and the challenges that testers are facing.

As the report shows, testing is not something done solely by testers, and is becoming more and more integrated with other activities in the software development life-cycle. In many organizations, senior software engineers, software architects, and team leads, are working with testers or have testers in their cross-functional team, or they are actually doing testing or managing testing.

As we see in the report, a lot of testing is done by "non-testers" who are being coached and trained by testers which can be challenging. It’s important for anyone involved in software development to know about the main challenges in testing, the things testers are worried about, and what testers have learned over the years.

From a methodology point, many teams are adopting agile, where in testing there's still a lot of waterfall. The confirmation that agile has become the defacto standard for testing (which confirms the trend towards agile the State of Testings reports have been showing in the past year) can help to work further on integrating testing activities with architecture and development in an agile and/or DevOps approach.

The shift left of testing work impacts activities in architecture and design, and even product management, as testers are getting more and more involved at that stage. It also encourages architects to think about the need for testing, and for test automation, to think about test architecture aspects.

The State of Testing report explores how testers learn and touches upon technical coaching and training. The learnings may be relevant for designers too, as they could be facing similar issues in their technical work (or have solutions where they can pair with testers).

The distinction between design and testing has been reducing over many years. Testers initially had their own networks and conferences for learning but these are opening up more and more for "non-testers" and are fostering collaboration, which is something that makes sense for developing better products faster.

One of the findings from the report is "more respect for testers," which is a cry out for the important work that testers do. It’s good to be aware that this is how many testers feel.

InfoQ readers will need to register themselves and will be signed up for the PractiTest newsletter before being able to download the 2021 State of Testing report.

InfoQ interviewed the organizers of the State of Testing survey, Joel Montvelisky from PractiTest and Lalit Bhamare from Tea-Time with Testers.

InfoQ: What are the major changes in the 2021 State of Testing report compared to previous reports?

Lalit Bhamare: The major change I would say has been about information we have tried to gather about the pandemic and how it has affected the testing profession, compared to previous reports.

For some other critical aspects we chose to continue finding more information to identify the patterns and trends.

Joel Montvelisky: When I look at the answers we got, other than the Covid-19 Pandemic as Lalit mentioned, we see the continuous shift towards more technical testing as more organizations adopt Agile and now DevOps as part of their process.

On the other hand, we also see answers aligned with "Shifting Left" trends, including more work around validation and even the gathering of user stories, as part of the tasks taken by testers within their teams.

InfoQ: What effect did the Covid-19 pandemic have on income?

Bhamare: Not as much as I feared it to be. And I am happy about it. Around 28% of the respondents reported being affected in terms of salary cuts or reduced work hours, or for some of them there was no annual pay-raise.

However, people also reported to have found better job opportunities and pay-raises as their employers saw growth in business. "Somebody’s loss is somebody’s gain" kind of situation, I would say.

Montvelisky: As Lalit mentioned, the effects seemed to be lower than what I was expecting. But I’m afraid that we have not seen the end of it and we will need to continue following up on this on our next State of Testing, to see if 2021 brought with it further changes that can be traced to the ongoing Covid Pandemic.

InfoQ: What methods do testers use to enhance their knowledge and sharpen their skills?

Montvelisky: The main sources of knowledge are relatively constant over the years. We see testing books, conferences, webinars and certifications always grading high on these questions.

One point that surprised me a little this year was the increase in people mentioning peer mentoring as one of their sources. The surprise did not come from the method itself (as I think it is one of the best ways to learn and teach testing) but from the increase in popularity especially this year when most of us have been working in relative physical isolation.

Bhamare: There has not been any major difference as compared to results from previous years. On the job training and conferences still remain key contributors to how people learn.

Interestingly enough, the effects of the pandemic and learning taking more of an online format seem to have encouraged people to learn from online platforms like Udemy, Test automation academy etc.

InfoQ: What changes do you see regarding testing models and principles that are being used? What may cause these changes?

Montvelisky: There is not much more to be said about Agile that is now on over 90% of organizations. DevOps keeps also expanding and I believe that as more organizations undergo "digital transformations" it will also become more widely used.

On a separate point, although not less important, we see that BDD keeps expanding as an approach and I think this is related to the increased adoption of automation as the main testing resource in many organizations.

Bhamare: Agile or Agile like continues to grow in terms of its use while there has been an increase in people who reported following BDD. One of the possible reasons could be the whole team approach and close collaboration between product and engineering teams, but that is just my guess.  

Knowing how exactly people are using BDD and how they follow it might tell us more about the possible changes. I would like to know how people do BDD at their workplace to be able to figure out what exactly it means to them, and the underlying causes of that shift.

InfoQ: What activities are being done by testers and developers as part of their work outside of their usual testing tasks?

Montvelisky: I see in general that we are really shifting both right and left on our testing and quality related tasks.

Shifting left is shown by the number of respondents working either on user stories or requirement-related tasks. Shifting right can be seen by the work being made on production issues and in our responsibility around the testing and development environments.

This means that we need to expand our horizons, both in business-related areas such as customer behaviour and market trends, as well on the technical aspects related to our server infrastructure and other possible production issues.

Bhamare: This year we have a new entry of "validating the user stories" and "risk analysis" which makes me positively surprised. This makes me happy since I firmly believe this helps the teams a lot more if they spend enough time, and thinking on requirements, risk analysis earlier in the process. Why I think this is good idea is better explained in methodology I preach, i.e. Quality-Conscious Software Delivery. Seems like people are realizing for themselves that investing early in these things helps them save a lot more if not done otherwise. The number of people reporting "test coaching and consulting" as an activity they perform continues to grow.

There is considerable decline in people who analyse production and other user data. Either this is automated, or their organisations don’t find it very useful. It would be interesting to see what exactly it is getting compensated with. I would assume they are investing more in early testing, risk analysis and also in testability perhaps.

InfoQ: What are the developments in the methodological approaches for testing?

Bhamare: There has been a continuous rise in test specification techniques that people use (equivalent partitioning or sampling or boundary value analysis e.g.). At the same time there is equal rise in use of exploratory testing techniques and session-based testing. Traditional script-based checking has gone a bit down while it grew up last year. This up-down in this particular area seems to be a topic of further investigation.

There are some people who do code walkthroughs, and surprisingly review of production logs has gone down. I am unable to figure out the possible reason behind this change. It would be interesting to find it out.

Montvelisky: In this section I actually see some stagnation on the techniques we use.  It seems like there is no big new methodological development and this is something we need to keep our eyes on.

We do see more and more adoption of automation and a shifting towards user story and production environments testing, and this in itself may be the big change as it will be better perceived with the passing of a couple of more years.

InfoQ: What are the biggest challenges that testing teams face? How do they deal with them?

Montvelisky: It is interesting to see professional testing training as an important challenge, and one that I believe we do not pay enough attention too. This is a direct result from the adoption of real Agile and specially DevOps methodologies, and as a requirement for organizations embarking on a whole team testing approach.

Also, coping with timelines is interesting; I believe this is still a side effect of most Agile and DevOps methodologies focusing mostly on development tasks, leaving testers to adapt as best as we can.

Bhamare: It pains me to say that early involvement of testing is still a challenge for many and that indicates the sad state of affairs with organisations that are still not able to solve this problem for their own good.

The time testers waste in fighting with the environment and test data management remains one of the biggest challenges for testers. Getting proper training is also a challenge for some.

All in all, not much seems to have changed in the challenges testers have been facing historically.

The reasons for some of these could be tied back to answers we received in the open question section, such as lack of respect for testers and the profession, organizational culture issues, lack of understanding of testing by non-testers, and so on.

InfoQ: What are the skills that hiring managers are looking for in testing jobs?

Bhamare: Apart from automation knowledge, communication skills and exploratory testing skills, there is heavy emphasis on soft skills.

Some of the managers are looking for skills with keyword driven tests, API testing and SQL query writing skills. Programming skills and logical thinking is being desired too.

Montvelisky: In addition to what Lalit mentioned, our managers are asking for more personal and professional flexibility in order to work with developers (outside our core testing tasks); I think this shows very clearly how the lines are blurring between our tasks and those of our development peers.

InfoQ: What new technologies or subjects for testing that will become important in 3-5 years from now have been brought up by the respondents?

Montvelisky: I think that the next step in the AI/ML expansion will be towards more versatile and intelligent robots.  We are already seeing it in warehouse management, security patrolling, and who can forget about self driving cars.

With this in mind, I think we will see a renaissance in embedded systems testing, and also this new branch of Intelligent Machine Testing will include parts of AI and hardware testing.

Bhamare: Machine learning naturally takes the first spot here. Among others, people have reported IoT and microservices, AI testing, VR , cloud tech and automation being at the center of it all.

InfoQ: There’s an open question in the survey on what testers would like to see changed in order to help them to do their job better. What came out of this, and what’s your advice for getting the software industry to act upon this?

Bhamare: At the top of the list, "more respect for testers" remains followed by "abolish the assumption that exploratory testers are less skilled". There is also a mention of 100% automation in all contexts being an unrealistic expectation. And empowering testers still remains the pain point.

My advice would be to educate non-testers about testing, especially if they get to decide about testing in their organisation. Having a programming background and expertise and getting to lead the engineering team (which now-a-days also has testers) does not make testing knowledge optional. I think the root cause of this problem is lack of awareness and education about skilled testing in the industry, especially among the decision makers who are responsible for deliveries and engineering teams.

Montvelisky: What caught my attention with this question was the way testers show the need to expand testing outside our organic teams, asking for greater adoption of whole-team testing and better coordination between Scrum teams.

I see this as a very mature understanding of the current situation, where we cannot be tasked with running all the testing in our teams, nor can we spend all our time testing if there are other quality related tasks we need to do.  

I see this aligned with the Modern Testing approach, that is, asking testers to look outside our traditional tasks in order to help our development organizations evolve into more efficient teams where everyone does testing as a part of their job, and where the process is aligned in order to deliver quality products faster to our users.

About the Interviewees

Lalit Bhamare is chief editor & co-founder of Tea-time with Testers magazine @Ttimewidtesters. He is the director at Association for Software Testing @AST_News, international keynote speaker and testing/quality coach. Learn more about Bhamare on his Tales of Testing blog.

Joel Montvelisky is co-founder and chief solution architect at PractiTest. A testing practitioner with over 20 years of experience, he is the chair of the OnlineTestConf and involved in a number of QA and testing-related projects worldwide.



Rate this Article