Crowdsourced Testing, Changing the Game

Posted by Vikas Hazrati on Aug 03, 2010

Sections

Process & Practices

Topics

Agile in the Enterprise ,

Agile ,

Agile Techniques

Tags

Testing

Crowdsourcing is the process of requesting a large group of community, a crowd, to perform a task which is traditionally done by a select set of people in an organization, most likely employees or contractors. Crowdsourced testing is the powerful combination of combining web and cloud economics with the effectiveness and efficiency of crowdsourcing. Could this be a game changer?

Israel Gat mentioned that the software testing process can be split into two parts

• Unit testing by the development team
• All other forms of testing including functional, load, regression, usability etc.

According to Israel, the latter process is where the game is changing and there are specialized software testing companies who are making efficient use of the web and the crowd. He mentioned that by the definition of testing, crowdsourced testing suits itself very well to a process like Kanban.

By definition, testing as a service involves handing over tasks from one party to another. No matter how closely a development team works with the party that carries out the testing, it is a stage-to-stage flow. Such flow lends itself to Kanban techniques in a natural manner.

Bob Walsh suggested how the crowdsourced testing could be a win-win for the organizations. According to him,

While members of a crowd devoted to quality assurance will all share a love of testing, they’re otherwise all pretty unique. For you, that’s a win! For example, it could be the tester in Hong Kong running Windows Server 2003 who discovers that your application crashes when it tries to read files containing unicode written Cantonese characters. Or it could be the tester in Brazil running Red Hat Enterprise Linux 3 who discovers that your application relies on features in glibc only found in version 4 or later.

Likewise, Yvette Francino mentioned the reason for a crowdsourced testing service to exist. According to Yvette,

It’s virtually impossible to test the vast number of devices and various configurations of software that Web-based software can run on today. Add to this the differences that may occur if the software is meant to be run anywhere and you have a major obstacle to traditional test methods. How can the code be tested effectively in every geographical region? The best alternative would be for people that are native to the country, people that are most like the end-users, to test the software.

Stanton Champion outlined a number of benefits of crowdsourced testing. These included,

• Access to diverse platforms, languages, and people
• Real insights from the real world, not just made up test case results
• Testing done by hundreds of people at the same time
• Rapid feedback right away

On similar lines, Fred Beringer mentioned that he is a big fan of crowdsourcing and the crowdsourced testing helps well to tackle, 

• The need an for a larger and flexible heterogeneous hardware environment primarily used for compliance and performance testing.
• The need to ensure adequate and flexible testing capacity to be able to cope with aggressive release timeframe.

Thus, crowdsourced testing seems to be an interesting concept which would help organizations leverage the heterogeneous power of the crowd. As Israel put it,

If crowdsourced testing indeed gets the traction I believe it would, it will accelerate the deconstruction and subsequent reformulation of the product delivery process 

1. Back to top

   We already have a term for this. It is called a Public Beta.

   by Jonathan Allen

   We already have a term for this. It is called a Public Beta.

   Reply

2. Back to top

   Re: We already have a term for this. It is called a Public Beta.

   by Vikas Hazrati

   True. For a public facing product based on a free(mium) model it could be valid case. But what about SaaS applications which are available to a subscribers. Would they like to do a public beta or rather crowdsource?

   Reply

3. Back to top

   Crowd sourced testing site

   by Avinasha S

   Top crowd sourced testing site: utest (Project model) and a relatively new one in contest model 99tests.com

   Reply

• Older >