What is CrowdTesting?
Are you an early adopter? I am. Great products put a great big smile on my face, but nothing gets my goat more than buggy apps. I have spent most of my multi-decade career leading and contributing to product teams, and have learned a thing or two about how to build apps. One common theme in absolutely every project, there has been a need for testing…and there is no shortage of testing types, quite the opposite. There is functional testing, UI testing, smoke testing, regression testing, QA testing, load testing, unit testing, acceptance testing, etc…the list goes on and on. The challenge has been figuring out a way to perform testing in an Agile way; fast, iteratively and cost effectively.
The incumbent methods have been to perform testing internally or outsourcing to a dedicated team. But there is a newish term being thrown around that describes a distributed way of testing apps and websites.
Here is my definition of crowdtesting:
Crowdtesting: Leveraging a global pool of individuals to voluntarily evaluate various aspects of digital products, often in exchange for compensation.
Wow, that is pretty vague. But that is the point, it is an extremely large and versatile discipline. While it is not ideal for every testing scenario, crowdtesting can be used in a huge variety of ways that cover nearly every testing discipline- from functional testing of mobile apps to QA testing of landing pages and acceptance testing for websites. How teams use crowdtesting to expand their existing testing capabilities is incredibly varied.
The key to this versatility is in the primary components or key tenets of crowdtesting. Let’s list all of the core tenets for crowdtesting and then break them down:
THE EIGHT TENETS OF CROWDTESTING
- Flexible
- Cost Efficient
- Fast
- Variable
- Real world
- Objective
- Digital
- Semi-secret
* In theory you could crowdtest physical products but that involves a product logistics layer that I am not diving into. (Sorry IoT, you will have to wait.)
Flexible. Crowdtesting is endlessly flexible. With the right platform, users have access to a massive pool of global participants. These testers often represent a rich and diverse cross section of society which ranges from professional testers with decades of experience to computer science students. Of course, this cross section can also include non-professionals that range from stay-at-home-parents to underemployed folks looking to make a few extra bucks. This is where selecting the right crowdtesting platform is important. They will provide a degree of quality control to ensure accurate and professional results regardless of the underlying crowd.
Cost Efficient. Some folks just want inexpensive labor to test their app cheaply. Fair enough. By going across geographic boundaries, you can often achieve dramatic cost efficiency. This doesn’t have to be offshoring…sometimes it can be locally. For example, we are based in San Francisco. The cost of labor here in the Bay Area is really high. But just a few hours North, South or East the cost of living drops and hired work can be much more affordable.
Fast. Because there are often many testers, the testing is parallelized which means it gets completed much faster than if a small team or individual were testing. This means that if a testing cycle used to take several days or a week, it can often be completed within hours with a finely tuned crowdtesting campaign.
Variable. A crowd by definition includes a variety of individuals. Once again, a quality crowdtesting platform will help you manage or guarantee this diversity. In a testing scenario it is often valued to have different device configurations, OS versions or network conditions. With crowdtesting, you get this all for free while in a lab you need to pay for each variation you want. The flip side of this is when you want to test the same scenario or configuration but your crowdtesting platform should also let you achieve this.
Real-World. There are two camps of thinking here. One is believe that Virtualized Testing Environments is sufficient to emulate real world scenarios. The other camp believes that nothing beats real-world scenarios for really understanding how an app will perform in the wild. Crowdtesting is an effective way to get boots on the ground to test your app in the real world.
Objective. There is an old saying, “never test your own software.” And it is sage advice. The people who write the software have certain expectations about how it should be used, and they test those flows. This creates a type of testing “blindness” where the developers are too close to the project to fully exercise it. Crowdtesting can solve this easily with external resources testing an app or website in ways that the creators never imagined. This is a powerful tool to expand your existing teams testing effectiveness.
Digital. This could open up a debate but for simplicity, I am going to say that crowdtesting applies to digital products. Of course, you could try to use a crowd to test physical products but I would argue that the number of products tested would require less of a crowd. It’s a volume thing, and testing 50 products is probably more of a team than a crowd.
Semi-Secret. Is this a bad thing or a good thing? It depends. If you are Electronic Arts and you are developing a top-secret game that the media absolutely cannot know about, then crowdtesting is not a good option. The general rule is that the crowd does not share much about their working experience- betraying that trust is in conflict with them gaining access to new tests and earning money. But the ideal app to utilize crowdtesting could be leaked to the press and there would not be any damage. So use caution when deciding if it is right for you.
Now that I have outlined the core tenets of crowdtesting, let’s talk about how crowdtesting can be used, and some of its shortcomings.
IDEAL USES FOR CROWDTESTING:
Functional Testing. The crowd is a superior choice for testing the functionality of a digital product because it can quickly test common and uncommon uses. We call this “discovering your blind spots” be an objective crowdtest will exercise your app in ways you never imagined.
UI and UX Testing. Designers and the UX team will create the visual style, layout and designs for your app. Developers will do their best to interpret these designs, but who is checking to make sure those designs are correct? How about how those designs translate across device types and screen sizes? Use the crowd.
Usability Testing. Do you want to get insight into how people use your app or what they think of it? Utilize the crowd to test the usability of your app, identify any challenges or blindspots. You can do this with pre-release apps or for apps that are already live. Its never too late to test and improve your usability.
Executing Test Scripts. While automation tools like Selenium are powerful, there is a cost associated with writing and maintaining these scripts. When a crowd of people execute the scripts, the scripts themselves don’t need to be as complicated or brittle as when a machine is interpreting them.
Load Testing. There are software tools that simulate load, but nothing beats the real deal of having dozens or hundreds or thousands of testers using an app organically from around the world. Coordinating this effort on your own is daunting, but with the right platform it should be easy.
Security Testing. Cyber threats are only increasing and important to make sure your digital products are secure. There are many tools that security experts have at their disposal for analyzing and monitoring their systems. Another powerful tool at their disposal is security crowdtesting. A professional crowd of security testers can offer a last line of analysis at reasonable rates.
Surveys. Want an opinion or feedback? Ask the crowd. There are two components to a good survey. 1) writing artful questions, 2) making sure you target the right audience. Make sure you get both of these right or your results could be very misleading.
Compatibility Testing. A diverse pool of testers who have a wide range of devices, operating systems, versions, configurations, languages and physical locations. This broad spectrum of real-world scenarios allows the crowd to detect compatibility issues in a cost effective manner.
HISTORY ANYONE?
If you are interested in where crowdtesting came from and how it evolved, here is a quick review.
SHORTCOMINGS OF CROWDTESTING:
Crowdtesting is powerful but it is not the right tool for every job. While the tests listed above are a perfect fit for crowdtesting, you should think twice before trying to crowdtesting the following:
Test Types to avoid:
Unit Testing. Your developers write these and they should be checked every build by your internal team.
Backend testing. Would you let a crowd into your house? Probably not. So don’t let the crowd behind your firewall or give them access to sensitive systems. Leave that testing to testers inside your circle of trust.
***
In addition to needing the right test type, there are other challenges that should be considered when crowdtesting. When leveraging a crowd some of the risks are obvious but some might be hidden or not fully understood. Here is a quick breakdown of items to keep in mind.
Confidentiality. Stating the obvious; when you share a project to the crowd you are sharing it with a large group of people outside of your organization. It is very challenging to hold the crowd accountable for their actions, such as sharing information publicly. Bottom line: there are some applications that are good candidates for crowdsourcing and some that are not. If absolute secrecy is a priority, you might not want to use crowdsourcing.
Quality. The crowd is an unknown quality. They might have taken tests, provided their work history and proved who they were on social media but you never really know who they are. I’m going to debunk a myth or two now. The best crowdsourcing platforms in the world know this and implement robust and ever evolving methods to keep the quality of the output of the crowd high. If I were a Mythbuster I would say “busted!” While the crowd has the potential to equate to low quality, any platform worth its fees should make that a mute point.
Domain Expertise. This is one of the biggest challenges. How do you educate strangers about the nuances and internal workings of your app? The simple answer is that test plans go a long way in conveying this information. However, test plans require work and teams will then often say “well, since I already made the test plan, I might as well just do the testing myself.” But this is where crowdtesting can really prove itself. You might not gain efficiencies for the first test, but future tests will really begin to show you a return on your investment of time. Retests, regression tests, etc. can be done quickly and efficiently with crowdtesting, especially once you have setup Test Scripts or a test plan.
Cultural Understanding. A crowd can be anywhere and that includes different countries. Sometimes that offers great cost savings and the ability to run tests 24 hours a day. But in some scenarios, a lack of local cultural knowledge can cause unwanted consequences in a test. This is easily mitigated with the right platform that allows you to select the geography of your testers.
FUN FACT: The price of buggy apps is huge. Some estimates claim that the US loses $50 Billion per year because of issues created by bugs!
In summary, crowdtesting is an exciting new domain. It holds the potential to provide flexible and varied solutions to customers if used properly. The burden from here falls onto the crowdtesting platforms to create an experience and deliver on that promise. A quality platform should provide a powerful mechanism for selecting your crowd, built in quality control for the work products, transparency on pricing.
Matthew Cordasco is the co-founder and CEO of MyCrowd QA. A crowdtesting platform built to solve the problems he and fellow co-founder Kirk Franzen have faced while managing product development teams for the past 20 years. When not ranting about crowdsourcing quality, you can find Matthew sailing on San Francisco Bay.
