Lack of test resources and the changing nature of software landscape has brought about a shift from traditional testing methods to a more dynamic way of testing called Crowdtesting. To understand more about how crowdtesting can impact software quality, InformationWeek spoke to Mayank Mittal, Director and Country Head, Pass Technologies, India, whose firm specializes in crowdsourcing testing. In India, the company is growing rapidly, and has set up two delivery centers in Mumbai and Bangalore within a year of starting operations. India’s growing prominence in the crowdsourcing testing community can also be seen from the fact that around 10-15 percent of the company’s crowdsourced testing community hail from India. Mayank shares his perspective on why he believes that crowdsourcing offers tremendous advantages over traditional testing models. Some edited excerpts follow: When compared to a traditional testing approach, what advantages does crowd sourced testing offer? In broad terms, the traditional testing is based on a set of formal methods and processes that govern software quality assurance. The software development company usually performs these activities in-house using their internal workforce. This method is limited in effectiveness because it is technically and economically challenging to cover an array of all possible scenarios in which a particular software application is eventually used. The growing number of mobile, web and browser platforms, device and system version and configurations under which the application should be tested and supported create an overhead and pose a quality assurance nightmare that software development companies should reconcile with the traditional approach. Even if it is feasible to perform these activities in-house, crowdtesting is much faster and cheaper than the formal approach. crowdtesting’s key differentiator is that it allows applications to be tested under real-world conditions – a case almost impossible to emulate with an in-house staff using a traditional testing approach. In addition to such benefits as enhanced quality, reduced costs and faster time-to-release, companies seeking to crowdtest their applications also benefit from fresh perspectives the tester community brings to the equation. A crowdtest team can consist of professional testers, novice testers, real application users and subject matter experts, testing the application simultaneously. In-house testers may develop biases as they grow familiar with the application(s) they regularly test; Crowdtesting is competitive and dynamic in nature, so by design, such biases are inherently factored out, which results in superior test outcomes compared to traditional testing. In terms of factors like time, cost and quality, how does crowd sourced testing score over traditional testing? It would help if you could provide measurable statistics? Crowdtesting is highly effective over traditional testing, achieving high economies of scale. In terms of cost, traditional testing is based on fixed price i.e. testers’ salaries and other overheads are fixed and predictable. In other words, the software development company incurs this overhead regardless of the testing outcome, coverage or quality of testing. With crowdtesting, these fixed costs are translated into variable costs. The software development company could decide on the number of testers, their skillsets, experience levels, etc., so crowdtesting is both flexible and controllable. Compensation for the testers is based on the quality and uniqueness of defects they report during the course of their testing activity. As a result, the quality and depth of testing is greatly improved at a fraction of the costs incurred through traditional testing. In a recent client project, we benchmarked the effectiveness of crowdtesting against traditional testing using two metrics – time and number of defects using a standard system configuration. In the first experiment, we measured the defect discovery pattern in terms of the number of material issues identified over a fixed time window (constant). We observed that a large number of high-impact bugs were discovered during the early on during the test cycle. Over a week testing period, our testers identified a total of 92 defects compared to 59 defects that our client identified using their in-house personnel. These figures imply that crowdtesting is almost two times more effective than traditional testing over a fixed time window. In a follow-up experiment, we fixed the number of defects and measured the time it took our community to identify all defects in this set. We observed that our community discovered all the issues within less over a week compared to over 2 weeks that our testing colleagues at our client’s company took. These figures strongly imply that crowdtesting is almost two times as fast as traditional in-house testing for a fixed number of defects. In actual crowdtesting projects, our testers subject the application under diverse system configurations and usage patterns. As a result, our clients experience performance gains in the order of several magnitudes in addition to speed and test coverage because testing is simultaneously performed on several platforms. These observations are hard evidence of our position on the effectiveness and efficiency of Crowdtesting over traditional testing. Can you give us a brief of your operations in India? Our crowdtesting approach is based on an Integrated Delivery Model with a combination of internal teams and professional freelance testers. In this model, our in-house team, team comprising mainly of project managers, test managers, domain experts and technical consultants, handles consulting, project management and reviewing of all deliverables submitted by the passbrains community. The passbrains community has a large mix of testers, ranging from novice testers to seasoned professionals with 20+ years of experience. We staff our customers’ projects with testers from the passbrains community based on a comprehensive selection process covering various metrics, such as testers’ skills, testing and specific domain experience, demographics, performance and many more factors to ensure optimal test coverage, quality and overall performance. In many projects, we blend professional and non-professional testers (users) in a test project to include consumers’ user experience reports with professional testing methods and to increase the number of permutations of use cases, scenarios and perspectives. We started our operations in India in March 2013 and within a short span of time, we have rapidly scaled our operations and successfully set up two delivery centers in Mumbai and Bangalore. In addition to our regional growth in India, our global crowdtesting platform hosts a diverse, international professional software testing community that is spread across 100+ countries and is continuously growing with more than 25,000+ members. Around 10-15 percent of the testing professionals in the community hail from India. With over two decades of testing experience under our belt, we offer a broad range of services that are attractive to both, large global enterprises and smaller firms or startups alike. Our extensive domain experience in such industries as Telecom, Healthcare, Financial Services, Media and Retail is particularly attractive to large enterprise customers that seek specific domain knowledge in addition to high quality testing. Simultaneously, our model is very attractive to Small and Medium scale enterprises that expect high quality professional testing services at competitive price points. In our hybrid this model, we use a mix of on-shore, offshore and crowdtesting for specific projects that we identify could immensely benefit from this setup. To identify which projects or applications are viable candidates for this constellation, we also offer our free CrowdConsult services to our customers. Presently, we are working with two types of customer segments: Firstly, Mobile and Web application development companies across various sectors and other software quality assurance and testing service providers. We are also partnering with many large IT enterprises, and enabling them to add crowdtesting in their portfolio.