Posts Tagged ‘Testing’

DARPA gamifies open-source software testing

June 10th, 2014

Secret-squirrel military tech bureau DARPA has designed a series of computer games which can help to verify open source software.

It is working on the games under the auspices of its Crowd Sourced Formal Verification programme.

The idea is to perform the soft of software verification which is generally conducted by technical experts.

“There are not enough human experts or available time to demonstrate that software is secure and reliable – so what we’ve done is repackage what human experts would normally do and produce tens of thousands of game levels for players on the internet to play games for us,” Matthew Barry, who is principal investigator at Kestrel Technology, the firm working with DARPA to develop the games, told

One of the games is called Circuit Bot and involves running missions to asteroids.

“Along the way they are assembling teams of robots that have different assignments. If you assemble the robot team in the correct order you get a certain amount of points. The results of those assemblies contribute toward the verification of open source software,” Barry continued. “We’re enhancing what a human expert has to do by using the internet crowd to play games.”

Five games are available to be played on the website Verigames and some of the software verified by gamers may even end up being used by DARPA itself and other government organisations.


Google testing software to take email encryption to the masses

June 5th, 2014

Google is testing a new browser extension that will be able to encrypt Gmail messages sent to and from Google Chrome, making it harder for someone to read them.

While email encryption software isn’t new, and Google already offers an encrypted connection for Gmail (shown as https on the address bar), the new service would encrypt the message content.

Google said it hoped the plug-in would make the process of encryption more accessible and therefore more widely used. Encryption software tools like PGP and GnuPG are freely available but are cumbersome for consumers.

Google’s plug-in, is called End-to-End, promising uninterrupted protection of data travelling between two parties.

According to a recent Google Transparency report, 40 to 50 per cent of emails sent from within their hosted accounts aren’t encrypted.

”We recognise that this sort of encryption will probably only be used for very sensitive messages or by those who need added protection. But we hope that the End-to-End extension will make it quicker and easier for people to get that extra layer of security should they need it,” the statement said.

The company has released the source code to the tech community to check for bugs and get feedback before a public launch.

The move follows Yahoo’s announcement in April it was moving towards a platform where all emails were encrypted by default.

Google’s source code release is part of a day of action scheduled for Friday called Reset the Net. That event aims at motivating internet users to “take privacy back” in light of mass surveillance operations by the US National Security Agency (NSA).

It will involve a range pro-privacy activities coordinated by the Electronic Frontier Foundation with Google, Reddit, Mozilla, Amnesty International and NSA whistleblower Edward Snowden.

“One year ago, we learnt that the internet is under surveillance, and our activities are being monitored to create permanent records of our private lives – no matter how innocent or ordinary those lives might be,” Mr Snowden said in a statement issued by his lawyer.

”Today, we can begin the work of effectively shutting down the collection of our online communications, even if the US Congress fails to do the same.”


Can crowdsourcing testing change the way software testing is done?

May 13th, 2014

Lack of test resources and the changing nature of software landscape has brought about a shift from traditional testing methods to a more dynamic way of testing called Crowdtesting. To understand more about how crowdtesting can impact software quality, InformationWeek spoke to Mayank Mittal, Director and Country Head, Pass Technologies, India, whose firm specializes in crowdsourcing testing. In India, the company is growing rapidly, and has set up two delivery centers in Mumbai and Bangalore within a year of starting operations. India’s growing prominence in the crowdsourcing testing community can also be seen from the fact that around 10-15 percent of the company’s crowdsourced testing community hail from India. Mayank shares his perspective on why he believes that crowdsourcing offers tremendous advantages over traditional testing models. Some edited excerpts follow: When compared to a traditional testing approach, what advantages does crowd sourced testing offer? In broad terms, the traditional testing is based on a set of formal methods and processes that govern software quality assurance. The software development company usually performs these activities in-house using their internal workforce. This method is limited in effectiveness because it is technically and economically challenging to cover an array of all possible scenarios in which a particular software application is eventually used. The growing number of mobile, web and browser platforms, device and system version and configurations under which the application should be tested and supported create an overhead and pose a quality assurance nightmare that software development companies should reconcile with the traditional approach. Even if it is feasible to perform these activities in-house, crowdtesting is much faster and cheaper than the formal approach. crowdtesting’s key differentiator is that it allows applications to be tested under real-world conditions – a case almost impossible to emulate with an in-house staff using a traditional testing approach. In addition to such benefits as enhanced quality, reduced costs and faster time-to-release, companies seeking to crowdtest their applications also benefit from fresh perspectives the tester community brings to the equation. A crowdtest team can consist of professional testers, novice testers, real application users and subject matter experts, testing the application simultaneously. In-house testers may develop biases as they grow familiar with the application(s) they regularly test; Crowdtesting is competitive and dynamic in nature, so by design, such biases are inherently factored out, which results in superior test outcomes compared to traditional testing. In terms of factors like time, cost and quality, how does crowd sourced testing score over traditional testing? It would help if you could provide measurable statistics? Crowdtesting is highly effective over traditional testing, achieving high economies of scale. In terms of cost, traditional testing is based on fixed price i.e. testers’ salaries and other overheads are fixed and predictable. In other words, the software development company incurs this overhead regardless of the testing outcome, coverage or quality of testing. With crowdtesting, these fixed costs are translated into variable costs. The software development company could decide on the number of testers, their skillsets, experience levels, etc., so crowdtesting is both flexible and controllable. Compensation for the testers is based on the quality and uniqueness of defects they report during the course of their testing activity. As a result, the quality and depth of testing is greatly improved at a fraction of the costs incurred through traditional testing. In a recent client project, we benchmarked the effectiveness of crowdtesting against traditional testing using two metrics – time and number of defects using a standard system configuration. In the first experiment, we measured the defect discovery pattern in terms of the number of material issues identified over a fixed time window (constant). We observed that a large number of high-impact bugs were discovered during the early on during the test cycle. Over a week testing period, our testers identified a total of 92 defects compared to 59 defects that our client identified using their in-house personnel. These figures imply that crowdtesting is almost two times more effective than traditional testing over a fixed time window. In a follow-up experiment, we fixed the number of defects and measured the time it took our community to identify all defects in this set. We observed that our community discovered all the issues within less over a week compared to over 2 weeks that our testing colleagues at our client’s company took. These figures strongly imply that crowdtesting is almost two times as fast as traditional in-house testing for a fixed number of defects. In actual crowdtesting projects, our testers subject the application under diverse system configurations and usage patterns. As a result, our clients experience performance gains in the order of several magnitudes in addition to speed and test coverage because testing is simultaneously performed on several platforms. These observations are hard evidence of our position on the effectiveness and efficiency of Crowdtesting over traditional testing. Can you give us a brief of your operations in India? Our crowdtesting approach is based on an Integrated Delivery Model with a combination of internal teams and professional freelance testers. In this model, our in-house team, team comprising mainly of project managers, test managers, domain experts and technical consultants, handles consulting, project management and reviewing of all deliverables submitted by the passbrains community. The passbrains community has a large mix of testers, ranging from novice testers to seasoned professionals with 20+ years of experience. We staff our customers’ projects with testers from the passbrains community based on a comprehensive selection process covering various metrics, such as testers’ skills, testing and specific domain experience, demographics, performance and many more factors to ensure optimal test coverage, quality and overall performance. In many projects, we blend professional and non-professional testers (users) in a test project to include consumers’ user experience reports with professional testing methods and to increase the number of permutations of use cases, scenarios and perspectives. We started our operations in India in March 2013 and within a short span of time, we have rapidly scaled our operations and successfully set up two delivery centers in Mumbai and Bangalore. In addition to our regional growth in India, our global crowdtesting platform hosts a diverse, international professional software testing community that is spread across 100+ countries and is continuously growing with more than 25,000+ members. Around 10-15 percent of the testing professionals in the community hail from India. With over two decades of testing experience under our belt, we offer a broad range of services that are attractive to both, large global enterprises and smaller firms or startups alike. Our extensive domain experience in such industries as Telecom, Healthcare, Financial Services, Media and Retail is particularly attractive to large enterprise customers that seek specific domain knowledge in addition to high quality testing. Simultaneously, our model is very attractive to Small and Medium scale enterprises that expect high quality professional testing services at competitive price points. In our hybrid this model, we use a mix of on-shore, offshore and crowdtesting for specific projects that we identify could immensely benefit from this setup. To identify which projects or applications are viable candidates for this constellation, we also offer our free CrowdConsult services to our customers. Presently, we are working with two types of customer segments: Firstly, Mobile and Web application development companies across various sectors and other software quality assurance and testing service providers. We are also partnering with many large IT enterprises, and enabling them to add crowdtesting in their portfolio.


Get Adobe Flash player