Summary
Mutation testing is a technique for systematically mutating source code in order to validate test suites. It works by making small changes to a program’s source code and then running a test suite; if the test suite passes on mutated code then a flag is raised. The goal is to check that a system’s test suite is sufficiently powerful to detect a large class of functionality-affecting changes, thereby helping ensure that the system functions as expected. Mutation testing is a fascinating topic with great potential that has valuable lessons for the broader software development community. I’ll begin this talk with a description of the theory behind mutation testing. We’ll look at how it works and the benefits it can provide. We’ll also consider some of the practical difficulties associated with the technique, including long runtimes and certain difficult classes of mutants. I’ll then move into an analysis of Cosmic Ray, a tool for mutation testing in Python. I'll demonstrate using Cosmic Ray to locate untested code not detected by traditional coverage techniques in an open source library – with surprising results.
Description
Mutation testing is a technique for systematically mutating source code in order to validate test suites. It works by making small changes to a program’s source code and then running a test suite; if the test suite passes on mutated code then a flag is raised. The goal is to check that a system’s test suite is sufficiently powerful to detect a large class of functionality-affecting changes, thereby helping ensure that the system functions as expected. Mutation testing is a fascinating topic with great potential that has valuable lessons for the broader software development community. I’ll begin this talk with a description of the theory behind mutation testing. We’ll look at how it works and the benefits it can provide. We’ll also consider some of the practical difficulties associated with the technique, including long runtimes and certain difficult classes of mutants. I’ll then move into an analysis of Cosmic Ray, a tool for mutation testing in Python. I'll demonstrate using Cosmic Ray to locate untested code not detected by traditional coverage techniques in an open source library – with surprising results.