Mutation Testing: A New Approach to Automatic Error-Detection

Table of Contents

• Introduction
• What is Mutation Testing?
• Classical Approaches To Mutation Testing
• How Insure++ Revolutionizes Mutation Testing to Find More Bugs Automatically
• Benefits of Mutation Testing

Introduction

Mutation Testing is a powerful method for finding errors in software programs. Until now, however, no tool has been able to bring this power to the everyday programmer. This changes with Insure++, the first tool to bring Mutation Testing's unique methodology to the error-detection field.

This paper will describe the process of Mutation Testing and how Insure++'s revolutionary approach benefits the entire software industry.

What is Mutation Testing?

Mutation Testing has been well-known to computer scientists for decades. It is the process of "re-writing" source code (hence the name Mutation) in order to flush out ambiguities that may exist in the code. These ambiguities can cause failures in software if not detected and fixed by the developer. Because these faults are often very subtle, they can easily pass conventional testing and debugging only to be passed on to the customer.

Although powerful, Mutation Testing is complicated and time-consuming to perform without an automated tool. Lacking a tool to bring its benefits to the commercial market, Mutation Testing has not spread far outside the research community. The longtime goal of researchers has been a testing system that is highly effective, efficient, and convenient to use.(1) This breakthrough has been achieved with the release of Insure++ v4.0, which incorporates Mutation Testing automatically into its error-detection technology. Now software developers and testers everywhere can enjoy the most powerful error-detection capabilities with the speed and ease-of-use of an automatic runtime debugging tool.

Classical Approaches To Mutation Testing

Mutation Testing originated as a means of creating more effective test suites. By creating more effective test suites, the tester can be more confident that his program is adequately tested.

The classical approach to Mutation Testing begins by creating many versions of a program. Each of these versions is "mutated" to introduce a single fault. These "mutant" programs are then run against test cases with the goal of causing each faulty version to fail. Each time a test case causes a faulty version to fail, that mutant is considered "killed" and the test case is saved. When all mutants have been killed, the saved test cases will comprise the test suite used to test the original program.

This approach to Mutation Testing is useful for creating test suites, but is little help for finding bugs in the original program. Insure++ takes a new approach to Mutation Testing, shifting the focus from the test suite to the program itself. This change enables Insure++ to find bugs that exist in the original source code.

How Insure++ Revolutionizes Mutation Testing to Find More Bugs Automatically

Insure++ takes a different approach to Mutation Testing, creating "equivalent" (rather than "faulty") mutants. This technique makes for more effective bug-checking, as any ambiguities that may exist in the original source code are exposed during mutation. Insure++'s patented Source Code Instrumentation (SCI) technology (patent #5,581,696) then provides in-depth debugging information that makes it fast and easy for the developer to find and fix bugs.

During the checking process, Insure++ parses, analyzes, and converts a developer's original source code into a new, functionally equivalent source code. This equivalent code is stored in a temporary file that is passed to the compiler. (It is important to note that Insure++ does not modify the original source and no user intervention is required. The process is completely automatic and invisible to the user.)

Once all of the developer's files have been "mutated" into equivalent code, they are linked into a final executable, ready for run-time error detection.

During the error-detection process, the "functionally equivalent" mutants are run in place of the original source code. Unlike faulty mutants, these "equivalent mutants" are expected to "pass" instead of fail. If the original program is correct, then the mutant can not be killed. If a mutant is killed, it means there is an error in the original program that must be fixed. Through this method, Insure++ is able to uncover ambiguities that are undetectable by any other method or tool.

Whenever a bug is found, Insure++ automatically reports the fault and the line responsible. The inclusion of debugging information is a major difference between classical Mutation Testing and the Mutation Testing performed by Insure++. This improvement is made possible through a process called code insertion in which Insure++ inserts a line of code before and after each line of mutated code. This extra code enables Insure++ to accurately check for and report errors found during program execution.

An additional benefit to Insure++'s approach to Mutation Testing is that users can create as many (or as few) mutants as they wish. Using Insure++'s flexible suppression mechanism, users can create equivalent mutants for any code they wish, from a single function to an entire project. Of course, the more mutants created, the more thorough the error-detection.

This new approach to Mutation Testing lets Insure++ meet the goals of effectiveness, efficiency, and convenience aspired to by researchers. More debugging information is provided to the developer, improving effectiveness, less time and computing is required, increasing efficiency, and for ultimate convenience, the entire process is made completely automatic.

Benefits of Mutation Testing

Mutation Testing brings a whole new level of error-detection to the software developer. This powerful method is able to uncover ambiguities in code previously thought impossible to detect automatically. By incorporating Mutation Testing into its state-of-the-art error-detection technology, Insure++ is able to flush out more faults than any other tool.

Software developers and testers using Insure++ will benefit enormously from Mutation Testing, as more bugs will be reported automatically with Insure++ than with any other tool. Errors that were once frustrating and time-consuming to find and fix can now be found automatically, and Insure++'s accurate, detailed error reports make debugging and maintenance fast and easy.

The customer also benefits from Mutation Testing, as the program he receives is less buggy and more reliable. This increased confidence will in turn benefit your company where it matters most-- the bottom line.

(1) Offutt, A.J. A Practical System for Mutation Testing: Help for the Common Programmer.