Mutation Testing: A New Approach to Automatic Error-Detection
Table of Contents
Mutation Testing is a powerful method for finding errors in software
programs. Until now, however, no tool has been able to bring this power
to the everyday programmer. This changes with
Insure++, the first tool to bring Mutation Testing's unique
methodology to the error-detection field.
This paper will describe the process of Mutation Testing and how
Insure++'s revolutionary approach benefits the entire software
What is Mutation Testing?
Mutation Testing has been well-known to computer scientists for
decades. It is the process of "re-writing" source code (hence
the name Mutation) in order to flush out ambiguities that may
exist in the code. These ambiguities can cause failures in software if
not detected and fixed by the developer. Because these faults are often
very subtle, they can easily pass conventional testing and debugging only
to be passed on to the customer.
Although powerful, Mutation Testing is complicated and time-consuming
to perform without an automated tool. Lacking a tool to bring its benefits
to the commercial market, Mutation Testing has not spread far outside the
research community. The longtime goal of researchers has been a testing
system that is highly effective, efficient, and convenient to use.(1) This
breakthrough has been achieved with the release of Insure++ v4.0,
which incorporates Mutation Testing automatically into its error-detection
technology. Now software developers and testers everywhere can enjoy the
most powerful error-detection capabilities with the speed and ease-of-use
of an automatic runtime debugging tool.
Classical Approaches To Mutation Testing
Mutation Testing originated as a means of creating more effective test
suites. By creating more effective test suites, the tester can be more
confident that his program is adequately tested.
The classical approach to Mutation Testing begins by creating many
versions of a program. Each of these versions is "mutated" to
introduce a single fault. These "mutant" programs are then run
against test cases with the goal of causing each faulty version to fail.
Each time a test case causes a faulty version to fail, that mutant is
considered "killed" and the test case is saved. When all
mutants have been killed, the saved test cases will comprise the test
suite used to test the original program.
This approach to Mutation Testing is useful for creating test suites,
but is little help for finding bugs in the original program.
Insure++ takes a new approach to Mutation Testing, shifting the
focus from the test suite to the program itself. This change enables
Insure++ to find bugs that exist in the original source code.
How Insure++ Revolutionizes Mutation Testing to Find More Bugs
Insure++ takes a different approach to Mutation Testing,
creating "equivalent" (rather than "faulty") mutants.
This technique makes for more effective bug-checking, as any ambiguities
that may exist in the original source code are exposed during mutation.
Insure++'s patented Source Code Instrumentation (SCI)
technology (patent #5,581,696) then provides in-depth debugging
information that makes it fast and easy for the developer to find and
During the checking process, Insure++ parses, analyzes, and
converts a developer's original source code into a new, functionally
equivalent source code. This equivalent code is stored in a temporary
file that is passed to the compiler. (It is important to note that
Insure++ does not modify the original source and no user
intervention is required. The process is completely automatic and
invisible to the user.)
Once all of the developer's files have been "mutated" into
equivalent code, they are linked into a final executable, ready for
run-time error detection.
During the error-detection process, the "functionally
equivalent" mutants are run in place of the original source code.
Unlike faulty mutants, these "equivalent mutants" are expected
to "pass" instead of fail. If the original program is correct,
then the mutant can not be killed. If a mutant is killed, it means there
is an error in the original program that must be fixed. Through this
method, Insure++ is able to uncover ambiguities that are
undetectable by any other method or tool.
Whenever a bug is found, Insure++ automatically reports the
fault and the line responsible. The inclusion of debugging information
is a major difference between classical Mutation Testing and the
Mutation Testing performed by Insure++. This improvement is made
possible through a process called code insertion in which
Insure++ inserts a line of code before and after each line of
mutated code. This extra code enables Insure++ to accurately
check for and report errors found during program execution.
An additional benefit to Insure++'s approach to Mutation
Testing is that users can create as many (or as few) mutants as they wish.
Using Insure++'s flexible suppression mechanism, users can create
equivalent mutants for any code they wish, from a single function to an
entire project. Of course, the more mutants created, the more thorough
This new approach to Mutation Testing lets Insure++ meet the
goals of effectiveness, efficiency, and convenience aspired to by
researchers. More debugging information is provided to the developer,
improving effectiveness, less time and computing is required, increasing
efficiency, and for ultimate convenience, the entire process is made
Benefits of Mutation Testing
Mutation Testing brings a whole new level of error-detection to the
software developer. This powerful method is able to uncover ambiguities
in code previously thought impossible to detect automatically. By
incorporating Mutation Testing into its state-of-the-art error-detection
technology, Insure++ is able to flush out more faults than any
Software developers and testers using Insure++ will benefit
enormously from Mutation Testing, as more bugs will be reported
automatically with Insure++ than with any other tool. Errors
that were once frustrating and time-consuming to find and fix can now
be found automatically, and Insure++'s accurate, detailed error
reports make debugging and maintenance fast and easy.
The customer also benefits from Mutation Testing, as the program he
receives is less buggy and more reliable. This increased confidence
will in turn benefit your company where it matters most-- the bottom
(1) Offutt, A.J. A Practical System for Mutation Testing: Help for
the Common Programmer.