Mutation Testing

Nowadays, systems are becoming more and more complex and development teams are under stress to complete the projects on time. Due to which, multiple developers work on the same project and due to system complexities source code also increases. Software testing becomes an important step in any software development life cycle as the code needs to be fault-free.

There are various white box techniques to test the quality of the source code, one of them is mutation testing.

Content

What is Mutation Testing?

Mutation testing is a structural, fault-based white box testing technique used to assess the fitness of the test suites which helps in finding any faults in the system. A mutation is a small change in the program. These changes are small so that they do not affect the high-level purpose of the system and basic low-level bugs can be found in the code.

Here we make small changes i.e. faults in the source code and check the test suits to see whether they can catch the faults or not. These faulty set of programs are called mutants. To check the test suite, faulty mutants are executed against the test suite. If the result is different from the original program, then we remove or kill the mutant. If the result is the same, then the mutant is kept alive.

Let’s try to get a detailed understanding of the steps to execute performed in mutation testing-

First, prepare test cases for the selected program where mutation testing needs to be applied. Run test cases against this program.
Create mutants by making changes in the program which will result in faults. Run a test case against this program.
Compare results of both the test runs i.e. test run of original and mutant programs.
If the results are different, we can assume that the test case was able to identify the fault in the program, this mutant will then be killed by the test case.
But if the results are the same, we can assume that the test case was unable to identify the fault in the program and the mutant will have to be kept alive.
Repeat steps 2 to 5 for the remaining test cases of the test suite.

Mutant testing works on the two hypotheses as discussed below-

1. Competent Programmer Hypothesis (CPH)

This hypothesis implies that the programmer is competent which means s/he will code programs close to perfection. And even if the faults are found, it is assumed that it is because of the minor syntactic errors which can be solved easily.

2. Coupling Effect

This hypothesis is based on the faults used in mutation testing. It states that the simple mutant with a simple fault is sensitive enough to couple to form other complex faults i.e. a test suite that can detect simple faults in the system will also detect a high percentage of complex faults.

Mutation Score

A mutation score is a ratio of the killed mutants to the total number of mutants. It is used to measure the quality of the test suite quantitatively; the ideal mutation score should be 100% or 1 i.e. all the mutants are killed by the test suites.

Mutation Score = (No. of killed mutants/total no. of mutants)

In %,

Mutation Score = (No. of killed mutants/total no. of mutants) * 100%

Types of Mutations

There are mainly three types of mutations.

1. Value Mutations – These types of mutations change values of constants or parameters. Example-

The original program is as follows-

Int x = 5, y = 10, total;
print ("Total = x^y");

Mutant program is as follows-

Int x = 55897, y = 10, total;
print ("Total = x^y");

2. Statement Mutations – These types of mutations change statements of the code either by removing them, replacing with other statements or changing the order of the statements.

The original program is as follows-

Int x, y, total;
If (x>y)
total = x+y;
else
total = x-y;

Mutant program is as follows-

Int x, y, total;
If (x>y)
total = x+y;
else
total = x*y;

3. Decision Mutations – These types of mutations change logical or arithmetic operators in the program. Example-

The original program is as follows-

Int x, y, total;
If (x>y)
total = x+y;
else
total = x-y;

Mutant program is as follows-

Int x, y, total;
If (x

Advantages of Mutation Testing

Let’s look at some of the advantages of mutation testing:

Mutation testing helps find weak programs in the code that are responsible for bugs.
It is also helpful in finding weak test cases (test cases that are not able to kill mutants).
It helps the development team to check the quality of the source code.
Mutation testing will help in increasing customer satisfaction as defects are found early in the development cycle and also at the code level.
It helps in gaining complete code coverage of the source code.

Disadvantages of Mutation Testing

The following are some disadvantages of mutation testing-

It is a time-consuming process as it involves creating a large number of mutants and executing them against test suites.
Manual mutation testing requires a large number of resources to create mutants and executing all of them; it becomes a laborious process.
It is not applicable for black-box testing, as it requires changes in the code in order to find the faults.
Resources should have a detailed knowledge of the automation tool(s) used for mutation testing.