Investigating the Utilizability of Memoization for Mutation Testing

Abstract

Software developers often write tests to verify that their programs behave as expected. However, it is difficult to assess the completeness and robustness of their test suite. Mutation testing is a powerful technique that is used to analyse and improve the quality of a test suite. It works by introducing small faults, called mutants, into a program and checking whether the test suite can detect these. The relative number of mutants caught is used to calculate a mutation score that indicates the quality of the test suite. Developers can use mutation analysis to examine the test results and identify which parts of the test suite can be improved. A major drawback of mutation testing, however, is the substantial time required to perform it. Although many advances have been made on the computational cost reduction of mutation testing, its efficiency remains suboptimal. This thesis investigates to which extent memoization can be utilized as a computation cost-reduction strategy for mutation testing in languages with stateful values. For this purpose, we have implemented memoization for a mutation testing tool named Stryker.NET. Our approach uses a shared memory space to maintain and exchange memoized state across multiple ndependent processes. We performed an empirical experiment on 15 real-world projects. We found that memoization using serialization can be utilized on roughly 4-22% of methods and expressions in a program. Our results indicate a varying degree of effectiveness across different projects when observing the ratio of memoization cache hits.