On effective and efficient mutation analysis for unit and integration testing

Abstract

Software testing is the most common technique to verify that a program meets certain quality standards. Sufficient manual testing is not only time consuming but also an error-prone task. Additionally, due to the rapidly growing size and complexity of software systems, automating the software testing process is desirable for more cost-effective testing. Besides automating the testing process, the quality of the applied testing strategy has to be assessed to achieve reliable results. This is of particular importance to ensure that the employed tests are also effective in terms of their fault-finding capabilities, and hence the results adequately reflect the quality of the software. By focusing on mutation analysis and testing with partial oracles, this thesis addresses the automation and assessment of unit and integrations tests. This thesis describes and evaluates approaches that improve the effectiveness and efficiency of mutation testing. While mutation testing is known to be computationally expensive and time consuming, it also lacks proper tool support for various purposes. Therefore, this thesis also presents a versatile and highly configurable mutation framework that implements the suggested approaches to enable further research as well as the application of mutation analysis for large software systems. The automation of software tests often results in the oracle problem, another crucial challenge in software testing. In an attempt to alleviate this problem, leveraging partial oracles seem to be viable solution but their adequacy for different testing purposes has not been examined sufficiently. Therefore, this thesis investigates whether partial oracles are in principal adequate for unit and integration testing. By employing mutation analysis for this purpose, the thesis also analyzes how such partial oracles can be improved.