On the Detection of Test Smells: A Metrics-based Approach for General Fixture and Eager Test

As a fine-grained defect detection technique, unit-testing introduces a strong dependency on the structure of the code. Accordingly, test co-evolution forms an additional burden to the software developer, which can be tempered by writing tests in a manner that makes them easier to change. Fortunately, we are able to concretely express what a good test is by exploiting the specific principles underlying unit testing. Analogous to the concept of code smells, violations of these principles are termed test smells. In this paper, we clarify the structural deficiencies encapsulated in test smells by formalizing core test concepts and their characteristics. To support the detection of two such test smells, General Fixture and Eager Test, we propose a set of metrics defined in terms of unit test concepts. We compare their detection effectiveness using manual inspection and through a comparison with human reviewing. While the latter is the traditional means for test quality assurance, our results indicate it not to be a reliable means for test smell detection. This work thus stresses the need for a more reliable detection mechanism, and provides an initial contribution through the validation of test smell metrics.