AmPyfier (Test Amplification for Python)

Test Amplification

Software test amplification is the act of strengthening manually written test-cases to exercise the boundary conditions of the system under test. It has been demonstrated extensively by the research community to work for the programming language Java, relying on the type system to safely transform the code under test.

The AnSyMo research group, has created the AmPyfier tool for the Python language (a programming languages without a type system ).

Example

Consider the following unit test which excercises a class under test SmallBank, representing a bank with operations deposit and withdraw. This test is just a happy day scenario where line 02 deposits 10 eur (and then asserts that the balance is indeed 10 in line 3) and then depossits twice 100 eur (lines 04 and 05) to arrive at a final balance of 210 (line 6)

01     def testDeposit(self):
02         self.b.deposit(10)
03         self.assertEqual(self.b.get_balance(), 10)
04         self.b.deposit(100)
05         self.b.deposit(100)
06         self.assertEqual(self.b.get_balance(), 210)

AmPyfier takes this test and amplifies it by adding extra assertions (lines 09 - 12) that verifies all observable output on the object under test. It modifies the deposit call in line 04 with a very large negative integer which is expected to raise an exception (lines 13-14). The rest of the test proceeds as before, however now with extra assertions (lines 16-17)

07     def testDeposit_amplified(self):
08         self.b.deposit(10)
09         self.assertEqual(self.b.get_transactions(), [10])
10         self.assertFalse(self.b.is_empty())
11         self.assertEqual(self.b.owner, 'Iwena Kroka')
12         self.assertEqual(self.b.get_balance(), 10)
13         with self.assertRaises(Exception):
14             self.b.deposit(-56313)
15         self.b.deposit(100)
16         self.assertEqual(self.b.get_transactions(), [10, 100])
17         self.assertFalse(self.b.is_empty())
18         self.assertEqual(self.b.owner, 'Iwena Kroka')
19         self.assertEqual(self.b.get_balance(), 110)

Validation

We evaluated Ampyfier on 11 open-source projects in the Python ecosystem containing 51 test classes. We show an average improvement of 9,19% for the mutation score in 30 test classes (~59%) and an average improvement of 3,08% for the coverage score in 27 test classes (~53%). In total 35 out of the 51 test classes (~69) were improved in regards to one or both of those acceptance criteria.

This illustrates that in AmPyfier really is capable of strengtening existing test suites.