Testing

Testing is an essential part of developing software independent of which approach is used. Within XP testing plays a vital role, as you can also look at XP as validation centric software development. All artefacts, which are created, are tested.

XP distinguishes between two categories of tests: acceptance tests and unit tests. The former is specified by the on-site customer, thus ensuring that the user stories are correctly implemented. Unit tests on the other hand are created during pair programming. They test individual parts of the system from a programmer's perspective.

In all cases the approach is always that the tests are written first, then the code. Please notice that the tests should not make any assumptions about the implementation. In other words: It is specified how the system will be used (which functions are called) and what the expected result is. How the system gets to the results, is of no interest from the test's perspective.

Validation Centric Software Development

Apart from these obvious tests (acceptance tests, unit tests) there are other activities within XP, where validation is an integral part.

Pair Programming contains an ongoing validation element. There is always a second person who does a permanent code review.

As prototypes are made available to the customer short term, he can immediately check, whether what was implemented matches what he wanted to have. This comparison does not happen only at the end of the release cycle [link?], but will be done several times during the release cycle, thus providing the programmers with immediate feedback. At the same time, this approach gives the customer the opportunity to directly influence the outcome.

What Will Be Tested?

You test everything that can fail. In other words: If no defects are detected after the system has been released, you have had enough tests. If there are defects, the set of tests is not sufficient; at least the set does not contain the necessary ones. If this happens, a test that reproduces a known defect has to be written. Then the system has to be modified until all tests (including the new one) run successfully.

As there is (almost) no system, that does no required 3rd party products such as libraries, we have also to answer the question, whether these 3rd products should be tested. An example:

A help desk system for customer service uses a database system. Should we write unit tests for the database system?

To answer the question we have a look at how the manufacturing industry has solved this problem. In the last two decades the verification of raw materials has been successively eliminated; instead quality assurance management processes at the suppliers were introduced an enhanced. Now, it became the task of the supplier to proove that the goods he delivered match the required specifications. Under normal circumstances the purchaser (the customer) can be sure that this is the case. This way on each step of the production chain a duplicate quality check has been removed. Looking at the value chain, significant improvements of the cost structure were achieved by avoiding duplicat quality checks.

The same can be applied to software development. We do not check wether a 3rd party product works perfectly under all conditions. This check is not our task. Our responsibility is, however, that we test every single functionality that we deliver to our customers. Running these tests we cannot avoid that 3rd party products are tested at the same time. We do have the advantage, that we test only those parts of the 3rd party products which we actually use.

In the case of the above mentioned help desk system entering a service request unavoidably leads to a transaction for the database system. Should the database system have a problem with regards to transations, we will notice it in most cases. And as we write tests before we code (as opposed to at the end of the release cycle), we detect such defects befor our system has been delivered to the customer. (Note: This approach does not work for systems which a) will be in production for a long time and b) are very expensive purchasing. I do not have a solution for this scenario.)

Tips

• Unit tests and acceptance tests should be refactored as well.
• If you think that testing is your ultimate justification for existence, then you should have a look at the standard books about testing written by Boris Beizer and Robert V. Binder. But note: From an XP perspective they overdo it!
• Testing not only serves finding defects. Its primary goal is avoiding defects.