3        Professional Testing Approaches

With the description of typical testing in the preceding section, it would seem that nothing else should be necessary.  And if that testing is expensive enough, why should management consider spending even more money on testing? 

The answer is accuracy.  Typical testing allows significant parts of a program to go untested.  Typical testing is like single entry bookkeeping, there is no way to check that it was done as well as you think it was done.  If the business cost of an undetected fault is sufficiently high, then the accuracy of typical testing may be insufficient to cover the business risk. If it does make business sense to extend the accuracy of testing, then there are several techniques that can be used to do so in addition to simply doing more typical testing, and these are discussed below. Conversely, if the business cost of an undetected fault is acceptable, then no further effort should be expended.

3.1        Positive Versus Negative Testing

Typical testing usually covers only positive cases.  In other words, we enter this correct transaction data and we see the same result in the migrated program as in the original program. 

Professional testing will also cover negative cases.  We put in every conceivable variant of an incorrect transaction, and ensure that it is properly rejected, from both the original and migrated versions.  This is particularly evident in testing online programs, where each field should have both no entry and incorrect data submitted for processing.  Where cross-field editing is done, each combination should be permuted across the affected fields.  Note that in executing negative testing, it frequently will be the case that faults in the original programs will be uncovered.

3.2        Coverage Analysis

This form of coverage analysis does not ensure that all lines of code will be executed. It only provides a way to measure the amount of coverage, and allows the knowledgeable application specialist to decide whether or not the coverage is sufficient.  His or her judgment might be incorrect, saying on the one hand that additional coverage is necessary when it is not (resulting in higher than necessary costs) or on the other hand that additional coverage is not necessary when it is (leading to a latent fault not being discovered until production testing when it is much more expensive to correct).  However, it does provide checks and balances on the testing process.

There are other forms of coverage analysis besides this report showing lines of code coverage, which is also known as branch and path coverage.  More sophisticated analyses can determine whether all permutations of branch paths have been taken, known as logical test path coverage analysis.  Other analyses can determine whether arithmetic statements have been executed with a full range of values.  And there are even more obscure situations that can be measured.  The type of coverage and the degree of coverage are suitable topics for a risk management discussion during migration project planning, but typically branch and path coverage will be sufficient for all but a very few cases.

Reviewing coverage analysis reports can be time consuming, and requires the attention of application knowledgeable people whose time is in heavy demand.  These report reviews are difficult to relegate to contract or staff personnel who do not have detailed knowledge of the application.  However, it is possible to optimize this and other aspects of project testing, which we will discuss in the next sections.

3.3        Construct Coverage Analysis

It is possible to say that certain constructs that occur within a given program or within the entire suite of programs should always be covered.  For example, if the migration project is going to be replacing indexed file access or an older database with a relational database management system, then all of the unique accesses of each file from the old data stores should be tested at least once amongst the coverage reports.  Or, programs that are updating the database should test all arithmetic constructs.  Et cetera.

These kind of construct coverage rules can be derived during project planning, and can be applied before submitting a final set of coverage analysis reports to subject matter experts.  These rules can be used to create a list of constructs that must be covered, and an analysis report created showing which mandatory constructs have not been covered at least once.

Note that the migration methodology impinges on testing at this point.  If the migration is being conducted using a fully automated process, then a single instance of each construct may be sufficient to prove the transformation of that construct correct.  If, however, the process is only partially automated, or if it is wholly manual, then there is the opportunity for the same construct to be transformed in different ways in different programs.  In this case, construct coverage analysis may be of limited if any value.

3.4        Complexity Analysis

Some programs require more testing than other programs, but is there a way to determine which ones those are?  Certainly, programs with more complex logic in the expression of their business rules will have more branch paths over which to ensure coverage.  However, it can also be argued that fairly straightforward reporting programs do not need the analytical scrutiny of primary transaction processing program.

It is possible to process program sources and derive various metrics regarding their complexity.  Of the many such metrics, perhaps the best known is McCabe’s Cyclomatic Complexity.[1]  If we derive this or an alternative metric for the entire library of sources under consideration, then we can apply the greatest testing to the most complex programs, and proportionally less testing for the less complex programs.

3.5        Risk Optimized Testing

Risk optimized testing says that the most testing should be applied to the programs where a fault will create the greatest negative impact.  Arguably, a report program need not have much more testing than a correctness test of the output, matching control totals and perhaps checking the sequencing and number of pages.  A program that extracts data for analysis or reporting by other programs probably needs more testing since an error in that program could propagate to other downstream programs.  And a program that updates the primary database should receive the most testing attention, with a careful analysis of the coverage reports.  Such a risk index can be derived fairly simply and can be combined with the complexity analysis discussed above. The exact algorithm for risk optimized testing will be unique to each project, but if the algorithm is derived as part of discussions during project planning it offers the opportunity for increased accuracy while controlling testing costs.