2 Conventional Testing for Migration Projects
Basically, migration testing can answer two
questions. “Does the migrated system produce the correct results?” Or,
“does the migrated system produce the same results as the old system?”
Answering the first question is the more typical approach to software
testing in general, and may be referred to as correctness or expected
results testing, if presented as a distinct form of testing at all.
Most commonly, when someone refers to software testing of any kind, they
mean the first: testing that produces results that can be shown to be
correct or that produces the results that are expected, given a
controlled set of inputs.
The process of answering the second question may be
referred to as comparison testing, equivalence testing, or regression
testing. In this less common form of testing, the same set of inputs
are given to both the old and the newly migrated sets of programs, and
the results of the new are compared or “regressed” against the results
of the old. If they are the same, which is not the same as being
correct, then the test is considered to have been passed. If the old
programs produced a result that is considered to be incorrect in some
fashion, that is of no concern so long as the new set of programs are
incorrect in the same way.
When a test plan specifies correctness testing,
then subject matter experts, people deeply knowledgeable in the use of
the system and perhaps in the implementation of the system, will have to
be intimately involved in the definition of the tests and, to a lesser
extent, the test executions as well. When a test plan specifies
comparison testing, subject matter experts will still be required, but
to a significantly lesser extent. In this case it will be often be
sufficient to demonstrate the use of the programs. In many cases,
normal everyday users of the system can be sufficient subject matter
experts for the definition of the tests.
The timely availability of subject matter experts
at key points of the project is critical to maintaining schedule and
controlling costs, which is why migration vendors may contractually
specify certain levels of availability and responsiveness. This is
particularly true with regard to test definition and execution. Subject
matter experts are frequently overscheduled, and can become a costly
bottleneck during the course of the project.
It is significantly more difficult when subject
matter experts are largely unavailable, or when there are no subject
matter experts available at all. Written documentation is frequently
incomplete, out of date, or simply wrong, if indeed any documentation is
available at all. In this circumstance a significant amount of time
will need to be spent learning the system before correctness tests can
be defined. This time may be impossible to quantify accurately in
advance. As a result, comparison tests will be easier to define,
generally from system operations. Even though easier, this will still
be a considerable effort, primarily in the logistics of assembling and
executing the tests.
An question that has to be considered early in the
planning with all approaches to testing is whether to use specially
constructed test data or to use production data. Where the run times of
batch programs are very large, it will be preferable to use test data.
Not only will test data be much smaller in volume, resulting in less
time waiting for a test run to finish, but it can be constructed to
include combinations of input data that only rarely occur in production
data. For example, there may be transactions that only occur at end of
month or end of year that could be defined specifically for the test
data.
Using production data for testing online programs
might seem to be less of an issue, since online programs typically
access only the data that is required, ignoring the bulk of the data.
While it is true that the run time of online tests will be no different
with production or test data, the setup times for the tests may be
considerable. Even after the one time conversion of data from the old
to the new data store has executed and been secured, which can be
considerable in many cases, we have to consider that the test image of
production data in use will have to be restored before each test. This
can take many hours, as opposed to a few minutes or less for low volume
test data, in direct proportion to the data volumes involved. These
database restoration times, summed over the total number of tests,
sharply limits of the number of tests that can be executed in a work
week and can have a dramatically negative impact on the project delivery
time and cost of testing.
On the other hand, producing low volume test data
that accurately reflects all of the circumstances that will be
experienced in testing with production data can be an extremely complex
and time consuming task, more than many technicians wish to invest even
when there will be a significant payback for doing so. Typical testing
will therefore consist of production data throughout, or whatever test
data can be derived quickly for early tests and production data used for
later tests. Therefore, typical testing will tend to use either test or
production data based on the concerns of knowledgeable technicians and
the time pressures that they are under at the moment. We recommend that
the decision about which to employ and when should be a subject for the
planning phase of a project.
Testing batch programs tends in be easier than
testing online programs because there is rarely any manual input
required, whereas online program testing demands a great deal of manual
input. However, easier does not necessarily mean easy.
All formal tests are described in scripts, both
batch scripts and online scripts. A batch script will be simpler than
an online script:
-
Specify the set of files and any database, noting any case
where the program can decide to access any other files based on its own
internal algorithms or via alternate JCL decks.
-
Secure an image of all files definitely or possibly input
to the program and any subroutines; this is referred to as the “before”
image.
-
Secure an image of the source to the program, including
all copy books and any subroutines with their copy books, then compile
the program and subroutines using these sources prior to execution.
This particular set of sources should be input to the migration process.
-
Secure the exact execution parameters, and execute the
program such that it is the only program updating shared files and/or
databases; if this is not possible then the programs that have to run
simultaneously will have to be run as a controlled set of programs in a
single script.
-
Sometimes, to minimize effort, a series of programs will
be run and only the results from the final program compared in the
tests. However, all intermediate files have to be secured in this case
anyway so that any discrepancies can be traced back to the creating
program.
-
Secure the “after” image of all output files plus any
database and/or any files that might be updated during the execution; if
in doubt secure them all.
-
Set up a post execution audit using system logs to ensure
that all files and/or databases actually used during execution were
secured, and that the correct images (before and/or after) were secured.
This can be tedious and time consuming, but otherwise project management
must expect that a number of test scripts will be invalid and will have
to be recreated. Alternatively, the test script data can be loaded onto
a test machine and the test re-executed, comparing against the secured
output. This script validation process is less tedious but also time
consuming.
The issues in setting up batch comparison test
scripts are primarily logistical. Securing all the files and
databases, and ensuring that the correct versions were secured without
contamination is a non-trivial task. A single error invalidates the
test. Then the secured batch test scripts must be archived and tracked,
at least one per program being migrated and sometimes several, when a
program can be executed in different modes or with varying inputs.
Then when the
source secured with the batch test script has been migrated, we can run
the comparison test.
-
All of the before data and the after data must be
converted to the new database and/or system platform, with the after
data segregated to use for post-execution comparisons.
-
Then the program or programs from the script must be
executed using the converted before data as input, updating the
converted files and databases as appropriate.
-
Then, the output files and any updated files and/or
database tables must be compared against the converted after data. The
comparison methodology may be a manual inspection of selected records
for consistency plus a count of the records, or it may use a specially
written program or a comparison utility to automatically compare the
files. Sometimes non‑meaningful differences (e.g., timestamps) must be
disregarded by the comparison tool in order to avoid false positives.
-
If the program fails the test, when it is corrected the
test needs to be run again until it passes.
A correctness test on a batch program avoids the logistical headache of
having to convert the data twice and run the comparison program or
utility against the various data stores. However, the batch test script
must define how the tester will determine whether the output is correct,
perhaps using a control total or some other aspect of the program’s
execution that is known to the programmer or operator creating the batch
test script. Note that this method is dependent on the depth of
knowledge of the person creating the test script for the accuracy of its
results.
Online comparison test scripts must do what a batch
test script does with regard to sources and before/after images of the
necessary data stores. In addition, the online test script will almost
always involve many programs being tested as a group, and it is
important to ensure that all possible types of transactions are input to
all the programs. The precise screen inputs and the precise screen
outputs need to be secured, either to hard copy or in a software tool
designed for the purpose.
Then, during the testing of the migrated programs,
the screen inputs must be precisely the same in precisely the same
sequence, and the outputs compared against the output screen images. A
number of software testing tools will automate this process, first of
capturing the data entered, and then replaying that data exactly into
the screen. However, it is important to note that these basic, online
capture/replay tools apply only to the screen image, not the
database access or update, and they do nothing whatsoever for batch
testing.
Note that these screen images can become voluminous
very quickly. When the script is completed, then the updated files
and/or databases are compared as in the batch comparison script.
If done rigorously, online correctness tests will
both compare the screen outputs and some type of algorithm to determine
whether the files and/or databases have their expected values updated.
However, these tests are frequently abbreviated to comparing only the
screen outputs, disregarding any validation of the data stored. These
tests are somewhat less difficult for testing personnel because a
keystroke error may be disregarded where it would require the test to be
restarted for a rigorous comparison test, but they require a greater
knowledge of the programs to be done to the same level of accuracy.
Testing is generally divided into 5 stages:
-
Unit tests, typically single program tests often with test data
if available.
-
System or run unit tests, in which programs that normally run in
a job stream or concurrently in an online environment are run together
in a test simulating production usage. Unit test scripts may be
combined, or new run unit test scripts created, as appropriate, using
test data if available and production data if not.
-
Parallel test, in which the old and new systems are run in
parallel with production data, with the same work being done on both
systems and the results compared.
-
Acceptance test, a formal sign-off that the system’s parallel
test has completed satisfactorily, usually involving an extended
demonstration to stakeholders not directly involved in the parallel
testing. The project is usually considered complete at this point.
-
Stress tests, in which the system is artificially driven at a
higher than normal transaction rate to determine whether any bottlenecks
exist that require tuning and to determine the effective capacity of the
new environment. These tests are seldom performed in a migration
project.
Interestingly, it is usually unit testing where the
greatest efforts are expended, and the greatest percentage of errors
found. It is not unusual for unit testing to consume 50-75% or more of
all testing, because of the logistical effort of creating, validating
and executing the tests.
When typical testing is being done by testers with
little knowledge of the application programs, comparison testing is
generally preferred because the testers will not know how to determine
whether the results are correct. If the test scripts are prepared in
sufficient detail by knowledgeable staff, then testers with less
knowledge can execute the tests. However, the greater effort is usually
in creating the test scripts rather than in executing them, so avoiding
the use of subject matter experts in creating detailed test scripts
necessarily involves significant compromise of testing accuracy. This
may be appropriate, but it should be done knowingly. In general,
comparison tests are preferred if subject matter experts cannot be
regularly involved in the details of testing and test evaluation
This description will perhaps give the flavor of
why even typical testing is expensive. There are a lot of hours
required to secure and verify each test script, and then to execute
them, particularly if using production data volumes. Since there will
be some number of faults, depending on the complexity of the programs
being migrated and the precise nature of the migration process, programs
failing their test will have to be re‑worked and then re-tested, perhaps
more than once.
|
