This module of the Test Harness is used to demonstrate empirically that the business logic in a program is unchanged by the actions of the project.  For any given input, the modified program will be shown to produce the identical output as the unmodified version of the program, and follow the same logic path to accomplish that output.

The Test Harness is an ideal complement to a fully automated source modification project, although it can certainly be used with a manual or semi-automated project as well.  Operationally, the Test Harness works similarly to most automated source modification systems, in that there is a parsing engine that reads the existing sources, consults a repository of system information and logic templates, and uses that information to apply code transformations to create new source codes.  This process can be illustrated like this:
 

Capture Instrumentation

The Test Harness is implemented with a capture/replay architecture, analogous but vastly more comprehensive than a screen capture and replay system like QAHiperstation or Playback from Compuware.

Business rule testing works by instrumenting all programs for capture, then compiling and executing these special program versions to produce a capture file for each program or for each module within a load module. Instrumenting a program refers to the process of adding logically neutral program code into a program to perform some specialized task during execution.

The capture file is functionally equivalent to a test package for that program (see conventional testing).  All inputs, outputs, logic paths, and state variables are stored in the capture file.  Indeed, if the capture covered less than 100% of the inputs and outputs, the replay process would not work.

This capture instrumentation process looks like the left side red branch in this illustration:
 

Note that the same, identical version of a program must be used for capture instrumentation, for validation instrumentation, and for modification by the project.  If a program version changes after the project commences, then the new version must be re-instrumented and the capture execution must be re-run.  We recommend that program versions should be locked down as the project gets underway. 

Where there are mandatory changes to programs that must occur during the project, we recommend that the project proceed with the original versions, and that any modifications be re-applied after the project completes.  There are several tools that make this post-project modification process easy and painless.

The capture file must be collected on the source machine, by compiling a temporary version of the program instrumented for data capture. This program is executed exactly as the non-instrumented version of the program, except that provision must be made for the capture file.  If a mainframe CICS program, the capture file must be added to the environment.  If a mainframe batch program, then a new DD card must be added to the normal job JCL. The capture file is usually a VSAM KSDS that can take capture from a variety of programs, but it can also be a single sequential file if that is more operationally convenient.  For non-mainframe captures, the file allocation method is dependent on the operating system and compiler in use.  In certain environments, such as IDMS or IMS, the capture methodology requires a couple of utility programs, but is logically identical.

The execution can use either test data or production data.  Generally, test data is preferred because smaller volumes are easier to handle operationally, but production data may give a greater degree of test coverage.  This is a topic that is discussed at length during planning.  Note that test data is accumulated from one execution to the next, so that test coverage can be added to at any time, so long as the program version does not change.

Once the capture is complete, it is file transferred as binary to the machine being used for testing, usually an Intel workstation.  After receipt, the transferred file is unpacked and checked for integrity with a utility.  The captured data from each different program is unpacked into separate files, each of which is used to drive first the validation and then the actual testing.
 

Validation and Test Coverage Analysis

Once the file is on the workstation, the validation source is compiled and readied for executin.  This special version is used solely to validate the internal integrity of the captured data, a much more thorough validation than that of the unpack utility, and (for COBOL capture) also to produce a test coverage report.  For non-COBOL capture, the test coverage is produced directly from the validated capture data.

The test coverage analysis report shows which parts of a program have been exercised and which have not.  For example:

Analysis of the test coverage report allows the capture of additional data to bring the test coverage to a level deemed adequate by subject matter experts.  Once the capture is declared adequate, then the capture version of the program under test, with its special logic, is deleted.  No instrumented logic is permanently added to any program.

Once there is a validated capture file with adequate coverage, then the modified version of the program is instrumented for replay, or test execution.  Replay is the process by which the data captured from the original version of the program is fed to the modified program's executing logic, and the results of the execution are tested against the original results.  Any discrepancy of data, logic path or state information is immediately obvious.

This instrumentation process looks like the right side red branch in this illustration:
 

Replay Execution

During capture, the entire execution environment must be present: the OS, any files and/or databases, the datacom environment.  To this environment is added the capture file.  During replay, the original environment is simulated by the replay logic using the data in the capture file. So, for example, during capture there would be a full batch JCL with one DD card added, but during replay only the capture file is input, plus a replay parameter file.  There is only one output: a report file.

When the replay program is set to executing by the Test Harness controlling logic, any initial data buffers (such as passed parameters) are initialized with data from the capture file, and then the program logic is set to executing.  At each logic branch, the capture file is consulted to see which path the original execution took, and compares that to the path being taken by the replay execution. 

At each point in which there is an input I/O operation of some kind, the results of the input, including the record area and the state variables, are set from the capture file.  "Input" is considered broadly here, as a system date/time call, an EXEC CICS command, a SQL SELECT, or indeed any command that causes data to be transferred into the program is treated identically to a read.  The only exception to this is in the case of a keyed read, in which case the keys being read are compared to the keys from the original execution, but otherwise the keyed read is handled identically to any other input operations.

At each point in which there is an output I/O operation of some kind, the data buffer being written is compared against the data buffer from the captured data, and any discrepancy, even a single bit discrepancy, is reported.  Note that since even date/time stamps are replayed faithfully from capture, there are no spurious discrepancies that have to be investigated.

Here is an example of a data discrepancy, taken from a replay log of a converted ADSO program.  The working storage dataname PARW8580-W01-REC: contained X'0D' (decimal 013) at this point in the capture, but the replay has value X'00' (decimal 000) at this point in the replay.
 

When there is a data discrepancy, frequently it can be traced to a logic path discrepancy. This is extremely useful when debugging a program exhibiting errors, because it is usually possible to pinpoint a problem in a couple of minutes, instead of hours or longer.

Here is an example of a logic path discrepancy: