| Source Code Instrumentation System Test Harness 2000 is based on our Source Code Instrumentation System (SCIS).
"Instrumentation" is a term that refers to the controlled introduction of new
code into a program or suite of programs to accomplish a particular purpose.
This usually refers to the introduction of new source code but there have been
object code instrumentation systems developed as well.
SCIS allows us to take any COBOL or COBOL-like source language and process it
to add code at relevant points within each source. This is frequently
before and after each specified I/O verb, but could be any COBOL, SQL, CICS or
DL/1 verb as well as certain relevant other points in a source, such as source
begin/end or paragraph begin/end.
This additional code can be capture logic or replay logic for the Test
Harness as discussed in detail below. But it can also be configured for
any special purpose. For example, we have used it to create standalone
test coverage analysis logic, to monitor the amount of CPU time used in a given
paragraph within an executing program, to monitor the elapsed time for each SQL
command within a program, and to create an extended set of
instructions to simplify the handling of XML within a program. To do this
we defined EXEC XML constructs as an extension to the COBOL language to provide
high speed, low overhead XML encoding and decoding routines, effectively a DOM
parser, as compared to the XML PARSE command which is effectively a SAX parser. SCIS can be used to fulfill any requirement for the controlled
insertion of logic into a body of source code, either temporary insertion or
permanent insertion.
SCIS in general, and the Test Harness specifically, can be applied to
any COBOL application system on any platform with any operating system:
IBM mainframe, non-IBM mainframe, mini-computer, mid-range system (e.g.,
AS/400), Unix system, or workstation. With some limitations, it
can be applied to projects converting from one language to COBOL.
For example, it has been used successfully in IDMS projects that convert
the ADSO language to COBOL, and could be trivially extended to IBM
assembler and to other COBOL-like 4GLs such as IDEAL, Natural, etc.
With some additional effort, it could be extended to non-COBOL
languages.
Test Harness 2000
Any given online program is considered to have 3 logical layers: the business
rule layer, the I/O layer, and the presentation (or user interface) layer.
Batch programs have 2 layers: the business rule layer and the I/O layer, since
there is no need for presentation in batch execution.
Test Harness 2000 separates the testing of each logical layer of a program.
The usage of the Test Harness for each layer is sufficiently different
operationally that we organize it as 3 separate products. Indeed, we find
that we may use only business rule testing in one project, only I/O testing in
another project, and all 3 in a third project. Which one to use depends on
what it is that the project is doing to the programs.
The Test Harness makes use of one critical feature that
differentiates a migration or conversion project from all conventional
application development projects:
-
A
conventional project seeks, in most cases, to introduce some change
into a system that changes the business functionality of the
affected programs. By definition, the before change version of
the system will function differently from the after change version
of the application.
-
A
migration or conversion project seeks to change some aspect of the
infrastructure of a system (language, database, etc.) but without
changing the business functionality. From a functional point
of view, the before change version of the system will function
identically to the after change version of the application.
Test
Harness 2000 takes advantage of this subtle difference by proving
analytically that two versions of a program are functionally identical.
When you have an infrastructure change the business functionality does
not change, and so fully automated testing works. However, when
you have a functionality change, all the Test Harness can do is document
where the execution produces different results - it cannot prove one way
or the other whether the change is correct. In other words,
automated testing proves equivalence, not correctness.
It is
often tempting for project managers to want to introduce some changes in
business functionality during a migration project. This is always
a false economy. The slight savings in programming effort is
offset 10:1 or more by the increased cost of testing imposed when
business functionality changes. When considered in light of the
discussion above, this makes sense. You can gain the cost and
thoroughness advantages of automated equivalence testing when you change
only the infrastructure but not the functionality, and you are forced
into the much more expensive manual correctness testing when you make
any changes to the functionality. It is much cheaper and quicker
to test first the migration or conversion change, and then to separately
change and test the functionality.
Key Benefits of Test Harness 2000
-
Substantially reduced effort in creating test packages - you just
execute a special version of the program and the test package is created
automatically.
-
Individual program coverage analysis reports quantifies adequacy of testing.
-
Project wide construct coverage
analysis assures that minimum test coverage adequacy has been achieved.
-
Key staff are involved only to determine adequacy of coverage,
dramatically reducing the time they must contribute to the project.
-
For vendors - improved quality,
reduced internal costs and quicker project acceptance.
-
For clients - substantially
reduced risk of undetected faults affecting production, reduced cost of
testing, and an earlier date to implement the project results into
production and secure its benefits.
Test Harness 2000 - Modules
- Test Harness 2000 - Business Rule Testing
-
Business rule testing works by instrumenting all programs for
capture, then compiling and executing these special program versions to
produce a capture file for each program or each module within a load
module. After the program is modified by the project, that version is
instrumented for replay, and the capture file is replayed through that
version. If the outputs and logic paths are identical for the same
inputs, the program is proven to be functionally equivalent in its business
rule logic. The capture file is also used to provide a coverage analysis report which
shows which parts of a program have been exercised and which have not.
This allows the capture of additional data to bring the test coverage to a
level deemed adequate by subject matter experts. Once the capture is
declared adequate, then the capture version of the program under test, with
its special logic, is deleted. No instrumented logic is permanently
added to any program.
It is noteworthy that Business Rule Replay (BRR) is performed on a
program by program basis, as all information necessary for the replay is
contained wholly within the capture file. Nothing external to the
program is required for replay. As a result, it is possible to capture
on a mainframe and replay on a workstation, with substantial cost and
productivity advantages.
Test
Harness 2000 - I/O Testing
I/O testing is relevant when a project
changes the operational data store, typically by moving from a
non-relational to a relational data store. I/O testing is
similar to business rule testing in that the programs are first
instrumented, compiled and executed for capture, and a subsequent
replay is performed.
However, it is significantly different
operationally. During replay, the actual calls to the database
will be executed in order to test the accuracy of the translation. As a result, all data for the database must
be secured and coordinated with the execution of the capture, to
allow the data to be converted and loaded into the database prior to
I/O replay testing.
There are two forms of I/O replay, single-threaded replay and
multi-threaded replay. Single threaded I/O replay requires
that the capture be single threaded, but if this condition can be
met then the I/O replay is very similar to business rule replay,
with the addition of the live I/O's to the database. In
general, single threaded capture and replay is recommended for batch
programs, and multi-threaded capture and replay is recommended for on-line programs.
Test Harness 2000 - Datacom Testing
Datacom testing can be performed in concert with I/O
replay. In this case, the online inputs and output from the various
online programs are extracted in temporal sequence, in a manner very
similar to multi-threaded I/O replay. But then the driver program
passes the messages inputs to minimally instrumented versions of the
converted programs, and compares the outputs to the originally
recorded outputs. Datacom testing tests only the user
interface directly, but indirectly tests all the business rule logic
and the I/O logic as well. Note that not all online features
of IBM CICS are supported.
|
