|
Testing is where the rubber meets the road in a
migration project, where we learn whether or not the project was indeed
done correctly. And, if the project was not done correctly, re-work and
re‑testing after failing initial testing is where cost overruns and late
delivery begin in a project with problems. Conversely, if the testing
is not done correctly, latent or expressed errors will appear in
production, resulting in even greater cost and delivery overruns than if
they had been found earlier in the project, when they are much less
expensive to fix.
Testing is also the most expensive part of a
project, both in money and in the time of the most knowledgeable
personnel who we would prefer to be applying their skills in tasks
creating new value rather than preserving old value. Depending on the
methodology employed for both migration and testing and on the accuracy
requirement for the testing, migration project testing can consume 40%
to 80% of a project’s total cost, including both external and internal
expenses.
In other words, for very high accuracy testing
requirements, there can be $4 or more spent on testing for every $1
spent on everything else. Yet, when planning a project, testing is
rarely given a proportionally prominent place in the project design
process. This is true even when choosing an alternate project strategy
could have dramatic impacts on the total project cost, time frame and
risk.
There is a direct relationship between cost and
accuracy. We have a tendency to think of a program as being “tested” or
“not tested”, whereas the reality is that a program may be 30% tested,
or 50% tested, or 90% tested. Except for very simple programs, it is
never the case in practice that a program will be 100% tested. The
increasing costs of testing ever more obscure and seemingly unlikely
cases invokes the law of diminishing returns, ensuring that we will
declare a program to be “good enough” tested long before we get close to
100%.
However, to the extent that a program is not 100%
tested, there is a residual risk that a latent error in the untested
part of the program will be expressed in production. And it is
difficult or impossible to predict in advance how significant that
latent error may be when it is finally expressed. Thus, budgeting for
testing is an exercise in balancing known, present day costs of testing
against unknown costs from possible future failures, which is a recipe
for either undertesting or overtesting. There is no objective,
scientific standard by which anyone can define what is “good enough.”
It is always to a greater or lesser extent intuitive and subjective.
Ultimately, then, migration project testing is an
exercise in risk management. However, technicians, both internal and
external, rarely think in terms of risk management. Indeed, it is very
uncommon for technicians who are not testing specialists to even measure
the extent of their testing, a well established process known as “test
coverage analysis” or simply “coverage analysis.” If you even don’t
measure your testing, how could you know how accurate it is?
Most testing is conducted with an ad hoc, seat of
the pants methodology, based on a combination of knowledge of the
application system internals and feedback from users of the
application. This is not to denigrate such efforts, as this is the way
most testing is done, and it is usually sufficient. But when it is
found to be insufficient in a given project, it is always too late to do
anything else except more of the same until the result is finally deemed
“good enough,” or until it is too late in the day and the system has to
go into production regardless, bugs and all. So, luck is a wild card
element in the results of every migration project, with a significant
dependency on the quality and complexity of the code being migrated, on
the migration solutions being applied to that code, and the talent and
experience of the project team.
So what is
management to think of all this? Management typically is less schooled
in testing theory and practice than the technicians, but management is
also better than most technicians at optimizing the use of business
resources and engaging in organized risk management in the conduct of a
project. Therefore, we recommend to our clients that senior management
be engaged at the highest level of planning a migration project, and
that progress be measured in a way that allows for resource optimization
and maximal risk reduction for the resources available.
In a typical project where management is not
engaged in the process, technical staff will be asked, “how much do you
need to budget for testing?” Since technical staff will tend to be
optimistic in most cases, and also because they anticipate resistance to
a significant budget request, there will be a strong tendency to
underbudget for testing. And indeed, in many cases management will
resist allocating even an insufficient budget, because of the lack of
black and white criteria from which an optimal budget can be derived.
It is a murky, gray process, governed by opinions rather than objective
facts. By the time a given budget is proven insufficient in practice,
the project will be too far along to cancel, and the additional budget
will simply have to be allocated.
For smaller projects, this may not be a bad
algorithm. Simply let the technicians test until the rate of faults
found comes to an acceptable level. If that occurs before a testing
budget is exhausted, well and good, but if not then there are no
surprises and no recriminations. In effect, in this case senior
management is accepting all the risk of faults in the process, with the
pleasant result that the price bid from migration vendors will be
minimized.
However, for larger projects, management may want
their migration vendor to share in the project risk or even to accept
all the project risk. Alternatively, management may be unsure about the
necessity or affordability of the project. Without locking down the
total cost, management may be unwilling to commit to the project at
all. Conversely, if a vendor is going to accept part or all of the
project risk, they will have to charge accordingly, which can result in
a very high price, particularly when the risk cannot be accurately
quantified in advance. A vendor is then forced to price on a worst case
basis, although there are vendors who will underprice the risk in order
to get a contract and then renegotiate if the project runs into
trouble. In this case, the client is left with the alternative of
paying more or litigating, and usually the costs, risks and delays from
litigating are greater than just paying up and getting the job finished.
For those projects where risk is to be taken
totally in-house or shared with a vendor, we recommend that business
management in consultation with technical staff set the budget for
testing, and set that budget based on primarily business not
technical criteria. Always, the question should be, “what could it
cost us if we miss a fault in one of the migrated programs?” From the
answer to that question, summed over the whole of the application being
migrated, should come the amount that management is willing to spend to
avoid the impact of such a fault. In other words, the decision should
be made in exactly the same way that insurance coverage and acceptable
premium payments are decided upon. Then, management should exercise
oversight to ensure that the budget is spent to yield the greatest risk
reduction for the resources expended. We discuss how to do this in more
detail in the body of this paper.
Both technical and
senior management may also have unrealistic expectations of a migration
vendor. Some of the faults revealed by testing will have been in the
programs for some time, but not expressed for various technical
reasons. When the migration process incidentally removes the barriers
that previously prevented their expression, faults will then appear in
production even though they were not introduced by the migration
process.
When a migration is done wholly or partly manually,
there will be variations in the way that solutions to migration issues
are implemented that subtly vary from one migration technician to
another and from one program to the next. In some cases these variations
will be the source of errors introduced by the migration process.
When automation is used to partly or wholly
introduce the solutions to migration issues into programs, there is a
tendency to expect that the automation will be perfect. This is
unrealistic. When mapping from one software/hardware environment to
another or from one language to another, there are always interacting
complexities that can never be perfectly specified in general case
transformation algorithms. Although automated migration processes do
vary in quality, principally based on the project experience and
technical sophistication of the automation employed, none will ever do
the job perfectly because none can ever anticipate every possible
combination of circumstances that require a specific solution. Every
project reveals a combination that has never been seen before in that
exact expression, and this will occur even with tools that have absorbed
the experience of several hundred projects. The technical problem is
simply too complex, regardless of what sales people might lead you to
believe. This is why all automated projects have an analysis and setup
phase where the tool is adapted to the specific circumstances of the
particular set of programs undergoing migration.
The advantages of automation are real and
compelling, in the elimination of much manual effort, in the consistency
of the results, and in the time it takes to complete the project. These
advantages are expressed in sharply reduced costs and in the
sophistication of the resulting transformed programs. However, there
will be faults, both faults inadvertently introduced into programs by
the process and previously latent faults in the programs that will now
be expressed. The question is how to find those faults quickly and
efficiently in order to control both cost and risk.
Ultimately, testing is rarely done scientifically
except by professional testers. In the sections that follow we will
describe “typical” testing and then thorough, high accuracy scientific
testing approaches. Both of these use conventional approaches to
migration testing. Then we will describe a non-conventional approach to
migration testing utilizing test automation, one that is uniquely
applicable only to migration projects and that can be used to produce
highly accurate results at lower cost.
We recommend that senior management carefully
consider the business and technical risks involved in a migration
project, and then choose the best strategy for both migration and
testing. In many cases, particularly smaller projects or projects where
an undetected fault poses minimal business risk, it may be appropriate
to accept all the project risk, which will result in the lowest price
from a migration vendor. In these cases, typical testing may be the
best choice.
In other cases, particularly larger projects and
projects where an undetected fault poses significant risk to human life
and/or the viability of the business, it may be appropriate to share the
project risk with the migration vendor, or to completely outsource the
project risk by engaging a professional testing organization. In these
cases, automation of migration testing can provide a compelling
alternative to the high cost of professional, scientific testing.
|
