A Proposal for

		       Porting the Common Lisp Test Suite

				      - - -

				  [Proposal #5]

				  Sept. 23, 1988 

			       David M. J. Saslav

				      - - -

Introduction:

	A significant portion of Common LISP functionality is not tested by the
	existing Common LISP regression test suite (found in SYS:VALID;).  This
	suite is run whenever a new system has been compiled.

Proposal:

	Part I:

	Spend one day or so surveying the test suite to determine which
	functionality is without test/validation sequences, attaching
	programmer-time	estimates for each major category of tests not
	currently in existence. (Already underway)

	Part II:

        Write test/validation regression tests for that portion of overall
	CommonLISP functionality not currently covered by the existing Common
	LISP test suite.

Rationale: 

In our case, the phrase "regression" has an important meaning: we must
ensure that the Falcon software is at least as good as, if not better
than, the existing Lambda system.  The existing test suite is the
starting point for a robust Common LISP regression suite, one which will
be usable over the entire lifetime of all future projects.

The existing test suite is healthy.  When first brought up on the
Lambda, it diagnostically located over twenty bugs as well as numerous
examples of anomalous behaviour requiring documentation.

The test suite needs immediate expanding, however.  If the first
comprehensive test of the complete Falcon system occurs after most (or
all) of the modules have been ported, newly-introduced problems will
take the maximum possible toll on developer time.  Full testing of the
core software gives an early indication of the scope and nature of
newly-introduced problems, thus minimizing the time required to solve
them.

While this kind of testing is supplemental to the testing that every
developer is responsible for within the particular development domain,
this regression suite should serve as a collecting place for these
tests.  They can be run every time the system changes substantially
(e.g., when the system is recompiled).  That is the meaning of the
phrase "regression" -- these tests are designed primarily to ensure that
the state of the software does not "regress", i.e., that the existing
software base never suffers damage from any major system development.

-keith/dmjs


============================== Commentary

[smh 30sep88]

Two issues to be considered:

[1]	I question whether the focus on Common Lisp is appropriate.
Although we might like to build (and advertise) a Common Lisp machine,
in fact or product will and must support both CL and Lambdoid Zetalisp.
Anything that can be said about the time savings dur to having a
regression test facility for CL goes just as much for Zetalisp.  Given
that we know we won't have resources to do a 100% job in creating a test
suite, then we owe it to ourselves to factor in Zetalisp functions and
constructions in order to prioritize them into our testing efforts.

	I am here thinking about language features such as (but by no
means limited to) locatives, dynamic closures, array leaders,
multiprocessing, and (ideally) flavors.  All these require special
support in both the compiler and low-level runtime system.  Having
regression tests for this support is just as useful as for basic CL
functionality.  Actually, I'd be willing to punt testing flavors (except
for the basic message sending and instance reference mechanisms) because
the system itself will test the high-level components adequately, but on
the other hand testing locatives is desirable because so many hidden
places in the system try to use them.

[2]	Obviously, barring some serious manpower additions, we will not
be able to complete a full validation/regression testing suite for our
project.  Given this, I feel we need to develop some rough estimates of
what portion of the job we might complete with plausible manpower
assignments.  We all agree that programmers *should* add tests as they
implement or debug some component of the system, but we all also
recognise that it is difficult to get programmers to do so.  This is
especially true if there are serious time pressures -- "externally
invisible" tasks like testing are usually the first to be compromised.
Therefore, for project scheduling purposes we need to decide on a
plausible surcharge ratio to factor into each implementor's time --
perhaps 10-20% ??  We should schedule that time for writing tests, and
then make completion of tests part of each milestone item.  If we just
leave test coding to the good graces of each implementor, it won't
happen.

	In any case, we can't leave all testing as part of the task of
the code writer.  Many areas of Lisp functionality are implemented
inside large files which will be cross compiled and downloded, and our
plans intend this to proceed *very* quickly once it gets started.
Coding any significant amount of testing as part of this task would
seriously alter our porting time estimates!

====================