Computer Algebra System Testing


In 4 steps, each 3 months long, you can get (much) more control over 
the problems in Maple using a methodology including (but not limited 
by) automated  problem identification. The anticipated numbers are 
many  thousands of bug manifestations (some of them are not "linearly 
independent"), in hundreds functions, including all mathematical 
functions, interface, connectivity, and, naturally, documentation.

The important part of the approach is GEMM.

GEMM, a heuristic-driven adaptive automated monitor, to be implemented, 
tries to

  *  identify the "minimal" form of the bug
  *  double check it
  *  locate the version where it occurred for the first time
  *  give when possible a very short proof that it IS a bug
  *  classify the problem
  *  add the classified problem to a proper database

In a nutshell, the project could be split into 4 steps each 3 months
long. As you had pointed out, at each step, you would have the next
chunk of information about the Maple's problems.

By the end of the project, you would have, totally,

* several thousand of bug manifestations, feature requests & indications
  on the various places where the behavior of Maple is now still user-
  acceptable but possibly should be modified to satisfy the customers'
  needs in the best way

* the electronic form of some 10 selected sources which, over the decade
  of my symbolic software testing practice, I have found most useful for
  black box testing. It could help you in the the next releases testing.

* a set of testing units (for details, please see below)

* a monitor program which controls these units (please see below)


Now about the details.

My machine is Athlon FX 1600+/512 Mb/42 Gb/12 Gb. I can run Maple under
any Windows clone and Linux.

I have my own precomputed database of 10000+ examples where the computer
algebra systems often yield the invalid answers. To construct it, I used

Analytic      Analytic-93, Analytic-91
Derive        5.06 beta, 5.05, 5.04, 4.11, 4.02, 3.11, 2.59, 2.57, 1.62
Maple         7, 6, V (releases 5, 4, 3)
Mathematica   4.2 beta, 4.1, 4.0, 3.0, 2.2, 1.2
MuPAD         2.5 beta, 2.0, 1.4
Reduce        3.6, 3.4.1, 3.3

This database enable us to start off the bug identification immediately.

The next steps are as follows.

........................................................................

STEP 1    Check the precomputed database.

It would take about 3 months.

You could expect about 1000 problem reports at this step.

........................................................................

STEP 2    Scan, convert into the Maple usable format, and verify the
          Maple's output for the following problem books

     1.  Calculus textbook by Dorogovtsev   (about 2,500 problems)
     2.  Calculus textbook by Kudryavtsev   (about 10,000 problems)
     3.  Calculus textbook by Demidovich    (about  4500 problems)
     4.  A selected collection of best
         10 high school level & Olympiad
         level textbooks                    (about 2,500 problems)
     5.  ODE textbook by Filippov           (about 1,200 problems)

         that is about 20,700 selected problems totally.

It would take about 3 months.

You could expect about 1000 problem reports at this step.

........................................................................

STEP 3    Program in the Maple language a set of procedures which will
          verify the most important for the Maple users mathematical
          functions (int, limit, factor, series, taylor, residue, sum,
          product, simplify & some other) over the set of ALL elementary
          & special functions & generate automatically a sizable part
          of the problem reports. Maybe, to be highly readable, a certain
          part of these reports would require, at this step, extra human
          editing. Automated sending system which is connected to the 
          automated testing system guarantees that you will receive the 
          report on a new-identified problem within several minutes 
          after it is done.

It would take about 3 months.

You could expect a surge in the total number of problem reports at this
step. It is difficult to predict the exact magnitude of the surge.

........................................................................

STEP 4    Program in the Maple language a monitor program which will
          try to call the units programmed at the STEP 3 in such a way
          that the performance of the complex would reach its maximum

It would take about 3 months.

You could expect a new surge in the total number of problem reports at
this step. Again, it is difficult to predict its exact magnitude.
E-mail: vb@cybertester.com