07/10/07 tscoffe:  This document is for discussion of rewriting the Xyce regression testing software.

Current configuration files for testing:

Scripts that actually start the testing:
/Net/Proj/Xyce/XTF/kickoff.sh
/Net/Proj/Xyce/XTF/kickoff_arrowhead.sh
/Net/Proj/Xyce/XTF/arrowhead_kickoff.sh

Files that define which environments to do testing on and specification of what configure & build options are needed for each environment:
/Net/Proj/Xyce/XTF/nightly/test_nightly_serial.cfg
/Net/Proj/Xyce/XTF/nightly/test_nightly_parallel.cfg
/Net/Proj/Xyce/XTF/nightly/test_nightly_serial_windows.cfg
/Net/Proj/Xyce/XTF/nightly/test_weekly_serial.cfg
/Net/Proj/Xyce/XTF/nightly/test_weekly_parallel.cfg
/Net/Proj/Xyce/XTF/nightly/reconfigure.cfg

Files that defined which tests are run on each environment:
/Net/Proj/Xyce/XTF/XTF/env_list_nightly_serial
/Net/Proj/Xyce/XTF/XTF/env_list_nightly_serial_windows
/Net/Proj/Xyce/XTF/XTF/env_list_nightly_parallel
/Net/Proj/Xyce/XTF/XTF/env_list_weekly_serial
/Net/Proj/Xyce/XTF/XTF/env_list_weekly_parallel

Location of perl on each machine (this could go away trivially by replacing perl's path with "/usr/bin/env perl")
/Net/Proj/Xyce/XTF/XTF/perl_path.cfig

Lists of tests that are used in the env_list files above (these are mostly maintained by the developers):
Xyce_Regression/TestScripts/TestLists/regr_nightly_parallel_klu
Xyce_Regression/TestScripts/TestLists/regr_nightly_serialonly
Xyce_Regression/TestScripts/TestLists/regr_nightly_serial
Xyce_Regression/TestScripts/TestLists/cert_nightly_parallelonly
Xyce_Regression/TestScripts/TestLists/cert_nightly_parallel
Xyce_Regression/TestScripts/TestLists/cert_nightly_serialonly
Xyce_Regression/TestScripts/TestLists/cert_verbose_serial
Xyce_Regression/TestScripts/TestLists/cert_nightly_serial
Xyce_Regression/TestScripts/TestLists/cert_windows_serial
Xyce_Regression/TestScripts/TestLists/cert_weekly_serial
Xyce_Regression/TestScripts/TestLists/cert_weekly_parallel
Xyce_Regression/TestScripts/TestLists/regr_weekly_serial
Xyce_Regression/TestScripts/TestLists/regr_weekly_parallel
Xyce_Regression/TestScripts/TestLists/regr_nightly_newdae_only
Xyce_Regression/TestScripts/TestLists/cert_noverbose_serial
Xyce_Regression/TestScripts/TestLists/regr_dakota_serial
Xyce_Regression/TestScripts/TestLists/valgrind
Xyce_RadRegression/TestScripts/TestLists/rad_cert_nightly_parallelonly
Xyce_RadRegression/TestScripts/TestLists/rad_regr_nightly_serialonly
Xyce_RadRegression/TestScripts/TestLists/rad_regr_nightly_serial
Xyce_RadRegression/TestScripts/TestLists/rad_regr_nightly_parallel
Xyce_RadRegression/TestScripts/TestLists/rad_cert_nightly_parallel
Xyce_RadRegression/TestScripts/TestLists/rad_cert_nightly_serial


Total number of files that must be modified (sometimes heavily) for release testing:  9
Total number of files that must be modified when adding a new machine:  7


07/10/07 tscoffe:  Consider the following configuration file structure:

Machine:   aji.sandia.gov
  Hardware:  Intel64
  OS:  Linux
  MaxProcs:  2
  Jobs:  Serial, Parallel

Machine:  lifshitz.sandia.gov
  Hardware:  Intel32
  OS:  Linux
  MaxProcs:  2
  Jobs:  Parallel

Machine:  winston.sandia.gov
  Hardware:  Intel32
  OS:  FreeBSD
  MaxProcs:  1
  Jobs:  Serial

Machine:  blackfoot.sandia.gov
  Hardware:  Intel32
  OS:  Linux
  MaxProcs:  1
  Jobs:  Serial

Machine:  S865885
  Hardware:  PPC
  OS:  OSX
  MaxProcs:  2
  Jobs:  Serial, Parallel

Machine:  kachi.sandia.gov
  Hardware:  Intel32
  OS:  OSX
  MaxProcs:  2
  Jobs:  Serial, Parallel

Machine:  arrowhead.srn.sandia.gov
  Hardware:  Intel32
  OS:  Windows
  MaxProcs:  1
  Jobs:  Serial

Machine:  tbird.sandia.gov
  Hardware:  Intel64
  OS:  Linux
  MaxProcs:  32
  Jobs:  Parallel

Testing Report Options:
  Mail To:  xyce-developers@sandia.gov

Xyce CVS:
  /home/msmeine/XTF/Xyce_Testing/Xyce

Xyce_Regression CVS:
  /Net/Proj/Xyce/XTF/Xyce_Testing/Xyce_Regression

Tests:
  Default Nightly Serial:
    +search
    +nightly_serial 
    -nightly_parallelonly
    +rad_nightly_serial
    -rad_nightly_parallelonly
    +valgrind

  Default Nightly Parallel:
    +search
    +nightly_parallel
    -nightly_serialonly
    +rad_nightly_parallel
    -rad_nightly_serialonly
    
  Default Weekly Serial:
    +weekly_serial

  Default Weekly Parallel:
    +weekly_parallel

  Rad Binary Nightly Serial Rad:
    +rad_nightly_serial

  Rad Binary Nightly Serial NonRad:
    +Default Nightly Serial Tests
    -rad_nightly_serial

  Rad Binary Nightly Parallel Rad:
    +rad_nightly_parallel

  Rad Binary Nightly Parallel NonRad:
    +Default Nightly Parallel Tests
    -rad_nightly_parallel
    -rad_nightly_serial

  All Debug:
    # No tests

  All Verbose:
    # No tests

Environments:
  Intel Linux Serial gcc34
  Intel Linux Serial gcc34 olddae
  Intel Linux Serial gcc34 noverbosity
  Intel Linux Serial gcc41 dakota
  Intel Linux Serial icc91
  Intel Linux Serial gcc34 norad
  Intel Linux Serial gcc34 norad norad
  Intel Linux Serial gcc34 norad norad olddae

  Intel64 Linux Serial icc91
  Intel64 Linux Serial icc91 olddae
  Intel64 Linux Serial icc91 noverbosity
  Intel64 Linux Serial gcc41 Dakota

  Intel FreeBSD Serial gcc34
  Intel FreeBSD Serial gcc34 olddae

  Intel OSX Serial icc91
  Intel OSX Serial icc91 olddae
  Intel OSX Serial icc91 noverbosity
  Intel OSX Serial icc91 noverbosity olddae
  Intel OSX Serial icc91 ALLDEBUG
  Intel OSX Serial icc91 dakota

  Intel Linux MPICH gcc34 
  Intel Linux MPICH gcc34 olddae
  Intel Linux MPICH gcc34 klu
  Intel Linux MPICH gcc34 norad
  Intel Linux MPICH gcc34 norad norad
  Intel Linux MPICH gcc34 norad norad klu olddae
  Intel Linux MPILAM gcc34 
  Intel Linux MPILAM gcc34 olddae

  Intel64 Linux MPICH icc91
  Intel64 Linux MPICH icc91 klu
  Intel64 Linux MPILAM gcc34  

  Intel OSX MPICH icc91 
  Intel OSX MPICH icc91 klu
  Intel OSX MPICH icc91 olddae

  Intel Windows Serial
  Intel Windows Serial olddae
  Intel Windows Serial norad
  Intel Windows Serial norad norad olddae

07/16/07 tscoffe: 
These testlists are getting out of hand, how about we consider putting all this
information about where & when to test in a file in each test directory?
The file could be named:  tags
Baseline:  # These define the baseline environments this test will run in
  Serial
  Parallel
  rad
  nightly
  weekly
  exclude  # don't run on a regular basis
Required:  # This test will only run in environments with all these tags
  verbose
  noverbose
  dakota
  windows
  NWCC
  klu
Optional: # These do not remove a test from normal testing
  valgrind
  olddae
  newdae

Lets put a few details out there:

A test that can run in serial and parallel and should be run nightly:
Baseline:
  serial
  parallel
  nightly

A test that can be run in serial but not parallel and should be run nightly:
Baseline:
  serial
  nightly

A test that should be run in serial, nightly, and with valgrind:
Baseline:
  serial
  nightly
Optional:
  valgrind

A test that should be run in serial or parallel, is a rad test, and should be tested on olddae and klu environments:
Baseline:
  serial
  parallel
  rad
  nightly
Optional:
  olddae
  klu

Is there a simpler way to get all of this???  Basically, there are required
tags and there are optional tags, these need to be dealt with separately.

If I do this, then I think run_xyce_regression should be able to take an
environment name and produce a list of tests for that environment.

I still need a unique name for each environment, and the hw_os_... convention
is one way to do that.  Here is a slight variation on this which should be
compatible with my tags above.

HW_OS_Mode_Compiler[_BuildOpts][-RunOpts]

HW:   Intel
OS:   Linux
Mode: MPILAM
Compiler:  icc91
BuildOpts:  noverbose,norad
RunOpts:  rad,klu

This would turn into:

Intel_Linux_MPILAM_icc91_noverbose_norad-rad-klu

Then you could give this to run_xyce_regression and have it create a test list.

How do I deal with optional and required tags?
Say I had a test with the following tags:
#Test A
Baseline:
  serial
  parallel
  rad
  nightly
Required:
  olddae

And I had another test with the following tags:
#Test B
Baseline:
  serial
  parallel
  rad
  nightly
Optional:
  olddae

Which environments will pick up these tests?
Intel_Linux_Serial_gcc34-olddae      Test A and Test B
Intel_Linux_Serial_gcc34                        Test B

This setup would have entries in the Xyce testing configuration file like:

Nightly:
  Machines:  aji, blackfoot, winston, kachi, lifshitz
  Environments:
    Intel_Linux_Serial_gcc34
      baseline: +nightly, +serial
    Intel_Linux_Serial_gcc34-olddae
      baseline: +nightly, +serial
    Intel_Linux_Serial_gcc34_noverbosity
      baseline: +nightly, +serial
      optional:  +noverbose
    Intel_Linux_Serial_gcc41_dakota
      baseline: +nightly, +serial
      required:  +dakota
    Intel_Linux_Serial_icc91
      baseline: +nightly, +serial
    Intel_Linux_MPICH_gcc34
      baseline: +nightly, +mpich
      optional: -klu
    Intel_Linux_MPICH_gcc34-klu
      baseline: +nightly, +mpich
      optional: +klu
    Intel_Linux_MPILAM_gcc34
      baseline: +nightly, +mpilam
      optional: -klu
    Intel64_Linux_Serial_icc91
      baseline: +nightly, +serial
    Intel64_Linux_Serial_icc91-olddae
      baseline: +nightly, +serial
    Intel64_Linux_Serial_icc91_noverbosity
      baseline: +nightly, +serial
      required: +noverbose
    Intel64_Linux_Serial_gcc41_dakota
      baseline: +nightly, +serial
      required: +dakota
    Intel64_Linux_MPICH_icc91
      baseline: +nightly, +mpich
      optional: -klu
    Intel64_Linux_MPICH_icc91-klu
      baseline: +nightly, +mpich
      optional: +klu
    Intel64_Linux_MPILAM_gcc34
      baseline: +nightly, +mpilam
      optional: -klu
    Intel_FreeBSD_Serial_gcc34
      baseline: +nightly, +serial
    Intel_FreeBSD_Serial_gcc34-olddae
      baseline: +nightly, +serial
    Intel_OSX_Serial_icc91
      baseline: +nightly, +serial
    Intel_OSX_Serial_icc91-olddae
      baseline: +nightly, +serial
    Intel_OSX_Serial_icc91_noverbosity
      baseline: +nightly, +serial
      required: +noverbose
    Intel_OSX_Serial_icc91_ALLDEBUG
      baseline: +nightly, +serial
      required: +ALLDEBUG
    Intel_OSX_Serial_icc91_dakota
      baseline: +nightly, +serial
      required: +dakota
    Intel_OSX_MPICH_icc91
      baseline: +nightly, +mpich
      optional: -klu
    Intel_OSX_MPICH_icc91-klu
      baseline: +nightly, +mpich
      optional: +klu

Weekly:
  Machines:  aji, blackfoot, winston, kachi, lifshitz
  Environments:
    Intel_Linux_Serial_gcc34
      baseline: +weekly, +serial
    Intel_Linux_Serial_gcc34-olddae
      baseline: +weekly, +serial
    Intel_Linux_Serial_gcc34_ALLDEBUG
      baseline: +weekly, +serial
      required: +ALLDEBUG
    Intel_Linux_Serial_gcc34_noverbosity
      baseline: +weekly, +serial
      required: +noverbose
    Intel_Linux_Serial_gcc41_Dakota
      baseline: +weekly, +serial
      required: +dakota
    Intel_Linux_MPICH_gcc34
      baseline: +weekly, +mpich
      optional: -klu
    Intel_Linux_MPICH_gcc34-olddae
      baseline: +weekly, +mpich
      optional: -klu
    Intel_Linux_MPILAM_gcc34
      baseline: +weekly, +mpilam
      optional: -klu
    Intel_Linux_MPILAM_gcc34-olddae
      baseline: +weekly, +mpilam
      optional: -klu
    Intel_Linux_MPICH_gcc34_norad
      baseline: +weekly, +mpich, -rad
      optional: -klu
    Intel_Linux_MPICH_gcc34_norad-olddae
      baseline: +weekly, +mpich, -rad
      optional: -klu
    Intel_Linux_MPICH_gcc34_norad-rad-klu-olddae
      baseline: +weekly, +mpich, +rad
    Intel64_Linux_Serial_icc91
      baseline: +weekly, +serial
    Intel64_Linux_Serial_icc91-olddae
      baseline: +weekly, +serial
    Intel64_Linux_Serial_icc91_ALLDEBUG
      baseline: +weekly, +serial
      required: +ALLDEBUG
    Intel64_Linux_Serial_icc91_noverbosity
      baseline: +weekly, +serial
      required: +noverbose
    Intel64_Linux_Serial_gcc41_Dakota
      baseline: +weekly, +serial
      required: +dakota
    Intel_FreeBSD_Serial_gcc34
      baseline: +weekly, +serial
    Intel_FreeBSD_Serial_gcc34-olddae
      baseline: +weekly, +serial
    Intel_OSX_Serial_icc91
      baseline: +weekly, +serial
    Intel_OSX_Serial_icc91-olddae
      baseline: +weekly, +serial
    Intel_OSX_Serial_icc91_ALLDEBUG
      baseline: +weekly, +serial
      required: +ALLDEBUG
    Intel_OSX_Serial_icc91_noverbosity-olddae
      baseline: +weekly, +serial
      required: +noverbose
    Intel_OSX_Serial_icc91_Dakota
      baseline: +weekly, +serial
      required: +dakota
    Intel_OSX_MPICH_icc91
      baseline: +weekly, +mpich
    Intel_OSX_MPICH_icc91-olddae
      baseline: +weekly, +mpich
    Intel_Linux_Serial_gcc34_norad
      baseline: +weekly, +serial, -rad
    Intel_Linux_Serial_gcc34_norad-olddae
      baseline: +weekly, +serial, -rad
    Intel_Linux_Serial_gcc34_norad-rad
      baseline: +weekly, +serial, +rad
    Intel_Linux_Serial_gcc34_norad-rad-olddae
      baseline: +weekly, +serial, +rad


This is still awefully long.  Maybe some other system for specifying the
environments would be good.

07/17/07 tscoffe:  
  KLU is a problem:  We have tests which need klu, but only in parallel.
  serialonly is a problem:  We have some tests in a subdir that are serialonly, and the rest are okay in parallel.

  The example config file above does not handle releases well.  I need separate
  blocks for different kinds of testing.  This block needs to include:
  Xyce CVS, Xyce_Regression CVS, MailTo subject, 

  To deal with these problems, lets back up a bit and consider the possibility
  that we'll still have to specify which tags we should specify for each
  environment.  Then we're just saying what kinds of tests we want to test, not
  the names of the tests themselves.

  Note:  we need build options for each environment.
