XYCE_VERIFY.PL(1)

SYNOPSIS
xyce_verify.pl [ --verbose ] [ --debug ] [ --goodres=good .step result file ] [ --testres=test .step result file ] [ --help ] netlist goodfile testfile [ plotfile ]

NAME
xyce_verify.pl

AUTHORS
Tom Russo, SNL, Electrical Systems Modeling Todd Coffey, SNL, Simulation Modeling Sciences

DESCRIPTION
xyce_verify.pl is a PERL script to compare files of SPICE-like simulation output producing RMS errors and RMS relative errors on standard out, optionally producing a file on plotfile for plotting. A netlist corresponding to the output must be given, as the output will be checked in various ways to see that it conforms to the netlist specification. "good" and "test" input files consist of lines of the form typical of Xyce(TM) .prn files, or output created in ChileSPICE using the

    print v(1) v(2) v(3) [...] > goodfile

command in the interactive interpreter. Lines consist of an integer index, followed by a time and any number of output variables. Input may be space, tab or comma delimited. If given data files from either Xyce(TM) or ChileSPICE, xyce_verify.pl will also check that the column names in the both data files agree with the output variables specified on the ".print" line of the netlist. Note that in the case of SPICE output, long .print lines will be paginated by the simulator in a way that xyce_verify.pl cannot parse --- it expects all the output variables for a given time or DC sweep step to be on one line. It is also legal to give xyce_verify.pl an unadorned datafile with no header or footer (perhaps generated by a non-spice code), but this disables the check that xyce_verify.pl does on the titles of the output columns. The first file listed on the command line will be taken to be the netlist from which the "test" file is generated. This netlist will be partially parsed and both the analysis (.DC or .TRAN,) and output (.PRINT) lines are used to determine what sort of error analysis to do. The second argument is taken as a file of "gold standard" data, and the third file will be taken to be a file to test against it.

What it does
xyce_verify.pl does some basic sanity checking on the files it is given, then peforms a detailed RMS error analysis of the data, comparing the "test" file to the "good" file. It exits with exit code 0 and no output if the two files match to within tolerances, and with a non-zero exit code and some output otherwise. For DC sweep netlists, xyce_verify.pl checks that both data files contain output for every step of the DC sweep requested by the netlist. This check makes some very restrictive assumptions about the netlist, and limits the applicability of the current version of xyce_verify.pl to DC sweep netlists that conform to those assumptions. See the Limitations section, below. DC sweeps of one or two sources are supported. If two sources are swept, xyce_verify.pl does a comparison of output by treating the output as a family of curves with the fast-varying source as the independent coordinate for each curve and the slowly-varying source as a parameter identifying the curve. For transient simulations, xyce_verify.pl checks that the simulator printed out an appropriate "end of simulation" footer in the output file. The only "end of simulation" footers it recognizes are those from Xyce(TM) and ChileSPICE. Since runs of different codes might produce output at different times but still be performing acceptably, xyce_verify.pl interpolates the "good" data to the times in the test data file to produce a "good" series that is evaluated at precisely the same times as the test series. At this time only linear interpolation is available. An RMS error norm is computed by integrating the square of (good(t)-test(t))/(reltol*good(t)+abstol), dividing by the time interval (or voltage sweep range), and taking the square root. reltol and abstol are described below. If a plot file name is specified, the file will contain lines of data useful for plotting the good and test data against each other, and the error measures. Each line of plot output contains, in order, the time (if a transient sweep), and then the test value, interpolated good value, difference, and relative error for each output variable, in columnar format delimited with spaces.

Specification of tolerances
xyce_verify.pl uses several different tolerances in its operation. Each may be specified independently for every signal present in the netlist. The tolerances, their use, their specification and their defaults are described in this section. IMPORTANT: The tolerances used by xyce_verify.pl are for error norms, and are often very different in use from the similarly-named tolerances in the simulators themselves. It is often completely incorrect to attempt to specify tolerances in xyce_verify.pl that are identical to the similarly-named quantities for the simulator. Specification of tolerances: Tolerances for individual signals are specified using netlist comments. If a line in the netlist starts with the five characters "*COMP", the remainder of the line is parsed as a "comparator tolerance specification." The format of *COMP lines is: *comp <output column name> [ reltol=<relative tolerance> ] [ abstol=<absolute tolerance> ] [ zerotol=<zero tolerance> ] [ absdifftol=<absolute difference tolerance> ] This format was chosen because its specification inside a netlist is interpreted as a comment by the simulators. This allows the netlist to be the full specification of the test without requiring a separate configuration file for tolerances of each test. For example:

     *comp v(1) reltol=0.02 abstol=1e-6
     .print tran v(1) v(2) v(3)

will set abstol to 1e-6 and reltol to 2% for signal v(1), leaving the tolerances for all other signals at their defaults. When run through Xyce(tm) the *COMP line will be ignored. If an expression is used on the print line to output a function of output signals, the precise expression used on the print line must also be used on the "*COMP" line. For example:

     *comp {1.0+v(1)} reltol=0.02 abstol=1e-6
     .print tran {1.0+v(1)} v(2) v(3)

It is a common error to forget this requirement and to use only "v(1)" on the "*COMP" line. "*COMP" lines apply only to signals that match exactly the expression given, and those including print expressions that are not output on the ".print" line are ignored. reltol is the relative tolerance used to compute the RMS error. In a rough sense it is the precision to which the overall signal is held, and reltol should be set to the threshold for failure of the test. It defaults to 1 percent, or 0.01. abstol is used to reduce the significance of errors in quantities that are already small. Its use in the denominator of the relative error term prevents the denominator from becoming too small and inflating the error norm. The default value of abstol is 1e-12. zerotol is the absolute value below which a number should be replaced with 0.0. absdifftol: if two numbers differ by an absolute value of absdifftol or less they are considered to have zero error. The default value of absdifftol is 1e-12.

Use with .STEP netlists
On 7 Oct 2004 a limited support for netlists with ".STEP" analysis statements was added. Please read carefully if you're trying to use xyce_verify.pl to compare runs that use .STEP. The .STEP capability assumes that one has both gold standard and test cases that have been generated with Xyce. No provision is available to compare Xyce output with some other simulator. xyce_verify.pl will work with up to two .STEP statements in a netlist. When run on a .STEP netlist and its output, xyce_verify.pl will split both the good and test input files into separate files, relying on the fact that Xyce will start the "Index" column over at zero for each step. These files are given a name that includes the base name of the input file, the current process ID number, and the string "_split." followed by the step number. It is tacitly assumed that this set of file names will be unique in the current directory and no attempt is made to check that the file already exists. The comparator then checks the ".res" file that contains the step parameters. It requires such a file for both the gold standard and test files. By default it will use the name of the netlist with ".res.gs" appended for the gold standard ".res" file, and the name of the netlist with ".res" appended for the test file. If these are not the correct names to be used, specify alternates with "--goodres=" and "--testres." Once xyce_verify.pl has checked that both the good and test result files contain a line for every step in the sweeps specified in the netlist, it simply loops over the list of split good and test files and compares them as if they were normal Xyce output. All behavior after processing the .res file is identical to processing for non-step netlists. That means that the netlist is subject to the same limitations that would pertain if ".STEP" were not used, including the requirements of the .PRINT line, described below. If a plotfile is specified on the command line, one such file will be produced for each step, using the specified plotfile name as a base name and appending the step number.

LIMITATIONS (read carefully!)
xyce_verify.pl was written to do quite a lot of detailed sanity checking of output, but it does have some rather severe limitations. Read this section carefully before using xyce_verify.pl to check simulation results. xyce_verify.pl assumes that one is using standard format output from Xyce and relies on that format for its sanity checking. It cannot work on PROBE or "NOINDEX" format output files. For transient simulations xyce_verify.pl interpolates the good dataset to the times in the test dataset. This interpolation is only as good as the good dataset: if the good dataset is not very smooth, is evaluated on an inappropriately coarse time mesh, or just isn't very "good," then meaningless comparisons can result. Since xyce_verify.pl does an interpolation of the good data to the test data's times, IT CANNOT BE USED TO CHECK THAT XYCE IS OUTPUTTING AT SPECIFIC TIMES!. If your test is that Xyce is outputting at specific intervals you MUST NOT use xyce_verify.pl to check, because as long as the output signals are the correct shape and interpolation is clean then xyce_verify.pl will declare victory. For DC sweeps the limitations are far more severe. In order to do its sanity checking of output, xyce_verify.pl has to make some assumptions about how netlist commands produce output. 1. If the negative node of a DC source is node 0 (ground), it is expected that the user will print the voltage of the postive node. If the positive node is ground, the user must print the voltage at the negative node. If neither positive node nor negative node is ground, the user must print the voltage drop across the source using the V(pos,neg) syntax. If the output does not conform to this expection a failure will be reported. 2. xyce_verify.pl checks to see that all steps of the requested DC sweeps are present in the output. In order to do its check, xyce_verify.pl expects that the sweep variables are the first items in the .PRINT line. In the case of two-variable sweeps the first column of output should be the faster varying sweep and the second column the slower. xyce_verify.pl checks the .DC line, the .print line, and the output files' headers to make sure this is true, and will exit without doing any comparison if it is not. See also the restriction above. 3. The comparator assumes that only one line of output will appear in the data files for every DC sweep step. Recent changes in Xyce do allow some special cases (PDE devices for example) where more than one line of output is produced for a single sweep step. These will result in xyce_verify.pl reporting failure due to what it considers an inconsistency between what was requested in the netlist and what was produced in the output.

Exit codes
Each error condition detected by xyce_verify.pl is signaled by use of a non-zero exit code. This allows use of xyce_verify.pl inside scripts.

    1: Data file not terminated with valid "end of run" string. 3: could not find interpolation program 4: interpolation program failed to run 20: Failed compare (exceeded tolerances on one or more signal) 30: Failed compare, missing data in test file. 66: unused 79: Assert failure, interpolator didn't produce correct number of interpolated points 85: Sweep values in good data file don't agree with those asked for in .DC line of netlist 86: DC sweep data file has step count inconsistent with .DC line in netlist 87: DC test file contains more steps than "good" file does 88: *COMP line contains unrecognized tolerance 89: DC test file missing steps 90: Failure to determine analysis type (DC or transient) 92: *COMP line found for variable not in the .PRINT line 93: Columns in datafile don't agree with those output by .PRINT line in netlist 94: .PRINT line type (TRAN or DC) in netlist doesn't agree with .DC or .TRAN analysis line 95: .PRINT line for a DC sweep does not contain the voltage of positive node of the swept source 97: End of file encountered before reading any data 98: End of file encountered while parsing SPICE header 99: End of file encountered before reading any non-blank lines

BUGS
Numerous, I'm sure. The relative error may be meaningless when the thing in which there is error is very small --- and no real attempt is made to account for that.

DISCLAIMER
This code was written to help us look more critically at the differences between SPICE and Xyce output. It is not ready to be relied upon for V&V purposes, but is currently being used for Xyce certification testing. Keep out of reach of children. May cause drowsiness, use caution while driving. Discontinue use if rash develops.

ROUTINES CALLED
None. Perl must be installed in the users's path in order for xyce_verify.pl to work.