RUNTEST

RUNTEST

Low-level Test Cases

(This is an edited copy of the runtest README which exists in the Hercules 'tests' sub-directory)

 

The tests directory contains test cases that are run by Hercules console commands. High-level test cases are run under an operating system, for example OS/j.

Test cases are run after Hyperion has been built, but before it is installed. make check runs the entire regression test reference.

With current Hyperion as of 2016-06-06, there are no longer any special considerations for testing crypto.

A test case is run stand-alone under hercules from a script specified with the -r flag. The script first sets up a program in core storage either by the r console command or by loading a core image by loadcore. It then issues the runtest command to start Hercules by a restart interrupt, the restart new PSW having been set up in the previous step. The script regains control when all CPUs have entered the stopped state. The remainder of the script inspects storage, registers, etc., to display data that are later inspected. The session log is then processed to check the results against with known good values.

You can create test cases in several ways:

• Entirely by hand using only a text editor. Many test cases have been built this way, but this usage has been superseded by use of an assembler that can generate core images directly.

• By using SATK's ASMA, which is an assembler that implements a subset of HLASM's functionality, but has the ability to generate a core image directly. Thus no post processing is required. Typically, you will write by hand a .tst file to load the .core file and do whatever is required.

• By a real /360 assembler, such as IFOX or HLASM. The object deck is converted to r console commands or a core file. This offers the full assembly language with a powerful macro capability, but at the expense of some post processing.

Validating the test case (specifying the desired results; the oracle in testing parlance) is the most difficult step if you do not have access to z/CMS. There may be ambiguities in the PoO or, perhaps more likely, you may misinterpret the manual. As IBM do not publish the suite the developers use to validate a new CPU model, you will need to find a friend who has access to "real iron". This real iron could be z/PDT.

There are two routes to obtaining an object module and validation script for z/CMS: From an ASSEMBLE file using an assembler or from a .tst file by the cvttst.rexx utility (very much still in the development phase).

By convention, the following file type extensions are used.

.tst

Files that have a tst extension are to be run by the runtest or runtest.cmd shell script. They are run and the Hercules console output is then inspected for correctness by an automated rexx script. You can create .tst files either "by hand", with the ASMA (SATK); or by assembly on a /360 system (VM, MVS, DOS), but then TEXT decks need to be processed into .tst files.

.subtst

Files that have a subtst extension are secondary helper scripts called by a primary tst script. They are used when more than one primary tst script consists of code that is virtually identical to another. Rather than duplicate the same code in multiple tst files, the common code is placed into a subtst file instead and "called" by each of the primary tst scripts (via the script command). Think of them as tst script "subroutines".

.assemble

Source files for test cases that are designed for assembly with HLASM. On z/VM with HLASM, the test cases can be prepared by BLDHTC EXEC (bldhtc.exec) on VM or by bldhtc.rexx on the workstation. ASMTST XEDIT (asmtst.xedit) is a sample macro to perform the build step from within the XEDIT session.

On VM/370, you can use the system assembler to assemble, but you will likely do the conversion on the workstation.

http://vm.marist.edu/~pipeline/zops.copy contains a stacked copy file for many new operation codes.

.listing

An assembler listing file produced by HLASM.

.asm

Source files for the .tst test cases that are designed for assembly by ASMA (Harold Grovesteen's SATK), https://github.com/s390guy/SATK . To clone: git@github.com:s390guy/SATK.git

When instructed accordingly, the assembler will produce a core image file (.core) that can be loaded with the loadcore console command.

This makefile snippet can automate the process. You must modify it with the location of the SATK and Hyperion build directories in your system.

.SUFFIXES: .text .tst .assemble .rexx .asm .core
USD:=/usr/data/src
APATH:=${USD}/SATK/tools

%.core: %.asm
   ASMPATH=. ${APATH}/asma.py -l $*.list -i $@ $<

While ASMA so obviously is superior to hand assembly, it does have a few limitations relative to HLASM that you should be aware of:

• Always check the documentation of the assembler instructions you intend to use to be sure they are indeed implemented. Notable omissions are AMODE and RMODE.

• ASMA does not support literal pools; you must collect your literals yourself and give them appropriate names. (This applies to programs that are converted from HLASM to ASMA.)

• No floating point constants. D'...' assembles as if FD'...' were specified, which is correct only for D'0'. Use hexadecimal constants to assemble floating point constants. The file dc-float.asm contains, in no particular order, a number of constants suitable for cut-and-paste. This file was generated from the HLASM listing file from an assembly of the float constants.

• Input is variable length ASCII. Continuation is specified by a backslash at the end of a line. The alternate format for macro instructions is not supported.

.list

An assembler listing file produced by ASMA.

.txt

Files that have a txt extension are manually executed test scripts invoked via the Hercules script command. Their output must be manually eyeballed to ensure they executed correctly. Usually they end by loading a disabled wait PSW. If the PSW's instruction address is 0 then the test case likely completed successfully.

.core

Files that have the extension .core are binary core image files loaded into main storage by the Hercules loadcore console command. They can be created by many means; one of them is ASMA as described above.

"runtest" shell script

The runtest UNIX shell script (runtest.cmd batch file on Windows) is used to run one or more (usually all) automated .tst scripts to verify proper Hercules functionality.

On UNIX, assuming that your are building outside the source tree, perhaps in the "atom" directory, the source directory might be ../hyperion. Use the following command to run and verify the .tst test cases:

  ../hyperion/tests/runtest

make check in the directory where you built Hercules will find stuff wherever you configured it.

On Windows the batch file runtest.cmd runs the test cases. Its first few parameters are identical (almost) to the parameters accepted by the UNIX runtest shell script, but supports many additional parameters as well. Enter runtest /? or --help for more information.

In either case, runtest first creates a composite set of test cases (allTests.testin in the current working directory) and then invokes Hercules to run them using the configuration file tests.conf and the rc file just created. When it completes, the console output (allTests.out) is then inspected by redtest.rexx and the results are reported to the user.

The runtest script return code is the number of failed test cases. A return code of 0 indicates success (no failed test cases). Any other return code indicates failure (one or more failed test cases).

The tail end of an early sample output is:

Test SSKE S/390 nextlast.   3 OK compares.   All pass.
Test SSKE S/390 lastpage.   3 OK compares.   All pass.
Test timeout.   2 OK compares.   All pass.
Done 138 tests.   All OK.
[/usr/data/src/hercules/atom]

The Linux runtest shell script accepts these flags:

-d

Specify the path to the test directory. The default is ${dirname $0}, perhaps ../hyperion/tests

-e

Specify the default extension for the input test files. tst is the default. The -e flag is active for all -f flags until the next -e flag if any.

-f

Specify the input file name perhaps including an extension. The default is -f *.tst to get the current behaviour. Multiple -f flags are supported. The default extension is appended if the word does not contain a period.

-h

Specify the object directory that contains the hercules executable. The default is parent of -d if there is no hercules executable in the current directory (that is, in-source build).

-l

Specify the name of a module to load dynamically into hercules. This is passed to Hercules.

-p

Specify the library path for loadable modules, i.e., device managers. This is passed to Hercules. Nota bene: Specifying this flag will prevent the crypto module from being loaded.

-q

Pass quiet to redtest.rexx to suppress details about test cases.

-r

Repeat the composite test script n times. There is no need to specify -x as hercules terminates without an explicit quit when in daemon mode.

-t

Passed to hercules (test timeout factor)

-v

Set variable for redtest.rexx (in fact, arguments to redtest.rexx). Use -v ptrsize=4 when running a 32 bit executable on an 64 bit system if no valid config.h can be found. You can also set private variables this way (but such variables will be unset when running make check). It is unspecified whether case is respected in variable names. -v quiet will make redtest.rexx more terse.

-w

Specify the work file name. The default is allTests.

-x

Do not append the quit command to the composite test script.

.tst file runtest command

The test a .tst script defines is normally begun via a special runtest script command which supports optional arguments indicating how the test should be started as well as the maximum amount of time it should run. Unlike the pause command, the runtest command waits only until all CPUs are in the stopped state.

  runtest   [restart|start|<oldpsw>]   [timeout]   [# comment...]

  <oldpsw> ::= external | svc | program | machine | io

The first argument is optional. If specified, it must be either restart, start, or an interrupt type. It specifies how the test should be started: via the Hercules restart console command or the start console command. The default is to start the test via the restart command. When an interrupt type is specified, the old psw for that type is copied to the restart new PSW before the restart command is issued to fire up the test case where it left off taking an interrupt for which the new PSW was disabled.

The second argument, which is also optional, defines the maximum amount of time (specified in a whole or fractional number of seconds) that the test is allowed to run. This is a safety feature to prevent runaway tests from running forever due to a bug in the test or within Hercules itself. Under normal circumstances all tests should end automatically the very moment they are done. If no timeout is specified, the default is 30 seconds.

Please also note that timeout values are multiplied by the test timeout factor value which is specified by the -t switch on the Hercules command. The test timeout factor value allows for running tests on systems that may be slower than the system they were originally designed for. Refer to the documentation describing Hercules's supported command line arguments.

The runtest script command is almost always immediately followed by Hercules console commands to display storage or registers, etc, as well as several of the below .tst file directives to verify the test case ran correctly.

.tst file directives

.tst files are executed as Hercules rc files, so they contain Hercules console commands. A number of loud comments (beginning with an asterisk) are interpretted by redtest.rexx as directives. All directives are case sensitive. Using *testcase for example accomplishes nothing.

*           Comment; ignored.  There must be a blank after the
            asterisk.

*Testcase   Identify beginning of a test case.

*Compare    An r command follows and the output should be stored for
            later comparison.

*Want       The expected r command output.  A string identifying the
            comparison may be enclosed in double quotes before the
            compare data.

*Gpr        A previous gpr console command should have been issued to
            display the general registers.  The specified register is
            compared with the hexadecimal data specified after the
            register number.  Specify a decimal register number with no
            leading zero.  The comment "#address" may be specified to
            notify the CMS test driver that the register contains a
            storage address.

*Key        Compare the specified key against the key displayed in the
            previous r command.

*Prefix     Compare the specified value to the contents of the prefix
            register displayed by a previous pr command.

*Program    A program check is expected, for example to verify that a
            privileged operation causes an 0c2.

*Hmsg       Specify the complete informational message string to
            compare against the last issued message, except for
            messages that are processed by redtest.rexx, plus messages
            7 and 1603.  A number is optional before the message
            identifier to specify the last but one, last but two, etc,
            see example below.

*Info       Synonym for *Hmsg.  Use if an informational message is
            expected.

*Error      Synonym for *Hmsg.  Use if an error message is expected.

*Explain    Specify  the explanatory text to write when the next
            comparison fails.  You can specify as many *Explains as you
            like for multiline messages.  The explain array is cleared
            once a test has been done, whether OK or not.

*Message    Print a message on the output.

*If         Specify a condition to test (see later).  The expression is
            evaluated by the REXX interpreter.   If the condition holds, the
            following directives are processed up to the matching *Else
            or *Fi.   If the condition does not hold, directives are
            ignored except that nested *If directives are parsed.

*Else       Flip the setting of the matching *If condition, unless
            nested in a suppressed *If.

*Fi         End an *If.   The process/ignore setting for directives is
            unstacked.

*Done       End of a test case.  Specify "nowait" if the test case does
            not end by loading a disabled PSW, for example, when it
            tests just console commands without running any code.

*Timeout    A timeout is expected.  Do not use.

logicimm.tst shows examples of all directives, except *Key, *Prefix, and *Program; see sske.tst for an example of *Key and *Program. See pr.tst for *Prefix. See mainsize.tst for conditional processing.

Predefined variables are specified as arguments to redtest.rexx and extracted from the features message, HHC01417I. When the runtest script (not to be confused with the runtest command above) finds config.h, it extracts the size of a pointer and sets the variable ptrsize to 4 or 8. For any facilities listed by message HHC01417I, a variable named by the facility name is set to 1, for example, cmpxchg1=1. Variables are specified in the expression by a dollar sign followed by the variable name; they are read only.

Example of conditional directives:

mainsize 2147483648b
*If $ptrsize = 4
*Error 1 HHC01430S Error in function configure_storage(2G): Cannot allocate memory
*Error   HHC02388E Configure storage error -1
*Else # 64 bit
*Info HHC17003I MAIN     storage is 2G (mainsize); storage is not locked
*Fi

Creating test files that can run on z/CMS as well as under Hercules

Running a test case on z/CMS is accomplished by DOTEST EXEC. It merges the object deck for the test case with a pipeline program that interprets the runtest directives, which are extracted from the object deck.

A test case to run on z/CMS can be generated in two ways:

1.    From assembler source.  The object deck is uploaded to z/CMS and
      used directly.  The Hercules (bare iron) test case is created from
      the object deck by bldhtc.rexx.

2.    From a hand-built tst script.  The test script is converted to a
      TEXT deck by cvttst.rexx.

There are a number of differences when a test case is run on CMS rather than on the bare iron:

1.    The machine is not all yours.  Well, it *is*, but you may not wish
      to exercise that prerogative.  You'd likely want CMS to remain
      after running the test case.

2.    GPR15 is the base register for an emulated main storage the size
      of the program (see agf.assemble).  The test case is entered with
      the return address in GPR14.

3.    The test case must not modify GPR11 (or it must save and restore
      it).

4.    Likewise, the test case must not modify GPR14 (or it must save and
      restore it).

5.    The *Gpr directive to verify an address must add the comment
      "#address" so that the value can be rebased.

The following applies to tst scripts that are generated from assembler source:

6.    When the script is finished, it should first check whether GPR14
      is nonzero and branch back rather than load a disabled PSW.

The following applies when cvttst.rexx generates an object deck from a hand-built tst script:

7.    Loading a disabled PSW is replaced by BR 14, so there is no reason
      to write dual path code in hand-built tst scripts.

8.    Loading control register 0 is suppressed, but the driver notes the
      fact that the AFPR enable bit is on and turns it on for the
      duration of the test.

9.    The test program must not reference storage beyond the highest
      address specified in any r console command (either loading data or
      displaying it).  The highest address is set as the control section
      size.

Here is an example of how to code a dual-path return in an assembler program:

 ltr 14,14     Have a return address?
 bnzr 14       return if so
 LPSWE WAITPSW     Load wait PSW
WAITPSW dc x'0002000180000000',ad(0) OK wait state PSW

Some programs contain a series of tests where the CPU is stopped after each test. This should be emulated by a branch back using GPR14, having stored the resume address in the SVC old PSW. This applies only to tests that are assembled from assembler source; cvttst.rexx knows how to deal with loading a disabled PSW.

swap equ * entry by bas 9,swap
 ltr 14,14  Running on CMS?
 jz bare
* Under z/CMS in 31-bit mode.  Simulate a SVC 255
 la 10,0(,9)                  Strip amode bit
 st 10,svcold+12
 basr 14,14                   Return to driver
 br 9                         Return to caller
bare equ *
 svc 255  We are in problem state
 br 9

The test cases are TEXT decks on CMS. When the .tst script is built from assembler source, the object deck is a by product; it should be uploaded to CMS without change. The test case is loaded at a page boundary by CMS.

The REXX script cvttst.rexx builds an object deck from a .tst script that is built by hand (notably the ieee floating point test cases). This process makes several changes to the program:

1.    References to the PSA are rebased to use GPR15 as a base register.
      For LA, this is done only when the displacement is above the first
      instruction in the program (which is obtained from the restart new
      PSW field).

2.    A LPSW[E] for a disabled wait is replaced by BR 14 to return.

3.    LCTL to enable advanced floating point is replaced by a no
      operation and a directive is emitted to have the test driver
      enable this flag and disable it again after the tests have all
      been run.

4.    The addresses in the various new PSW fields in the PSA are marked
      as relocatable when the PSW is not a disabled wait.  This will
      relocate the restart new PSW and make sure that the test case
      fires up the right place.  For other new PSWs this is of little
      practical application since the real interrupts will be fielded by
      CMS.

5.    Distinguishing between an instruction and a constant is in general
      an untractable problem, but it is assumed that there is one "r"
      command for each instruction and for each constant.  Clearly, a
      lenght that is not one of 2, 4, 6 must refer to a constant.  If
      the first two bits of the data are not congruent with the length
      of the area, it must be a constant.  Further, data where the first
      byte is either '00'x or 'ff'x is also a constant.

      This leaves negative short precision constants.  If the displacement
      part of the value is less than '200'x it is taken as a constant.
      I'll likely have to invent better algorithms.

cvttst is still very much experimental.