API

The shell script API source is located at src/main/shell and the installation puts it at ${libdir}/$(PACKAGE_NAME)/.

To make the API available in your shell scripts simply source the util shell file in your script:

source /path/to/install/prefix/libdir/submit-scripts/util.sh

Configuration

There are two environment variables controlling the read and write buffer sizes:

export R_BUF_SIZE=${R_BUF_SIZE:-1M}
export W_BUF_SIZE=${W_BUF_SIZE:-1M}

As you can see you can individually set them, if unset they default to 1 MiB.

Bailout Behaviour

You can define bailout with the trap builtin, where bailout is a predefined shell function:

trap 'bailout $LINENO $?' ERR

Since everything works a bit differently with pipes there is also a predefined handler them:

foo | bar | baz | ...
pipe_bailout $LINENO

IO

These are various functions for reading and writing files. The short version is: use read-from and write-to which will detect the file type by themselves and use the appropriate compressors / decompressors.

These functions are intended to be used in pipes or in process substitutions. Some examples:

• the usual way

  app -i input -o output
  

• reading and writing with pipes, using pigz for de/compression

  module load pigz
  read-from input.gz | app | write-to output.gz
  

  this works only if app is able to read from STDIN and write to STDOUT and if no special file operations are performed, e.g. seeking

• using process substitution

  app -i <(read-from input) -o >(output)
  

  this works only if app is able to read from STDIN and write to STDOUT and if no special file operations are performed, e.g. seeking

  process substitution should be applied if there are more than one files to be read or written (STDIN/STDOUT can take care of only one file each), e.g.

  app -i <(read-from input) -i <(read-from input2 -i <(read-from input3) -o >(output)
  

Reading

The following functions are defined for reading files, all sending their output to STDOUT:

# simply reading and dumping to STDOUT (using R_BUF_SIZE as defined above)
read-from-dd

# also filters through gzip
read-from-gzip

# also filters through pigz (possibly parallel)
read-from-pigz

# using a filter if file type matches
# - *.gz using unpigz or gunzip
read-from

Writing

The following functions are defined for writing files, all getting their input from STDIN:

# simply reading from STDIN and dumping to file (using W_BUF_SIZE as defined above)
write-to-dd

# also filters through gzip
write-to-gzip

# also filters through pigz (possibly parallel)
write-to-pigz

# using a filter if file type matches
# - *.gz using pigz or gzip
write-to

Tracing

You can trace your applications with strace. The predifined tracer function takes care of writing the trace output file efficiently and also compresses it, since this output can get quite large for long-term jobs and depends on the amount of syscalls generated. The generated output is then ready to be parsed by the strace-analyzer submit script.

The tracer function is best used only optionally, since it can impact performance quite a bit, controlled by an environment variable in a submit script:

${TRACE:+tracer} app [app-args]