Struggles with afterscripts

[Cesar48]

Hi,

I'm currently working with gourd and I tried to do some postprocessing, specifically through afterscripts (though I could rewrite it to use the postprocessing workflow, or next according to #32, but it seems that would require me to wait for the mentioned documentation).
Currently I've downgraded my script to be exactly what it says in the documentation:

gourd.toml:

[program.program_name]
binary = "/path/to/binary/pumpkin-cli"
arguments = ["several", "different", "flags"]
afterscript = "./script.sh"

script.sh:

echo $1 > $2

However, upon running, I (repeatedly) get the following error message:

warn: No output found for afterscript of run #1: caused by: Could not run the afterscript at "/full/correct/path/to/script.sh" with job results at "/full/correct/path/to/experiments/44/0/1/stdout"
 |
 |  help: Check that the afterscript is correct and job results exist at "/full/correct/path/to/experiments/44/0/1/stdout"
 |
 |  Check that the afterscript correctly places the output in a file.

What am I missing? Or, perhaps more precisely: is something missing in the documentation? Or did I simply read it wrong?

[andtsa]

Thanks for pointing this out! The issue is with the error message, this is addressed in #27.

Try running chmod +x ./script.sh

Please let me know if it works, or if there's any additional steps needed! We'll add them to the help message of this error

[Cesar48]

Script.sh already had rwxrwxrwx permissions. Just in case I ran the command anyways, but it did not seem to change anything

[andtsa]

@Cesar48 apologies for the delay! After some debugging and trying to reproduce your issue I came to the conclusion that debugging afterscript issues like yours is unnecessarily difficult, and called for a rework. The idea is stdout and stderr of the afterscript execution are both captured, which previously was not the case. this also means that afterscript output can be instead printed to stdout, which I hope will be easier to work with. Reading the job output is still from a file, which is now the $1 and only argument passed to the afterscript.

If you have some time, please try out the new version (#41), and let me know if there are any issues you encountered there! We are getting very close to releasing v1.2.0, so most things should be stable on #41.

[Cesar48]

EDIT: I know the reason for some things described below.

Currently, gourd expects the afterscript to be an executable (binary), rather than a bash script. This is not something I assumed, though I guess it does make it much more versatile, so that's nice.

Futhermore I've answered one of my own questions: it does report that output expected in: "/full/path/to/gourd/gourd/experiments/afterscripts", but the folder appears to be empty. This is somewhat worrying, as the current afterscript is cat (for testing purposes), and my stdout only shows the gourd status update....

Though with this information I might be able to get everything to work nonetheless; if gourd expects an executable, I'll try to give its older version one, and see if that works just as nicely.

EDIT2: I've found the output. It's now stored in the [run].lock file

(End of edit)

Thanks! However, it seems I'm currently getting a different error, still preventing me from making progress:

error: Exec format error (os error 8)
caused by: Could not spawn child cd "/full/path/to/gourd/gourd/experiments/49/0/0" && "/full/path/to/gourd/gourd/script.sh" "/full/path/to/gourd/gourd/experiments/49/0/0/stdout"

My first search told me this might be because I'm using WSL (though it did seem to say only WSL1 has the issue; I'm using 2), but whether this is indeed the case will require further research. (EDIT: see above, I misinterpreted the error message; thought it meant that the executor encountered a format error (instead of exec file format))

Some minor additional questions: is the output still captured into files at any point? I saw (in code) that it did save the output into a variable, but is it also written into a file at some point? Or will I have to manually extract the relevant parts of a local run, and extract the remote results from the slurm stdout?

[andtsa]

Currently, gourd expects the afterscript to be an executable (binary), rather than a bash script. This is not something I assumed, though I guess it does make it much more versatile, so that's nice.

on unix it can still accept shell scripts if they begin with a shebang (#!/bin/sh). A few months ago we were considering automatically detecting (missing) shebangs and giving appropriate responses (either working around their absence or giving a different error message), but this was postponed as it is not trivial to do completely and correctly.

Futhermore I've answered one of my own questions: it does report that output expected in: "/full/path/to/gourd/gourd/experiments/afterscripts", but the folder appears to be empty. This is somewhat worrying, as the current afterscript is cat (for testing purposes), and my stdout only shows the gourd status update....

that error must be updated, thanks for pointing out. I didn't quite understand "and my stdout only shows the gourd status update....", can you elaborate?

Though with this information I might be able to get everything to work nonetheless; if gourd expects an executable, I'll try to give its older version one, and see if that works just as nicely.

if you do so before we fix this version, could you maybe give some suggestions on which parts where easy to work with and should be kept and which not?

EDIT2: I've found the output. It's now stored in the [run].lock file

this is intended to avoid creating one file for every run on delft blue's slow file system. do you think you'd need explicit access to the stdout/stderr in an easier to reach place?

error: Exec format error (os error 8)
caused by: Could not spawn child cd "/full/path/to/gourd/gourd/experiments/49/0/0" && "/full/path/to/gourd/gourd/script.sh" "/full/path/to/gourd/gourd/experiments/49/0/0/stdout"

if still possible, could you share the surrounding prints so I can add an explanation that goes with this error?

[Cesar48]

on unix it can still accept shell scripts if they begin with a shebang (#!/bin/sh)

I missed this mainly because the tutorial mentions a bash script containing only the cat $1 > $2 command, NOT the shebang. To be honest I've hardly ever needed files with a shebang up to this point in my life (usually more high-level software) and as such it was nog the first thing that came to mind. Maybe this would be good to update in the tutorial, or maybe even add as a warning in case running fails.

I didn't quite understand "and my stdout only shows the gourd status update....", can you elaborate?

Upon running gourd status, I expected to be able to see the afterscript results somewhere, either directly or as a file location (generally you'll run them without access to the stdout, though I suppose these results might be saved to the gourd_{slurm_id}.out file, are they?). Since I only saw the 'normal' output, I was wondering where the afterscript results were.

if you do so before we fix this version, could you maybe give some suggestions on which parts where easy to work with and should be kept and which not?

Honestly for me specifically it was just the missing shebang. You could make more explicit that it needs to be an executable and give some (comprehensive) examples, but I don't have much more feedback than that

this is intended to avoid creating one file for every run on delft blue's slow file system. do you think you'd need explicit access to the stdout/stderr in an easier to reach place?

Maybe the best way would be to get the option to print the output somewhere. I think it is a very suitable solution to save it in the lock file, as it keeps all the data in the same place and saved I/O, but doing so makes it much less accessible to anyone using the "regular" CLI. I think gourd status uses other data from the lock file as well right? Maybe add an additional argument/command to show the afterscript output?

if still possible, could you share the surrounding prints so I can add an explanation that goes with this error?

That's the entire output. The line before is me running gourd status -i {run} and the line after is a new prompt