-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
automatically remove parallelization files #285
Comments
Would also probably make sense to update the .gitignore file so that even if the files get left around they wouldn't be committed in a git environment |
Picking up this thread again. I would also prefer having these files deleted as part of the clean up of the model directory. IF there are users who would really like these files to not be deleted, could there be a global option in bbi where this can be turned on or off (default setting should then be that the files are removed). Thoughts @seth127 ? |
Thanks for getting this going again @jacobleander. I think this is the right idea:
We currently have a |
Also, @kyleam if we decide to do this through @jacobleander if you or anyone else have any thoughts on that ^, feel free to comment on that issue as well. The original intention of the |
Under --parallel, NONMEM creates several WK_* files. These are worker files created for parallelization problems and are used as file buffers. The NONMEM Users Guide Introduction to NONMEM 7.5.0 (pg 73) says One can alternatively assess empirically whether file buffers are used, by beginning the run, allowing perhaps one iteration to transpire, then from another command window do a directory search for FILE*, (or WK* for worker files in parallelization problems, section I.72 Parallel Computing (NM72) If any of the FILExx do not have 0 size, then they are being used. Interrupt the analysis, then increase the appropriate LIM value with the $SIZES record [...] So, a non-empty file is a signal that the user probably wants to make adjustments so that everything fits into memory. An empty file, on the other hand, is safe for us to clean up [*]. Due to the regexp match and the extra size condition, this doesn't work nicely as part of getCleanableFileList(). filesToCleanup() already has special handling for some parallelization, so add the WK handling there. [*] NONMEM cleans up FILE* buffer files after a run. I'm not sure why it doesn't also clean up WK*. Re: #285
I've put up a draft at gh-306. There's an issue with the CI that needs to be sorted out before considering merging that in.
Conceptually I think that makes sense, but there's already a spot in
With gh-306, I focused just on the |
Under --parallel, NONMEM creates several WK_* files. These are worker files created for parallelization problems and are used as file buffers. The NONMEM Users Guide Introduction to NONMEM 7.5.0 (pg 73) says One can alternatively assess empirically whether file buffers are used, by beginning the run, allowing perhaps one iteration to transpire, then from another command window do a directory search for FILE*, (or WK* for worker files in parallelization problems, section I.72 Parallel Computing (NM72) If any of the FILExx do not have 0 size, then they are being used. Interrupt the analysis, then increase the appropriate LIM value with the $SIZES record [...] So, a non-empty file is a signal that the user probably wants to make adjustments so that everything fits into memory. An empty file, on the other hand, is safe for us to clean up [*]. Due to the regexp match and the extra size condition, this doesn't work nicely as part of getCleanableFileList(). filesToCleanup() already has special handling for some parallelization, so add the WK handling there. [*] NONMEM cleans up FILE* buffer files after a run. I'm not sure why it doesn't also clean up WK*. Re: #285
With the merge of gh-306,
As I said above, I'm not sure about what, if anything, to do with those. I'll wait a bit for further feedback and thoughts on that, but my preference would be to close this issue and either 1) leave the |
Thanks @kyleam
I agree with closing this issue now, and I would lean towards just opening a fresh one to consider what to do with the |
When running a NONMEM model in parallel, many folders with the prefix WK_ are produced. It would be nice if bbi automatically removed these files. There are also files produced after every run with the extension
.o#
and.po#
where the # is the number assigned by the grid. I think these could also be automatically handled by bbi to prevent buildup of junk files in version control when saving a whole model directory.The text was updated successfully, but these errors were encountered: