-
Notifications
You must be signed in to change notification settings - Fork 51
fix: redownload weights only on PytorchStreamReader error #686
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: redownload weights only on PytorchStreamReader error #686
Conversation
I think this is still a problem. Interrupting on the first start causes error ".... is a directory" |
@maciejmajek I added one minor naming fix (didn't affect anything). It works on my side, maybe @jmatejcz could check. |
src/rai_extensions/rai_open_set_vision/rai_open_set_vision/agents/base_vision_agent.py
Outdated
Show resolved
Hide resolved
src/rai_extensions/rai_open_set_vision/rai_open_set_vision/agents/base_vision_agent.py
Outdated
Show resolved
Hide resolved
@pawel-kotowski, we looked at #678 during Wednesday's dev sync meeting. I was added as a reviewer since I was seeing the CUDA error somewhat frequently when running the RAI manipulation demo a few months ago. Your changes look solid overall, good work! In the PR description, you mentioned still experiencing CUDA errors even with your fixes - is that correct? I haven't been able to reproduce the error with your changes. If you're still seeing the issue, could you run the attached cuda_error_debug.py to help gather diagnostic information? |
@maciejmajek, how can we reproduce the issue you're experiencing? I tried interrupting the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested again and works! Thank you!
dd1f53e
to
fa8a153
Compare
Purpose
Proposed Changes
RuntimeError
regardless of error type) inBaseVisionAgent
classIssues
Testing
CUDA_VISIBLE_DEVICES