Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Program hanging when input is throttling by maxsize and exception raised in stage #110

Open
chengjinluo opened this issue Sep 12, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@chengjinluo
Copy link

chengjinluo commented Sep 12, 2023

Describe the bug
Please refer to the code example.

After some investigation, I found all process workers exits successfully.
But the initial worker thread which feed the input numbers into queue stuck on the block call of multiprocessing.Queue.put, and it can not get the exception raised by stopit.

Manually edit use_thread default value to False in pypeln.process.worker.start_workers will be a workaround (will start initial worker as a process instead of thread, and it will be terminated by signal).

It seems that pypeln.thread.Queue.IterableQueue has a different implementation, it will try put operation with timeout so that the thread is able to get the exception raised by stopit.

Minimal code to reproduce

import pypeln as pl

def proc(x):
    print(x, flush=True)
    if x == 10:
        assert False

if __name__ == '__main__':
    stage = pl.process.map(proc, list(range(100)), maxsize=4, workers=4)
    list(stage)

Expected behavior
Program exits with exception.

Library Info
os: ubuntu 22.04
python=3.10.12
pypeln=0.4.9

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

@chengjinluo chengjinluo added the bug Something isn't working label Sep 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant