Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow PrimitiveJob to use Python processes rather than threads in _submit. #12989

Open
kshyatt-aws opened this issue Aug 19, 2024 · 2 comments
Open
Labels
mod: primitives Related to the Primitives module type: feature request New feature or request

Comments

@kshyatt-aws
Copy link

What should we add?

In primitive_job.py:

def _submit(self):
        if self._future is not None:
            raise JobError("Primitive job has been submitted already.")

        executor = ThreadPoolExecutor(max_workers=1)  # pylint: disable=consider-using-with
        self._future = executor.submit(self._function, *self._args, **self._kwargs)
        executor.shutdown(wait=False)

Any backend that's using PrimitiveJob is locked into using Python's threading, which can cause locks when interacting with other non-Python languages with their own thread schedulers. Python processes don't have this problem (because they avoid the GIL). This behaviour is currently blocking me when using a custom backend I've written. It'd be great if the PrimitiveJob could obey the preference set in the QISKIT_PARALLEL environment variable.

@kshyatt-aws kshyatt-aws added the type: feature request New feature or request label Aug 19, 2024
@jakelishman jakelishman added the mod: primitives Related to the Primitives module label Aug 19, 2024
@kshyatt-aws
Copy link
Author

kshyatt-aws commented Aug 19, 2024

Another option which might unblock me would be to allow the user to supply an initializer option to the ThreadPoolExecutor

@t-imamichi
Copy link
Member

It makes sense to control the thread pool, e.g., do not use ThreadPoolExecutor if QISKIT_IN_PARALLEL is True.
It is related to #12963. Since PrimitiveJob currently stores the result and status in the ThreadPoolExecutor, we need to store them directly in PrimitiveJob.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
mod: primitives Related to the Primitives module type: feature request New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants