Skip to content

BrianPugh/lox

Folders and files

NameName
Last commit message
Last commit date

Latest commit

24d2fa5 · Jun 12, 2024
Jun 12, 2024
Jun 30, 2019
Apr 7, 2022
Jun 12, 2024
Jun 12, 2024
Apr 7, 2022
May 21, 2019
Jun 12, 2024
Jun 12, 2024
May 21, 2019
Jun 12, 2024
Apr 7, 2022
Mar 11, 2021
Jun 18, 2019
Jul 1, 2021
May 9, 2022
Nov 25, 2020
Dec 18, 2021
Apr 7, 2022
Jun 12, 2024
Jun 12, 2024
Jul 21, 2020

Repository files navigation

assets/lox_200w.png

https://circleci.com/gh/BrianPugh/lox.svg?style=svg Documentation Status

Threading and multiprocessing made easy.

Lox provides decorators and synchronization primitives to quickly add concurrency to your projects.

Installation

pip3 install --user lox

Features

  • Multithreading: Powerful, intuitive multithreading in just 2 additional lines of code.
  • Multiprocessing: Truly parallel function execution with the same interface as multithreading.
  • Synchronization: Advanced thread synchronization, communication, and resource management tools.

Todos

  • All objects except lox.process are for threads. These will eventually be multiprocess friendly.

Usage

Easy Multithreading

>>> import lox
>>>
>>> @lox.thread(4) # Will operate with a maximum of 4 threads
... def foo(x,y):
...     return x*y
>>> foo(3,4) # normal function calls still work
12
>>> for i in range(5):
...     foo.scatter(i, i+1)
-ignore-
>>> # foo is currently being executed in 4 threads
>>> results = foo.gather() # block until results are ready
>>> print(results) # Results are in the same order as scatter() calls
[0, 2, 6, 12, 20]

Or, for example, if you aren't allowed to directly decorate the function you would like multithreaded/multiprocessed, you can just directly invoke the decorator:

>>> # Lets say we don't have direct access to this function
... def foo(x, y):
...     return x * y
...
>>>
>>> def my_func():
...     foo_threaded = lox.thread(foo)
...     for i in range(5):
...         foo_threaded.scatter(i, i + 1)
...     results = foo_threaded.gather()
...     # foo is currently being executed in default 50 thread executor pool
...     return results
...

This also makes it easier to dynamically control the number of thread/processes in the executor pool. The syntax is a little weird, but this is just explicitly invoking a decorator that has optional arguments:

>>> # Set the number of executer threads to 10
>>> foo_threaded = lox.thread(10)(foo)

Easy Multiprocessing

>>> import lox
>>>
>>> @lox.process(4)  # Will operate with a pool of 4 processes
... def foo(x, y):
...     return x * y
...
>>> foo(3, 4)  # normal function calls still work
12
>>> for i in range(5):
...     foo.scatter(i, i + 1)
...
-ignore-
>>> # foo is currently being executed in 4 processes
>>> results = foo.gather()  # block until results are ready
>>> print(results)  # Results are in the same order as scatter() calls
[0, 2, 6, 12, 20]

Progress Bar Support (tqdm)

>>> import lox
>>> from random import random
>>> from time import sleep
>>>
>>> @lox.thread(2)
... def foo(multiplier):
...     sleep(multiplier * random())
...
>>> for i in range(10):
>>>     foo.scatter(i)
>>> results = foo.gather(tqdm=True)
90%|████████████████████████████████▌        | 9/10 [00:03<00:00,  1.32it/s]
100%|███████████████████████████████████████| 10/10 [00:06<00:00,  1.46s/it]