Zmq freerun

[sneakers-the-rat]

Fix: #69
Fix: #96

Builds on: #127

Opening as an early draft just to track progress.

currently at a point where the iterators exhaust and prematurely close the runner

• Separate start/stop and init/deinit - have intermediate state whre nodes stay inited but aren't actively running. also need to keep scheduler and store after failures, so probably move clearing those to the init method.
• iter method to iterate through results
• run method with unlimited n
• run method with fixed n
• run method with iter
• test store clearing on all nodes
• figure out why zmqrunner command node can escape pytest stdout capture

Along the way we resolved a few other things

• Figured out that one source of the zmq thread/async deadlock might be threading locks held within the logging modules when the processes are forked, so ensured that loggers in child processes create their own file and stdout loggers. this ends up being much cleaner than the prior implementation, where we not have one logging file per process rather than one per process per logger, which created an ungodly spray of log files
• along the same lines, we also added a 'ping' loop to the zmq startup, where sometimes (esp on mac in the github runner) the nodes are slow to start listening to events in pub/sub sockets. this loop prompts them to re-identify themselves, which then sends additional announce messages, which then causes them to update their states, which then appropriately releases the "ready" lock in the command node.
• this let us turn off debug logging by default in testing, which was a weird crutch that kept things running on mac in the CI runners...

---

📚 Documentation preview 📚: https://noob--129.org.readthedocs.build/en/129/

[coveralls]

coverage: 87.551% (+1.3%) from 86.203%
when pulling 197fbd0 on zmq-freerun
into 90ed7df on main.

[codspeed-hq]

Merging this PR will not alter performance

✅ 7 untouched benchmarks
⏩ 7 skipped benchmarks¹

---

_{Comparing zmq-freerun (197fbd0) with main (90ed7df)}

Footnotes

1. 7 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[raymondwjang]

is this ok to be not nonlocal? clearly it's working since tests are passing but why

[sneakers-the-rat]

not sure what you mean? this is nonlocal because it's referring to finished which is defined in the outer function scope, and we are modifying it, so it needs to be nonlocal (otherwise assigning to it would just create finished in the inner _done scope.

[raymondwjang]

oh sorry i was asking about work_ready

[sneakers-the-rat]

Aha my b. No thats fine since we dont rebind work_ready