`hyperopt` fails when run in parallel on macOS

[CaptSolo]

When annif hyperopt is run on macOS in parallel it fails with the following error message:

annif hyperopt nllsh-mllm-en --trials 2 --jobs 2 txt-validation-en

[...]

[W 2026-01-01 19:07:17,060] Trial 1 failed with parameters: {'min_samples_leaf': 29, 'max_leaf_nodes': 1402, 'max_samples': 0.7878340014898446} because of the following error: TypeError("'NoneType' object is not subscriptable").
Traceback (most recent call last):
  File "/Users/captsolo/Documents/Code/LNB_Annif_1.4/.venv/lib/python3.12/site-packages/optuna/study/_optimize.py", line 201, in _run_trial
    value_or_values = func(trial)
                      ^^^^^^^^^^^
  File "/Users/captsolo/Documents/Code/LNB_Annif_1.4/.venv/lib/python3.12/site-packages/annif/backend/hyperopt.py", line 67, in _objective_wrapper
    return cls.objective(trial, cls.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/captsolo/Documents/Code/LNB_Annif_1.4/.venv/lib/python3.12/site-packages/annif/backend/mllm.py", line 59, in objective
    model.fit(args["train_x"], args["train_y"])
              ~~~~^^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable

Full error message:

• annif_error_msg.log

Related:

• https://groups.google.com/g/annif-users/c/b3twYAn33mg

[CaptSolo]

Here is the analysis of this problem using Claude Code:

The problem is with how the multiprocessing pool is initialized. The init method is called in the main process (line 133 in hyperopt.py), but when worker processes are spawned, they don't inherit the class variable args because each process has its own memory space.

The problem is in Annif/annif/backend/hyperopt.py:143-152

When the multiprocessing pool is created, it doesn't pass an initializer function to set up the worker processes. This means that while objective_args is prepared and init() is called in the main process (line 133), the worker processes don't have access to cls.args.

I will check if its suggested changes fix the problem.

[CaptSolo]

Changing line 144 in annif/backend/hyperopt.py from:

with pool_class(jobs) as pool:

... to:
with pool_class(jobs, initializer=self._objective.init, initargs=(objective_args,)) as pool:

... fixed the issue (on macOS). I hope it does not break Annif for other operating systems.

[osma]

Hi @CaptSolo , thanks for reporting the issue.

The analysis is correct; this happens because on macOS, the multiprocessing module uses spawn instead of fork as the default start method for worker processes. The Annif code implicitly relies on fork semantics - that worker subprocesses will inherit global state, including static variables such as cls.args in this case. This is very efficient when those data structures are large, which is often the case with Annif backends. That's why we have implemented this solution (see also BaseWorker in parallel.py). But it is apparently a bit fragile. And now I see (on the multiprocessing page linked above) that on Python 3.14+, the default start method has been changed from fork to forkserver also for Linux, which I think would break this functionality, so we may have to do something in the near future for Linux as well...

The fix that you suggested should work, but I think that it may hurt performance and possibly increase overall memory usage because the initialization will now have to be done separately for each worker. There is another way to fix the problem; change MP_START_METHOD to "fork" in parallel.py and see if that helps.

I think it would be helpful if you could do some benchmarking on macOS to compare the different solutions in terms of total RAM usage and performance (total wall clock time & CPU time). I would like to know the relative performance of running a significant sized hyperopt job (a hundred or more trials) using:

1. single-process (--jobs 1)
2. multi-process (--jobs equal to your number of CPU cores) using your initializer fix
3. multi-process using MP_START_METHOD="fork"

Can you try these and report back the results? Then we would have a better idea of how to fix this, not just for macOS, but potentially other OSes as well.