ENH: Give informative error when Edge filters select no inputs

[mateuszpawlik]

I used the example model to try execute FitLins on our data. I only renamed the trial_type values from word to w and from pseudoword to pw. Unforutnately I missed it in one place, in the last Edges item. FitLins broke with the following crash report.

Node: fitlins_wf.loader
Working directory: /work/fitlins_wf/loader

Node inputs:

database_path = /work/dbcache
model = {'Name': 'words_pseudowords', 'BIDSModelVersion': '1.0.0', 'Description': '', 'Input': {'task': 'lexdec'}, 'Nodes': [{'Level': 'run', 'Name': 'subject', 'GroupBy': ['subject'], 'Transformations': {'Transformer': 'pybids-transforms-v1', 'Instructions': [{'Name': 'Factor', 'Input': ['trial_type']}, {'Name': 'Convolve', 'Input': ['trial_type.w', 'trial_type.pw'], 'Model': 'spm'}]}, 'Model': {'X': ['trial_type.w', 'trial_type.pw', 'framewise_displacement', 'trans_x', 'trans_y', 'trans_z', 'rot_x', 'rot_y', 'rot_z', 'a_comp_cor_00', 'a_comp_cor_01', 'a_comp_cor_02', 'a_comp_cor_03', 'a_comp_cor_04', 'a_comp_cor_05', 1]}, 'DummyContrasts': {'Conditions': ['trial_type.w', 'trial_type.pw'], 'Test': 't'}, 'Contrasts': [{'Name': 'word_gt_pseudo', 'ConditionList': ['trial_type.w', 'trial_type.pw'], 'Weights': [1, -1], 'Test': 't'}, {'Name': 'task_vs_baseline', 'ConditionList': ['trial_type.w', 'trial_type.pw'], 'Weights': [0.5, 0.5], 'Test': 't'}]}, {'Level': 'dataset', 'Name': 't-test', 'GroupBy': ['contrast'], 'Model': {'X': [1], 'Type': 'glm'}, 'DummyContrasts': {'Test': 't'}}, {'Level': 'dataset', 'Name': 'F-test', 'GroupBy': [], 'Model': {'X': ['trial_type.w', 'trial_type.pw'], 'Type': 'glm'}, 'Contrasts': [{'Name': 'any_words', 'ConditionList': ['trial_type.w', 'trial_type.pw'], 'Weights': [[1, 0], [0, 1]], 'Test': 'F'}]}], 'Edges': [{'Source': 'subject', 'Destination': 't-test', 'Filter': {'contrast': ['task_vs_baseline', 'word_gt_pseudo']}}, {'Source': 'subject', 'Destination': 'F-test', 'Filter': {'contrast': ['trial_type.word', 'trial_type.pseudoword']}}]}
selectors = {'desc': 'preproc', 'space': 'MNI152NLin2009cAsym'}

Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 521, in run
    result = self._run_interface(execute=True)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 639, in _run_interface
    return self._run_command(execute)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node loader.

Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run
    runtime = self._run_interface(runtime)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/fitlins/interfaces/bids.py", line 250, in _run_interface
    self._results['all_specs'] = self._load_graph(runtime, graph)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/fitlins/interfaces/bids.py", line 284, in _load_graph
    self._load_graph(runtime, graph, child.destination, outputs, **child.filter)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/fitlins/interfaces/bids.py", line 258, in _load_graph
    specs = node.run(inputs, group_by=node.group_by, **filters)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/bids/modeling/statsmodels.py", line 459, in run
    node_output = BIDSStatsModelsNodeOutput(
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/bids/modeling/statsmodels.py", line 576, in __init__
    df = reduce(merge_dfs, dfs)
TypeError: reduce() of empty sequence with no initial value

After renaming the remaining trial_type values, FitLins worked. Though, it took me hours to figure it out because the error has not indicated the real problem whatsoever.

It would be nice to validate the model json file and report issues like this in a user friendly way.

[adelavega]

Thanks. I agree, unfortunately a lot of these runtime errors yield very cryptic error messages.

We're working on a BIDS StatsModel validator that would try to catch more of these issues upfront.
That said, variable names are always tough because it requires us to fully predict the inputs/outputs of each node.

[effigies]

We are planning a dry-run mode of FitLins that would allow us at least to detect this during pipeline creation time. But I agree that this particular error is going to be hard to detect. A good one to try to solve, though.

Actually, another read-through, and this shouldn't be as hard as all that. It's should be something like:

try:
    specs = node.run(inputs, group_by=node.group_by, **filters)
except TypeError as e:
    if "empty sequence" in str(e):
        raise ValueError(f"Unable to initialize node - probably bad filters {filters}") from e
    raise

[adelavega]

@effigies I think it would be useful to also build this into the validator: https://github.com/bids-standard/bids-stats-model-schema
That is, instead of just checking the structure of their model, also variables etc...

Could fitlins run the validator prior to execution, to prevent duplicate error checking?

[effigies]

We can definitely validate the schema, and if it's possible to validate the edge filters to determine that their values appear in the model outputs, cool. I just doubt it's possible.

PyBIDS might be able to raise more informative errors directly, though.

[adelavega]

Yep, agree. That said, we might want to consider adding "warnings" to the validator that can guess at potential issues like this, or maybe it's not as hard as we think to pre-compute the output names of variables (e.g. if there's no transformations, esp at the first level)