Attempt to generate plausability loss fails using ExplainableBoostingClassifier.

[sabri-manai]

The used dataset is attached for reproducibility:
ready_for_acfx.csv

Description:
The system fails to generate plausability loss after uploading the attached dataset, removing null values, and selecting ExplainableBoostingClassifier.

Logs:

ValueError: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you're on Streamlit Cloud, click on 'Manage app' in the lower right of your app).
Traceback:
File "/mount/src/acfx/src/app/pages/3_Adjacency_Generation.py", line 70, in <module>
    casual_model = train_causal_model(st.session_state.selected_X)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mount/src/acfx/src/app/pages/3_Adjacency_Generation.py", line 12, in train_causal_model
    causal_model.fit(X)
File "/home/adminuser/venv/lib/python3.11/site-packages/lingam/direct_lingam.py", line 76, in fit
    X = check_array(X)
        ^^^^^^^^^^^^^^
File "/home/adminuser/venv/lib/python3.11/site-packages/sklearn/utils/validation.py", line 1055, in check_array
    array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/adminuser/venv/lib/python3.11/site-packages/sklearn/utils/_array_api.py", line 839, in _asarray_with_order
    array = numpy.asarray(array, order=order, dtype=dtype)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/adminuser/venv/lib/python3.11/site-packages/pandas/core/generic.py", line 2070, in __array__
    return np.asarray(self._values, dtype=dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[she3r]

Several issues with the provided dataset.

1. You try to input non-numerical dataset. e.g 'params_normalize_type', which is a string-type column. So you ignored the "Please preprocess your data to meet the classifier's fit() requirements. Without preprocessing, fitting may fail or be in inefficient depending on the classifier." warning in the Data Selection page. You were able to fit to EBM as it supports fitting to string-valued categorical features, but other scikitlearn's blackboxes do not have this ability. (Error while trying to initialize a classifier only with (KNeighborsClassifier, RandomForestClassifier, LogisticRegression) #9).
2. We require categorical values to be numerical (integers) for optuna sampling too - in 'Evaluation settings', we set the bounds for each feature to search over (Error running ExplainableBoostingClassifier #11)
3. With this dataset, it is not possible to calculate directlingam adjacency, the data simply needs preprocessing to numerical values. (Attempt to generate plausability loss fails using ExplainableBoostingClassifier. #10)
   Please find attached preprocessing notebook that will transform the dataset to an acceptable form

ready_for_acfx.csv
ready_for_acfx_data_processing.ipynb

You did however touch upon a subject of how to treat categorical values with generating adjacency and calculating adjacency loss. We've decided to make it more explicit to encorce using proper blackbox with proper data. Additionally, we want the adjacency to make more sense, inputing lingam's adjacency with both categorical and continous data is problematic. On the other hand, the API is not prepared for the transformed categorical data (like OH).

The following changes were made to some things more explicit:

1. If you have non-numerical features (columns) an error message will appear: "non-numerical feature found: params_booster. For any kind of categorical feature, you need to preprocess first"
   2.non-continous features (automatically detected) will is turned off from evaluation by default (but one can still make it turned on)
2. If any non-continous feature is available, only EBM will be available as it is the only blackbox that is prepared for unprocessed categorical data (e.g processed by transforming to OH)
3. For any non-continous features included in dataset, calculating adjacency will be skipped. A message in 'Adjacency Generation' page will appear: "For categorical features, the adjacency is set to 0. These features are skipped for causality calculation as DirectLinGam doesn't support categorical features. Order of categorical features is irrelevant for evaluation"
4. In the CCF method's API, a warning will appear if the adjacency matrix is implying causability of non-continous features, as they are skipped in calculating the causability loss. <- this is still subject to change, as we are still discussing alternative ways to calculate the causality other than lingam.

[sabri-manai]

After using the obtained dataset, running the attached notebook, I am still unable to run it, as it crashes in the evaluation settings step now.

[she3r]

Sorry, but I am not able to reproduce the issue.

1. Did you remove NaNs in the 'Data Selection' step?
2. Can you provide the whole log?
3. Which feature did you choose as the label? Can you provide a minimal amount of features included as available in evaluation in 'Data Selection' step that will allow me to reproduce the issue?
4. If you're testing on the online deployment, please try to reproduce the issue locally using the newest code on 'develop' branch. Maybe the problem you are encountering was handled already.

[sabri-manai]

I was able to get the counterfactuals using this dataset:

acfx_table.csv

However, there is a problem when I try to include the nominal features.
Here are the full logs:

File "/mount/src/acfx/src/app/pages/5_Run.py", line 261, in <module> cfs = acfx.counterfactual( ^^^^^^^^^^^^^^^^^^^^ File "/home/adminuser/venv/lib/python3.11/site-packages/acfx/ACFX.py", line 198, in counterfactual return generate_cfs(query_instance=query_instance, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/adminuser/venv/lib/python3.11/site-packages/acfx/evaluation/ccfs.py", line 360, in generate_cfs cf = _generate_single_cf(query_instance, desired_class, adjacency_matrix, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/adminuser/venv/lib/python3.11/site-packages/acfx/evaluation/ccfs.py", line 224, in _generate_single_cf return __generate_single_cf(query_instance=query_instance, desired_class=desired_class, adjacency_matrix=adjacency_matrix, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/adminuser/venv/lib/python3.11/site-packages/acfx/evaluation/ccfs.py", line 162, in __generate_single_cf sampled_trials = sample_from_model(Xdesired, bounds, desired_class, features_order, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/adminuser/venv/lib/python3.11/site-packages/acfx/evaluation/ccfs.py", line 141, in sample_from_model cf = clip_values(cf, bounds) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/adminuser/venv/lib/python3.11/site-packages/acfx/evaluation/ccfs.py", line 96, in clip_values return {feature: np.clip(value, pbounds[feature][0], pbounds[feature][1]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/adminuser/venv/lib/python3.11/site-packages/acfx/evaluation/ccfs.py", line 96, in <dictcomp> return {feature: np.clip(value, pbounds[feature][0], pbounds[feature][1]) ~~~~~~~^^^^^^^^^

Any idea why this is happening, please?

[she3r]

I think i managed to reproduce the issue. Once initialized, pbounds are not reset appropriately when the the set of features taken for evaluation change (in the 'Data selection' page). The fix consists in resetting the pbounds dictionary keys each time the ‘Evaluation settings’ page is loaded.

[she3r]

I published a fix, please let me know if it helped:
https://pypi.org/manage/project/acfx/release/0.3.5/
Commit:
5b9b3d0