Issues in the Evaluation

[liztrg]

I have been trying to replicate the results from this repository following the tutorial on a Windows, yet have been running into many issues when running the evaluation code. I've deployed a Windows vm on Azure and only installed the conda version for python 3.8, so I wouldnt have conflicts with other versions of conda, python, etc, i have installed on my actual pc. Anyone has any tips for the getting the replication to work?

[Yura52]

Hi! I don't have experience running this codebase on Windows, but I can try to help if you describe the problems in more details and provide error messages.

[liztrg]

Thank you! I have been able to replicate the Resnet, FT-T, MLP, and XGBoost results for the California Housing and Adult datasets. I don't have a GPU available, so I'm running on a CPU. The results are quite close, except for the XGBoost model for the CH dataset. I have followed the same procedure as for the Adult dataset, and the tutorial in the readme file, yet the results are quite different. I haven't been able to pin point yet the source of the difference. Maybe you have some insight?

[Yura52]

The first hypothesis that comes to my mind is the "tree_method" parameter of XGBoost (example). When you switch to CPU, you are likely to use a different value for this parameter (since "gpu_hist" is a GPU-specific value), and this affects the actual algorithm that is building the decision trees.

P.S. In the paper, I don't remember using the "CH" dataset (see Table 1).