some questions about hyperparameter spaces

[Xuyike]

Dear authers,
Thank you for your GOOD, which has provided great help to learners. I have some questions about the results in the new leaderboard.
1.Is this result just the experimental result under the OOD validation/OOD test setting?
2.What is the scope of the hyperparameter spaces? I find some results are worse than the performance in the paper with larger hyperparameter spaces.
3.I have a point that I don't know whether it is correct or not? As the hyperparameter spaces continue to grow, the selected model will overfit the valid set data, and the distribution of the valid set and the test set are quite different, so the better the effect of the super-parameter combination on the valid set will lead to the worse performance of the test set.It means that how to select a model is still a very important part in OOD generalization.

Sorry to bother you,Look forward to your reply~Thanks！
Best Regards,
xuyike

[CM-BF]

Hi Xuyike,

Thank you for your questions!

1. Yes, the results are chosen under the OOD validation set, and the posted results are from the OOD test.
2. The scope of the hyperparameter spaces can be found in this config folder. For example, this is the grid hyperparameter searching space of GSAT.
3. I agree with your opinion. How to select a model is indeed a critical problem. But this problem may not be so sever as what you expect. This is because (without training under-fit situations) the selected model can fit well to BOTH training and validation distributions, while training and validation distributions are quite different. Given an assumption that the distribution shifts from training to validation share the same direction (to be more specific, the same spurious features) with the distribution shifts from training to test, the well fitted validation distribution indicates the good model generalization ability on this type of distribution shifts, implying better OOD test resutls.
   However, when this assumption is violated, OOD validation may not act effectively.

Please let me know if any further questions or any suggestions. 😄

Best,
Shurui

[Xuyike]

Dear authers,
Thank you very much for your reply！
By the way, there may be a mistake in the code when I run the Twitch dataset.I am not sure whether this judgment is correct, so I hope to discuss it with you.
First, the twitch dataset is a Binary classification, the loss is calculated with 'loss = config.metric.loss_func(raw_pred, targets, reduction='none') * mask' in BaseOOD.py.However, the dimension returned by this loss function(binary_cross_entropy_with_logits) is [N, 1], and the dimension of mask is [N],so the dimension of loss becomes [N,N]. So far, this may only be a problem of matrix dimension expansion，however，when running loss = loss * node_norm * mask.sum() if config.model.model_level == 'node' else loss,the value in node_norm cannot match each node, which causes an error in GraphSAINTRandomWalkSampler. This is also why the loss is so large in training stage.
Solution:loss = config.metric.loss_func(raw_pred, targets, reduction='none').squeeze() * mask. Add a squeeze().
At the same time, another place needs to be modified,loss: torch.tensor = self.config.metric.loss_func(raw_preds, targets, reduction='none') * mask(basic_pipeline.py/Pipeline/evaluate).
Result:After I modify the code, under the parameter settings in the original paper, the performance of Twitch can be improved.
SET:Twitch / language /covariate

I don't know if I can explain clearly. The above views are only in the task of node classification, excluding graph classification.

Look forward to your reply~Thanks！
Best Regards,
xuyike

[CM-BF]

Hi Xuyike,

Hope you are doing well! I've checked the problem and believe you are right. :) This problem can be solved by reshaping the mask and node_norm to make them the same shape as the targets. Squeezing the loss can cause an additional problem in graph-level binary classification tasks since it will lead to different loss and mask shapes in these tasks. Therefore, I prefer to regard the targets as the anchor and align the mask and node_norm to it, leading to clear and consistent expressions. We have updated the code to correct it, and this update will be reflected in our new leaderboard.

We appreciate your suggestions and contributions! Please let me know if any further questions. :)

Best,
Shurui

[Xuyike]

Hi Shurui,
Thanks to GOOD, it has helped me a lot in my research and brought me great convenience. I hope it will become more and more popular.
Best,
xuyike