About Grid Search on CASH problems

[ajoeajoe]

is it Grid Search can solve CASH problems with NNI , it seems that it is usually used for hyper-parameters optimization, have you guys have finished some revision for Grid Search for solving CASH problems.

about Cash problems can refer to ：#1178

[gaocegege]

Grid Search can be used to solve CASH problems. Maybe the paper https://arxiv.org/pdf/1904.12054.pdf 4.1 section could give you information about it.

Grid search will perform the exhaustive combinatorial search, thus personally I think it does not perform good in this scenario. It seems that SMBO algorithms are more popular than grid search in CASH area.

[QuanluZhang]

Thanks @gaocegege for the response. Yes, Grid Search can be used to solve CASH problems. In CASH problems, candidate algorithms (for selection) usually have different hyper-parameters. Thus, it usually requires a nested search space to formulate a CASH problem. For example,

{
    "models": {
        "_type": "choice",
        "_value": [
            {
                "_name": "SVD",
                "n_components": {
                    "_type": "choice",
                    "_value": [1, 2, 3]
                },
                "n_iter": {
                    "_type": "choice",
                    "_value": [4, 5, 6]
                }
            },
            {
                "_name": "DecisionTreeRegressor",
                "criterion": {
                    "_type": "choice",
                    "_value": ["mse", "friedman_mse", "mae"]
                },
                "min_weight_frac": {
                    "_type": "quniform",
                    "_value": [0, 1, 0.1]
                }
            }
        ]
    }
}

This example means choosing an algorithm from SVD and DecisionTreeRegressor, as well as their hyper-parameters. As @gaocegege said, Grid Search could solve this problem with exhaustive combinatorial search. Note that the current Grid Search in NNI has not supported nested search space yet, but it is easy to support it. We encourage external contribution on this feature.