[GSOC] hyperopt suggestion service logic update

[shashank-iitbhu]

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2374

Checklist:

• Docs included if any changes are user facing

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tenzen-y]

/area gsoc

[andreyvelich]

Thank you for this @shashank-iitbhu!
I left a few comments

[andreyvelich]

Please keep the same name as for parameter_type

Suggested change

	DISTRIBUTION_UNKNOWN = 0;
	UNKNOWN_DISTRIBUTION = 0;

[tenzen-y]

Suggested change

	DISTRIBUTION_UNKNOWN = 0;
	DISTRIBUTION _UNSPECIFIED = 0;

I would like to select the UNSPECIFIED suffix here.
Please see: https://google.aip.dev/126

[andreyvelich]

Make sense, @tenzen-y should we rename other gRPC parameters to UNSPECIFIED ?

[tenzen-y]

Changing released gRPC, it indicates losing backward compatibility.
So, I would like to keep using the existing API for released protocolbuffers API, @andreyvelich WDYT?

[andreyvelich]

Since these gRPC APIs are not exposed to the end-users, do you still think that we should not change the existing APIs ?
It only affects users who build their own Suggestion service.

[tenzen-y]

Since these gRPC APIs are not exposed to the end-users, do you still think that we should not change the existing APIs ?

Almost correct. Additionally, when users keep using the removed Suggestion Services like the Chocolate Suggestion, users face the same problem.

So, can we collect feedback on the dedicated issue outside of here?

[andreyvelich]

Sure, let's followup on this in the issue, and rename it after few months if we don't get any feedback.
@shashank-iitbhu Please can you create an issue to track it ?

[tenzen-y]

SGTM

[shashank-iitbhu]

Sure, let's followup on this in the issue, and rename it after few months if we don't get any feedback. @shashank-iitbhu Please can you create an issue to track it ?

Sure, I will create a separate issue to track the renaming of other gRPC parameters to UNSPECIFIED.

[andreyvelich]

If parameter is int, why we can't support other distributions like lognormal ?

[shashank-iitbhu]

Distributions like uniform quniform loguniform normal etc return float values. They are designed to sample from a range of values that can take any real number (float), which might not make sense if we're looking for an integer value.
Although we can definitely add support for these distributions when parameter is int also. Should we do this?

[andreyvelich]

@tenzen-y @kubeflow/wg-training-leads @shashank-iitbhu Should we round this float value to int if user wants to use this distribution and int parameter type ?

[tenzen-y]

@tenzen-y @kubeflow/wg-training-leads @shashank-iitbhu Should we round this float value to int if user wants to use this distribution and int parameter type ?

SGTM
Users can specify the double parameter type if they want to compute more exactly.
But, documentation of this restriction for int parameter type would be better.

[shashank-iitbhu]

I followed this article to determine the value of sigma from min and max.
cc @tenzen-y @andreyvelich

[andreyvelich]

Maybe we should add this article to the comments. WDYT @tenzen-y @johnugeorge ?

[tenzen-y]

I do not want to depend on the individual article. Instead of that, it would be better to add an actual mathematical description here as a comment.

[shashank-iitbhu]

@tenzen-y I have added two new parameters, weight_decay and dropout_rate, to the Hyperopt example and passed them to mnist.py, but I haven't used them in the Net class yet in the train and test functions. If you check the logs for this e2e test, the maximum value of the loss metrics is an enormously large number. I can't figure out what I'm missing. Also tested this locally.

[shashank-iitbhu]

@tenzen-y

katib/pkg/suggestion/v1beta1/hyperopt/base_service.py

Lines 265 to 266 in 867c40a

	if param.type == INTEGER:
	assignments.append(Assignment(param.name, int(vals[param.name][0])))

here the float values sampled from the distribution get converted to int for INT parameter type.

[tenzen-y]

Shouldn't we use the fixed value when the min and max are scalers the same as Nevergrad, right?

[shashank-iitbhu]

Yeah we can, but that was an edge case when min and max are not defined in case of nevergrad.

    elif isinstance(param, (p.Log, p.Scalar)):
        if (param.bounds[0][0] is None) or (param.bounds[1][0] is None):
            if isinstance(param, p.Scalar) and not param.integer:
                return hp.lognormal(label=param_name, mu=0, sigma=1)

For example,

    - name: batch_size
      parameterType: int
      feasibleSpace:
        min: "32"
        max: "64"
        distribution: "logNormal"

The above parameter will be sampled out from this graph:

where u=3.8123 and sigma=0.3465 are calculated by putting min=32 and max=64 in our code. and E(X) represents the mean which is 48 in our case.

[tenzen-y]

That makes sense.
In that case, could you address the cases where min or max is not specified, as well as nevergrad?

https://github.com/facebookresearch/nevergrad/blob/a2006e50b068fe598e0f3d7dab9c9bcf6cf97e00/nevergrad/optimization/externalbo.py#L61-L64

[tenzen-y]

@shashank-iitbhu This is still pending.

[shashank-iitbhu]

katib/pkg/webhook/v1beta1/experiment/validator/validator.go

Lines 287 to 290 in 867c40a

	if param.FeasibleSpace.Max == "" && param.FeasibleSpace.Min == "" {
	allErrs = append(allErrs, field.Required(parametersPath.Index(i).Child("feasibleSpace").Child("max"),
	fmt.Sprintf("feasibleSpace.max or feasibleSpace.min must be specified for parameterType: %v", param.ParameterType)))
	}

The webhook validator requires feasibleSpace.max or feasibleSpace.min to be specified.

[tenzen-y]

But when either min or max is empty, this validation does not reject the request, right?
So, shouldn't we implement the special case in the Suggestion Service?

[shashank-iitbhu]

Yes, the validation webhook does not reject the request when either min or max is empty. But I created an example where:

    - name: batch_size
      parameterType: int
      feasibleSpace:
        min: "32"
        distribution: "logNormal"

For this, the experiment is being created but the suggestion service is not sampling out any value hence the trials are not running, though handled this case (when either min or max are not specified) in pkg/suggestion/v1beta1/hyperopt/base_service.py.
Do we need to check experiment_defaults.go file?
https://github.com/kubeflow/katib/blob/867c40a1b0669446c774cd6e770a5b7bbf1eb2f1/pkg/apis/controller/experiments/v1beta1/experiment_defaults.go

[andreyvelich]

Maybe we should add this article to the comments. WDYT @tenzen-y @johnugeorge ?