When it comes to minimizing costs in a business, what we need is an optimized business model. The right hyperparameter tuning technique impacts the model performance, and the model performing optimally is the one that helps achieve the business objective it's built for. The main techniques of tuning model hyperparameters for optimization are random search, grid search, and Bayesian search.
Grid search systematically explores all combinations within a predefined grid, while random search randomly samples hyperparameters to cover a range of possibilities. An illustrative image is given below.
The different techniques are discussed in Chapter 9 of my book. The chapter's title is exploring optimization techniques for machine learning.
Buy at Amazon: https://a.co/d/iRN1WK5
The very premise of machine learning relies on a form of function optimization, so inputs can be most accurately mapped to expected output. Bayesian optimization is best when the objective function is complex. This process keeps updating the hyperparameter configuration until an optimal point (global minimum) is reached in the search space.
More about Hyperparameter tuning in ML using Bayesian Optimization: https://cloud.google.com/blog/products/ai-machine-learning/hyperparameter-tuning-cloud-machine-learning-engine-using-bayesian-optimization
Note: The 'sonar' dataset is used for classification task. The 'auto-insurance' dataset is used for regression task. Look up the codes directory.