Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementation of hyperopt model selection #1976

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

Cmurilochem
Copy link
Collaborator

@Cmurilochem Cmurilochem commented Mar 4, 2024

This PR aims to implement #1962 into validphys vp-hyperoptplot.py.

Updates

The model selection following to the min_chi2_max_phi algorithm (refactored from best_chi2_worst_phi2 in #1962) could be done in two ways:

1. via a basic script that imports validphys hyperopt_dataframe

An example is shown below:

from pathlib import Path

from validphys.hyperoptplot import hyperopt_dataframe

hyperopt_name = 'hyperopt_10_rep_chi2_average'

# Get the current directory as a Path object
current_directory = Path.cwd()
path = current_directory / hyperopt_name

args = {
    'debug': True,
    'filter': [],
    'hyperopt_folder': path,
    'loss_target': 'min_chi2_max_phi',       # select Juan & Roy's algorithm
    'max_phi_n_models': 10,                       # select the n lowest values of 1/phi
    'val_multiplier': 0.0,
    'threshold': 3.0,
    'combine': False,
    'autofilter': [],
    'include_failures': False
}

all_data, best_setup, best_models = hyperopt_dataframe(args)

where best_setup corresponds to the model which shows the lowest 1/phi among those with the lowest chi2, while best_models is a pandas DataFrame containing all max_phi_n_models.

2. via vp-hyperoptplot

by running:

vp-hyperoptplot -l min_chi2_max_phi --max_phi_n_models 10 hyperopt_10_rep_chi2_average -t 3

which would launch an html file in your browser with some statistics and plots (I thinks this could be shared within the NNPDF server later on)

Screenshot 2024-03-08 at 16 54 36 (2)

TODO: add a table in the html file containing the detailed specs of best_models.

@Cmurilochem Cmurilochem added validphys enhancement New feature or request labels Mar 8, 2024
@Cmurilochem Cmurilochem force-pushed the dev_hyperopt_model_selection branch 2 times, most recently from d7a3873 to e5294e8 Compare March 12, 2024 16:48
@Cmurilochem Cmurilochem force-pushed the dev_hyperopt_model_selection branch from 21c5a06 to 695c722 Compare May 27, 2024 06:43
@Cmurilochem Cmurilochem force-pushed the dev_hyperopt_model_selection branch 2 times, most recently from 2ea055a to aa61cb7 Compare June 12, 2024 07:12
@scarlehoff
Copy link
Member

Can we review plus merge this now @Radonirinaunimi or do we want to wait for ref. replies?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Automated selection of models via a best_chi2_worse_phi2 algorithm
3 participants