Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Polish analysis of a multiclosure test #1982

Open
wants to merge 61 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
9f2f9d4
Init analysis
andreab1997 Feb 20, 2024
be93df4
Add other functions
andreab1997 Feb 20, 2024
c041049
Add other functions again
andreab1997 Feb 20, 2024
b6abb53
Init analysis
andreab1997 Feb 20, 2024
c580a9a
Add other functions
andreab1997 Feb 20, 2024
d3c99b2
Add other functions again
andreab1997 Feb 20, 2024
82a9ff8
Remove ipdbs and make sure multiclosure_inconsistent work
comane Jun 10, 2024
8d635b4
Add multiclosure comparefits
andreab1997 Feb 22, 2024
0b6acae
Add sklearn to pyproject
andreab1997 Feb 26, 2024
e6936db
Scikit added to conda
comane Jun 10, 2024
baa99a3
added PCA bias variance example to validphys2/examples
comane Mar 5, 2024
a739964
l2 condition number plot
comane Mar 5, 2024
72ad203
added l2 condition number plot
comane Mar 5, 2024
1909114
removed unused functions pca_covmat, dataset_fits_gaussian_parameters
comane Mar 5, 2024
66f293f
updated multiclosure analysis with l2 plots
comane Mar 5, 2024
90c553a
variance pdf has to be taken from the same fits
comane Mar 5, 2024
d245451
use np.nan and set default EVR to 0.99
comane Mar 5, 2024
d09018a
cleaned bias variance table
comane Mar 5, 2024
d9b94b8
added docs to func + removed sklearn preprocessing as unused
comane Mar 5, 2024
acf3f91
added single point dataset to example report
comane Mar 5, 2024
2de09b8
added bootstrap of bias distribution
comane Mar 11, 2024
325ad8f
added rbv plots as function of lambda
comane Mar 11, 2024
95d608e
added ratio bias variance to multi closure report
comane Mar 12, 2024
070e3f6
forgot ratio_bias_variance template
comane Mar 12, 2024
7079292
fmt = 'o' for ratio bias variance plot
comane Mar 12, 2024
49d0484
added functions for computing Rbv using old definition + runcard with…
comane Mar 15, 2024
9d5f63e
Piazza pulita 1: removed old plotting functions that were never used
comane Mar 17, 2024
3644cb8
Piazza pulita 2: removed principal_components_variance_distribution_d…
comane Mar 17, 2024
462e63d
moved single data colorbar plots to multiclosure.py/_output.py
giovannidecrescenzo Mar 17, 2024
7d696af
scikit-dep >=1.4.1 in pyproject file
comane Apr 11, 2024
37b835c
use mean theory covmat for PCA -> gives more stable values of variance
comane Apr 22, 2024
8b4d5c5
rewrote functions using new variance definition (covmat computed by a…
comane Jun 10, 2024
5606f3e
removed sklearn dep from conda recipe meta file
comane Apr 23, 2024
ee8bf09
removed variancepdf as unused
comane Jun 10, 2024
7144ac9
added check_multifit_replicas check
comane Jun 10, 2024
ce5ff2d
use plotting dataset labels for rbv vs lambda titles
comane Jun 10, 2024
328a892
added hlines for rbv = 1
comane Jun 10, 2024
1f4eacc
added bootstrapped_internal_multiclosure_dataset_loader for tuple of …
comane Jun 11, 2024
8689a18
bootstrap of PCA regularised multiclosure tests
comane Jun 11, 2024
2690bc9
bootstrap for internal_data_loadeer objecgs
comane Jun 11, 2024
cc77ed9
added bootstrapped_principal_components_bias_variance_dataset
comane Jun 12, 2024
493e769
added bootstrap table to report
comane Jun 12, 2024
fb4fb4e
changed defaults of bootstrap
comane Jun 12, 2024
b154273
added title for single data point in latex mode
Jun 13, 2024
a1d6a08
Add PCA on corr matrix on full dataset
andreab1997 Jun 18, 2024
d1808ec
fixed inconsistency with single data point and suggest different way …
giovannidecrescenzo Jun 19, 2024
7c20ec8
added delta plots
giovannidecrescenzo Jun 19, 2024
6f2a8ef
use consistent bootstrap def and separate table datasets from table data
comane Jun 21, 2024
75773d4
fixed PCA of correlation matrix for full dataset
comane Jun 21, 2024
cb61d22
compute rbv scan using bootstrap uncertainty quantification
comane Jun 21, 2024
07b16a6
added full data bootstrapped table
comane Jun 25, 2024
099af95
slight change in def of delta hist
Jun 26, 2024
89398ef
added definition of delta in line with eq. 2.22
comane Jun 26, 2024
8d71a2e
added definition of delta in line with eq. 2.22
comane Jun 26, 2024
8df13b2
added rbv scan for full dataset
comane Jun 26, 2024
defcd96
added bootstrapped xi indicator function for full dataset
comane Jun 27, 2024
e03c29e
removed unused import from vp_multiclosure.py script
comane Jul 24, 2024
5b03cf4
removed new lines
comane Jul 24, 2024
ddc3c0d
removed unused variables
comane Jul 24, 2024
e4fe438
_covmats as array instead of list of arrays
comane Jul 24, 2024
7cfdd9a
added eigendecomposition function
comane Jul 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ wiki-upload = "validphys.scripts.wiki_upload:main"
vp-get = "validphys.scripts.vp_get:main"
vp-comparefits = "validphys.scripts.vp_comparefits:main"
vp-fitrename = "validphys.scripts.vp_fitrename:main"
vp-multiclosure = "validphys.scripts.vp_multiclosure:main"
vp-checktheory = "validphys.scripts.vp_checktheory:main"
vp-pdfrename = "validphys.scripts.vp_pdfrename:main"
vp-pdffromreplicas = "validphys.scripts.vp_pdffromreplicas:main"
Expand Down
65 changes: 65 additions & 0 deletions validphys2/examples/pca_bias_variance_report.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
meta:
title: PCA report for bias and variance of multiclosure fits.
author: Mark N. Costantini
keywords: [multiclosure, PCA, bias, variance]


dataset_inputs:
- {dataset: ATLAS_SINGLETOP_7TEV_TCHANNEL-XSEC, variant: legacy}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you using the legacy versions? Is that necessary?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fits that I use in that runcard were done using th 200. I think this is the reason, why I added the legacy version

- {dataset: HERA_NC_225GEV_EP-SIGMARED, variant: legacy}
- {dataset: HERA_NC_251GEV_EP-SIGMARED, variant: legacy}
- {dataset: HERA_NC_300GEV_EP-SIGMARED, variant: legacy}
- {dataset: HERA_NC_318GEV_EP-SIGMARED, variant: legacy}
- {dataset: HERA_CC_318GEV_EM-SIGMARED, variant: legacy}
- {dataset: HERA_CC_318GEV_EP-SIGMARED, variant: legacy}



theoryid: 200
use_cuts: internal
use_t0: True
t0pdfset: 210223-mw-000_fakepdf


explained_variance_ratio: 0.99

##### Bootstrap parameters
n_fit_max: 8
n_fit: 8

n_rep_max: 100
n_rep: 60

n_boot_multiclosure: 100
rng_seed_mct_boot: 1


fits:
- 25_5_2023_19_47_5_dis_pt1_mnc_commit_4d5d473c_filterseed_415295
- 25_5_2023_19_49_38_dis_pt1_mnc_commit_4d5d473c_filterseed_120750
- 25_5_2023_19_50_27_dis_pt1_mnc_commit_4d5d473c_filterseed_754299
- 25_5_2023_19_51_19_dis_pt1_mnc_commit_4d5d473c_filterseed_361864
- 25_5_2023_19_52_54_dis_pt1_mnc_commit_4d5d473c_filterseed_516909
- 25_5_2023_19_52_7_dis_pt1_mnc_commit_4d5d473c_filterseed_539528
- 25_5_2023_19_53_41_dis_pt1_mnc_commit_4d5d473c_filterseed_307907
- 25_5_2023_19_54_28_dis_pt1_mnc_commit_4d5d473c_filterseed_279829


template_text: |

PCA report for bias and variance of multiclosure fits.
------------------------------------------------------

## Table of bias and variance
{@table_bias_variance_datasets@}


## Bootstrapped table for full dataset
{@bootstrapped_table_bias_variance_data@}

## L2 condition number
{@plot_l2_condition_number@}


actions_:
- report(main=true)
2 changes: 2 additions & 0 deletions validphys2/src/validphys/closuretest/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,5 @@
from validphys.closuretest.multiclosure_pdf_output import *
from validphys.closuretest.multiclosure_preprocessing import *
from validphys.closuretest.multiclosure_pseudodata import *
from validphys.closuretest.inconsistent_closuretest.multiclosure_inconsistent import *
from validphys.closuretest.inconsistent_closuretest.multiclosure_inconsistent_output import *
Loading