Merge branch 'master' into dev_hyperopt_model_selection

.github/workflows/fitbot.yml

@@ -10,7 +10,7 @@ on:
 env:
   N3FIT_MAXNREP: 20 # total number of replicas to fit
   POSTFIT_NREP: 16 # requested replicas for postfit
-  REFERENCE_SET: NNBOT-c0b68779b-2024-05-16 # reference set for exact results
+  REFERENCE_SET: NNBOT-06f20cf6a-2024-07-26 # reference set for exact results
   STABLE_REFERENCE_SET: NNBOT-c0f99b7b3-2024-02-28 # reference set for last tag
   CONDA_PY: 312
   PYTHONHASHSEED: "0"

conda-recipe/meta.yaml

@@ -19,7 +19,7 @@ requirements:
         - pip
     run:
         - python >=3.9,<3.13
-        - tensorflow >=2.10
+        - tensorflow >=2.10,<2.17 # 2.17 works ok but the conda-forge package for macos doesn't
         - psutil # to ensure n3fit affinity is with the right processors
         - hyperopt
         - mongodb
@@ -36,7 +36,7 @@ requirements:
         - requests
         - prompt_toolkit
         - validobj
-        - pineappl >=0.7.3
+        - pineappl >=0.8.2
         - eko >=0.14.2
         - fiatlux
         - sphinx >=5.0.2,<6 # documentation. Needs pinning temporarily due to markdown

doc/sphinx/source/data/dataset-naming-convention.rst

@@ -28,26 +28,51 @@ constructed following this [Backus–Naur form]::
 Experiments
 ===========
 
+- `ATHENA`: A Totally Hermetic Electron Nucleus Apparatus 
+proposed for IP6 at the Electron-Ion Collider
 - `ATLAS <https://home.cern/science/experiments/atlas>`_: A Large Toroidal
   Aparatus
-- BCDMS: TODO
-- CHORUS: TODO
+- BCDMS: Muon-proton collider
+- `CDF <https://www.fnal.gov/pub/tevatron/experiments/cdf.html>`_: Collider Detector at Fermilab
+- CHORUS: CERN Hybrid Oscillation Research apparatUS
 - `CMS <https://home.cern/science/experiments/cms>`_: Compact Muon Solenoid
-- E605: TODO
-- E866: TODO
-- E906: TODO
-- EMC: TODO
+- `COMPASS <https://home.cern/science/experiments/compass>`_: Common Muon and Proton Apparatus 
+for Structure and Spectroscopy
+- DO: Tevatron collider experiment 
+- `E142 <https://inspirehep.net/experiments/1108817>`_: measurement of the neutron 
+spin dependent structure function at SLAC
+- `E143 <https://inspirehep.net/experiments/1108679>`_: measurement of the nucleon 
+spin structure in the end stattion at SLAC
+- `E154 <https://inspirehep.net/experiments/1108588>`_: successor of E142 
+using a polarized HE3 target
+- `E155 <https://inspirehep.net/experiments/1108587>`_: successor of E143
+- E605: Fixed-target experiment at Fermilab
+- E866: NuSea, also Fermilab.
+- E906: SeaQuest, successor of E866
+- EMC: European Muon Collaboration
+- `H1 <https://h1.desy.de/>`_: HERA experiment
 - `HERA <https://dphep.web.cern.ch/accelerators/hera>`: Hadron Elektron Ring
   Anlage. While technically speaking this is an accelerator, this string is
   used for the combined analyses of H1 and ZEUS.
+- HERMES: HERA experiment on nucleon spin properties
+- `E06-010 <https://hallaweb.jlab.org/experiment/transversity/>`_: JLAB experiment
+- `E97-110 <https://hallaweb.jlab.org/experiment/E97-110/>`_: JLAB experiment
+- EG1b: JLAB experiment from the CLAS collaboration
+- EG1-DVCS: JLAB experiment from the CLAS collaboration
 - `LHCB <https://home.cern/science/experiments/lhcb>`_:
-- NMC: TODO
+- NMC: New Muon Collaboration, successor of EMC.
 - `NNPDF <https://nnpdf.mi.infn.it/>`_: This experiment name is used for two
   purposes:
-
   1. for auxiliary datasets needed in the PDF fit, for instance `INTEG` and `POS`
   2. for predictions used in NNPDF papers to study the impact of PDFs in processes not included in its PDF fit
-- NUTEV: TODO
+- NUTEV: heavy target neutrino detector
+- `PHENIX <https://www.bnl.gov/rhic/phenix.php>`_: Pioneering High Energy Nuclear 
+Interaction eXperiment at RHIC
+- SLAC: Stanford Linear Accelerator Center
+- SMC: Spin Muon Collaboration
+- SMCSX: Low-x Spin Muon Collaboration
+- `STAR <https://www.bnl.gov/rhic/star.php>`_: Solenoidal Tracker at RHIC
+- ZEUS: HERA experiment
 
 
 
@@ -73,8 +98,8 @@ Processes
   production)
 - `WP`: production of a single positively-charged lepton (charged current
   off-shell Drell–Yan)
-- `WPZ`: production of three leptons (WZ-diboson production)
-- `Z0PT`: production of two same-flavor opposite-sign leptons with non-zero
+- `Z0J`: production of two same-flavor opposite-sign leptons with non-zero
   total transverse momentum (Z-boson pt spectrum)
+- `WJ`: same for W (W-boson pt spectrum)
 
 `Backus–Naur form <https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form>`_

doc/sphinx/source/figuresofmerit/index.rst

@@ -155,7 +155,7 @@ HERACOMB_SIGMARED_C dataset to 100 by adding the following to the runcard:
 .. code-block:: yaml
 
     dataset_inputs:
-        - {dataset: HERACOMB_SIGMARED_C, frac: 0.75, weight: 100}
+        - {dataset: HERA_NC_318GEV_EAVG_CHARM-SIGMARED, frac: 0.75, variant: legacy, weight: 100}
 
 
 Experimental, validation, and training 𝜒²

doc/sphinx/source/n3fit/hyperopt.rst

@@ -336,15 +336,15 @@ hyperopt configuration dictionary).
 .. code-block:: yaml
 
         dataset_inputs:
-        - {dataset: NMCPD_dw_ite, frac: 0.75}
-        - {dataset: NMC, frac: 0.75}
-        - {dataset: SLACP_dwsh, frac: 0.75}
-        - {dataset: SLACD_dw_ite, frac: 0.75}
-        - {dataset: BCDMSP_dwsh, frac: 0.75}
-        - {dataset: BCDMSD_dw_ite, frac: 0.75}
-        - {dataset: HERACOMBNCEP575, frac: 0.75}
-        - {dataset: HERACOMBCCEM, frac: 0.75}
-        - {dataset: HERACOMBCCEP, frac: 0.75}
+        - {dataset: NMC_NC_NOTFIXED_DW_EM-F2, frac: 0.75, variant: legacy}
+        - {dataset: NMC_NC_NOTFIXED_P_EM-SIGMARED, frac: 0.75, variant: legacy}
+        - {dataset: SLAC_NC_NOTFIXED_P_DW_EM-F2, frac: 0.75, variant: legacy}
+        - {dataset: SLAC_NC_NOTFIXED_D_DW_EM-F2, frac: 0.75, variant: legacy}
+        - {dataset: BCDMS_NC_NOTFIXED_P_DW_EM-F2, frac: 0.75, variant: legacy}
+        - {dataset: BCDMS_NC_NOTFIXED_D_DW_EM-F2, frac: 0.75, variant: legacy}
+        - {dataset: HERA_NC_251GEV_EP-SIGMARED, frac: 0.75, variant: legacy}
+        - {dataset: HERA_CC_318GEV_EM-SIGMARED, frac: 0.75, variant: legacy}
+        - {dataset: HERA_CC_318GEV_EP-SIGMARED frac: 0.75, variant: legacy}
 
         hyperscan_config:
           use_tries_from: 210508-hyperopt_for_paper
@@ -353,9 +353,9 @@ hyperopt configuration dictionary).
           target: fit_future_tests
           partitions:
           - datasets:
-            - HERACOMBCCEP
-            - HERACOMBCCEM
-            - HERACOMBNCEP575
+            - HERA_CC_318GEV_EP-SIGMARED
+            - HERA_CC_318GEV_EM-SIGMARED
+            - HERA_NC_251GEV_EP-SIGMARED
           - datasets:
 
         parallel_models: true

doc/sphinx/source/n3fit/runcard_detailed.rst

@@ -26,11 +26,11 @@ The first thing one finds when building a fit runcard for
 .. code-block:: yaml
 
     dataset_inputs:
-        - { dataset: SLACP_dwsh, frac: 0.5}
-        - { dataset: NMCPD_dw, frac: 0.5 }
-        - { dataset: ATLASZPT8TEVMDIST, frac: 0.75, sys: 10, cfac: [QCD] }
+        - { dataset: SLAC_NC_NOTFIXED_P_DW_EM-F2, frac: 0.5, variant: legacy}
+        - { dataset: NMC_NC_NOTFIXED_DW_EM-F2, frac: 0.5, variant: legacy }
+        - { dataset: ATLAS_Z0J_8TEV_PT-M, frac: 0.75, variant: legacy_10}
+
 
-        
 The `dataset_inputs` key contains a list of dictionaries defining the datasets
 to be used in the fit as well as their options (which are detailed in :ref:`datasetspec-core-label`).
 
@@ -41,7 +41,7 @@ The fraction of events that are considered for the training and validation sets
 .. code-block:: yaml
 
     dataset_inputs:
-    - { dataset: SLACP_dwsh, frac: 0.75}
+    - { dataset: SLAC_NC_NOTFIXED_P_DW_EM-F2, frac: 0.75, variant: legacy}
   
 It is possible to run a fit with no validation set by setting the fraction to ``1.0``, in this case the training set will be used as validation set.
 
@@ -54,7 +54,7 @@ to fix it such that it is the same for all replicas with ``same_trvl_per_replica
 
     trvlseed: 7
     same_trvl_per_replica: true
-                
+
 
 .. _preprocessing-label:
 
@@ -92,26 +92,26 @@ Setting the ``trainable`` flag to ``False`` is equivalent to recovering the old
             - { fl: t8,  smallx: [0.56,1.29], largex: [1.45,3.03] }
             - { fl: cp,  smallx: [0.12,1.19], largex: [1.83,6.70] }
 
-It  is important to determine the correct values for the ``largex`` and ``smallx`` preprocessing 
-ranges. For example setting a poor range for those parameters can result in a conflict with the 
-:ref:`positivity <positivity>` or :ref:`integrability <integrability>` constraints, making it such 
-that no replicas can satisfy those constraints. In most cases when changes are made to a runcard, 
-they will have a relatively small effect on the required preprocessing ranges. This includes common 
-variations to runcards such as changing the datasets, or settings related to the training of the 
-neural network. In these cases :ref:`running an iterated fit <run-iterated-fit>` is likely the 
+It  is important to determine the correct values for the ``largex`` and ``smallx`` preprocessing
+ranges. For example setting a poor range for those parameters can result in a conflict with the
+:ref:`positivity <positivity>` or :ref:`integrability <integrability>` constraints, making it such
+that no replicas can satisfy those constraints. In most cases when changes are made to a runcard,
+they will have a relatively small effect on the required preprocessing ranges. This includes common
+variations to runcards such as changing the datasets, or settings related to the training of the
+neural network. In these cases :ref:`running an iterated fit <run-iterated-fit>` is likely the
 easiest way to obtain a satisfactory range of the preprocessing. However, in some cases, such as for
-example a change of PDF basis where the preprocessing ranges obtain a different meaning entirely, 
-we don't know what a good starting point for the ranges would be. One way to identify good ranges 
-is by opening up the ``smallx`` and ``large`` parameters for large ranges and setting 
-``trainable: True``. This way the preprocessing exponents will be considered part of the free 
+example a change of PDF basis where the preprocessing ranges obtain a different meaning entirely,
+we don't know what a good starting point for the ranges would be. One way to identify good ranges
+is by opening up the ``smallx`` and ``large`` parameters for large ranges and setting
+``trainable: True``. This way the preprocessing exponents will be considered part of the free
 parameters of the model, and as such they will be fitted by the optimization algorithm.
 
-NNPDF4.0 fits are run with ``trainable: False``, because trainable preprocessing exponents can lead 
+NNPDF4.0 fits are run with ``trainable: False``, because trainable preprocessing exponents can lead
 to an underestimation of the PDF uncertainties in the extrapolation domain. So after determining a
-reasonable range for the preprocessing exponents, a new runcard should be generated using 
-``vp-nextfitruncard`` as explained in :ref:_run-iterated-fit. In this runcard one should then 
-manually set ``trainable: False`` for all preprocessing exponents before running the iterated fit. 
-It can take more than one iteration before the iterated fits have converged to stable values for the 
+reasonable range for the preprocessing exponents, a new runcard should be generated using
+``vp-nextfitruncard`` as explained in :ref:_run-iterated-fit. In this runcard one should then
+manually set ``trainable: False`` for all preprocessing exponents before running the iterated fit.
+It can take more than one iteration before the iterated fits have converged to stable values for the
 preprocessing ranges.
 
 Note that the script ``vp-nextfitruncard`` automatically enforces some constraints
@@ -160,7 +160,7 @@ In this case the ``nodes_per_layer`` parameter represents the nodes each one of
 
 - **One network per flavour** (``layer_type: dense_per_flavour``):
 
-This mode is designed to behave as the methodology for NNPDF before 3.1 where each flavour has a separated identical network. 
+This mode is designed to behave as the methodology for NNPDF before 3.1 where each flavour has a separated identical network.
 
 In this case the ``nodes_per_layer`` parameter represents the nodes each layer of each flavour has. For instance ``[5, 3, 8]`` means that the first step is a list of 8 layers of shape ``(2x5)``, while the second layer is again a list that matches the previous one (i.e., 8 layers) with layers of shape ``(5x3)`` while the last layer has two task. The output of each layer should be one single element (i.e., 8 ``(3x1)`` layers) and then concatenate them all so that the final output of the neural network will be a 8-elements tensor. A report comparing the ``dense`` and ``dense_per_flavour`` architectures can be found  `here <https://vp.nnpdf.science/q6Rm1Q_rTguJwKsLOZFoig==/>`_
 
@@ -219,7 +219,7 @@ Note that by defining the positivity in this way all datasets will share the sam
 
 It is also possible to not define the positivity hyperparameters (or define them only partially).
 In this case ``n3fit`` will set the initial Lagrange multiplier as ``initial`` (default: 1.0)
-while the ``multiplier`` will be such that after the last epoch the final Lagrange multiplier 
+while the ``multiplier`` will be such that after the last epoch the final Lagrange multiplier
 equals the ``maxlambda`` defined for the dataset.
 
 Finally we have the positivity threshold, which is set to ``1e-6`` by default.
@@ -233,11 +233,11 @@ this value, it will be tagged as ``POS_VETO`` and the replica removed from postf
 
 Integrability
 -------------
-Integrability in ``n3fit`` is enforced through a Lagrange multiplier, this is 
-the same basic concept as how positivity is enforced, and therefore the 
-input in the runcard is analogous to the case of positivity where one can 
-apply the integrability contraints through an optional ``integrability`` 
-dictionary as (not that as opposed to positivity, for integrability no 
+Integrability in ``n3fit`` is enforced through a Lagrange multiplier, this is
+the same basic concept as how positivity is enforced, and therefore the
+input in the runcard is analogous to the case of positivity where one can
+apply the integrability contraints through an optional ``integrability``
+dictionary as (not that as opposed to positivity, for integrability no
 threshold value can be set):
 
 .. code-block:: yaml
@@ -307,7 +307,7 @@ Logging details will include the value of the loss for each experiment over time
 the values of the weights of the NN,
 as well as a detailed analysis of the amount of time that TensorFlow spent on each operation.
 
-          
+
 .. _parallel-label:
 
 Running fits in parallel
@@ -325,8 +325,8 @@ this can be done with the `same_trvl_per_replica: true` runcard flag.
 
 In other words, in order to run several replicas in parallel in a machine
 (be it a big CPU or, most likely, a GPU)
-it is necessary to modify the ``n3fit`` runcard by adding the following two
-top-level options:
+it is necessary to modify the ``n3fit`` runcard by adding the following
+top-level option:
 
 .. code-block:: yaml
 
@@ -337,7 +337,7 @@ And then run ``n3fit`` with a replica range to be parallelized
 (in this case from replica 1 to replica 4).
 
 .. code-block:: bash
-  
+
    n3fit runcard.yml 1 -r 4
 
 
@@ -412,8 +412,7 @@ It is however possible to disable them by setting to false the ``sum_rules`` fla
 
     fitting:
       sum_rules: False
-      
+
 
 It is also possible to impose just the valence or the momentum sum rules by using the
 ``VSR`` or ``MSR`` flags, respectively (``True`` is equal to ``All``).
-

doc/sphinx/source/tutorials/polarized_fits.rst

@@ -33,6 +33,42 @@ as a boundary condition while ``n_std`` specifies the shift in terms of the stan
 deviation to be applied to the PDF central values. If ``n_std=0.0`` then the
 PDF central values will be used to constrain their polarized counterparts.
 
+.. note::
+
+   This will also be the PDF set to be used to compute observable predictions
+   during the fit.
+
+Instead, the unpolarized PDF set to be used to compute the `t0` covariance matrix need to be specified under the ``datacuts`` entry:
+
+.. code-block:: yaml
+
+  ...
+  datacuts:
+    t0pdfset: NNPDFpol11_100   # PDF set to generate t0 covmat
+    unpolarized_bc: NNPDF40_nnlo_pch_as_01180
+    q2min: 1.00                # Q2 minimum
+    w2min: 4.00                # W2 minimum
+  ...
+
+And if a theory covariance matrix is included in the fit
+(see :ref:`how to include a theory covariance matrix in a fit <_thcov_tutorial>`
+for more details), then the unpolarized PDF set also needs to be specified in the
+``theorycovmatconfig`` together with the polarized PDF to be used:
+
+.. code-block:: yaml
+
+  # Define the unpolarized PDF set to be used to compute predictions with validphys actions
+  ...
+  theorycovmatconfig:
+    point_prescription: '7 point'
+    theoryids:
+      from_: scale_variation_theories
+    pdf: NNPDFpol11_100
+    unpolarized_bc: NNPDF40_nnlo_pch_as_01180
+    use_thcovmat_in_fitting: true
+    use_thcovmat_in_sampling: true
+  ...
+
 Given that polarized fits require different fitting bases and different theory
 constraints, the fields under the ``fitting`` key require some adjustments.
 Specifically:
@@ -41,7 +77,7 @@ Specifically:
 
   ...
   fitting:
-    fitbasis: POLARIZED_EVOL
+    fitbasis: POLARIZED_EVOL_CMP
     sum_rules: TSR
     basis:
     - {fl: sng, trainable: false, smallx: [1.094, 1.118], largex: [1.46, 3.003]}
@@ -89,8 +125,8 @@ and then followed by the basis type (for example ``EVOL`` or ``FLAVOUR``).
 
    where the subscript :math:`r` indicates that the random seed per replica is always
    different. In practice, when imposing the positivity at the level of PDFs, we enforce
-   the constraints on the flavor combination :math:`\left( \Delta f_i + \Delta \bar{f}_i \right)`,
-   that is :math:`(\Delta) \mathcal{F}_k \equiv (\Delta) f_k + (\Delta) \bar{f}_k`.
+   the constraints on the individual flavor :math:`\Delta f_i` (or respectively :math:`\Delta \bar{f}_i`),
+   that is :math:`(\Delta) \mathcal{F}_k \equiv (\Delta) f_k` (or respectively :math:`(\Delta) \mathcal{F}_k \equiv (\Delta) \bar{f}_k`).
 
 Once the runcard is ready, the user can follow the guidelines :ref:`here <run-n3fit-fits>`
 to set up and run fits.
@@ -106,8 +142,6 @@ can simply be done by supplementing a flag to the ``evolven3fit``:
 
   evolven3fit evolve $runcard_folder --use_polarized
 
-Alternatively, the user can explicitly specify the path to the EKO using the flag ``--load``.
-
 
 Comparing polarized fits
 ------------------------