correct sorting of level1 data

[comane]

PR due to issue #1755

[Zaharid]

not that it matters much, but we should be doing this with a dict rather than a list, to avoid quadratic behavior.

[Zaharid]

We could add sort=False to groupby. AFAICT the behaviour is not documented, as keeping the order, so I would not touch the rest in any way.

[comane]

sort=False does not seem to solve the problem

[comane]

Apart from the ordering of the level1 commondata list, the following bug is fixed in the present PR.
The bug is due to experiments_index and can be visualised from the snippet below

from validphys.api import API
from validphys.pseudodata import make_level1_data


inp = {
    "fit": "210623_mnc_disonly_linear_1000",
    "dataset_inputs": {"from_": "fit"},
    "theoryid":200,
    "use_cuts": "internal",
    "fakepdf": "NNPDF40_nnlo_as_01180"
}

data = API.data(**inp)
experiments_index =  API.experiments_index(**inp) # API.data_index(**inp)
level0_commondata_wc = API.level0_commondata_wc(**inp)



replica = make_level1_data(
    data=data, 
    level0_commondata_wc=level0_commondata_wc,
    filterseed=2,
    experiments_index=experiments_index,
    sep_mult=True
)




print("Ratio level 1/ level 0: all exps")
print(replica[4].central_values / level0_commondata_wc[4].central_values)

Gives:

Ratio level 1/ level 0: all exps
entry
1        4.586865
2        4.472092
3        4.856960
4        4.761477
5        4.783476
          ...    
250    205.996344
251    199.764029
252    201.027175
253    221.003710
254    207.598561
Name: data, Length: 248, dtype: float64

[Zaharid]

@comane the diff is unreadable now unfortunately. Could you please make it so only your changes show up?

[comane]

@comane the diff is unreadable now unfortunately. Could you please make it so only your changes show up?

Hi @Zaharid, not sure on how to do that. The diffs appears this way because I had to rebase the branch.
The changed files, however, are: filters.py, pseudodata.py and results.py

[RoyStegeman]

When you opened this PR the additional commits were not yet in master. Perhaps changing the base branch to something else and reverting it to master corrects the diffs in this interface? Just a guess though.

The diffs appears this way because I had to rebase the branch.

Maybe I'm just misinterpreting what you mean, but there is a genuine difference between git rebase and git merge. It looks like you only merged this branch.

[Zaharid]

Indeed the generic approach is something like

git switch <this branch>
git rebase origin/master
git push --force-with-lease

Depending on how creative you have been in the current branch, it may be simpler to start over and git cherry-pick the relevant commits.