Agree on a minimum requirement for reproducible analysis

[arm61]

This would probably include:

• data files
• a programmatic (not a plot) description of the fitted SLD profile

[arm61]

https://github.com/reflectivity/reproducibility/blob/master/minimum_requirements.md

[mdoucet]

We should probably define what we mean be reproducible.

We could also establish a Reproducibility Checklist to be submitted as supplemental info.
This is something that exists in other fields (example: https://doi.org/10.31222/osf.io/9sm4x).

Here are a few items to consider:

• The reduced data should be available.
• A reference describing the reduction process should be cited it it exists.
• If custom code was used to extract information, if should also be available.
• Some fitting packages like refl1d produce a summary output (a json file in the case of refl1d) that contains provenance info. This should be included. For instance, although parameter limits are never quoted in publications, they are often used to constrain models to physical solutions. Knowing those limits would help understand the assumptions used and help someone else reproduce the results.
• The supporting data, parameters, and code should be in a repo and tagged with its own DOI to ensure that the version used is available.

[andyfaff]

It's certainly a disappointing aspect of existing literature that many articles don't have the mechanistic code (tools) to reproduce them. Not because I really want to reproduce the analysis per-se, but because I want to stand on their shoulders and use their approaches to analyse data.
For example, I've been interesting in trying to implement Sivia's max entropy methods. How much time would've been saved if that code was readily available?
I'm also interested in characterising polymer brushes, and there have been several papers that implement a model with the tool then not being available. Which means that extra work is required to reimplement a tool that I can then make available to the general community. It's then difficult to publish a paper using/explaining the new tool for various reasons.

[christykinane]

Hi Andy, i know Devinder Sivia, I will just ask him if the code still exists? He is basically retired or nearing retirment, but will see what he says.

[andyfaff]

I intended it more as more of an example, but it would be good to see what was originally used (it was 30 years ago).

[mdoucet]

@andyfaff Exactly. It's not very hard to put our analysis code on github. Plus, it has the added benefit of being easier if I want to use it later or give it to someone else. Giving the version used a DOI might not strictly be necessary I guess. It's more of a way to ensure that you can see what was actually used 10 years later even if the code base has evolved.

[christykinane]

Hi Andy, i have had a reply from Devinder, i have added the sivia, webster paper he mentions. And i will proceed to hunt down my group leader and interogate him about the Slab_fit/refine codes.
Cheers

Christy

"Dear Christy,

Yes, I did write the programs SLAB_FIT and SLAB_REFINE about 30 years ago, and John Webster used them at ISIS a few years later. He probably has a lot more experience of using those programs than I do, because he was trying to do real science with them.

I have attached a paper that John and I published in 1998, which shows the results of analysing some data from Mark Geoghegan with both SLAB_FIT/SLAB_REFINE in section 3 and my VOLFMEM MaxEnt program in section 4 (the later makes use of proprietary code, MEMSYS).

Arwell Hughes has been doing much more sophisticated Bayesian analysis of reflectivity data than I ever did, making use of robust Monte Carlo algorithms that were not practical (in terms of computer speed) when I originally wrote SLAB_FIT/SLAB_REFINE.

Cheers,
Devinder.
Sivia_Webster_1998.pdf

"

[arm61]

This is my favourite paper ^

[andyfaff]

Yes, it's great work. I'm interested in how $\alpha$ is figured out for equation 12. The difficulties are experienced on pg333:

"A simple extension for traditional ... through the use of fuzzy pixels[10]. While we leave the details to Ref[5]."

I would have loved to seen the 'simple' code that was used for the analysis. The details from ref 5 don't include code. Supporting information didn't really get going in 1998 I guess. Meaning that the reader has to reimplement analysis code for themselves.

[andyfaff]

One of my favourite papers is doi:10.1016/j.physb.2004.11.058