Structure of repository

[londumas]

The structure of the repository is a but messy, if I can be allowed to say that.
If you want this repository, and this code to live long, it might need a bit of clean-up so that anybody starting to use it, can understand easily how to navigate it and run it.

• The readme looks great, but it would be nice to have the code appear as such. You can look how to do that from the readme of picca, it is the same way as in tickets: https://github.com/igmhub/picca/edit/master/README.md
• The repository structure could be as follows: scripts, bin, py/lyacolore, tutorials, ...

What do you think?
Of course all of that is very subjective.
I think also, it would be nice to add:

• the code to Codacy, in order to clean-up the different formatting, and have it more standard.
  https://app.codacy.com/projects
• add Travis tests, with a simple example

[andreufont]

Hi @londumas - I agree that James will have to do some work on the structure, it will help others to take over after his project.

[andreufont]

Sorry, I wanted to cancel the comment that I was writing, and ended up closing the issue :-)

I wanted to say that I agree with @londumas , once we are happy with the final version of bias/beta tunning we should spend some time working on documentation and code structure. I think the goal should be to have a good public version by the time we put the paper online.

[jfarr03]

Yes I'd definitely agree with this - I'm trying to just keep it working and under control for now, but it will need a tidy up and restructure over the next few months before the paper is up!

[londumas]

@jfarr03 and @TEtourneau , here is what we do for picca:

• bin: executables, e.g. picca_cf.py
• data: results from running the code, e.g. correlation_DR12
• docs: necessary to get the automatic doc (not yet implemented)
• etc: necessary files to run the code, e.g. Lyacolore power spectrum
• py/picca: library, e.g. picca_cf.py
• tutorials: tutorials to run and analyse: e.g. jupyther and scripts
  Does that make sense?

[londumas]

@jfarr03, it would be also useful to add a version, as I have implemented in picca, so that we can tag before sending 100 mocks, and know very well with which version the production was made.

[londumas]

@jfarr03, I would like to test if I can run on my laptop the code, what command line should I send?

[jfarr03]

If you modify the input options in "run_process_colore.sh" appropriately then simply executing that script should do the job. You'd need to download some CoLoRe output onto your laptop first - if you just want a very quick test then the file LyaCoLoRe/example_data/raw_colore_1000/out_srcs_s1_0.fits should do the job.

Let me know if any of the options in that script aren't clear - I haven't used it in a little while as I have a separate script for running it on multiple nodes at NERSC

[londumas]

Would this code run on ant machine? Could you give me the full series of command oine to produce a small bix for the tests? I will set it up.

[jfarr03]

Would you like to run CoLoRe on your machine? You can do but it's not feasible to generate large skewers because of the memory requirements - NERSC or similar is required for that.

Just using ./run_process_colore.sh will run on your machine as long as you change all of the options so that they point to the right places. I can try and set most of it up and then you could just add the paths you want for your output if that would be helpful?

[londumas]

In order to set up the tests, we need to run the code. Even a very small box would do the job. Would that be possible or too much work? And yea it is better if you set up the comnand line.

[jfarr03]

Sure - the file I mentioned before LyaCoLoRe/example_data/raw_colore_1000/out_srcs_s1_0.fits contains 1000 output skewers from CoLoRe, so I think that should do the job.

I guess it depends on exactly what the tests will do. Do you have an idea of this? Then we can determine how many skewers we need etc to test it.

[jfarr03]

There were some issues with running run_process_colore.sh so I made a new script run_process_colore_example.sh which should run very straightforwardly. All you would need to do is edit the first couple of lines to make sure it knows the path to LyaCoLoRe and how many processors to use. It returns a couple of errors for me relating to DLAs at the moment but I'm not sure why - I'll work on that soon.

In the meantime: is this script roughly what you were looking for?

[londumas]

@jfarr03, I do not have the time to have a look at the file. Here is what I think the tests should do:

• run CoLoRe to get a small box (is that possible)
• run LyaCoLoRe to get the spectra and the qso, rand-qso, dla, rand-dla catalogs
• compare the version of the branch of these files to their reference one in master, and see if they are the same.
  What do you think?