5a Clean up Simulation parameters

[vagechirkov]

Hey, I think it makes sense to refactor the simulation parameters a bit because at the moment I don't understand how I can define and manipulate them. My main problem is that there are several places where parameters are defined (e.g., .env file, playgroundtool, cs_playgroundtool) and it's not easy for me to manipulate the default parameters. I think the problem got worse when we have multiple projects with different .env files. It seems to me that this issue has already been addressed in this PR #48. But I believe we can do a major refactoring in a separate PR (or multiple PRs). @mezdahun let me know what you think and how you think I can proceed with this issue.

[mezdahun]

Hi There, yes I fully agree with you! We should do this indeed. As the env files are currently necessary for the HPC experiments and this should not be changed due to our limited possibilities of deploying the software in singularity containers, I would go somehow along the line of:

• keeping the SW to start and read env variables from a central .env file from the root folder for all subprojects. The behavior should be that no matter which subproject I run, no matter if playground-tool or simulation, or headless, it is uses the current .env framework. This should be under the hood and the user should not worry about changing/deleting/replacing .env files
• let's build a separate unified submodule in the stack e.g. "config" that has the following functionalities: providing a higher level interface for the user to set default variables to the playground tool and the simulations, describes these parameters and translates them to .env variables and files.
• We can use e.g. yaml or json files to create this higher level config language and collect all .env related functionalities there too.
• Upon starting the SW we only call functions of this submodule that takes care of .env files, variables and initialization.

This was the idea with the playground tool default_params thing as well, but unfortunately I didn't really have the capacity to finish that much, so this higher level translation of env parameters is mixed with reading env parameters as they are which is super messy :(

What do you think, do you have maybe another suggestion? I am really open to that until we can keep the central .env file.

[vagechirkov]

I would suggest using a great Pydantic library (https://pydantic-docs.helpmanual.io/) to handle the settings. I don't think this contradicts the requirement to keep the .env file at the root of the project. See also this post with more examples for the Pydantic BaseSettings class: https://towardsdatascience.com/how-to-manage-configurations-like-a-boss-34e224f3bb4

I can open a PR with a new configuration and we can continue working on it in that PR.

[mezdahun]

It looks awesome from a first quick 👀. If it allows us to clean up this mess and keep the .env framework in place it would be a prefect solution. Do you have experience with it?

[vagechirkov]

Yep, I have some experience with Pydantic and I think this is a great tool that can help clean up settings.