Plans for aiida-lammps

[chrisjsewell]

This morning I met with @JPchico and co from Sandvik Coromant, and we discussed what the plans are for moving forward on this plugin.
Here are the key points we want to address:

• The current Calculation/Workflow structure is too constrictive to harness the full capabilities of LAMMPS and enact "real-world" workflows.

  • @JPchico and co will put together some "exemplar" workflows that they would like to achieve, together with the LAMMPS input files that would be required, then we can start to think about what AiiDA components could best achieve these.
  • pyiron also has some LAMMPS based workflows that could be looked at for inspiration
  • Incorporating the new aiida-core BaseRestartWorkchain would also be desirable
  • One issue, not generally seen in DFT, is that of stochastic outputs. How should these be dealt with by provenance tracking / caching? e.g. you may not necessarily want to re-use outputs from a single calculation (which could have non-deterministic outputs) but rather re-use the outputs from a workflow that averages outputs from a number of calculations

• We would like the plugin to be easier to use for "non-experts"

  • Having integration with potential libraries, such as OpenKIM
  • "Protocols" similar to those in https://github.com/aiidateam/aiida-common-workflows/
  • Looking at building AiiDAlab applets

• We would like to have wider buy-in by the LAMMPS/MD community

  • Advertise on the LAMMPS mailing list (probably after tackling some of the points above, so that it is in good shape for people to try)

[abelcarreras]

Hi @chrisjsewell, thanks for inform about the future plans of the repository

[ltalirz]

Just to mention that @RaffaelaCabriolu may also have a use case for the aiida-lammps plugin (for computing the diffusion coefficient)

One issue, not generally seen in DFT, is that of stochastic outputs. How should these be dealt with by provenance tracking / caching? e.g. you may not necessarily want to re-use outputs from a single calculation (which could have non-deterministic outputs) but rather re-use the outputs from a workflow that averages outputs from a number of calculations

For provenance tracking I don't see an issue.
For caching I can see the issue - if the randomness (except for computational randomness related to numerical precision) comes from a random number generator, I think it's a good practice to include the seeds of the RNG in the input (at which point caching works again).
However, even if the seed is not provided and the input is identical, the caching control is already very configurable, i.e. you can turn it off per calculation class or even temporarily using a context manager.

[JPchico]

Just to mention that @RaffaelaCabriolu may also have a use case for the aiida-lammps plugin (for computing the diffusion coefficient)

This is fantastic! The more people the better.

One issue, not generally seen in DFT, is that of stochastic outputs. How should these be dealt with by provenance tracking / caching? e.g. you may not necessarily want to re-use outputs from a single calculation (which could have non-deterministic outputs) but rather re-use the outputs from a workflow that averages outputs from a number of calculations

For provenance tracking I don't see an issue.
For caching I can see the issue - if the randomness (except for computational randomness) comes from a random number generator, I think it's a good practice to include the seeds of the RNG in the input (at which point caching works again).

I would need to check if this is indeed possible in lammps. Normally for this kind of stochastic noise one is able to pass in the seed.

However, even if the seed is not provided and the input is identical, the caching control is already very configurable, i.e. you can turn it off per calculation class or even temporarily using a context manager.

In the case of not having the seed, would this not cause that one is attaching an output which is strictly speaking not the result from the inputs, i.e. equal except for the effects of stochastic noise. In theory this should be okay for most quantities, but there might be a case for which shit is not the case.

[mjclarke94]

However, even if the seed is not provided and the input is identical...

When computing across multiple processors/nodes, simply providing the seed may not be sufficient. Take this example from the delete_atoms command:

For style porosity a specified fraction of atoms are deleted within the specified region. For example, if fraction is 0.1, then 10% of the atoms will be deleted. The atoms to delete are chosen randomly. There is no guarantee that the exact fraction of atoms will be deleted, or that the same atoms will be deleted when running on different numbers of processors.

From what I can tell, how the domain is decomposed will vary not only based on the number of cores available but also the number of cores per node which will vary on a per machine basis. The same will be true for potentially more influential random processes like initialisation of velocities. You can manually define how the system should be decomposed, but that seems like a good way to kill off any remaining "non-experts"!

TLDR: seeding in lammps is occasionally reassuring but pointless.

[JPchico]

From what I can tell, how the domain is decomposed will vary not only based on the number of cores available but also the number of cores per node which will vary on a per machine basis. The same will be true for potentially more influential random processes like initialisation of velocities. You can manually define how the system should be decomposed, but that seems like a good way to kill off any remaining "non-experts"!

Those are very good points. If one needs to set the system decomposition by hand will make it very hard to use for most users, which is of course less than ideal.

This means than caching is for sure an issue, as even when one specifies the seed there is no guarantee that the outputs correspond to the inputs given. Of course in principle the certain average quantities should not be "too far off" even if others such as trajectories can be quite different.

[RaffaelaCabriolu]

Hi all, and thanks to Leopold for introducing me to the conversation! Yes, for us would be very interesting to use LAMMPS in Aiida and we can provide a test case with examples to calculate the zero loading diffusion coefficients. I am quite busy with some deadlines up to the end of next week. After that, I will get back to the chat and to you later to see how this is proceeding. Thanks to all.

[ltalirz]

On the seeding/reproducibility discussion:

There are at least two fundamentally different aspects of "randomness" here and we should distinguish between them.

1. Generating statistically independent results via different starting conditions (e.g. to sample some statistical distribution). If the seed of the RNG is not part of the input, then caching these calculations can lead to problems: uncached calculations would result in statistically independent results, while reusing the cached calculation always gives the same result. It is therefore useful to be able to specify the RNG seed in the input of the AiiDA plugin.

2. Numerical values of results depending on aspects like parallelization, compiler, CPU architecture, ...
   Here, randomness is a side-effect of the computational environment, but cannot be guaranteed a priori.
   I see no problem with caching here, even if in reality the calculation might give a different result from the cached one.

[mjclarke94]

Assuming that there is going to be some analysis-workflows component to the common-workflows, having some wrapper around the trajectories which allows them to be interpreted as aiida-core Trajectory objects might be useful. The compression trick you've used is brilliant, but would likely make data generated in aiida-lammps incompatible with generic trajectory analysis workflows.

[jan-janssen]

@chrisjsewell As you already mention pyiron this might also be the point to think about a generic interface to make simulation codes available in various simulation frameworks at the same time, similar to optimade for atomic databases. With the majority of simulation codes being available as conda-forge packages - at least for Linux - we already achieved the first step. It is now easier than ever to test a simulation code like abinit, cp2k, LAMMPS, quantum espresso or siesta. The next steps from my perspective is defining an interface which is reasonable for multiple simulation frameworks. Such a universal interface could handle the conversion of units and the parsing of input and output files while each framework internally can use a different user interface, data storage and job management.

[chrisjsewell]

can not we develop a universal parser

ah well this relates to why I linked https://molssi-qc-schema.readthedocs.io/; I'd say there should not be a universal parser, there should be a universal interface, e.g. input/output JSON schemas, which any parser would implement

[chrisjsewell]

I have to admit I still struggle to get an overview of the different activities, still I am very happy to see the wide adoption of conda-forge.

and indeed thanks for your work on the conda-forge recipes!

[jan-janssen]

ah well this relates to why I linked https://molssi-qc-schema.readthedocs.io/; I'd say there should not be a universal parser, there should be a universal interface, e.g. input/output JSON schemas, which any parser would implement

That would also be an option - I just want to emphasise that from the pyiron side we are very much interested in moving towards universal interfaces/ parsers and conda was the first step. So maybe it makes sense to discuss this topic in a larger group at some later stage.

[chrisjsewell]

Yeh that sounds great, let me know if/when you want to arrange such a discussion 👍

[JPchico]

I think that having some sort of universal interface as @chrisjsewell mentions is a great idea. This is something that we discussed before, specially for a code like LAMMPS, where the input file is more of a pseudo-code. How would one tackle this issue? Since there is quite a bit of of functionality in LAMMPS which cannot easily be codified.

[JPchico]

@chrisjsewell something that we have been discussing is the level of "workflow" logic that one should allow LAMMPS itself to cover and which level should aiida-lammp treat.

LAMMPS works quite differently than DFT codes which have a very well defined input with a finite set of parameters that can easily be controlled. This is not the cause in LAMMPS where the input is basically a script that gives a series of instructions that allow one to perform either a simple calculation or a workflow consisting of several calculations.

One thing that would be good to settle before trying to come up with a generic interface is what is the level of "scripting" that one allows in a given LAMMPS calculation. E.g. in the majority of the DFT plugins there are workflows that handle the logic behind several types of calculations, such as relaxation, equation of states, etc. but one can submit a "calculation" (probably via a BaseRestartWorkchain) which has access to all the options available in a single calculation (e.g. one can do a relaxation by hand by passing all the appropriate parameters, one shouldn't but it is possible)

Once that answer that question, one can much more easily design a "common" interface.

The solution that several frameworks such as ASE and Pyiron (@jan-janssen please correct me if I'm wrong) seems to be that only a very small set of predefined variables can be passed to the basic LAMMPS calculation, and all the other logic is off-loaded to either workflow management suites or local scripting. This is of course the simplest option, as very few modifications (if any) of aiida-lammps would be necessary, however one looses quite a lot of the options available in LAMMPS, and then performing something that is outside of the predefined options would necessitate the definition of a workflow or some more complex scripting.

Another completely different option (and something I do not think is very good from the AiiDA perspective is to do what seems to be done inside atomate where one can pass the entire lammps input file instead of just variables in the file. This of course gives you a lot of flexibility, but I think destroys a lot of the reproducibility, and traceability aspects at the core of AiiDA, so I think that this is not something that one should do. Instead something more in-between these solutions, where some flexibility is given would be ideal.

Something that I plan to be dealing with quite soon is with the implementation of base workchains that make use of the BaseRestartWorkchain so that some internal logic can be applied to failed calculations that are recoverable (e.g. no convergence achieved due to lack of time steps). However, it would be good to try to come to an agreement on the "scripting" level acceptable in aiida-lammps to avoid having to refactor anything more than necessary.

[chrisjsewell]

Just to give a quick heartbeat on this (and I will try to give some more thought in the coming days/week): I certainly agree with the points you make; the balance between flexibility and "structure" for the workflows is crucial 👍

[JPchico]

Yes I think that achieving the correct balance is key, specially in codes like LAMMPS. I'm myself not sure on what the balance should be, and it would be great to hear what some of the other people that want to use aiida-lammps thinks, because if we tackle that, then making workflows for the most common use-cases would be much easier.

[JPchico]

Hello everyone!

TL;DR We are trying to refactor the plugin to make it more flexible. If you have ideas/comments/suggestions/concerns please feel free to check the issues and let us know what you think.

As you might have made a little milestone to refactor the plugin to make it more flexible.

The basic idea is to change what is three calculations now into a single base calculation, in which the user can run any single stage LAMMPS calculation, depending of which parameters they have passed, i.e. instead of running an optimize calculation for relaxation, you pass the box/relax fix with the parameters you want and then that calculation will be ran. The role of the current optimize, forceand md calculations would instead be upgraded to Workflows.

Why this change? The idea is that at the Calculation level, one can have any options that one wants, for a single stage LAMMPS run, i.e. whatever fixes and computes you want (taking care of that certain fixes do not work if you run a minimization for example). While in the workflow level, a set of predefined parameters would let you tackle very specific issues. By making the Calculation very flexible, building these workflows would be simpler, as one would not have to worry about whether a compute or fix is present in the calculation that one is using.

You can see that in the milestone I have been gathering certain issues that could be used to improve aiida-lammps in this refactoring. However, I would very much like to hear what other users of aiida-lammps have to say. I have noticed that @mjclarke94 , @jan-janssen, @RaffaelaCabriolu and @ltalirz have some nice use cases as well as some ideas that could be used to improve the plugin. So if you can please take a look at the issues, and feel free to let us know what you think of these changes, as well of other things that we have not thought of.

If you are curious on what some of the actual implementation ideas are, I have made a fork of the plugin and I'm doing the development there, currently dealing mostly with the input generation, I think that making this flexible would be key. I'm mostly using the same philosophy from @abelcarreras but without the separation of types of runs, instead having certain things to be mutually exclusive, e.g. you cannot run an md and a minimize step in the same stage, this would require different runs.

All in all please let me know what you think, and if you want to help us out hacking away we are more than happy to collaborate!