Add Codecov

[jngrad]

Description of changes:

• move slow tests to a scheduled job called samples
  • fixes Reduce feedback time from the testsuite #56
  • the workflow can also be triggered manually
• add code coverage in CI using the Codecov GitHub Action
• add test cases (partial fix for Improve the code coverage of pyMBE #58)

[jngrad]

The pyMBE.py script has 55% code coverage at the time of writing. See #58 for the features that require our attention.

If we could get every pyMBE community member to contribute unit tests for 5 features and invest no more than 20 min of developer time per unit test, we should be able to reach 90% code coverage in pyMBE.py in about a week. We don't need to write simulation scripts, just a few np.testing.assert_allclose() or pd.testing.assert_frame_equal() checks. Convenience scripts under the lib/ folder might require more effort, since they need either synthetic data or a minimal ESPResSo simulation script as input.

The report is available online. You can generate the code coverage report locally with the following commands:

make unit_tests COVERAGE=1
make coverage_html
firefox coverage/index.html

This draft PR will be rebased once all dependency issues are sorted out. It is currently designed around Codecov, because I'm already familiar with this service, but I am open to alternatives. For completeness, here are the main competing services: Codecov, Coveralls, Codacy. Here is a brief comparison of these services.

[pm-blanco]

@jngrad a million thanks for addressing this point. I will add this to our tomorrow's meeting schedule so we can start splitting the workload of developing the tests.

[jngrad]

@pm-blanco While we wait for the Pylint PR to be merged, could you please give Codecov access to the pyMBE organization? Right now I'm using my own Codecov token, which means only the commits on my fork can upload coverage reports. Here is the official quick start page. You will need to follow step 1 using your GitHub account to sign up and establish access rights (the app needs to get read access to the organization and write access to write a green check mark in the commit metadata) and step 2 to store the secret token in the pyMBE-dev/pyMBE repository settings. That's it. The YAML file described in step 2 is already available in this PR. Steps 3-6 are already done in this PR too.

[pm-blanco]

@jngrad I belive it should be done now!

[codecov]

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment

Thanks for integrating Codecov - We've got you covered ☂️

[jngrad]

The code coverage reports are now uploaded to https://app.codecov.io/gh/pyMBE-dev/pyMBE and written to the commit status (green checkmark or red cross). New features require a unit test, otherwise code coverage will drop and CI will fail.

Things to keep in mind:

• for branches from this repository, the secret token is used, and uploads always succeed
• for PRs from forks, the secret token isn't revealed for security reasons, and tokenless mode is used: uploads may fail
  • GitHub has a shared pool of free actions for tokenless users; when exhausted, wait for it to refill and re-run job
  • action log: Upload failed: {"detail":"Tokenless has reached GitHub rate limit. Please upload using a token: https://docs.codecov.com/docs/adding-the-codecov-token. Expected available in 358 seconds."}
  • using OpenID Connect probably would not help us here, because it requires a secret too (for implementation details: codecov-action@d42a336:src/buildExec.ts#L47-L54)
• the samples job runs long tests and samples that reproduce paper results
  • runs biweekly at 6:20 AM to avoid rush hours: cron schedule 20 6 5,20 * *
    • scheduled workflows may start several minutes late if the rush hour is too strong, and might be cancelled if there isn't enough cloud capacity to absorb the backlog (quite rare, https://www.githubstatus.com/)
    • rush hour is every hour at 0 min, hence the extra 20 min
  • can be manually triggered
    • the manual action will only become visible in the Action tab once the workflow is part of the main branch!
    • I can confirm it works on branches, but I haven't tried on PRs (which are actually branches with a prefix)

[pm-blanco]

Looks excellent to me, thank you very much @jngrad. With this PR now we can proceed to develop more unit tests for pyMBE more efficiently. I added the information in this PR to our wiki so it can be consulted by all contributors.