Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate why small transfer is slow in AsyncSshTransport #6737

Open
khsrali opened this issue Jan 29, 2025 · 2 comments
Open

Investigate why small transfer is slow in AsyncSshTransport #6737

khsrali opened this issue Jan 29, 2025 · 2 comments
Labels
type/performance Issue related to how quickly AiiDA works

Comments

@khsrali
Copy link
Contributor

khsrali commented Jan 29, 2025

Here #6626 (comment)

I ran some tests comparing small file transfers between core.ssh_async and core.ssh
The one for core.ssh_async is slower.

I originally thought because the I/O for small tasks are not really blocking that much so that one is slow because it has async over heads.

But I may be wrong, so a more careful look is needed. Suggestions from @giovannipizzi

  • One can check the scaling by limiting async workers to 2-4-8, etc.
  • Try to change number of files to see the scaling
  • Or it might be slow becasue BeeGfs has limits on concurrent tasks thor server (where I perfomed those test)
@danielhollas danielhollas added the type/performance Issue related to how quickly AiiDA works label Feb 10, 2025
@danielhollas
Copy link
Collaborator

@khsrali If you could include the scripts that you used for the original benchmarking (and any additional details about your approach) I could at least try to reproduce your results.

@khsrali
Copy link
Contributor Author

khsrali commented Feb 12, 2025

@danielhollas fair point 😅
I'll polish my script adding comments, etc. and will put it here soon.. stay tuned

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/performance Issue related to how quickly AiiDA works
Projects
None yet
Development

No branches or pull requests

2 participants