Clearing cuda cache during interpolation is slow

[AntoineTheb]

Hi @EmmaRenauld !

I was profiling my training and noticed that "torch._C._cuda_emptyCache()" was slowing down my tracking process quite a lot.

Looking at dwi_ml.data.processing.volume.interpolation.interpolate_volume_in_neighborhood (https://github.com/scil-vital/dwi_ml/blob/master/dwi_ml/data/processing/volume/interpolation.py#L166), the comment does mention that it may be a bit low but saves on memory. Would it be possible to make clearing the cache optional ? Or leave it to the user ? Personally I never encountered memory leaks due to this.

Thanks !

[EmmaRenauld]

Ok, yes. Do a PR, and I will test it both locally and on Beluga, to see if my things still run correctly. I remember that I was not sure at all how to manage all this.
It's not really exactly memory leaks. It's just that there are variables that are deleted in the code but not yet in the memory.
But also, it could be very different in torch 2.0. Did you have time to see if you can install everything with it? I could merge PR 234 before we test the interpolation.