[PB-3405]: feat/limit-webdav-concurrent-petitions

[larryrider]

• Limited WebDAV concurrent petitions to avoid backend timeouts

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[sg-gs]

@larryrider

I think you misunderstood the timeout issue. The issue could happen more frequently when you do long-lived requests, but even in small requests, the timeout can happen and it is not a matter of the number of concurrent uploads in this case.

The issue is that the connection could be stalled, due to a variety of reasons. As multipart transfers allow to repeat just the transfer of a chunk (either download or upload) if it fails or is stalled, you need to control precisely that and if happens, re-attempt the request.

TL;DR: You need to set the timeout max allowed seconds on the library doing the transfer. Often it's https (Node.js native) which has a parameter for that. This requires a change that involves the library you are using to perform uploads (inxt-js)

[larryrider]

@larryrider

I think you misunderstood the timeout issue. The issue could happen more frequently when you do long-lived requests, but even in small requests, the timeout can happen and it is not a matter of the number of concurrent uploads in this case.

The issue is that the connection could be stalled, due to a variety of reasons. As multipart transfers allow to repeat just the transfer of a chunk (either download or upload) if it fails or is stalled, you need to control precisely that and if happens, re-attempt the request.

TL;DR: You need to set the timeout max allowed seconds on the library doing the transfer. Often it's https (Node.js native) which has a parameter for that. This requires a change that involves the library you are using to perform uploads (inxt-js)

Oh I see.. Even so, I believe this change can help avoid overwhelming the system with multiple calls, as it handles concurrent WebDAV calls to avoid an overload of requests to the backend.

Additionally, I've submitted another PR that implements multipart uploads, which I believe will significantly improve the issues we've been encountering.

The solution that implements retries for individual chunks that you mentioned looks good for me, we can implement it in another task too

[larryrider]

closing it as this task was already fixed in other PR