Is TimeOut handled in getFile API?

[panta-123]

Dear All,
At Belle II , we saw a case when the files are downloaded using getFile method:
While downloading multiple files, download got stuck for some hours in a single file. This means the whole download got stuck. Isn't there any mechanism of Timeout in getFile method, so that that specific file download fails but other files in the list can go on with the download.

[fstagni]

No, there is no timeout. A timeout would depend on a number of factors (protocol, file size, network, storage) and it would not be a practical choice to make.
Also, skipping one file would be a rather dangerous idea if not implemented properly.

[panta-123]

Good point. But can we have timeout option in the API, so that the caller of API can set it as it wants. Default value is NONE.

[chaen]

Sorry I missed that question.
There is indeed no parameters you can pass. I'd have to see how intrusive it is to add one, but I am not against per se.

There is however at the moment a transfer timeout which is estimated:

DIRAC/src/DIRAC/Resources/Storage/GFAL2_StorageBase.py

Lines 162 to 172 in 14dc98d

	def _estimateTransferTimeout(self, fileSize):
	"""Dark magic to estimate the timeout for a transfer
	The values are set empirically and seem to work fine.
	They were evaluated with gfal1 and SRM.
	:param fileSize: size of the file in bytes we want to transfer
	:return: timeout in seconds
	"""
	return int(fileSize / self.MIN_BANDWIDTH * 4 + 310)

This can however indeed give large values...
I am currently having discussions with the gfal people about timeout management in general, I'll come up with something once this discussions settles.

One thing to be noted though: the timeout is at the protocol level. That means that if you have n possible protocols to try, you will have n times the timeout

[iueda]

The timeout we are talking about here is to skip transfers that often stall , or even do not start, with a certain SE.
Larger timeout for larger files is good. Here we are talking about very small files (user job output), usually downloaded in 1-2 sec, while download from the SE often stall and it takes "hours" according to end-user who reported the issue.
I do not have a clear technical information on the download problem, but it may well be related to the lack of available movers that has been causing failures in FTS transferes and discussed in a GGUS (https://ggus.eu/index.php?mode=ticket_info&ticket_id=158552).
For FTS transfers, we still use SRM, and the perf-marker timeout (360) kills transfers.
For downloading (DIRAC.DMS.getFile), we use davs and it waits for "hours" (according to end-user).
The above, with self.MIN_BANDWIDTH = 0.5 * (1024 * 1024), would give 314 (sec?) for a file size 0.5MB, i.e. 5 min. for such small files.
I am wondering why they claim their download gets stuck for "hours".

[chaen]

Ah, I think you are getting hit by something we found out very recently while digging with the gfal developers: https://its.cern.ch/jira/browse/DMC-1339
So in the case of downloading files with webdav, the timeout parameter set in the transfer is ignored, and it uses another timeout that we do not set (HTTPS OPERATION_TIMEOUT).
Good news is that this is fixed in the next gfal2 patch release (end of september), and we can then very quickly make a DIRACOS2 release. It will solve the problem.