DsWriter should create ZIP files directly in archive storage

[RKrahl]

Consider an ids.server to be configured as two level storage with storage unit Dataset. When the DsWriter creates a ZIP file to be stored into archive storage, it creates the ZIP file in the cache first and then copies this file to archive storage. The intermediate step in the cache is not needed, this causes all data to hit the disk once more then necessary. The DsWriter should write the ZIP file right into archive storage instead.

[fisherab]

The DsWriter and DsRestorer both make use of the datasetCache. This is not really a cache but just some temporary storage. It used to be a real cache in a much earlier version of the code. It was then kept to reduce the risk of getting the main or archive storage in an inconsistent state due to I/O errors. If this code is removed then code will need to be added to clean up any half built structures if an exception is thrown.

What do you think?

[RKrahl]

Dealing with I/O errors is a somewhat broader issue then this one. The question is, what should be done in case of errors on the one hand and what does the current code in ids.server do on the other. I'm afraid, both is not the same in all cases.

Considering the particular case of the DsWriter, if I understand things correctly, in the case of an error, the DsWriter aborts, writes an error message to the log and that's it. The data from main storage will not be written to archive in this case. If the next action in the process queue concerning this dataset will be an ARCHIVE request, the data will be erased from main storage and are gone forever, without any notice. If the next action happens to be a WRITE request and the DsWriter succeeds this time, everything is fine and the previous error will have had no effect at all.

I agree that writing the ZIP file directly into archive storage doesn't make things any better in the case of an I/O error, because the half written ZIP file will most likely be corrupt then. But the intermediate step of writing into cache space isn't sufficient either to save the data. I guess, we should reconsider error handling more in general and postpone this Issue until we came to a conclusion there.

[RKrahl]

Depends on #79.