3rd party dcache implementation

[DavidTrachy]

I am creating a 3rd party open source dcache implementation and have found that there are missing calls in at least main.c for plugging in a dcache library provider, initializing a dcache provider and adding index-data to the dcache provider as it is read from the index partition. There may be other files as well that need modifications. If anyone has a provider and is willing to share the files it would be much appreciated.

[piste-jp]

The purposes of the dcache are

1. Return a metadata without tape mount
2. Reduce the memory foot print of on memory meta-data

At this time, we don't think dcache I/F have a big value in the single drive environment. Because it is no demand to return meta-data from dcache without tape mount. And it is really bad idea to fill a tape with tens or hundreds million files, that idea might waste the capacity of tape.

On the other hand, the dcache is a kind of cache. So we don't have any intent to build it from the index on tape directly. It shall be created (rebuilt) from read metadata from the index on tape.

Of course, IBM Spectrum Archive LE has such kind of codes but I'm sorry to say we cannot provide it to the open source project at this time.

[DavidTrachy]

In regards to value of dcache. We have customers who have small files and require that we archive files exactly as they are, in other words will not allow us to bundle them into say .zip or .tar files. We also have systems that have a dozen or more tapes drives loaded at one time and therefore memory consumption is becoming an issue. Why this has become more concerning is that the tape capacities have increased such that when we remaster say LTO5 tapes onto LTO9 tapes our memory usage goes up by a factor of 6x. With the possibility of 40 TB+ tapes in the future we see this problem only becoming much worse and basically rendering LTFS unusable for many archival environments. I understand that IBM does not want to provide their dcache code present in their enterprise version however better definition of the functions defined in dcache.c sure would be useful. From a community standpoint if anyone else has this requirement I certainly would be open to partnering with them in the creation of an open source version of dcache.

[piste-jp]

I understand that IBM does not want to provide their dcache code present in their enterprise version however better definition of the functions defined in dcache.c sure would be useful. From a community standpoint if anyone else has this requirement I certainly would be open to partnering with them in the creation of an open source version of dcache.

It's OK to develop it by your lead and to open the PR for merge it but I can't help you. Because (as you know) I'm working in IBM.

IBM allows me to make activities on this project unless I don't make any negative impact the IBM product. One more thing is I'm a developer of the dcache module bundled into IBM Spectrum Archive LE, so I cannot re-implement it in the open source world because it means the source code leak from IBM.

In this topic, I cannot say anything anymore. Please help him if someone can join in this idea without any restriction.

[KengoSawa2]

Thanks to IBM LTFS OSS project.

Hi, I am a user of IBM Spectrum Archive LE for archiving system in Japanese post production.

I am also keeping watch on this issue.

There is a requirement in the video and medical industries to store large numbers of sequentially numbered images without having to tar or zip them.

The number of files can be quite large, as in the example of archiving a single-frame, single-file film to LTO.
I have been helped by LTFSLE,
but given the migration,
it is natural to want dentry support.

My LTFSLE usages:
https://www.lespace.co.jp/Product-Detail/8UCUPUoU
(sorry, only in Japanese)

Although I write programs for file copy software,
I don't have the technical skills to support the internal implementation of dentry.
I hope to come up with some good ideas on this issue in the future:)