Module for filtering exports #80
Conversation
src/Filter.cc
Outdated
| // Parse the provided file to configure the class | ||
| // | ||
| // We understand the following options: | ||
| // - filter.trace [all|error|warning|info|debug|non] |
| path.data()); | ||
| return -EISDIR; | ||
| } | ||
| return std::invoke(fn, wrapPI, std::forward<Args>(args)...); |
There was a problem hiding this comment.
I could see that wrapPI comes from XrdOssWrapper.hh and is XrdOss &wrapPI, but it's hard to grok what this is doing here. Can you give a few hints?
| &XrdOss::Lfn2Pfn), | ||
| Path, buff, blen); | ||
| } | ||
| const char *FilterFileSystem::Lfn2Pfn(const char *Path, char *buff, int blen, |
There was a problem hiding this comment.
A lot of these function names and purposes are very cryptic to me, and while XrdOss.hh provides some definitions, I find those cryptic in their own way. This is probably because I don't have a deep understanding of all the filesystem operations going on here, but it does make me wonder whether they all actually translate into meaningful operations in S3/HTTP land. What do logical and physical file names mean in that space? Or do these overrides need to be filled in simply for the sake of completeness?
If we can find the time to sit down and go over these concepts in a bit more depth, I think I'd greatly benefit from the mentorship. If we can't find the time, are you able to link me to any accessible learning materials that could help me fill in my knowledge gaps?
There was a problem hiding this comment.
Indeed, let's sit down and chat! Always glad to delve deeper here.
That said, for completion:
- I'm trying to override all the functions for completeness. Even if the underlying plugins in this repository don't implement them, I see
libXrdOssFilter.sobeing of general use: one might stack it with a POSIX-based plugin where this is implemented. - LFN2PFN - or "logical file name to physical file name" is jargon for translating between the path exported by the various XRootD protocols and the path used within the OSS. The XRootD framework takes care of the simplest of transformations (such as stripping out a prefix) but it's possible to do more arbitrarily complex transformations. For example, CMS historically has allowed administrators to provide a list of regular expressions to do pattern matching to transform paths. That said, I believe it's more common to use a separate plugin (as in https://github.com/opensciencegrid/xrootd-cmstfc) than to implement this in the OSS layer.
- There are some functions I struggle with myself, such as the various "stat of a storage space". Again, those are overridden for completeness since they're just filtering the input location.
The new plugin, `libXrdOssFilter.so`, allows the administrator to only permit the opening / listings of files and directories that match specified globs or prefixes. This can be used to keep some storage from being visible via XRootD (even with the appropriate permissions). For example, to hide all files and directories beginning with "." from being exported, one could setup: ``` ofs.osslib ++ libXrdOssFilter.so filter.glob /** ``` (This policy would be quite difficult to implement via the existing authorization frameworks)
The `filter.prefix` approach is meant to provide a simpler variant of `filter.glob` (for administrators who feel they may get tripped up). To prevent unexpected behaviors, ensure that there are no glob metacharacters in the provided path and that the path is normalized.
|
Hi @bbockelm ! I'm still digesting this PR since it's built on top of stuff I don't understand yet. I'll leave a comment or two but you should proceed when you're ready. |
This PR (note: it's built on top of #79 and hence it contains those commits -- ignore them for review purposes or wait until that PR is merged) adds a new module,
libXrdOssFilter, for managing what objects are exported via XRootD.libXrdOssFiltercan be stacked on top of the HTTP or S3 OSS plugin and provides reasonably fine-grained control over what data is accessible. The administrator can limit things to specific prefixes or only objects matching a certain glob (e.g.,/ncar/dataset*/**or/foo/*.png). It supports the "globstar" operator, allowing multiple path hierarchies to be matched (e.g.,/pelican/**/*.gowill export all*.gofiles underneath/pelican). The integration test contains a valid XRootD configuration file demonstrating this.Includes reasonable unit test and a simple end-to-end integration test built on top of the S3 integration test.