-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add tag namespace concept #715
Conversation
Something else I haven't tried to tackle here yet is to avoid collisions between namespaced and non-namespaced tags. Since using a namespace simply adds a path prefix to the tag's location on disk, it is easy enough to have collisions, but it would be a good idea that tag "foo/bar" and tag "bar" in namespace "foo" are distinct entities that can coexist. This could be accomplished by putting a fixed prefix like ".namespaces/" at the beginning of all tag namespaces on disk, and then disallowing that as a legal name of a non-namespaced name. So like:
There's still the possibility of collisions between namespaces that are prefixes of other namespaces, though. We can require namespaces be a single flat name (no |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #715 +/- ##
==========================================
+ Coverage 53.55% 55.97% +2.42%
==========================================
Files 258 234 -24
Lines 20466 18318 -2148
==========================================
- Hits 10960 10254 -706
+ Misses 9506 8064 -1442 ☔ View full report in Codecov by Sentry. |
If we go in the direction of not putting runtimes into the tag namespace, where I ran into a small snag was here: The problem being that this code will want to clear the tag namespace on whatever repo it gets called with, but the signature here means this can only get an |
From the meeting today:
|
4aba1aa
to
ab41fe4
Compare
@rydrman I removed the depth part from the namespace. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you think it would be a lot of work for other repository types to support tag namespaces? Most have an FS repo at the backend, anyway or something like RPC can just send it in the request and default to None
.
I foresee us trying to use this feature in the future and immediately hitting that issues (since we use almost no direct FS repos)
crates/spfs/src/runtime/storage.rs
Outdated
// Keep runtime tags out of the tag namespace. Allow `spfs runtime` | ||
// to view and operate on all runtimes on the host. | ||
inner.as_tag_mut().set_tag_namespace(None); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a good argument to be made for putting all runtime tags into a specific namespace so that they don't busy-up the main one for users (though it wouldn't be backwards-compatible)
17cf840
to
02dce3a
Compare
I rebased this but haven't started addressing the notes yet. But I want to point out how this feature didn't really integrate with the pinned repository feature, since a pinned repository is a wrapper for another repo (via Arc) it wasn't possible to implement the new The first obvious thing I tried was to use Edit: Also to clarify, it is possible to set a tag namespace on a repo and then pin it, but currently not the other way around. |
ba0705f
to
76b0e56
Compare
I implemented RPC in my latest commit. Have a look at the commit message for more details. |
76b0e56
to
4d57ce5
Compare
Could we make the tag storage modification non require This layers on my previous thoughts about replacing the handle enum with For now, I think it's probably okay as a trade-off |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me, I think the only part that feels like it's missing is the ability to provide a namespace in url form for remote repos. I'm picturing a url query param like I added for the pinning stuff in open_repository
...
From the meeting today:
|
4d57ce5
to
7ccdf3a
Compare
I added the RemoteConfig stuff. |
From #390, the desire is to have different users (or CI jobs on the same runner) have a certain amount of isolation from each other. In particular, `spk ls` is expected to not contain any "local" packages that don't belong to the current user/CI job. This commit experiments with this idea. Only the local repo is expected to have an optional tag namespace set, so tar and rpc repos have not been changed. Setting the tag namespace is possible via the spfs config file, but it is expected to typically be set via the environment variable $SPFS_STORAGE_TAGNAMESPACE instead. It may be reasonable to add a [spk] config option to make the tag namespace default to the current user's username. A user sharing a host with other users are unlikely to want to see any local packages made by other users. There is an open question on if spfs runtimes should be subject to the tag namespace. This code currently does make them subject to the tag namespace. Therefore, if a user has a tag namespace specified, `spfs runtime ls` would only show the runtimes created in that same tag namespace, and those runtimes would be hidden from other users/other tag namespaces. This is in the same spirit of users being isolated from each other, however an argument could be made that it is important from an admin perspective to be able to list _all_ the runtimes on a host. `spfs clean` is a special case that always ignores any tag namespace. It should be able to find _all_ tags in the repo and avoid coming to any false conclusions about what objects are garbage on the basis of only seeing tags inside a tag namespace. Signed-off-by: J Robert Ray <jrray@jrray.org>
Storage::new will start needing a mutable reference to the repo in order to make it ignore the current tag namespace. Signed-off-by: J Robert Ray <jrray@jrray.org>
Signed-off-by: J Robert Ray <jrray@jrray.org>
Use a character that is an illegal character for users, '#', so we can prevent users from creating tag names that could be misinterpreted as namespace names. To prevent different namespaces from getting mixed up, include the depth of the namespace component in the directory name. For example the tag "hi" in namespace "foo/bar" would produce this on disk: /spfs-repos/local/tags/foo#ns.0 /spfs-repos/local/tags/foo#ns.0/bar#ns.1 /spfs-repos/local/tags/foo#ns.0/bar#ns.1/hi.tag This prevents tag "bar/hi" in namespace "foo" from colliding with tag "hi" in namespace "foo/bar" since these would have different filenames: - tags/foo#ns.0/bar/hi.tag - tags/foo#ns.0/bar#ns.1/hi.tag Signed-off-by: J Robert Ray <jrray@jrray.org>
Give ls-tags some awareness of the existence of namespace entries so it can hide these from its output. Note that iter_tag_streams doesn't make use of `EntryType` and will walk into any namespace directories it find, which for the time being is good because it still allows `spfs clean` to find all tags in all namespaces. Signed-off-by: J Robert Ray <jrray@jrray.org>
Verify that TagSpec will reject a name in the new namespace naming convention (it already does). Signed-off-by: J Robert Ray <jrray@jrray.org>
The test code requires a way to modify the tag namespace of a TempRepo, but since it can be changed through an Arc, add a `with_tag_namespace` method to make a duplicate handle to an existing repo but with a different namespace set. This required putting the TempDir inside an Arc. Signed-off-by: J Robert Ray <jrray@jrray.org>
Confirm that tags with names that could possibly collide with tag namespace names are really distinct and don't collide. Signed-off-by: J Robert Ray <jrray@jrray.org>
This was deemed unnecessary for disambiguation purposes and would only be useful for supporting nesting namespaces, which we do not expect to support at this time. Signed-off-by: J Robert Ray <jrray@jrray.org>
Signed-off-by: J Robert Ray <jrray@jrray.org>
This requires adding "*_in_namespace" flavors of the tag methods, because the rpc server needed a way to process requests for tags in a namespace without introducing race conditions by setting the tag namespace on its repo handle. The server is "stateless" in the sense that setting the tag namespace on the client side only changes state on the client side, and then any requests sent to the server include the name of the prevailing tag namespace as part of the request. Signed-off-by: J Robert Ray <jrray@jrray.org>
Signed-off-by: J Robert Ray <jrray@jrray.org>
Leverage the fact that Config contains the tag address. Signed-off-by: J Robert Ray <jrray@jrray.org>
As part of transitioning to use RelativePath instead of Path, create wrapper types to partially hide the underlying type from the public API. Signed-off-by: J Robert Ray <jrray@jrray.org>
Make it easier to add more query options. Signed-off-by: J Robert Ray <jrray@jrray.org>
Signed-off-by: J Robert Ray <jrray@jrray.org>
7ccdf3a
to
b9a33a1
Compare
From #390, the desire is to have different users (or CI jobs on the same
runner) have a certain amount of isolation from each other. In particular,
spk ls
is expected to not contain any "local" packages that don'tbelong to the current user/CI job.
This commit experiments with this idea. Only the local repo is expected to
have an optional tag namespace set, so tar and rpc repos have not been
changed. Setting the tag namespace is possible via the spfs config file,
but it is expected to typically be set via the environment variable
$SPFS__STORAGE__TAG_NAMESPACE instead.
It may be reasonable to add a [spk] config option to make the tag
namespace default to the current user's username. A user sharing a host
with other users are unlikely to want to see any local packages made by
other users.
There is an open question on if spfs runtimes should be subject to the tag
namespace. This code currently does make them subject to the tag namespace.
Therefore, if a user has a tag namespace specified,
spfs runtime ls
wouldonly show the runtimes created in that same tag namespace, and those
runtimes would be hidden from other users/other tag namespaces. This is in
the same spirit of users being isolated from each other, however an
argument could be made that it is important from an admin perspective to
be able to list all the runtimes on a host.
spfs clean
is a special case that always ignores any tag namespace. Itshould be able to find all tags in the repo and avoid coming to any false
conclusions about what objects are garbage on the basis of only seeing
tags inside a tag namespace.
Signed-off-by: J Robert Ray jrray@jrray.org