Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bit too late #35

Closed
wants to merge 14 commits into from
Closed

Bit too late #35

wants to merge 14 commits into from

Conversation

sorki
Copy link
Member

@sorki sorki commented Mar 10, 2019

Lots of things mainly

  • path hashing + tests
  • store now uses hashed paths
  • tries to test itself against reference implementations (some of the tests started to fail, need to debug further)
  • remote store accepts socket path and store path
  • attempt to integrate with Nix.Derivation library

Sorry for getting this out late but I've got stuck last time mainly due to derivations and misunderstanding of paths - caused some work duplication but it should be merge-able.

@sorki sorki mentioned this pull request Mar 10, 2019
@sorki
Copy link
Member Author

sorki commented Mar 10, 2019

digestText16 can be implemented by means of printAsBase16 (Digest bs) = printHashBytes16 bs depending on whether we want to depend on base16-bytestring or not (performance might be better I guess).

I've also added printHashAlgo as it's required for building path strings - should use Proxies similar to addToStore?

@@ -34,12 +37,23 @@ processOutput = go decoder
case ctrl of
e@(Error _ _) -> return [e]
Last -> return [Last]
Read n -> do
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StoreMonad now embeds Maybe Handle in its StateT so we can provide handle and use it to provide data when other side requests tunneling.

@@ -76,6 +76,7 @@ class HasDigest (a :: HashAlgorithm) where
initialize :: AlgoCtx a
update :: AlgoCtx a -> BS.ByteString -> AlgoCtx a
finalize :: AlgoCtx a -> Digest a
hashName :: T.Text
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a sensible place to put hashName (I think others have put this under its own typeclass, but simply using HashDigest seems better). HasDigest is not a great typeclass name under this change though. What's a name for this class that's neither (a) too specific for the methods it provides, (b) as general as "Hash"?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree that this should be under HasDigest (which I've renamed ValidAlgo in my branch). Not every hash has a canonical name, e.g. the truncated ones don't (and shouldn't!).

-- | Convert any Digest to a base32-encoded string.
-- This is not used in producing store path hashes
printAsBase32 :: Digest a -> T.Text
printAsBase32 (Digest bs) = printHashBytes32 bs

-- | Print lowercased name of the hashing algorithm
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove this function? It perfectly overlaps with hashName.

adjustName n | outputId == "out" = n
adjustName (PathName name) | otherwise = PathName $ T.concat [ name, T.pack "-", outputId ]

type Recursive = Bool
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: data Recursive = Recursive | NotRecursive preferred over type alias.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually:

-- | Whether the hash is computedover the serialisation of the output path
-- or over the contents of the single non-executable regular file
-- See thesis page 107
data OutputHashMode = Recursive | Flat

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still being broken up into manageable chunks,but see 974a944#diff-d6eea410e6c564641deccfa88ae99d1fR101

makeStored :: StoreDir -> Path -> Stored Path
makeStored sl p = (sl, p)

type PathType = Text
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we create a new datatype for this? I think it would be:

-- | Store object type is used when creating store paths, to differentiate
-- between paths for sources and derivation outputs (see thesis page 94)
data StoreObjectType =
    Source
  | Output Text -- ^ A build output (e.g. `Output "out"`)

or

-- | A single derivation build output (e.g. "out", "docs" etc) 
newtype BuildOutputName = BuildOutputName { getBuildOutputName :: Text }

-- | Store object type is used when creating store paths, to differentiate
-- between paths for sources and derivation outputs (see thesis page 94)
data StoreObjectType =
    Source
  | Output BuildOutputName -- ^ A build output (e.g. `Output (BuildOutputName "out")`)

-- build <s> string which is hashed and used in makeStorePath
-- <s> = "<pathType>:<hash_algo>:<base16_hash>:<storeDir>:<pathName>"
-- (exposed for testing purposes only)
makePathDigestString :: (HasDigest a) => PathType -> PathName -> Digest a -> StoreDir -> Text
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this function is only used in makeStorePath, can we move it into a where clause in makeStorePath? That could help to reduce the largish number of functions with similar-sounding names at the top level.

@imalsogreg
Copy link
Collaborator

Lot of good stuff here, @sorki!

where
base = T.pack . takeBaseName . BSC.unpack . BSL.toStrict $ p
parts = T.breakOn "-" base
digest = Digest . BSC.pack . T.unpack . fst $ parts
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is incorrect and needs #34

@sorki
Copy link
Member Author

sorki commented Mar 19, 2019

I would like to rebase and fix this at some point - probably after the Path changes are merged.

@sorki
Copy link
Member Author

sorki commented Mar 5, 2020

Superseded by #59. All of the stuff from review is now addressed.

@sorki sorki closed this Mar 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants