DIP-273 Content Addressing #280

wesbiggs · 2024-07-01T19:51:55Z

Problem

DSNP should provide affordances for finding content that do not rely on a single DNS-based point of failure for content hosting.
More in the original discussion: #273

Solution

Enhance the specification to treat URLs in Announcements as suggestions rather than canonical locations for content.
Provide a simple and well-specified set of hashes and encodings that can be used consistently throughout the protocol.
Use IPFS CIDv1 specifically for locating profiles.

Change summary:

Broaden ProfileResource definition so that different types of profile resources can use different, possibly distributed, file systems via a generic contentAddress field
Simplify multihashes to use base32 encoding only and sha2-256 or blake3 as the hashing algorithm
Update various example hashes in line with this
Update to pre-1.3.0 versioning and sync prerelease changelogs for other recent additions to the spec
Change spec language to expand on how to treat content hash + URL pairs.
Announcements use the same base32 encoded multihash for the contentHash and targetContentHash fields.

* Update the definition of DSNP Content Hash to include DSNP CIDs * Change ProfileResource to use bytes, not string, for cid field, in line with Announcement usage * Update Activity Content Hash extension to include CID option * Update various example hashes to use CIDv1 * Update to pre-1.3.0 versioning and sync prerelease changelogs

audreyt · 2024-07-03T16:27:59Z

pages/DSNP/Identifiers.md

@@ -69,9 +70,9 @@ In order for DSNP applications to interoperate, the required functionality is li

 - CID version: MUST be version 1, in order to distinguish CIDs from simple multihash values in situations where either may be used
 - Hash algorithm: MUST be `sha2-256` or `blake2b-256`
- Encoding: MUST be `base58btc` or `base32`
 - Codec: MUST be `dag-pb` for data 256*1024 bytes or longer; `raw` for data less than 256*1024 bytes


Adding backslash before * helps markdown to disambiguate:

Codec: MUST be dag-pb for data 256*1024 bytes or longer; raw for data less than 256*1024 bytes

…an option

…ported types only. Announcements have been updated to use strings for contentHash and targetContentHash fields.

wilwade

Everything looks good with the exception of some of the example hashes.

wilwade · 2024-08-12T12:56:21Z

pages/ActivityContent/Associated/Hash.md


 ```json
 {
  "hash": [
-    "QmQNHNfHnbgJJ6nK4UPx2VtTUCafAKCbqZJ6ZRYUGjoeFj",
-    "2DrjgbGgSsXRhTiBWckoVwBFC6H4qiBWWNumSsRwdUt82YnTdN"
+    "bciqgb7tvyyqn6tw4amtdj7funn5ockup6ldrkap4o5ubd77od6awi4y",


I'm getting a different value for the hash of the dsnp_whitepaper than this is giving me.

Perhaps it is the older version of the whitepaper the hash is of?

Different hash setup that is reading the file differently than I did?

I tried sha256sum on mac and a random online hasher.

The digest I expected from hashing the file was 0x36d37cf984565f2151ccea050e0bcecf44801b4c89971654c7308dc2338ce7ea

However the digest from this multihash appears to be 0x60fe75c620df4edc032634fcb46b7ae12a8ff2c71501fc776811ffee1f816473

Yep, my mistake. Will fix.

wilwade · 2024-08-12T12:58:36Z

pages/ActivityContent/Associated/Hash.md

-    "QmQNHNfHnbgJJ6nK4UPx2VtTUCafAKCbqZJ6ZRYUGjoeFj",
-    "2DrjgbGgSsXRhTiBWckoVwBFC6H4qiBWWNumSsRwdUt82YnTdN"
+    "bciqgb7tvyyqn6tw4amtdj7funn5ockup6ldrkap4o5ubd77od6awi4y",
+    "bdyqdua4t4pxgy37mdmjyqv3dejp5betyqsznimpneyujsur23yubzna"


Same for this one, so I think it might just be we are using different files.

wilwade

Looks good to me!

shannonwells

Read through the referenced Issue thread, read through the changes, looks ok by me.

saraswatpuneet · 2024-08-12T22:01:36Z

pages/DSNP/BatchPublications.md

+| emoji | `BYTE_ARRAY` | `STRING` | `UTF8` |
+| fromId | `INT64` | `INT(64, false)` | `UINT_64` |
+| inReplyTo | `BYTE_ARRAY` | `STRING` | `UTF8` |
+| targetContentHash | `BYTE_ARRAY` | `STRING` | `UTF8` |


Note: Confirming on gateway side this is already updated

shannonwells

Re-approved

shannonwells

This time for sure

wesbiggs force-pushed the proposal/DIP-273_Content_Addressing branch from bbce502 to 9332b55 Compare July 1, 2024 19:54

wesbiggs requested a review from wilwade July 1, 2024 19:56

wesbiggs mentioned this pull request Jul 1, 2024

DIP-273 Content hash semantics and format normalization #273

Closed

wesbiggs force-pushed the proposal/DIP-273_Content_Addressing branch from 9332b55 to c571222 Compare July 1, 2024 20:41

Merge branch 'main' into proposal/DIP-273_Content_Addressing

a6f5ac2

wesbiggs requested a review from harry-evans July 1, 2024 20:59

audreyt reviewed Jul 3, 2024

View reviewed changes

Wes Biggs added 2 commits July 10, 2024 22:17

Revert to multihash only, but with blake3 in place of blake2b-256 as …

7da7a6d

…an option

Improve description of usage of content hash vs URL

1329f7a

wilwade mentioned this pull request Jul 17, 2024

[bug] contentHash uses incorrect encoding ProjectLibertyLabs/content-publishing-service#149

Closed

wilwade mentioned this pull request Jul 30, 2024

Add back in hash validation ProjectLibertyLabs/gateway#276

Closed

Wes Biggs added 7 commits August 2, 2024 13:02

Update ProfileResource

09b606b

Fix content URI definition; edit versioning on start page

47aad29

"DSNP Content Hash" now refers to the base32-encoded multihash of sup…

51c0218

…ported types only. Announcements have been updated to use strings for contentHash and targetContentHash fields.

Lint

9a43e28

Break out base32 multibase as a Serialization

2543500

Formatting issues

eda2857

Tweaking decimal examples that bothered me

f3ba6e3

wilwade reviewed Aug 12, 2024

View reviewed changes

Fix hashes for dsnp_whitepaper.pdf

a080418

wilwade approved these changes Aug 12, 2024

View reviewed changes

wesbiggs enabled auto-merge (squash) August 12, 2024 20:11

shannonwells approved these changes Aug 12, 2024

View reviewed changes

saraswatpuneet approved these changes Aug 12, 2024

View reviewed changes

saraswatpuneet reviewed Aug 12, 2024

View reviewed changes

wesbiggs requested a review from shannonwells August 13, 2024 17:17

shannonwells approved these changes Aug 13, 2024

View reviewed changes

wesbiggs merged commit 0a81372 into main Aug 13, 2024
1 check passed

wesbiggs deleted the proposal/DIP-273_Content_Addressing branch August 13, 2024 17:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DIP-273 Content Addressing #280

DIP-273 Content Addressing #280

wesbiggs commented Jul 1, 2024 •

edited

Loading

audreyt Jul 3, 2024

wilwade left a comment

wilwade Aug 12, 2024

wesbiggs Aug 12, 2024

wilwade Aug 12, 2024

wilwade left a comment

shannonwells left a comment •

edited

Loading

saraswatpuneet Aug 12, 2024

shannonwells left a comment

shannonwells left a comment

DIP-273 Content Addressing #280

DIP-273 Content Addressing #280

Conversation

wesbiggs commented Jul 1, 2024 • edited Loading

Problem

Solution

Change summary:

audreyt Jul 3, 2024

Choose a reason for hiding this comment

wilwade left a comment

Choose a reason for hiding this comment

wilwade Aug 12, 2024

Choose a reason for hiding this comment

wesbiggs Aug 12, 2024

Choose a reason for hiding this comment

wilwade Aug 12, 2024

Choose a reason for hiding this comment

wilwade left a comment

Choose a reason for hiding this comment

shannonwells left a comment • edited Loading

Choose a reason for hiding this comment

saraswatpuneet Aug 12, 2024

Choose a reason for hiding this comment

shannonwells left a comment

Choose a reason for hiding this comment

shannonwells left a comment

Choose a reason for hiding this comment

wesbiggs commented Jul 1, 2024 •

edited

Loading

shannonwells left a comment •

edited

Loading