Update serializations of numbers vs byte data (#114)

* Update serializations of numbers vs byte data * Dependabot updates * Add additional generic Data Type field for the announcement specs * Use UINT_64 for parquet types
LibertyDSNP · Aug 23, 2021 · 142e3ff · 142e3ff
1 parent d062138
commit 142e3ff
Show file tree

Hide file tree

Showing 11 changed files with 6,129 additions and 11,640 deletions.
diff --git a/package-lock.json b/package-lock.json
diff --git a/pages/Announcements/Overview.md b/pages/Announcements/Overview.md
@@ -30,18 +30,36 @@ Most serializations use outside standards, but some require additional clarifica
 
 ### hexadecimal
 
+Used to represent bytes.
+
 - MUST use 0-9,a-f representation
 - MUST be lowercase
 - MUST be prefixed with a `0x`
-- MUST NOT have spaces
-- MUST NOT have any zero padding
+- MUST NOT have spaces or separators
+- MUST have two characters per byte in addition to the `0x` characters
 
 | Bytes | Invalid | Valid |
 | --- | --- | --- |
-| 2 | `0x0123` | `0x123` |
-| 2 | `123h` | `0x123` |
-| 8 | `0x0000000000000abc` | `0xabc` |
-| 32 | `0x00003e34c4325f4461b9355027b314f3eb56d31af549f7da7bd9ef1ce951651e` | `0x3e34c4325f4461b9355027b314f3eb56d31af549f7da7bd9ef1ce951651e` |
+| 2 | `0x123` | `0x0123` |
+| 2 | `123h` | `0x0123` |
+| 2 | `0x0ABC` | `0x0abc` |
+| 8 | `0xabc` | `0x0000000000000abc` |
+| 32 | `0x3e34c4325f4461b9355027b314f3eb56d31af549f7da7bd9ef1ce951651e` | `0x00003e34c4325f4461b9355027b314f3eb56d31af549f7da7bd9ef1ce951651e` |
+
+### decimal
+
+Used to represent integers.
+Strings are used to avoid issues with different implementations of numbers.
+
+- MUST use 0-9 representation
+- MUST NOT have spaces or separators
+- MUST be a string
+
+| Invalid | Why | Valid |
+| --- | --- | --- |
+| `0x123` | Must be decimal | `"291"` |
+| 291 | Must be a string | `"291"` |
+| `291n` | `BigInt(291)` serialization appends an `n`  | `"291"` |
 
 ## Duplicate Handling
 

diff --git a/pages/Announcements/Signatures.md b/pages/Announcements/Signatures.md
@@ -28,26 +28,24 @@ All parties interacting with Announcements should independently validate signatu
 
 #### Example
 
-```json
-{
-  "announcementType": 1,
-  "fromId": "0x12345",
-  "contentHash": "0x67890",
-  "url": "https://www.dsnp.org/",
-  "createdAt": "2021-07-31T10:11:12"
-}
-```
+| Field | Value |
+| --- | --- |
+| announcementType | `1` |
+| contentHash | `0x9c22ff5f21f0b81b113e63f7db6da94fedef11b2119b4088b89664fb9a3cb658` |
+| createdAt | `1627726272000` |
+| fromId | `74565` |
+| url | `https://www.dsnp.org/` |
 
 Expected serialization:
 
 ```
-\x19Ethereum Signed Message:\n64announcementType0x1contentHash0x67890createdAt0x17afc0bd600fromId0x12345urlhttps://www.dsnp.org/
+\x19Ethereum Signed Message:\n64announcementType1contentHash0x9c22ff5f21f0b81b113e63f7db6da94fedef11b2119b4088b89664fb9a3cb658createdAt1627726272000fromId74565urlhttps://www.dsnp.org/
 ```
 
 Serialization in hexadecimal:
 
 ```
-0x19457468657265756d205369676e6564204d6573736167653a0a3936616e6e6f756e63656d656e7454797065307831636f6e74656e7448617368307836373839306372656174656441743078313761666330626436303066726f6d49643078313233343575726c68747470733a2f2f7777772e64736e702e6f72672f
+0x19457468657265756d205369676e6564204d6573736167653a0a313531616e6e6f756e63656d656e745479706531636f6e74656e74486173683078396332326666356632316630623831623131336536336637646236646139346665646566313162323131396234303838623839363634666239613363623635386372656174656441743136323737323632373230303066726f6d4964373435363575726c68747470733a2f2f7777772e64736e702e6f72672f
 ```
 
 ### Hashing
@@ -59,7 +57,7 @@ Serialization in hexadecimal:
 For the previous example, the resulting hexadecimal hash MUST match:
 
 ```
-0xe998171b9eedfe13a181aa158c7b2dbb739af9e5ca062cc5822e668be1314478
+0xabaae4d8fda61c1b9cf481ef784158ebae5cea36f5cb7d1242987553a6dc6aa8
 ```
 
 
@@ -79,15 +77,15 @@ Private Key: `0xd9d3b5afb7765ffd9f047fd0d1d9b47d4d538b6a56f1cf29dc160ab9c6d30aa3
 ```
 {
   v: '0x1c',
-  r: '0xa34e5f6ba5f133cc1c8dfed613ad913f07dc5dff38c92278f9253c07ff43bd1d',
-  s: '0x3f86a862db3db7223a2d2b530dd15cbdc450fb2394917f1f413f4a102822deca',
+  r: '0x2e05b0f769b0344a58a06718f90f5d605878b6d5e9e14e1f235de24b399cfe42',
+  s: '0x7135a0b704862a8bc2847c4bb9f78bb43f707d427f0ba19bb43f66d5666934c9',
 }
 ```
 
 The compressed form of the above being this (`r + s + v`):
 
 ```
-0xa34e5f6ba5f133cc1c8dfed613ad913f07dc5dff38c92278f9253c07ff43bd1d3f86a862db3db7223a2d2b530dd15cbdc450fb2394917f1f413f4a102822deca1c
+0x2e05b0f769b0344a58a06718f90f5d605878b6d5e9e14e1f235de24b399cfe427135a0b704862a8bc2847c4bb9f78bb43f707d427f0ba19bb43f66d5666934c91c
 ```
 
 ## Validating a Signature

diff --git a/pages/Announcements/Types/Broadcast.md b/pages/Announcements/Types/Broadcast.md
@@ -10,14 +10,14 @@ A Broadcast Announcement is a way to send a public message to everyone.
 
 ## Fields
 
-| Field | Description | Serialization | Parquet Type | Bloom Filter |
-| ----- | ----------- | ------------- | ------------ | ------------ |
-| announcementType | Announcement Type Enum (`2`) | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no |
-| contentHash | keccak-256 hash of content stored at URL | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
-| createdAt | milliseconds since Unix epoch | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT64` | no
-| fromId | id of the user creating the announcement | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
-| url | content URL | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `BYTE_ARRAY` | no
-| signature | creator signature | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
+| Field | Description | Data Type | Serialization | Parquet Type | Bloom Filter |
+| ----- | ----------- | --------- | ------------- | ------------ | ------------ |
+| announcementType | Announcement Type Enum (`2`) | enum | [decimal](/Announcements/Overview#decimal) | `INT32` | no |
+| contentHash | keccak-256 hash of content stored at URL | 32 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
+| createdAt | milliseconds since Unix epoch | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | no
+| fromId | id of the user creating the announcement | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | YES
+| url | content URL | UTF-8 | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `UTF8` | no
+| signature | creator signature | 65 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
 
 ## Field Requirements
 
@@ -27,7 +27,6 @@ A Broadcast Announcement is a way to send a public message to everyone.
 
 ### contentHash
 
-- MUST be 32 bytes in length
 - MUST be the [keccak-256 hash](https://keccak.team/files/Keccak-submission-3.pdf) of the bytes of the reference at the url
 
 ### createdAt

diff --git a/pages/Announcements/Types/GraphChange.md b/pages/Announcements/Types/GraphChange.md
@@ -10,14 +10,14 @@ A Graph Change Announcement is for publishing relationship state changes for a u
 
 ## Fields
 
-| Field | Description | Serialization | Parquet Type | Bloom Filter |
-| ----- | ----------- | ------------- | ------------ | ------------ |
-| announcementType | Announcement Type Enum (`1`) | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no |
-| changeType | Type of relationship change | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no
-| createdAt | milliseconds since Unix epoch | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT64` | no
-| fromId | id of the user creating the relationship | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
-| objectId | id of the target of the relationship | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
-| signature | creator signature | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
+| Field | Description | Data Type | Serialization | Parquet Type | Bloom Filter |
+| ----- | ----------- | --------- | ------------- | ------------ | ------------ |
+| announcementType | Announcement Type Enum (`1`) | enum | [decimal](/Announcements/Overview#decimal) | `INT32` | no |
+| changeType | Type of relationship change | enum | [decimal](/Announcements/Overview#decimal) | `INT32` | no
+| createdAt | milliseconds since Unix epoch | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | no
+| fromId | id of the user creating the relationship | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | YES
+| objectId | id of the target of the relationship | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | YES
+| signature | creator signature | 65 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
 
 ## Field Requirements
 

diff --git a/pages/Announcements/Types/Profile.md b/pages/Announcements/Types/Profile.md
@@ -13,11 +13,11 @@ The reference content *MUST be of profile type*.
 
 | Field | Description | Serialization | Parquet Type | Bloom Filter |
 | ----- | ----------- | ------------- | ------------ | ------------ |
-| announcementType | Announcement Type Enum (`5`) | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no |
+| announcementType | Announcement Type Enum (`5`) | [decimal](/Announcements/Overview#decimal) | `INT32` | no |
 | contentHash | keccak-256 hash of content stored at URL | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
-| createdAt | milliseconds since Unix epoch | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT64` | no
-| fromId | id of the user creating the announcement | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
-| url | Profile content URL | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `BYTE_ARRAY` | no
+| createdAt | milliseconds since Unix epoch | [decimal](/Announcements/Overview#decimal) | `UINT_64` | no
+| fromId | id of the user creating the announcement | [decimal](/Announcements/Overview#decimal) | `UINT_64` | YES
+| url | Profile content URL | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `UTF8` | no
 | signature | creator signature | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
 
 ## Field Requirements

diff --git a/pages/Announcements/Types/Reaction.md b/pages/Announcements/Types/Reaction.md
@@ -10,14 +10,14 @@ A Reaction Announcement is for publishing emoji reactions to anything with a [DS
 
 ## Fields
 
-| Field | Description | Serialization | Parquet Type | Bloom Filter |
-| ----- | ----------- | ------------- | ------------ | ------------ |
-| announcementType | Announcement Type Enum (`4`) | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no |
-| createdAt | milliseconds since Unix epoch | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT64` | no
-| emoji | the encoded reaction | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `BYTE_ARRAY` | YES
-| fromId | id of the user creating the relationship | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
-| inReplyTo | Target [DSNP Announcement URI](/Identifiers#dsnp-announcement-uri) | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `BYTE_ARRAY` | YES
-| signature | creator signature | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
+| Field | Description | Data Type | Serialization | Parquet Type | Bloom Filter |
+| ----- | ----------- | --------- | ------------- | ------------ | ------------ |
+| announcementType | Announcement Type Enum (`4`) | enum | [decimal](/Announcements/Overview#decimal) | `INT32` | no |
+| createdAt | milliseconds since Unix epoch | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | no
+| emoji | the encoded reaction | UTF-8 | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `UTF8` | YES
+| fromId | id of the user creating the relationship | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | YES
+| inReplyTo | Target [DSNP Announcement URI](/Identifiers#dsnp-announcement-uri) | UTF-8 | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `UTF8` | YES
+| signature | creator signature | 65 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
 
 ## Field Requirements
 

diff --git a/pages/Announcements/Types/Reply.md b/pages/Announcements/Types/Reply.md
@@ -11,15 +11,15 @@ but includes an `inReplyTo` field for noting it as a reply to a given [DSNP Anno
 
 ## Fields
 
-| Field | Description | Serialization | Parquet Type | Bloom Filter |
-| ----- | ----------- | ------------- | ------------ | ------------ |
-| announcementType | Announcement Type Enum (`3`) | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no |
-| contentHash | keccak-256 hash of content stored at URL | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
-| createdAt | milliseconds since Unix epoch | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT64` | no
-| fromId | id of the user creating the announcement | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
-| inReplyTo | Target [DSNP Announcement URI](/Identifiers#dsnp-announcement-uri) | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `BYTE_ARRAY` | YES
-| url | content URL | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `BYTE_ARRAY` | no
-| signature | creator signature | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
+| Field | Description | Data Type | Serialization | Parquet Type | Bloom Filter |
+| ----- | ----------- | --------- | ------------- | ------------ | ------------ |
+| announcementType | Announcement Type Enum (`3`) | enum | [decimal](/Announcements/Overview#decimal) | `INT32` | no |
+| contentHash | keccak-256 hash of content stored at URL | 32 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
+| createdAt | milliseconds since Unix epoch | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | no
+| fromId | id of the user creating the announcement | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | YES
+| inReplyTo | Target [DSNP Announcement URI](/Identifiers#dsnp-announcement-uri) | UTF-8 | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `UTF8` | YES
+| url | content URL | UTF-8 | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `UTF8` | no
+| signature | creator signature | 65 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
 
 ## Field Requirements
 
@@ -29,7 +29,6 @@ but includes an `inReplyTo` field for noting it as a reply to a given [DSNP Anno
 
 ### contentHash
 
-- MUST be 32 bytes in length
 - MUST be the [keccak-256 hash](https://keccak.team/files/Keccak-submission-3.pdf) of the bytes of the reference at the url
 
 ### createdAt

diff --git a/pages/Announcements/Types/Tombstone.md b/pages/Announcements/Types/Tombstone.md
@@ -17,14 +17,14 @@ It is NOT possible to revert a tombstone.
 
 ## Fields
 
-| Field | Description | Serialization | Parquet Type | Bloom Filter |
-| ----- | ----------- | ------------- | ------------ | ------------ |
-| announcementType | Announcement Type Enum (`0`) | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no |
-| createdAt | milliseconds since Unix epoch | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT64` | no
-| fromId | id of the user creating the announcement and tombstoned announcement | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
-| targetAnnouncementType | target tombstoned announcement type | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no |
-| targetSignature | target announcement signature to tombstone  | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
-| signature | creator signature | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
+| Field | Description | Data Type | Serialization | Parquet Type | Bloom Filter |
+| ----- | ----------- | --------- | ------------- | ------------ | ------------ |
+| announcementType | Announcement Type Enum (`0`) | enum | [decimal](/Announcements/Overview#decimal) | `INT32` | no |
+| createdAt | milliseconds since Unix epoch | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | no
+| fromId | id of the user creating the announcement and tombstoned announcement | 64 bit unsigned integer | [decimal](/Announcements/Overview#hecimal) | `UINT_64` | YES
+| targetAnnouncementType | target tombstoned announcement type | enum | [decimal](/Announcements/Overview#decimal) | `INT32` | no |
+| targetSignature | target announcement signature to tombstone | 65 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
+| signature | creator signature | 65 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
 
 ## Field Requirements
 

diff --git a/pages/Identifiers.md b/pages/Identifiers.md
@@ -7,13 +7,13 @@ route: /Identifiers
 
 ## DSNP User Id
 
-- Is 8 bytes in size
+- 64 bit Unsigned Integer
 - MUST be registered in the [Identity Registry](/Identity/Registry)
-- MUST be serialized as [hexadecimal](/Announcements/Overview#hexadecimal)
+- MUST be serialized as [decimal](/Announcements/Overview#decimal)
 
 ## DSNP Content Hash
 
-- Is 32 bytes in size
+- MUST be 32 bytes in size
 - MUST be a [keccak-256 hash](https://keccak.team/files/Keccak-submission-3.pdf) of the bytes of the content
 - MUST be serialized as [hexadecimal](/Announcements/Overview#hexadecimal)
 
@@ -28,13 +28,13 @@ It is used to identify a user via a URI.
 
 ### Example
 ```
-dsnp://0x1234567890abcdef
+dsnp://1311768467294899700
 ```
 
 | part | value |
 | ---- | ----- |
 | Scheme | `dsnp://` |
-| User Id | `0x1234567890abcdef` |
+| User Id | `1311768467294899700` |
 
 ## DSNP Announcement URI
 
@@ -45,11 +45,11 @@ Any [Announcement Types](/Announcements/Overview#announcement-types) with a `fro
 
 ### Example
 ```
-dsnp://0x1234567890/0x1234567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef
+dsnp://78187493520/0x1234567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef
 ```
 
 | part | value |
 | ---- | ----- |
 | Scheme | `dsnp://` |
-| User Id | `0x1234567890` |
+| User Id | `78187493520` |
 | Content Hash | `0x1234567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef` |
diff --git a/pages/index.md b/pages/index.md
@@ -5,7 +5,7 @@ route: /
 
 # DSNP Specification
 
-## Version 0.9.2
+## Version 0.10.0
 
 Welcome to the Decentralized Social Networking Protocol (DSNP) specification!
 Here you can find a detailed documentation regarding the current state of the protocol, previous iterations and proposals for future extensions.