Consensus indexer converts data to CID

[marc-aurele-besner]

User description

Consensus indexer converts data to CID

This PR aims to minimize data utilization in consensus tables (especially extrinsics and events).
When we upload data on the network, using auto-drive or anything that follows auto-dag-data models, this data gets indexed by the consensus indexer as the argument of the extrinsic (args column in consensus.extrinsics table).
This data can be indexed in a second table (args column in consensus.events table), if the extrinsics use is system.remarkWithEvent.

Therefore these changes convert the data into a CID if possible and replace the args by { "cid": "bafkr6i..." } and add the CID in a new column.

• add auto-dag-data in consensus indexer and update other auto-sdk depencencies
• add cid column in extrinsics and events table (as index)
• add logic to try to parse the cid from the data and store it in db

---

PR Type

enhancement, configuration changes

---

Description

• Added CID handling to consensus indexer for extrinsics and events, minimizing data size by replacing args with CID where applicable.
• Updated database schema to include cid column and indexes in consensus.events and consensus.extrinsics tables.
• Introduced utility functions for CID parsing and hex conversion.
• Updated package dependencies to include auto-dag-data and updated versions of existing dependencies.
• Extended GraphQL schema to support CID field in Extrinsic and Event types.

---

Changes walkthrough 📝

	Relevant files
Enhancement	12 files db.ts Add CID parameter to database functions for extrinsics and events indexers/mainnet/consensus/src/mappings/db.ts Added cid parameter to createAndSaveExtrinsic and createAndSaveEvent functions. Updated function calls to include cid. +6/-2      helper.ts Implement CID parsing utility function                                      indexers/mainnet/consensus/src/mappings/helper.ts Imported CID-related functions from auto-dag-data. Added parseDataToCid function to parse data into CID. +18/-1    mappingHandlers.ts Integrate CID parsing into extrinsic and event handling    indexers/mainnet/consensus/src/mappings/mappingHandlers.ts Integrated parseDataToCid to handle extrinsics and events. Updated logic to replace args with CID if applicable. +35/-4    types.ts Define types for CID and modified arguments                            indexers/mainnet/consensus/src/mappings/types.ts Added new types Cid, ModifiedArgs, and ParsedArgs. +8/-0      utils.ts Add utility function for hex to Uint8Array conversion        indexers/mainnet/consensus/src/mappings/utils.ts Added hexToUint8Array utility function. +8/-0      db.ts Add CID parameter to database functions for extrinsics and events indexers/taurus/consensus/src/mappings/db.ts Added cid parameter to createAndSaveExtrinsic and createAndSaveEvent functions. Updated function calls to include cid. +6/-2      helper.ts Implement CID parsing utility function                                      indexers/taurus/consensus/src/mappings/helper.ts Imported CID-related functions from auto-dag-data. Added parseDataToCid function to parse data into CID. +18/-1    mappingHandlers.ts Integrate CID parsing into extrinsic and event handling    indexers/taurus/consensus/src/mappings/mappingHandlers.ts Integrated parseDataToCid to handle extrinsics and events. Updated logic to replace args with CID if applicable. +35/-4    types.ts Define types for CID and modified arguments                            indexers/taurus/consensus/src/mappings/types.ts Added new types Cid, ModifiedArgs, and ParsedArgs. +8/-0      utils.ts Add utility function for hex to Uint8Array conversion        indexers/taurus/consensus/src/mappings/utils.ts Added hexToUint8Array utility function. +8/-0      schema.graphql Extend GraphQL schema with CID field                                          indexers/mainnet/consensus/schema.graphql Added cid field to Extrinsic and Event types. +3/-1      schema.graphql Extend GraphQL schema with CID field                                          indexers/taurus/consensus/schema.graphql Added cid field to Extrinsic and Event types. +3/-1
Configuration changes	1 files init-db.sql Add CID column and indexes to database schema                        indexers/db/docker-entrypoint-initdb.d/init-db.sql Added cid column to consensus.events and consensus.extrinsics tables. Created indexes for cid in both tables. +4/-2
Dependencies	2 files package.json Update package dependencies for consensus indexer                indexers/mainnet/consensus/package.json Updated dependencies for auto-consensus, auto-dag-data, and auto-utils. +3/-2      package.json Update package dependencies for consensus indexer                indexers/taurus/consensus/package.json Updated dependencies for auto-consensus, auto-dag-data, and auto-utils. +3/-2

---

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

[netlify]

✅ Deploy Preview for dev-astral canceled.

Name	Link
🔨 Latest commit	e2e51f5
🔍 Latest deploy log	https://app.netlify.com/sites/dev-astral/deploys/673f40ed9cff8400082f389e

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Error Handling The error handling in parseDataToCid function might not be sufficient. Consider enhancing the error handling to manage different types of failures more gracefully. Input Validation The hexToUint8Array function throws an error for odd-length hex strings but does not handle other potential invalid inputs. Consider adding more comprehensive input validation.

[github-actions]

PR Code Suggestions ✨

No code suggestions found for the PR.

[socket-security]

New dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/@autonomys/auto-dag-data@1.0.8	None	+4	555 kB	clostao

View full report↗︎