[PAY-3881][PAY-3883] Add indexing/endpoints for separated collectibles data

[schottra]

Description

This is the discovery-side work for splitting collectibles info out of cid_data. Since the primary goal is to get us off cid_data, I opted not to do a big redesign of how we store or return this info and left it in the format we use inside of cid_data.

• Added a table to store the raw collectibles data objects that would normally live inside cid_data for users.
  • This is a 1:1 relationship so user_id is the primary key
  • Using the same blocknumber foreign key behavior to allow us to revert/cascade delete.
• Added a migration to pull out the existing collectibles objects into rows for their corresponding users. This is < 400 users on prod at the moment.
• Added endpoint for fetching collectibles for a given user. It will just look up the data and return the object or null
• Added indexing logic for creating/updating. Either action is supported since they both result in the same operation.
  • There is no support for Delete. The existing paradigm is to just set it to an empty object.
  • Similar to the existing user update indexing, we will set has_collectibles on the user based on whether the data has any keys in it
  • User update indexing will also update the collectibles_data table for now until we remove support for that (which will be a few client releases after we start using these new transactions)

How Has This Been Tested?

• Integration tests
• Used local client against local dev to send existing user update transactions and new style of transactions to verify that both methods are supported.

[changeset-bot]

🦋 Changeset detected

Latest commit: 0138d35

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@audius/sdk	Minor
@audius/sdk-legacy	Patch
@audius/sp-actions	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[raymondjacobson]

LGTM, though I do think we need to probably have some logic in here

audius-protocol/packages/discovery-provider/src/tasks/index_nethermind.py

Line 414 in 90ff3da

def revert_block(session: Session, block_to_revert: Block):

We might not handle reverts elegantly in general, and this actually should go away once we're on core b/c of instant finality. Anyway... just putting for posterity

[schottra]

LGTM, though I do think we need to probably have some logic in here

audius-protocol/packages/discovery-provider/src/tasks/index_nethermind.py

Line 414 in 90ff3da

def revert_block(session: Session, block_to_revert: Block):

We might not handle reverts elegantly in general, and this actually should go away once we're on core b/c of instant finality. Anyway... just putting for posterity

If I understand how revert works (and it's possible I don't), I think it's handling it correctly. Since we're using the add_records functionality and we have a defined entity type, it should generate an entry for it in the revert_blocks row at the end of processing the block. And we use cascade delete on the new table to make sure we remove any records that were inserted when the block is deleted.
I think the only specialized logic that is left in revert_blocks is for tracks/playlists which are still using is_current?

But I will take a deep dive on it and see if I can verify that it's doing what we need it to.

[gitguardian]

⚠️ GitGuardian has uncovered 4 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
11648676	Triggered	Generic High Entropy Secret	e6eda81	dev-tools/environment/audiusd-1.env	View secret
11648678	Triggered	Generic High Entropy Secret	e6eda81	dev-tools/environment/audiusd-4.env	View secret
11648679	Triggered	Generic High Entropy Secret	e6eda81	dev-tools/environment/audiusd-3.env	View secret
11648680	Triggered	Generic High Entropy Secret	e6eda81	dev-tools/environment/audiusd-2.env	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secrets safely. Learn here the best practices.
3. Revoke and rotate these secrets.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}