Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: fetching jobs and collections from asset caches #1207

Merged
merged 16 commits into from
Jul 31, 2024

Conversation

Yashk767
Copy link
Contributor

Description

  • Stored jobs and collections in a cache and updating the cache on every collection/job update or creation.
  • Faster fetching [O(1)] of jobs from cache for every collection/job resulting in quicker execution of commit process.

Fixes https://linear.app/interstellar-research/issue/RAZ-713

How Has This Been Tested?

Tested with running a staker and creating create and update jobs/collections transactions and checked whether event listeners are listening correctly and updating the cache with the expected update value.

@Yashk767 Yashk767 force-pushed the yash/raz-713-cache-jobs-and-collections branch from 89dbc2a to 5dda2e9 Compare April 30, 2024 09:34
@Yashk767 Yashk767 marked this pull request as ready for review April 30, 2024 10:06
cmd/eventListeners.go Outdated Show resolved Hide resolved
cmd/eventListeners.go Outdated Show resolved Hide resolved
@Yashk767 Yashk767 requested a review from hrishikeshio July 19, 2024 11:34
cmd/commit.go Show resolved Hide resolved
@ashish10677 ashish10677 marked this pull request as draft July 25, 2024 11:35
@Yashk767 Yashk767 marked this pull request as ready for review July 26, 2024 10:55
@ashish10677 ashish10677 self-requested a review July 29, 2024 09:19
cmd/eventListeners.go Show resolved Hide resolved
cmd/eventListeners.go Show resolved Hide resolved
cmd/eventListeners.go Outdated Show resolved Hide resolved
core/types/vote.go Outdated Show resolved Hide resolved
utils/asset.go Outdated Show resolved Hide resolved
@ashish10677 ashish10677 self-requested a review July 31, 2024 12:38
@Yashk767 Yashk767 merged commit 20424c4 into feature/v1.2.0 Jul 31, 2024
3 checks passed
Yashk767 added a commit to Yashk767/razor-go that referenced this pull request Aug 9, 2024
* refactor: used the same keystore instance to fetch pKey everytime  (razor-network#1200)

* refactor: used regex instead of ks.Accounts to match keystore file

* refactor: added more extensive realistic testing for fetching of private key

* ci: ignore test-accounts directory in coverage

* refactor: added test when multiple timestamps keystore file present for same account

* refactor: covered lower and upper case addresses

* revert: optimised fetching key from keystore dir

* fix: reused same keystore instance for fetching pkey every time

* refactor: added realistic tests for fetching of pKey

* refactor: removed unused accounts() from interface

* refactor: used AccountManager struct with a keystore as a method caller

* refactor: fix tests

* refactor: removed commented test code

* refactor: returned interface instead of type struct instance from AccountManagerForKeystore()

* refactor: fixed create tests

* refactor: compared expected and returned pKey values in tests

* refactor: requested changes

* fix: changed keyword to const for defining constants (razor-network#1206)

* fix: changed keyword to const for defining constants

* fix: fixed lint error by adding explicit type for ProcessRequestRetryDelay

* refactor: updated RPC to op sepolia in testing post request

* refactor: used a common http client for all the requests (razor-network#1201)

* refactor: used a common http client throughout for all the APIs

* fix: handled goroutine exit

* refactor: golangci-lint fix

* refactor: defined http client fields in constants.go

* refactor: defined http client as a field to HttpClient struct used as a method caller

* refactor: revert InvokeFunctionWithTimeout() changes

* refator: fixed tests

* refactor: moved httpClient as the first parameter in api.go functions signature

* refactor: removed exporting of client field from HttpClient struct

* refactor: changed var to const keyword for http client constants

* refactor: Optimised number of `GetBlockNumber` and `GetStakerId` eth calls (razor-network#1204)

* refactor: Used getStakerId() only once

* refactor: used exisiting blockNumber from Vote() in dependent functions

* refactor: fixed tests after header being passed as a parameter

* fix: removed continous retries if error in fetching block number in logger

* refactor: added tests when context is cancelled for Vote()

* refactor: fixed vote tests

* fix: updated default config values to support both 20 min and shorter 5 min epoch  (razor-network#1209)

* fix: updated config default max, min values to support longer epoch

* fix: updated default config values in config.sh if builded from source

* feat: Implemented batching of multiple getStakeSnapshot RPC calls (razor-network#1218)

* feat: Implemented batching of multiple getStakeSnapshot RPC calls

* refactor: added method caller to batch call functions and added it to interface

* refactor: added new and fixed exisiting tests

* refactor: fixed log with correct staker Id associated for stake

* refactor: moved getStakeSnapshot string to constants

* refactor: implemented generic batch call function

* refactor: used generic batch call for getStakeSnapshot

* refactor: removed timeout and address parameter from GetBiggestStakeAnId()

* refactor: fetching jobs and collections from asset caches (razor-network#1207)

* feat: added jobs/collections cache structs

* feat: added support for using jobs/collections caches

* refactor: added time constants required for job/collection cache updates

* fix: fixed resetting of cache every expiry interval

* refactor: fixed error log

* refactor: fix utils tests

* refactor: added httpClient instance in commitParams struct

* refactor: removed resetting of cache as its not required

* feat: checked for job/collection events at the start of commit state in main go routine

* refactor: used FilterLogswithRetry and fixed other tests

* refactor: fixed benchmark

* refactor: moved all the event names to constants

* refactor: requested changes

* refactor: removed custom http client struct and directly used inbuilt http.Client

* refcator: updated default logFileMaxAge and logFileMaxBackups (razor-network#1221)

* chore: updated default deployment parameters and version for `v1.2.0` (razor-network#1222)

* chore: updated default addresses and chainId to mainnet addresses and chainId

* chore: updated testnet addresses to staging europa 20 min epoch

* chore: updated version to v1.2.0

* refactor: fixed tests due to state length change to 1200
Yashk767 added a commit that referenced this pull request Aug 9, 2024
* refactor: used the same keystore instance to fetch pKey everytime  (#1200)

* refactor: used regex instead of ks.Accounts to match keystore file

* refactor: added more extensive realistic testing for fetching of private key

* ci: ignore test-accounts directory in coverage

* refactor: added test when multiple timestamps keystore file present for same account

* refactor: covered lower and upper case addresses

* revert: optimised fetching key from keystore dir

* fix: reused same keystore instance for fetching pkey every time

* refactor: added realistic tests for fetching of pKey

* refactor: removed unused accounts() from interface

* refactor: used AccountManager struct with a keystore as a method caller

* refactor: fix tests

* refactor: removed commented test code

* refactor: returned interface instead of type struct instance from AccountManagerForKeystore()

* refactor: fixed create tests

* refactor: compared expected and returned pKey values in tests

* refactor: requested changes

* fix: changed keyword to const for defining constants (#1206)

* fix: changed keyword to const for defining constants

* fix: fixed lint error by adding explicit type for ProcessRequestRetryDelay

* refactor: updated RPC to op sepolia in testing post request

* refactor: used a common http client for all the requests (#1201)

* refactor: used a common http client throughout for all the APIs

* fix: handled goroutine exit

* refactor: golangci-lint fix

* refactor: defined http client fields in constants.go

* refactor: defined http client as a field to HttpClient struct used as a method caller

* refactor: revert InvokeFunctionWithTimeout() changes

* refator: fixed tests

* refactor: moved httpClient as the first parameter in api.go functions signature

* refactor: removed exporting of client field from HttpClient struct

* refactor: changed var to const keyword for http client constants

* refactor: Optimised number of `GetBlockNumber` and `GetStakerId` eth calls (#1204)

* refactor: Used getStakerId() only once

* refactor: used exisiting blockNumber from Vote() in dependent functions

* refactor: fixed tests after header being passed as a parameter

* fix: removed continous retries if error in fetching block number in logger

* refactor: added tests when context is cancelled for Vote()

* refactor: fixed vote tests

* fix: updated default config values to support both 20 min and shorter 5 min epoch  (#1209)

* fix: updated config default max, min values to support longer epoch

* fix: updated default config values in config.sh if builded from source

* feat: Implemented batching of multiple getStakeSnapshot RPC calls (#1218)

* feat: Implemented batching of multiple getStakeSnapshot RPC calls

* refactor: added method caller to batch call functions and added it to interface

* refactor: added new and fixed exisiting tests

* refactor: fixed log with correct staker Id associated for stake

* refactor: moved getStakeSnapshot string to constants

* refactor: implemented generic batch call function

* refactor: used generic batch call for getStakeSnapshot

* refactor: removed timeout and address parameter from GetBiggestStakeAnId()

* refactor: fetching jobs and collections from asset caches (#1207)

* feat: added jobs/collections cache structs

* feat: added support for using jobs/collections caches

* refactor: added time constants required for job/collection cache updates

* fix: fixed resetting of cache every expiry interval

* refactor: fixed error log

* refactor: fix utils tests

* refactor: added httpClient instance in commitParams struct

* refactor: removed resetting of cache as its not required

* feat: checked for job/collection events at the start of commit state in main go routine

* refactor: used FilterLogswithRetry and fixed other tests

* refactor: fixed benchmark

* refactor: moved all the event names to constants

* refactor: requested changes

* refactor: removed custom http client struct and directly used inbuilt http.Client

* refcator: updated default logFileMaxAge and logFileMaxBackups (#1221)

* chore: updated default deployment parameters and version for `v1.2.0` (#1222)

* chore: updated default addresses and chainId to mainnet addresses and chainId

* chore: updated testnet addresses to staging europa 20 min epoch

* chore: updated version to v1.2.0

* refactor: fixed tests due to state length change to 1200

* refactor: removed redundant and incorrect log for staker id (#1226)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants