Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: prototype versioned app module upgrades #3729

Draft
wants to merge 142 commits into
base: main
Choose a base branch
from

Conversation

rootulp
Copy link
Collaborator

@rootulp rootulp commented Jul 24, 2024

Closes #3625

Issues encountered

  1. Modules register errors via errors.Register. Since the multiplexer imports multiple celestia-app versions, all errors get registered twice resulting in a: panic: error with code 2 is already registered: "duplicate". To resolve, we need to:
    1. Make errors unique so that they don't conflict across app versions
    2. Modify errors.Register to not panic if a duplicate error is registered. Did this in https://github.com/rootulp/cosmos-sdk/releases/tag/errors%2Fv1.4.0
  2. panic: concrete type *types.MsgPayForBlobs has already been registered under typeURL /, cannot register *types.MsgRegisterEVMAddress under same typeURL. This usually means that there are conflicting modules registering different concrete types for a same interface implementation.
    1. Option A: bump the proto package versions (did in this PR)
    2. Option B: Create a unique go.mod for each package and de-duplicate imports. Note: we'll have to extract the x/blob package from this repo so that it isn't present in the root-level celestia-app go module. Use the v2 version because it has types that aren't present in the v1 version.
    3. Option C: Make the registries distinct per state machine

Remaining tasks

  • Remove v3 upgrade height because the upgrade from v2 -> v3 will use the signaling mechanism.
  • Extract the testnode refactors to a separate PR or remove them entirely
  • Update this branch to latest on main. Extract the previous changes to a new v2 branch.
  • Revert the unnecessary replace statements in celestia-app/node/go.mod
  • Add a logger to multiplexer. Convert print statements to logger.Debug statements.
  • Figure out a long-term solution so that two state machines (v2 and v3) can invoke config.Seal() or move it to some place where it is only invoked once. Resolved via feat: expose config.IsSealed cosmos-sdk#414
  • Rename node to multiplexer
  • Figure out a long term solution for proto conflicts for SDK registry.
  • Remove the diff in v2 that extends the range for all modules to v3. Why do module migrations need to occur in Commit of the block height before the upgrade height? In order to satisfy that constraint, the v2 state machine has to be updated so that modules support the range v1 to v3. If we remove v2 supporting v3 modules then we hit this error
  • Update Multiplexer to not hardcode v2 app version initialization and instead run Info() get the version and then start that state machine
  • Support module migrations
  • Modify the multiplexer CLI so that it replaces the existing celestia-appd start CLI

FLUPs

  • Add unit tests
  • Add e2e tests
  • For v4: remove most of the app version checks, the versioned constants, the module manager, etc.

Notes

  • To run this locally you need the v2.x locally then from the repo root:
make build-node && ./build/node

Screenshot

multiplexer

@rootulp rootulp self-assigned this Jul 24, 2024
$ ./node
panic: error with code 2 is already registered: "duplicate"

goroutine 1 [running]:
cosmossdk.io/errors.RegisterWithGRPCCode({0x105b93b3f, 0x3}, 0x2, 0x2, {0x105b9a8f5, 0x9})
	/Users/rootulp/go/pkg/mod/cosmossdk.io/errors@v1.0.1/errors.go:43 +0x240
cosmossdk.io/errors.Register(...)
	/Users/rootulp/go/pkg/mod/cosmossdk.io/errors@v1.0.1/errors.go:35
github.com/celestiaorg/celestia-app/x/qgb/types.init()
	/Users/rootulp/go/pkg/mod/github.com/celestiaorg/celestia-app@v1.13.0/x/qgb/types/errors.go:8 +0x128
@rootulp

This comment was marked as resolved.

@rootulp
Copy link
Collaborator Author

rootulp commented Jul 24, 2024

New issue:

panic: concrete type *types.MsgPayForBlobs has already been registered under typeURL /, cannot register *types.MsgRegisterEVMAddress under same typeURL. This usually means that there are conflicting modules registering different concrete types for a same interface implementation

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/codec/types.(*interfaceRegistry).registerImpl(0x14000181dd0, {0x104623f80?, 0x0?}, {0x104536430, 0x1}, {0x104aaaab0, 0x1400126c9c0})
	/Users/rootulp/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.23.0-sdk-v0.46.16/codec/types/interface_registry.go:151 +0x2ec
github.com/cosmos/cosmos-sdk/codec/types.(*interfaceRegistry).RegisterImplementations(0x14000181dd0, {0x104623f80, 0x0}, {0x140012dc4b0?, 0x104abe2e8?, 0x14000181dd0?})
	/Users/rootulp/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.23.0-sdk-v0.46.16/codec/types/interface_registry.go:115 +0x84
github.com/celestiaorg/celestia-app/x/qgb/types.RegisterInterfaces({0x104abe2e8, 0x14000181dd0})
	/Users/rootulp/go/pkg/mod/github.com/celestiaorg/celestia-app@v1.13.0/x/qgb/types/codec.go:17 +0x9c
github.com/celestiaorg/celestia-app/x/qgb.AppModuleBasic.RegisterInterfaces(...)
	/Users/rootulp/go/pkg/mod/github.com/celestiaorg/celestia-app@v1.13.0/x/qgb/module.go:58
github.com/celestiaorg/celestia-app/app/encoding.MakeConfig({0x1400107ea00, 0x14, 0x10466f380?})
	/Users/rootulp/go/pkg/mod/github.com/celestiaorg/celestia-app@v1.13.0/app/encoding/encoding.go:37 +0x1cc
github.com/celestiaorg/celestia-app/node/utils.NewAppV1()
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/node/utils/apps.go:27 +0x170
github.com/celestiaorg/celestia-app/node/utils.GetApps()
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/node/utils/apps.go:14 +0x1c
github.com/celestiaorg/celestia-app/node/cmd.init.func1(0x1400078b900?, {0x10390c42e?, 0x4?, 0x10390c432?})
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/node/cmd/root.go:29 +0x24
github.com/spf13/cobra.(*Command).execute(0x105d7b520, {0x14000114060, 0x0, 0x0})
	/Users/rootulp/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0x840
github.com/spf13/cobra.(*Command).ExecuteC(0x105d7b520)
	/Users/rootulp/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/rootulp/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/celestiaorg/celestia-app/node/cmd.Execute()
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/node/cmd/root.go:46 +0x24
main.main()
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/node/main.go:6 +0x1c

@cmwaters
Copy link
Contributor

This was the error that I first got when trying to unfork the SDK. There is a global proto registry and thus when we import multiple versions we end up with this panic. One option would be to go mod the modules that way there's not two copies of x/blob for instance but both state machines point to the same version. The other option would be to modify the register code such that if it saw the same type url name, it would check the fields (using reflect maybe) and so long as they were exactly the same it would ignore the panic

@rootulp
Copy link
Collaborator Author

rootulp commented Jul 25, 2024

Thanks for your help!

One option would be to go mod the modules that way there's not two copies of x/blob for instance but both state machines point to the same version.

What if celestia-app v3 needs x/blob/v2 and celestia-app v1 needs x/blob/v1, then I think we'll hit the same issue where both modules try to register the same types in the global proto registry.

The other option would be to modify the register code such that if it saw the same type url name, it would check the fields (using reflect maybe) and so long as they were exactly the same it would ignore the panic

This one seems more hacky but potentially do-able. The concern is if celestia-app v2's x/blob actually does modify a field from a message type and the two types conflict on MsgTypeUrl but differ on fields. IMO if an SDK message's fields do change, that would be a breaking change of the x/blob package which we can workaround by defining a v2 for that type and module. In other words if MsgPayForBlobs changes we need to introduce MsgPayForBlobsV2 with a unique MsgTypeURL.

@cmwaters
Copy link
Contributor

What if celestia-app v3 needs x/blob/v2 and celestia-app v1 needs x/blob/v1, then I think we'll hit the same issue where both modules try to register the same types in the global proto registry.

If we created a x/blob/2 we would bump the proto version so the msg url would look like celestia.blob.v2.MsgPayForBlob

@rootulp
Copy link
Collaborator Author

rootulp commented Jul 31, 2024

I've tried a few times to create a Go module from x/blob but the gopls in VSCode messes up the import paths. Currently hitting:

$ make build-node
--> Building celestia-app/node and outputting binary to build/node
# github.com/celestiaorg/celestia-app/v2/app/ante
../app/ante/ante.go:69:12: undefined: blobante.NewMaxTotalBlobSizeDecorator
../app/ante/ante.go:73:12: undefined: blobante.NewBlobShareDecorator

error while importing github.com/celestiaorg/celestia-app/v2/app: ambiguous import: found package github.com/celestiaorg/celestia-app/x/blob in multiple modules:
github.com/celestiaorg/celestia-app v1.13.0 (/Users/rootulp/go/pkg/mod/github.com/celestiaorg/celestia-app@v1.13.0/x/blob)
github.com/celestiaorg/celestia-app/x/blob v0.0.0 (/Users/rootulp/git/rootulp/celestiaorg/celestia-app/x/blob)
@rootulp
Copy link
Collaborator Author

rootulp commented Aug 1, 2024

I thought the issue might be because v1.x has an x/blob that isn't a Go module yet so I repeated the steps above on the v1.x branch and hit this issue:

go: github.com/celestiaorg/celestia-app/x/blob: ambiguous import: found package github.com/celestiaorg/celestia-app/x/blob in multiple modules:
	github.com/celestiaorg/celestia-app v1.14.0 (/Users/rootulp/go/pkg/mod/github.com/celestiaorg/celestia-app@v1.14.0/x/blob)
	github.com/celestiaorg/celestia-app/x/blob (/Users/rootulp/git/rootulp/celestiaorg/celestia-app/x/blob)
go: github.com/celestiaorg/celestia-app/x/blob/ante: ambiguous import: found package github.com/celestiaorg/celestia-app/x/blob/ante in multiple modules:
	github.com/celestiaorg/celestia-app v1.14.0 (/Users/rootulp/go/pkg/mod/github.com/celestiaorg/celestia-app@v1.14.0/x/blob/ante)
	github.com/celestiaorg/celestia-app/x/blob (/Users/rootulp/git/rootulp/celestiaorg/celestia-app/x/blob/ante)
go: github.com/celestiaorg/celestia-app/x/blob/client/cli: ambiguous import: found package github.com/celestiaorg/celestia-app/x/blob/client/cli in multiple modules:
	github.com/celestiaorg/celestia-app v1.14.0 (/Users/rootulp/go/pkg/mod/github.com/celestiaorg/celestia-app@v1.14.0/x/blob/client/cli)
	github.com/celestiaorg/celestia-app/x/blob (/Users/rootulp/git/rootulp/celestiaorg/celestia-app/x/blob/client/cli)
go: github.com/celestiaorg/celestia-app/x/blob/client/testutil: ambiguous import: found package github.com/celestiaorg/celestia-app/x/blob/client/testutil in multiple modules:
	github.com/celestiaorg/celestia-app v1.14.0 (/Users/rootulp/go/pkg/mod/github.com/celestiaorg/celestia-app@v1.14.0/x/blob/client/testutil)
	github.com/celestiaorg/celestia-app/x/blob (/Users/rootulp/git/rootulp/celestiaorg/celestia-app/x/blob/client/testutil)
go: github.com/celestiaorg/celestia-app/x/blob/keeper: ambiguous import: found package github.com/celestiaorg/celestia-app/x/blob/keeper in multiple modules:
	github.com/celestiaorg/celestia-app v1.14.0 (/Users/rootulp/go/pkg/mod/github.com/celestiaorg/celestia-app@v1.14.0/x/blob/keeper)
	github.com/celestiaorg/celestia-app/x/blob (/Users/rootulp/git/rootulp/celestiaorg/celestia-app/x/blob/keeper)
go: github.com/celestiaorg/celestia-app/x/blob/types: ambiguous import: found package github.com/celestiaorg/celestia-app/x/blob/types in multiple modules:
	github.com/celestiaorg/celestia-app v1.14.0 (/Users/rootulp/go/pkg/mod/github.com/celestiaorg/celestia-app@v1.14.0/x/blob/types)
	github.com/celestiaorg/celestia-app/x/blob (/Users/rootulp/git/rootulp/celestiaorg/celestia-app/x/blob/types)
go: github.com/celestiaorg/celestia-app/x/blob/types/test: ambiguous import: found package github.com/celestiaorg/celestia-app/x/blob/types/test in multiple modules:
	github.com/celestiaorg/celestia-app v1.14.0 (/Users/rootulp/go/pkg/mod/github.com/celestiaorg/celestia-app@v1.14.0/x/blob/types/test)
	github.com/celestiaorg/celestia-app/x/blob (/Users/rootulp/git/rootulp/celestiaorg/celestia-app/x/blob/types/test)

I tried cleaning the go mod cache and a few different variations of importing the x/blob package from the root go.mod. See https://github.com/rootulp/celestia-app/tree/rp/go-mod-x-blob

@rootulp
Copy link
Collaborator Author

rootulp commented Aug 1, 2024

The original error message

panic: concrete type *types.MsgPayForBlobs has already been registered under typeURL /, cannot register *types.MsgRegisterEVMAddress under same typeURL. This usually means that there are conflicting modules registering different concrete types for a same interface implementation

makes me think that both the types MsgPayForBlobs and MsgRegisterEVMAddress don't have a proto.MessageName defined because this line should suffix it to / so I would've expected to his this when registering both of those types in just one application.

@cmwaters
Copy link
Contributor

cmwaters commented Aug 2, 2024

MsgPayForBlobs and MsgRegisterEVMAddress don't have a proto.MessageName defined

That's strange. This isn't a problem in main. These two would be registered under different names

@rootulp
Copy link
Collaborator Author

rootulp commented Oct 21, 2024

$ ./scripts/single-node.sh
celestia-app home: /Users/rootulp/.celestia-app
celestia-app version: 1.28.0-140-g28c8edde3

Do you want to delete existing /Users/rootulp/.celestia-app and start a new local testnet? [y/n]
y
Deleting /Users/rootulp/.celestia-app...
Initializing validator and node config files...
Adding a new key to the keyring...
Adding genesis account...
Creating a genesis tx...
Collecting genesis txs...
Tracing is set up with the ability to pull traced data from the node on the address http://127.0.0.1:26661
Starting celestia-app...
panic: failed to load state at height 0; no commit info found (latest height: 0): invalid request

@rootulp
Copy link
Collaborator Author

rootulp commented Oct 21, 2024

Hitting an error I resolved earlier on this PR

$ ./build/node

$ ./build/node
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.core.v1.da.DataAvailabilityHeader
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.core.v1.proof.ShareProof
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.core.v1.proof.RowProof
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.core.v1.proof.NMTProof
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.core.v1.proof.Proof
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.core.v1.tx.TxStatusRequest
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.core.v1.tx.TxStatusResponse
warning: error with code 1 is already registered: "invalid signal version because signal version can not be less than the current version"
warning: error with code 3 is already registered: "invalid upgrade version"
warning: error with code 2 is already registered: "upgrade is already pending"
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.signal.v1.QueryVersionTallyRequest
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.signal.v1.QueryVersionTallyResponse
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.signal.v1.QueryGetUpgradeRequest
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.signal.v1.QueryGetUpgradeResponse
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.signal.v1.MsgSignalVersion
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.signal.v1.MsgSignalVersionResponse
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.signal.v1.MsgTryUpgrade
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.signal.v1.MsgTryUpgradeResponse
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.signal.v1.Upgrade
warning: error with code 2 is already registered: "duplicate"
warning: error with code 6 is already registered: "empty"
warning: error with code 12 is already registered: "no bonded validators in active set"
warning: error with code 13 is already registered: "invalid validator address in current valset %v"
warning: error with code 14 is already registered: "discovered invalid EVM address stored for validator %v"
warning: error with code 15 is already registered: "generated invalid valset"
warning: error with code 16 is already registered: "attestation is not a valset request"
warning: error with code 18 is already registered: "attestation not found"
warning: error with code 22 is already registered: "nil attestation"
warning: error with code 26 is already registered: "couldn't unmarshall attestation from store"
warning: error with code 27 is already registered: "the provided nonce is higher than the latest attestation nonce"
warning: error with code 28 is already registered: "there is no valset before attestation nonce 1"
warning: error with code 29 is already registered: "no data commitment has been generated for the provided height"
warning: error with code 30 is already registered: "data commitment not found"
warning: error with code 31 is already registered: "the latest attestation nonce has still not been defined in store"
warning: error with code 32 is already registered: "invalid data commitment window"
warning: error with code 33 is already registered: "the earliest available nonce after pruning has still not been defined in store"
warning: error with code 34 is already registered: "the requested nonce has been pruned"
warning: error with code 35 is already registered: "unknown attestation type"
warning: error with code 36 is already registered: "the provided evm address is not a valid hex address"
warning: error with code 37 is already registered: "the provided evm address already exists"
warning: error with code 38 is already registered: "EVM address not found"
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.qgb.v1.MsgRegisterEVMAddress
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.qgb.v1.MsgRegisterEVMAddressResponse
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.qgb.v1.BridgeValidator
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.qgb.v1.Valset
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.qgb.v1.DataCommitment
warning: error with code 11110 is already registered: "cannot use reserved namespace IDs"
warning: error with code 11111 is already registered: "invalid namespace length"
warning: error with code 11112 is already registered: "data must be multiple of shareSize"
warning: error with code 11113 is already registered: "actual blob size differs from that specified in the MsgPayForBlob"
warning: error with code 11114 is already registered: "committed to invalid square size: must be power of two"
warning: error with code 11115 is already registered: "unexpected error calculating commitment for share"
warning: error with code 11116 is already registered: "invalid commitment for share"
warning: error with code 11117 is already registered: "cannot use parity shares namespace ID"
warning: error with code 11118 is already registered: "cannot use tail padding namespace ID"
warning: error with code 11119 is already registered: "cannot use transaction namespace ID"
warning: error with code 11122 is already registered: "invalid share commitments: all relevant square sizes must be committed to"
warning: error with code 11123 is already registered: "unsupported share version"
warning: error with code 11124 is already registered: "cannot use zero blob size"
warning: error with code 11125 is already registered: "mismatched number of blobs per MsgPayForBlob"
warning: error with code 11126 is already registered: "no MsgPayForBlobs found in blob transaction"
warning: error with code 11127 is already registered: "namespace of blob and its respective MsgPayForBlobs differ"
warning: error with code 11128 is already registered: "failure to parse a transaction from its protobuf representation"
warning: error with code 11129 is already registered: "not yet supported: multiple sdk.Msgs found in BlobTx"
warning: error with code 11130 is already registered: "number of each component in a MsgPayForBlobs must be identical"
warning: error with code 11131 is already registered: "no blobs provided"
warning: error with code 11132 is already registered: "no namespaces provided"
warning: error with code 11133 is already registered: "no share versions provided"
warning: error with code 11134 is already registered: "no blob sizes provided"
warning: error with code 11135 is already registered: "no share commitments provided"
warning: error with code 11136 is already registered: "invalid namespace"
warning: error with code 11137 is already registered: "invalid namespace version"
warning: error with code 11138 is already registered: "total blob size too large"
warning: error with code 11139 is already registered: "blob(s) too large"
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.blob.v1.EventPayForBlobs
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.blob.v1.GenesisState
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.blob.v1.Params
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.blob.v1.QueryParamsRequest
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.blob.v1.QueryParamsResponse
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.blob.v1.MsgPayForBlobs
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.blob.v1.MsgPayForBlobsResponse
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.mint.v1.GenesisState
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.mint.v1.Minter
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.mint.v1.GenesisTime
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.mint.v1.QueryInflationRateRequest
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.mint.v1.QueryInflationRateResponse
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.mint.v1.QueryAnnualProvisionsRequest
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.mint.v1.QueryAnnualProvisionsResponse
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.mint.v1.QueryGenesisTimeRequest
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.mint.v1.QueryGenesisTimeResponse
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.minfee.v1.GenesisState
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.minfee.v1.QueryNetworkMinGasPrice
2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.minfee.v1.QueryNetworkMinGasPriceResponse
warning: error with code 91710 is already registered: "parameter can not be modified"
panic: concrete type *types.MsgTryUpgrade has already been registered under typeURL /, cannot register *types.MsgSignalVersion under same typeURL. This usually means that there are conflicting modules registering different concrete types for a same interface implementation

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/codec/types.(*interfaceRegistry).registerImpl(0x1400000f578, {0x106b03280?, 0x0?}, {0x1069f36f0, 0x1}, {0x107004a30, 0x1400000f920})
	/Users/rootulp/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.25.0-sdk-v0.46.16/codec/types/interface_registry.go:151 +0x2ec
github.com/cosmos/cosmos-sdk/codec/types.(*interfaceRegistry).RegisterImplementations(0x1400000f578, {0x106b03280, 0x0}, {0x140004df290?, 0xffffffffffffffff?, 0x1?})
	/Users/rootulp/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.25.0-sdk-v0.46.16/codec/types/interface_registry.go:115 +0x84
github.com/celestiaorg/celestia-app/v3/x/signal/types.RegisterInterfaces({0x107018148, 0x1400000f578})
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/x/signal/types/codec.go:21 +0x110
github.com/celestiaorg/celestia-app/v3/x/signal.AppModuleBasic.RegisterInterfaces(...)
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/x/signal/module.go:76
github.com/cosmos/cosmos-sdk/types/module.BasicManager.RegisterInterfaces(0x140003a02e0?, {0x107018148, 0x1400000f578})
	/Users/rootulp/go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.25.0-sdk-v0.46.16/types/module/module.go:84 +0x74
github.com/celestiaorg/celestia-app/v3/app/encoding.MakeConfig({0x14000799428, 0x1, 0x6?})
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/app/encoding/encoding.go:37 +0x1cc
github.com/celestiaorg/celestia-app/v3/test/util/genesis.NewDefaultGenesis()
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/test/util/genesis/genesis.go:64 +0x4c
github.com/celestiaorg/celestia-app/v3/test/util/testnode.DefaultConfig()
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/test/util/testnode/config.go:128 +0x34
github.com/celestiaorg/celestia-app/node/utils.GetConfig()
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/node/utils/config.go:19 +0x1b4
github.com/celestiaorg/celestia-app/node/cmd.init.func1(0x140004b1d00?, {0x105fb8009?, 0x4?, 0x105fb800d?})
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/node/cmd/root.go:22 +0x38
github.com/spf13/cobra.(*Command).execute(0x1084669a0, {0x14000112050, 0x0, 0x0})
	/Users/rootulp/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0x834
github.com/spf13/cobra.(*Command).ExecuteC(0x1084669a0)
	/Users/rootulp/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/rootulp/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/celestiaorg/celestia-app/node/cmd.Execute()
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/node/cmd/root.go:56 +0x24
main.main()
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/node/main.go:6 +0x1c

@rootulp
Copy link
Collaborator Author

rootulp commented Oct 21, 2024

Problematic code in Cosmos SDK lives here.

@rootulp
Copy link
Collaborator Author

rootulp commented Oct 22, 2024

Added some debug logs to Cosmos SDK. Strange that registerImpl is being invoked with a typeURL of empty string.

registerImpl: <nil>, /cosmos.tx.v1beta1.Tx,
registerImpl: <nil>, /cosmos.crypto.ed25519.PubKey, PubKeyEd25519{}
registerImpl: <nil>, /cosmos.crypto.secp256k1.PubKey, PubKeySecp256k1{}
registerImpl: <nil>, /cosmos.crypto.multisig.LegacyAminoPubKey,
registerImpl: <nil>, /cosmos.crypto.secp256k1.PrivKey,
registerImpl: <nil>, /cosmos.crypto.ed25519.PrivKey,
registerImpl: <nil>, /cosmos.crypto.secp256r1.PubKey, %!v(PANIC=String method: runtime error: invalid memory address or nil pointer dereference)
registerImpl: <nil>, /,
registerImpl: <nil>, /,
panic: concrete type *types.MsgTryUpgrade has already been registered under typeURL /, cannot register *types.MsgSignalVersion under same typeURL. This usually means that there are conflicting modules registering different concrete types for a same interface implementation

@rootulp
Copy link
Collaborator Author

rootulp commented Oct 22, 2024

More logs:

RegisterImplementations: <nil>, []
RegisterImplementations: <nil>, []
RegisterImplementations: <nil>, []
RegisterImplementations: <nil>, []
impl: , typeURL /cosmos.tx.v1beta1.Tx
registerImpl: <nil>, /cosmos.tx.v1beta1.Tx,
RegisterImplementations: <nil>, []
RegisterImplementations: <nil>, []
RegisterImplementations: <nil>, [PubKeyEd25519{}]
impl: PubKeyEd25519{}, typeURL /cosmos.crypto.ed25519.PubKey
registerImpl: <nil>, /cosmos.crypto.ed25519.PubKey, PubKeyEd25519{}
RegisterImplementations: <nil>, [PubKeySecp256k1{}]
impl: PubKeySecp256k1{}, typeURL /cosmos.crypto.secp256k1.PubKey
registerImpl: <nil>, /cosmos.crypto.secp256k1.PubKey, PubKeySecp256k1{}
RegisterImplementations: <nil>, []
impl: , typeURL /cosmos.crypto.multisig.LegacyAminoPubKey
registerImpl: <nil>, /cosmos.crypto.multisig.LegacyAminoPubKey,
RegisterImplementations: <nil>, []
RegisterImplementations: <nil>, []
impl: , typeURL /cosmos.crypto.secp256k1.PrivKey
registerImpl: <nil>, /cosmos.crypto.secp256k1.PrivKey,
RegisterImplementations: <nil>, []
impl: , typeURL /cosmos.crypto.ed25519.PrivKey
registerImpl: <nil>, /cosmos.crypto.ed25519.PrivKey,
RegisterImplementations: <nil>, [%!v(PANIC=String method: runtime error: invalid memory address or nil pointer dereference)]
impl: %!v(PANIC=String method: runtime error: invalid memory address or nil pointer dereference), typeURL /cosmos.crypto.secp256r1.PubKey
registerImpl: <nil>, /cosmos.crypto.secp256r1.PubKey, %!v(PANIC=String method: runtime error: invalid memory address or nil pointer dereference)
RegisterImplementations: <nil>, [ ]
impl: , typeURL /cosmos.bank.v1beta1.MsgSend
registerImpl: <nil>, /cosmos.bank.v1beta1.MsgSend,
impl: , typeURL /cosmos.bank.v1beta1.MsgMultiSend
registerImpl: <nil>, /cosmos.bank.v1beta1.MsgMultiSend,
RegisterImplementations: <nil>, []
impl: , typeURL /cosmos.bank.v1beta1.SendAuthorization
registerImpl: <nil>, /cosmos.bank.v1beta1.SendAuthorization,
RegisterImplementations: <nil>, [<nil>]
impl: <nil>, typeURL /cosmos.bank.v1beta1.MsgSend
registerImpl: <nil>, /cosmos.bank.v1beta1.MsgSend, <nil>
RegisterImplementations: <nil>, [<nil>]
impl: <nil>, typeURL /cosmos.bank.v1beta1.MsgSendResponse
registerImpl: <nil>, /cosmos.bank.v1beta1.MsgSendResponse, <nil>
RegisterImplementations: <nil>, [<nil>]
impl: <nil>, typeURL /cosmos.bank.v1beta1.MsgMultiSend
registerImpl: <nil>, /cosmos.bank.v1beta1.MsgMultiSend, <nil>
RegisterImplementations: <nil>, [<nil>]
impl: <nil>, typeURL /cosmos.bank.v1beta1.MsgMultiSendResponse
registerImpl: <nil>, /cosmos.bank.v1beta1.MsgMultiSendResponse, <nil>
RegisterImplementations: <nil>, []
impl: , typeURL /cosmos.bank.v1beta1.Supply
registerImpl: <nil>, /cosmos.bank.v1beta1.Supply,
RegisterImplementations: <nil>, []
impl: , typeURL /cosmos.crisis.v1beta1.MsgVerifyInvariant
registerImpl: <nil>, /cosmos.crisis.v1beta1.MsgVerifyInvariant,
RegisterImplementations: <nil>, [<nil>]
impl: <nil>, typeURL /cosmos.crisis.v1beta1.MsgVerifyInvariant
registerImpl: <nil>, /cosmos.crisis.v1beta1.MsgVerifyInvariant, <nil>
RegisterImplementations: <nil>, [<nil>]
impl: <nil>, typeURL /cosmos.crisis.v1beta1.MsgVerifyInvariantResponse
registerImpl: <nil>, /cosmos.crisis.v1beta1.MsgVerifyInvariantResponse, <nil>
RegisterImplementations: <nil>, []
impl: , typeURL /
registerImpl: <nil>, /,
RegisterImplementations: <nil>, []
RegisterImplementations: <nil>, [account_number: "0"
address: ""
pub_key: null
sequence: "0"
]
impl: account_number: "0"
address: ""
pub_key: null
sequence: "0"
, typeURL /cosmos.auth.v1beta1.BaseAccount
registerImpl: <nil>, /cosmos.auth.v1beta1.BaseAccount, account_number: "0"
address: ""
pub_key: null
sequence: "0"

RegisterImplementations: <nil>, [<nil>]
impl: <nil>, typeURL /celestia.blob.v1.MsgPayForBlobs
registerImpl: <nil>, /celestia.blob.v1.MsgPayForBlobs, <nil>
RegisterImplementations: <nil>, [<nil>]
impl: <nil>, typeURL /celestia.blob.v1.MsgPayForBlobsResponse
registerImpl: <nil>, /celestia.blob.v1.MsgPayForBlobsResponse, <nil>
RegisterImplementations: <nil>, []
impl: , typeURL /
registerImpl: <nil>, /,
panic: concrete type *types.MsgPayForBlobs has already been registered under typeURL /, cannot register *types.MsgRegisterEVMAddress under same typeURL. This usually means that there are conflicting modules registering different concrete types for a same interface implementation

@rootulp
Copy link
Collaborator Author

rootulp commented Oct 22, 2024

To resolve we can extract each state machine module into it's own Go module and refactor the state machines to import the newly extracted Go module. The dependency graph would look something like this

diagram

@rootulp
Copy link
Collaborator Author

rootulp commented Oct 22, 2024

In order to extract x/minfee we need to refactor it to not import celestia-app/v3.

@rootulp
Copy link
Collaborator Author

rootulp commented Oct 22, 2024

I extracted a new Go module for x/minfee, refactored the existing PR and v2.x branch to use the extracted x/minfee module and now there are no more duplicate proto registry logs:

-2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.minfee.v1.GenesisState
-2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.minfee.v1.QueryNetworkMinGasPrice
-2024/10/21 15:50:57 proto: duplicate proto type registered: celestia.minfee.v1.QueryNetworkMinGasPriceResponse

so I think we can pursue this path.

Note: we need to either:

  1. Extract the state machine modules to a new repo so that they can be imported by a user via an unambiguous path.
  2. Publish multiple Go modules from the same repo. This seems preferable although a bit more tricky to set up.

Next steps

  1. Repeat the x/minfee extraction on my fork and a new branch separate from the multiplexed app work.
  2. Create releases of x/minfee and celestia-app in order to verify that celestia-app can import x/minfee without a replace directive. This involves sourcing multiple modules in a single repo
  3. [Optional] verify that consumers can import x/minfee unambiguously. Currently if it's nested under celestia-app/x/minfee then Go tries to import that package from the celestia-app v1.x release which doesn't have a Go module for x/minfee. That's why this PR made the Go module path celestia-app/v3/x/minfee but that's not consumable by prior state machine versions (i.e. v2).

@rootulp rootulp mentioned this pull request Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Prototype versioned app module rolling upgrades
2 participants