@rjan90 To follow up with testing importing a Lotus exported snapshot

I bumped in to this today, but by tinkering with the lotus-shed gas-estimation tool, same issue. Some related threads about drand stuff:

• feat: daemon: import: only setup stmgr if validating chain #11084 - avoid the problem by iffing it off
• feat: cli: gas estimation tooling #9654 - a hack introduced here to make the introduced feature "work" but that was later removed
• fix: daemon: set real beacon schedule when importing chain #11080 - where the real drand client was introduced
• [skip changelog] chore: drand: cleanup legacy and unused configs #12272 - this is kind of related, but it's not the only cause, this was removing clients from the configs that weren't needed - this removed client config for DrandMainnet but DrandIncentinet clients were already removed at this point.

Here's what's going on:

1. Every different drand beacon needs a way of fetching the randomness, on mainnet we have 3 throughout the history of the chain
2. In reality we only need to be using the latest beacon live, we've used the rest of the beacons to fetch randomness and baked it into the chain so we should never need to fetch fresh randomness for arbitrary historical points
3. We still set up a beacon for each of these different points in history even though we only use one
4. Initialising a beacon requires there to be a client, i.e. a way of talking to the remote beacon API, otherwise it fails, here: https://github.com/drand/drand/blob/v1.5.11/client/client.go#L52-L53
5. Our configs only have HTTP endpoints set for the latest drand beacon (quicknet), the others don't, so when we initialise the other 2, we don't provide any options with clients and we get the above error.
6. But, the important bit: it works when you're running Lotus daemon because we are able to provide a PubSub instance, which can be set up as a "client":

   lotus/chain/beacon/drand/drand.go

   Lines 113 to 118 in b83bf69

   	opts = append(opts, gclient.WithPubsub(ps))
   	} else {
   	log.Info("drand beacon without pubsub")
   	}
   	client, err := dclient.Wrap(clients, opts...)

The other places we run BeaconScheduleFromDrandSchedule and don't provide a PubSub (last argument is nil), we encounter this problem.

These old networks don't exist anymore (at least incentinet certainly doesn't), so even pubsub should fail if it was trying to actually use it. The endpoints were removed from config because they shouldn't even work anymore.

Possible solutions:

• I don't think we should need these old beacons, maybe if we don't have a client we just don't initialise it? If there's something about the beacon we need in order to deal with ancient tipsets, then maybe we could get this addressed on the drand side so it's more lax about needing a client to initialise?
  • There might be some deeper surgery needed here to unwind the chain of beacons, why do we have them all? Were they just not retired? There's probably a short period of time after a switch where we need the old one but now we have 2 old ones that shouldn't be used.
• Provide a fake client just to make it setup? Alternatively a fake pubsub might do the trick?

Digging a bit further, I believe we need ancient beacons in order to validate old block headers. Our syncer does a ValidateBlock and part of the calls ValidateBlockValues which uses the schedule to find the beacon for that epoch and does a VerifyEntry on that which calls VerifyBeacon (in the drand code).

So we're back to this problem of no network so being unable to initialise drand beacons where we no longer want (or can) talk to them.

@AnomalRoil can you advise on the right way forward here? See my previous comment for context. Do we need to fake a network client to init? Should I open an issue over on drand to make it possible to init without a client? Would drand@2.x fix this?

Noop client being attempted here: #12830