No addresses of peer after successful kademlia lookup?

[Wiezzel]

I am trying to do a seemingly very simple thing: look up a peer in kademlia DHT, and dial it. I have noticed though that sometimes despite the kad query being successful, I am unable to dial the peer due to lack of known addresses. I would expect kad behaviour to be able to provide the peer's address via handle_pending_outbound_connection, but it seems not to be the case.

So e.g. I get this event:

OutboundQueryProgressed { id: QueryId(105), result: GetClosestPeers(Ok(GetClosestPeersOk { key: [0, 36, 8, 1, 18, 32, 51, 206, 172, 17, 103, 233, 54, 137, 218, 98, 96, 255, 147, 88, 160, 27, 156, 44, 34, 116, 163, 39, 25, 94, 149, 197, 254, 71, 18, 76, 56, 22], peers: [PeerId("12D3KooWDJbk4fUwkKFoMwev7YyUFx7pPAXbMyVkRH7sH42nv4vq"), PeerId("12D3KooWMztQS97kK73G5YJsxBVZFYUSBzicdEgKQHqY9Z88sr6U"), PeerId("12D3KooWNFZodcxrVkcCVyURKLahUeiFW1bZYP4mwsT3cjJrjC7J"), PeerId("12D3KooWPYs9wpdBGTAmAzo89DjNXFSYJoUrmshKbrjQXXNvcY8p"), PeerId("12D3KooWPAmD5miwBXdziFC9J2SnLqN4Sdh3vGW8BNKRyGoELTne"), PeerId("12D3KooWCoSjkfvmC6vpd5AfmBWJiP5RSnFfKoN3otyUmVtD1B5P"), PeerId("12D3KooWL6vN6dLXxVdLawuyJDANugbuySxzZFXmZVKPwYWUVfMN"), PeerId("12D3KooWQtAGvDDgxkGmYzHBjofbtBRVP3zu18jJZ711E3YYnoUD"), PeerId("12D3KooWQYZeE4eKuDBzaZCb6Kp4PyzDc2BUjNfQHsBYGVzckiTg"), PeerId("12D3KooWDCsqJrLMJ2ojsH98i37E8EY7mWDuqJkJ74AcjZghvNtk"), PeerId("12D3KooWLqHwBnqJNrKLSU5hCiqCUGcmKghwyQfr3JSpRDmAn6eR"), PeerId("12D3KooWHuo573Bf1GkStqW6LnawrszguUT3UiyVevgpSj6o2kH3"), PeerId("12D3KooWBNmKEks2ZcASztvxuLbXc5caSRbfd2Ukv4WLFRJyeXnz"), PeerId("12D3KooWHJfa5ekQMPeEZuvgoNKdxm9W6ZWJDarMcKGF5VHkwDhg"), PeerId("12D3KooWR52LhKYkcK5iDQyXBfnZ3bsUB6q9L8pcsDGxa4faLx7c"), PeerId("12D3KooWDVAv9AWQBR94bdkbKGtMjzgHCqmHyDwTaHsu3XyF6RV2"), PeerId("12D3KooWCg8QwJS44iFupwH8K6nFFfM4kwKKzQfMH3j3cxRixMp9"), PeerId("12D3KooWM5YXQWEzXRv2W9kbaoCbmgiExdyqp7zTRqKmdbyk9fq1"), PeerId("12D3KooWD9NHY4Lkm8GT312GpsDSA8f2qrRTXwx5HUfNPPy3hTKp"), PeerId("12D3KooWJpXJ4JBkob65Y8ok9228PAFc94Uia4Sccp5M68M6jXfd")] })), stats: QueryStats { requests: 79, success: 64, failure: 15, start: Some(Instant { tv_sec: 7203206, tv_nsec: 41789436 }), end: Some(Instant { tv_sec: 7203211, tv_nsec: 356204910 }) }, step: ProgressStep { count: 1, last: true } }

And right after this I see:

Failed to dial 12D3KooWDJbk4fUwkKFoMwev7YyUFx7pPAXbMyVkRH7sH42nv4vq: NoAddresses

How is that possible? Is this a bug or do I need to do something more in my code?

[dariusc93]

I've seen this when the node is in client mode. By default, the mode is automatically set on if an external address is provided. If none is provided, mode will be set to client mode, making it unable to accept queries, however once an external address is provided, either manually (either through Swarm::add_external_address or emitting ToSwarm::ExternalAddrConfirmed in a behaviour) or through other means like autonat (determining the node can be reached), relay reservation or upnp, it will be set to server mode. Same with the other peer(s).

[Wiezzel]

@dariusc93 You mean the querying node? I'm pretty sure it's in server mode, but will double check that. Also, this doesn't apply to all queries, only some of them.

[Wiezzel]

Yea, the node has external address set on startup, so that's an issue with client/server mode.

[guillaumemichel]

1. Does it consistently fail with all peer ids?

2. Do you share any common transport with the node you are trying to dial?

3. Are the nodes you are trying to dial all using the same libp2p implementation? Or does it just seem random?

[Wiezzel]

Does it consistently fail with all peer ids?

No. Sometimes, with some peer ids. Seems random.

Do you share any common transport with the node you are trying to dial?

Yes, all nodes are using QUIC transport only.

Are the nodes you are trying to dial all using the same libp2p implementation?

Yes, they all use rust-libp2p.

[guillaumemichel]

Does it consistently fail with all peer ids?

No. Sometimes, with some peer ids. Seems random.

So for a given peer id, sometimes it works and sometimes not? Or is it consistent that a peer id will always succeed/fail?

How many peers are you connected to in your setup? And what is the kad bucket size?

[Wiezzel]

So for a given peer id, sometimes it works and sometimes not? Or is it consistent that a peer id will always succeed/fail?

It doesn't seem consistent.

How many peers are you connected to in your setup? And what is the kad bucket size?

I am connected to ~200 peers. The kad bucket size is default (I'm not sure what is the default value).

[guillaumemichel]

It is likely that the bucket to which the target peer belong is already full, so the addresses aren't stored in the kad routing table.

This is indeed a serious issue.

I can see two ways to address it:

• implement a Peerstore (see misc: add a PeerStore behaviour #4103)
• emit a FromSwarm::NewExternalAddrOfPeer event (context in feat(swarm): allow behaviours to share addresses of peers #4371). This event could be emitted either a) for the target of the FIND_NODE RPC only or b) for all discovered peers in a request (probably overkill). The caller of the kademlia lookup then needs to handle the addresses.

The Peerstore seems to be the right approach, although it may not be straightforward to implement. Emitting a FromSwarm::NewExternalAddrOfPeer event could be a temporary quick fix.

[guillaumemichel]

For a network of ~200 stable peers, you should be able to lookup and dial only ~50% of them because the peers falling in a full bucket aren't recorded.

[Wiezzel]

For a network of ~200 stable peers, you should be able to lookup and dial only ~50% of them because the peers falling in a full bucket aren't recorded.

Well, I'd like to be able to dial all of them. Don't need to keep all addresses cached all the time though. Perhaps the least recently queried could be evicted from buckets??

[Wiezzel]

Also, could I increase the bucket size as temporary workaround?

[guillaumemichel]

Also, could I increase the bucket size as temporary workaround?

Yes, you can se the bucket size to be the network size (e.g 200), and it should work (if this is actually the root of the problem).

Perhaps the least recently queried could be evicted from buckets??

The logic of Kademlia is to keep to most stable peers in the buckets. Peers in the buckets are periodically queried to make sure they are still online, and offline peers eventually get replaced by fresh peers. Once the buckets have selected a set of stable peers, any newly queried peer belonging to this bucket will be dropped, because we already have enough good peers.