Node Discovery using a custom DHT

[St4NNi]

I wanted to start a discussion about bringing DHT capabilities directly into iroh, building on the work in the iroh-dht-experiment repository.

Currently, the available discovery methods are (m)DNS, pkarr, or mainline DHT. Except for mDNS (which is local-only), all of them require additional server infrastructure or setup to enable node discovery. This means you can't build a purely iroh-based system for internet-wide discovery without depending on external services.

The iroh-dht-experiment repo showcases a Kademlia implementation with a 32-byte keyspace working over iroh connections. This solves key limitations with mainline DHT: the 20-byte keyspace requires hashing our BLAKE3 identifiers and could be a starting point for a custom iroh DHT discovery.

Would it be possible to create an iroh-only DHT discovery based on this? I would love to build a system that only uses iroh nodes (and maybe relays) and that enables discovery, similar to what the libp2p kad and identify protocols can do. This would give applications the option to run fully distributed systems without requiring DNS servers, pkarr infrastructure, or bootstrapping into the mainline DHT.

If this makes sense as a direction, what would the integration path look like? Should it be a separate crate or something more deeply embedded in iroh itself?

[matheus23]

I think there's a little bit of a bootstrapping problem with this.
iroh-dht-experiment relies on Endpoint::connect with only EndpointIds (prev. NodeId).
And it seems that discovering neighbors in the iroh-dht-experiment means that you're only discovering its EndpointId, and not its full EndpointAddr (prev. NodeAddr).

So you need another system still to resolve these iroh endpoint public keys into IP addresses. That said, I think this might only be a shortcoming of the current iroh-dht-experiment, and should be fixable: The Lookup should be changed to return an EndpointAddr instead of only an EndpointId. (Or perhaps return only a single optional IPv4 & IPv6 addr, or optional relay, if we want to reduce the scope/potential for abuse a bit here.)

If that problem is solved, then I think something like that can be done, although I don't think it's going to be much better than the current system. Practically, I think running multiple iroh-dns-servers redundantly and installing a random subset of them as concurrent discovery services in each iroh endpoint probably makes the system just as resilient with a lot less overhead.

In any case, at this point this is likely to be another helper crate IMHO.