Decentralized Search Problem

[hoffmabc]

One of the biggest problems in the original OpenBazaar model was decentralized search. The first iteration of the marketplace had a very slow, unstable real-time search protocol that just flooded the network with search commands. Our second piece was a centralized indexer that worked better, but was susceptible to centralized censorship and required server infrastructure.

I'm proposing that a better way would be something more akin to BEP-51 for BitTorrent where nodes are required to respond with which listings/metadata they currently host, but it would be optional to serve as an indexer service. Crawlers for Mainline DHT (BT) have been able to index around 500k torrents in less than an hour so the network for the marketplace wouldn't be unreasonable based on that rough approximation, especially in the beginning. I think we can do a lot better with a strong indexer built on Rust and a lighter weight DHT and network protocol than BitTorrent.

[hoffmabc]

For additional context: https://github.com/cpacia/obcrawler

[cpacia]

Here was the general idea that I was going for with the ob3 design...

• Listings were still seeded on ipfs with the store root published on ipns.
• Every time a user updated anything about their store they would update their ipns record in the DHT as well as publish the new record on
  a dedicated pubsub channel.
• Regular users could just ignore that pubsub channel, but anyone running a network crawler could subscribe to it.
• This gives the network crawler a real-time update stream from the network.
• It also allows the network crawler to index every ipns record on the network, including the expiration time of the record. And to crawl the node immediately after they update their stores.
• As back-up the network crawler could periodically crawl the DHT to look for new nodes, but assuming the pubsub works correctly, this shouldn't turn up any unknown peers.
• The crawler would also track lastseen, lastcrawled and periodically try to crawl nodes that it hasn't seen in a while. This also shouldn't be expected to turn up anything new if the pubsub is working correctly.
• When new or updated user data is found by the crawler it would stream it over a gRPC endpoint and include the ipns record expiration date.
• Anyone who wants to run a search engine would only need to run this crawler, subscribe to the streaming endpoint, and ingest the data into their search index. They would also need to respect the expiration date that the crawler stamps the data with to avoid returning stale search results.
• Since the crawler/search index knows the most current ipns record for each user, it can always return IPFS path in its results. ipfs paths resolve substantially faster than ipns paths so this represents a large speed boost.
• The only time an ipns path would ever be needed is if the user copied someone's store link from an external app and pasted it into the url bar.
• To ensure the availability of the ipns records in this case the crawlers can pin all un-expired ipns records since they have every record in their db.

This code did all this last I worked on it https://github.com/cpacia/obcrawler