Bitswap Retrieval, Sync version by hannahhoward · Pull Request #86 · filecoin-project/lassie

hannahhoward · 2023-02-15T00:03:39Z

Goals

Get bitswap retrieval working

Implementation

more information coming (to be delivered sync for now)

add context key based blockstore and routing implementations that limit results for a bitswap session to the retrieval in question

rvagg · 2023-02-15T01:31:40Z

pkg/types/types.go

 	// retrieval was requested via peer ID
 	StorageProviderId() peer.ID
+	// Protocol
+	Protocols() []multicodec.Code


We've gone plural for Protocols but singular for StorageProviderId. Some events (Candidates*) return an empty peerID for a call to StorageProviderId() because they have a plural form that makes sense for them, StorageProviderIds(). Aren't we going to end up in a situation where the majority of events have a single protocol in the slice returned from this? How many will it make sense to have the plural form?

arguably, the right thing here is to just return retrieval candidates, IMHO

but actually, I think storage provider id indicates that an event is associated with a specific storage provider. protocols indicates all the protocols it applies to. Honestly? I'm not sure what the right thing is.

pkg/retriever/bitswaphelpers/indexerrouting.go

pkg/retriever/bitswaphelpers/multiblockstore.go

pkg/retriever/bitswaphelpers/multiblockstore_test.go

pkg/retriever/bitswapretriever.go

pkg/retriever/retriever.go

rvagg · 2023-02-15T02:13:39Z

pkg/retriever/retriever.go

+			CandidateSplitter:   NewProtocolSplitter([]multicodec.Code{multicodec.TransportGraphsyncFilecoinv1, multicodec.TransportBitswap}),
+			CandidateRetrievers: candidateRetrievers,


still not a fan of how these two arrays have to match up for this to work

and, what's going to happen when our candidateRetrievers only has graphsync but the splitter is considering both? Do we need a NullRetriever() to take its place or something?

no it's setup properly to just ignore

rvagg · 2023-02-15T02:19:57Z

pkg/retriever/retriever.go

-			CoordinationKind: types.RaceCoordination,
+			CandidateSplitter:   NewProtocolSplitter([]multicodec.Code{multicodec.TransportGraphsyncFilecoinv1, multicodec.TransportBitswap}),
+			CandidateRetrievers: candidateRetrievers,
+			CoordinationKind:    types.RaceCoordination,


This is going to get weird if/when we have both strategies using the same linksystem.
The CAR(s) shouldn't double-write if both bitswap and graphsync both fetch the same block and try and put it, but we ought to test that this works cleanly and we don't end up with weird CARs.

yea I know we'll fix in the streaming PR

Co-authored-by: Rod Vagg <rod@vagg.org>

cleanup errors on the multiblockstore

to be safe, make the changes to counts in bitswap retriever atomic

go.mod

pkg/retriever/bitswaphelpers/multiblockstore.go

rvagg · 2023-02-15T06:42:40Z

pkg/retriever/bitswapretriever_test.go

+	"github.com/stretchr/testify/require"
+)
+
+func TestBitswapRetriever(t *testing.T) {


some nice tricks in here to make the test less verbose 👌

Co-authored-by: Rod Vagg <rod@vagg.org>

…sterify fix: improve flakes, make test runs faster

fetches from two seperate peers over bitswap, had to change the provider

codecov-commenter · 2023-02-15T23:09:46Z

Codecov Report

Merging #86 (3ba9704) into main (1744b10) will increase coverage by 9.18%.
The diff coverage is 78.07%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #86      +/-   ##
==========================================
+ Coverage   50.09%   59.27%   +9.18%     
==========================================
  Files          36       42       +6     
  Lines        2703     3202     +499     
==========================================
+ Hits         1354     1898     +544     
+ Misses       1288     1204      -84     
- Partials       61      100      +39

Impacted Files	Coverage Δ
cmd/lassie/daemon.go	`0.00% <0.00%> (ø)`
cmd/lassie/fetch.go	`0.00% <0.00%> (ø)`
cmd/lassie/main.go	`0.00% <0.00%> (ø)`
pkg/indexerlookup/candidatefinder.go	`0.00% <0.00%> (ø)`
pkg/internal/libp2p.go	`0.00% <0.00%> (ø)`
...kg/internal/testutil/collectingeventlsubscriber.go	`80.00% <ø> (ø)`
pkg/internal/testutil/mockcandidatefinder.go	`23.80% <ø> (ø)`
pkg/internal/testutil/mockclient.go	`50.69% <ø> (ø)`
pkg/retriever/graphsyncretriever.go	`90.52% <ø> (ø)`
pkg/retriever/bitswaphelpers/countinglinksystem.go	`64.00% <64.00%> (ø)`
... and 16 more

rvagg · 2023-02-15T23:25:54Z

pkg/internal/itest/testpeer/generator.go

+	g.seq++
+	p, err := p2ptestutil.RandTestBogusIdentity()
+	if err != nil {
+		panic("FIXME") // TODO change signature


what's going on here, are we tied to a particular API for this struct or is this just an annoying TODO to get to later because it's too cumbersome for now?

copied from go-bitswap # oy

pkg/internal/itest/bitswapfetch_test.go

kylehuntsman · 2023-02-15T23:34:59Z

cmd/lassie/daemon.go

-	httpServer, err := httpserver.NewHttpServer(cctx.Context, address, port)
+
+	// create a lassie instance
+	lassie, err := lassie.NewLassie(cctx.Context, lassie.WithProviderTimeout(20*time.Second))
+	if err != nil {
+		return err
+	}
+
+	httpServer, err := httpserver.NewHttpServer(cctx.Context, lassie, address, port)


kylehuntsman · 2023-02-15T23:36:06Z

pkg/lassie/lassie.go

+	if cfg.Host == nil {
+		var err error
+		cfg.Host, err = internal.InitHost(ctx, multiaddr.StringCast("/ip4/0.0.0.0/tcp/6746"))
+		if err != nil {
+			return nil, err
+		}


Co-authored-by: Rod Vagg <rod@vagg.org>

various minor fixes to optimize behavior

hannahhoward added 2 commits February 14, 2023 12:03

feat(bitswap): add helpers to fit in go-bitswap

ebfd862

add context key based blockstore and routing implementations that limit results for a bitswap session to the retrieval in question

feat(bitswap): add bitswap retriever impl

2137ea4

rvagg reviewed Feb 15, 2023

View reviewed changes

pkg/retriever/bitswaphelpers/indexerrouting.go Outdated Show resolved Hide resolved

rvagg reviewed Feb 15, 2023

View reviewed changes

pkg/retriever/bitswaphelpers/indexerrouting.go Outdated Show resolved Hide resolved

rvagg reviewed Feb 15, 2023

View reviewed changes

pkg/retriever/bitswaphelpers/multiblockstore.go Outdated Show resolved Hide resolved

rvagg reviewed Feb 15, 2023

View reviewed changes

pkg/retriever/bitswaphelpers/multiblockstore.go Outdated Show resolved Hide resolved

rvagg reviewed Feb 15, 2023

View reviewed changes

pkg/retriever/bitswaphelpers/multiblockstore.go Outdated Show resolved Hide resolved

rvagg reviewed Feb 15, 2023

View reviewed changes

pkg/retriever/bitswaphelpers/multiblockstore.go Outdated Show resolved Hide resolved

rvagg reviewed Feb 15, 2023

View reviewed changes

pkg/retriever/bitswaphelpers/multiblockstore_test.go Show resolved Hide resolved

rvagg reviewed Feb 15, 2023

View reviewed changes

pkg/retriever/bitswapretriever.go Outdated Show resolved Hide resolved

rvagg reviewed Feb 15, 2023

View reviewed changes

pkg/retriever/bitswapretriever.go Outdated Show resolved Hide resolved

rvagg reviewed Feb 15, 2023

View reviewed changes

pkg/retriever/retriever.go Outdated Show resolved Hide resolved

rvagg reviewed Feb 15, 2023

View reviewed changes

hannahhoward and others added 9 commits February 14, 2023 19:16

feat(retriever): add bitswap retriever test

ec93d7c

Update pkg/retriever/bitswaphelpers/indexerrouting.go

63a177c

Co-authored-by: Rod Vagg <rod@vagg.org>

Update pkg/retriever/bitswaphelpers/indexerrouting.go

eb7ff13

Co-authored-by: Rod Vagg <rod@vagg.org>

Update pkg/retriever/retriever.go

980be5f

Co-authored-by: Rod Vagg <rod@vagg.org>

refactor(bitswaphelpers): respond PR comments

57b272c

cleanup errors on the multiblockstore

fix(deps): get all deps in line

cd6c923

fix(retriever): add atomics

a858f39

to be safe, make the changes to counts in bitswap retriever atomic

style(lint): remove unused type

da02d45

style(lint): fix unkeyed fields

17d6cf7

hannahhoward marked this pull request as ready for review February 15, 2023 03:46

rvagg reviewed Feb 15, 2023

View reviewed changes

go.mod Show resolved Hide resolved

rvagg reviewed Feb 15, 2023

View reviewed changes

pkg/retriever/bitswaphelpers/multiblockstore.go Show resolved Hide resolved

fix: improve flakes, make test runs faster

185384b

rvagg reviewed Feb 15, 2023

View reviewed changes

rvagg mentioned this pull request Feb 15, 2023

feat: add ErrNotFound ipld/go-ipld-prime#494

Open

hannahhoward and others added 5 commits February 15, 2023 05:15

Update pkg/retriever/bitswaphelpers/multiblockstore.go

531d3de

Co-authored-by: Rod Vagg <rod@vagg.org>

Merge pull request #87 from filecoin-project/rvagg/test-robust-and-fa…

78c0f94

…sterify fix: improve flakes, make test runs faster

feat(itest): add integration test for bitswap

b643c79

fetches from two seperate peers over bitswap, had to change the provider

style(lint): fix lint errors

017fb14

fix(itest): speed up itest

827fcd7

hannahhoward force-pushed the feat/bitswap branch from ed8b56d to 827fcd7 Compare February 15, 2023 23:05

rvagg reviewed Feb 15, 2023

View reviewed changes

pkg/internal/itest/bitswapfetch_test.go Show resolved Hide resolved

rvagg approved these changes Feb 15, 2023

View reviewed changes

kylehuntsman approved these changes Feb 15, 2023

View reviewed changes

Update pkg/internal/itest/bitswapfetch_test.go

288e9db

Co-authored-by: Rod Vagg <rod@vagg.org>

rvagg approved these changes Feb 16, 2023

View reviewed changes

hannahhoward added 3 commits February 15, 2023 20:02

feat(bitswap): add timeouts, further testing

ce79d61

various minor fixes to optimize behavior

Merge branch 'feat/bitswap-local' into feat/bitswap

e93ce02

fix(retriever): increase timeout

3ba9704

hannahhoward merged commit d470217 into main Feb 16, 2023

rvagg deleted the feat/bitswap branch February 20, 2023 21:28

		CandidateSplitter: NewProtocolSplitter([]multicodec.Code{multicodec.TransportGraphsyncFilecoinv1, multicodec.TransportBitswap}),
		CandidateRetrievers: candidateRetrievers,

Conversation

hannahhoward commented Feb 15, 2023

Goals

Implementation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Feb 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Feb 15, 2023 •

edited

Loading