fatal error: concurrent map iteration and map write

[shanliu]

fatal error: concurrent map iteration and map write

goroutine 55604 [running]:
reflect.mapiternext(0x49fa8f?)
/usr/lib/golang/src/runtime/map.go:1380 +0x19
reflect.(*MapIter).Next(0xc000066c20?)
/usr/lib/golang/src/reflect/value.go:1924 +0x7e
encoding/json.mapEncoder.encode({0xc000066ce0?}, 0xc0004cee80, {0xbb4900?, 0xc000266530?, 0x6?}, {0x7?, 0x0?})
/usr/lib/golang/src/encoding/json/encode.go:797 +0x33e
encoding/json.structEncoder.encode({{{0xc000286900?, 0xc000066e68?, 0x4160d0?}, 0xc00029cd20?}}, 0xc0004cee80, {0xbf7ce0?, 0xc000266460?, 0x4383d6?}, {0x0, 0x1})
/usr/lib/golang/src/encoding/json/encode.go:759 +0x1f4
encoding/json.ptrEncoder.encode({0x414ba5?}, 0xc0004cee80, {0xb6e780?, 0xc000266460?, 0xb6e780?}, {0xbf?, 0x1?})
/usr/lib/golang/src/encoding/json/encode.go:943 +0x21c
encoding/json.(*encodeState).reflectValue(0xc000066f98?, {0xb6e780?, 0xc000266460?, 0x7fb0a34a0108?}, {0x20?, 0x57?})
/usr/lib/golang/src/encoding/json/encode.go:358 +0x78
encoding/json.(*encodeState).marshal(0x2?, {0xb6e780?, 0xc000266460?}, {0x99?, 0xc4?})
/usr/lib/golang/src/encoding/json/encode.go:330 +0xfa
encoding/json.Marshal({0xb6e780, 0xc000266460})
/usr/lib/golang/src/encoding/json/encode.go:161 +0xe5
github.com/openzipkin/zipkin-go/model.SpanModel.MarshalJSON({{{0x0, 0xec061f7ddeaaf40}, 0x73ff63cecd0bba2a, 0xc00003f118, 0x0, 0xc000612b5c, {0x0, 0x0}, {0x0, 0x0}}, ...})
/root/go/pkg/mod/github.com/openzipkin/zipkin-go@v0.4.0/model/span.go:121 +0x2c9
gitlab.xx.com/hsbproject/hsbcommon/hsbservice.(*HsbListFileZipkinReporter).Send(0xc000012100, {{{0x0, 0xec061f7ddeaaf40}, 0x73ff63cecd0bba2a, 0xc00003f118, 0x0, 0xc000612b5c, {0x0, 0x0}, {0x0, ...}}, ...})
/root/go/pkg/mod/gitlab.xx.com/hsbproject/hsbcommon@v0.1.137/hsbservice/zipkin.go:19 +0x5f
github.com/openzipkin/zipkin-go.(*spanImpl).Finish(0xd95f60?)
/root/go/pkg/mod/github.com/openzipkin/zipkin-go@v0.4.0/span_implementation.go:83 +0xf7
github.com/openzipkin/zipkin-go/middleware/http.(*transport).RoundTrip(0xc00028ad20, 0xc0005ddc00)
/root/go/pkg/mod/github.com/openzipkin/zipkin-go@v0.4.0/middleware/http/transport.go:217 +0x10d3
net/http.send(0xc0005ddc00, {0xd8d6a0, 0xc00028ad20}, {0x8?, 0xc54c80?, 0x0?})
/usr/lib/golang/src/net/http/client.go:252 +0x5f7
net/http.(*Client).send(0xc000618fc0, 0xc0005ddc00, {0x4160d0?, 0xc000067908?, 0x0?})
/usr/lib/golang/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0xc000618fc0, 0xc0005ddc00)
/usr/lib/golang/src/net/http/client.go:716 +0x8fb
net/http.(*Client).Do(...)
/usr/lib/golang/src/net/http/client.go:582
github.com/openzipkin/zipkin-go/middleware/http.(*Client).DoWithAppSpan(0xc0002b8f00, 0xc0005ddb00, {0xc941a9, 0x15})
/root/go/pkg/mod/github.com/openzipkin/zipkin-go@v0.4.0/middleware/http/client.go:123 +0x32c
gitlab.xx.com/hsbproject/hsbcommon/hsbmodule.(*HsbRestBuild).executeRequest(0xc000533b80, {0xd925e0, 0xc0004db300?}, 0xc0003146e0, {0xc000360780, 0x39}, 0xc00023e6f0?, {0xd94398, 0x12794b8}, 0xc000620e10, ...)
/root/go/pkg/mod/gitlab.xx.com/hsbproject/hsbcommon@v0.1.137/hsbmodule/hsb_client.go:187 +0x899
gitlab.xx.com/hsbproject/hsbcommon/hsbmodule.(*HsbRestBuild).BuildRequest(0xc000533b80, {0xd925e0, 0xc0004db300}, 0x9f2c20?, 0xd96f90?, {0xbb4540, 0xc000620db0}, 0x9f26f6?)
/root/go/pkg/mod/gitlab.xx.com/hsbproject/hsbcommon@v0.1.137/hsbmodule/hsb_client.go:131 +0x527
github.com/hsbteam/rest_client.(*RestClient).Do.func1()
/root/go/pkg/mod/gitlab.xx.com/hsbproject/rest_client@v0.0.53/rest_client.go:148 +0x83
created by github.com/hsbteam/rest_client.(*RestClient).Do
/root/go/pkg/mod/gitlab.xx.com/hsbproject/rest_client@v0.0.53/rest_client.go:141 +0x1e5

[basvanbeek]

I'd like to understand why tags are still being modified if already set to completion. Can this race condition from outside of the library be fixed at the source. We're being asked here (#218) to guard against external race conditions at the expense of performance of the library.

[php-lsys]

When the timeout and cancel occur at the same time in the context, it is difficult to find the place where the tag is triggered.@basvanbeek

[php-lsys]

There is a small probability that panic will be triggered when using the following code

//gCtx *gin.Context
//req *http.Request
//"github.com/openzipkin/zipkin-go/middleware/http"
httpClient, _ := http.NewClient(tracer, zipkinhttp.WithClient(&http.Client{
}), zipkinhttp.ClientTrace(true))
req = req.WithContext(gCtx.Request.Context())
resp, err := httpClient.DoWithAppSpan(req, "***")

@basvanbeek But I'm not sure where the " tags " change was made.

[php-lsys]

I'd like to understand why tags are still being modified if already set to completion. Can this race condition from outside of the library be fixed at the source. We're being asked here (#218) to guard against external race conditions at the expense of performance of the library.

Or is it possible to put the following interface

type Reporter interface {
	Send(model.SpanModel) // Send Span data to the reporter
	Close() error // Close the reporter
}

Modify this to

type Reporter interface {
	Send(model.SpanModel,mtx sync.RWMutex) // Send Span data to the reporter.
	Close() error // Close the reporter
}

I'm doing RLock when I implement the Reporter.

[shanliu]

zipkin-go/middleware/http/transport.go

Lines 180 to 195 in e84b2cf

	sptr.c = &httptrace.ClientTrace{
	GetConn: sptr.getConn,
	GotConn: sptr.gotConn,
	PutIdleConn: sptr.putIdleConn,
	GotFirstResponseByte: sptr.gotFirstResponseByte,
	Got100Continue: sptr.got100Continue,
	DNSStart: sptr.dnsStart,
	DNSDone: sptr.dnsDone,
	ConnectStart: sptr.connectStart,
	ConnectDone: sptr.connectDone,
	TLSHandshakeStart: sptr.tlsHandshakeStart,
	TLSHandshakeDone: sptr.tlsHandshakeDone,
	WroteHeaders: sptr.wroteHeaders,
	Wait100Continue: sptr.wait100Continue,
	WroteRequest: sptr.wroteRequest,
	}

The above callback function may still be triggered when the following error occurs:

zipkin-go/middleware/http/transport.go

Lines 217 to 218 in e84b2cf

	sp.Finish()
	return nil, err

This will eventually lead to the occurrence of: fatal error: concurrent map iteration and map write
@basvanbeek

[codefromthecrypt]

should be fixed in v0.4.3. please comment if this didn't solve it