"remote sync match failed" error in cadence worker log

[zhangkyou]

{"level":"error","ts":"2021-09-03T11:50:29.265+0800","msg":"Failed to add task","service":"cadence-matching","component":"matching-engine","wf-task-list-name":"custom","wf-task-list-type":0,"error":"RemoteSyncMatchedError{Message: remote sync match failed}","wf-task-list-name":"custom","wf-task-list-type":0,"logging-call-at":"taskListManager.go:259","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/common/log/loggerimpl/logger.go:131\ngithub.com/uber/cadence/service/matching.(*taskListManagerImpl).AddTask\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/matching/taskListManager.go:259\ngithub.com/uber/cadence/service/matching.(*matchingEngineImpl).AddDecisionTask\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/matching/matchingEngine.go:271\ngithub.com/uber/cadence/service/matching.(*handlerImpl).AddDecisionTask\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/matching/handler.go:191\ngithub.com/uber/cadence/service/matching.ThriftHandler.AddDecisionTask\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/service/matching/thriftHandler.go:66\ngithub.com/uber/cadence/.gen/go/matching/matchingserviceserver.handler.AddDecisionTask\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/.gen/go/matching/matchingserviceserver/server.go:248\ngo.uber.org/yarpc/encoding/thrift.thriftUnaryHandler.Handle\n\t/Users/didi/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/encoding/thrift/inbound.go:64\ngo.uber.org/yarpc/internal/observability.(*Middleware).Handle\n\t/Users/didi/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/internal/observability/middleware.go:166\ngo.uber.org/yarpc/api/middleware.unaryHandlerWithMiddleware.Handle\n\t/Users/didi/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/api/middleware/inbound.go:71\ngo.uber.org/yarpc/api/transport.InvokeUnaryHandler\n\t/Users/didi/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/api/transport/handler_invoker.go:70\ngo.uber.org/yarpc/transport/tchannel.handler.callHandler\n\t/Users/didi/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/transport/tchannel/handler.go:233\ngo.uber.org/yarpc/transport/tchannel.handler.handle\n\t/Users/didi/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/transport/tchannel/handler.go:121\ngo.uber.org/yarpc/transport/tchannel.handler.Handle\n\t/Users/didi/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/transport/tchannel/handler.go:110\ngithub.com/uber/tchannel-go.channelHandler.Handle\n\t/Users/didi/go/pkg/mod/github.com/uber/tchannel-go@v1.16.0/handlers.go:126\ngithub.com/uber/tchannel-go.(*Connection).dispatchInbound\n\t/Users/didi/go/pkg/mod/github.com/uber/tchannel-go@v1.16.0/inbound.go:203"}

How can I investigate this error?

[longquanzheng]

See #4363