Too many go routines causing OOM #1645
Comments
|
@astralblue I remember you looked at this problem last week and proposed to add thread pool for the go routines. We can talk about this after you come back. |
|
@rlan35 the real culprit here is the deadlocks we were seeing in our message processing, when #504 was written. It was supposedly a short-term fix to avoid having to look for the deadlocks. I objected that we should fix these deadlocks proper instead of converting them into goroutine leaks and sweeping them under the rug, which is unfortunately what the team decided to do. I can and will build goroutine pools to use in these situations, but a fixed-size pool will surely these deadlocks back. Please take care of them on your side: Set the limit to a low number like 1 or 2 you'll be able to quickly find many of these. |
|
One of our community members also reported a similar issue. harmony-one/pangaea-community#77 |
|
@harmony-ek , the original change wasn't for deadlock. When I was working on the fix of consensus issue, I clearly found that leader wasn't able to process the message fast enough. Even though validators sent out the signed message, leader didn't appear to process them. I went back to original network code and found we were using the go routine to parallelize the processing. Yes, it too many messages are here, it is causing go routing OOM leakage. I don't remember any deadlock issue then. Some interesting read I found. https://blog.golang.org/pipelines What do you think? |
|
I stumbled upon the following repo & article when I was looking into the OOM reaping that happened on Pangea:
The article is from a blockchain dev over at Orbs detailing how he profiled their Go codebase to look for memory leaks. Not sure if it's of any use, but just wanted to share. |
|
Eugene last pr #1710 is fixing this |
atal error: runtime: out of memory
runtime stack:
runtime.throw(0x11277d3, 0x16)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/panic.go:617 +0x72
runtime.sysMap(0xc06c000000, 0x4000000, 0x20395f8)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/mem_linux.go:170 +0xc7
runtime.(*mheap).sysAlloc(0x2016600, 0x100000, 0x2016610, 0x80)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/malloc.go:633 +0x1cd
runtime.(*mheap).grow(0x2016600, 0x80, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/mheap.go:1232 +0x42
runtime.(*mheap).allocSpanLocked(0x2016600, 0x80, 0x2039608, 0x2016608)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/mheap.go:1150 +0x3a7
runtime.(*mheap).alloc_m(0x2016600, 0x80, 0x101, 0x7ffa00000000)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/mheap.go:977 +0xc2
runtime.(*mheap).alloc.func1()
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/mheap.go:1048 +0x4c
runtime.(*mheap).alloc(0x2016600, 0x80, 0x10101, 0x7ffaa0ea2370)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/mheap.go:1047 +0x8a
runtime.largeAlloc(0x100000, 0x101, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/malloc.go:1055 +0x99
runtime.mallocgc.func1()
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/malloc.go:950 +0x46
runtime.systemstack(0x7ffb3effcef0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/proc.go:1153
goroutine 34017 [running]:
runtime.systemstack_switch()
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/asm_amd64.s:311 fp=0xc001bb4b78 sp=0xc001bb4b70 pc=0x45e530
runtime.mallocgc(0x100000, 0xf45da0, 0xc06bf3c201, 0xc06bf3c2a8)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/malloc.go:949 +0x872 fp=0xc001bb4c18 sp=0xc001bb4b78 pc=0x412792
runtime.makeslice(0xf45da0, 0x100000, 0x100000, 0x20)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/slice.go:49 +0x6c fp=0xc001bb4c48 sp=0xc001bb4c18 pc=0x4496fc
github.com/libp2p/go-buffer-pool.(*BufferPool).Get(0xc0003f7880, 0xebe14, 0xc01c559440, 0xc000154700, 0x7ffb724891f0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-buffer-pool@v0.0.2/pool.go:72 +0x168 fp=0xc001bb4c98 sp=0xc001bb4c48 pc=0xd19178
github.com/libp2p/go-buffer-pool.(*Buffer).getBuf(0xc009eae030, 0xebe14, 0xc009eae000, 0xc001bb4d18, 0x412a98)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-buffer-pool@v0.0.2/buffer.go:115 +0x75 fp=0xc001bb4cd0 sp=0xc001bb4c98 pc=0xd18475
github.com/libp2p/go-buffer-pool.(*Buffer).grow(0xc009eae030, 0x420, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-buffer-pool@v0.0.2/buffer.go:86 +0x102 fp=0xc001bb4d28 sp=0xc001bb4cd0 pc=0xd18262
github.com/libp2p/go-buffer-pool.(*Buffer).Grow(...)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-buffer-pool@v0.0.2/buffer.go:198
github.com/libp2p/go-yamux.(*Stream).readData(0xc009eae000, 0x100000000000000, 0x20040000, 0x7ffb72446ef0, 0xc002de3800, 0xc, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/stream.go:420 +0x10b fp=0xc001bb4e08 sp=0xc001bb4d28 pc=0xd8306b
github.com/libp2p/go-yamux.(*Session).handleStreamMessage(0xc003c3dad0, 0x100000000000000, 0xc020040000, 0xc, 0xc)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:574 +0x31d fp=0xc001bb4f00 sp=0xc001bb4e08 pc=0xd8084d
github.com/libp2p/go-yamux.(*Session).recvLoop(0xc003c3dad0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:525 +0x172 fp=0xc001bb4fb0 sp=0xc001bb4f00 pc=0xd80242
github.com/libp2p/go-yamux.(*Session).recv(0xc003c3dad0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:486 +0x2b fp=0xc001bb4fd8 sp=0xc001bb4fb0 pc=0xd8009b
runtime.goexit()
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc001bb4fe0 sp=0xc001bb4fd8 pc=0x460601
created by github.com/libp2p/go-yamux.newSession
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:129 +0x388
goroutine 1 [select (no cases), 2481 minutes]:
github.com/harmony-one/harmony/node.(*Node).StartServer(0xc00007ac00)
/data/harmony/node/node.go:334 +0x20
main.main()
/data/harmony/cmd/harmony/main.go:504 +0xc8b
goroutine 20 [syscall, 2481 minutes]:
os/signal.signal_recv(0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/os/signal/signal_unix.go:29 +0x41
goroutine 21 [chan receive, 2481 minutes]:
github.com/harmony-one/harmony/core.(*txSenderCacher).cache(0xc00016a0a0)
/data/harmony/core/tx_cacher.go:63 +0x97
created by github.com/harmony-one/harmony/core.newTxSenderCacher
/data/harmony/core/tx_cacher.go:55 +0x9f
goroutine 22 [chan receive, 2481 minutes]:
github.com/harmony-one/harmony/core.(*txSenderCacher).cache(0xc00016a0a0)
/data/harmony/core/tx_cacher.go:63 +0x97
created by github.com/harmony-one/harmony/core.newTxSenderCacher
/data/harmony/core/tx_cacher.go:55 +0x9f
goroutine 29 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0000c6c80)
/home/ec2-user/go/pkg/mod/go.opencensus.io@v0.21.0/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
/home/ec2-user/go/pkg/mod/go.opencensus.io@v0.21.0/stats/view/worker.go:32 +0x57
goroutine 31 [select]:
github.com/harmony-one/harmony/core.(*BlockChain).update(0xc000afc000)
/data/harmony/core/blockchain.go:1632 +0xfc
created by github.com/harmony-one/harmony/core.NewBlockChain
/data/harmony/core/blockchain.go:209 +0x6dc
goroutine 7 [chan receive, 2481 minutes]:
github.com/rjeczalik/notify.(*nonrecursiveTree).dispatch(0xc00007ccc0, 0xc00007cc00)
/home/ec2-user/go/pkg/mod/github.com/rjeczalik/notify@v0.9.2/tree_nonrecursive.go:36 +0xb6
created by github.com/rjeczalik/notify.newNonrecursiveTree
/home/ec2-user/go/pkg/mod/github.com/rjeczalik/notify@v0.9.2/tree_nonrecursive.go:29 +0xdd
goroutine 8 [chan receive, 2481 minutes]:
github.com/rjeczalik/notify.(*nonrecursiveTree).internal(0xc00007ccc0, 0xc00007cc60)
/home/ec2-user/go/pkg/mod/github.com/rjeczalik/notify@v0.9.2/tree_nonrecursive.go:81 +0x58
created by github.com/rjeczalik/notify.newNonrecursiveTree
/home/ec2-user/go/pkg/mod/github.com/rjeczalik/notify@v0.9.2/tree_nonrecursive.go:30 +0x109
goroutine 28 [select, 2481 minutes]:
github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0xc0001b0d80)
/home/ec2-user/go/pkg/mod/github.com/ipfs/go-log@v0.0.1/writer/writer.go:76 +0x11c
created by github.com/ipfs/go-log/writer.NewMirrorWriter
/home/ec2-user/go/pkg/mod/github.com/ipfs/go-log@v0.0.1/writer/writer.go:41 +0xb9
goroutine 12 [chan receive, 352 minutes]:
github.com/natefinch/lumberjack.(*Logger).millRun(0xc000442480)
/home/ec2-user/go/pkg/mod/github.com/natefinch/lumberjack@v2.0.0+incompatible/lumberjack.go:379 +0x58
created by github.com/natefinch/lumberjack.(*Logger).mill.func1
/home/ec2-user/go/pkg/mod/github.com/natefinch/lumberjack@v2.0.0+incompatible/lumberjack.go:390 +0x79
goroutine 13 [chan receive]:
github.com/harmony-one/harmony/internal/memprofiling.MaybeCallGCPeriodically.func1()
/data/harmony/internal/memprofiling/lib.go:103 +0x35
created by github.com/harmony-one/harmony/internal/memprofiling.MaybeCallGCPeriodically
/data/harmony/internal/memprofiling/lib.go:100 +0x35
goroutine 14 [chan receive, 1 minutes]:
github.com/harmony-one/harmony/internal/memprofiling.MaybeCallGCPeriodically.func2()
/data/harmony/internal/memprofiling/lib.go:113 +0x31
created by github.com/harmony-one/harmony/internal/memprofiling.MaybeCallGCPeriodically
/data/harmony/internal/memprofiling/lib.go:110 +0x4d
goroutine 15 [select, 21 minutes]:
github.com/libp2p/go-libp2p-peerstore/pstoremem.(*memoryAddrBook).background(0xc0000eed00)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.1.3/pstoremem/addr_book.go:81 +0x11c
created by github.com/libp2p/go-libp2p-peerstore/pstoremem.NewAddrBook
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.1.3/pstoremem/addr_book.go:71 +0x190
goroutine 57 [IO wait, 21 minutes]:
internal/poll.runtime_pollWait(0x7ffb722e6ea8, 0x72, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000120418, 0x72, 0x0, 0x0, 0x11173d0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000120400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:384 +0x1ba
net.(*netFD).accept(0xc000120400, 0xc000065dd8, 0xc000065db0, 0xc000065de0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0000ae1a0, 0xc0001aebe8, 0xc000065e01, 0xc000065e10)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0000ae1a0, 0xc000065e70, 0xccbd0a, 0xc0000c6b90, 0x5)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/tcpsock.go:260 +0x48
github.com/multiformats/go-multiaddr-net.(*maListener).Accept(0xc00061be00, 0x113d57b, 0x27, 0xc05422a570, 0x3)
/home/ec2-user/go/pkg/mod/github.com/multiformats/go-multiaddr-net@v0.0.1/net.go:243 +0x3a
github.com/libp2p/go-tcp-transport.(*lingerListener).Accept(0xc00061be40, 0x0, 0x0, 0xc000442cc0, 0x14aa9e0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-tcp-transport@v0.1.0/tcp.go:42 +0x37
github.com/libp2p/go-libp2p-transport-upgrader.(*listener).handleIncoming(0xc000442cc0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/listener.go:77 +0xdd
created by github.com/libp2p/go-libp2p-transport-upgrader.(*Upgrader).UpgradeListener
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/upgrader.go:47 +0x187
goroutine 56 [chan receive, 2481 minutes]:
github.com/libp2p/go-libp2p-transport-upgrader.(*listener).Accept(0xc000442c00, 0x12fa7e8, 0x149cd00, 0xc000442c00, 0xc00008b980)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/listener.go:144 +0x51
github.com/libp2p/go-libp2p-swarm.(*Swarm).AddListenAddr.func2(0x149cd00, 0xc000442c00, 0xc00008b980)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_listen.go:79 +0x10b
created by github.com/libp2p/go-libp2p-swarm.(*Swarm).AddListenAddr
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_listen.go:69 +0x243
goroutine 51 [select, 1 minutes]:
github.com/libp2p/go-libp2p/p2p/protocol/identify.(*ObservedAddrSet).worker(0xc00061d5c0, 0x149c740, 0xc00044b280)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.3.1/p2p/protocol/identify/obsaddr.go:128 +0x189
created by github.com/libp2p/go-libp2p/p2p/protocol/identify.NewObservedAddrSet
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.3.1/p2p/protocol/identify/obsaddr.go:64 +0xd0
goroutine 54 [select, 2481 minutes]:
github.com/libp2p/go-libp2p-circuit.(*RelayListener).Accept(0xc000442ba0, 0x0, 0x0, 0xc0000381a0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-circuit@v0.1.1/listen.go:26 +0xd6
github.com/libp2p/go-libp2p-transport-upgrader.(*listener).handleIncoming(0xc000442c00)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/listener.go:77 +0xdd
created by github.com/libp2p/go-libp2p-transport-upgrader.(*Upgrader).UpgradeListener
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/upgrader.go:47 +0x187
goroutine 53 [select, 2481 minutes]:
github.com/libp2p/go-libp2p/p2p/protocol/identify.(*IDService).handleEvents(0xc00044e0e0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.3.1/p2p/protocol/identify/id.go:112 +0x18e
created by github.com/libp2p/go-libp2p/p2p/protocol/identify.NewIDService
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.3.1/p2p/protocol/identify/id.go:87 +0x5e8
goroutine 59 [chan receive, 21 minutes]:
github.com/libp2p/go-libp2p-transport-upgrader.(*listener).Accept(0xc000442cc0, 0x12fa7f0, 0xc00008b980, 0x14abd60, 0xc019c11ec0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/listener.go:144 +0x51
github.com/libp2p/go-libp2p-swarm.(*Swarm).AddListenAddr.func2(0x149cd00, 0xc000442cc0, 0xc00008b980)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_listen.go:79 +0x10b
created by github.com/libp2p/go-libp2p-swarm.(*Swarm).AddListenAddr
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_listen.go:69 +0x243
goroutine 43 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0000ba340)
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0xf6
created by github.com/syndtr/goleveldb/leveldb.openDB
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x431
goroutine 42 [select, 348 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0000ba340)
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0xcd
created by github.com/syndtr/goleveldb/leveldb.openDB
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x40f
goroutine 62 [select]:
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).background(0xc0000ccf20, 0x14aa5e0, 0xc000442d20)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.3.1/p2p/host/basic/basic_host.go:327 +0x15c
github.com/jbenet/goprocess.(*process).Go.func1(0xc0004208a0, 0xc000442d20, 0xc000442d80)
/home/ec2-user/go/pkg/mod/github.com/jbenet/goprocess@v0.1.3/impl-mutex.go:134 +0x3c
created by github.com/jbenet/goprocess.(*process).Go
/home/ec2-user/go/pkg/mod/github.com/jbenet/goprocess@v0.1.3/impl-mutex.go:133 +0x1f7
goroutine 63 [select]:
github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).heartbeatTimer(0xc00044b500)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/gossipsub.go:369 +0x228
created by github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).Attach
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/gossipsub.go:76 +0x52
goroutine 64 [select]:
github.com/libp2p/go-libp2p-pubsub.(*validation).validateWorker(0xc00061dfb0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/validation.go:177 +0x129
created by github.com/libp2p/go-libp2p-pubsub.(*validation).Start
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/validation.go:94 +0x5b
goroutine 65 [runnable]:
github.com/libp2p/go-libp2p-pubsub.(*validation).validateWorker(0xc00061dfb0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/validation.go:177 +0x129
created by github.com/libp2p/go-libp2p-pubsub.(*validation).Start
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/validation.go:94 +0x5b
goroutine 49 [select, 21 minutes]:
github.com/dgraph-io/badger/y.(*WaterMark).process(0xc0000b2bc0, 0xc000620a40)
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/y/watermark.go:207 +0x2d1
created by github.com/dgraph-io/badger/y.(*WaterMark).Init
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/y/watermark.go:74 +0xc9
goroutine 11185172 [chan send, 10 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc055c7b385, 0x198, 0x19b, 0xc05bb02270, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 93 [chan receive, 390 minutes]:
github.com/harmony-one/harmony/node.(*Node).DoBeaconSyncing.func1(0xc00007ac00)
/data/harmony/node/node_syncing.go:167 +0x32
created by github.com/harmony-one/harmony/node.(*Node).DoBeaconSyncing
/data/harmony/node/node_syncing.go:164 +0x54
goroutine 46 [chan receive, 2481 minutes]:
github.com/natefinch/lumberjack.(*Logger).millRun(0xc0004423c0)
/home/ec2-user/go/pkg/mod/github.com/natefinch/lumberjack@v2.0.0+incompatible/lumberjack.go:379 +0x58
created by github.com/natefinch/lumberjack.(*Logger).mill.func1
/home/ec2-user/go/pkg/mod/github.com/natefinch/lumberjack@v2.0.0+incompatible/lumberjack.go:390 +0x79
goroutine 45 [select, 396 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0000ba340)
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0x12e
created by github.com/syndtr/goleveldb/leveldb.openDB
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x5a7
goroutine 44 [select, 348 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc0000ba340)
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x23a
created by github.com/syndtr/goleveldb/leveldb.openDB
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x585
goroutine 82 [select]:
github.com/libp2p/go-libp2p-pubsub.(*PubSub).processLoop(0xc0000e2d80, 0x149c780, 0xc0000aa018)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/pubsub.go:277 +0x4d9
created by github.com/libp2p/go-libp2p-pubsub.NewPubSub
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/pubsub.go:206 +0x9ae
goroutine 130 [select, 2481 minutes]:
github.com/dgraph-io/badger/y.(*WaterMark).process(0xc0000b2e40, 0xc000620a40)
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/y/watermark.go:207 +0x2d1
created by github.com/dgraph-io/badger/y.(*WaterMark).Init
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/y/watermark.go:74 +0xc9
goroutine 83 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc0003c21c0)
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0x121
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x176
goroutine 32 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc0003c2460)
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0x121
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x176
goroutine 86 [select, 390 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0000ba4e0)
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0xcd
created by github.com/syndtr/goleveldb/leveldb.openDB
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x40f
goroutine 87 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0000ba4e0)
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0xf6
created by github.com/syndtr/goleveldb/leveldb.openDB
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x431
goroutine 88 [select, 390 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc0000ba4e0)
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x23a
created by github.com/syndtr/goleveldb/leveldb.openDB
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x585
goroutine 89 [select, 392 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0000ba4e0)
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0x12e
created by github.com/syndtr/goleveldb/leveldb.openDB
/home/ec2-user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x5a7
goroutine 90 [select]:
github.com/harmony-one/harmony/core.(*BlockChain).update(0xc000aff000)
/data/harmony/core/blockchain.go:1632 +0xfc
created by github.com/harmony-one/harmony/core.NewBlockChain
/data/harmony/core/blockchain.go:209 +0x6dc
goroutine 91 [select]:
github.com/harmony-one/harmony/core.(*TxPool).loop(0xc001328540)
/data/harmony/core/tx_pool.go:299 +0x280
created by github.com/harmony-one/harmony/core.NewTxPool
/data/harmony/core/tx_pool.go:271 +0x704
goroutine 33 [select]:
github.com/libp2p/go-libp2p-pubsub.(*Subscription).Next(0xc000622550, 0x149c780, 0xc0000aa018, 0x30, 0xc00a690e70, 0xdf7364)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/subscription.go:39 +0xd8
github.com/harmony-one/harmony/p2p/host/hostv2.(*GroupReceiverImpl).Receive(0xc00136f4f0, 0x149c780, 0xc0000aa018, 0xc000446105, 0x6f, 0x7b, 0xc00a690e70, 0x22, 0x0, 0x0)
/data/harmony/p2p/host/hostv2/hostv2.go:93 +0x5d
github.com/harmony-one/harmony/node.(*Node).ReceiveClientGroupMessage(0xc00007ac00)
/data/harmony/node/node_handler.go:87 +0x83
created by github.com/harmony-one/harmony/node.New
/data/harmony/node/node.go:445 +0x890
goroutine 98 [select]:
github.com/libp2p/go-libp2p-pubsub.(*Subscription).Next(0xc000622280, 0x149c780, 0xc0000aa018, 0x30, 0xc06bf2e7e0, 0xdf7364)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/subscription.go:39 +0xd8
github.com/harmony-one/harmony/p2p/host/hostv2.(*GroupReceiverImpl).Receive(0xc00136f430, 0x149c780, 0xc0000aa018, 0xc06bee76c5, 0x198, 0x19b, 0xc06bf2e7e0, 0x22, 0x0, 0x0)
/data/harmony/p2p/host/hostv2/hostv2.go:93 +0x5d
github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage(0xc00007ac00)
/data/harmony/node/node_handler.go:67 +0x83
created by github.com/harmony-one/harmony/node.New
/data/harmony/node/node.go:448 +0x8b5
goroutine 99 [select]:
github.com/libp2p/go-libp2p-pubsub.(*Subscription).Next(0xc000622460, 0x149c780, 0xc0000aa018, 0x30, 0xc025a28690, 0xdf7364)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/subscription.go:39 +0xd8
github.com/harmony-one/harmony/p2p/host/hostv2.(*GroupReceiverImpl).Receive(0xc00136f490, 0x149c780, 0xc0000aa018, 0xc000446485, 0x6f, 0x7b, 0xc025a28690, 0x22, 0x0, 0x0)
/data/harmony/p2p/host/hostv2/hostv2.go:93 +0x5d
github.com/harmony-one/harmony/node.(*Node).ReceiveGlobalMessage(0xc00007ac00)
/data/harmony/node/node_handler.go:48 +0x83
created by github.com/harmony-one/harmony/node.New
/data/harmony/node/node.go:452 +0x8da
goroutine 100 [chan send, 2480 minutes]:
github.com/harmony-one/harmony/node.(*Node).bootstrapConsensus(0xc00007ac00)
/data/harmony/node/node_handler.go:593 +0xec
created by github.com/harmony-one/harmony/node.New
/data/harmony/node/node.go:459 +0x952
goroutine 92 [sleep]:
runtime.goparkunlock(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/proc.go:307
time.Sleep(0x12a05f200)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/time.go:105 +0x159
github.com/harmony-one/harmony/node.(*Node).DoBeaconSyncing(0xc00007ac00)
/data/harmony/node/node_syncing.go:194 +0xcd
created by github.com/harmony-one/harmony/node.(*Node).SupportBeaconSyncing
/data/harmony/node/node_syncing.go:253 +0x3f
goroutine 11443704 [chan send]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc04ddf0005, 0x198, 0x19b, 0xc01ca7ade0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11330029 [chan send, 3 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc02381a345, 0x198, 0x19b, 0xc034ba6e10, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 96 [IO wait, 705 minutes]:
internal/poll.runtime_pollWait(0x7ffb722e6c38, 0x72, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000120b98, 0x72, 0x0, 0x0, 0x11173d0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000120b80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:384 +0x1ba
net.(*netFD).accept(0xc000120b80, 0xc0213e8701, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000fb6028, 0xc000fb0e88, 0xc000fb0e90, 0x18)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000fb6028, 0x12fb0d8, 0xc000631800, 0x14a7620, 0xc0213e8788)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/tcpsock.go:260 +0x48
google.golang.org/grpc.(*Server).Serve(0xc000631800, 0x1498580, 0xc000fb6028, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/google.golang.org/grpc@v1.21.1/server.go:587 +0x1ef
github.com/harmony-one/harmony/api/service/syncing/downloader.(*Server).Start.func1(0xc000631800, 0x1498580, 0xc000fb6028)
/data/harmony/api/service/syncing/downloader/server.go:54 +0x43
created by github.com/harmony-one/harmony/api/service/syncing/downloader.(*Server).Start
/data/harmony/api/service/syncing/downloader/server.go:53 +0x1f7
goroutine 97 [chan receive, 48 minutes]:
github.com/harmony-one/harmony/node.(*Node).SendNewBlockToUnsync(0xc00007ac00)
/data/harmony/node/node_syncing.go:295 +0x66
created by github.com/harmony-one/harmony/node.(*Node).SupportSyncing
/data/harmony/node/node_syncing.go:271 +0xce
goroutine 114 [sleep]:
runtime.goparkunlock(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/proc.go:307
time.Sleep(0x2540be400)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/time.go:105 +0x159
github.com/harmony-one/harmony/node.(*Node).DoSyncing(0xc00007ac00, 0xc000afc000, 0xc0001a8b40, 0xc000038101)
/data/harmony/node/node_syncing.go:247 +0xdf
created by github.com/harmony-one/harmony/node.(*Node).SupportSyncing
/data/harmony/node/node_syncing.go:274 +0xa7
goroutine 984591 [IO wait]:
internal/poll.runtime_pollWait(0x7ffb64739828, 0x72, 0xffffffffffffffff)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0109f1d18, 0x72, 0x0, 0x4, 0xffffffffffffffff)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0109f1d00, 0xc010b9e4d0, 0x4, 0x4, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0109f1d00, 0xc010b9e4d0, 0x4, 0x4, 0x6cb296, 0xc0103efa40, 0xc)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc001013190, 0xc010b9e4d0, 0x4, 0x4, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/net.go:177 +0x69
io.ReadAtLeast(0x7ffb647b9150, 0xc00a4ffce0, 0xc010b9e4d0, 0x4, 0x4, 0x4, 0xc0103efa4c, 0xc001ab2d30, 0xd1932f)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:310 +0x88
io.ReadFull(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:329
github.com/libp2p/go-msgio.ReadLen(0x7ffb647b9150, 0xc00a4ffce0, 0xc010b9e4d0, 0x4, 0x4, 0xc033f50b40, 0xc001ab2d98, 0xc001ab2d78)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/num.go:27 +0x81
github.com/libp2p/go-msgio.(*reader).nextMsgLen(0xc010b9e4c0, 0x12fbec8, 0xc010b9e4e8, 0x40)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/msgio.go:178 +0x78
github.com/libp2p/go-msgio.(*reader).NextMsgLen(0xc010b9e4c0, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/msgio.go:173 +0x7f
github.com/libp2p/go-libp2p-secio.(*etmReader).Read(0xc00ae1c6e0, 0xc010a92b10, 0xc, 0xc, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/rw.go:163 +0xd2
io.ReadAtLeast(0x7ffb72446ef0, 0xc000736e00, 0xc010a92b10, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:310 +0x88
io.ReadFull(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:329
github.com/libp2p/go-yamux.(*Session).recvLoop(0xc00b6f80d0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:507 +0xd4
github.com/libp2p/go-yamux.(*Session).recv(0xc00b6f80d0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:486 +0x2b
created by github.com/libp2p/go-yamux.newSession
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:129 +0x388
goroutine 131 [select]:
github.com/dgraph-io/badger.(*DB).updateSize(0xc0003bb500, 0xc000620b00)
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/db.go:998 +0x12f
created by github.com/dgraph-io/badger.Open
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/db.go:284 +0xa78
goroutine 132 [select]:
github.com/dgraph-io/badger.(*levelsController).runWorker(0xc00044e700, 0xc000620cc0)
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/levels.go:348 +0x1fc
created by github.com/dgraph-io/badger.(*levelsController).startCompact
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/levels.go:329 +0x89
goroutine 133 [select]:
github.com/dgraph-io/badger.(*levelsController).runWorker(0xc00044e700, 0xc000620cc0)
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/levels.go:348 +0x1fc
created by github.com/dgraph-io/badger.(*levelsController).startCompact
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/levels.go:329 +0x89
goroutine 134 [chan receive, 2481 minutes]:
github.com/dgraph-io/badger.(*DB).flushMemtable(0xc0003bb500, 0xc000620ce0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/db.go:912 +0xb3
github.com/dgraph-io/badger.Open.func4(0xc0000105f8)
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/db.go:298 +0x37
created by github.com/dgraph-io/badger.Open
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/db.go:297 +0x14b7
goroutine 71 [select, 2481 minutes]:
github.com/dgraph-io/badger.(*DB).doWrites(0xc0003bb500, 0xc00061af80)
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/db.go:705 +0x3f8
created by github.com/dgraph-io/badger.Open
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/db.go:332 +0xfbf
goroutine 72 [chan receive, 2481 minutes]:
github.com/dgraph-io/badger.(*valueLog).waitOnGC(0xc0003bb688, 0xc00061afa0)
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/value.go:1337 +0x67
created by github.com/dgraph-io/badger.Open
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/db.go:335 +0x1087
goroutine 73 [select, 2481 minutes]:
github.com/dgraph-io/badger.(*publisher).listenForUpdates(0xc000620a60, 0xc00061afc0)
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/publisher.go:65 +0x158
created by github.com/dgraph-io/badger.Open
/home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger@v1.6.0-rc1/db.go:338 +0x114f
goroutine 2195209 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleSendingMessages(0xc0000e2d80, 0x149c780, 0xc0000aa018, 0x14abce0, 0xc0025d7540, 0xc002ef9020)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:118 +0x27c
created by github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleNewPeer
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:79 +0x404
goroutine 11451193 [chan send]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc05d905ba5, 0x198, 0x19b, 0xc05afabe60, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 37134 [select, 3 minutes]:
github.com/libp2p/go-yamux.(*Session).AcceptStream(0xc00a129ad0, 0xc0018720e0, 0x0, 0x43ccef)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:225 +0xb6
github.com/libp2p/go-libp2p-yamux.(*conn).AcceptStream(0xc00a129ad0, 0xc00b07afa0, 0x18, 0xc0012ced80, 0xe2916a)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-yamux@v0.2.1/yamux.go:38 +0x2c
github.com/libp2p/go-libp2p-swarm.(*Conn).start.func1(0xc00b4507e0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_conn.go:98 +0xf0
created by github.com/libp2p/go-libp2p-swarm.(*Conn).start
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_conn.go:93 +0x3f
goroutine 11264318 [chan send, 6 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc055513865, 0x198, 0x19b, 0xc055ec3260, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11454335 [chan send]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc00a2ddd45, 0x198, 0x19b, 0xc0205d4030, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11439342 [chan send]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc015de71e5, 0x198, 0x19b, 0xc0329376e0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 80 [select, 1289 minutes]:
github.com/harmony-one/harmony/api/service/discovery.(*Service).contactP2pPeers(0xc00052d3f0)
/data/harmony/api/service/discovery/service.go:90 +0x35a
created by github.com/harmony-one/harmony/api/service/discovery.(*Service).Run
/data/harmony/api/service/discovery/service.go:73 +0x3f
goroutine 11407609 [chan send, 1 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc00ea4d525, 0x198, 0x19b, 0xc05de3a570, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11405144 [chan send, 1 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc01cb71d45, 0x198, 0x19b, 0xc0457052c0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11362397 [chan send, 2 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc049457865, 0x198, 0x19b, 0xc048c68b10, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 2698177 [select, 1 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc002f29b80)
/home/ec2-user/go/pkg/mod/google.golang.org/grpc@v1.21.1/clientconn.go:1058 +0x380
created by google.golang.org/grpc.(*addrConn).connect
/home/ec2-user/go/pkg/mod/google.golang.org/grpc@v1.21.1/clientconn.go:767 +0xb6
goroutine 199 [select, 2481 minutes]:
github.com/libp2p/go-yamux.(*Stream).Read(0xc000136000, 0xc00052a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/stream.go:113 +0x1a0
github.com/libp2p/go-libp2p-swarm.(*Stream).Read(0xc0001a8dc0, 0xc00052a000, 0x1000, 0x1000, 0x0, 0xc0012d4e28, 0x412a98)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_stream.go:64 +0x64
bufio.(*Reader).fill(0xc000472a20)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadByte(0xc000472a20, 0xc00044bb40, 0x1000, 0x60)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:249 +0x39
encoding/binary.ReadUvarint(0x147ff00, 0xc000472a20, 0x1000, 0x1000, 0x7ffb72446fa8)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/encoding/binary/varint.go:110 +0x7b
github.com/gogo/protobuf/io.(*varintReader).ReadMsg(0xc00044bb40, 0x1497640, 0xc0000b41c0, 0x148e380, 0xc00044bb40)
/home/ec2-user/go/pkg/mod/github.com/gogo/protobuf@v1.2.1/io/varint.go:110 +0x3e
github.com/libp2p/go-libp2p-pubsub.(*PubSub).handlePeerEOF(0xc0000e2d80, 0x149c780, 0xc0000aa018, 0x14abce0, 0xc0001a8dc0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:91 +0x106
created by github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleNewPeer
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:80 +0x45a
goroutine 162 [select, 350 minutes]:
github.com/libp2p/go-yamux.(*Session).AcceptStream(0xc0000a3520, 0xc00b4fd0a0, 0x0, 0x43ccef)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:225 +0xb6
github.com/libp2p/go-libp2p-yamux.(*conn).AcceptStream(0xc0000a3520, 0xc000649fa0, 0x18, 0xc0012cfc80, 0xe2916a)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-yamux@v0.2.1/yamux.go:38 +0x2c
github.com/libp2p/go-libp2p-swarm.(*Conn).start.func1(0xc000484360)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_conn.go:98 +0xf0
created by github.com/libp2p/go-libp2p-swarm.(*Conn).start
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_conn.go:93 +0x3f
goroutine 2044584 [IO wait]:
internal/poll.runtime_pollWait(0x7ffb72450648, 0x72, 0xffffffffffffffff)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0037ba218, 0x72, 0x0, 0x4, 0xffffffffffffffff)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0037ba200, 0xc00e3d3dd0, 0x4, 0x4, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0037ba200, 0xc00e3d3dd0, 0x4, 0x4, 0x6cb296, 0xc02ddb7540, 0xc)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0013c3870, 0xc00e3d3dd0, 0x4, 0x4, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/net.go:177 +0x69
io.ReadAtLeast(0x7ffb647b9150, 0xc00b682c00, 0xc00e3d3dd0, 0x4, 0x4, 0x4, 0xc02ddb754c, 0xc0021dad30, 0xd1932f)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:310 +0x88
io.ReadFull(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:329
github.com/libp2p/go-msgio.ReadLen(0x7ffb647b9150, 0xc00b682c00, 0xc00e3d3dd0, 0x4, 0x4, 0xc02b44d000, 0xc0021dad98, 0xc0021dad78)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/num.go:27 +0x81
github.com/libp2p/go-msgio.(*reader).nextMsgLen(0xc00e3d3dc0, 0x12fbec8, 0xc00e3d3de8, 0x40)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/msgio.go:178 +0x78
github.com/libp2p/go-msgio.(*reader).NextMsgLen(0xc00e3d3dc0, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/msgio.go:173 +0x7f
github.com/libp2p/go-libp2p-secio.(*etmReader).Read(0xc01263e8c0, 0xc000e69630, 0xc, 0xc, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/rw.go:163 +0xd2
io.ReadAtLeast(0x7ffb72446ef0, 0xc00b01ec00, 0xc000e69630, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:310 +0x88
io.ReadFull(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:329
github.com/libp2p/go-yamux.(*Session).recvLoop(0xc0022324e0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:507 +0xd4
github.com/libp2p/go-yamux.(*Session).recv(0xc0022324e0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:486 +0x2b
created by github.com/libp2p/go-yamux.newSession
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:129 +0x388
goroutine 555 [select, 162 minutes]:
github.com/libp2p/go-yamux.(*Session).AcceptStream(0xc0012cc270, 0xc0034bc380, 0x0, 0x43ccef)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:225 +0xb6
github.com/libp2p/go-libp2p-yamux.(*conn).AcceptStream(0xc0012cc270, 0xc000feefa0, 0x18, 0xc00047a300, 0xe2916a)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-yamux@v0.2.1/yamux.go:38 +0x2c
github.com/libp2p/go-libp2p-swarm.(*Conn).start.func1(0xc0012f9da0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_conn.go:98 +0xf0
created by github.com/libp2p/go-libp2p-swarm.(*Conn).start
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_conn.go:93 +0x3f
goroutine 1285 [select, 2481 minutes]:
github.com/libp2p/go-yamux.(*Session).AcceptStream(0xc0016c5380, 0xc001625a40, 0x0, 0x43ccef)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:225 +0xb6
github.com/libp2p/go-libp2p-yamux.(*conn).AcceptStream(0xc0016c5380, 0xc0011dc7a0, 0x18, 0xc00047a480, 0xe2916a)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-yamux@v0.2.1/yamux.go:38 +0x2c
github.com/libp2p/go-libp2p-swarm.(*Conn).start.func1(0xc00176c240)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_conn.go:98 +0xf0
created by github.com/libp2p/go-libp2p-swarm.(*Conn).start
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_conn.go:93 +0x3f
goroutine 11188307 [chan send, 10 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc0603c0685, 0x198, 0x19b, 0xc06225d920, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 3258868 [IO wait]:
internal/poll.runtime_pollWait(0x7ffb64799ec0, 0x72, 0xffffffffffffffff)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0127cb818, 0x72, 0x0, 0x4, 0xffffffffffffffff)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0127cb800, 0xc00f2b3990, 0x4, 0x4, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0127cb800, 0xc00f2b3990, 0x4, 0x4, 0x6cb296, 0xc034f6f840, 0xc)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0010139a8, 0xc00f2b3990, 0x4, 0x4, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/net.go:177 +0x69
io.ReadAtLeast(0x7ffb647b9150, 0xc002142300, 0xc00f2b3990, 0x4, 0x4, 0x4, 0xc034f6f84c, 0xc0095f5d30, 0xd1932f)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:310 +0x88
io.ReadFull(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:329
github.com/libp2p/go-msgio.ReadLen(0x7ffb647b9150, 0xc002142300, 0xc00f2b3990, 0x4, 0x4, 0xc00e274800, 0xc0095f5d98, 0xc0095f5d78)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/num.go:27 +0x81
github.com/libp2p/go-msgio.(*reader).nextMsgLen(0xc00f2b3980, 0x12fbec8, 0xc00f2b39a8, 0x40)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/msgio.go:178 +0x78
github.com/libp2p/go-msgio.(*reader).NextMsgLen(0xc00f2b3980, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/msgio.go:173 +0x7f
github.com/libp2p/go-libp2p-secio.(*etmReader).Read(0xc000b5d220, 0xc009c3cba0, 0xc, 0xc, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/rw.go:163 +0xd2
io.ReadAtLeast(0x7ffb72446ef0, 0xc003237200, 0xc009c3cba0, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:310 +0x88
io.ReadFull(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:329
github.com/libp2p/go-yamux.(*Session).recvLoop(0xc0019d12b0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:507 +0xd4
github.com/libp2p/go-yamux.(*Session).recv(0xc0019d12b0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:486 +0x2b
created by github.com/libp2p/go-yamux.newSession
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:129 +0x388
goroutine 136 [IO wait]:
internal/poll.runtime_pollWait(0x7ffb722e69c8, 0x72, 0xffffffffffffffff)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0001a0498, 0x72, 0x0, 0x4, 0xffffffffffffffff)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0001a0480, 0xc00044b990, 0x4, 0x4, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0001a0480, 0xc00044b990, 0x4, 0x4, 0x6cb296, 0xc022edee00, 0xc)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000010530, 0xc00044b990, 0x4, 0x4, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/net.go:177 +0x69
io.ReadAtLeast(0x7ffb647b9150, 0xc000fb4360, 0xc00044b990, 0x4, 0x4, 0x4, 0xc022edee0c, 0xc001243d30, 0xd1932f)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:310 +0x88
io.ReadFull(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:329
github.com/libp2p/go-msgio.ReadLen(0x7ffb647b9150, 0xc000fb4360, 0xc00044b990, 0x4, 0x4, 0xc01ba1aec0, 0xc001243d98, 0xc001243d78)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/num.go:27 +0x81
github.com/libp2p/go-msgio.(*reader).nextMsgLen(0xc00044b980, 0x12fbec8, 0xc00044b9a8, 0x40)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/msgio.go:178 +0x78
github.com/libp2p/go-msgio.(*reader).NextMsgLen(0xc00044b980, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/msgio.go:173 +0x7f
github.com/libp2p/go-libp2p-secio.(*etmReader).Read(0xc0003a8000, 0xc000aa0040, 0xc, 0xc, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/rw.go:163 +0xd2
io.ReadAtLeast(0x7ffb72446ef0, 0xc0001b4200, 0xc000aa0040, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:310 +0x88
io.ReadFull(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:329
github.com/libp2p/go-yamux.(*Session).recvLoop(0xc0000a3520, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:507 +0xd4
github.com/libp2p/go-yamux.(*Session).recv(0xc0000a3520)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:486 +0x2b
created by github.com/libp2p/go-yamux.newSession
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:129 +0x388
goroutine 137 [select]:
github.com/libp2p/go-yamux.(*Session).sendLoop(0xc0000a3520, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:438 +0x1c7
github.com/libp2p/go-yamux.(*Session).send(0xc0000a3520)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:384 +0x2b
created by github.com/libp2p/go-yamux.newSession
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:130 +0x3aa
goroutine 11421407 [chan send, 1 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc0496b6825, 0x198, 0x19b, 0xc0637db8f0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 1707 [select, 2481 minutes]:
github.com/libp2p/go-yamux.(*Stream).Read(0xc0003c2ee0, 0xc0019a0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/stream.go:113 +0x1a0
github.com/libp2p/go-libp2p-swarm.(*Stream).Read(0xc0006239f0, 0xc0019a0000, 0x1000, 0x1000, 0x64, 0x0, 0xc0019773b0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_stream.go:64 +0x64
github.com/multiformats/go-multistream.(*lazyServerConn).Read(0xc001996030, 0xc0019a0000, 0x1000, 0x1000, 0xc000442e98, 0xc00042d030, 0xc001990e48)
/home/ec2-user/go/pkg/mod/github.com/multiformats/go-multistream@v0.1.0/lazyServer.go:32 +0x76
github.com/libp2p/go-libp2p/p2p/host/basic.(*streamWrapper).Read(0xc0004e8ee0, 0xc0019a0000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.3.1/p2p/host/basic/basic_host.go:766 +0x52
bufio.(*Reader).fill(0xc00193c6c0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadByte(0xc00193c6c0, 0x0, 0x0, 0x60)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:249 +0x39
encoding/binary.ReadUvarint(0x147ff00, 0xc00193c6c0, 0x63, 0xd8901d, 0xc001990ed8)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/encoding/binary/varint.go:110 +0x7b
github.com/gogo/protobuf/io.(*varintReader).ReadMsg(0xc0003a7240, 0x1497640, 0xc00042d030, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/gogo/protobuf@v1.2.1/io/varint.go:110 +0x3e
github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleNewStream(0xc0000e2d80, 0x14abde0, 0xc0004e8ee0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:36 +0xcf
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).SetStreamHandler.func1(0xc001364b90, 0xe, 0x7ffb72447070, 0xc0004e8ee0, 0xe, 0xc0004208c0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.3.1/p2p/host/basic/basic_host.go:394 +0x9d
created by github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).newStreamHandler
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.3.1/p2p/host/basic/basic_host.go:292 +0x6a3
goroutine 11242663 [chan send, 8 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc04dd76825, 0x198, 0x19b, 0xc03e5eb3e0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 184 [select, 2481 minutes]:
github.com/libp2p/go-yamux.(*Stream).Read(0xc0003c2620, 0xc0003c9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/stream.go:113 +0x1a0
github.com/libp2p/go-libp2p-swarm.(*Stream).Read(0xc000616cd0, 0xc0003c9000, 0x1000, 0x1000, 0x43578c, 0xc00123ecd8, 0x45d0d0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_stream.go:64 +0x64
github.com/multiformats/go-multistream.(*lazyServerConn).Read(0xc0003b65a0, 0xc0003c9000, 0x1000, 0x1000, 0xc00123eeb8, 0xc000032230, 0xc00123ee48)
/home/ec2-user/go/pkg/mod/github.com/multiformats/go-multistream@v0.1.0/lazyServer.go:32 +0x76
github.com/libp2p/go-libp2p/p2p/host/basic.(*streamWrapper).Read(0xc0001925a0, 0xc0003c9000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.3.1/p2p/host/basic/basic_host.go:766 +0x52
bufio.(*Reader).fill(0xc000472960)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadByte(0xc000472960, 0x0, 0x0, 0x60)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:249 +0x39
encoding/binary.ReadUvarint(0x147ff00, 0xc000472960, 0x0, 0xd8901d, 0xc00123eed8)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/encoding/binary/varint.go:110 +0x7b
github.com/gogo/protobuf/io.(*varintReader).ReadMsg(0xc00044bb00, 0x1497640, 0xc000032230, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/gogo/protobuf@v1.2.1/io/varint.go:110 +0x3e
github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleNewStream(0xc0000e2d80, 0x14abde0, 0xc0001925a0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:36 +0xcf
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).SetStreamHandler.func1(0xc000448270, 0xe, 0x7ffb72447070, 0xc0001925a0, 0xe, 0xc0004208c0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.3.1/p2p/host/basic/basic_host.go:394 +0x9d
created by github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).newStreamHandler
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.3.1/p2p/host/basic/basic_host.go:292 +0x6a3
goroutine 11414501 [chan send, 1 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc010aab045, 0x198, 0x19b, 0xc048fcc3f0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 23476 [select, 2480 minutes]:
github.com/libp2p/go-yamux.(*Stream).Read(0xc003741c00, 0xc001737000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/stream.go:113 +0x1a0
github.com/libp2p/go-libp2p-swarm.(*Stream).Read(0xc002da59f0, 0xc001737000, 0x1000, 0x1000, 0xc002e11a90, 0xc0018f7628, 0x412a98)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_stream.go:64 +0x64
bufio.(*Reader).fill(0xc00345ae40)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadByte(0xc00345ae40, 0xc0031c8680, 0x1000, 0x60)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:249 +0x39
encoding/binary.ReadUvarint(0x147ff00, 0xc00345ae40, 0x1000, 0x1000, 0x7ffb72446fa8)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/encoding/binary/varint.go:110 +0x7b
github.com/gogo/protobuf/io.(*varintReader).ReadMsg(0xc0031c8680, 0x1497640, 0xc00308b420, 0x148e380, 0xc0031c8680)
/home/ec2-user/go/pkg/mod/github.com/gogo/protobuf@v1.2.1/io/varint.go:110 +0x3e
github.com/libp2p/go-libp2p-pubsub.(*PubSub).handlePeerEOF(0xc0000e2d80, 0x149c780, 0xc0000aa018, 0x14abce0, 0xc002da59f0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:91 +0x106
created by github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleNewPeer
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:80 +0x45a
goroutine 11239210 [chan send, 8 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc05c800d05, 0x198, 0x19b, 0xc05c3bfa40, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11344556 [chan send, 3 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc0281c76c5, 0x198, 0x19b, 0xc0268cced0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11387990 [chan send, 2 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc049c2c825, 0x198, 0x19b, 0xc05423f8c0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11248906 [chan send, 8 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc0006dd865, 0x198, 0x19b, 0xc059a56ae0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11427949 [chan send, 1 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc027e34ea5, 0x198, 0x19b, 0xc028f166c0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 660 [runnable]:
github.com/harmony-one/bls/ffi/go/bls._Cfunc_blsSignatureDeserialize(0xc06bed07e0, 0xc00305cea0, 0x60, 0x60)
_cgo_gotypes.go:548 +0x4e
github.com/harmony-one/bls/ffi/go/bls.(*Sign).Deserialize.func1(0xc06bed07e0, 0xc00fd58c90, 0x60)
/home/ec2-user/go/pkg/mod/github.com/harmony-one/bls@v0.0.5/ffi/go/bls/bls.go:398 +0xec
github.com/harmony-one/bls/ffi/go/bls.(*Sign).Deserialize(0xc06bed07e0, 0xc00305cea0, 0x60, 0x60, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/harmony-one/bls@v0.0.5/ffi/go/bls/bls.go:398 +0xaa
github.com/harmony-one/harmony/consensus.ParseViewChangeMessage(0xc02b9cb3b0, 0xc04a13f680, 0x0, 0x0)
/data/harmony/consensus/pbft_log.go:277 +0x42c
github.com/harmony-one/harmony/consensus.(*Consensus).onViewChange(0xc00013c000, 0xc02b9cb3b0)
/data/harmony/consensus/view_change.go:200 +0x43
github.com/harmony-one/harmony/consensus.(*Consensus).handleMessageUpdate(0xc00013c000, 0xc04e180b66, 0x197, 0x19a)
/data/harmony/consensus/consensus_v2.go:81 +0x1db
github.com/harmony-one/harmony/consensus.(*Consensus).Start.func1(0xc00013c000, 0xc0004d0000, 0xc001079a40, 0xc000fedc20, 0xc0010799e0)
/data/harmony/consensus/consensus_v2.go:1175 +0x824
created by github.com/harmony-one/harmony/consensus.(*Consensus).Start
/data/harmony/consensus/consensus_v2.go:1028 +0x67
goroutine 228948 [select]:
github.com/libp2p/go-yamux.(*Stream).Read(0xc003d83ce0, 0xc002e33000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/stream.go:113 +0x1a0
github.com/libp2p/go-libp2p-swarm.(*Stream).Read(0xc00f5b1270, 0xc002e33000, 0x1000, 0x1000, 0x43578c, 0xc0095e1cd8, 0x45d0d0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_stream.go:64 +0x64
github.com/multiformats/go-multistream.(*lazyServerConn).Read(0xc003459a40, 0xc002e33000, 0x1000, 0x1000, 0xc0095e1eb8, 0xc00b90d960, 0xc0095e1e48)
/home/ec2-user/go/pkg/mod/github.com/multiformats/go-multistream@v0.1.0/lazyServer.go:32 +0x76
github.com/libp2p/go-libp2p/p2p/host/basic.(*streamWrapper).Read(0xc003a3d7a0, 0xc002e33000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.3.1/p2p/host/basic/basic_host.go:766 +0x52
bufio.(*Reader).fill(0xc002fa1d40)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadByte(0xc002fa1d40, 0x0, 0x0, 0x60)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:249 +0x39
encoding/binary.ReadUvarint(0x147ff00, 0xc002fa1d40, 0xa7, 0xd8901d, 0xc0095e1ed8)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/encoding/binary/varint.go:110 +0x7b
github.com/gogo/protobuf/io.(*varintReader).ReadMsg(0xc0021eeb00, 0x1497640, 0xc00b90d960, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/gogo/protobuf@v1.2.1/io/varint.go:110 +0x3e
github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleNewStream(0xc0000e2d80, 0x14abde0, 0xc003a3d7a0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:36 +0xcf
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).SetStreamHandler.func1(0xc003d47520, 0xe, 0x7ffb72447070, 0xc003a3d7a0, 0xe, 0xc0004208c0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.3.1/p2p/host/basic/basic_host.go:394 +0x9d
created by github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).newStreamHandler
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.3.1/p2p/host/basic/basic_host.go:292 +0x6a3
goroutine 198 [select, 2481 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleSendingMessages(0xc0000e2d80, 0x149c780, 0xc0000aa018, 0x14abce0, 0xc0001a8dc0, 0xc000484600)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:118 +0x27c
created by github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleNewPeer
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:79 +0x404
goroutine 2568 [select, 2481 minutes]:
github.com/libp2p/go-yamux.(*Session).AcceptStream(0xc00172d790, 0xc001aa07e0, 0x0, 0x43ccef)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:225 +0xb6
github.com/libp2p/go-libp2p-yamux.(*conn).AcceptStream(0xc00172d790, 0xc001a41fa0, 0x18, 0xc000fb2600, 0xe2916a)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-yamux@v0.2.1/yamux.go:38 +0x2c
github.com/libp2p/go-libp2p-swarm.(*Conn).start.func1(0xc00190de60)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_conn.go:98 +0xf0
created by github.com/libp2p/go-libp2p-swarm.(*Conn).start
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_conn.go:93 +0x3f
goroutine 11329931 [chan send, 3 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc02010fa05, 0x198, 0x19b, 0xc0240e5d70, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11177347 [chan send, 10 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc040f19525, 0x198, 0x19b, 0xc02737dec0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11311588 [chan send, 4 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc043a889c5, 0x198, 0x19b, 0xc05d9e22a0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11390421 [chan send, 2 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc05e0bab65, 0x198, 0x19b, 0xc05c741350, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 2698131 [chan receive, 1799 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc0000e1080)
/home/ec2-user/go/pkg/mod/google.golang.org/grpc@v1.21.1/clientconn.go:1107 +0x5a7
created by google.golang.org/grpc.(*addrConn).connect
/home/ec2-user/go/pkg/mod/google.golang.org/grpc@v1.21.1/clientconn.go:767 +0xb6
goroutine 11453526 [chan send]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc024725a05, 0x198, 0x19b, 0xc0251d1a10, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11201242 [chan send, 10 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc057b9fba5, 0x198, 0x19b, 0xc04af52150, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11413985 [chan send, 1 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc01214eea5, 0x198, 0x19b, 0xc045ce5a10, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 500627 [select, 2362 minutes]:
github.com/libp2p/go-yamux.(*Stream).Read(0xc011abd6c0, 0xc002a80000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/stream.go:113 +0x1a0
github.com/libp2p/go-libp2p-swarm.(*Stream).Read(0xc01148a1e0, 0xc002a80000, 0x1000, 0x1000, 0x0, 0xc001580e28, 0x412a98)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_stream.go:64 +0x64
bufio.(*Reader).fill(0xc00259faa0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadByte(0xc00259faa0, 0xc003339540, 0x1000, 0x60)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:249 +0x39
encoding/binary.ReadUvarint(0x147ff00, 0xc00259faa0, 0x1000, 0x1000, 0x7ffb72446fa8)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/encoding/binary/varint.go:110 +0x7b
github.com/gogo/protobuf/io.(*varintReader).ReadMsg(0xc003339540, 0x1497640, 0xc00308a850, 0x148e380, 0xc003339540)
/home/ec2-user/go/pkg/mod/github.com/gogo/protobuf@v1.2.1/io/varint.go:110 +0x3e
github.com/libp2p/go-libp2p-pubsub.(*PubSub).handlePeerEOF(0xc0000e2d80, 0x149c780, 0xc0000aa018, 0x14abce0, 0xc01148a1e0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:91 +0x106
created by github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleNewPeer
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:80 +0x45a
goroutine 4043148 [select, 330 minutes]:
github.com/libp2p/go-yamux.(*Session).AcceptStream(0xc00158fd40, 0xc001592700, 0x0, 0x43ccef)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:225 +0xb6
github.com/libp2p/go-libp2p-yamux.(*conn).AcceptStream(0xc00158fd40, 0xc002d31fa0, 0x18, 0xc000fda180, 0xe2916a)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-yamux@v0.2.1/yamux.go:38 +0x2c
github.com/libp2p/go-libp2p-swarm.(*Conn).start.func1(0xc01a02afc0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_conn.go:98 +0xf0
created by github.com/libp2p/go-libp2p-swarm.(*Conn).start
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_conn.go:93 +0x3f
goroutine 11342848 [chan send, 3 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc02280ba05, 0x198, 0x19b, 0xc03f7361e0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 668 [IO wait, 2481 minutes]:
internal/poll.runtime_pollWait(0x7ffb647b9f08, 0x72, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000ae9d98, 0x72, 0x0, 0x0, 0x11173d0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000ae9d80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:384 +0x1ba
net.(*netFD).accept(0xc000ae9d80, 0xc00004b900, 0x20110c0, 0x7ffb724891f0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc001012980, 0xc0004a3eb0, 0x412a98, 0x30)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc001012980, 0x106f660, 0xc001456390, 0xfa3ee0, 0x1ed4ad0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/tcpsock.go:260 +0x48
net/http.(*Server).Serve(0xc00144ba00, 0x1498580, 0xc001012980, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2859 +0x22d
created by github.com/ethereum/go-ethereum/rpc.StartWSEndpoint
/home/ec2-user/go/pkg/mod/github.com/ethereum/go-ethereum@v1.8.27/rpc/endpoints.go:80 +0x48f
goroutine 11364951 [chan send, 2 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc00b9d3525, 0x198, 0x19b, 0xc04da9f380, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11414506 [chan send, 1 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc0118beea5, 0x198, 0x19b, 0xc048fcdad0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11215824 [chan send, 9 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc05869b385, 0x198, 0x19b, 0xc05fec6fc0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 30829 [select]:
github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleSendingMessages(0xc0000e2d80, 0x149c780, 0xc0000aa018, 0x14abce0, 0xc009724e60, 0xc009756a80)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:118 +0x27c
created by github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleNewPeer
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:79 +0x404
goroutine 11280044 [chan send, 6 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc017e31d45, 0x198, 0x19b, 0xc02a0ab170, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 30693 [IO wait]:
internal/poll.runtime_pollWait(0x7ffb722e6dd8, 0x72, 0xffffffffffffffff)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc003c1e418, 0x72, 0x0, 0x4, 0xffffffffffffffff)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc003c1e400, 0xc00974f0d0, 0x4, 0x4, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc003c1e400, 0xc00974f0d0, 0x4, 0x4, 0x4, 0xc000faec38, 0xc001a58000)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000af618, 0xc00974f0d0, 0x4, 0x4, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/net.go:177 +0x69
io.ReadAtLeast(0x7ffb647b9150, 0xc003f40e40, 0xc00974f0d0, 0x4, 0x4, 0x4, 0x12, 0xc000faed68, 0x432bad)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:310 +0x88
io.ReadFull(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:329
github.com/libp2p/go-msgio.ReadLen(0x7ffb647b9150, 0xc003f40e40, 0xc00974f0d0, 0x4, 0x4, 0xc00120c6c0, 0x0, 0xc000faed78)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/num.go:27 +0x81
github.com/libp2p/go-msgio.(*reader).nextMsgLen(0xc00974f0c0, 0x12fbec8, 0xc00974f0e8, 0xc003384000)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/msgio.go:178 +0x78
github.com/libp2p/go-msgio.(*reader).NextMsgLen(0xc00974f0c0, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/msgio.go:173 +0x7f
github.com/libp2p/go-libp2p-secio.(*etmReader).Read(0xc003929360, 0xc003ede5d0, 0xc, 0xc, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/rw.go:163 +0xd2
io.ReadAtLeast(0x7ffb72446ef0, 0xc0039d7400, 0xc003ede5d0, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:310 +0x88
io.ReadFull(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:329
github.com/libp2p/go-yamux.(*Session).recvLoop(0xc002aeaa90, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:507 +0xd4
github.com/libp2p/go-yamux.(*Session).recv(0xc002aeaa90)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:486 +0x2b
created by github.com/libp2p/go-yamux.newSession
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:129 +0x388
goroutine 11264559 [chan send, 6 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc051283865, 0x198, 0x19b, 0xc05d8885d0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11390397 [chan send, 2 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc05e8fda05, 0x198, 0x19b, 0xc05ecb37d0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11221826 [chan send, 9 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc04895c005, 0x198, 0x19b, 0xc04b29f6b0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11439436 [chan send]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc00b9b8b65, 0x198, 0x19b, 0xc02ecce3f0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11247237 [chan send, 8 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc047149d45, 0x198, 0x19b, 0xc01e04cde0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 500902 [select]:
github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleSendingMessages(0xc0000e2d80, 0x149c780, 0xc0000aa018, 0x14abce0, 0xc00c3c8820, 0xc0039eb500)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:118 +0x27c
created by github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleNewPeer
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:79 +0x404
goroutine 664 [select, 2481 minutes]:
github.com/harmony-one/harmony/core.(*ChainIndexer).updateLoop(0xc00048a3c0)
/data/harmony/core/chain_indexer.go:314 +0x11f
created by github.com/harmony-one/harmony/core.NewChainIndexer
/data/harmony/core/chain_indexer.go:119 +0x49a
goroutine 666 [chan receive, 1 minutes]:
github.com/harmony-one/harmony/internal/hmyapi/filters.(*PublicFilterAPI).timeoutLoop(0xc001224dc0)
/data/harmony/internal/hmyapi/filters/api.go:65 +0x122
created by github.com/harmony-one/harmony/internal/hmyapi/filters.NewPublicFilterAPI
/data/harmony/internal/hmyapi/filters/api.go:55 +0x149
goroutine 11346367 [chan send, 3 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc02c9376c5, 0x198, 0x19b, 0xc00bcc7080, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 532093 [select, 2354 minutes]:
github.com/libp2p/go-yamux.(*Session).AcceptStream(0xc0012cd860, 0xc00bf69ce0, 0x0, 0x43ccef)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:225 +0xb6
github.com/libp2p/go-libp2p-yamux.(*conn).AcceptStream(0xc0012cd860, 0xc00232dfa0, 0x18, 0xc001614a80, 0xe2916a)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-yamux@v0.2.1/yamux.go:38 +0x2c
github.com/libp2p/go-libp2p-swarm.(*Conn).start.func1(0xc003b57980)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_conn.go:98 +0xf0
created by github.com/libp2p/go-libp2p-swarm.(*Conn).start
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_conn.go:93 +0x3f
goroutine 31434 [select, 2480 minutes]:
github.com/libp2p/go-yamux.(*Stream).Read(0xc0038a6d20, 0xc0097ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/stream.go:113 +0x1a0
github.com/libp2p/go-libp2p-swarm.(*Stream).Read(0xc009461180, 0xc0097ff000, 0x1000, 0x1000, 0xd88f0c, 0xc0027fee28, 0x412a98)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_stream.go:64 +0x64
bufio.(*Reader).fill(0xc0097b4c00)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadByte(0xc0097b4c00, 0xc0031e3700, 0x1000, 0x60)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:249 +0x39
encoding/binary.ReadUvarint(0x147ff00, 0xc0097b4c00, 0x1000, 0x1000, 0x7ffb72446fa8)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/encoding/binary/varint.go:110 +0x7b
github.com/gogo/protobuf/io.(*varintReader).ReadMsg(0xc0031e3700, 0x1497640, 0xc009448e00, 0x148e380, 0xc0031e3700)
/home/ec2-user/go/pkg/mod/github.com/gogo/protobuf@v1.2.1/io/varint.go:110 +0x3e
github.com/libp2p/go-libp2p-pubsub.(*PubSub).handlePeerEOF(0xc0000e2d80, 0x149c780, 0xc0000aa018, 0x14abce0, 0xc009461180)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:91 +0x106
created by github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleNewPeer
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:80 +0x45a
goroutine 596 [IO wait]:
internal/poll.runtime_pollWait(0x7ffb722e68f8, 0x72, 0xffffffffffffffff)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0001a0618, 0x72, 0x0, 0x4, 0xffffffffffffffff)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0001a0600, 0xc00162d590, 0x4, 0x4, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0001a0600, 0xc00162d590, 0x4, 0x4, 0x6cb296, 0x45c740, 0xc000049648)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000108b0, 0xc00162d590, 0x4, 0x4, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/net.go:177 +0x69
io.ReadAtLeast(0x7ffb647b9150, 0xc000562e10, 0xc00162d590, 0x4, 0x4, 0x4, 0xbfd7c3d3, 0xc000413d68, 0x432bad)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:310 +0x88
io.ReadFull(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:329
github.com/libp2p/go-msgio.ReadLen(0x7ffb647b9150, 0xc000562e10, 0xc00162d590, 0x4, 0x4, 0xc001dbd580, 0xc000413d98, 0xc000413d78)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/num.go:27 +0x81
github.com/libp2p/go-msgio.(*reader).nextMsgLen(0xc00162d580, 0x12fbec8, 0xc00162d5a8, 0x40)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/msgio.go:178 +0x78
github.com/libp2p/go-msgio.(*reader).NextMsgLen(0xc00162d580, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/msgio.go:173 +0x7f
github.com/libp2p/go-libp2p-secio.(*etmReader).Read(0xc001628820, 0xc000ab2700, 0xc, 0xc, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/rw.go:163 +0xd2
io.ReadAtLeast(0x7ffb72446ef0, 0xc0005cc000, 0xc000ab2700, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:310 +0x88
io.ReadFull(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:329
github.com/libp2p/go-yamux.(*Session).recvLoop(0xc0012cc270, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:507 +0xd4
github.com/libp2p/go-yamux.(*Session).recv(0xc0012cc270)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:486 +0x2b
created by github.com/libp2p/go-yamux.newSession
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:129 +0x388
goroutine 11419600 [chan send, 1 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc040f1e825, 0x198, 0x19b, 0xc0633fe6f0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 662 [select, 7 minutes]:
github.com/harmony-one/harmony/node.(*Node).WaitForConsensusReadyV2.func1(0xc001079b00, 0xc00007ac00, 0xc001079aa0, 0xc000087bc0)
/data/harmony/node/node_newblock.go:40 +0x46b
created by github.com/harmony-one/harmony/node.(*Node).WaitForConsensusReadyV2
/data/harmony/node/node_newblock.go:27 +0x5d
goroutine 11251921 [chan send, 7 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc0461f1385, 0x198, 0x19b, 0xc0549a5620, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11402733 [chan send, 1 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc04bfff1e5, 0x198, 0x19b, 0xc0591eb440, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11259024 [chan send, 7 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc01aed31e5, 0x198, 0x19b, 0xc049043680, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11261049 [chan send, 7 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc018eea685, 0x198, 0x19b, 0xc04fcc3380, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11411263 [chan send, 1 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc01202ed05, 0x198, 0x19b, 0xc00b734c90, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11244952 [chan send, 8 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc00e204ea5, 0x198, 0x19b, 0xc03036db60, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11403037 [chan send, 1 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc050d99385, 0x198, 0x19b, 0xc044fe9bf0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11340010 [chan send, 3 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc046dcba05, 0x198, 0x19b, 0xc05ddcdc20, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11435150 [chan send]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc05e60f045, 0x198, 0x19b, 0xc0630ca270, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11393401 [chan send, 1 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc059651525, 0x198, 0x19b, 0xc04df838f0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11391074 [chan send, 1 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc05b0f3a05, 0x198, 0x19b, 0xc05832ccf0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11259840 [chan send, 7 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc04dfe8685, 0x198, 0x19b, 0xc040eb6090, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11352191 [chan send, 2 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc058596b65, 0x198, 0x19b, 0xc05fb338f0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11387127 [chan send, 2 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc042550825, 0x198, 0x19b, 0xc05dd4d560, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11245423 [chan send, 8 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc000566b65, 0x198, 0x19b, 0xc003aeea20, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 24017 [select, 2480 minutes]:
github.com/libp2p/go-yamux.(*Stream).Read(0xc00393ab60, 0xc002b43000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/stream.go:113 +0x1a0
github.com/libp2p/go-libp2p-swarm.(*Stream).Read(0xc00318b360, 0xc002b43000, 0x1000, 0x1000, 0xc00176d500, 0xc0095a8e28, 0x412a98)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_stream.go:64 +0x64
bufio.(*Reader).fill(0xc0034b3d40)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadByte(0xc0034b3d40, 0xc003414e00, 0x1000, 0x60)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:249 +0x39
encoding/binary.ReadUvarint(0x147ff00, 0xc0034b3d40, 0x1000, 0x1000, 0x7ffb72446fa8)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/encoding/binary/varint.go:110 +0x7b
github.com/gogo/protobuf/io.(*varintReader).ReadMsg(0xc003414e00, 0x1497640, 0xc002d94b60, 0x148e380, 0xc003414e00)
/home/ec2-user/go/pkg/mod/github.com/gogo/protobuf@v1.2.1/io/varint.go:110 +0x3e
github.com/libp2p/go-libp2p-pubsub.(*PubSub).handlePeerEOF(0xc0000e2d80, 0x149c780, 0xc0000aa018, 0x14abce0, 0xc00318b360)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:91 +0x106
created by github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleNewPeer
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:80 +0x45a
goroutine 11268658 [chan send, 6 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc0255b5a05, 0x198, 0x19b, 0xc025da0330, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11223554 [chan send, 9 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc043c3c825, 0x198, 0x19b, 0xc02bda0180, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 500512 [IO wait]:
internal/poll.runtime_pollWait(0x7ffb64739758, 0x72, 0xffffffffffffffff)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc011a1ce98, 0x72, 0x0, 0x4, 0xffffffffffffffff)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc011a1ce80, 0xc0033389d0, 0x4, 0x4, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc011a1ce80, 0xc0033389d0, 0x4, 0x4, 0x6cb296, 0xc011988900, 0xc)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00b186240, 0xc0033389d0, 0x4, 0x4, 0x0, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/net/net.go:177 +0x69
io.ReadAtLeast(0x7ffb647b9150, 0xc003f1c000, 0xc0033389d0, 0x4, 0x4, 0x4, 0xc01198890c, 0xc0030ebd30, 0xd1932f)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:310 +0x88
io.ReadFull(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:329
github.com/libp2p/go-msgio.ReadLen(0x7ffb647b9150, 0xc003f1c000, 0xc0033389d0, 0x4, 0x4, 0xc0023f0540, 0xc0030ebd98, 0xc0030ebd78)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/num.go:27 +0x81
github.com/libp2p/go-msgio.(*reader).nextMsgLen(0xc0033389c0, 0x12fbec8, 0xc0033389e8, 0x40)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/msgio.go:178 +0x78
github.com/libp2p/go-msgio.(*reader).NextMsgLen(0xc0033389c0, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-msgio@v0.0.4/msgio.go:173 +0x7f
github.com/libp2p/go-libp2p-secio.(*etmReader).Read(0xc00a9732c0, 0xc00d96d8d0, 0xc, 0xc, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/rw.go:163 +0xd2
io.ReadAtLeast(0x7ffb72446ef0, 0xc011b20800, 0xc00d96d8d0, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:310 +0x88
io.ReadFull(...)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/io/io.go:329
github.com/libp2p/go-yamux.(*Session).recvLoop(0xc00a934f70, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:507 +0xd4
github.com/libp2p/go-yamux.(*Session).recv(0xc00a934f70)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:486 +0x2b
created by github.com/libp2p/go-yamux.newSession
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/session.go:129 +0x388
goroutine 11249076 [chan send, 8 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc011f79ba5, 0x198, 0x19b, 0xc05fff0db0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11265249 [chan send, 6 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc0537009c5, 0x198, 0x19b, 0xc042f9f0b0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 558 [select, 2481 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleSendingMessages(0xc0000e2d80, 0x149c780, 0xc0000aa018, 0x14abce0, 0xc000ab1810, 0xc000ab7aa0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:118 +0x27c
created by github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleNewPeer
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:79 +0x404
goroutine 10245334 [select, 52 minutes]:
github.com/libp2p/go-yamux.(*Stream).Read(0xc0037400e0, 0xc0034d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-yamux@v1.2.3/stream.go:113 +0x1a0
github.com/libp2p/go-libp2p-swarm.(*Stream).Read(0xc01ad543c0, 0xc0034d3000, 0x1000, 0x1000, 0xc0313f6520, 0xc001725628, 0x412a98)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.1/swarm_stream.go:64 +0x64
bufio.(*Reader).fill(0xc028f22780)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadByte(0xc028f22780, 0xc02d163800, 0x1000, 0x60)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:249 +0x39
encoding/binary.ReadUvarint(0x147ff00, 0xc028f22780, 0x1000, 0x1000, 0x7ffb72446fa8)
/home/ec2-user/.gimme/versions/go1.12.linux.amd64/src/encoding/binary/varint.go:110 +0x7b
github.com/gogo/protobuf/io.(*varintReader).ReadMsg(0xc02d163800, 0x1497640, 0xc02a454460, 0x148e380, 0xc02d163800)
/home/ec2-user/go/pkg/mod/github.com/gogo/protobuf@v1.2.1/io/varint.go:110 +0x3e
github.com/libp2p/go-libp2p-pubsub.(*PubSub).handlePeerEOF(0xc0000e2d80, 0x149c780, 0xc0000aa018, 0x14abce0, 0xc01ad543c0)
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:91 +0x106
created by github.com/libp2p/go-libp2p-pubsub.(*PubSub).handleNewPeer
/home/ec2-user/go/pkg/mod/github.com/libp2p/go-libp2p-pubsub@v0.1.1/comm.go:80 +0x45a
goroutine 11474197 [chan send]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc03e4b0685, 0x198, 0x19b, 0xc05db729c0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
goroutine 11358263 [chan send, 2 minutes]:
github.com/harmony-one/harmony/node.(*Node).ConsensusMessageHandler(...)
/data/harmony/node/node_handler.go:602
github.com/harmony-one/harmony/node.(*Node).messageHandler(0xc00007ac00, 0xc021592b65, 0x198, 0x19b, 0xc02eda99b0, 0x22)
/data/harmony/node/node_handler.go:130 +0x190
created by github.com/harmony-one/harmony/node.(*Node).ReceiveGroupMessage
/data/harmony/node/node_handler.go:72 +0x177
/////////////////////////////
Message handling and p2p are creating a lot of go routines: (e.g. ConsensusMessageHandler)
The text was updated successfully, but these errors were encountered: