Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] - Node Process Killed When Calculating Next Epoch's Leadership Schedule #5861

Open
Qifeng-Wu99 opened this issue May 28, 2024 · 4 comments
Open

[BUG] - Node Process Killed When Calculating Next Epoch's Leadership Schedule #5861

Qifeng-Wu99 opened this issue May 28, 2024 · 4 comments
Labels
Stale type: bug Something is not working

Comments

@Qifeng-Wu99
Copy link

Qifeng-Wu99 commented May 28, 2024 •
edited
Loading

Internal/External
External otherwise.

Area
Other Any other topic (Delegation, Ranking, ...).

Summary
Node process killed when calculating next epoch's leadership schedule

Steps to reproduce
Steps to reproduce the behavior:

  1. Start a node with
/usr/local/bin/cardano-node run +RTS -N -A16m -qg -qb -RTS --topology ${TOPOLOGY} --database-path ${DB_PATH} --socket-path ${SOCKET_PATH} --host-addr ${HOSTADDR} --port ${PORT} --config ${CONFIG} --shelley-kes-key ${KES} --shelley-vrf-key ${VRF} --shelley-operational-certificate ${CERT}
  1. Calculate the next epoch's leadership schedule with
cardano-cli query leadership-schedule \
--mainnet \
--genesis $NODE_HOME/shelley-genesis.json \
--stake-pool-id $(cat $NODE_HOME/stakepoolid.txt) \
--vrf-signing-key-file $NODE_HOME/vrf.skey \
--next
--socket-path db/socket
  1. The node process is killed a few minutes after execute step 2, and the node is restarted automatically
  2. The output of step 2 pop out during the restarting of the node process and is empty:
SlotNo                          UTC Time
-------------------------------------------------------------
  1. The Restarting of the node takes time longer than usual

Expected behavior
The output should looks like below

SlotNo                          UTC Time
-------------------------------------------------------------
     4073                   2021-12-29 17:26:54.998001755 UTC
     4126                   2021-12-29 17:27:00.298001755 UTC
     4206                   2021-12-29 17:27:08.298001755 UTC

System info (please complete the following information):

  • OS Name: Ubuntu

  • OS Version: 20.04

  • Node version:
    cardano-node 8.9.2 - linux-x86_64 - ghc-8.10
    git rev 424983fa186786397f5a99539f51710abf62c37b

  • CLI version:
    cardano-cli 8.20.3.0 - linux-x86_64 - ghc-8.10
    git rev 424983fa186786397f5a99539f51710abf62c37b

  • CPU: 8 core / 2.0 GHz

  • RAM: 32 GiB

  • Disk: 400 GiB

Screenshots and attachments
Part of the log related this issue is shown below, before and after the node process is killed:

May 28 13:31:36 ada-node cardano-node[1118012]: [ada-node:cardano.node.ChainDB:Notice:14260] [2024-05-28 13:31:36.23 UTC] Chain extended, new tip: 5a5299bed0f203b79e62eb5a23ac1a9d4b3a249e69bfd90cc40b6dea276ab027 at slot 1>
May 28 13:31:37 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:37.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:37 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:37.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:38 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:38.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:38 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:38.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:39 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:39.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:39 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:39.04 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:40 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:40.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:40 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:40.04 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:41 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:41.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:41 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:41.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:42 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:42.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:42 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:42.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:43 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:43.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:43 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:43.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:44 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:44.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:44 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:44.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:45 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:45.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:45 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:45.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:46 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:46.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:46 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:46.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:47 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:47.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:47 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:47.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:48 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:48.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:48 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:48.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:49 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:49.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:49 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:49.06 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:50 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:50.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:50 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:50.08 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:51 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:51.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:51 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:51.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:51 ada-node cardano-node[1118012]: [ada-node:cardano.node.ChainDB:Notice:14260] [2024-05-28 13:31:51.39 UTC] Chain extended, new tip: 22c923a0aab9b9656f877a87ad47d3635c31aeb30a3d2f724ad3afd7b47a3dd3 at slot 1>
May 28 13:31:52 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:52.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:52 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:52.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:53 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:53.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:53 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:53.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:54 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:54.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:54 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:54.06 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:55 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:55.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:55 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:55.06 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:56 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:56.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:56 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:56.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:57 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:57.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:57 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:57.04 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:58 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:58.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:58 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:58.04 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:59 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:59.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:59 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:59.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:32:01 ada-node cardano-node[1117999]: /home/ada/cardano-my-node/startCardanoNode.sh: line 11: 1118012 Killed                  /usr/local/bin/cardano-node run +RTS -N -A16m -qg -qb -RTS --topology ${TOPOLOGY} --d>
May 28 13:32:07 ada-node cardano-node[2985725]: Node configuration: NodeConfiguration {ncSocketConfig = SocketConfig {ncNodeIPv4Addr = Last {getLast = Just 0.0.0.0}, ncNodeIPv6Addr = Last {getLast = Nothing}, ncNodePortNu>
May 28 13:32:07 ada-node cardano-node[2985725]: Listening on http://127.0.0.1:12798
May 28 13:32:07 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.protocol:Notice:11] [2024-05-28 13:32:07.54 UTC] Byron; Shelley
May 28 13:32:07 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.version:Notice:11] [2024-05-28 13:32:07.54 UTC] 8.9.1
May 28 13:32:07 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.commit:Notice:11] [2024-05-28 13:32:07.54 UTC] da945ea983d4722a9ffe54250edba9a193a57cf0
May 28 13:32:07 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.nodeStartTime:Notice:11] [2024-05-28 13:32:07.54 UTC] 2024-05-28 13:32:07.541878989 UTC
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.systemStartTime:Notice:11] [2024-05-28 13:32:07.54 UTC] 2017-09-23 21:44:51 UTC
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotLengthByron:Notice:11] [2024-05-28 13:32:07.54 UTC] 20s
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.epochLengthByron:Notice:11] [2024-05-28 13:32:07.54 UTC] 21600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotLengthShelley:Notice:11] [2024-05-28 13:32:07.54 UTC] 1s
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.epochLengthShelley:Notice:11] [2024-05-28 13:32:07.54 UTC] 432000
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotsPerKESPeriodShelley:Notice:11] [2024-05-28 13:32:07.54 UTC] 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotLengthAllegra:Notice:11] [2024-05-28 13:32:07.54 UTC] 1s
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.epochLengthAllegra:Notice:11] [2024-05-28 13:32:07.54 UTC] 432000
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotsPerKESPeriodAllegra:Notice:11] [2024-05-28 13:32:07.54 UTC] 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotLengthMary:Notice:11] [2024-05-28 13:32:07.54 UTC] 1s
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.epochLengthMary:Notice:11] [2024-05-28 13:32:07.54 UTC] 432000
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotsPerKESPeriodMary:Notice:11] [2024-05-28 13:32:07.54 UTC] 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotLengthAlonzo:Notice:11] [2024-05-28 13:32:07.54 UTC] 1s
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.epochLengthAlonzo:Notice:11] [2024-05-28 13:32:07.54 UTC] 432000
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotsPerKESPeriodAlonzo:Notice:11] [2024-05-28 13:32:07.54 UTC] 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotLengthBabbage:Notice:11] [2024-05-28 13:32:07.54 UTC] 1s
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.epochLengthBabbage:Notice:11] [2024-05-28 13:32:07.54 UTC] 432000
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotsPerKESPeriodBabbage:Notice:11] [2024-05-28 13:32:07.54 UTC] 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotLengthConway:Notice:11] [2024-05-28 13:32:07.54 UTC] 1s
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.epochLengthConway:Notice:11] [2024-05-28 13:32:07.54 UTC] 432000
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotsPerKESPeriodConway:Notice:11] [2024-05-28 13:32:07.54 UTC] 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.22 UTC] Config path /home/ada/cardano-my-node/config.json, Network magic NetworkMagic {unNetworkMagic = 76482407>
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.22 UTC] Era Byron, Slot length 20s, Epoch length 21600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] Era Shelley, Slot length 1s, Epoch length 432000, Slots per KESPeriod 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] Era Allegra, Slot length 1s, Epoch length 432000, Slots per KESPeriod 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] Era Mary, Slot length 1s, Epoch length 432000, Slots per KESPeriod 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] Era Alonzo, Slot length 1s, Epoch length 432000, Slots per KESPeriod 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] Era Babbage, Slot length 1s, Epoch length 432000, Slots per KESPeriod 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] Era Conway, Slot length 1s, Epoch length 432000, Slots per KESPeriod 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Warning:11] [2024-05-28 13:32:08.23 UTC] You are using an early release of peer-to-peer capabilities, please report any issues.
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] startup time: 1716903129
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC]
May 28 13:32:08 ada-node cardano-node[2985725]: node addresses:          0.0.0.0:6000
May 28 13:32:08 ada-node cardano-node[2985725]: local socket:            /home/ada/cardano-my-node/db/socket
May 28 13:32:08 ada-node cardano-node[2985725]: node-to-node versions:   [7,8,9,10,11,12,13]
May 28 13:32:08 ada-node cardano-node[2985725]: node-to-client versions: [9,10,11,12,13,14,15]
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] initiator and responder diffusion mode
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC]
May 28 13:32:08 ada-node cardano-node[2985725]: Local Root Groups:
May 28 13:32:08 ada-node cardano-node[2985725]:   (HotValency {getHotValency = 1},WarmValency {getWarmValency = 1},[(RelayAccessAddress "159.203.146.191" 6000,(DoNotAdvertisePeer,IsNotTrustable))])
May 28 13:32:08 ada-node cardano-node[2985725]: Public Roots:
May 28 13:32:08 ada-node cardano-node[2985725]:   
May 28 13:32:08 ada-node cardano-node[2985725]: Don't use ledger to get root peers.
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:08.23 UTC] Started opening Chain DB
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:08.23 UTC] Started opening Immutable DB
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:08.55 UTC] Validating chunk no. 0 out of 5800. Progress: 0.00%
May 28 13:32:09 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:09.26 UTC] Validated chunk no. 0 out of 5800. Progress: 0.02%
May 28 13:32:09 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:09.26 UTC] Validating chunk no. 1 out of 5800. Progress: 0.02%
May 28 13:32:10 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:10.22 UTC] Validated chunk no. 1 out of 5800. Progress: 0.03%
May 28 13:32:10 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:10.22 UTC] Validating chunk no. 2 out of 5800. Progress: 0.03%
May 28 13:32:11 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:11.17 UTC] Validated chunk no. 2 out of 5800. Progress: 0.05%
May 28 13:32:11 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:11.17 UTC] Validating chunk no. 3 out of 5800. Progress: 0.05%
May 28 13:32:12 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:12.18 UTC] Validated chunk no. 3 out of 5800. Progress: 0.07%
May 28 13:32:12 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:12.18 UTC] Validating chunk no. 4 out of 5800. Progress: 0.07%

Additional context
I noticed this issue recently, the epoch leader calculation was fine in April this year.
@Qifeng-Wu99 Qifeng-Wu99 added the type: bug Something is not working label May 28, 2024
@rdlrt
Copy link

rdlrt commented May 28, 2024

You likely ran out of memory, on a typical system you can check dmesg -T | grep [c]ardano-node

@Qifeng-Wu99
Copy link
Author

You likely ran out of memory, on a typical system you can check dmesg -T | grep [c]ardano-node

Thanks for the comment, the command you suggested gives the following output, which indicates that the node process was killed due to OOM:

[Tue May 28 18:03:19 2024] [2985725]  1000 2985725 268450990  4969040 39989248        0             0 cardano-node
[Tue May 28 18:03:19 2024] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/cardano-node.service,task=cardano-node,pid=2985725,uid=1000
[Tue May 28 18:03:19 2024] Out of memory: Killed process 2985725 (cardano-node) total-vm:1073803960kB, anon-rss:19876160kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:39052kB oom_score_adj:0

However, this is unexpected since the server I am using has RAM larger than suggested by Cardano official docs, and the server is purely used for Cardano node operating.

Would you suggest any solutions other than upgrade RAM space?

@syndeton
Copy link

If you're using -xn for any RTS options for your cardano-node run command, that might be the issue. I have to remove that option before running leaderlogs or node will crash with OOM.

@github-actions GitHub Actions
Copy link

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

@github-actions github-actions bot added the Stale label Jun 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Stale type: bug Something is not working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rdlrt @syndeton @Qifeng-Wu99