Node going down and need a reboot after 2-3 days

[Ducky-001]

The full node has crashed consistently every 2-3 days since i got it. It may be because it runs out of memory after a while. When I start it
it uses around 36-38% memory. After running for 13 hours or so it looks like this:

[Ducky-001]

[Ducky-001]

[MrChrisJ]

Ok thanks for reporting this. I setup Node #3 in the flat yesterday and will leave it running and see if I can recreate the same errors.

During testing I did get some crashing of bitcoin daemon but not more frequently that once per week and that was with the GUI running sometimes so I could tweet the stats. So 2-3 days seems a bit much to me. Let's see if I can reproduce the same and if not then worst comes to the worst I can send you replacement.

Might also be good to tweak raspi-config and the overclocking as well. Also if you plan to run in headless mode you can re-allocate more memory to system.

[Ducky-001]

Thanks!

The node is still running, but memory is getting thight

[MrChrisJ]

Ok and I am assuming you haven't made any changes to the Pi since you received it?

I am getting about half the memory usage after 14 hours of operation:

[MrChrisJ]

Ah ok I just noticed that you have more connections because I did not remap my port forwarding on my router to the new IP address. I have done that now and I dealing outbound connections as well. This I expect will increase the memory load.

In the bitcoin.conf file there is maxconnections= field which we can use to limit the max number on the #fullnode project. I will report back once I have gathered some more data.

[Ducky-001]

Yeah, I added the two lines, that's the only change. I came home now and it was died

[johnp1954]

Mine dies too after a couple of days

[MrChrisJ]

Ok. Update my end. I am on Day 3, more memory usage up to 72% now but no sign of crashing just yet:

Can you guys @johnp1954 & @Ducky-001 restart Bitcoin again with bitcoind -daemon command? Let me know if it throws up any errors.

[ghost]

I tweaked the bitcoin.conf file with the following options ... seeing as tho is only a low power computer

checklevel=1
limitfreerelay=10
maxconnections=24
minrelaytxfee=0.0001

[Emzy]

Using the new Jessie Raspberian with systemd as init-system would be an easy fix.
systemd will automatically restart a service that died.

File: /etc/systemd/system/bitcoind.service

[Unit]
Description=Bitcoins distributed currency daemon
After=network.target

[Service]
User=bitcoin
Group=bitcoin

Type=forking
PIDFile=/var/lib/bitcoind/bitcoind.pid
ExecStart=/usr/bin/bitcoind -daemon -pid=/var/lib/bitcoind/bitcoind.pid \
-conf=/etc/bitcoin/bitcoin.conf -datadir=/var/lib/bitcoind -disablewallet

Restart=always
PrivateTmp=true
TimeoutStopSec=60s
TimeoutStartSec=2s
StartLimitInterval=120s
StartLimitBurst=5

[Install]
WantedBy=multi-user.target

A quick hack, I have running on one small VPS Server, is starting bitcoind with this command:
while true ; do sleep 1; bitcoind ; done

[MrChrisJ]

Hey thanks @Emzy, these fullnodes run on Wheezy, would this still work?

[johnp1954]

Just got home and the fullnode has stopped what command do you want me to type in

John

On 6 Jan 2016, at 17:54, Chris Ellis notifications@github.com wrote:

Hey thanks @Emzy, these fullnodes run on Wheezy, would this still work?

—
Reply to this email directly or view it on GitHub.

[MrChrisJ]

Ok after 4 days my fullnode is still running fine. Not sure why I am not able to reproduce the same errors that you guys are getting but I will leave it on in case.

In the mean time here is a new bitcoin.conf file with changes made thanks to @MyVeryOwn and @johnp1954
https://gist.github.com/MrChrisJ/013bc7e14902b1c21208

[MrChrisJ]

Hey @johnp1954,

The command:
bitcoind -daemon

[Emzy]

The "hack" works. But there is no systemd on Wheezy, so the clean way is only available on Jessie.
Maybe, it will be the best to use the newer Jessie anyway.

[johnp1954]

From: Chris Ellis
Sent: Wednesday, January 06, 2016 5:58 PM
To: MrChrisJ/fullnode
Cc: johnp1954
Subject: Re: [fullnode] Node going down and need a reboot after 2-3 days (#4)

Hey @johnp1954,

The command:
bitcoind -daemon

—
Reply to this email directly or view it on GitHub.

[johnp1954]

[Ducky-001]

Ill try the command next time it crashes. Until now I have just pulled the plug to restart it.

[ghost]

checklevel=1 relates to the verification of blocks ...

I used the lowest setting to get the blockchain downloaded/sync'd as quick as i could and i was also not too concerned with historic transactions .. just getting the node up ... I am now going to load my machine with the default level again and see what it performs like

[Emzy]

add file /usr/local/bin/bitcoind-restarter.sh

#!/bin/bash
while true ; do sleep 15; bitcoind ; done

chmod +x /usr/local/bin/bitcoind-restarter.sh
change the line in /etc/rc.local

...
su pi -c '/usr/local/bin/bitcoind-restarter.sh &'
...

For all of this to work right it is impotent that the command bitcoind doesn't go in the background!
This is achieved by putting 'daemon=0' in the config file .bitcoin/bitcoin.conf

All the bash script is doing: wait 15 seconds and then start bitcoind (in foreground), if it dies the loop will start again with waiting 15 seconds and starting bitcoind.

[Merlinsmatrix]

Hi Chris, #19 node

[Ducky-001]

My node is running "better" since the last reboot. I have not changed anything but it's still at 75% after 4 days with 14 connections. So the memory use seems to vary a lot. Thanks for all the suggestions made. I will try to add the hack nest time it goes down. Btw, can this be of any help:

Reducing bitcoind memory usage https://gist.github.com/laanwj/efe29c7661ce9b6620a7

[MrChrisJ]

Hey @Merlinsmatrix can you setup a new issue. You are just doing some commands in the wrong order.

[MrChrisJ]

Hey @Emzy thanks for that. I have now put the script on Node 3 and will see how it goes. Entering 7 days without crashing so far.

[MrChrisJ]

@Ducky-001 it looks like on that list you found many of those improvements don't apply now we are on 0.11.2. So along with @MyVeryOwn's suggestions which I put in place here: https://gist.github.com/MrChrisJ/70e063f48c4370096c1c I think we have about all the efficiency gains we are going to get.

The only thing that might improve is the dbcache field which we could add to the conf file to see if it makes a difference. Try it out and see.

[MrChrisJ]

@Emzy I have tested your script and it works well at 30 second intervals. I have figured the rc.local file thus:
https://gist.github.com/MrChrisJ/5d828760e0eb14e5d87d

This gives the user a choice as I have commented out the alternative option with just the start on boot.

Your script does not seem to appear to interfere with QT's operation but only gives the user 30 seconds to start QT:

Let me know your thoughts everyone. I will make this the next default image for the next batch of #Fullnodes

[ghost]

I have commented out the checklevel=1 line (as it relates to verification of blocks) and my fullnoode has been running for days without an issue - am going to leave it alone to reach double digit day numbers of uptime.

[oktoshi]

You can set up a crontab to clean up the cached ram every 24/48hrs, this prevent the nodes from crashing.

How to:
(If dont have cron)
$ sudo apt-get install cron
then:
$ sudo crontab -e (this will open cron for edition)
and add this line at the end:
0 4 * * * root sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

and click Control + X, Y and Enter to save,
This will install the new cron job and will run on automatic every day at 4:00am and will clean the Ram,
preventing the node from crashing.

You could also increase the swap size from 100 to 1024 to help bring a bit more stability and speed to the system.

[ghost]

my node has been up for 8 days now and shows over 2000 mins of cpu time by bitcoind.. no reboot scripts just bitcoin.conf tweaks as posted above

[Ducky-001]

I have upgraded to the new bitcoin.conf, and the node seems to be running stable now. Much better than it was for sure:

[ralphtheninja]

How to:
(If dont have cron)
$ sudo apt-get install cron
then:
$ sudo crontab -e (this will open cron for edition)
and add this line at the end:
0 4 * * * root sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

and click Control + X, Y and Enter to save,

This could also be abstracted into a script (so you don't need to do all the manual work).

cc @MrChrisJ