VON-nodes failures

[Durai46]

Hi, Running the network using ./mange scripts and seeing an intermittent failures on 2 or 1 ports when VON-nodes startup.

Error:

start_indy_node Node2 0.0.0.0 9703 0.0.0.0 9704
2022-10-10 07:16:08,950|INFO|notifier_plugin_manager.py|Found notifier plugins: []
2022-10-10 07:16:09,027|INFO|notifier_plugin_manager.py|Found notifier plugins: []
2022-10-10 07:16:09,079|INFO|looper.py|Starting up indy-node
2022-10-10 07:16:09,105|INFO|ledger.py|Starting ledger...
2022-10-10 07:16:09,112|INFO|ledger.py|Recovering tree from transaction log
2022-10-10 07:16:09,126|INFO|ledger.py|Recovered tree in 0.01454660099989269 seconds
2022-10-10 07:16:09,143|INFO|ledger.py|Starting ledger...
2022-10-10 07:16:09,149|INFO|ledger.py|Recovering tree from hash store of size 4
2022-10-10 07:16:09,149|INFO|ledger.py|Recovered tree in 0.00036303600018072757 seconds
2022-10-10 07:16:09,166|INFO|ledger.py|Starting ledger...
2022-10-10 07:16:09,176|INFO|ledger.py|Recovering tree from hash store of size 39
2022-10-10 07:16:09,177|INFO|ledger.py|Recovered tree in 0.0009142939998127986 seconds
2022-10-10 07:16:09,196|INFO|ledger.py|Starting ledger...
2022-10-10 07:16:09,209|INFO|ledger.py|Recovering tree from hash store of size 2524
2022-10-10 07:16:09,217|INFO|ledger.py|Recovered tree in 0.008502250000219647 seconds
2022-10-10 07:16:09,268|NOTIFICATION|node_bootstrap.py|BLS: BLS Signatures will be used for Node Node2
2022-10-10 07:16:09,269|INFO|pool_manager.py|Node2 sets node Node1 (Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv) order to 5
2022-10-10 07:16:09,269|INFO|pool_manager.py|Node2 sets node Node2 (8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb) order to 5
2022-10-10 07:16:09,269|INFO|pool_manager.py|Node2 sets node Node3 (DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya) order to 5
2022-10-10 07:16:09,269|INFO|pool_manager.py|Node2 sets node Node4 (4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA) order to 5
2022-10-10 07:16:09,275|INFO|notifier_plugin_manager.py|Found notifier plugins: []
2022-10-10 07:16:09,276|INFO|notifier_plugin_manager.py|Found notifier plugins: []
2022-10-10 07:16:09,277|NOTIFICATION|looper.py|Looper shutting down now...
2022-10-10 07:16:09,288|NOTIFICATION|looper.py|Looper shut down in 0.010 seconds.
start_indy_node Node2 0.0.0.0 9703 0.0.0.0 9704
2022-10-10 07:16:52,483|INFO|notifier_plugin_manager.py|Found notifier plugins: []
2022-10-10 07:16:52,560|INFO|notifier_plugin_manager.py|Found notifier plugins: []
2022-10-10 07:16:52,608|INFO|looper.py|Starting up indy-node
2022-10-10 07:16:52,631|INFO|ledger.py|Starting ledger...
2022-10-10 07:16:52,637|INFO|ledger.py|Recovering tree from transaction log
2022-10-10 07:16:52,650|INFO|ledger.py|Recovered tree in 0.012840225000218197 seconds
2022-10-10 07:16:52,664|INFO|ledger.py|Starting ledger...
2022-10-10 07:16:52,669|INFO|ledger.py|Recovering tree from hash store of size 4
2022-10-10 07:16:52,669|INFO|ledger.py|Recovered tree in 0.0003230899997106462 seconds
2022-10-10 07:16:52,685|INFO|ledger.py|Starting ledger...
2022-10-10 07:16:52,690|INFO|ledger.py|Recovering tree from hash store of size 39
2022-10-10 07:16:52,691|INFO|ledger.py|Recovered tree in 0.0005423120001069037 seconds
2022-10-10 07:16:52,686|INFO|ledger.py|Starting ledger...
2022-10-10 07:16:52,693|INFO|ledger.py|Recovering tree from hash store of size 2524
2022-10-10 07:16:52,703|INFO|ledger.py|Recovered tree in 0.009407756000200607 seconds
2022-10-10 07:16:52,745|NOTIFICATION|node_bootstrap.py|BLS: BLS Signatures will be used for Node Node2
2022-10-10 07:16:52,745|INFO|pool_manager.py|Node2 sets node Node1 (Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv) order to 5
2022-10-10 07:16:52,746|INFO|pool_manager.py|Node2 sets node Node2 (8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb) order to 5
2022-10-10 07:16:52,746|INFO|pool_manager.py|Node2 sets node Node3 (DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya) order to 5
2022-10-10 07:16:52,746|INFO|pool_manager.py|Node2 sets node Node4 (4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA) order to 5
2022-10-10 07:16:52,752|INFO|notifier_plugin_manager.py|Found notifier plugins: []
2022-10-10 07:16:52,753|INFO|notifier_plugin_manager.py|Found notifier plugins: []
Traceback (most recent call last):
  File "/home/indy/.pyenv/versions/3.6.13/bin/start_indy_node", line 19, in <module>
    client_ip=sys.argv[4], client_port=int(sys.argv[5]))
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/utils/node_runner.py", line 50, in run_node
    ha=node_ha, cliha=client_ha)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/node.py", line 87, in __init__
    bootstrap_cls=bootstrap_cls)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/plenum/server/node.py", line 188, in __init__
    self.bootstrapper = self._bootstrap_node(bootstrap_cls, storage)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/plenum/server/node.py", line 3164, in _bootstrap_node
    bootstrapper.init(domain_storage=storage)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/plenum/server/ledgers_bootstrap.py", line 80, in init
    self._init_common_managers()
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/node_bootstrap.py", line 243, in _init_common_managers
    self.node.restarter = self.init_restarter()
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/node_bootstrap.py", line 238, in init_restarter
    self.node.config)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/node_maintainer.py", line 39, in __init__
    self._defaultLog(dataDir, config)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/restarter.py", line 26, in _defaultLog
    return RestartLog(file_path=log)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/restart_log.py", line 20, in __init__
    delimiter='\t'
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/action_log.py", line 147, in __init__
    self._load()
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/action_log.py", line 177, in _load
    types=self._event_types
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/action_log.py", line 29, in unpack
    return cls(*next(reader), *args, **kwargs)
_csv.Error: line contains NULL byte
Traceback (most recent call last):
  File "/home/indy/.pyenv/versions/3.6.13/bin/start_indy_node", line 19, in <module>
    client_ip=sys.argv[4], client_port=int(sys.argv[5]))
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/utils/node_runner.py", line 50, in run_node
    ha=node_ha, cliha=client_ha)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/node.py", line 87, in __init__
    bootstrap_cls=bootstrap_cls)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/plenum/server/node.py", line 188, in __init__
    self.bootstrapper = self._bootstrap_node(bootstrap_cls, storage)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/plenum/server/node.py", line 3164, in _bootstrap_node
    bootstrapper.init(domain_storage=storage)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/plenum/server/ledgers_bootstrap.py", line 80, in init
    self._init_common_managers()
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/node_bootstrap.py", line 243, in _init_common_managers
    self.node.restarter = self.init_restarter()
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/node_bootstrap.py", line 238, in init_restarter
    self.node.config)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/node_maintainer.py", line 39, in __init__
    self._defaultLog(dataDir, config)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/restarter.py", line 26, in _defaultLog
    return RestartLog(file_path=log)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/restart_log.py", line 20, in __init__
    delimiter='\t'
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/action_log.py", line 147, in __init__
    self._load()
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/action_log.py", line 177, in _load
    types=self._event_types
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/indy_node/server/action_log.py", line 29, in unpack
    return cls(*next(reader), *args, **kwargs)
_csv.Error: line contains NULL byte
2022-10-10 07:16:52,754|NOTIFICATION|looper.py|Looper shutting down now...
2022-10-10 07:16:52,764|NOTIFICATION|looper.py|Looper shut down in 0.011 seconds.