Accounts are not indexed to DB

[alexqrid]

Tried to run harvester both in docker and outside docker but the harvester didn't index accounts. How can I fix it? @arjanz

[arjanz]

Could you provide some more context, like which chain, docker-compose environment, error logs etc?

[alexqrid]

I tried both chain: kusama using docker-compose defined below:

docker-compose

version: '3.2'

services:

  harvester-api:
    build: .
    image: &app polkascan-harvester
    ports:
      - '8000:8000'
    volumes:
      - '.:/usr/src/app'
    command: ./start.sh
    environment: &env
      - CELERY_BROKER=redis://redis:6379/0
      - CELERY_BACKEND=redis://redis:6379/0
      - PYTHONPATH=/usr/src/app
      - ENVIRONMENT=dev
    depends_on:
      - redis
      - mysql
      - substrate-node

  harvester-worker:
    build: .
    image: *app
    volumes:
      - '.:/usr/src/app'
    command: celery -A app.tasks worker --loglevel=INFO
    environment: *env
    depends_on:
      - redis
      - mysql

  harvester-beat:
    build: .
    image: *app
    volumes:
      - '.:/usr/src/app'
    command: celery -A app.tasks beat --loglevel=INFO  --schedule="data/celerybeat-schedule" --pidfile="data/celerybeat.pid"
    environment: *env
    depends_on:
      - redis

  harvester-monitor:
    build: .
    image: *app
    ports:
      - '5555:5555'
    command:  flower -A app.tasks --port=5555 --broker=redis://redis:6379/0
    depends_on:
      - redis

  redis:
    image: redis:3.2.11

  mysql:
    image: mysql:latest
    volumes:
      - './data/mysql:/var/lib/mysql'
    ports:
      - '33061:3306'
    environment:
      - MYSQL_ROOT_PASSWORD=root
      - MYSQL_DATABASE=polkascan

  substrate-node:
    image: polkasource/substrate-client:polkadot-latest
    ports:
      - '30333:30333'
      - '9933:9933'
      - '9944:9944'
    command: --rpc-port 9933 --rpc-external --pruning=archive --rpc-cors all --chain=polkadot

I have changed the image of the substrate-node as the image defined in docker-compose.yml in the repo doesn't exist anymore (polkasource/substrate-alexander:latest) and it couldn't be retrieved from dockerhub. As you can see I also started the node in the archive mode.

Indexer database state for the kusama chain

Logs of containters

For the chain polkadot I tried to run indexer outside of the docker, i.e., I installed all dependency on the server and changed appropriate network configs of the app (ip address of the mysql,redis, polkadot node). After successfully indexing a few blocks the indexer started to complain about missing block number 775.

Logs from polkadot indexer

[2021-03-10 08:05:14,951: INFO/ForkPoolWorker-4] Task app.tasks.accumulate_block_recursive[308f7dd5-085c-451f-8c22-d3b757f5ea48] succeeded in 0.059940158389508724s: {'result': '0 blocks added', 'lastAddedBlockHash': '0xb5f2894a2f5f309e2436d208186841cfbdb7fd7876c4a8c42c118cb753323e7c', 'sequencerStartedFrom': False}
[2021-03-10 08:05:15,199: INFO/ForkPoolWorker-2] Task app.tasks.start_sequencer[e878e1bf-2ef8-4abc-b460-61a1bc283718] succeeded in 0.3258209628984332s: {'result': 'Block #775 is missing.. stopping check '}
[2021-03-10 08:05:24,827: INFO/MainProcess] Received task: app.tasks.start_harvester[75e025ac-6e29-4acc-98db-359d3c9704c9]  
[2021-03-10 08:05:24,871: INFO/MainProcess] Received task: app.tasks.start_sequencer[729cf19b-b90a-4737-ac5e-0143800e9b78]  
[2021-03-10 08:05:24,881: INFO/ForkPoolWorker-6] Task app.tasks.start_harvester[75e025ac-6e29-4acc-98db-359d3c9704c9] succeeded in 0.05178449023514986s: {'result': 'Harvester job started', 'block_sets': [{'start_block_hash': '0xb5f2894a2f5f309e2436d208186841cfbdb7fd7876c4a8c42c118cb753323e7c', 'end_block_hash': None}], 'sequencer_task_id': '729cf19b-b90a-4737-ac5e-0143800e9b78'}
[2021-03-10 08:05:24,881: INFO/MainProcess] Received task: app.tasks.accumulate_block_recursive[c6d1674a-bc02-4e59-93de-c58d7f438158]  
[2021-03-10 08:05:24,952: WARNING/ForkPoolWorker-2] . Skipped 0xb5f2894a2f5f309e2436d208186841cfbdb7fd7876c4a8c42c118cb753323e7c
[2021-03-10 08:05:24,954: INFO/ForkPoolWorker-2] Task app.tasks.accumulate_block_recursive[c6d1674a-bc02-4e59-93de-c58d7f438158] succeeded in 0.06298000365495682s: {'result': '0 blocks added', 'lastAddedBlockHash': '0xb5f2894a2f5f309e2436d208186841cfbdb7fd7876c4a8c42c118cb753323e7c', 'sequencerStartedFrom': False}
[2021-03-10 08:05:25,336: INFO/ForkPoolWorker-8] Task app.tasks.start_sequencer[729cf19b-b90a-4737-ac5e-0143800e9b78] succeeded in 0.46332085970789194s: {'result': 'Block #775 is missing.. stopping check '}
[2021-03-10 08:05:34,828: INFO/MainProcess] Received task: app.tasks.start_harvester[f4cfe305-868c-4000-aea9-f8bb0f60cf24]  
[2021-03-10 08:05:34,866: INFO/MainProcess] Received task: app.tasks.start_sequencer[f75bb887-eb51-44e5-b725-57506452841f]  
[2021-03-10 08:05:34,872: INFO/MainProcess] Received task: app.tasks.accumulate_block_recursive[f59b16f5-c31e-4848-b3d2-d1f2b36bd1e1]  
[2021-03-10 08:05:34,872: INFO/ForkPoolWorker-4] Task app.tasks.start_harvester[f4cfe305-868c-4000-aea9-f8bb0f60cf24] succeeded in 0.04259246960282326s: {'result': 'Harvester job started', 'block_sets': [{'start_block_hash': '0xb5f2894a2f5f309e2436d208186841cfbdb7fd7876c4a8c42c118cb753323e7c', 'end_block_hash': None}], 'sequencer_task_id': 'f75bb887-eb51-44e5-b725-57506452841f'}

Database state of the polkadot chain

However the main problem is that the blocks, extrinsics, etc. are indexed but accounts are not. As a result running explorer-api can not response on any account info query.

[alexqrid]

@arjanz any suggestions?

[boreyapp]

@alexqrid do you found the solution yet?
I meet similar problem like you too. Hope you can seed me some light...
http://172.104.177.231:8080/node-template/account

[alexqrid]

@boreyapp Unfortunately I could not and tried subscan but also have had some problems with it. Anyway subscan is better than polkascan, IMHO.