From e351ef25c70b5d958c46bacdcb7583fef288d80d Mon Sep 17 00:00:00 2001 From: Gesof Date: Wed, 24 Jun 2020 15:55:04 +0300 Subject: [PATCH] Updated cluster-setup.rst Added the options required to running the storage workers so they won't show warnings like: [db-worker] Can't identify DB worker type (no-scoring False, no-batches True, no-incoming False) --- docs/source/topics/cluster-setup.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/source/topics/cluster-setup.rst b/docs/source/topics/cluster-setup.rst index 281c95690..c708126c8 100644 --- a/docs/source/topics/cluster-setup.rst +++ b/docs/source/topics/cluster-setup.rst @@ -127,11 +127,11 @@ First, let's start storage worker: :: # start DB worker only for batch generation # use single instance for every 10 partitions - $ python -m frontera.worker.db --config [db worker config module] --no-incoming --partitions 0 1 + $ python -m frontera.worker.db --config [db worker config module] --no-scoring --no-incoming --partitions 0 1 # Optionally, start next one dedicated to spider log processing. - $ python -m frontera.worker.db --no-batches --config [db worker config module] + $ python -m frontera.worker.db --no-batches --no-incoming --config [db worker config module] Next, let's start strategy workers, one process per spider log partition: :: @@ -159,3 +159,4 @@ You should end up with N spider processes running. Also :setting:`SPIDER_PARTITI You're done, crawler should start crawling. Any component can be restarted any time, without major data loss. However, for pausing its enough to stop batch gen only. +