Nextcloud scan crashes

[fallingcats]

I tried scanning a nextcloud account for photos and it seems to not do anything. When checking the log files I found ownphotos has most likely crashed:

14:38:18 default: nextcloud.directory_watcher.scan_photos(<User: bernhard>) (3e0be643-9798-494f-8dcf-b3a4abeb30c0)
14:38:18 Sent heartbeat to prevent worker timeout. Next one should arrive within 420 seconds.
14:38:18 Sent heartbeat to prevent worker timeout. Next one should arrive within 35 seconds.
14:38:18 django.db.utils.IntegrityError: duplicate key value violates unique constraint "api_longrunningjob_job_id_key"
DETAIL:  Key (job_id)=(3e0be643-9798-494f-8dcf-b3a4abeb30c0) already exists.

Traceback (most recent call last):
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
psycopg2.IntegrityError: duplicate key value violates unique constraint "api_longrunningjob_job_id_key"
DETAIL:  Key (job_id)=(3e0be643-9798-494f-8dcf-b3a4abeb30c0) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/miniconda/lib/python3.6/site-packages/rq/worker.py", line 793, in perform_job
    rv = job.perform()
  File "/miniconda/lib/python3.6/site-packages/rq/job.py", line 599, in perform
    self._result = self._execute()
  File "/miniconda/lib/python3.6/site-packages/rq/job.py", line 605, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/code/nextcloud/directory_watcher.py", line 50, in scan_photos
    job_type=LongRunningJob.JOB_SCAN_PHOTOS)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/query.py", line 413, in create
    obj.save(force_insert=True, using=self.db)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/base.py", line 718, in save
    force_update=force_update, update_fields=update_fields)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/base.py", line 748, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/base.py", line 831, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/base.py", line 869, in _do_insert
    using=using, raw=raw)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/query.py", line 1136, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1289, in execute_sql
    cursor.execute(sql, params)
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 100, in execute
    return super().execute(sql, params)
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/miniconda/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "api_longrunningjob_job_id_key"
DETAIL:  Key (job_id)=(3e0be643-9798-494f-8dcf-b3a4abeb30c0) already exists.

Traceback (most recent call last):
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
psycopg2.IntegrityError: duplicate key value violates unique constraint "api_longrunningjob_job_id_key"
DETAIL:  Key (job_id)=(3e0be643-9798-494f-8dcf-b3a4abeb30c0) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/miniconda/lib/python3.6/site-packages/rq/worker.py", line 793, in perform_job
    rv = job.perform()
  File "/miniconda/lib/python3.6/site-packages/rq/job.py", line 599, in perform
    self._result = self._execute()
  File "/miniconda/lib/python3.6/site-packages/rq/job.py", line 605, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/code/nextcloud/directory_watcher.py", line 50, in scan_photos
    job_type=LongRunningJob.JOB_SCAN_PHOTOS)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/query.py", line 413, in create
    obj.save(force_insert=True, using=self.db)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/base.py", line 718, in save
    force_update=force_update, update_fields=update_fields)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/base.py", line 748, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/base.py", line 831, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/base.py", line 869, in _do_insert
    using=using, raw=raw)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/query.py", line 1136, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1289, in execute_sql
    cursor.execute(sql, params)
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 100, in execute
    return super().execute(sql, params)
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/miniconda/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "api_longrunningjob_job_id_key"
DETAIL:  Key (job_id)=(3e0be643-9798-494f-8dcf-b3a4abeb30c0) already exists.

14:38:18 Invoking exception handler <bound method Worker.move_to_failed_queue of <rq.worker.Worker object at 0x7fa53d214c18>>
14:38:18 Moving job to 'failed' queue

I would like to add that it I think it would have worked the first time I tried setting up ownphotos but because I am doing this in a vm I had allocated to little disk space it ran out of free space.
Then I deleted all docker containers and volumes by running

docker stop ownphotos-backend ownphotos-db ownphotos-frontend ownphotos-proxy ownphotos-redis
docker rm ownphotos-backend ownphotos-db ownphotos-frontend ownphotos-proxy ownphotos-redis
docker volume prune

Did some data from the previous install survive or do you really get a clean install when following up with docker-compose up -d?
I am asking because as I said the nextcloud scan looked good until I ran out of disk space but I just can't find what would be different now.

[rechner]

I'm seeing the same database integrity error with my install after a number of days between initial import and trying to scan NextCloud again, though I notice that all backend job queues die with the same error, even after recreating the containers and clearing the old jobs from the api_longrunningjob table.

Edit: Seems to also be related to this - #48 (comment)