Deployment failed due to disk space error when merging into develop

[PouyaMohseni]

OSError: [Errno 28] No space left on device is received when a branch is merged to develop.

[dchiller]

Is this an error in the GitHub action that deploys to staging? If so, we probably need to clear the docker build cache there.

[yinanazhou]

I added a cron job to clean docker build cache before. It's probably something else. I'll check later.

[dchiller]

I just did a few things to clean up disk space.

1. Prune docker build cache (already done by cron job, but since I was in there I just did it manually): 1.3GB
2. Pruned docker dangling images: 1.2GB
3. There were a bunch of anonymous volumes that I also pruned: 8GB

Because of 3, we will have to re-index data.

[dchiller]

I just re-ran the failed job, and deployment was successful. Re-indexing data now.

[yinanazhou]

Oh I wonder what those volumes are. They are not cleaned by the cron job?

[yinanazhou]

I'm still getting Server Error 500 on staging. Tried to run migrate, but got this error:

Traceback (most recent call last):
  File "/virtual-instrument-museum/vim-app/manage.py", line 22, in <module>
    main()
  File "/virtual-instrument-museum/vim-app/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/migrations/operations/fields.py", line 303, in database_forwards
    schema_editor.alter_field(
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/backends/base/schema.py", line 830, in alter_field
    self._alter_field(
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/backends/postgresql/schema.py", line 287, in _alter_field
    super()._alter_field(
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/backends/base/schema.py", line 982, in _alter_field
    self.execute(
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/backends/postgresql/schema.py", line 48, in execute
    return super().execute(sql, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/backends/base/schema.py", line 201, in execute
    cursor.execute(sql, params)
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 87, in _execute
    return self.cursor.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/virtual-instrument-museum/.venv/lib/python3.11/site-packages/psycopg/cursor.py", line 737, in execute
    raise ex.with_traceback(None)
django.db.utils.ProgrammingError: column "status" does not exist

Any recent model changes could lead to this?

[PouyaMohseni]

status was removed in this migration: 0008_rename_status_instrumentname_verification_status_and_more

migrations.RenameField(
            model_name="instrumentname",
            old_name="status",
            new_name="verification_status",
        ),

Not very recent, it was created in 2025-08-07.

[yinanazhou]

This is because of migration history mismatch:

• Database recorded 0008_remove_instrumentname_is_alias_and_more as applied
• Actual migration file is 0008_rename_status_instrumentname_verification_status_and_more.py
• Database was partially migrated: verification_status existed, but status did not (causing the rename to fail), and the unique constraint was missing

Fixed by correcting the migration record and adding the missing constraint. Closing this as solved.