Add History Archival feature

[davelopez]

Implementation of #14771

Depends on #16075 and #16143

UI

There is a new option on the histories menu to let you "archive" a history (moving it out of your active histories)

When Archiving a history, depending on the server configuration, we are presented with 2 options, one to Keep the storage space and another to free the storage space taken by the history.

Keep storage space (option A)

This option will just mark the history as being Archived and will no longer be shown in the regular histories listing or the "History Selector".
Some characteristics of this kind of archived histories:

• The contents of the history will remain as they are.
• The UI will not allow certain operations on the history (the API will still allow them until a Read Only mode for histories is implemented on the backend as a future feature)
• The history can be restored at any moment, effectively moving it back to your active histories.

Free storage space (option B)

This option will allow the user to package & export the history to a permanent remote source as a backup snapshot and then purge the history and its contents to free up storage space.

After generating the export record the user should acknowledge that the original history will be permanently deleted and then proceed to archive it.

Some characteristics of this kind of archived histories:

• If there is already a recent (up-to-date) export record to a permanent file source you don't need to generate another one.
• These histories are purged. So they are read-only by definition.
• You can always import a new copy out of the associated export record in case you want to restore it, this will import back the backup snapshot that you associated when archiving it.

Archived Histories

Just a simple list where you can explore your archived histories and restore or reimport them as necessary.

---

Backend

The backend implementation builds on top of #14839, this allows to optionally associate an export record at the time of archival so the actual contents of the history can be purged.

Only export records targeting a permanent file source are considered for archival backup.

Changes in the database

• The history table now contains 2 new columns:
  • archived: just a flag to indicate that this history is no longer part of the active histories of the user.
  • archive_export_id: the optional ID of an export record that contains a persisted backup snapshot of the history.

New API routes

• GET /api/histories/archived: returns the list of archived histories of the user. Supports filtering and pagination as other similar endpoints.
• POST /api/histories/{history_id}/archive: marks a history as archived. Optionally it can associate an export record and purge the history.
• PUT /api/histories/{history_id}/archive/restore: will restore the history (unarchive, move it back to your active histories). Purged histories are not restored by default but you can still "force" the unarchival by using the force=true query parameter. It will be then like one of your regular purged histories.

---

How to test the changes?

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[hexylena]

This is fantastic! looks like it addresses both of the main use cases, how exciting!

[jdavcs]

@davelopez so here's what's happening with the migrations error. There are two. The first one (the one we discussed) happens because of the index parameter is ignored (ref: see gray box). A separate statement is needed.

Once that one is fixed, another error is raised, that one is SQLite-specific: handling foreign keys in batch operations mode requires separate statements (here's a detailed discussion).

So, my suggestions are as follows:

1. I think separating this into 2 revisions might be a little cleaner, given the constraints? (that's how I tested, but having everything in 1 file should work too)
2. For the archived column:

def upgrade():
    with transaction():
        add_column(
            table_name,
            sa.Column(column_name, sa.Boolean(), default=False, server_default=sa.false()),
        )
        create_index(index_name, table_name, [column_name])

def downgrade():
    with transaction():
        drop_index(index_name, table_name)
        drop_column(table_name, column_name)

3. For the archive_export_id column:

fk_name = f"fk_{table_name}_{column_name}"

column = sa.Column(
    column_name,
    sa.Integer,
    nullable=True,
    default=None,
)

def upgrade():
    with transaction():
        add_column(table_name, column)
        create_foreign_key(fk_name, table_name, ref_table_name, [column_name], ["id"])

def downgrade():
    with transaction():
        drop_constraint(fk_name, table_name)
        drop_column(table_name, column_name)

(We definitely need an automated naming convention - to ensure that the indexes and constraints automatically created from our model definitions match those created via migrations. It's on my todo list.)

[davelopez]

Thank you again for all the help @jdavcs!

I think separating this into 2 revisions might be a little cleaner, given the constraints? (that's how I tested, but having everything in 1 file should work too)

I decided to keep both in the same revision file, but I'm happy to split them if this is important. The updated commit is 110361d.

Update: looks like I'm still doing something wrong... https://github.com/galaxyproject/galaxy/actions/runs/4957569426/jobs/8869412400?pr=16003

[jdavcs]

Update: looks like I'm still doing something wrong... https://github.com/galaxyproject/galaxy/actions/runs/4957569426/jobs/8869412400?pr=16003

It's not your code - it's related to the naming convention I mentioned in my previous comment. The test starts out when the foreign key constraint had been already added by SQLAlchemy (as per the model definition). It was added with a different name - so when the test tries to run a downgrade, the constraint is not found. (if you were to start on a previous version - as you did in development - make the change and add the revision, then run upgrade - then the constraint would have the name you gave it and all would've been tip-top! However, since the test starts with a blank database (as it should), the constraint name specified in the revision script is never applied.

Changing the name to match what's generated is easy for postgres (history_archive_export_id_fkey). However, as an added bonus, we have to deal with SQLite as well, which does things differently: to quote Alembic's docs, "SQLite, unlike any other database, allows constraints to exist in the database that have no identifying name." [ref]. The correct solution is an automated naming convention - Alembic has a whole section on The Importance of Naming Constraints, which, of course, was "not urgent" at the time... I'll fix this.

[davelopez]

I've fixed all the conflicts and updated de db migration script commit with the latest revision. This should be ready to go if someone has time to review it 🙇‍♂️

[dannon]

This is really great work @davelopez. It looks like it's working well, to me, so let's get it in and just iterate on it if anything pops up during testing!