Clean Datastore Tables Job

[JVickery-TBS]

feat(jobs): added job to delete datastore tables;

• Added config option clean_datastore_tables.
• Added job to delete datastore tables from unsupported formats.

This adds the possible feature (controlled by config option) to delete datastore tables for unsupported formats. This is useful in the case that a user updates a Resource to no longer be a datastore resource, but the tables remain. So this helps keep the size of the datastore database down a bit

See: ckan#196

[JVickery-TBS]

@ThrawnCA did feedback from ckan#196 here too

[JVickery-TBS]

@ThrawnCA .lower() also removed from the debug logging on resource format

[duttonw]

It does seem like this should be working code, but we all would be more confident if some tests were added.

Reviewing what you are wanting. I was thought experimenting if this logic could live inside the 'ckanext-datastore' since its about 'cleanup' of said data stores. Sadly a quick look through said codebase, it does not cover the hooks required for this nor has the ability to know what formats we allow to ingest in a common bases between datapusher/xloader.

[duttonw]

This might help in getting your discrete tests on this function without needing to also test the xloader 'is file supported' function.

@pytest.fixture
def mock_toolkit_config(request):
    with mock.patch('ckan.plugins.toolkit.config.get') as mock_get:
        mock_get.return_value = request.param
        yield mock_get

@pytest.fixture
def mock_xloader_formats(request):
    with mock.patch('your_module.XLoaderFormats.is_it_an_xloader_format') as mock_is_xloader_format:
        mock_is_xloader_format.return_value = request.param
        yield mock_is_xloader_format


@pytest.mark.parametrize("toolkit_config_value, xloader_formats_value, expected_result", [
    (True, True, True),
    (True, False, False), 
    (False, True, False),
    (False, False, False),
]) <!-- this allows you to test all the combinations with less code, you might also want to pass in the url_type/datastore_active flags also OR you can go with individual discrete tests with nice names-->
def test_should_remove_unsupported_resource_from_datastore(
    mock_toolkit_config, mock_xloader_formats, toolkit_config_value, xloader_formats_value, expected_result
):
    # Setup mock data
    res_dict = {
        'format': 'some_format',
        'url_type': 'upload',
        'datastore_active': True,
        'extras': {'datastore_active': True}
    }

    # Call the function
    result = _should_remove_unsupported_resource_from_datastore(res_dict)

    # Assert the result based on the logic paths covered
    assert result == expected_result
    ```

[JVickery-TBS]

@duttonw @ThrawnCA okay got those automated tests in now. Thanks so much for giving me the code! My python knowledge would have never gotten there hahaha.

I did add in the url_type and datastore_active params

As for the greedy-ness of this functionality, I think it is kind of greedy in general to trash datastore tables if the Resource Format changes. But users can always just re-upload to the datastore if they need to

[JVickery-TBS]

@ThrawnCA @duttonw okay I think the tests should be good now. I had to play around with the fixtures a bit, and seems like we do need the yields in the fixtures if your patching class methods