Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/ckan 2.10 python 3 #21

Draft
wants to merge 26 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
0fb5c28
fix: call auth-function before resource_show
stefina Aug 19, 2023
67ce430
feat: Use IClick interface instead of CkanCommand
bellisk Aug 21, 2023
87248ed
style: Remove repeated references to dryrun option
bellisk Aug 21, 2023
c0420cd
fix: Fix typo
bellisk Aug 21, 2023
f52c191
fix: Remove hardcoded debug line
bellisk Aug 21, 2023
a412cc5
feat: Set command names to match old ones
bellisk Aug 21, 2023
ba94491
docs: Update README
bellisk Aug 21, 2023
099f191
fix: Configure command options as they were in old code
bellisk Aug 21, 2023
6269b51
feat: Use click.echo for command output
bellisk Aug 22, 2023
961dc04
feat: Add black and isort as dev requirements
bellisk Aug 22, 2023
f435bb7
style: Run black and isort
bellisk Aug 22, 2023
c364fb7
fix: Fix references to imported error
bellisk Aug 22, 2023
8da33d2
style: Rename ambiguous variable 'l' (PEP 8 E741)
bellisk Aug 22, 2023
cbf6a20
feat: Replace Travis with GitHub Actions
bellisk Aug 22, 2023
cb65d4b
fix: Fix path in test.yml
bellisk Aug 22, 2023
84bbebe
feat: Remove old commands.py that somehow got added back in
bellisk Aug 22, 2023
19c2a33
feat: Don't use a bare 'except' clause
bellisk Aug 22, 2023
acf99bb
fix: Remove unused variable + function call
bellisk Aug 22, 2023
b7c3809
update: Update flake8 version
bellisk Aug 22, 2023
c97d45c
fix: Remove incorrect output statement
bellisk Aug 23, 2023
1ffba5a
chore: Remove unused requirement
bellisk Aug 23, 2023
819c211
chore: Clean up references to shacl
bellisk Aug 23, 2023
7753346
Merge branch 'feat/ckan-2.10-python-3' into ckan-210-resolve-merge-co…
stefina Sep 5, 2023
4c50139
Merge pull request #26 from opendata-swiss/ckan-210-resolve-merge-con…
stefina Sep 5, 2023
f3052f2
fix: define resource_path
stefina Sep 5, 2023
d036bb1
feat: Update test.ini to work with docker container
bellisk Feb 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
name: Tests
on: [push, pull_request]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v3
with:
python-version: '3.10'
- name: Install requirements
run: pip install -r $GITHUB_WORKSPACE/dev-requirements.txt
- name: Check syntax
run: |
isort --diff --check ckanext/
black --diff --check ckanext/
flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics --exclude ckan
19 changes: 0 additions & 19 deletions .travis.yml

This file was deleted.

33 changes: 19 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,36 +4,41 @@ ckanext-ogdchcommands
CKAN extension for DCAT-AP Switzerland. This extension provides two CKAN plugins:

- `ogdch_cmd`: providing commands to run in the background
- `ogdch_admin`: admin tools mainly for handeling of solr
- `ogdch_admin`: admin tools mainly for handling of solr

## Requirements

- CKAN 2.8+
- CKAN 2.10+
- ckanext-switzerland
- ckanext-harvest
- ckanext-datastore

## `ogdch_cmd` Paster Commands
## `ogdch_cmd` Commands

To see the help text for any command:

### Command to cleanup the datastore database.
[Datastore currently does not delete tables](https://github.com/ckan/ckan/issues/3422) when the corresponding resource is deleted.
This command finds these orphaned tables and deletes its rows to free the space in the database.
```bash
ckan -c /var/www/ckan/development.ini ogdch [command] --help
```

### Command to clean up the datastore database.
[Datastore currently does not delete tables](https://github.com/ckan/ckan/issues/3422) when the corresponding resource
is deleted. This command finds these orphaned tables and deletes its rows to free the space in the database.
It is meant to be run regularly by a cronjob.

```bash
paster --plugin=ckanext-ogdchcommands ogdch cleanup_datastore -c /var/www/ckan/development.ini
ckan -c /var/www/ckan/development.ini ogdch cleanup_datastore
```

## Command to cleanup the resources.
## Command to clean up the resources.
When datasets are harvested, we try to reuse the existing resources, but not all of them are
reused. Some old resources remain with the state 'deleted'. These orphaned resources can be
deleted with this command. It is meant to be run regularly by a cronjob.
It will also delete all files from the filestore that are associated with the orphaned resources.
It also comes with a dryrun option.

```bash
paster --plugin=ckanext-ogdchcommands ogdch cleanup_resources -c /var/www/ckan/development.ini
ckan -c /var/www/ckan/development.ini ogdch cleanup_resources
```

### Command to cleanup resource-files from the filestore.
Expand All @@ -53,24 +58,24 @@ and from the dependent table package_extra_revision.
The command comes with a dryrun option.

```bash
paster --plugin=ckanext-ogdchcommands ogdch cleanup_extras publishers --dryrun -c /var/www/ckan/development.ini
ckan -c /var/www/ckan/development.ini ogdch cleanup_extras publishers --dryrun
```

## Command to cleanup the harvest jobs.
## Command to clean up the harvest jobs.
This commands deletes the harvest jobs and objects per source and overall leaving only the latest n,
where n and the source are optional arguments. The command is supposed to be used in a cron job to
provide for a regular cleanup of harvest jobs, so that the database is not overloaded with unneeded data
of past job runs. It has a dryrun option so that it can be tested what will get be deleted in the
database before the actual database changes are performed.

```bash
paster --plugin=ckanext-ogdchcommands ogdch cleanup_harvestjobs [{source_id}] [--keep={n}}] [--dryrun] -c /var/www/ckan/development.ini
ckan -c /var/www/ckan/development.ini ogdch cleanup_harvestjobs [{source_id}] [--keep={n}}] [--dryrun]
```

### Command to publish private datasets that have a scheduled-date.
This command will look for private datasets that have the `scheduled`-field set and will publish it if it is due.
```bash
paster --plugin=ckanext-ogdchcommands ogdch publish_scheduled_datasets [--dryrun] -c /var/www/ckan/development.ini
ckan -c /var/www/ckan/development.ini ogdch publish_scheduled_datasets [--dryrun]
```

## Command to clear stale harvest sources.
Expand All @@ -79,7 +84,7 @@ that was not active for a given amount of days (default 30 days).
The command is supposed to be used in a cron job and to check all harvest sources.

```bash
paster --plugin=ckanext-ogdchcommands ogdch clear_stale_harvestsources [--keep_harvestsource_days={n}}] -c /var/www/ckan/development.ini
ckan -c /var/www/ckan/development.ini ogdch clear_stale_harvestsources [--keep_harvestsource_days={n}}]
```

## `ogdch_admin` Admin Tools
Expand Down
79 changes: 0 additions & 79 deletions bin/travis-build.bash

This file was deleted.

12 changes: 0 additions & 12 deletions bin/travis-run.sh

This file was deleted.

2 changes: 2 additions & 0 deletions ckanext/__init__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# this is a namespace package
try:
import pkg_resources

pkg_resources.declare_namespace(__name__)
except ImportError:
import pkgutil

__path__ = pkgutil.extend_path(__path__, __name__)
Loading
Loading