Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding skills. See #23 #28

Merged
merged 3 commits into from
May 7, 2024
Merged

Adding skills. See #23 #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
95a6edf
adding skills. See #23
ManuelAudisio May 3, 2024
5ca9974
adding skills. See #23
Edocampa May 7, 2024
9d2598c
Fix: #23. Use a separate virtuoso instance. Add tests. Add skills.
ioggstream May 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified esco/esco.json.gz
View file
Open in desktop
Binary file not shown.
Binary file modified esco/esco_o.json.gz
View file
Open in desktop
Binary file not shown.
Binary file modified esco/esco_s.json.gz
View file
Open in desktop
Binary file not shown.
1 change: 1 addition & 0 deletions esco/sparql.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ def load_esco(self, categories=None):
"http://data.europa.eu/esco/isced-f/06",
"http://data.europa.eu/esco/skill/243eb885-07c7-4b77-ab9c-827551d83dc4",
"http://data.europa.eu/esco/skill/b590d4e5-7c62-4b4a-abc2-c270b482e0ce",
"http://data.europa.eu/esco/skill/bec4359e-cb92-468f-a997-8fb28e32fba9",
]

categories = "\n".join([f"<{uri}>" for uri in categories])
Expand Down
9 changes: 6 additions & 3 deletions tests/test_localdb_base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,10 @@ def test_get_skill(db):
assert skill["skillType"] == "skill"


def test_search_skill_label(db):
products = {"ansible", "JBoss", "Bash"}
@pytest.mark.parametrize(
"products,expected_results",
[({"ansible", "JBoss", "Bash"}, 3), ({"agile", "scrum", "kanban"}, 1)],
)
def test_search_skill_label(db, products, expected_results):
Comment on lines +43 to +47
Copy link
Contributor

@ioggstream ioggstream May 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verify that both old and new skills are accessible, using a parametric test.

skills = db.search_products(products)
assert len(skills) >= 3
assert len(skills) >= expected_results
11 changes: 10 additions & 1 deletion tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@ commands =
python -m pytest tests/ {posargs}


#
# Install in the tox environment all dependencies
# and run the model generation.
# Note that this won't run tests.
#
[testenv:model]
deps =
.
Expand All @@ -42,12 +47,16 @@ allowlist_externals =
sitepackages = True

commands_pre =
python -m spacy download en_core_web_trf
# Eventually download en_core_web_trf.
python -c "import spacy; m='en_core_web_trf'; spacy.util.is_package(m) or spacy.cli.download(m)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We download the model only if it's not already there. This saves a lot of time (and CO2)

# Ensure sparql is populated.
pytest tests/test_sparql.py
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test if sparql is ok before rebuilding the model. If this fails, then the sparql server is not up and running.


commands =
python model/model.py
rm generated/output -fr
mkdir -p generated/output
pytest tests/test_localdb_base.py
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after the build, we test that new skills are there.

python -m spacy package ./generated/en_core_web_trf_esco_ner ./generated/output --build wheel

bash -c '[ "{posargs}" == "upload" ] && python -m spacy huggingface-hub push generated/output/en_core_web_trf_esco_ner*/dist/*.whl'
Expand Down
Loading