Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[IMPORT] Include import documentation in GeoNature documention #2970

Closed
wants to merge 120 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
120 commits
Select commit Hold shift + click to select a range
5a13925
import schema
bouttier Dec 3, 2023
accae28
import to synthese
bouttier Dec 3, 2023
3cb2f60
update synthese import after checks rework
bouttier Dec 11, 2023
d7049df
import to occhab
bouttier Dec 11, 2023
ceef3de
rework uuid checks
bouttier Dec 18, 2023
c1090e7
add imports backend code from gn_module_import
bouttier Dec 18, 2023
160557b
add dependencies required for imports
bouttier Dec 18, 2023
9867b32
bump pandas 1.3.5 → 2.1.4
bouttier Dec 18, 2023
2beea9a
add redis service required by import
bouttier Dec 18, 2023
4c5c293
run test for PR on branch feat/import
bouttier Dec 21, 2023
9df80b7
feat(import) moving import frontend into the core
Pierre-Narcisi Dec 20, 2023
3886646
Feat(import) remove ng_module from import
Pierre-Narcisi Dec 20, 2023
d6bdde0
lint(import) apply prettier to imported code
bouttier Dec 21, 2023
c69baf8
Fix(destination) add type occhab to module occhab
Pierre-Narcisi Dec 21, 2023
4f80359
fix(celery): move session.remove() in app context
bouttier Jan 9, 2024
0a7818c
permissions: skip caching when no request context
bouttier Jan 9, 2024
18aba42
fix(occhab): defaults nomenclature for 'ALL' orga
bouttier Jan 9, 2024
889156f
chore(occhab): complete models
bouttier Jan 9, 2024
74b9e3e
fix(tests/imports): determinist ordering
bouttier Jan 10, 2024
64a03ac
fix(tests/imports): use M5 instead of M10
bouttier Jan 10, 2024
20e0d94
fix(imports): several checks corrections
bouttier Jan 9, 2024
e0e376b
feat(imports): add parent/child related checks
bouttier Jan 9, 2024
3e6c00f
feat(imports): add parent related error code
bouttier Dec 28, 2023
756287c
feat(imports): add preprocess transient data hook
bouttier Jan 9, 2024
8748af5
chore(imports): concat_dates takes columns
bouttier Jan 9, 2024
49ec338
feat(import/occhab): add id_station_source
bouttier Dec 28, 2023
c21020f
feat(imports/occhab): add id_import FK in pr_occhab
bouttier Dec 28, 2023
208c381
feat(imports/occhab): fix checks and add import
bouttier Dec 18, 2023
f4e44a7
tests(import/occhab): improve coverage
bouttier Dec 28, 2023
ba28a58
feat(imports/occhab): preprocess transient data
bouttier Jan 9, 2024
4502684
add merge revision following rebase on develop
bouttier Jan 11, 2024
98cd3ba
tests: propagates eager task exceptions
bouttier Jan 16, 2024
9a901ee
tests(import/occhab): assert processed
bouttier Jan 16, 2024
3a531fc
fix(import/occhab): insert fields comment
bouttier Jan 16, 2024
aea31f2
import: rm synthese specific column on list
bouttier Jan 17, 2024
e4047ea
feat(back): list all imports
andriacap Jan 16, 2024
6489172
test(back): get list all import
andriacap Jan 16, 2024
25ecd67
refact: remove useless return
andriacap Jan 18, 2024
57b9559
fix(tests/import): fixtures
bouttier Jan 25, 2024
c3908de
fixtures: associate datasets to occhab
bouttier Jan 25, 2024
f5c3276
fix(occhab): fix edit habitats
Pierre-Narcisi Jan 25, 2024
f222089
Fix(occhab) remove import error du to default value creating a non su…
Pierre-Narcisi Jan 25, 2024
990e72a
Feat(destinations) adding destination to modal and moving jdd to uplo…
Pierre-Narcisi Jan 11, 2024
3cd69da
Feat(datasets) managing datasets controls
Pierre-Narcisi Jan 12, 2024
c037349
Feat(import) fetch module in destination for datasets
Pierre-Narcisi Jan 15, 2024
5860933
Feat(destinations) adding destination to modal and moving jdd to uplo…
Pierre-Narcisi Jan 11, 2024
56c34e3
Feat(import) fetch module in destination for datasets
Pierre-Narcisi Jan 15, 2024
492d40c
Feat(destination) add destination tu api calls in frontend, using rou…
Pierre-Narcisi Jan 17, 2024
8e39af4
Feat(destination) Multilingue and fix navigation
Pierre-Narcisi Jan 18, 2024
4b5ac9a
feat(notifications): make notifs for import-done destination-generic
VincentCauchois Jan 23, 2024
66e6314
feat: add tab to handle multiple entities in fields-mapping-step
edelclaux Feb 2, 2024
2fda2b6
lint: apply prettier to fields-mapping-step.component.html
edelclaux Feb 2, 2024
0aafeb5
fix(import): access to import report from several places
Pierre-Narcisi Feb 1, 2024
6100cb3
fix(import): download file and delete import from imports list
Pierre-Narcisi Feb 1, 2024
ad898b4
feat: remove specific id_source col
andriacap Jan 30, 2024
f5d1910
feat: add optional destination to getImportList
andriacap Jan 30, 2024
3915d42
feat: add select destination component
andriacap Jan 30, 2024
b8fa0f7
feat: add column destination to list import
andriacap Jan 30, 2024
112495f
test(back): add test on sort destination.code
andriacap Jan 31, 2024
6782bee
style: change ux filter destination and language
andriacap Jan 31, 2024
c4e1b52
feat: fieldmapping synchro
edelclaux Feb 2, 2024
2d9da5b
fix backend test
jacquesfize Feb 14, 2024
df89603
feat: remove taxa_count and add stats in report
andriacap Feb 13, 2024
379f338
front: add upload to config i18n
andriacap Feb 13, 2024
6588df2
feat: add btn "Upload" to synthese component
andriacap Feb 13, 2024
293ee45
fix(style): html and css for occhab page
andriacap Feb 13, 2024
494cf2d
feat: add btn "Upload" to OCCHAB page
andriacap Feb 13, 2024
105f9fc
style(front): apply prettier
andriacap Feb 13, 2024
66050cd
refact: use different way to use module_code
andriacap Feb 14, 2024
6ab2255
feat: add conditional display btn based on cruved
andriacap Feb 14, 2024
b22ef55
fix: add comma for css
andriacap Feb 14, 2024
d19d2a8
style(front): apply prettier
andriacap Feb 14, 2024
5dd96d2
feat(front): add cruved Import srvc into occhab
andriacap Feb 19, 2024
612c00f
feat(front): add cruved Import srvc into synthese
andriacap Feb 19, 2024
30ae245
Feat(destination) getting destination in modules
andriacap Feb 19, 2024
96e6444
feat(import): add statistics display in import list
edelclaux Feb 13, 2024
39a31a6
lint(import): apply linting to import-list-component
edelclaux Feb 15, 2024
1951f1c
feat(import): move geom_from_area_code to synthese imports
edelclaux Feb 20, 2024
e65268e
feat: group field by entity in report
andriacap Feb 7, 2024
b751ffe
feat(import): use geom_4326 instead of transform(geom, 4326)
edelclaux Feb 20, 2024
b3d632f
doc: add todos to use geom_4326 instead of transform(geom, 4326)
edelclaux Feb 20, 2024
dfc46ab
feat(import): add statistics labels
bouttier Feb 5, 2024
935472c
feat(import): add counting statistics in import report
VincentCauchois Jan 31, 2024
ab79dc7
import: geo checks fixes
bouttier Feb 21, 2024
8530035
feat/add-constraint-t_habitats.id_nomenclature_community_interest
edelclaux Feb 7, 2024
7720654
feat(checks) manage multi in checks
Pierre-Narcisi Feb 12, 2024
9856ac0
feat(metadata) fix import button
Pierre-Narcisi Feb 14, 2024
d69b7cb
feat: add test for line number errors
andriacap Feb 15, 2024
4b102c1
import: simplify errors csv test
bouttier Feb 21, 2024
d67c253
feat(import): add entity handling in error report
edelclaux Feb 22, 2024
8ab25ef
fix(import): entity & errors
bouttier Feb 22, 2024
66b1a11
tests(import): assert errors entity
bouttier Feb 22, 2024
cee819e
feat(import): enum based error in import step component + remove comm…
edelclaux Feb 12, 2024
f582f88
feat(import): improve error displayed
edelclaux Feb 12, 2024
9382151
fix search on import list
jacquesfize Feb 22, 2024
36801e9
fix lint
jacquesfize Feb 22, 2024
80c53a3
debug import list filters
jacquesfize Feb 29, 2024
2658f5e
fix(import) fix design flaws
Pierre-Narcisi Mar 6, 2024
58106f5
fix lint
jacquesfize Mar 6, 2024
d4a3178
doc(test): add todos for occhab impor test
edelclaux Feb 20, 2024
3b3c90a
fix: label stats in report pdf
andriacap Feb 28, 2024
7bee951
merge alembic branch
jacquesfize Mar 8, 2024
d87e880
fix rebase error
jacquesfize Mar 8, 2024
e7cca18
feat (mapping, import) renaming data service and variables (less ambi…
jacquesfize Mar 18, 2024
5c0a608
add CruvedWithScope interface + function to parse the latter to boole…
jacquesfize Mar 18, 2024
79d4949
refactor
jacquesfize Mar 18, 2024
3320f64
Feat(mapping) controling entities controls and adding a badge
Pierre-Narcisi Mar 18, 2024
632af60
(fix) badge computation no longer block the from validation process
jacquesfize Mar 18, 2024
bf5a4c6
lint + remove console.log
jacquesfize Mar 18, 2024
4e3e558
remove last trace of synthese in fields-mapping component
jacquesfize Mar 19, 2024
e8dde46
feat(import list, statistic): enable generic tooltip in the import list
jacquesfize Mar 15, 2024
5789ecc
(fix) typo
jacquesfize Mar 15, 2024
f992232
feat(import report filename) : add item in import configuration
jacquesfize Mar 15, 2024
bd588cf
feat(import report filename) : add function to generate report filename
jacquesfize Mar 15, 2024
d7e1849
feat(import report filename) : add import default value in default_co…
jacquesfize Mar 15, 2024
4630670
fix lint + refact
jacquesfize Mar 15, 2024
e3f5dd7
indicate unused import configuration variable
jacquesfize Mar 15, 2024
63815f7
update defaut_config toml
jacquesfize Mar 15, 2024
3acdfe7
Feat(mapping) sort mapping and put a ng-select
Pierre-Narcisi Mar 15, 2024
a0e943c
(import,doc) : import orginal documention from gn_module_import
jacquesfize Mar 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/cypress.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ on:
- master
- hotfixes
- develop
- fixtestfront
pull_request:
branches:
- master
- hotfixes
- develop
- feat/import

jobs:
mount_app_and_run_cypress:
Expand Down
10 changes: 10 additions & 0 deletions .github/workflows/pytest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ on:
- master
- hotfixes
- develop
- feat/import

jobs:
build:
Expand Down Expand Up @@ -46,6 +47,15 @@ jobs:
--health-interval 10s
--health-timeout 5s
--health-retries 5
redis:
image: redis
ports:
- 6379:6379
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5

steps:
- uses: actions/checkout@v4
Expand Down
1 change: 1 addition & 0 deletions backend/geonature/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,7 @@ def load_current_user():
("geonature.core.gn_profiles.routes:routes", "/gn_profiles"),
("geonature.core.sensitivity.routes:routes", None),
("geonature.core.notifications.routes:routes", "/notifications"),
("geonature.core.imports.blueprint:blueprint", "/import"),
]:
module_name, blueprint_name = blueprint_path.split(":")
blueprint = getattr(import_module(module_name), blueprint_name)
Expand Down
5 changes: 4 additions & 1 deletion backend/geonature/celery_app.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
from .app import create_app
from .utils.celery import celery_app as app
from .utils.module import iter_modules_dist
from .utils.env import db


flask_app = create_app()
Expand All @@ -9,7 +10,9 @@
class ContextTask(app.Task):
def __call__(self, *args, **kwargs):
with flask_app.app_context():
return self.run(*args, **kwargs)
result = self.run(*args, **kwargs)
db.session.remove()
return result


app.Task = ContextTask
Expand Down
3 changes: 2 additions & 1 deletion backend/geonature/core/gn_commons/routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ def list_modules():
query = (
select(TModules)
.options(joinedload(TModules.objects))
.options(joinedload(TModules.destination))
.where(TModules.module_code.notin_(exclude))
.order_by(TModules.module_order.asc())
.order_by(TModules.module_label.asc())
Expand All @@ -79,7 +80,7 @@ def list_modules():
# HACK : on a besoin d'avoir le module GeoNature en front pour l'URL de la doc
if module.module_code == "GEONATURE":
module_allowed = True
module_dict = module.as_dict(fields=["objects"])
module_dict = module.as_dict(fields=["objects", "destination.code"])
# TODO : use has_any_permissions instead - must refactor the front
module_dict["cruved"] = {
action: get_scope(action, module_code=module.module_code, bypass_warning=True)
Expand Down
1 change: 1 addition & 0 deletions backend/geonature/core/gn_monitoring/routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ def get_site_areas(id_site):
params = request.args

query = (
# TODO@LAreas.geom_4326
select(corSiteArea, func.ST_Transform(LAreas.geom, 4326))
.join(LAreas, LAreas.id_area == corSiteArea.c.id_area)
.where(corSiteArea.c.id_base_site == id_site)
Expand Down
20 changes: 13 additions & 7 deletions backend/geonature/core/gn_permissions/tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

import sqlalchemy as sa
from sqlalchemy.orm import joinedload
from flask import g
from flask import has_request_context, g

from geonature.core.gn_commons.models import TModules
from geonature.core.gn_permissions.models import (
Expand Down Expand Up @@ -50,9 +50,12 @@ def _get_user_permissions(id_role):
def get_user_permissions(id_role=None):
if id_role is None:
id_role = g.current_user.id_role
if id_role not in g._permissions_by_user:
g._permissions_by_user[id_role] = _get_user_permissions(id_role)
return g._permissions_by_user[id_role]
if has_request_context():
if id_role not in g._permissions_by_user:
g._permissions_by_user[id_role] = _get_user_permissions(id_role)
return g._permissions_by_user[id_role]
else:
return _get_user_permissions(id_role)


def _get_permissions(id_role, module_code, object_code, action_code):
Expand Down Expand Up @@ -93,9 +96,12 @@ def get_permissions(action_code, id_role=None, module_code=None, object_code=Non
object_code = "ALL"

ident = (id_role, module_code, object_code, action_code)
if ident not in g._permissions:
g._permissions[ident] = _get_permissions(*ident)
return g._permissions[ident]
if has_request_context():
if ident not in g._permissions:
g._permissions[ident] = _get_permissions(*ident)
return g._permissions[ident]
else:
return _get_permissions(*ident)


def get_scope(action_code, id_role=None, module_code=None, object_code=None, bypass_warning=False):
Expand Down
Loading
Loading