Skip to content

Commit

Permalink
Merge pull request #45 from LocationMind/last-change-before-release
Browse files Browse the repository at this point in the history
Last change before release
  • Loading branch information
MatRouillard authored Nov 7, 2024
2 parents 1e2158b + 8cdc800 commit def8c06
Show file tree
Hide file tree
Showing 16 changed files with 519 additions and 405 deletions.
107 changes: 78 additions & 29 deletions Documentation/Test_saudi_arabia.md

Large diffs are not rendered by default.

22 changes: 11 additions & 11 deletions Documentation/dev-doc.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ This documentation provides information on how to modify various scripts to eith
- [Adding a new theme](#adding-a-new-theme)
- [GeoDataCompare](#geodatacompare)
- [Change input datasets](#change-input-datasets)
- [Adding a new criterion to the Dashboard](#adding-a-new-criterion-to-the-dashboard)
- [Adding a new theme to the DashBoard](#adding-a-new-theme-to-the-dashboard)
- [Adding a new criterion to the dashboard](#adding-a-new-criterion-to-the-dashboard)
- [Adding a new theme to the dashboard](#adding-a-new-theme-to-the-dashboard)
- [Creating the new theme class](#creating-the-new-theme-class)
- [Adding elements to the DashBoard](#adding-elements-to-the-dashboard)
- [Adding elements to the dashboard](#adding-elements-to-the-dashboard)

# Data Integration Process

Expand Down Expand Up @@ -453,11 +453,11 @@ To add this theme to the dashboard, please refer to the [GeoDataCompare](#adding

# GeoDataCompare

This section explains how to implement changes made in the data integration process into the DashBoard.
This section explains how to implement changes made in the data integration process into the dashboard.

## Change input datasets

If another dataset is used, it is quite easy to change it in the DashBoard.
If another dataset is used, it is quite easy to change it in the dashboard.

First, it is necessary to add a specific class in the [datasets.py](../src/GeoDataCompare/datasets.py) file.
Here is a template for a new dataset:
Expand Down Expand Up @@ -488,7 +488,7 @@ This issue will likely be addressed later.

One might also want to change both [help.md](../src/GeoDataCompare/help.md) and [licenses.md](../src/GeoDataCompare/licenses.md) files to include information about the new dataset.

## Adding a new criterion to the Dashboard
## Adding a new criterion to the dashboard

To add a new criterion in the database, there are two necessary steps:

Expand Down Expand Up @@ -602,7 +602,7 @@ Several things are important to notice here:

- `columnCriterion`: Name of the column on which the indicator value is calculated. It is not mandatory if, for instance, only geometrical operations are necessary for the indicator.

- `displayNameCriterion`: Name of the criterion to display in the indicator card in the Dashboard. Usually, it is the name of the criterion with a unit (e.g. km, km², %), but it is not mandatory.
- `displayNameCriterion`: Name of the criterion to display in the indicator card in the dashboard. Usually, it is the name of the criterion with a unit (e.g. km, km², %), but it is not mandatory.

- `icon`: [Font Awesome](https://fontawesome.com/icons/) icon name. It is possible to search for an icon and copy the name of the icon in order to use it directly in the application.

Expand Down Expand Up @@ -689,9 +689,9 @@ Then, the changes should be applied to the new criterion as well.

If the style is a range style, like the connected components, please refer to the part of the code where the sidebar components are added ([here](../src/GeoDataCompare/app.py#L462)) and to the [`colorRange`](../src/GeoDataCompare/app.py#L1203) function at the end of the script to create new components and possibly new functions to change the style.

## Adding a new theme to the DashBoard
## Adding a new theme to the dashboard

To add a new theme to the DashBoard, in theory, only the [theme.py](../src/GeoDataCompare/theme.py) file should be modified, and a new class should be created.
To add a new theme to the dashboard, in theory, only the [theme.py](../src/GeoDataCompare/theme.py) file should be modified, and a new class should be created.
However, if a new theme is added, there would likely be a new general value added (e.g., number of elements in the base layer for this theme).

### Creating the new theme class
Expand Down Expand Up @@ -775,9 +775,9 @@ quality_criteria_choices = {

You would also be likely to change the `Dataset` class in the [dataset.py](../src/GeoDataCompare/datasets.py) and its subclasses in order to add the new layer directly in the `Dataset` implementation, such as adding a new attribute `layerName: str` in the `Dataset` class, and then setting its value in every subclass with `self.layerName = "layer_{}"` in the `__init__` function.

### Adding elements to the DashBoard
### Adding elements to the dashboard

It is easy to add an element to the DashBoard. Three steps are necessary for this:
It is easy to add an element to the dashboard. Three steps are necessary for this:

1. First, add two new variables to all classes in the [general_values](../src/GeoDataCompare/general_values.py) script, such as `nbLayerDatasetA: str` and `nbLayerDatasetB: str`. For the `DefaultGeneralValues`, set their values to an empty string in the `__init__` function. For the `GeneralValues`, set their values to `self.nbLayerDatasetA = self.getNbRowTable(engine, self.datasetA.layerTable.format(area.lower()))` and `self.nbLayerDatasetB = self.getNbRowTable(engine, self.datasetB.layerTable.format(area.lower()))`.

Expand Down
91 changes: 57 additions & 34 deletions Documentation/user-doc.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# User doc: Quality criteria and dashboard

This file provides information on how to use the Dashboard from scratch. It includes: formatting and preparing the database, adding a new area, downloading and integrating data into the database, launching, and using the Dashboard.
This file provides information on how to use the Dashboard from scratch.
It includes: formatting and preparing the database, adding a new area, downloading and integrating data into the database, launching, and using the Dashboard.

Refer to the [developer documentation](dev-doc.md) for information on how to add a criterion to the dashboard, for instance.

Expand All @@ -24,7 +25,7 @@ Refer to the [developer documentation](dev-doc.md) for information on how to add

# Necessary components

It was explained in another markdown, but to run the application, two main components are required:
To run the application, two main components are required:

- A PostgreSQL database with PostGIS extension;
- Python
Expand All @@ -50,19 +51,22 @@ The database used in this repository is:
- `user`: `postgres`
- `password`: `postgres`

These information can be changed by modifying the [.env](../.env) file directly.
This information can be changed by modifying the [.env](../.env) file directly.

### Common model

To compare OSM and OMF data, a common model was created (for transportation data), mainly based on the OMF transportation model. The model is as follows:
To compare OSM and OMF data, a common model was created (for transportation data), mainly based on the OMF transportation model.
The model is as follows:

![Common model for OSM and OMF dataset](./Images/common_model.png)

### Schemas

Four schemas are used in this database:

- `public`: The default schema, where PostGIS and PgRouting are installed. It also means that, to use these functions, the `public.` prefix must be used to avoid conflicts and problems. Additionally, the bounding box table is located in the public schema for easy access by the others.
- `public`: The default schema, where PostGIS and PgRouting are installed.
It also means that, to use these functions, the `public.` prefix must be used to avoid conflicts and problems.
Additionally, the bounding box table is located in the public schema for easy access by the others.

- `osm`: As the name suggests, this schema contains all tables for OpenStreetMap data.

Expand All @@ -85,7 +89,7 @@ During the installation, one will be able to install PostGIS as well, so it shou
### Create the Database and Schemas

If PgAdmin is being used, one can easily add a new database.
Make sure to call it `pgrouting` to avoid having to change the database name in other files.
Make sure to give the same name here and in the [`.env`](../.env) file too (default name to `pgrouting`)
Otherwise, one can create a database in the command line, using psql:

Connect to the default database with the postgres user: `psql -U postgres`.
Expand All @@ -108,7 +112,7 @@ CREATE SCHEMA IF NOT EXISTS omf;
CREATE SCHEMA IF NOT EXISTS results;
```

*Note*: It is sufficient to create the database, as in the [data_integration.py](../src/Assessment/data_integration.py#L24), these SQL commands are executed (inside the `utils.initialisePostgreSQL()` function).
*Note*: It is sufficient to create the database, as in the [data_integration.py](../src/Assessment/data_integration.py#L24), these SQL commands are executed using the `utils.initialisePostgreSQL()` function.

## Python

Expand All @@ -128,11 +132,9 @@ To do so, run these commands after downloading Python:
**Create virtual environment**
```cmd
python -m venv .venv
.venv\Scripts\activate
```

**Activate / deactivate**

**Activate / Deactivate**
```cmd
.venv\Scripts\activate
Expand All @@ -145,7 +147,6 @@ python.exe -m pip install --upgrade pip
```

**Install dependencies**

```cmd
pip install pip-tools && pip-compile Requirements\requirements.in && pip install -r Requirements\requirements.txt
```
Expand Down Expand Up @@ -200,11 +201,21 @@ If modifications to these areas are desired to download data from other location
The important part is to only add elements to the `bboxs` array.
Each element has two mandatory attributes:

- `bbox`: The bounding box of the area, in CSV format. Data will be extracted from this area. The [bounding box tool](https://boundingbox.klokantech.com/) can be used to easily create a bounding box, and the result can be copied and pasted directly into CSV format. Currently, extended areas are not readily available, so it is recommended to use bounding boxes of approximately 8 x 8 km. Direct measurement on the bounding box tool is not possible, but [this website](https://www.freemaptools.com/measure-distance.htm) can be used to approximately measure the bounding box. Ensure there are no spaces between numbers and commas.
- `bbox`: The bounding box of the area, in CSV format.
Data will be extracted from this area.
This [bounding box tool](https://boundingbox.klokantech.com/) can be used to easily create a bounding box, and the result can be copied and pasted directly into CSV format.
Currently, extended areas are not really available, so it is recommended to use bounding boxes of approximately 8 x 8 km.
Direct measurement on the bounding box tool is not possible, but [this website](https://www.freemaptools.com/measure-distance.htm) can be used to approximately measure the bounding box.
Ensure there are no spaces between numbers and commas.

- `name`: The name of the area. It should start with a capital letter, and the remaining characters must be in lowercase. No spaces or special accents should be used. Although inconvenient, this constraint is currently unavoidable. Additionally, the name should be unique to avoid issues during processing.
- `name`: The name of the area.
It should start with a capital letter, and the remaining characters must be in lowercase.
No spaces or special accents should be used.
Although inconvenient, this constraint is currently unavoidable.
Additionally, the name should be unique to avoid issues during processing.

All areas listed in this file will be downloaded and used in the other scripts, so remove any that are not desired for download. If an area has already been downloaded, there will be no issues.
All areas listed in this file will be downloaded and used in the other scripts, so remove any that are not desired for download.
If an area has already been downloaded, there will be no issues.

## Custom Database Connection

Expand All @@ -217,7 +228,8 @@ In the Python files, the database connection can take one of three forms:
- `initialiseDuckDB()`: Initializes DuckDB and connects it to a PostgreSQL database.

These functions are defined in the [utils.py](../src/Utils/utils.py) script.
They require one argument, a path to a `.env` file. If no path is provided, the default path will be:
They require one argument, a path to a `.env` file.
If no path is provided, the default path will be:

```python
os.path.join(os.getcwd(), '.env')
Expand All @@ -230,15 +242,16 @@ However, if it is not, the path to the `.env` file will need to be specified in

The connection parameters are:

- `POSTGRES_DATABASE`: Name of the database. Defaults to `pgrouting`.

- `POSTGRES_HOST`: IP address of the host. Defaults to `127.0.0.1`.

- `POSTGRES_USER`: User name. Defaults to `postgres`.

- `POSTGRES_PASSWORD`: Password. Defaults to `postgres`.

- `POSTGRES_PORT`: Port to connect to. Defaults to `5432`.
- `POSTGRES_DATABASE`: Name of the database.
Defaults to `pgrouting`.
- `POSTGRES_HOST`: IP address of the host.
Defaults to `127.0.0.1`.
- `POSTGRES_USER`: User name.
Defaults to `postgres`.
- `POSTGRES_PASSWORD`: Password.
Defaults to `postgres`.
- `POSTGRES_PORT`: Port to connect to.
Defaults to `5432`.

The same environment file is used for the data integration, quality assessment, and application processes.

Expand All @@ -247,42 +260,52 @@ The same environment file is used for the data integration, quality assessment,
Data corresponding to the chosen areas can be downloaded using the [data_integration.py](../src/Assessment/data_integration.py) script.
Certain attribute values in this file can be modified, such as:

- [`createBoundingBoxTable`](../src/Assessment/data_integration.py#L20): If `True`, it will create the bounding box table, even if it has already been created. Defaults to `True`.
- [`createBoundingBoxTable`](../src/Assessment/data_integration.py#L20): If `True`, it will create the bounding box table, even if it has already been created.
Defaults to `True`.

- [`skip<theme>Check`](../src/Assessment/data_integration.py#L58): If `True`, it will recreate all layers of each area for the specified theme (one of `Graph`, `Building`, or `Place`). Otherwise, layers will be created only if they have not been created yet. Defaults to `False`.
- [`skip<theme>Check`](../src/Assessment/data_integration.py#L58): If `True`, it will recreate all layers of each area for the specified theme (one of `Graph`, `Building`, or `Place`).
Otherwise, layers will be created only if they have not been created yet.
Defaults to `False`.

- [`ox.settings.overpass_settings`](../src/Assessment/data_integration.py#L38): This setting is used to limit OSM data to a specific date. The default date is `2024-06-07T23:59:59Z`. This date can be changed if desired, but it is preferable to select a date approximately the same as the one used by the `overturemaps.py` tool (the default date corresponds to the 2024-06-13-beta.1 release).
- [`ox.settings.overpass_settings`](../src/Assessment/data_integration.py#L38): This setting is used to limit OSM data to a specific date.
The default date is `2024-09-30T23:59:59Z`.
This date can be changed if desired, but it is preferable to select a date approximately the same as the one used by the `overturemaps.py` tool (the default date corresponds to the 2024-10-23.0 release).

Normally, no additional changes should be necessary (such as altering template names for the layer, schema names, or the path to the bbox file, etc.).
If changes are required, they should be made consistently across the different files.

The script can be run with this command:

```cmd
python Python\Assessment\data_integration.py
python src\Assessment\data_integration.py
```

Or directly by running the script in an IDE of choice.

## Quality Assessment Criteria

Once the data has been downloaded, the necessary scripts can be run to assess the quality of the different layers.
Currently, the criteria are focused on graph data. The Python script is [quality_assessment.py](../src/Assessment/quality_assessment.py), that uses the function in the [quality.py](../src/Assessment/quality.py) script.
Currently, the criteria are focused on graph data.
The Python script is [quality_assessment.py](../src/Assessment/quality_assessment.py), that uses the function in the [quality.py](../src/Assessment/quality.py) script.

Certain attribute values can be modified, such as:

- [`fileName`](../src/Assessment/quality_assessment.py#L20): The name of the markdown file that will be produced. Defaults to `Automatic_result.md`.
- [`fileName`](../src/Assessment/quality_assessment.py#L20): The name of the markdown file that will be produced.
Defaults to `Automatic_result.md`.

- [`pathSave`](../src/Assessment/quality_assessment.py#L22): The path to save the markdown file. If the folder does not exist, an error will be returned. Defaults to `./Data/Results/<fileName>`, where `<fileName>` is the previously mentioned variable.
- [`pathSave`](../src/Assessment/quality_assessment.py#L22): The path to save the markdown file.
If the folder does not exist, an error will be returned.
Defaults to `./Data/Results/<fileName>`, where `<fileName>` is the previously mentioned variable.

- [`bounding_box_table`](../src/Assessment/quality_assessment.py#L59): The name of the bounding box table in the public schema of the database. Defaults to `bounding_box`.
- [`bounding_box_table`](../src/Assessment/quality_assessment.py#L59): The name of the bounding box table in the public schema of the database.
Defaults to `bounding_box`.

Other variables can also be changed, but this is not recommended, especially since changes would need to be updated in the dashboard as well.

As with the previous script, this script can be run with this command:

```cmd
python Python\Assessment\quality_assessment.py
python src\Assessment\quality_assessment.py
```

Or directly by running the script in an IDE of choice.
Expand All @@ -295,7 +318,7 @@ Calculating everything may take some time, but the results will be printed to th
To run the application, since the requirements are already downloaded, one can simply use this command:

```
shiny run .\Python\Web\Dashboard\app.py
shiny run .\src\Web\Dashboard\app.py
```

Then, open your browser and navigate to [http://127.0.0.1:8000](http://127.0.0.1:8000).
Expand Down
21 changes: 21 additions & 0 deletions LICENSE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2024 LocationMind Inc.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Loading

0 comments on commit def8c06

Please sign in to comment.