From 8ee7a278b52fe48d3be50d320519d35c51c00776 Mon Sep 17 00:00:00 2001 From: Neelima Mohanty <97789856+neelimagoogly@users.noreply.github.com> Date: Fri, 7 Jul 2023 12:45:06 +0530 Subject: [PATCH 01/18] Created FAQ.md --- docs/FAQ.md | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 docs/FAQ.md diff --git a/docs/FAQ.md b/docs/FAQ.md new file mode 100644 index 00000000..c9ba4e31 --- /dev/null +++ b/docs/FAQ.md @@ -0,0 +1,74 @@ +# Frequently Asked Questions (FAQs) + +## Users +**Q : What is fAIr?** + +A : fAIr is an open-source toolkit developed by the Humanitarian OpenStreetMap Team (HOT) that enables the integration of artificial intelligence (AI) into humanitarian mapping workflows. It provides AI models and tools to automate mapping tasks, improving efficiency and accuracy. + +
+ +**Q : How can I use fAIr in my mapping projects?** + +A : To use fAIr, you can follow the installation instructions provided in the documentation. Once installed, you can utilize the pre-trained AI models for tasks such as road network extraction, building footprint detection, or land use classification. The documentation offers examples and guidelines to help you get started. + +
+ +**Q : What types of data can I use with fAIr?** + +A : fAIr primarily works with satellite imagery and aerial photographs. You can use various types of imagery sources, including public satellite imagery datasets, to perform mapping tasks using the AI models provided. The documentation provides information on the compatible data formats and recommended data sources. + +
+ +**Q : Can I customize the AI models in fAIr for my specific needs?** + +A : Yes, fAIr is designed to be customizable. The toolkit provides instructions and guidelines on how to train and fine-tune models with your own datasets. This allows you to adapt the models to suit your specific mapping requirements and improve their performance on your target areas or features of interest. + +
+ +**Q : Are there any limitations to using fAIr?** + +A : While fAIr offers powerful AI models, it's important to note that the accuracy of the results may vary depending on the quality and resolution of the input imagery and the complexity of the mapping task. Some tasks may require additional manual editing or validation. It's recommended to review the documentation and understand the limitations of each model before use. + +
+ +**Q : How can I contribute feedback or report issues with fAIr?** + +A : You can contribute to the improvement of fAIr by providing feedback, reporting issues, or suggesting new features through the project's GitHub repository. By creating an issue, you can share your observations or difficulties faced while using fAIr, allowing the developers to address them and enhance the toolkit. + +
+ +**Q : Can I integrate fAIr with other mapping tools or platforms?** + +A : Yes, fAIr is designed to be modular and can be integrated into existing mapping workflows and platforms. It provides interoperability with tools like OpenStreetMap, QGIS, and ArcGIS. You can leverage the AI models from fAIr within these tools or use the outputs of fAIr in combination with other mapping datasets and tools. + +
+ +## Developers + +**Q : How can I contribute to the development of fAIr?** + +A : You can contribute to fAIr's development by following the guidelines provided in the project's GitHub repository. You can contribute code, suggest new features, or help improve documentation. By submitting pull requests, you can propose changes and collaborate with the project maintainers and the community. + +
+ +**Q : What programming languages and frameworks are used in fAIr?** + +A : fAIr frontend is made with React and backend with Django.fAIr is primarily developed using Python, which is widely used in machine learning and AI. It leverages popular frameworks such as PyTorch and TensorFlow for model development and training. Familiarity with Python and these frameworks will be beneficial if you plan to work on fAIr's development. + +
+ +**Q : Are there any specific coding standards or guidelines for contributing to fAIr?** + +A : Yes, fAIr follows coding standards and guidelines outlined in the project's repository. These guidelines cover aspects such as code formatting, documentation, and testing. Adhering to these standards helps maintain code consistency and facilitates collaboration among contributors. + +
+ +**Q : Can I propose new AI models or enhancements to existing models?** + +A : Absolutely! The fAIr project welcomes contributions related to new AI models or enhancements to existing models. If you have developed a new model that aligns with the project's goals or if you have ideas to improve the current models, you can propose them through the GitHub repository and engage in discussions with the community or you can create them at [here](https://fair-dev.hotosm.org/ai-models) . + +
+ +**Q : What kind of support can I expect as a developer contributing to fAIr?** + +A : As a developer contributing to fAIr, you can receive support and guidance from the project maintainers and the community. The maintainers can help address technical questions, review your contributions, and provide feedback. The community also offers a collaborative environment where you can interact with other contributors and share knowledge. From 7643a09d50ed404c7372e1a658c4e1221483264a Mon Sep 17 00:00:00 2001 From: Neelima Mohanty <97789856+neelimagoogly@users.noreply.github.com> Date: Fri, 7 Jul 2023 13:44:34 +0530 Subject: [PATCH 02/18] Update FAQ.md --- docs/FAQ.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/FAQ.md b/docs/FAQ.md index c9ba4e31..1eb3c407 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -9,13 +9,13 @@ A : fAIr is an open-source toolkit developed by the Humanitarian OpenStreetMap T **Q : How can I use fAIr in my mapping projects?** -A : To use fAIr, you can follow the installation instructions provided in the documentation. Once installed, you can utilize the pre-trained AI models for tasks such as road network extraction, building footprint detection, or land use classification. The documentation offers examples and guidelines to help you get started. +A : Building for now but to use fAIr, you can follow the installation instructions provided in the documentation. Once installed, you can utilize the pre-trained AI models for tasks such as road network extraction, building footprint detection, or land use classification. The documentation offers examples and guidelines to help you get started.
**Q : What types of data can I use with fAIr?** -A : fAIr primarily works with satellite imagery and aerial photographs. You can use various types of imagery sources, including public satellite imagery datasets, to perform mapping tasks using the AI models provided. The documentation provides information on the compatible data formats and recommended data sources. +A : fAIr primarily works with satellite imagery and aerial photographs. You can use various types of imagery sources which compiles with opensource imagery license to perform mapping tasks using the AI models provided. The documentation provides information on the compatible data formats and recommended data sources.
@@ -39,7 +39,7 @@ A : You can contribute to the improvement of fAIr by providing feedback, reporti **Q : Can I integrate fAIr with other mapping tools or platforms?** -A : Yes, fAIr is designed to be modular and can be integrated into existing mapping workflows and platforms. It provides interoperability with tools like OpenStreetMap, QGIS, and ArcGIS. You can leverage the AI models from fAIr within these tools or use the outputs of fAIr in combination with other mapping datasets and tools. +A : Yes, fAIr is designed to be modular and can be integrated into existing mapping workflows and platforms. It provides modular pip packages/ API endpoints that can be used in other scripts and can be integrated as plugins on different platforms such as Qgis, Josm etc. You can leverage the AI models from fAIr within these tools or use the outputs of fAIr in combination with other mapping datasets and tools.
@@ -53,7 +53,7 @@ A : You can contribute to fAIr's development by following the guidelines provide **Q : What programming languages and frameworks are used in fAIr?** -A : fAIr frontend is made with React and backend with Django.fAIr is primarily developed using Python, which is widely used in machine learning and AI. It leverages popular frameworks such as PyTorch and TensorFlow for model development and training. Familiarity with Python and these frameworks will be beneficial if you plan to work on fAIr's development. +A : fAIr frontend is made with React and backend with Django.fAIr is primarily developed using Python, which is widely used in machine learning and AI. It leverages popular framework TensorFlow for model development and training. Familiarity with Python and these frameworks will be beneficial if you plan to work on fAIr's development.
@@ -65,7 +65,7 @@ A : Yes, fAIr follows coding standards and guidelines outlined in the project's **Q : Can I propose new AI models or enhancements to existing models?** -A : Absolutely! The fAIr project welcomes contributions related to new AI models or enhancements to existing models. If you have developed a new model that aligns with the project's goals or if you have ideas to improve the current models, you can propose them through the GitHub repository and engage in discussions with the community or you can create them at [here](https://fair-dev.hotosm.org/ai-models) . +A : Absolutely! The fAIr project welcomes contributions related to new AI models or enhancements to existing models. If you have developed a new model that aligns with the project's goals or if you have ideas to improve the current models, you can propose them through the GitHub repository and engage in discussions with the community . Once approved community will be able go use it .
From f5f3daab2277d8d18ef8f9cc87683705f42ba328 Mon Sep 17 00:00:00 2001 From: Neelima Mohanty <97789856+neelimagoogly@users.noreply.github.com> Date: Thu, 13 Jul 2023 10:45:28 +0530 Subject: [PATCH 03/18] Create Contribution.md I have created the contribution doc . @kshitijrajsharma , @omranlm and @petya-kangalova please review it. --- docs/Contribution.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 docs/Contribution.md diff --git a/docs/Contribution.md b/docs/Contribution.md new file mode 100644 index 00000000..ae598298 --- /dev/null +++ b/docs/Contribution.md @@ -0,0 +1,29 @@ +## :hugs: Welcome + +:+1::tada: First off, I'm really glad you're reading this, because we need volunteer developers to help improve the fAIr! :tada::+1: + +We welcome and encourage contributors of all skill levels and we are committed to making sure your participation is inclusive, enjoyable and rewarding. If you have never contributed to an open source project before, we are a good place to start and will make sure you are supported every step of the way. If you have **any** questions, please ask! + +There are many ways to contribute to this repo: + +## Testing + +Adding test cases, or simply testing out existing functionality. + +## Report bugs and suggest improvements + +The [issue queue](https://github.com/hotosm/fAIr/issues) is the best way to get started. There are issue templates for BUGs and FEATURES that you can use, or you can create your own. Once you have submitted an issue, it will be assigned one label out of the following [label categories](https://github.com/hotosm/fAIr/labels). If you are wondering where to start, you can filter by the GoodFirstIssue label. + +## Code contributions + +Create pull requests (PRs) for changes that you think are needed. We would really appreciate your help! + +## Documentation contributions + +Create pull requests (PRs) for changes that you think are needed to the documentation of fAIr.As of now you can find the documentation work at the [docs](https://github.com/hotosm/fAIr/tree/master/docs) directory. + + +## :handshake: Thank you + +Thank you very much in advance for your contributions!! Please ensure you refer to our [Code of Conduct](https://github.com/hotosm/fAIr/blob/master/docs/Code-of-Conduct.md). +If you've read the guidelines, but you are still not sure how to contribute on Github, please reach out to us via our Slack #geospatial-tech-and-innovation. From f6ef8ddf55a82ecd6bd2dafcb8bc02b5d3abb50e Mon Sep 17 00:00:00 2001 From: Neelima Mohanty <97789856+neelimagoogly@users.noreply.github.com> Date: Thu, 13 Jul 2023 11:46:37 +0530 Subject: [PATCH 04/18] Updated frontend readme.md --- frontend/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/frontend/README.md b/frontend/README.md index b49e3bab..e03099e9 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -1,3 +1,5 @@ +# Frontend Deployment for Development + ![example workflow](https://github.com/omranlm/TDB/actions/workflows/frontend_build.yml/badge.svg) ## Getting Started @@ -11,10 +13,10 @@ This project is tested with Node v14.17.3 and Node v16.14.2 ### Create .env file inside root dir -Put your credentials inside .env by following .env [sample](https://github.com/omranlm/TDB/blob/main/frontend/.env%20sample) +Put your credentials inside .env by following .env [sample](https://github.com/omranlm/TDB/blob/main/frontend/.env%20sample) . ### Start development server `npm start` -#### Note : Build and Run Backend First before building frontend +#### Note : Build and Run Backend first by following [this](https://github.com/hotosm/fAIr/blob/master/backend/README.md) before building frontend . From 7f8bc8d5d197b46b31beb88a625cc7ed84920ec3 Mon Sep 17 00:00:00 2001 From: Neelima Mohanty <97789856+neelimagoogly@users.noreply.github.com> Date: Fri, 14 Jul 2023 19:18:16 +0530 Subject: [PATCH 05/18] Updated FAQ.md --- docs/FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/FAQ.md b/docs/FAQ.md index 1eb3c407..e4bfea66 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -65,7 +65,7 @@ A : Yes, fAIr follows coding standards and guidelines outlined in the project's **Q : Can I propose new AI models or enhancements to existing models?** -A : Absolutely! The fAIr project welcomes contributions related to new AI models or enhancements to existing models. If you have developed a new model that aligns with the project's goals or if you have ideas to improve the current models, you can propose them through the GitHub repository and engage in discussions with the community . Once approved community will be able go use it . +A : Absolutely! The fAIr project welcomes contributions related to new AI models or enhancements to existing models. If you have developed a new model that aligns with the project's goals or if you have ideas to improve the current models, you can propose them through the GitHub repository at [fAIr-utilities](https://github.com/hotosm/fAIr-utilities) and engage in discussions with the community . Once approved community will be able go use it .
From 0a10cfd0f2f64846c1455ed0cb11539d5934149a Mon Sep 17 00:00:00 2001 From: Neelima Mohanty <97789856+neelimagoogly@users.noreply.github.com> Date: Mon, 17 Jul 2023 17:07:26 +0530 Subject: [PATCH 06/18] Update Contribution.md --- docs/Contribution.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Contribution.md b/docs/Contribution.md index ae598298..147d4d73 100644 --- a/docs/Contribution.md +++ b/docs/Contribution.md @@ -1,6 +1,6 @@ ## :hugs: Welcome -:+1::tada: First off, I'm really glad you're reading this, because we need volunteer developers to help improve the fAIr! :tada::+1: +:+1::tada: First off, we are really glad you're reading this, because we need volunteer developers to help improve the fAIr! :tada::+1: We welcome and encourage contributors of all skill levels and we are committed to making sure your participation is inclusive, enjoyable and rewarding. If you have never contributed to an open source project before, we are a good place to start and will make sure you are supported every step of the way. If you have **any** questions, please ask! @@ -16,7 +16,7 @@ The [issue queue](https://github.com/hotosm/fAIr/issues) is the best way to get ## Code contributions -Create pull requests (PRs) for changes that you think are needed. We would really appreciate your help! +Fork repo , Maintain your local changes on branch and Create pull requests (PRs) for changes that you think are needed. We would really appreciate your help! ## Documentation contributions From 699bb33aa771587f54d1def2be7b4d6443b8c488 Mon Sep 17 00:00:00 2001 From: Neelima Mohanty <97789856+neelimagoogly@users.noreply.github.com> Date: Mon, 17 Jul 2023 17:09:31 +0530 Subject: [PATCH 07/18] Update README.md --- frontend/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/frontend/README.md b/frontend/README.md index b49e3bab..e03099e9 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -1,3 +1,5 @@ +# Frontend Deployment for Development + ![example workflow](https://github.com/omranlm/TDB/actions/workflows/frontend_build.yml/badge.svg) ## Getting Started @@ -11,10 +13,10 @@ This project is tested with Node v14.17.3 and Node v16.14.2 ### Create .env file inside root dir -Put your credentials inside .env by following .env [sample](https://github.com/omranlm/TDB/blob/main/frontend/.env%20sample) +Put your credentials inside .env by following .env [sample](https://github.com/omranlm/TDB/blob/main/frontend/.env%20sample) . ### Start development server `npm start` -#### Note : Build and Run Backend First before building frontend +#### Note : Build and Run Backend first by following [this](https://github.com/hotosm/fAIr/blob/master/backend/README.md) before building frontend . From be04d27b8b6d92ab471e285070cc4f055b0a387f Mon Sep 17 00:00:00 2001 From: Neelima Mohanty <97789856+neelimagoogly@users.noreply.github.com> Date: Mon, 17 Jul 2023 18:11:39 +0530 Subject: [PATCH 08/18] Create Contribution.md --- docs/Contribution.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 docs/Contribution.md diff --git a/docs/Contribution.md b/docs/Contribution.md new file mode 100644 index 00000000..147d4d73 --- /dev/null +++ b/docs/Contribution.md @@ -0,0 +1,29 @@ +## :hugs: Welcome + +:+1::tada: First off, we are really glad you're reading this, because we need volunteer developers to help improve the fAIr! :tada::+1: + +We welcome and encourage contributors of all skill levels and we are committed to making sure your participation is inclusive, enjoyable and rewarding. If you have never contributed to an open source project before, we are a good place to start and will make sure you are supported every step of the way. If you have **any** questions, please ask! + +There are many ways to contribute to this repo: + +## Testing + +Adding test cases, or simply testing out existing functionality. + +## Report bugs and suggest improvements + +The [issue queue](https://github.com/hotosm/fAIr/issues) is the best way to get started. There are issue templates for BUGs and FEATURES that you can use, or you can create your own. Once you have submitted an issue, it will be assigned one label out of the following [label categories](https://github.com/hotosm/fAIr/labels). If you are wondering where to start, you can filter by the GoodFirstIssue label. + +## Code contributions + +Fork repo , Maintain your local changes on branch and Create pull requests (PRs) for changes that you think are needed. We would really appreciate your help! + +## Documentation contributions + +Create pull requests (PRs) for changes that you think are needed to the documentation of fAIr.As of now you can find the documentation work at the [docs](https://github.com/hotosm/fAIr/tree/master/docs) directory. + + +## :handshake: Thank you + +Thank you very much in advance for your contributions!! Please ensure you refer to our [Code of Conduct](https://github.com/hotosm/fAIr/blob/master/docs/Code-of-Conduct.md). +If you've read the guidelines, but you are still not sure how to contribute on Github, please reach out to us via our Slack #geospatial-tech-and-innovation. From a87af6b5e0f3c3bd7aadc98a6d955b10ea60edd6 Mon Sep 17 00:00:00 2001 From: Neelima Mohanty <97789856+neelimagoogly@users.noreply.github.com> Date: Mon, 17 Jul 2023 18:12:30 +0530 Subject: [PATCH 09/18] Update README.md --- frontend/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/frontend/README.md b/frontend/README.md index b49e3bab..e03099e9 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -1,3 +1,5 @@ +# Frontend Deployment for Development + ![example workflow](https://github.com/omranlm/TDB/actions/workflows/frontend_build.yml/badge.svg) ## Getting Started @@ -11,10 +13,10 @@ This project is tested with Node v14.17.3 and Node v16.14.2 ### Create .env file inside root dir -Put your credentials inside .env by following .env [sample](https://github.com/omranlm/TDB/blob/main/frontend/.env%20sample) +Put your credentials inside .env by following .env [sample](https://github.com/omranlm/TDB/blob/main/frontend/.env%20sample) . ### Start development server `npm start` -#### Note : Build and Run Backend First before building frontend +#### Note : Build and Run Backend first by following [this](https://github.com/hotosm/fAIr/blob/master/backend/README.md) before building frontend . From c65d7249c897735611fc3d059e947c2244af09a5 Mon Sep 17 00:00:00 2001 From: Omran NAJJAR Date: Wed, 19 Jul 2023 14:30:53 +0200 Subject: [PATCH 10/18] Adding validation for the OAM URL --- .../Layout/TrainingDS/DatasetEditor/TileServerList.js | 7 ++++++- .../Layout/TrainingDS/DatasetNew/DatasetNew.js | 9 ++++++++- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/frontend/src/components/Layout/TrainingDS/DatasetEditor/TileServerList.js b/frontend/src/components/Layout/TrainingDS/DatasetEditor/TileServerList.js index 3f1846cf..f6889205 100644 --- a/frontend/src/components/Layout/TrainingDS/DatasetEditor/TileServerList.js +++ b/frontend/src/components/Layout/TrainingDS/DatasetEditor/TileServerList.js @@ -63,6 +63,10 @@ const TileServerList = (props) => { setInputError(false); const res = await axios.get(url.replace("/{z}/{x}/{y}", "")); + if (!res) { + setInputError("Invalid OAM Link"); + return; + } if (res.error) setInputError(res.error.response.statusText); props.addImagery(res.data, url); @@ -74,7 +78,8 @@ const TileServerList = (props) => { return res.data; } catch (e) { console.log("isError"); - setInputError(e); + // setInputError(e); + setInputError("Invalid OAM Link"); } finally { setLoading(false); } diff --git a/frontend/src/components/Layout/TrainingDS/DatasetNew/DatasetNew.js b/frontend/src/components/Layout/TrainingDS/DatasetNew/DatasetNew.js index d914bb24..de4aeaf5 100644 --- a/frontend/src/components/Layout/TrainingDS/DatasetNew/DatasetNew.js +++ b/frontend/src/components/Layout/TrainingDS/DatasetNew/DatasetNew.js @@ -12,6 +12,7 @@ const DatasetNew = (props) => { const navigate = useNavigate(); const [oamURL, setOAMURL] = useState(); const { accessToken } = useContext(AuthContext); + const regUrl = /^(https?|chrome):\/\/[^\s$.?#].[^\s]*$/; const saveDataset = async () => { try { const body = { @@ -118,7 +119,13 @@ const DatasetNew = (props) => { console.log("save"); mutate(); }} - disabled={DSName.trim() === "" || !oamURL || isLoading} + disabled={ + !oamURL || + !regUrl.test(oamURL) || + DSName.trim() === "" || + !oamURL.endsWith("{z}/{x}/{y}") || + isLoading + } > Create Training Dataset From 1601827b3a5d0434e5a4b91b2d2c9f324739eeba Mon Sep 17 00:00:00 2001 From: kshitijrajsharma Date: Sun, 23 Jul 2023 20:57:21 +0545 Subject: [PATCH 11/18] added revised docker compose and env samples --- backend/docker-compose.yml | 71 ------------------------------- backend/docker_sample_env | 18 ++++---- docker-compose.yml | 82 ++++++++++++++++++++++++++++++++++++ frontend/.env sample | 2 +- frontend/Dockerfile.frontend | 15 +++++++ 5 files changed, 108 insertions(+), 80 deletions(-) delete mode 100644 backend/docker-compose.yml create mode 100644 docker-compose.yml create mode 100644 frontend/Dockerfile.frontend diff --git a/backend/docker-compose.yml b/backend/docker-compose.yml deleted file mode 100644 index 34f254a4..00000000 --- a/backend/docker-compose.yml +++ /dev/null @@ -1,71 +0,0 @@ -version: '3.8' - -services: - - postgres: - restart: always - image: postgis/postgis - container_name: pgsql - environment: - - POSTGRES_DB=ai - - POSTGRES_USER=postgres - - POSTGRES_PASSWORD=admin - ports: - - '5434:5432' - volumes: - - ./postgres-data:/var/lib/postgresql/data - - redis: - image: redis - container_name: redis - ports: - - "6379:6379" - - app: - build: . - container_name: api - command: python manage.py runserver 0.0.0.0:8000 - deploy: - resources: - reservations: - devices: - - driver: nvidia - capabilities: [gpu] - ports: - - 8000:8000 - volumes: - - .:/app - # - C:/Users/ommrn/Documents/HOT/fAIr-utilities:/RAMP_HOME - depends_on: - - redis - - postgres - - worker: - build: . - container_name: worker - command: celery -A aiproject worker --loglevel=INFO - deploy: - resources: - reservations: - devices: - - driver: nvidia - capabilities: [gpu] - volumes: - - .:/app - # - C:/Users/ommrn/Documents/HOT/fAIr-utilities:/RAMP_HOME - depends_on: - - app - - redis - - postgres - - - worker-dashboard: - build: . - container_name: flower - command: celery --broker=redis://redis:6379// flower --address=0.0.0.0 --port=5000 - ports: - - 5500:5000 - depends_on: - - app - - redis - - worker \ No newline at end of file diff --git a/backend/docker_sample_env b/backend/docker_sample_env index 21bf789c..c57331a2 100644 --- a/backend/docker_sample_env +++ b/backend/docker_sample_env @@ -1,17 +1,19 @@ +DEBUG=True SECRET_KEY=yl2w)c0boi_ma-1v5)935^2#&m*r!1s9z9^*9e5co^08_ixzo6 DATABASE_URL=postgis://postgres:admin@pgsql:5432/ai -EXPORT_TOOL_API_URL=http://44.203.33.53:8000/raw-data/current-snapshot/ -CORS_ALLOWED_ORIGINS=http://localhost:3000 -GDAL_LIBRARY_PATH='' -MAXAR_CONNECT_ID= +EXPORT_TOOL_API_URL=https://galaxy-api.hotosm.org/v1/raw-data/current-snapshot/ +CORS_ALLOWED_ORIGINS=http://127.0.0.1:3000 OSM_CLIENT_ID= OSM_CLIENT_SECRET= OSM_URL=https://www.openstreetmap.org OSM_SCOPE=read_prefs -OSM_LOGIN_REDIRECT_URI=http://127.0.0.1:8000/api/v1/auth/callback/ +OSM_LOGIN_REDIRECT_URI=http://127.0.0.1:3000/authenticate/ OSM_SECRET_KEY= CELERY_BROKER_URL="redis://redis:6379/0" CELERY_RESULT_BACKEND="redis://redis:6379/0" -RAMP_HOME="/home/kshitij/hotosm/fAIr-utilities" -TRAINING_WORKSPACE="/home/kshitij/hotosm/fAIr/backend/training" -TESTING_TOKEN= \ No newline at end of file +RAMP_HOME="/RAMP_HOME" +TRAINING_WORKSPACE="/TRAINING_WORKSPACE" + +TESTING_TOKEN= +GDAL_LIBRARY_PATH='' +MAXAR_CONNECT_ID= \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 00000000..0080def3 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,82 @@ +version: '3.8' + +services: + postgres: + restart: always + image: postgis/postgis + container_name: pgsql + environment: + - POSTGRES_DB=ai + - POSTGRES_USER=postgres + - POSTGRES_PASSWORD=admin + ports: + - '5434:5432' + + redis: + image: redis + container_name: redis + ports: + - "6379:6379" + + backend-api: + build: + context: ./backend + dockerfile: Dockerfile + container_name: api + command: python manage.py runserver 0.0.0.0:8000 + deploy: + resources: + reservations: + devices: + - driver: nvidia + capabilities: [gpu] + ports: + - 8000:8000 + volumes: + - ${RAMP_HOME}:/RAMP_HOME + - ${TRAINING_WORKSPACE}:/TRAINING_WORKSPACE + depends_on: + - redis + - postgres + + backend-worker: + build: + context: ./backend + dockerfile: Dockerfile + container_name: worker + command: celery -A aiproject worker --loglevel=INFO + deploy: + resources: + reservations: + devices: + - driver: nvidia + capabilities: [gpu] + volumes: + - ${RAMP_HOME}:/RAMP_HOME + - ${TRAINING_WORKSPACE}:/TRAINING_WORKSPACE + depends_on: + - backend-api + - redis + - postgres + + worker-dashboard: + image: mher/flower + container_name: flower + command: celery --broker=redis://redis:6379// flower --address=0.0.0.0 --port=5000 + ports: + - 5500:5000 + depends_on: + - backend-api + - redis + - backend-worker + + frontend: + build: + context: ./frontend + dockerfile: Dockerfile.frontend + container_name: frontend + command: npm start -- --host 0.0.0.0 --port 3000 + ports: + - 3000:3000 + depends_on: + - backend-api \ No newline at end of file diff --git a/frontend/.env sample b/frontend/.env sample index 5964a03f..c0447728 100644 --- a/frontend/.env sample +++ b/frontend/.env sample @@ -1,4 +1,4 @@ REACT_APP_CONNECT_ID= REACT_APP_TM_API=https://tasking-manager-tm4-production-api.hotosm.org/api/v2/projects/PROJECT_ID/tasks/ REACT_APP_ENV=Dev -REACT_APP_API_BASE=http://localhost:8000/api/v1 \ No newline at end of file +REACT_APP_API_BASE=http://127.0.0.1:8000/api/v1 \ No newline at end of file diff --git a/frontend/Dockerfile.frontend b/frontend/Dockerfile.frontend new file mode 100644 index 00000000..c852d1e4 --- /dev/null +++ b/frontend/Dockerfile.frontend @@ -0,0 +1,15 @@ + +FROM node:16.14.2 + +WORKDIR /app + + +COPY . /app + + +RUN npm install --legacy-peer-deps + + +# RUN npm run build + +# EXPOSE 3000 From 98c74ebcd1cf8f196d83496a57806d56d4e99291 Mon Sep 17 00:00:00 2001 From: kshitijrajsharma Date: Mon, 24 Jul 2023 14:19:16 +0545 Subject: [PATCH 12/18] Added installation docs --- Readme.md | 5 ++ docs/Docker-installation.md | 124 ++++++++++++++++++++++++++ frontend/{.env sample => .env_sample} | 0 3 files changed, 129 insertions(+) create mode 100644 docs/Docker-installation.md rename frontend/{.env sample => .env_sample} (100%) diff --git a/Readme.md b/Readme.md index a8c6db0c..b47904f7 100644 --- a/Readme.md +++ b/Readme.md @@ -25,3 +25,8 @@ To eliminate model biases, fAIr is built to work with the local communities and See below a suggested product roadmap [subject to change] that provides high-level overview for planned work. ![image](https://user-images.githubusercontent.com/98902727/218769416-b3c71d3b-7c20-4d40-ab1e-88442d06445d.png) + +## Local Installation [DEV] + +Checkout Docker Installation [docs](./docs/Docker-installation.md)## Local Installation [DEV] + diff --git a/docs/Docker-installation.md b/docs/Docker-installation.md new file mode 100644 index 00000000..e90ac797 --- /dev/null +++ b/docs/Docker-installation.md @@ -0,0 +1,124 @@ +Docker Compose is created with redis , worker , postgis database , api and frontend all in one making it easy for development . For production it is not recommended + +## [DEV] Installation With Docker + +1. Clone Repo + + ``` + git clone https://github.com/hotosm/fAIr.git + ``` + +2. Get Docker Compose Installed + + If docker is not installed , Install it from [here](https://docs.docker.com/engine/install/) + ``` + docker-compose --version + ``` + +3. Check your Graphics + + fAIr works best with graphics card. It is highly recommended to use graphics card . It might not work with CPU only . Nvidia Graphics cards are tested + + You need to make sure you can see your graphics card details and can be accessed through docker by installing necessary drivers + + By following command you can see your graphics and graphics driver details + ``` + nvidia-smi + ``` + +4. Clonse Base Model and Create RAMP_HOME + + - Create a new folder called RAMP , outside fAIr + + ``` + mkdir ramp + ``` + - Download BaseModel Checkpoint from [here](https://drive.google.com/file/d/1wvJhkiOrSlHmmvJ0avkAdu9sslFf5_I0/view?usp=sharing) + + ``` + pip install gdown + gdown --fuzzy https://drive.google.com/file/d/1wvJhkiOrSlHmmvJ0avkAdu9sslFf5_I0/view?usp=sharing + ``` + - Clone Ramp Code + + ``` + git clone https://github.com/kshitijrajsharma/ramp-code-fAIr.git ramp-code + ``` + - Unzip downloaded basemodel and move inside ramp-code/ramp + + ``` + unzip checkpoint.tf.zip -d ramp-code/ramp + ``` + - Export Env variable for RAMP_HOME + Grab the file path of folder we created earlier ```ramp``` and export it as env variable + ``` + export RAMP_HOME=/home/YOUR_RAMP_LOCATION + ``` + eg : export RAMP_HOME=/home/kshitij/ramp + +5. Register your Local setup to OSM + + - Go to [OpenStreetMap](https://www.openstreetmap.org/) , Login/Create Account + - Click on your Profile and Hit ```My Settings``` + - Navigate to ```Oauth2 Applications``` + - Register new application + - Check permissions for ```Read user preferences``` and Redirect URI to be ```http://127.0.0.1:3000/authenticate/``` , Give it name as ```fAIr Dev Local``` + - You will get ```OSM_CLIENT_ID``` , ```OSM_CLIENT_SECRET``` Copy them + +6. Create Env variables + - Create a file ```.env``` in backend with [docker_sample_env](../backend/docker_sample_env) content + ``` + cp docker_sample_env .env + ``` + - Fill out the details of ```OSM_CLIENT_ID``` &```OSM_CLIENT_SECRET``` in .env file and generate a unique key & paste it to ```OSM_SECRET_KEY``` (It can be random for dev setup) + + Leave rest of the items as it is unless you know what you are doing + - Export ```TRAINING_WORKSPACE``` Env + Training workspace is the folder where fAIr will store its training files + for eg : + ``` + export TRAINING_WORKSPACE=/home/kshitij/hotosm/fAIr/trainings + ``` + - Create ```.env``` in /frontend + ``` + cp .env_sample .env + ``` + You can leave it as it is for dev setup + +7. Build & Run containers + + ``` + docker compose build + ``` + + ``` + docker compose up + ``` + +8. Run Migrations + + See Running containers grab their ID and launch bash to make migrations (This is needed for the first time to set database) + + docker container ps + + Grab Container ID & Open Bash + + docker exec -it CONTAINER_ID bash + + + Once Bash is promoted hit following commands + + python manage.py makemigrations + python manage.py makemigrations login + python manage.py makemigrations login + python manage.py migrate + +9. Play and Develop + + Restart containers + + ``` + docker compose restart + ``` + + Frontend will be available on 5000 port , Backend will be on 8000 , Flower will be on 5500 \ No newline at end of file diff --git a/frontend/.env sample b/frontend/.env_sample similarity index 100% rename from frontend/.env sample rename to frontend/.env_sample From be743d7fa08645219ba671c4b72ea2fbf7fe022d Mon Sep 17 00:00:00 2001 From: kshitijrajsharma Date: Mon, 24 Jul 2023 14:20:41 +0545 Subject: [PATCH 13/18] install hot fair utilities latest --- backend/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/backend/requirements.txt b/backend/requirements.txt index b270b66f..6cae52f9 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -17,5 +17,5 @@ django_celery_results==2.4.0 flower==1.2.0 validators==0.20.0 gpxpy==1.5.0 -hot-fair-utilities==1.0.51 +hot-fair-utilities geojson2osm==0.0.1 \ No newline at end of file From 14958569110276cfad915e46984d4054115724ca Mon Sep 17 00:00:00 2001 From: kshitijrajsharma Date: Mon, 24 Jul 2023 18:16:35 +0545 Subject: [PATCH 14/18] Fixed typo and backend build --- .github/workflows/backend_build.yml | 5 ++--- docs/Docker-installation.md | 2 +- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/.github/workflows/backend_build.yml b/.github/workflows/backend_build.yml index 9744c901..66ac5098 100644 --- a/.github/workflows/backend_build.yml +++ b/.github/workflows/backend_build.yml @@ -51,9 +51,8 @@ jobs: - name: Install gdal run: | - sudo apt-get -y install gdal-bin libgdal-dev python3-gdal && sudo apt-get -y autoremove && sudo apt-get clean - pip install GDAL==$(gdal-config --version) --global-option=build_ext --global-option="-I/usr/include/gdal" - + sudo apt-get update && sudo apt-get -y install gdal-bin libgdal-dev python3-gdal && sudo apt-get -y autoremove && sudo apt-get clean + pip install GDAL==$(gdal-config --version) - name: Install ramp dependecies run: | cd ramp-code && cd colab && make install diff --git a/docs/Docker-installation.md b/docs/Docker-installation.md index e90ac797..239c945f 100644 --- a/docs/Docker-installation.md +++ b/docs/Docker-installation.md @@ -12,7 +12,7 @@ Docker Compose is created with redis , worker , postgis database , api and fron If docker is not installed , Install it from [here](https://docs.docker.com/engine/install/) ``` - docker-compose --version + docker compose version ``` 3. Check your Graphics From b108accb20ebb99933996cfcb15e084ff2de8ca3 Mon Sep 17 00:00:00 2001 From: kshitijrajsharma Date: Mon, 24 Jul 2023 18:55:52 +0545 Subject: [PATCH 15/18] Install latest version of hot fair utilties --- .github/workflows/backend_build.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/backend_build.yml b/.github/workflows/backend_build.yml index 66ac5098..6cdd8202 100644 --- a/.github/workflows/backend_build.yml +++ b/.github/workflows/backend_build.yml @@ -58,7 +58,7 @@ jobs: cd ramp-code && cd colab && make install - name: Install fair utilities - run: pip install hot-fair-utilities==1.0.41 + run: pip install hot-fair-utilities - name: Install Psycopg2 run: | From 8150430e40dc462063cb8e022273b094896d65c6 Mon Sep 17 00:00:00 2001 From: kshitijrajsharma Date: Mon, 24 Jul 2023 19:12:42 +0545 Subject: [PATCH 16/18] Install tensorflow before hot fair utilties --- .github/workflows/backend_build.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/backend_build.yml b/.github/workflows/backend_build.yml index 6cdd8202..ff506399 100644 --- a/.github/workflows/backend_build.yml +++ b/.github/workflows/backend_build.yml @@ -57,6 +57,9 @@ jobs: run: | cd ramp-code && cd colab && make install + - name: Install tensorflow + run: pip install tensorflow==2.9.2 + - name: Install fair utilities run: pip install hot-fair-utilities From 35ef1e2fb1223bb6789480dd8660de9e3af7beff Mon Sep 17 00:00:00 2001 From: kshitijrajsharma Date: Thu, 27 Jul 2023 12:12:58 +0545 Subject: [PATCH 17/18] Added fix for Docker file to see logs and installation doc --- Readme.md | 3 +-- backend/Dockerfile | 2 +- docker-compose.yml | 20 ++++++++++++-------- docs/Docker-installation.md | 16 +++++++++------- 4 files changed, 23 insertions(+), 18 deletions(-) diff --git a/Readme.md b/Readme.md index b47904f7..9a557b1c 100644 --- a/Readme.md +++ b/Readme.md @@ -28,5 +28,4 @@ See below a suggested product roadmap [subject to change] that provides high-lev ## Local Installation [DEV] -Checkout Docker Installation [docs](./docs/Docker-installation.md)## Local Installation [DEV] - +Checkout Docker Installation [docs](./docs/Docker-installation.md) diff --git a/backend/Dockerfile b/backend/Dockerfile index c8f1cbbb..b7e40872 100644 --- a/backend/Dockerfile +++ b/backend/Dockerfile @@ -28,6 +28,6 @@ RUN pip install setuptools --upgrade COPY requirements.txt requirements.txt RUN pip install -r requirements.txt -RUN mkdir /app + WORKDIR /app COPY . /app \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml index 0080def3..65817a0a 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,4 +1,4 @@ -version: '3.8' +version: "3.8" services: postgres: @@ -10,8 +10,8 @@ services: - POSTGRES_USER=postgres - POSTGRES_PASSWORD=admin ports: - - '5434:5432' - + - "5434:5432" + redis: image: redis container_name: redis @@ -20,7 +20,7 @@ services: backend-api: build: - context: ./backend + context: ./backend dockerfile: Dockerfile container_name: api command: python manage.py runserver 0.0.0.0:8000 @@ -33,6 +33,7 @@ services: ports: - 8000:8000 volumes: + - ./backend:/app - ${RAMP_HOME}:/RAMP_HOME - ${TRAINING_WORKSPACE}:/TRAINING_WORKSPACE depends_on: @@ -41,7 +42,7 @@ services: backend-worker: build: - context: ./backend + context: ./backend dockerfile: Dockerfile container_name: worker command: celery -A aiproject worker --loglevel=INFO @@ -52,6 +53,7 @@ services: - driver: nvidia capabilities: [gpu] volumes: + - ./backend:/app - ${RAMP_HOME}:/RAMP_HOME - ${TRAINING_WORKSPACE}:/TRAINING_WORKSPACE depends_on: @@ -72,11 +74,13 @@ services: frontend: build: - context: ./frontend + context: ./frontend dockerfile: Dockerfile.frontend container_name: frontend command: npm start -- --host 0.0.0.0 --port 3000 ports: - - 3000:3000 + - 3000:3000 depends_on: - - backend-api \ No newline at end of file + - backend-api + volumes: + - ./frontend:/app diff --git a/docs/Docker-installation.md b/docs/Docker-installation.md index 239c945f..7fdd8d01 100644 --- a/docs/Docker-installation.md +++ b/docs/Docker-installation.md @@ -56,6 +56,13 @@ Docker Compose is created with redis , worker , postgis database , api and fron ``` eg : export RAMP_HOME=/home/kshitij/ramp + - Export ```TRAINING_WORKSPACE``` Env + Training workspace is the folder where fAIr will store its training files + for eg : + ``` + export TRAINING_WORKSPACE=/home/kshitij/hotosm/fAIr/trainings + ``` + 5. Register your Local setup to OSM - Go to [OpenStreetMap](https://www.openstreetmap.org/) , Login/Create Account @@ -73,12 +80,7 @@ Docker Compose is created with redis , worker , postgis database , api and fron - Fill out the details of ```OSM_CLIENT_ID``` &```OSM_CLIENT_SECRET``` in .env file and generate a unique key & paste it to ```OSM_SECRET_KEY``` (It can be random for dev setup) Leave rest of the items as it is unless you know what you are doing - - Export ```TRAINING_WORKSPACE``` Env - Training workspace is the folder where fAIr will store its training files - for eg : - ``` - export TRAINING_WORKSPACE=/home/kshitij/hotosm/fAIr/trainings - ``` + - Create ```.env``` in /frontend ``` cp .env_sample .env @@ -110,7 +112,7 @@ Docker Compose is created with redis , worker , postgis database , api and fron python manage.py makemigrations python manage.py makemigrations login - python manage.py makemigrations login + python manage.py makemigrations core python manage.py migrate 9. Play and Develop From 05315f1a7a5da6c817b83600273819d474f736cb Mon Sep 17 00:00:00 2001 From: Kshitij Raj Sharma <36752999+kshitijrajsharma@users.noreply.github.com> Date: Thu, 27 Jul 2023 12:36:10 +0545 Subject: [PATCH 18/18] Update docker-compose.yml --- docker-compose.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/docker-compose.yml b/docker-compose.yml index 65817a0a..263647eb 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -82,5 +82,3 @@ services: - 3000:3000 depends_on: - backend-api - volumes: - - ./frontend:/app