Skip to content

Commit

Permalink
Staticbuild v1.3.0 update (#690)
Browse files Browse the repository at this point in the history
* Harden images and reorg

* fix timezone with package (#680)

* Joyride tutorials update (#679)

* Merged in latest v1.2.1

* Updates to the joyride tutorials and some minor bugfixes.

* Revert Markdown update to fix current jest testing error

* Updated React Markdown library and fixed the jest test errors.

* Some modifications to remove the build and linting errors that came up after the merge with latest branch.

* Load frontend config before rendering App

Stop tests from leaking calls to globus.org

* Some fixes to get the react front-end to build and get the postgres to start up. There are still errors with django session store.

* Fixed some linting errors so frontend should build correctly.

* Updated the local config file used for local testing. The new file .local_config, contains the configuration used when running Metagrid locally. Updated the manage_metagrid script to run Metagrid from the single docker-compose file. Updated some settings to use llnl instead of ornl for local development and testing. Removed some unused environment variable files after after testing that they aren't used. Left the default variables in place because Metagrid fails to build when several of the defaults aren't set, still working to troubeshoot why. Globus transfer steps in the front-end are working, however transfers seem to fail and need to be resolved.

* Switch from CRA to Vite and clean up tests (#692)

* Switch from CRA to Vite and clean up tests

* Fixed issue with loading screen continuing even after cancel button click. Updated local test config. Globus transfers still not working.

---------

Co-authored-by: Zach Price <pricezt@ornl.gov>
Co-authored-by: downiec <42552189+downiec@users.noreply.github.com>

* Staticbuild v1.3 fixes (#697)

* Fix proxy and redirects for globus auth on docker compose

* Various lint and test fixes

* Update yarn.lock

* Some changes made to try and get globus transfers working.

* Update docs to match new config (#707)

* Updated the local config file used for local testing. The new file .local_config, contains the configuration used when running Metagrid locally. Updated the manage_metagrid script to run Metagrid from the single docker-compose file. Updated some settings to use llnl instead of ornl for local development and testing. Removed some unused environment variable files after after testing that they aren't used. Left the default variables in place because Metagrid fails to build when several of the defaults aren't set, still working to troubeshoot why. Globus transfer steps in the front-end are working, however transfers seem to fail and need to be resolved.

* Switch from CRA to Vite and clean up tests (#692)

* Switch from CRA to Vite and clean up tests

* Fixed issue with loading screen continuing even after cancel button click. Updated local test config. Globus transfers still not working.

---------

Co-authored-by: Zach Price <pricezt@ornl.gov>
Co-authored-by: downiec <42552189+downiec@users.noreply.github.com>

* Fix proxy and redirects for globus auth on docker compose

* Various lint and test fixes

* Update yarn.lock

* Some changes made to try and get globus transfers working.

* Update docs to match new config
Auto generate configuration documentation with pre-commit hooks

* Revert yarn.lock, fix more tests

* Includes some updates and minor fixes so that everything builds and runs correctly. Removed the GLOBUS_REDIRECT setting and did some cleanup of the manage_metagrid.sh functions.

* Updated the test files to no longer use the act() functions, as they are deprecated. Cleaned up and fixed some failing tests.

* Fix esgsearch json tests

* Minor test fixes to see if tests will pass on github (they pass locally).

---------

Co-authored-by: downiec <42552189+downiec@users.noreply.github.com>
Co-authored-by: Zach Price <pricezt@ornl.gov>

---------

Co-authored-by: Zach Price <pricezt@ornl.gov>
Co-authored-by: bstrdsmkr <bstrdsmkr@gmail.com>

---------

Co-authored-by: Zach Price <pricezt@ornl.gov>
Co-authored-by: Sasha Ames <sashakames@users.noreply.github.com>
Co-authored-by: bstrdsmkr <bstrdsmkr@gmail.com>
  • Loading branch information
4 people authored Jan 16, 2025
1 parent 931420d commit e6b309f
Show file tree
Hide file tree
Showing 261 changed files with 8,927 additions and 37,264 deletions.
27 changes: 11 additions & 16 deletions .github/workflows/backend.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,22 +10,17 @@ defaults:
working-directory: backend

env:
DJANGO_SETTINGS_MODULE: metagrid.config.local
DOMAIN_NAME: http://localhost:8000
DJANGO_SECURE_SSL_REDIRECT: False
CORS_ORIGIN_WHITELIST: http://localhost:3000
KEYCLOAK_URL: https://esgf-login.ceda.ac.uk/
KEYCLOAK_REALM: esgf
KEYCLOAK_CLIENT_ID: metagrid-localhost
DATABASE_URL: pgsql://postgres:postgres@localhost:5432/postgres
REACT_APP_SEARCH_URL: https://esgf-node.llnl.gov/esg-search/search
REACT_APP_WGET_API_URL: https://esgf-node.llnl.gov/esg-search/wget
REACT_APP_ESGF_NODE_STATUS_URL: https://aims2.llnl.gov/metagrid-backend/proxy/status
REACT_APP_ESGF_SOLR_URL: https://esgf-fedtest.llnl.gov/solr
DJANGO_LOGIN_REDIRECT_URL: http://localhost:3000/search
DJANGO_LOGOUT_REDIRECT_URL: http://localhost:3000/search
GLOBUS_CLIENT_KEY: ${{ secrets.GLOBUS_KEY }}
GLOBUS_CLIENT_SECRET: ${{ secrets.GLOBUS_SECRET }}
DJANGO_SETTINGS_MODULE: config.settings
PGHOST: localhost
PGPASSWORD: postgres
PGUSER: postgres
METAGRID_SEARCH_URL: https://esgf-node.llnl.gov/esg-search/search
METAGRID_WGET_URL: https://esgf-node.llnl.gov/esg-search/wget
METAGRID_STATUS_URL: https://esgf-node.llnl.gov/proxy/status
METAGRID_SOLR_URL: https://esgf-node.llnl.gov/esg-search
METAGRID_SOCIAL_AUTH_GLOBUS_KEY: ${{ secrets.GLOBUS_KEY }}
METAGRID_SOCIAL_AUTH_GLOBUS_SECRET: ${{ secrets.GLOBUS_SECRET }}
METAGRID_GLOBUS_CLIENT_ID: 21982de0-2b7a-4ba8-bef5-5606ae098201

jobs:
build:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/containers.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ jobs:
cache-from: type=gha
cache-to: type=gha,mode=max
context: frontend/
file: frontend/docker/production/react/Dockerfile
file: frontend/Dockerfile
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.metadata.outputs.tags }}
labels: ${{ steps.metadata.outputs.labels }}
Expand Down Expand Up @@ -66,7 +66,7 @@ jobs:
cache-from: type=gha
cache-to: type=gha,mode=max
context: backend/
file: backend/docker/production/django/Dockerfile
file: backend/Dockerfile
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.metadata.outputs.tags }}
labels: ${{ steps.metadata.outputs.labels }}
12 changes: 1 addition & 11 deletions .github/workflows/frontend.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,6 @@ defaults:
run:
working-directory: frontend

env:
REACT_APP_KEYCLOAK_REALM: esgf
REACT_APP_KEYCLOAK_URL: https://esgf-login.ceda.ac.uk/
REACT_APP_KEYCLOAK_CLIENT_ID: metagrid-localhost

jobs:
build:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -46,13 +41,8 @@ jobs:
- name: Run Tests
env:
RELEASE: dev
ENV_FILE: .envs/.react
HTML_PATH: public
run: |
# Replaces react-scripts substitution during build for index.html and generates runtime_env.js
docker/production/react/entrypoint
yarn test:coverage
run: yarn test:coverage

- name: Upload Coverage Report
uses: codecov/codecov-action@v3
Expand Down
10 changes: 4 additions & 6 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ metagrid_configs/backups
# .nfs files are created when an open file is removed but is still being accessed
.nfs*


### VisualStudioCode template
**/.vscode/*

Expand Down Expand Up @@ -80,8 +79,6 @@ crashlytics.properties
crashlytics-build.properties
fabric.properties



### Windows template
# Windows thumbnail cache files
Thumbs.db
Expand All @@ -106,7 +103,6 @@ $RECYCLE.BIN/
# Windows shortcuts
*.lnk


### macOS template
# General
*.DS_Store
Expand Down Expand Up @@ -135,7 +131,6 @@ Network Trash Folder
Temporary Items
.apdisk


### SublimeText template
# Cache files for Sublime Text
*.tmlanguage.cache
Expand Down Expand Up @@ -168,7 +163,6 @@ bh_unicode_properties.cache
# https://packagecontrol.io/packages/sublime-github
GitHub.sublime-settings


### Vim template
# Swap
[._]*.s[a-v][a-z]
Expand All @@ -189,4 +183,8 @@ tags
.env
.envs/*
!.envs/.local/
docker-compose.*-overlay.yml

# Documentation artifacts
docs/site
frontend/public/runtime_env.js
109 changes: 73 additions & 36 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,44 +1,81 @@
exclude: "docs|node_modules|migrations|.git|.tox"
default_stages: [commit]
default_stages: [ commit ]
fail_fast: true

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml

# Back-end
# ------------------------------------------------------------------------------
- repo: https://github.com/pycqa/flake8
rev: 7.1.1
hooks:
- id: flake8
args: ["--config=backend/setup.cfg"]
additional_dependencies: [flake8-isort]
# Back-end
# ------------------------------------------------------------------------------
- repo: https://github.com/pycqa/flake8
rev: 7.1.1
hooks:
- id: flake8
args: [ "--config=backend/setup.cfg" ]
additional_dependencies: [ flake8-isort ]

- repo: https://github.com/psf/black
rev: 24.10.0
hooks:
- id: black
args: ["--config=backend/pyproject.toml"]
- repo: https://github.com/psf/black
rev: 24.10.0
hooks:
- id: black
args: [ "--config=backend/pyproject.toml" ]

- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.12.0
hooks:
- id: mypy
additional_dependencies: ["types-requests"]
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.12.0
hooks:
- id: mypy
additional_dependencies: [ "types-requests" ]

# Front-end
# ------------------------------------------------------------------------------
# Runs the local node `lint` script to avoid dependencies being out of sync with `frontend/package.json`
# https://github.com/pre-commit/pre-commit/issues/945
- repo: local
hooks:
- id: eslint_and_tsc
name: eslint_and_tsc
language: system
files: .+(js|jsx|ts|tsx|json)$
entry: bash -c 'cd frontend && yarn precommit'
- repo: https://github.com/radeklat/settings-doc
rev: '4.3.1'
hooks:
- id: settings-doc-markdown
name: Generate backend settings markdown
language: system
entry: env PYTHONPATH=$PYTHONPATH:backend settings-doc generate --output-format markdown
args:
- '--class'
- 'config.settings.site_specific.MetagridBackendSettings'
- '--update'
- 'docs/docs/users/configurable_environment_variables.md'
- '--between'
- '<!-- start generated backend settings markdown -->'
- '<!-- end generated backend settings markdown -->'
- '--templates'
- 'docs/custom_templates'
- '--heading-offset'
- '3'

- id: settings-doc-markdown
name: Generate frontend settings markdown
language: system
entry: env PYTHONPATH=$PYTHONPATH:backend settings-doc generate --output-format markdown
args:
- '--class'
- 'config.settings.site_specific.MetagridFrontendSettings'
- '--update'
- 'docs/docs/users/configurable_environment_variables.md'
- '--between'
- '<!-- start generated frontend settings markdown -->'
- '<!-- end generated frontend settings markdown -->'
- '--templates'
- 'docs/custom_templates'
- '--heading-offset'
- '3'

# Front-end
# ------------------------------------------------------------------------------
# Runs the local node `lint` script to avoid dependencies being out of sync with `frontend/package.json`
# https://github.com/pre-commit/pre-commit/issues/945
- repo: local
hooks:
- id: eslint_and_tsc
name: eslint_and_tsc
language: system
files: .+(js|jsx|ts|tsx|json)$
entry: bash -c 'cd frontend && yarn precommit'
13 changes: 11 additions & 2 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,14 @@
"eslint.workingDirectories": ["./frontend/"],
// Jest
// -----------------------------
"jest.pathToJest": "yarn test:watch",
"jest.rootPath": "./frontend/src",
"jest.jestCommandLine": "yarn jest",
"jest.monitorLongRun": "off",
"jest.enable": true,
"jest.runMode": {
"type": "on-demand",
"coverage": true
},
// HTML, CSS, JSON
// -----------------------------
"[html]": {
Expand All @@ -77,5 +84,7 @@
"[jsonc]": {
"editor.defaultFormatter": "esbenp.prettier-vscode"
},
"python.analysis.extraPaths": ["backend/venv/bin/python"]
"python.analysis.extraPaths": ["backend/venv/bin/python"],
"python.testing.pytestEnabled": true,
"python.testing.pytestArgs": ["backend"]
}
42 changes: 0 additions & 42 deletions backend/.envs/.django

This file was deleted.

14 changes: 0 additions & 14 deletions backend/.envs/.keycloak

This file was deleted.

5 changes: 0 additions & 5 deletions backend/.envs/.postgres

This file was deleted.

24 changes: 8 additions & 16 deletions backend/docker/production/django/Dockerfile → backend/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,23 +17,15 @@ RUN addgroup --system django \
&& adduser --system --ingroup django django

# Requirements are installed here to ensure they will be cached.
COPY ./requirements /requirements
RUN pip3 install --no-cache-dir -r /requirements/production.txt \
&& rm -rf /requirements
COPY requirements /requirements
RUN pip3 install --no-cache-dir -r /requirements/local.txt

COPY ./docker/production/django/entrypoint /entrypoint
RUN sed -i 's/\r$//g' /entrypoint
RUN chmod +x /entrypoint
RUN chown django /entrypoint
COPY . /app

COPY ./docker/production/django/start /start
RUN sed -i 's/\r$//g' /start
RUN chmod +x /start
RUN chown django /start
COPY --chown=django:django . /app

USER django
RUN mkdir -p /app/staticfiles/.well-known \
&& cyclonedx-py requirements requirements/base.txt --output-format json --outfile /app/staticfiles/.well-known/bom

WORKDIR /app

ENTRYPOINT ["/entrypoint"]
USER django
EXPOSE 5000
CMD ["/usr/local/bin/gunicorn", "-c", "gunicorn_conf.py", "config.wsgi", "--chdir=/app"]
24 changes: 24 additions & 0 deletions backend/config/settings/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
from typing import Any, Iterable

from .site_specific import MetagridBackendSettings, MetagridFrontendSettings
from .static import DjangoStaticSettings


def __dir__() -> Iterable[str]:
"""The list of available options are retrieved from
the dict view of our DjangoSettings object.
"""
return (
DjangoStaticSettings.model_fields.keys()
| MetagridBackendSettings.model_fields.keys()
| MetagridFrontendSettings.model_fields.keys()
)


def __getattr__(name: str) -> Any:
"""Turn the module access into a DjangoSettings access"""
_combined_settings = DjangoStaticSettings().model_dump()
_combined_settings |= MetagridBackendSettings().model_dump()
_combined_settings |= MetagridFrontendSettings().model_dump()

return _combined_settings[name]
Loading

0 comments on commit e6b309f

Please sign in to comment.