Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fetcher DAG development #496

Merged
merged 39 commits into from
Aug 23, 2023
Merged

Fetcher DAG development #496

merged 39 commits into from
Aug 23, 2023

Commits on Aug 17, 2023

  1. breaking: Update env var FETCHER_DATA_DEST to file://<path to rikolti… 

    … data>
    @amywieliczka
    amywieliczka committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    87a6318 View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2023

  1. breaking: Update env vars MAPPER_DATA_DEST, MAPPER_DATA_SRC 

    for local development use file://<path to rikolti data>
    @amywieliczka
    amywieliczka committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    1ab3f03 View commit details
    Browse the repository at this point in the history

  2. resolve import errors

    @amywieliczka
    amywieliczka committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    3b08c15 View commit details
    Browse the repository at this point in the history

  3. sample dags using taskflow or pythonoperator styles

    @amywieliczka
    amywieliczka committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    f26feeb View commit details
    Browse the repository at this point in the history

  4. starting a fetcher dag

    @amywieliczka
    amywieliczka committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    5a4ba2f View commit details
    Browse the repository at this point in the history

  5. sample of using airflow variables; writing to local disk

    @amywieliczka
    amywieliczka committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    cf3688b View commit details
    Browse the repository at this point in the history

  6. mkdir sample task

    @amywieliczka
    amywieliczka committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    1628157 View commit details
    Browse the repository at this point in the history

  7. sample of environment vars using startup.sh

    @amywieliczka
    amywieliczka committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    811b086 View commit details
    Browse the repository at this point in the history

  8. update FETCHER_DATA_DEST config for airflow

    @amywieliczka
    amywieliczka committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    679b5a1 View commit details
    Browse the repository at this point in the history

  9. remove startup.sh from version control

    @amywieliczka
    amywieliczka committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    9704750 View commit details
    Browse the repository at this point in the history

  10. make 26284 default collection for development ease

    @amywieliczka
    amywieliczka committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    9e47ab7 View commit details
    Browse the repository at this point in the history

  11. update README to reflect removal of the airflow folder

    @amywieliczka
    amywieliczka committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    17e7738 View commit details
    Browse the repository at this point in the history

  12. ignore startup.sh

    @amywieliczka
    amywieliczka committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    c055257 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2023

  1. fetcher settings requires python-dotenv

    @amywieliczka
    amywieliczka committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    420a873 View commit details
    Browse the repository at this point in the history

  2. dag imports should be based in rikolti

    @amywieliczka
    amywieliczka committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    50ba895 View commit details
    Browse the repository at this point in the history

  3. use the module's current package as import_module's package

    @amywieliczka
    amywieliczka committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    bd389f6 View commit details
    Browse the repository at this point in the history

  4. fetcher and mapper environment variable examples

    @amywieliczka
    amywieliczka committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    f3a8164 View commit details
    Browse the repository at this point in the history

  5. update usage instructions

    @amywieliczka
    amywieliczka committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    dc5d7f3 View commit details
    Browse the repository at this point in the history

  6. repair calling metadata_fetcher.lambda_function as __main__

    @amywieliczka
    amywieliczka committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    c38f9b8 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'data_dest' of github.com:ucldc/rikolti into dag-develop… 

    …ment
    @amywieliczka
    amywieliczka committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    64d8125 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2023

  1. added validator env variables

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    bbaa9ec View commit details
    Browse the repository at this point in the history

  2. Update README to describe env.example

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    1f7c3c2 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #495 from ucldc/data_dest 

    breaking: update local env variables
    @amywieliczka
    amywieliczka authored Aug 23, 2023
    Configuration menu
    Copy the full SHA
    7b84312 View commit details
    Browse the repository at this point in the history

  4. sample dags using taskflow or pythonoperator styles

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    d2d8932 View commit details
    Browse the repository at this point in the history

  5. starting a fetcher dag

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    78df21a View commit details
    Browse the repository at this point in the history

  6. sample of using airflow variables; writing to local disk

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    6cca3b9 View commit details
    Browse the repository at this point in the history

  7. mkdir sample task

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    d822370 View commit details
    Browse the repository at this point in the history

  8. sample of environment vars using startup.sh

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    26df0fe View commit details
    Browse the repository at this point in the history

  9. update FETCHER_DATA_DEST config for airflow

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    f29b2d8 View commit details
    Browse the repository at this point in the history

  10. remove startup.sh from version control

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    a8389bb View commit details
    Browse the repository at this point in the history

  11. make 26284 default collection for development ease

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    1257749 View commit details
    Browse the repository at this point in the history

  12. update README to reflect removal of the airflow folder

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    d6d4060 View commit details
    Browse the repository at this point in the history

  13. ignore startup.sh

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    c1d7a28 View commit details
    Browse the repository at this point in the history

  14. fetcher settings requires python-dotenv

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    806ccf2 View commit details
    Browse the repository at this point in the history

  15. dag imports should be based in rikolti

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    674d267 View commit details
    Browse the repository at this point in the history

  16. use the module's current package as import_module's package

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    9d2c004 View commit details
    Browse the repository at this point in the history

  17. Updated README with startup.sh, RIKOLTI_DATA_HOME, instructions

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    d795caa View commit details
    Browse the repository at this point in the history

  18. Merge branch 'dag-development' of github.com:ucldc/rikolti into dag-d… 

    …evelopment
    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    c43357e View commit details
    Browse the repository at this point in the history

  19. ruff-fix

    @amywieliczka
    amywieliczka committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    a1b1332 View commit details
    Browse the repository at this point in the history