From 9af814fdba2b4b17d568f8c5723c7aec0edd2fe4 Mon Sep 17 00:00:00 2001 From: teppialy <52909563+ttattl@users.noreply.github.com> Date: Sat, 26 Nov 2022 01:27:34 +0700 Subject: [PATCH] =?UTF-8?q?=E2=9C=A8Release=201.1=20`backend`=20(#224)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * 👮 add Python linter CI Signed-off-by: Pascal Marco Caversaccio * renovate.json for pip Signed-off-by: Pascal Marco Caversaccio * increase setup-python gh actions version Signed-off-by: Pascal Marco Caversaccio * Feat/refactor auth service sam app (#165) * Update:set dev env * Update: add role gateway * Update endpoint for dev environment * Update: Divide config prod and dev * Create prod and dev mode for sam app * update: refactor dev env * donwload reference image * Update: auth service sam app * Update: tempplate auth * fix: change datatype of template * Update: auth services sam app * Transport layyer for auth service * Update: apigateway for auth service * Add: forgotpassword lambda * Auth Transport layer: Update lambda invoke * Update: Google IdentityProvider * Update: change CognitoUserpoolIdentityProvider of google * Update: reafactor project services * Add: const file * Update: create project service discovery * Update: Api Gateway for project services * Update: Projects Service Transport layer * Add ses policy on auth service * delete unused comment and add try except * Add: emplate mail * pdate: Api gateway for template-invite-mail * Fix: typos * Update : auth service Co-authored-by: teppialy <52909563+ttattl@users.noreply.github.com> * refactor: + fix env dev + add requirements + remove unused code * refactor: remove old logic of ai_caller * refactor: remove key * Feat/refactor sam app (#169) * Refactor: webhoodbot and invitemail * update: login social Co-authored-by: teppialy <52909563+ttattl@users.noreply.github.com> * Feat/refactor sam app (#170) * Refactor: webhoodbot and invitemail * update: login social * Update Identity pool * Create: identity permission * Delete unused feature * Resolved merge conflict * Add Policy authenticated S3 Co-authored-by: teppialy <52909563+ttattl@users.noreply.github.com> * feat: list prebuild dataset * fix: named with stage param * Feat/refactor sam app (#171) * Refactor: webhoodbot and invitemail * update: login social * Update Identity pool * Create: identity permission * Delete unused feature * Resolved merge conflict * Add Policy authenticated S3 * refactor project * refactor api gw of project service and Add thumbnail feat * resoveled merge conflict Co-authored-by: teppialy <52909563+ttattl@users.noreply.github.com> * feat: init create project from prebuild dataset * fix: role name * fix: auth env variable (#174) * feat: update create sample project * Fix/refactor sam app (#175) * fix: auth env variable * Update: refactor database * fix: env paramter * fix: conflict * fix api gateway * refactor: update named convention * fix: mirror * Feat/thumbnail (#176) * update database project * Update output database * fix: minor Bugs * add eventbridge for thumbnail * Update: eventbridge thumbnail * fix: auth role identity pass value * Feat/thumbnail (#178) * update database project * Update output database * fix: minor Bugs * add eventbridge for thumbnail * Update: eventbridge thumbnail * fix: typos * Update: increase memory * fix: wrong named * intergation: data model (#179) * fix: wrong named * fix: (prebuild dataset) random images * fix: exception message return * fix: (prebuild dataset) check limit max project quantity * Feat/refactor/minior bug (#183) * cognito: update custom domain * feedback sam * change name custom domain * add thumbnail key * change domain * fix: (create prebuild project) parse Decimal field * fix: minor bug (#184) * fix: minor bug * fix minor bug * feat: init for annotation structure * feat: update confing and build script for both application * feat: (annotation)(project) clone project from daita to annotation * Merge branch 'feat/update_build_flow' * feat: add build instruction * feat: auto export config for FE * fix: data-flow subnet join, fix const db * update: move certificate arn to outside * fix: response github oauth2 (#190) * update certificate * Update (#191) * Feat/fix minor bug (#192) * Update * Update: parameter redirect url * fix: wrong assign upload update * Update recaptcha (#193) * update: flow ai caller use ec2 * update: enable thumnail code * fix: circle update sam template for thumnail * feat: api for annotation app * doc: add doc for url endpoint of annotation app * fix: (clone project) mirro bug field name * feat: add class of category, create default category and class when clone project * feat: (anno project info) add class list return in response body * feat: add API for add class * fix: return gen_status when get project info * feat: grant permission to annotation s3 bucket for cognito user * update: get label info * Feat/ecs segmentation (#201) * add template ecs task * feat: fix run ecs task with private subnet * Update: ecs-segmentation app * Update: ecs trigger * supdate: env variable * ecs app * change name sample image * update: ecs-segmentation * move ecs-app to annototation project * update : storage layer * add ec2 role * Update * Update * fix typos * delete unused file * update : connect efs to container * fix: typos * sqs trigger upload image * Update :ecs segmentation * devide batch on dynamodb Co-authored-by: teppialy * feat: integrate ecs segmentation to dev * delete project anno (#204) Co-authored-by: teppialy <52909563+ttattl@users.noreply.github.com> * feat: (annotation) delete project api * feat: (annotation) api upload data to project * fix: (annotation api) upload check, upload update * feat: check AI segementation progress * feat: update infra for ecs fix: hardcode in preprocessing * Add: info id_token (#216) * fix: add catch error on StartECSDecompressTask step (#213) Co-authored-by: dachanh * feat: init infra for ecs_ai_caller * fix: typo fix: decrease max ec2 of ecs from 4->2 * feat: add parameter table user from daita to annotation * feat: split config build dev and prod * Feature : Notify complete AI segmentation task (#217) * update * update * fix Co-authored-by: teppialy <52909563+ttattl@users.noreply.github.com> * feat: check limit number image when creatiing project from prebuild project * fix: hardcode table list ec2 const * fix mirror bugs * feat: move send email to core function * feat: remove hardcode for token slack feedback * feat: remove hardcode secret key * refactor: clean config, code for prod * refactor: move Image_AI_segmentation URL to main config * fix: add output config for FE s3 annotation * fix (#220) Co-authored-by: teppialy <52909563+ttattl@users.noreply.github.com> * update: config maximum character of project description * fix: grammar refactor: const MAX_NUM_IMAGES_IN_ORIGINAL * fix: duplicate config sam s3 for dev * fix: (update project info) mirror bugs * refactor: hardcode in config for captcha * fix api log out (#222) * fix * update Signed-off-by: teppialy <52909563+ttattl@users.noreply.github.com> Co-authored-by: teppialy <52909563+ttattl@users.noreply.github.com> * refactor: move to config max size of ec2 autoscalling created * fix: mirror bug * fix: mirror bug * fix (#223) * fix: mirror bug * fix: mirror bug * fix: link to reference image * refactor: remove hardcode ai caller calculate reference image * fix (#225) * Update (#226) Signed-off-by: Pascal Marco Caversaccio Signed-off-by: teppialy <52909563+ttattl@users.noreply.github.com> Co-authored-by: Pascal Marco Caversaccio Co-authored-by: Pascal Marco Caversaccio Co-authored-by: BEdaita <102169046+BEdaita@users.noreply.github.com> Co-authored-by: dachanh --- .github/workflows/lint.yml | 35 + .gitignore | 8 + .../AI_service/AI_service/__init__.py | 4 - .../AI_service/AI_service/asgi.py | 16 - .../AI_service/AI_service/settings.py | 129 - .../AI_service/AI_service/urls.py | 29 - .../AI_service/AI_service/worker.py | 8 - .../AI_service/AI_service/wsgi.py | 16 - .../AI_service/app/__init__.py | 1 - ai_caller_service_v1/AI_service/app/admin.py | 3 - ai_caller_service_v1/AI_service/app/apps.py | 6 - .../AI_service/app/migrations/0001_initial.py | 28 - ai_caller_service_v1/AI_service/app/models.py | 3 - .../AI_service/app/scripts/check_stop_ec2.py | 63 - .../AI_service/app/scripts/delete_all_keys.py | 6 - .../AI_service/app/scripts/dynamodb/images.py | 165 - .../app/scripts/dynamodb/methods.py | 27 - .../AI_service/app/scripts/dynamodb/task.py | 91 - .../AI_service/app/scripts/event_loop.py | 67 - .../AI_service/app/scripts/logging_api.py | 25 - .../AI_service/app/scripts/s3.py | 127 - .../AI_service/app/scripts/utils.py | 183 - ai_caller_service_v1/AI_service/app/tests.py | 1 - ai_caller_service_v1/AI_service/app/views.py | 383 -- .../AI_service/kill_worker.sh | 2 - ai_caller_service_v1/AI_service/manage.py | 22 - .../AI_service/requirements.txt | 38 - ai_caller_service_v1/AI_service/worker.sh | 9 - ai_caller_service_v1/build.sh | 6 - ai_caller_service_v1/deploy.sh | 3 - .../add-class/hdler_add_class.py | 59 + .../hdler_create_label_category.py | 63 + .../hdler_get_file_info_n_label.py | 68 + .../list-class/hdler_list_class.py | 57 + .../save-label/hdler_save_label.py | 59 + .../template_annotaion_service.yaml | 138 + .../api-service/api_route_define.yaml | 200 + .../api-service/template_api_service.yaml | 78 + annotation-app/build_annotation.sh | 84 + annotation-app/db-service/db_template.yaml | 194 + .../hdler_stream_data_annotation.py | 87 + .../hdler_task_segmentation.py | 65 + .../ecs-segment-app/controller.yaml | 226 + .../ecs-segment-app/ecs_segementation.yaml | 179 + annotation-app/ecs-segment-app/network.yaml | 252 + annotation-app/ecs-segment-app/role.yaml | 229 + .../statemachine/ecs_task.asl.yaml | 57 + .../functions/hdler_download_image_to_efs.py | 65 + .../hdler_send_email_to_identityid.py | 103 + .../functions/hdler_updoad_image.py | 128 + annotation-app/ecs-segment-app/storage.yaml | 78 + annotation-app/ecs-segment-app/template.yaml | 149 + .../hdler_check_si_segm_progress.py | 60 + .../clone-project/hdler_project_clone.py | 141 + .../delete-project/hdler_delete_project.py | 55 + .../hdler_get_project_info.py | 81 + .../list-data/hdler_list_data.py | 62 + .../list-project/hdler_list_project.py | 56 + .../upload-check/hdler_upload_check.py | 63 + .../upload-update/hdler_upload_update.py | 82 + .../functions/hdler_move_s3_data.py | 104 + .../functions/hdler_update_input_data.py | 106 + .../functions/hdler_update_sumary_db.py | 114 + .../sm_clone_project.asl.yaml | 49 + .../template_project_service.yaml | 274 + annotation-app/samconfig.toml | 19 + annotation-app/template_annotation_app.yaml | 257 + core_service/api_gateway/de_api_project.py | 3 +- core_service/aws_lambda/lambda_service.py | 3 +- .../project/code/credential_login.py | 8 +- .../project/code/forgot_password.py | 23 +- .../aws_lambda/project/code/invite_friend.py | 7 +- core_service/aws_lambda/project/code/login.py | 7 - .../aws_lambda/project/code/login_social.py | 2 +- .../project/code/presigned_url_s3.py | 72 +- .../aws_lambda/project/code/project_info.py | 6 +- .../project/code/slack_webhook_feedback.py | 137 +- .../aws_lambda/project/common/config.py | 30 +- .../aws_lambda/project/common/error.py | 6 +- core_service/aws_lambda/project/de_lambda.py | 5 - .../project/de_lambda_auth_funcs.py | 41 +- .../project/de_lambda_mail_funcs.py | 6 +- .../aws_lambda/project/de_lambda_s3_funcs.py | 3 +- .../aws_lambda/project/de_lambda_webhook.py | 33 +- .../project/packages/slack/__init__.py | 14 + .../project/packages/slack/deprecation.py | 14 + .../project/packages/slack/errors.py | 10 + .../project/packages/slack/py.typed | 0 .../project/packages/slack/rtm/__init__.py | 6 + .../project/packages/slack/rtm/client.py | 6 + .../packages/slack/signature/__init__.py | 5 + .../packages/slack/signature/verifier.py | 71 + .../project/packages/slack/version.py | 1 + .../project/packages/slack/web/__init__.py | 11 + .../packages/slack/web/async_base_client.py | 165 + .../packages/slack/web/async_client.py | 15 + .../slack/web/async_internal_utils.py | 199 + .../slack/web/async_slack_response.py | 187 + .../project/packages/slack/web/base_client.py | 497 ++ .../packages/slack/web/classes/__init__.py | 10 + .../packages/slack/web/classes/actions.py | 13 + .../packages/slack/web/classes/attachments.py | 9 + .../packages/slack/web/classes/blocks.py | 13 + .../slack/web/classes/dialog_elements.py | 14 + .../packages/slack/web/classes/dialogs.py | 5 + .../packages/slack/web/classes/elements.py | 27 + .../slack/web/classes/interactions.py | 137 + .../packages/slack/web/classes/messages.py | 1 + .../packages/slack/web/classes/objects.py | 19 + .../packages/slack/web/classes/views.py | 7 + .../project/packages/slack/web/client.py | 4 + .../project/packages/slack/web/deprecation.py | 30 + .../packages/slack/web/internal_utils.py | 52 + .../packages/slack/web/slack_response.py | 6 + .../packages/slack/webhook/__init__.py | 7 + .../packages/slack/webhook/async_client.py | 119 + .../project/packages/slack/webhook/client.py | 116 + .../packages/slack/webhook/internal_utils.py | 40 + .../slack/webhook/webhook_response.py | 13 + .../slack_sdk-3.17.2.dist-info/INSTALLER | 1 + .../slack_sdk-3.17.2.dist-info/LICENSE | 21 + .../slack_sdk-3.17.2.dist-info/METADATA | 350 ++ .../slack_sdk-3.17.2.dist-info/RECORD | 303 ++ .../slack_sdk-3.17.2.dist-info/REQUESTED | 0 .../packages/slack_sdk-3.17.2.dist-info/WHEEL | 6 + .../slack_sdk-3.17.2.dist-info/top_level.txt | 2 + .../project/packages/slack_sdk/__init__.py | 53 + .../slack_sdk/aiohttp_version_checker.py | 23 + .../packages/slack_sdk/audit_logs/__init__.py | 11 + .../slack_sdk/audit_logs/async_client.py | 5 + .../slack_sdk/audit_logs/v1/__init__.py | 4 + .../slack_sdk/audit_logs/v1/async_client.py | 356 ++ .../slack_sdk/audit_logs/v1/client.py | 362 ++ .../slack_sdk/audit_logs/v1/internal_utils.py | 40 + .../packages/slack_sdk/audit_logs/v1/logs.py | 640 +++ .../slack_sdk/audit_logs/v1/response.py | 34 + .../packages/slack_sdk/errors/__init__.py | 58 + .../packages/slack_sdk/http_retry/__init__.py | 48 + .../slack_sdk/http_retry/async_handler.py | 89 + .../http_retry/builtin_async_handlers.py | 90 + .../slack_sdk/http_retry/builtin_handlers.py | 89 + .../builtin_interval_calculators.py | 44 + .../packages/slack_sdk/http_retry/handler.py | 80 + .../http_retry/interval_calculator.py | 12 + .../packages/slack_sdk/http_retry/jitter.py | 24 + .../packages/slack_sdk/http_retry/request.py | 36 + .../packages/slack_sdk/http_retry/response.py | 23 + .../packages/slack_sdk/http_retry/state.py | 21 + .../packages/slack_sdk/models/__init__.py | 58 + .../slack_sdk/models/attachments/__init__.py | 588 +++ .../slack_sdk/models/basic_objects.py | 113 + .../slack_sdk/models/blocks/__init__.py | 95 + .../models/blocks/basic_components.py | 526 ++ .../slack_sdk/models/blocks/block_elements.py | 1482 ++++++ .../slack_sdk/models/blocks/blocks.py | 494 ++ .../packages/slack_sdk/models/dialoags.py | 29 + .../slack_sdk/models/dialogs/__init__.py | 921 ++++ .../slack_sdk/models/messages/__init__.py | 85 + .../slack_sdk/models/messages/message.py | 77 + .../slack_sdk/models/metadata/__init__.py | 30 + .../slack_sdk/models/views/__init__.py | 232 + .../packages/slack_sdk/oauth/__init__.py | 19 + .../oauth/authorize_url_generator/__init__.py | 63 + .../oauth/installation_store/__init__.py | 10 + .../installation_store/amazon_s3/__init__.py | 344 ++ .../async_cacheable_installation_store.py | 136 + .../async_installation_store.py | 92 + .../cacheable_installation_store.py | 136 + .../oauth/installation_store/file/__init__.py | 245 + .../installation_store/installation_store.py | 96 + .../oauth/installation_store/internals.py | 32 + .../installation_store/models/__init__.py | 7 + .../oauth/installation_store/models/bot.py | 126 + .../installation_store/models/installation.py | 217 + .../installation_store/sqlalchemy/__init__.py | 356 ++ .../installation_store/sqlite3/__init__.py | 611 +++ .../redirect_uri_page_renderer/__init__.py | 71 + .../slack_sdk/oauth/state_store/__init__.py | 12 + .../oauth/state_store/amazon_s3/__init__.py | 69 + .../oauth/state_store/async_state_store.py | 13 + .../oauth/state_store/file/__init__.py | 71 + .../oauth/state_store/sqlalchemy/__init__.py | 75 + .../oauth/state_store/sqlite3/__init__.py | 96 + .../oauth/state_store/state_store.py | 13 + .../slack_sdk/oauth/state_utils/__init__.py | 41 + .../oauth/token_rotation/__init__.py | 5 + .../oauth/token_rotation/async_rotator.py | 142 + .../slack_sdk/oauth/token_rotation/rotator.py | 135 + .../slack_sdk/proxy_env_variable_loader.py | 24 + .../project/packages/slack_sdk/py.typed | 0 .../packages/slack_sdk/rtm/__init__.py | 572 +++ .../packages/slack_sdk/rtm/v2/__init__.py | 5 + .../packages/slack_sdk/rtm_v2/__init__.py | 391 ++ .../packages/slack_sdk/scim/__init__.py | 23 + .../packages/slack_sdk/scim/async_client.py | 5 + .../packages/slack_sdk/scim/v1/__init__.py | 6 + .../slack_sdk/scim/v1/async_client.py | 398 ++ .../packages/slack_sdk/scim/v1/client.py | 418 ++ .../packages/slack_sdk/scim/v1/default_arg.py | 5 + .../packages/slack_sdk/scim/v1/group.py | 78 + .../slack_sdk/scim/v1/internal_utils.py | 149 + .../packages/slack_sdk/scim/v1/response.py | 263 + .../packages/slack_sdk/scim/v1/types.py | 27 + .../packages/slack_sdk/scim/v1/user.py | 214 + .../packages/slack_sdk/signature/__init__.py | 71 + .../slack_sdk/socket_mode/__init__.py | 11 + .../slack_sdk/socket_mode/aiohttp/__init__.py | 439 ++ .../slack_sdk/socket_mode/async_client.py | 168 + .../slack_sdk/socket_mode/async_listeners.py | 20 + .../slack_sdk/socket_mode/builtin/__init__.py | 5 + .../slack_sdk/socket_mode/builtin/client.py | 289 ++ .../socket_mode/builtin/connection.py | 452 ++ .../socket_mode/builtin/frame_header.py | 47 + .../socket_mode/builtin/internals.py | 403 ++ .../packages/slack_sdk/socket_mode/client.py | 157 + .../slack_sdk/socket_mode/interval_runner.py | 33 + .../slack_sdk/socket_mode/listeners.py | 17 + .../packages/slack_sdk/socket_mode/request.py | 57 + .../slack_sdk/socket_mode/response.py | 28 + .../socket_mode/websocket_client/__init__.py | 258 + .../socket_mode/websockets/__init__.py | 252 + .../project/packages/slack_sdk/version.py | 2 + .../packages/slack_sdk/web/__init__.py | 9 + .../slack_sdk/web/async_base_client.py | 216 + .../packages/slack_sdk/web/async_client.py | 4416 ++++++++++++++++ .../slack_sdk/web/async_internal_utils.py | 209 + .../slack_sdk/web/async_slack_response.py | 192 + .../packages/slack_sdk/web/base_client.py | 580 +++ .../project/packages/slack_sdk/web/client.py | 4407 ++++++++++++++++ .../packages/slack_sdk/web/deprecation.py | 30 + .../packages/slack_sdk/web/internal_utils.py | 301 ++ .../slack_sdk/web/legacy_base_client.py | 559 +++ .../packages/slack_sdk/web/legacy_client.py | 4418 +++++++++++++++++ .../slack_sdk/web/legacy_slack_response.py | 223 + .../packages/slack_sdk/web/slack_response.py | 189 + .../packages/slack_sdk/webhook/__init__.py | 11 + .../slack_sdk/webhook/async_client.py | 264 + .../packages/slack_sdk/webhook/client.py | 277 ++ .../slack_sdk/webhook/internal_utils.py | 45 + .../slack_sdk/webhook/webhook_response.py | 16 + core_service/deploy_core_service.py | 14 +- core_service/dynamoDB/de_dynamodb.py | 48 +- .../ai_caller_service_template.yaml | 338 +- .../handlers/crontab/consumer/app.py | 4 +- .../handlers/crontab/stop_ec2/app.py | 6 +- .../download_task/download_task/app.py | 87 +- .../get_reference_images/__init__.py | 0 .../download_task/get_reference_images/app.py | 86 + .../get_reference_images/requirements.txt | 1 + .../download_task/merge_download/app.py | 30 +- .../generate_step/CompleteRequestAI/app.py | 161 +- .../generate_step/HandleBatchToS3/app.py | 24 +- .../handlers/generate_step/RequestAI/app.py | 85 +- .../generate_step/generate_task/app.py | 59 +- .../generate_step/generate_task/ec2.py | 4 +- .../glob_output_files/__init__.py | 0 .../generate_step/glob_output_files/app.py | 14 + .../generate_step/updateStatusTask/app.py | 11 +- .../generate_batch/app.py | 41 +- .../hdler_preprocess_input_request.py | 43 + .../ai_caller_ecs/sm_ai_caller_ecs.asl.yaml | 25 + .../download_state_machine.asl.yaml | 33 +- .../generate_state_machine.asl.copy1.yaml | 147 - .../generate_state_machine.asl.yaml.copy | 118 - .../auth-service/CognitoClient/template.yaml | 104 + .../api-defs/daita_http_api.yaml | 72 + .../github_openid_token_wrapper/__init__.py | 0 .../github_openid_token_wrapper/app.py | 29 + .../requirements.txt | 1 + .../__init__.py | 0 .../github_openid_userinfo_wrapper/app.py | 49 + .../requirements.txt | 1 + .../functions/login_social/app.py | 125 + .../CognitoUserPool/template.yaml | 199 + .../auth-service/IdentityPool/template.yaml | 17 + .../auth-service/RoleIdentity/template.yaml | 116 + daita-app/build.sh | 107 - daita-app/build_daita.sh | 171 + .../core-service/api-defs/daita_http_api.yaml | 418 +- .../core-service/core_service_template.yaml | 573 ++- .../auth_service/auth_confirm/__init__.py | 0 .../handlers/auth_service/auth_confirm/app.py | 62 + .../confirm_code_forgot_password/__init__.py | 0 .../confirm_code_forgot_password/app.py | 69 + .../auth_service/credential_login/__init__.py | 0 .../auth_service/credential_login/app.py | 268 + .../credential_login/requirements.txt | 2 + .../auth_service/forgot_password/__init__.py | 0 .../auth_service/forgot_password/app.py | 106 + .../forgot_password/requirements.txt | 1 + .../handlers/auth_service/login/__init__.py | 0 .../handlers/auth_service/login/app.py | 255 + .../auth_service/login/requirements.txt | 1 + .../login_refresh_token/__init__.py | 0 .../auth_service/login_refresh_token/app.py | 118 + .../login_refresh_token/requirements.txt | 1 + .../auth_service/login_social/__init__.py | 0 .../handlers/auth_service/login_social/app.py | 127 + .../handlers/auth_service/logout/__init__.py | 0 .../handlers/auth_service/logout/app.py | 53 + .../auth_service/logout/requirements.txt | 1 + .../auth_service/mail_service/__init__.py | 0 .../handlers/auth_service/mail_service/app.py | 65 + .../resend_confirmcode/__init__.py | 0 .../auth_service/resend_confirmcode/app.py | 79 + .../handlers/auth_service/sign_up/__init__.py | 0 .../handlers/auth_service/sign_up/app.py | 148 + .../auth_service/sign_up/requirements.txt | 1 + .../auth_service/template_mail/__init__.py | 0 .../auth_service/template_mail/app.py | 73 + .../template_mail/requirements.txt | 1 + .../cli/check_daita_token/__init__.py | 0 .../handlers/cli/check_daita_token/app.py | 34 + .../cli/check_existence_file/__init__.py | 0 .../handlers/cli/check_existence_file/app.py | 53 + .../cli/cli_upload_project/__init__.py | 0 .../handlers/cli/cli_upload_project/app.py | 133 + .../cli/create_decompress_task/__init__.py | 0 .../cli/create_decompress_task/app.py | 56 + .../cli/create_presignurl_zip/__init__.py | 0 .../handlers/cli/create_presignurl_zip/app.py | 74 + .../handlers/feedback/presignUrl/__init__.py | 0 .../handlers/feedback/presignUrl/app.py | 53 + .../feedback/slack_webhook/__init__.py | 0 .../handlers/feedback/slack_webhook/app.py | 175 + .../feedback/slack_webhook/requirements.txt | 4 + .../generate/daita_upload_token/__init__.py | 0 .../generate/daita_upload_token/app.py | 63 + .../handlers/generate/generate_images/app.py | 119 +- .../handlers/generate/split_data/app.py | 22 +- .../project/apply_param_expert_mode/app.py | 2 +- .../project/create_prj_fr_prebuild/app.py | 145 + .../project/delete_images/__init__.py | 0 .../{ => project}/delete_images/app.py | 54 +- .../project/delete_project/__init__.py | 0 .../{ => project}/delete_project/app.py | 12 +- .../project/list_prebuild_dataset/app.py | 55 + .../project_asy_create_sample/__init__.py | 0 .../project/project_asy_create_sample/app.py | 143 + .../project/project_create/__init__.py | 0 .../handlers/project/project_create/app.py | 115 + .../project_download_create/__init__.py | 0 .../project/project_download_create/app.py | 111 + .../project_download_update/__init__.py | 0 .../project/project_download_update/app.py | 78 + .../handlers/project/project_info/__init__.py | 0 .../handlers/project/project_info/app.py | 170 + .../handlers/project/project_list/__init__.py | 0 .../handlers/project/project_list/app.py | 66 + .../project/project_list_data/__init__.py | 0 .../handlers/project/project_list_data/app.py | 115 + .../project/project_list_info/__init__.py | 0 .../handlers/project/project_list_info/app.py | 116 + .../project/project_sample/__init__.py | 0 .../handlers/project/project_sample/app.py | 136 + .../project/project_update_info/__init__.py | 0 .../project/project_update_info/app.py | 131 + .../project/project_upload_check/__init__.py | 0 .../project/project_upload_check/app.py | 110 + .../handlers/send-mail/reference-email/app.py | 112 + .../hdler_send_email_to_identityid.py | 105 + .../handlers/thumbnail/divide_batch/app.py | 31 + .../handlers/thumbnail/resize_image/app.py | 110 + .../thumbnail/resize_image/requirements.txt | 1 + .../thumbnail/trigger_thumbnail/__init__.py | 0 .../thumbnail/trigger_thumbnail/app.py | 47 + .../thumbnail_step_function.asl.yaml | 35 + .../statemachine/compress_download.asl.yml | 3 +- .../compress-download-app/template.yaml | 51 +- .../statemachine/decompress_file.asl.yml | 10 +- .../decompress-upload-app/template.yaml | 57 +- daita-app/dataflow-service/template.yaml | 48 +- daita-app/db-service/db_template.yaml | 344 +- .../ecs-ai-caller-app/ecs_segementation.yaml | 208 + daita-app/ecs-ai-caller-app/role.yaml | 137 + .../statemachine/ecs_task.asl.yaml | 48 + .../functions/hdler_download_image_to_efs.py | 65 + .../functions/hdler_updoad_image.py | 60 + .../template_ecs_ai_caller.yaml | 68 + .../health_check_service.yaml | 5 +- .../hdler_project_upload_update.py | 238 + .../functions/hdler_move_s3_data.py | 117 + .../functions/hdler_update_input_data.py | 101 + .../functions/hdler_update_sumary_db.py | 114 + .../project_service_template.yaml | 182 + .../sm_create_project_fr_prebuild.asl.yaml | 49 + .../functions/api_handler}/calculate/app.py | 2 +- .../functions/api_handler}/get_info/app.py | 2 +- .../functions/api_handler}/get_status/app.py | 2 +- .../reference_image_service.yaml | 77 +- daita-app/samconfig.toml | 82 +- .../shared-layer/commons/python/config.py | 187 +- .../shared-layer/commons/python/const.py | 44 + .../commons/python/custom_mail.py | 141 + .../commons/python/error_messages.py | 86 +- .../commons/python/identity_check.py | 7 +- .../commons/python/lambda_base_class.py | 35 +- .../commons/python/load_const_lambda_func.py | 16 + .../python/load_env_lambda_function.py | 46 + .../models/annotaition/anno_category_info.py | 46 + .../models/annotaition/anno_class_info.py | 91 + .../models/annotaition/anno_data_model.py | 238 + .../annotaition/anno_label_info_model.py | 80 + .../models/annotaition/anno_project_model.py | 240 + .../annotaition/anno_project_sum_model.py | 174 + .../commons/python/models/base_model.py | 50 + .../commons/python/models/data_model.py | 22 +- .../commons/python/models/event_model.py | 78 + .../commons/python/models/feedback_model.py | 24 + .../models/generate_daita_upload_token.py | 110 + .../python/models/generate_task_model.py | 109 +- .../python/models/prebuild_dataset_model.py | 49 + .../commons/python/models/project_model.py | 9 +- .../python/models/project_sum_model.py | 54 +- .../shared-layer/commons/python/s3_utils.py | 44 +- .../shared-layer/commons/python/thumbnail.py | 8 + .../shared-layer/commons/python/utils.py | 179 +- .../commons/python/verify_captcha.py | 21 + daita-app/template.yaml | 240 +- docs/build_instruction.md | 20 + docs/url-endpoint-annotation-app.md | 299 ++ docs/url-endpoint-aws-service.md | 75 +- download_service/app_download.py | 52 - download_service/download_task.py | 346 -- download_service/requirements.txt | 49 - download_service/settings.py | 6 - download_service/task_worker.py | 17 - env.dev | 8 + infrastructure-def-app/build_infra_app.sh | 86 + infrastructure-def-app/network.yaml | 250 + infrastructure-def-app/samconfig.toml | 19 + infrastructure-def-app/storage.yaml | 79 + infrastructure-def-app/template_infra.yaml | 65 + main_build.sh | 72 + main_config_dev.cnf | 61 + main_config_prod.cnf | 63 + requirements.txt | 2 + 437 files changed, 50727 insertions(+), 3235 deletions(-) create mode 100644 .github/workflows/lint.yml delete mode 100644 ai_caller_service_v1/AI_service/AI_service/__init__.py delete mode 100644 ai_caller_service_v1/AI_service/AI_service/asgi.py delete mode 100644 ai_caller_service_v1/AI_service/AI_service/settings.py delete mode 100644 ai_caller_service_v1/AI_service/AI_service/urls.py delete mode 100644 ai_caller_service_v1/AI_service/AI_service/worker.py delete mode 100644 ai_caller_service_v1/AI_service/AI_service/wsgi.py delete mode 100644 ai_caller_service_v1/AI_service/app/__init__.py delete mode 100644 ai_caller_service_v1/AI_service/app/admin.py delete mode 100644 ai_caller_service_v1/AI_service/app/apps.py delete mode 100644 ai_caller_service_v1/AI_service/app/migrations/0001_initial.py delete mode 100644 ai_caller_service_v1/AI_service/app/models.py delete mode 100644 ai_caller_service_v1/AI_service/app/scripts/check_stop_ec2.py delete mode 100644 ai_caller_service_v1/AI_service/app/scripts/delete_all_keys.py delete mode 100644 ai_caller_service_v1/AI_service/app/scripts/dynamodb/images.py delete mode 100644 ai_caller_service_v1/AI_service/app/scripts/dynamodb/methods.py delete mode 100644 ai_caller_service_v1/AI_service/app/scripts/dynamodb/task.py delete mode 100644 ai_caller_service_v1/AI_service/app/scripts/event_loop.py delete mode 100644 ai_caller_service_v1/AI_service/app/scripts/logging_api.py delete mode 100644 ai_caller_service_v1/AI_service/app/scripts/s3.py delete mode 100644 ai_caller_service_v1/AI_service/app/scripts/utils.py delete mode 100644 ai_caller_service_v1/AI_service/app/tests.py delete mode 100644 ai_caller_service_v1/AI_service/app/views.py delete mode 100644 ai_caller_service_v1/AI_service/kill_worker.sh delete mode 100644 ai_caller_service_v1/AI_service/manage.py delete mode 100644 ai_caller_service_v1/AI_service/requirements.txt delete mode 100644 ai_caller_service_v1/AI_service/worker.sh delete mode 100644 ai_caller_service_v1/build.sh delete mode 100644 ai_caller_service_v1/deploy.sh create mode 100644 annotation-app/annotation-service/api-handler-functions/add-class/hdler_add_class.py create mode 100644 annotation-app/annotation-service/api-handler-functions/create-label-category/hdler_create_label_category.py create mode 100644 annotation-app/annotation-service/api-handler-functions/get-file-info-n-label/hdler_get_file_info_n_label.py create mode 100644 annotation-app/annotation-service/api-handler-functions/list-class/hdler_list_class.py create mode 100644 annotation-app/annotation-service/api-handler-functions/save-label/hdler_save_label.py create mode 100644 annotation-app/annotation-service/template_annotaion_service.yaml create mode 100644 annotation-app/api-service/api_route_define.yaml create mode 100644 annotation-app/api-service/template_api_service.yaml create mode 100644 annotation-app/build_annotation.sh create mode 100644 annotation-app/db-service/db_template.yaml create mode 100644 annotation-app/ecs-segment-app/api-handler-functions/hdler_stream_data_annotation.py create mode 100644 annotation-app/ecs-segment-app/api-handler-functions/hdler_task_segmentation.py create mode 100644 annotation-app/ecs-segment-app/controller.yaml create mode 100644 annotation-app/ecs-segment-app/ecs_segementation.yaml create mode 100644 annotation-app/ecs-segment-app/network.yaml create mode 100644 annotation-app/ecs-segment-app/role.yaml create mode 100644 annotation-app/ecs-segment-app/statemachine/ecs_task.asl.yaml create mode 100644 annotation-app/ecs-segment-app/statemachine/functions/hdler_download_image_to_efs.py create mode 100644 annotation-app/ecs-segment-app/statemachine/functions/hdler_send_email_to_identityid.py create mode 100644 annotation-app/ecs-segment-app/statemachine/functions/hdler_updoad_image.py create mode 100644 annotation-app/ecs-segment-app/storage.yaml create mode 100644 annotation-app/ecs-segment-app/template.yaml create mode 100644 annotation-app/project-service/api-handler-functions/check-ai-segmentation/hdler_check_si_segm_progress.py create mode 100644 annotation-app/project-service/api-handler-functions/clone-project/hdler_project_clone.py create mode 100644 annotation-app/project-service/api-handler-functions/delete-project/hdler_delete_project.py create mode 100644 annotation-app/project-service/api-handler-functions/get-project-info/hdler_get_project_info.py create mode 100644 annotation-app/project-service/api-handler-functions/list-data/hdler_list_data.py create mode 100644 annotation-app/project-service/api-handler-functions/list-project/hdler_list_project.py create mode 100644 annotation-app/project-service/api-handler-functions/upload-check/hdler_upload_check.py create mode 100644 annotation-app/project-service/api-handler-functions/upload-update/hdler_upload_update.py create mode 100644 annotation-app/project-service/statemachine/clone_project_data/functions/hdler_move_s3_data.py create mode 100644 annotation-app/project-service/statemachine/clone_project_data/functions/hdler_update_input_data.py create mode 100644 annotation-app/project-service/statemachine/clone_project_data/functions/hdler_update_sumary_db.py create mode 100644 annotation-app/project-service/statemachine/clone_project_data/sm_clone_project.asl.yaml create mode 100644 annotation-app/project-service/template_project_service.yaml create mode 100644 annotation-app/samconfig.toml create mode 100644 annotation-app/template_annotation_app.yaml create mode 100644 core_service/aws_lambda/project/packages/slack/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack/deprecation.py create mode 100644 core_service/aws_lambda/project/packages/slack/errors.py rename ai_caller_service_v1/AI_service/app/migrations/__init__.py => core_service/aws_lambda/project/packages/slack/py.typed (100%) create mode 100644 core_service/aws_lambda/project/packages/slack/rtm/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack/rtm/client.py create mode 100644 core_service/aws_lambda/project/packages/slack/signature/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack/signature/verifier.py create mode 100644 core_service/aws_lambda/project/packages/slack/version.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/async_base_client.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/async_client.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/async_internal_utils.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/async_slack_response.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/base_client.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/classes/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/classes/actions.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/classes/attachments.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/classes/blocks.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/classes/dialog_elements.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/classes/dialogs.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/classes/elements.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/classes/interactions.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/classes/messages.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/classes/objects.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/classes/views.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/client.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/deprecation.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/internal_utils.py create mode 100644 core_service/aws_lambda/project/packages/slack/web/slack_response.py create mode 100644 core_service/aws_lambda/project/packages/slack/webhook/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack/webhook/async_client.py create mode 100644 core_service/aws_lambda/project/packages/slack/webhook/client.py create mode 100644 core_service/aws_lambda/project/packages/slack/webhook/internal_utils.py create mode 100644 core_service/aws_lambda/project/packages/slack/webhook/webhook_response.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/INSTALLER create mode 100644 core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/LICENSE create mode 100644 core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/METADATA create mode 100644 core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/RECORD rename daita-app/core-service/functions/handlers/delete_images/__init__.py => core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/REQUESTED (100%) create mode 100644 core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/WHEEL create mode 100644 core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/top_level.txt create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/aiohttp_version_checker.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/audit_logs/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/audit_logs/async_client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/async_client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/internal_utils.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/logs.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/response.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/errors/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/http_retry/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/http_retry/async_handler.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/http_retry/builtin_async_handlers.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/http_retry/builtin_handlers.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/http_retry/builtin_interval_calculators.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/http_retry/handler.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/http_retry/interval_calculator.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/http_retry/jitter.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/http_retry/request.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/http_retry/response.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/http_retry/state.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/models/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/models/attachments/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/models/basic_objects.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/models/blocks/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/models/blocks/basic_components.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/models/blocks/block_elements.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/models/blocks/blocks.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/models/dialoags.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/models/dialogs/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/models/messages/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/models/messages/message.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/models/metadata/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/models/views/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/authorize_url_generator/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/amazon_s3/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/async_cacheable_installation_store.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/async_installation_store.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/cacheable_installation_store.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/file/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/installation_store.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/internals.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/models/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/models/bot.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/models/installation.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/sqlalchemy/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/sqlite3/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/redirect_uri_page_renderer/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/amazon_s3/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/async_state_store.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/file/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/sqlalchemy/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/sqlite3/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/state_store.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/state_utils/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/token_rotation/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/token_rotation/async_rotator.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/oauth/token_rotation/rotator.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/proxy_env_variable_loader.py rename daita-app/core-service/functions/handlers/delete_project/__init__.py => core_service/aws_lambda/project/packages/slack_sdk/py.typed (100%) create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/rtm/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/rtm/v2/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/rtm_v2/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/scim/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/scim/async_client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/scim/v1/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/scim/v1/async_client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/scim/v1/client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/scim/v1/default_arg.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/scim/v1/group.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/scim/v1/internal_utils.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/scim/v1/response.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/scim/v1/types.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/scim/v1/user.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/signature/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/aiohttp/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/async_client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/async_listeners.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/connection.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/frame_header.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/internals.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/interval_runner.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/listeners.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/request.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/response.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/websocket_client/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/socket_mode/websockets/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/version.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/web/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/web/async_base_client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/web/async_client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/web/async_internal_utils.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/web/async_slack_response.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/web/base_client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/web/client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/web/deprecation.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/web/internal_utils.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/web/legacy_base_client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/web/legacy_client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/web/legacy_slack_response.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/web/slack_response.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/webhook/__init__.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/webhook/async_client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/webhook/client.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/webhook/internal_utils.py create mode 100644 core_service/aws_lambda/project/packages/slack_sdk/webhook/webhook_response.py create mode 100644 daita-app/ai-caller-service/functions/handlers/download_task/get_reference_images/__init__.py create mode 100644 daita-app/ai-caller-service/functions/handlers/download_task/get_reference_images/app.py create mode 100644 daita-app/ai-caller-service/functions/handlers/download_task/get_reference_images/requirements.txt create mode 100644 daita-app/ai-caller-service/functions/handlers/generate_step/glob_output_files/__init__.py create mode 100644 daita-app/ai-caller-service/functions/handlers/generate_step/glob_output_files/app.py create mode 100644 daita-app/ai-caller-service/statemachine/ai_caller_ecs/functions/preprocess_input_request/hdler_preprocess_input_request.py create mode 100644 daita-app/ai-caller-service/statemachine/ai_caller_ecs/sm_ai_caller_ecs.asl.yaml delete mode 100644 daita-app/ai-caller-service/statemachine/generate_state_machine.asl.copy1.yaml delete mode 100644 daita-app/ai-caller-service/statemachine/generate_state_machine.asl.yaml.copy create mode 100644 daita-app/auth-service/CognitoClient/template.yaml create mode 100644 daita-app/auth-service/CognitoUserPool/api-defs/daita_http_api.yaml create mode 100644 daita-app/auth-service/CognitoUserPool/functions/github_openid_token_wrapper/__init__.py create mode 100644 daita-app/auth-service/CognitoUserPool/functions/github_openid_token_wrapper/app.py create mode 100644 daita-app/auth-service/CognitoUserPool/functions/github_openid_token_wrapper/requirements.txt create mode 100644 daita-app/auth-service/CognitoUserPool/functions/github_openid_userinfo_wrapper/__init__.py create mode 100644 daita-app/auth-service/CognitoUserPool/functions/github_openid_userinfo_wrapper/app.py create mode 100644 daita-app/auth-service/CognitoUserPool/functions/github_openid_userinfo_wrapper/requirements.txt create mode 100644 daita-app/auth-service/CognitoUserPool/functions/login_social/app.py create mode 100644 daita-app/auth-service/CognitoUserPool/template.yaml create mode 100644 daita-app/auth-service/IdentityPool/template.yaml create mode 100644 daita-app/auth-service/RoleIdentity/template.yaml delete mode 100755 daita-app/build.sh create mode 100755 daita-app/build_daita.sh create mode 100644 daita-app/core-service/functions/handlers/auth_service/auth_confirm/__init__.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/auth_confirm/app.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/confirm_code_forgot_password/__init__.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/confirm_code_forgot_password/app.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/credential_login/__init__.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/credential_login/app.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/credential_login/requirements.txt create mode 100644 daita-app/core-service/functions/handlers/auth_service/forgot_password/__init__.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/forgot_password/app.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/forgot_password/requirements.txt create mode 100644 daita-app/core-service/functions/handlers/auth_service/login/__init__.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/login/app.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/login/requirements.txt create mode 100644 daita-app/core-service/functions/handlers/auth_service/login_refresh_token/__init__.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/login_refresh_token/app.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/login_refresh_token/requirements.txt create mode 100644 daita-app/core-service/functions/handlers/auth_service/login_social/__init__.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/login_social/app.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/logout/__init__.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/logout/app.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/logout/requirements.txt create mode 100644 daita-app/core-service/functions/handlers/auth_service/mail_service/__init__.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/mail_service/app.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/resend_confirmcode/__init__.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/resend_confirmcode/app.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/sign_up/__init__.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/sign_up/app.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/sign_up/requirements.txt create mode 100644 daita-app/core-service/functions/handlers/auth_service/template_mail/__init__.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/template_mail/app.py create mode 100644 daita-app/core-service/functions/handlers/auth_service/template_mail/requirements.txt create mode 100644 daita-app/core-service/functions/handlers/cli/check_daita_token/__init__.py create mode 100644 daita-app/core-service/functions/handlers/cli/check_daita_token/app.py create mode 100644 daita-app/core-service/functions/handlers/cli/check_existence_file/__init__.py create mode 100644 daita-app/core-service/functions/handlers/cli/check_existence_file/app.py create mode 100644 daita-app/core-service/functions/handlers/cli/cli_upload_project/__init__.py create mode 100644 daita-app/core-service/functions/handlers/cli/cli_upload_project/app.py create mode 100644 daita-app/core-service/functions/handlers/cli/create_decompress_task/__init__.py create mode 100644 daita-app/core-service/functions/handlers/cli/create_decompress_task/app.py create mode 100644 daita-app/core-service/functions/handlers/cli/create_presignurl_zip/__init__.py create mode 100644 daita-app/core-service/functions/handlers/cli/create_presignurl_zip/app.py create mode 100644 daita-app/core-service/functions/handlers/feedback/presignUrl/__init__.py create mode 100644 daita-app/core-service/functions/handlers/feedback/presignUrl/app.py create mode 100644 daita-app/core-service/functions/handlers/feedback/slack_webhook/__init__.py create mode 100644 daita-app/core-service/functions/handlers/feedback/slack_webhook/app.py create mode 100644 daita-app/core-service/functions/handlers/feedback/slack_webhook/requirements.txt create mode 100644 daita-app/core-service/functions/handlers/generate/daita_upload_token/__init__.py create mode 100644 daita-app/core-service/functions/handlers/generate/daita_upload_token/app.py create mode 100644 daita-app/core-service/functions/handlers/project/create_prj_fr_prebuild/app.py create mode 100644 daita-app/core-service/functions/handlers/project/delete_images/__init__.py rename daita-app/core-service/functions/handlers/{ => project}/delete_images/app.py (81%) create mode 100644 daita-app/core-service/functions/handlers/project/delete_project/__init__.py rename daita-app/core-service/functions/handlers/{ => project}/delete_project/app.py (92%) create mode 100644 daita-app/core-service/functions/handlers/project/list_prebuild_dataset/app.py create mode 100644 daita-app/core-service/functions/handlers/project/project_asy_create_sample/__init__.py create mode 100644 daita-app/core-service/functions/handlers/project/project_asy_create_sample/app.py create mode 100644 daita-app/core-service/functions/handlers/project/project_create/__init__.py create mode 100644 daita-app/core-service/functions/handlers/project/project_create/app.py create mode 100644 daita-app/core-service/functions/handlers/project/project_download_create/__init__.py create mode 100644 daita-app/core-service/functions/handlers/project/project_download_create/app.py create mode 100644 daita-app/core-service/functions/handlers/project/project_download_update/__init__.py create mode 100644 daita-app/core-service/functions/handlers/project/project_download_update/app.py create mode 100644 daita-app/core-service/functions/handlers/project/project_info/__init__.py create mode 100644 daita-app/core-service/functions/handlers/project/project_info/app.py create mode 100644 daita-app/core-service/functions/handlers/project/project_list/__init__.py create mode 100644 daita-app/core-service/functions/handlers/project/project_list/app.py create mode 100644 daita-app/core-service/functions/handlers/project/project_list_data/__init__.py create mode 100644 daita-app/core-service/functions/handlers/project/project_list_data/app.py create mode 100644 daita-app/core-service/functions/handlers/project/project_list_info/__init__.py create mode 100644 daita-app/core-service/functions/handlers/project/project_list_info/app.py create mode 100644 daita-app/core-service/functions/handlers/project/project_sample/__init__.py create mode 100644 daita-app/core-service/functions/handlers/project/project_sample/app.py create mode 100644 daita-app/core-service/functions/handlers/project/project_update_info/__init__.py create mode 100644 daita-app/core-service/functions/handlers/project/project_update_info/app.py create mode 100644 daita-app/core-service/functions/handlers/project/project_upload_check/__init__.py create mode 100644 daita-app/core-service/functions/handlers/project/project_upload_check/app.py create mode 100644 daita-app/core-service/functions/handlers/send-mail/reference-email/app.py create mode 100644 daita-app/core-service/functions/handlers/send-mail/send-email-identity-id/hdler_send_email_to_identityid.py create mode 100644 daita-app/core-service/functions/handlers/thumbnail/divide_batch/app.py create mode 100644 daita-app/core-service/functions/handlers/thumbnail/resize_image/app.py create mode 100644 daita-app/core-service/functions/handlers/thumbnail/resize_image/requirements.txt create mode 100644 daita-app/core-service/functions/handlers/thumbnail/trigger_thumbnail/__init__.py create mode 100644 daita-app/core-service/functions/handlers/thumbnail/trigger_thumbnail/app.py create mode 100644 daita-app/core-service/statemachine/thumbnail_step_function.asl.yaml create mode 100644 daita-app/ecs-ai-caller-app/ecs_segementation.yaml create mode 100644 daita-app/ecs-ai-caller-app/role.yaml create mode 100644 daita-app/ecs-ai-caller-app/statemachine/ecs_task.asl.yaml create mode 100644 daita-app/ecs-ai-caller-app/statemachine/functions/hdler_download_image_to_efs.py create mode 100644 daita-app/ecs-ai-caller-app/statemachine/functions/hdler_updoad_image.py create mode 100644 daita-app/ecs-ai-caller-app/template_ecs_ai_caller.yaml create mode 100644 daita-app/project-service/functions/api_handler/hdler_project_upload_update.py create mode 100644 daita-app/project-service/functions/hdler_move_s3_data.py create mode 100644 daita-app/project-service/functions/hdler_update_input_data.py create mode 100644 daita-app/project-service/functions/hdler_update_sumary_db.py create mode 100644 daita-app/project-service/project_service_template.yaml create mode 100644 daita-app/project-service/statemachine/sm_create_project_fr_prebuild.asl.yaml rename daita-app/{core-service/functions/handlers/reference_image => reference-image-service/functions/api_handler}/calculate/app.py (96%) rename daita-app/{core-service/functions/handlers/reference_image => reference-image-service/functions/api_handler}/get_info/app.py (93%) rename daita-app/{core-service/functions/handlers/reference_image => reference-image-service/functions/api_handler}/get_status/app.py (94%) create mode 100644 daita-app/shared-layer/commons/python/const.py create mode 100644 daita-app/shared-layer/commons/python/custom_mail.py create mode 100644 daita-app/shared-layer/commons/python/load_const_lambda_func.py create mode 100644 daita-app/shared-layer/commons/python/load_env_lambda_function.py create mode 100644 daita-app/shared-layer/commons/python/models/annotaition/anno_category_info.py create mode 100644 daita-app/shared-layer/commons/python/models/annotaition/anno_class_info.py create mode 100644 daita-app/shared-layer/commons/python/models/annotaition/anno_data_model.py create mode 100644 daita-app/shared-layer/commons/python/models/annotaition/anno_label_info_model.py create mode 100644 daita-app/shared-layer/commons/python/models/annotaition/anno_project_model.py create mode 100644 daita-app/shared-layer/commons/python/models/annotaition/anno_project_sum_model.py create mode 100644 daita-app/shared-layer/commons/python/models/base_model.py create mode 100644 daita-app/shared-layer/commons/python/models/event_model.py create mode 100644 daita-app/shared-layer/commons/python/models/feedback_model.py create mode 100644 daita-app/shared-layer/commons/python/models/generate_daita_upload_token.py create mode 100644 daita-app/shared-layer/commons/python/models/prebuild_dataset_model.py create mode 100644 daita-app/shared-layer/commons/python/thumbnail.py create mode 100644 daita-app/shared-layer/commons/python/verify_captcha.py create mode 100644 docs/build_instruction.md create mode 100644 docs/url-endpoint-annotation-app.md delete mode 100644 download_service/app_download.py delete mode 100644 download_service/download_task.py delete mode 100644 download_service/requirements.txt delete mode 100644 download_service/settings.py delete mode 100644 download_service/task_worker.py create mode 100644 env.dev create mode 100644 infrastructure-def-app/build_infra_app.sh create mode 100644 infrastructure-def-app/network.yaml create mode 100644 infrastructure-def-app/samconfig.toml create mode 100644 infrastructure-def-app/storage.yaml create mode 100644 infrastructure-def-app/template_infra.yaml create mode 100644 main_build.sh create mode 100644 main_config_dev.cnf create mode 100644 main_config_prod.cnf create mode 100644 requirements.txt diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml new file mode 100644 index 0000000..17081db --- /dev/null +++ b/.github/workflows/lint.yml @@ -0,0 +1,35 @@ +name: 👮‍♂️ Lint + +on: [push, pull_request] + +concurrency: + group: ${{github.workflow}}-${{github.ref}} + cancel-in-progress: true + +jobs: + unit: + runs-on: ${{ matrix.os }} + strategy: + matrix: + os: + - ubuntu-latest + architecture: + - 'x64' + python-version: + - '3.8' + + steps: + - name: Checkout + uses: actions/checkout@v3 + + - name: Setup Python + uses: actions/setup-python@v4 + with: + python-version: ${{ matrix.python-version }} + architecture: ${{ matrix.architecture }} + + - name: Check formatting with Black + uses: psf/black@stable + with: + options: "--check --verbose" + src: "./" diff --git a/.gitignore b/.gitignore index 8533557..a5cac4e 100644 --- a/.gitignore +++ b/.gitignore @@ -170,3 +170,11 @@ daita-app/output.txt /daita-app/.aws-sam/ +# key file +*.cer +output.txt +output_daita.txt + +output_fe_config.txt +annotation-app/output_anno.txt +output_daita.txt diff --git a/ai_caller_service_v1/AI_service/AI_service/__init__.py b/ai_caller_service_v1/AI_service/AI_service/__init__.py deleted file mode 100644 index 8a5e60a..0000000 --- a/ai_caller_service_v1/AI_service/AI_service/__init__.py +++ /dev/null @@ -1,4 +0,0 @@ -#./__init__.py -from __future__ import absolute_import, unicode_literals -from AI_service.worker import app as celery_app -__all__ = ['celery_app'] diff --git a/ai_caller_service_v1/AI_service/AI_service/asgi.py b/ai_caller_service_v1/AI_service/AI_service/asgi.py deleted file mode 100644 index 774fdcc..0000000 --- a/ai_caller_service_v1/AI_service/AI_service/asgi.py +++ /dev/null @@ -1,16 +0,0 @@ -""" -ASGI config for AI_service project. - -It exposes the ASGI callable as a module-level variable named ``application``. - -For more information on this file, see -https://docs.djangoproject.com/en/3.2/howto/deployment/asgi/ -""" - -import os - -from django.core.asgi import get_asgi_application - -os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'AI_service.settings') - -application = get_asgi_application() diff --git a/ai_caller_service_v1/AI_service/AI_service/settings.py b/ai_caller_service_v1/AI_service/AI_service/settings.py deleted file mode 100644 index e38101d..0000000 --- a/ai_caller_service_v1/AI_service/AI_service/settings.py +++ /dev/null @@ -1,129 +0,0 @@ -""" -Django settings for AI_service project. - -Generated by 'django-admin startproject' using Django 3.2.9. - -For more information on this file, see -https://docs.djangoproject.com/en/3.2/topics/settings/ - -For the full list of settings and their values, see -https://docs.djangoproject.com/en/3.2/ref/settings/ -""" -import os -from pathlib import Path - -# Build paths inside the project like this: BASE_DIR / 'subdir'. -BASE_DIR = Path(__file__).resolve().parent.parent - - -# Quick-start development settings - unsuitable for production -# See https://docs.djangoproject.com/en/3.2/howto/deployment/checklist/ - -# SECURITY WARNING: keep the secret key used in production secret! -SECRET_KEY = 'django-insecure-2@+jwjg)d@-pjue29@y-e@f!)r7n%p3p#hq98)xccdqiyq186d' - -# SECURITY WARNING: don't run with debug turned on in production! -DEBUG = True - -ALLOWED_HOSTS = ['0.0.0.0'] - - -# Application definition -CELERY_BROKER_URL = 'redis://localhost/10' - -INSTALLED_APPS = [ - 'rest_framework', - 'app', - 'django.contrib.admin', - 'django.contrib.auth', - 'django.contrib.contenttypes', - 'django.contrib.sessions', - 'django.contrib.messages', - 'django.contrib.staticfiles', -] - -MIDDLEWARE = [ - 'django.middleware.security.SecurityMiddleware', - 'django.contrib.sessions.middleware.SessionMiddleware', - 'django.middleware.common.CommonMiddleware', - 'django.middleware.csrf.CsrfViewMiddleware', - 'django.contrib.auth.middleware.AuthenticationMiddleware', - 'django.contrib.messages.middleware.MessageMiddleware', - 'django.middleware.clickjacking.XFrameOptionsMiddleware', -] - -ROOT_URLCONF = 'AI_service.urls' - -TEMPLATES = [ - { - 'BACKEND': 'django.template.backends.django.DjangoTemplates', - 'DIRS': [], - 'APP_DIRS': True, - 'OPTIONS': { - 'context_processors': [ - 'django.template.context_processors.debug', - 'django.template.context_processors.request', - 'django.contrib.auth.context_processors.auth', - 'django.contrib.messages.context_processors.messages', - ], - }, - }, -] - -WSGI_APPLICATION = 'AI_service.wsgi.application' - - -# Database -# https://docs.djangoproject.com/en/3.2/ref/settings/#databases - -DATABASES = { - 'default': { - 'ENGINE': 'django.db.backends.sqlite3', - 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'), - - } -} - - -# Password validation -# https://docs.djangoproject.com/en/3.2/ref/settings/#auth-password-validators - -AUTH_PASSWORD_VALIDATORS = [ - { - 'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator', - }, - { - 'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator', - }, - { - 'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator', - }, - { - 'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator', - }, -] - - -# Internationalization -# https://docs.djangoproject.com/en/3.2/topics/i18n/ - -LANGUAGE_CODE = 'en-us' - -TIME_ZONE = 'UTC' - -USE_I18N = True - -USE_L10N = True - -USE_TZ = True - - -# Static files (CSS, JavaScript, Images) -# https://docs.djangoproject.com/en/3.2/howto/static-files/ - -STATIC_URL = '/static/' - -# Default primary key field type -# https://docs.djangoproject.com/en/3.2/ref/settings/#default-auto-field - -DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField' diff --git a/ai_caller_service_v1/AI_service/AI_service/urls.py b/ai_caller_service_v1/AI_service/AI_service/urls.py deleted file mode 100644 index 88426ef..0000000 --- a/ai_caller_service_v1/AI_service/AI_service/urls.py +++ /dev/null @@ -1,29 +0,0 @@ -"""AI_service URL Configuration - -The `urlpatterns` list routes URLs to views. For more information please see: - https://docs.djangoproject.com/en/3.2/topics/http/urls/ -Examples: -Function views - 1. Add an import: from my_app import views - 2. Add a URL to urlpatterns: path('', views.home, name='home') -Class-based views - 1. Add an import: from other_app.views import Home - 2. Add a URL to urlpatterns: path('', Home.as_view(), name='home') -Including another URLconf - 1. Import the include() function: from django.urls import include, path - 2. Add a URL to urlpatterns: path('blog/', include('blog.urls')) -""" -from django.conf.urls import url, include -from django.contrib import admin -from django.urls import path -from rest_framework import routers -from app.views import * -# router = routers.SimpleRouter() -# router.register(r'posts',PostDetailUpdateApiView,basename='Posts') -# router.register(r'posts',PostListCreateAPIView,basename='Posts') -urlpatterns = [ - url(r'^admin/',admin.site.urls), - url(r'^v1/api/request_ai',AI_service_request), - url(r'^v1/api/check_healthy',AI_service_check_healthy) -] - diff --git a/ai_caller_service_v1/AI_service/AI_service/worker.py b/ai_caller_service_v1/AI_service/AI_service/worker.py deleted file mode 100644 index d95b166..0000000 --- a/ai_caller_service_v1/AI_service/AI_service/worker.py +++ /dev/null @@ -1,8 +0,0 @@ -from celery import Celery -from celery import shared_task -import os -import time - -os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'AI_service.settings') # DON'T FORGET TO CHANGE THIS ACCORDINGLY -app = Celery('AI_service',backend='redis://127.0.0.1:6379/2' , broker='redis://127.0.0.1:6379/3') -app.config_from_object('django.conf:settings', namespace='CELERY') diff --git a/ai_caller_service_v1/AI_service/AI_service/wsgi.py b/ai_caller_service_v1/AI_service/AI_service/wsgi.py deleted file mode 100644 index ffe4f6e..0000000 --- a/ai_caller_service_v1/AI_service/AI_service/wsgi.py +++ /dev/null @@ -1,16 +0,0 @@ -""" -WSGI config for AI_service project. - -It exposes the WSGI callable as a module-level variable named ``application``. - -For more information on this file, see -https://docs.djangoproject.com/en/3.2/howto/deployment/wsgi/ -""" - -import os - -from django.core.wsgi import get_wsgi_application - -os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'AI_service.settings') - -application = get_wsgi_application() diff --git a/ai_caller_service_v1/AI_service/app/__init__.py b/ai_caller_service_v1/AI_service/app/__init__.py deleted file mode 100644 index 8b13789..0000000 --- a/ai_caller_service_v1/AI_service/app/__init__.py +++ /dev/null @@ -1 +0,0 @@ - diff --git a/ai_caller_service_v1/AI_service/app/admin.py b/ai_caller_service_v1/AI_service/app/admin.py deleted file mode 100644 index 8c38f3f..0000000 --- a/ai_caller_service_v1/AI_service/app/admin.py +++ /dev/null @@ -1,3 +0,0 @@ -from django.contrib import admin - -# Register your models here. diff --git a/ai_caller_service_v1/AI_service/app/apps.py b/ai_caller_service_v1/AI_service/app/apps.py deleted file mode 100644 index ed327d2..0000000 --- a/ai_caller_service_v1/AI_service/app/apps.py +++ /dev/null @@ -1,6 +0,0 @@ -from django.apps import AppConfig - - -class AppConfig(AppConfig): - default_auto_field = 'django.db.models.BigAutoField' - name = 'app' diff --git a/ai_caller_service_v1/AI_service/app/migrations/0001_initial.py b/ai_caller_service_v1/AI_service/app/migrations/0001_initial.py deleted file mode 100644 index f44aa36..0000000 --- a/ai_caller_service_v1/AI_service/app/migrations/0001_initial.py +++ /dev/null @@ -1,28 +0,0 @@ -# Generated by Django 3.2.9 on 2021-11-29 08:49 - -from django.db import migrations, models - - -class Migration(migrations.Migration): - - initial = True - - dependencies = [ - ] - - operations = [ - migrations.CreateModel( - name='Post', - fields=[ - ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('title', models.CharField(max_length=120)), - ('content', models.TextField()), - ('read_time', models.IntegerField(default=0)), - ('updated', models.DateTimeField(auto_now=True)), - ('created', models.DateTimeField(auto_now_add=True)), - ], - options={ - 'ordering': ['-created', '-updated'], - }, - ), - ] diff --git a/ai_caller_service_v1/AI_service/app/models.py b/ai_caller_service_v1/AI_service/app/models.py deleted file mode 100644 index 71a8362..0000000 --- a/ai_caller_service_v1/AI_service/app/models.py +++ /dev/null @@ -1,3 +0,0 @@ -from django.db import models - -# Create your models here. diff --git a/ai_caller_service_v1/AI_service/app/scripts/check_stop_ec2.py b/ai_caller_service_v1/AI_service/app/scripts/check_stop_ec2.py deleted file mode 100644 index 3ddd9c2..0000000 --- a/ai_caller_service_v1/AI_service/app/scripts/check_stop_ec2.py +++ /dev/null @@ -1,63 +0,0 @@ -import redis -import os -import logging -import json -import datetime as dt -import logging.handlers as Handlers -import boto3 - -EXPIRED = 60*31 +30 -# loggingDir ='./ec2_job' -# os.makedirs(loggingDir,exist_ok=True) - - - - -def stop_ec2(ec2_id): - client = boto3.client('lambda') - print("[DEBUG] stop ec2: {}".format(ec2_id)) - # log.debug("[DEBUG] stop ec2: {}".format(ec2_id)) - try : - response = client.invoke( - FunctionName="staging-balancer-asy-stop", - InvocationType="RequestResponse", - Payload=json.dumps({"ec2_id":ec2_id}) - ) - print("[INFO] info request stop ec2: {}".format(json.loads(response['Payload'].read()))) - # log.info("[INFO] info request stop ec2: {}".format(json.loads(response['Payload'].read()))) - except Exception as e : - print("[ERROR] error request stop ec2: {}".format(e)) - # log.error("[ERROR] error request stop ec2: {}".format(e)) - - -def event_handler(msg): - try: - key = msg["data"].decode("utf-8") - if 'instanceKey' in key: - print(key) - key = key.replace("instanceKey:","") - # arr = [json.loads(conn.lindex(key, i)) for i in range(0, conn.llen(key))] - print('{} --- {}'.format(key,conn.llen(key))) - if not conn.llen(key): - stop_ec2(key) - conn.delete('instanceKey:'+key) - conn.delete(key) - print('delete completed {}'.format(key)) - else: - print('Expired continue: {}'.format(key)) - conn.set('instanceKey:'+key,'EX',EXPIRED) - except Exception as e: - print(e) - pass - - -conn = redis.Redis(host='localhost', - - port=6379, - - db=0) -pubsub = conn.pubsub() -conn.config_set('notify-keyspace-events', 'Ex') -pubsub.psubscribe(**{"__keyevent@0__:expired":event_handler}) -pubsub.run_in_thread(sleep_time=0.01) -print("Running : worker redis subscriber ...") diff --git a/ai_caller_service_v1/AI_service/app/scripts/delete_all_keys.py b/ai_caller_service_v1/AI_service/app/scripts/delete_all_keys.py deleted file mode 100644 index a67ee6c..0000000 --- a/ai_caller_service_v1/AI_service/app/scripts/delete_all_keys.py +++ /dev/null @@ -1,6 +0,0 @@ -import redis -conn = redis.Redis(host='localhost',port=6379,db=0) -print(conn.keys()) -for key in conn.keys(): - conn.delete(key) -print("DELETE COMPLETED") \ No newline at end of file diff --git a/ai_caller_service_v1/AI_service/app/scripts/dynamodb/images.py b/ai_caller_service_v1/AI_service/app/scripts/dynamodb/images.py deleted file mode 100644 index 31afad6..0000000 --- a/ai_caller_service_v1/AI_service/app/scripts/dynamodb/images.py +++ /dev/null @@ -1,165 +0,0 @@ -import boto3 -import random -from boto3.dynamodb.conditions import Key, Attr -from datetime import datetime - -TBL_data_original = 'data_original' -TBL_data_proprocess= 'data_preprocess' -TBL_PROJECT = 'projects' -MAX_NUMBER_GEN_PER_IMAGES = 5 -USER_POOL_ID = 'us-east-2_6Sc8AZij7' -IDENTITY_POOL_ID = 'us-east-2:639788f0-a9b0-460d-9f50-23bbe5bc7140' - -MAX_TIMES_GENERATED_IMAGES = 5 -def convert_current_date_to_iso8601(): - my_date = datetime.now() - return my_date.isoformat() - -def dydb_update_project_data_type_number(db_resource, identity_id, project_name, data_type, data_number, times_generated): - try: - table = db_resource.Table(TBL_PROJECT) - response = table.update_item( - Key={ - 'identity_id': identity_id, - 'project_name': project_name, - }, - ExpressionAttributeNames= { - '#DA_TY': data_type - }, - ExpressionAttributeValues = { - ':va': data_number, - ':da': convert_current_date_to_iso8601(), - ':tg': times_generated - }, - UpdateExpression = 'SET #DA_TY = :va , updated_date = :da, times_generated = :tg' - ) - except Exception as e: - print('Error: ', repr(e)) - raise - if response.get('Item', None): - return response['Item'] - return None - -def dydb_get_project(db_resource, identity_id, project_name): - try: - table = db_resource.Table(TBL_PROJECT) - response = table.get_item( - Key={ - 'identity_id': identity_id, - 'project_name': project_name, - }, - ProjectionExpression= 'project_id, s3_prefix, times_generated' - ) - except Exception as e: - print('Error: ', repr(e)) - raise - if response.get('Item', None): - return response['Item'] - return None -def dydb_update_class_data(table, project_id, filename, classtype): - response = table.update_item( - Key={ - 'project_id': project_id, - 'filename': filename, - }, - ExpressionAttributeNames= { - '#CT': 'classtype', - }, - ExpressionAttributeValues = { - ':ct': classtype, - - }, - UpdateExpression = 'SET #CT = :ct' - ) -def aws_get_identity_id(id_token): - identity_client = boto3.client('cognito-identity') - PROVIDER = f'cognito-idp.{identity_client.meta.region_name}.amazonaws.com/{USER_POOL_ID}' - - try: - identity_response = identity_client.get_id( - IdentityPoolId=IDENTITY_POOL_ID, - Logins = {PROVIDER: id_token}) - except Exception as e: - print('Error: ', repr(e)) - raise - - identity_id = identity_response['IdentityId'] - - return identity_id - -class ImageLoader(object): - def __init__(self): - self.db_resource = boto3.resource('dynamodb') - ''' - info_image: - id_token - project_id - augment_code - data_number - data_type - project_name - ''' - def __call__(self,info_image): - id_token = info_image['id_token'] - project_id = info_image['project_id'] - ls_methods_id = info_image['augment_code'] - project_name = info_image['project_name'] - data_type = info_image['data_type'] - data_number = info_image['data_number'] - try: - identity_id = aws_get_identity_id(id_token) - except Exception as e: - print('Error: ', repr(e)) - return {},e - # get type of process - type_method = 'PREPROCESS' - if 'AUG' in ls_methods_id[0]: - type_method = 'AUGMENT' - elif 'PRE' in ls_methods_id[0]: - type_method = 'PREPROCESS' - else: - raise(Exception('list method is not valid!')) - infor = dydb_get_project(self.db_resource, identity_id, project_name) - s3_prefix = infor['s3_prefix'] - - if data_type == 'ORIGINAL': - table_name = TBL_data_original - elif data_type == 'PREPROCESS': - table_name = TBL_data_proprocess - else: - raise(Exception('data_type is not valid!')) - - if type_method == 'PREPROCESS': - table_name = TBL_data_original - - # get list data - table = self.db_resource.Table(table_name) - response = table.query( - KeyConditionExpression = Key('project_id').eq(project_id), - ProjectionExpression='filename, s3_key', - ) - ls_data = response['Items'] - - if type_method == 'PREPROCESS': - ls_process = [item['s3_key'] for item in ls_data] # use all data in original for preprocessing - elif type_method == 'AUGMENT': - random.shuffle(ls_data) - ls_process = [] - ls_train = [] - ls_val = [] - ls_test = [] - for idx, data in enumerate(ls_data): - if idx: - output_dir : : diretory of gen image - info_request : : id_token : - project_id : - project_name - project_name - identity_id - process_type :: AUGMENT or PREPROCESS - aug : - # error_message_queue : : put error message into queue to log - total_file : the image numbers generate -""" - -def uploading_image(input_uploading_image): - s3_instance = S3(input_uploading_image['project_prefix'],input_uploading_image['process_type']) - s3_key, s3_info = s3_instance.upload_image(input_uploading_image['project_prefix'], - input_uploading_image['method_db'], - input_uploading_image['process_type'], - input_uploading_image['list_aug'], - input_uploading_image['output_dir'], - input_uploading_image['total_file']) - update_pro_info = { - 'id_token': input_uploading_image['info_request']['id_token'], - 'project_id': input_uploading_image['info_request']['project_id'], - 'project_name':input_uploading_image['info_request']['project_name'], - 's3_key': s3_key, - 'identity_id': input_uploading_image['info_request']['identity_id'] , - 'process_type':input_uploading_image['process_type'] - } - try : - request_update_proj(update_pro_info,s3_info) - except Exception as e : - raise Exception('request update project {}'.format(e)) - -@shared_task(bind=True,retries=1,autoretry_for=(Exception,),exponential_backoff=2,retry_backoff=True,retry_jitter=False,default_retry_delay=1,max_retries=15) -def request_AI_task(self,host,input_json): - logInfo.info("[INFO] The time Retry {} max retries : {} \n".format(self.request.retries,self.max_retries)) - if self.request.retries >= self.max_retries: - logError.error("[ERROR] MAX RETRIES Request Max Retry : {} , Retry times : {} \n".format(self.max_retries,self.request.retries)) - return {"Error_retry":"error"} - try : - outputs = requests.post(host,json=input_json) - except Exception as e: - logError.error("[ERROR] request AI: {}".format(input_json)) - raise Exception(e) - # check enough image - - total_image = input_json['num_augments_per_image']*len(input_json['images_paths']) if 'num_augments_per_image' in input_json else len(input_json['images_paths']) - ## if the task get enough image but response status code is 500 - if get_number_files(input_json['output_folder']) == total_image: - return {"message":"enough"} - - if outputs.status_code == 500: - logError.error("[ERROR] Request AI error : {}".format(outputs.text)) - raise Exception(outputs.text) - - - return outputs.json() \ No newline at end of file diff --git a/ai_caller_service_v1/AI_service/app/tests.py b/ai_caller_service_v1/AI_service/app/tests.py deleted file mode 100644 index 0fabb9f..0000000 --- a/ai_caller_service_v1/AI_service/app/tests.py +++ /dev/null @@ -1 +0,0 @@ -from django.test import TestCase \ No newline at end of file diff --git a/ai_caller_service_v1/AI_service/app/views.py b/ai_caller_service_v1/AI_service/app/views.py deleted file mode 100644 index 939e4ab..0000000 --- a/ai_caller_service_v1/AI_service/app/views.py +++ /dev/null @@ -1,383 +0,0 @@ -import os -import json -from re import L -import boto3 -from django.db import models -from django.db.models import query -from django.shortcuts import render -import requests -import glob -from django.views.decorators.csrf import csrf_exempt -from rest_framework.response import Response -from rest_framework import status -from rest_framework.decorators import api_view, renderer_classes -from rest_framework.generics import ListCreateAPIView, RetrieveUpdateDestroyAPIView -from rest_framework import viewsets -from rest_framework.pagination import PageNumberPagination -from rest_framework.permissions import IsAuthenticated -from rest_framework import serializers -from .scripts.utils import request_update_proj , get_number_files , ThreadRequest, uploading_image , done_task_ec2 , insert_KV_EC2 -from .scripts.dynamodb.task import TasksModel -from .scripts.dynamodb.images import ImageLoader -from .scripts.dynamodb.methods import get_data_method -from .scripts.s3 import S3 -import time -from celery import shared_task,current_task -from celery.result import AsyncResult -import threading -import queue as Queue -from .scripts.logging_api import logDebug , logInfo, logError - -field_input =['id_token','project_name','augment_code','project_id','num_augments_per_image','identity_id'] -image_loader = ImageLoader() -host = { - 'PREPROCESS':'http://{}:8000/ai', - 'AUGMENT':'http://{}:8000/ai' -} -try : - method_db = get_data_method() -except Exception as e : - logError.error("[ERROR] Error of get data from method table :{}\n".format(e)) - -@csrf_exempt -@api_view(('POST',)) -def AI_service_request(request): - data = json.loads(request.body) - for it in field_input: - if not it in data: - return Response(status=status.HTTP_400_BAD_REQUEST,data={ - "message": "Miss "+str(it)+" parameter", - "data": {}, - "error": False - }) - res_dynamodb = manage_update_task.apply_async(queue='high',kwargs={'data':data}) - while not res_dynamodb.successful(): - time.sleep(0.1) - test = res_dynamodb.get() - data['project_prefix'] = test['project_prefix'] - res = manage_task.apply_async(queue='low',kwargs={'data':data,'task_s3_id':test['task_s3_id'],'task_id':res_dynamodb.task_id,'num_images':test['num_images']}) - - return Response(status=status.HTTP_200_OK,data={ - "message": "", - "data": {'task_id':res_dynamodb.task_id}, - "error": True - }) - -@csrf_exempt -@api_view(('POST',)) -def AI_service_check_healthy(request): - data = json.loads(request.body) - if not 'task_id' in data: - return Response(status=status.HTTP_400_BAD_REQUEST,data={ - "message": "Miss task_id parameter", - "data": {}, - "error": True - }) - res = AsyncResult(data['task_id']) - - return Response(status=status.HTTP_200_OK,data={ - "message": "", - "data": {'status':res.state}, - "error": False - }) -""" -project_prefix -process_type -images -augment_code -task_id -""" -@shared_task -def download_s3_task(info_s3): - info = {} - s3_instance = S3(info_s3['project_prefix'],info_s3['process_type']) - try: - info = s3_instance.download_images(info_s3['images'],info_s3['augment_code'],info_s3['task_id']) - except Exception as e: - logError.debug("[ERROR] Task ID download images S3: {} - Error: {}\n".format(info_s3['task_id'],e)) - return info - return info - -@shared_task -def manage_update_task(data): - - task_id = current_task.request.id - logInfo.debug("[INFO] Task ID: {} - Info: {}\n".format(task_id,data)) - - try : - info_image , err = image_loader({ - 'id_token' : data['id_token'], - 'project_id' : data['project_id'], - 'augment_code': data['augment_code'], - 'data_number': data['data_number'], - 'data_type' : data['data_type'], - 'project_name': data['project_name'] - }) - if err != None: - logError.error("[ERROR] Error image loader : {}".format(err)) - return - except Exception as e: - logError.error("[ERROR] Error image loader : {}".format(e)) - return - logDebug.debug("[DEBUG] info_image: {}".format(info_image['images'])) - project_prefix = info_image['project_prefix'] - images = list(set(info_image['images'])) - - augment_code = data['augment_code'] - project_id = data['project_id'] - identity_id = data['identity_id'] - process_type = 'AUGMENT' if 'AUG' in augment_code[0] else 'PREPROCESS' - num_augments_per_image = data['num_augments_per_image'] - - - - # async task because of avoid timeout lambda - task_s3 = download_s3_task.apply_async(queue='download_image',kwargs={'info_s3':{ - 'project_prefix' : project_prefix, - 'process_type' : process_type, - 'images' : images, - 'augment_code': augment_code, - 'task_id': task_id - }}) - - num_images = len(images) - num_gens = num_images if process_type == 'PREPROCESS' else num_augments_per_image*num_images - try : - dynamodb = TasksModel() - dynamodb.create_item(identity_id,task_id,project_id ,num_gens,process_type,"","") - except Exception as e : - logError.error("[ERROR] Task ID: {} - Dynamodb Create Task: {}\n".format(task_id,e)) - return - - return {"task_s3_id":task_s3.task_id,'num_gens':num_gens,'project_prefix':project_prefix,'num_images':num_images} - -@shared_task -def manage_task(data,task_s3_id,task_id,num_images): - dynamodb = TasksModel() - id_token = data['id_token'] - project_prefix = data['project_prefix'] - project_name = data['project_name'] - augment_code = data['augment_code'] - project_id = data['project_id'] - identity_id = data['identity_id'] - process_type = 'AUGMENT' if 'AUG' in augment_code[0] else 'PREPROCESS' - num_augments_per_image = data['num_augments_per_image'] - current_pwd = os.path.join('/home/ec2-user/efs',task_id) - - payload_getIP = json.dumps({"identity_id":data["identity_id"],"task_id":task_id,"num_process_image":num_images }) - - # get IP - - IP = '' - EC2_ID = '' - - # update status preparing dynamodb : - ##PREPARING_HARDWARE update dynamodb######### - dynamodb.update_process(task_id=task_id,identity_id=identity_id,num_finish=0,status='PREPARING_HARDWARE') - logInfo.info("[INFO] TASK ID : {} UPDATE STATUS PREPARING_HARDWARE".format(task_id)) - ############################################### - ##### Request lambda get IP - try : - client = boto3.client('lambda') - response = client.invoke( - FunctionName="staging-balancer-asy-get-ip", - InvocationType="RequestResponse", - Payload=payload_getIP - ) - - json_result = json.loads(response['Payload'].read()) - logDebug.debug("[DEBUG] Response staging-balancer-asy-get-ip: {}".format(json_result)) - logInfo.info("[INFO] staging-balancer-asy-get-ip: {}".format(json_result)) - body =json.loads(json_result['body']) - logInfo.info("[INFO] staging-balancer-asy-get-ip body: {} and type : {}".format(json_result['body'],type(json_result['body']))) - IP = body['data']['ip'] - EC2_ID = body['data']['ec2_id'] - - except Exception as e : - logError.error("[ERROR] staging-balancer-asy-get-ip: {}".format(e)) - #update_process_error(task_id,identity_id,IP,EC2_ID) - dynamodb.update_process_error(task_id=task_id,identity_id=identity_id,IP="",EC2_ID="") - return {"message":"error"} - ############################################################## - # because front end check status after 5 second - time.sleep(6) - # fisinsh get IP - - ## prepare data S3 update dynamodb : PREPARING_DATA######### - dynamodb.update_process(task_id=task_id,identity_id=identity_id,num_finish=0,status='PREPARING_DATA') - logInfo.info("[INFO] TASK ID : {} UPDATE STATUS PREPARING_DATA".format(task_id)) - - res_download_s3_task = AsyncResult(task_s3_id) - while not res_download_s3_task.successful(): - time.sleep(0.01) - # because front end check status after 5 second - time.sleep(6) - info = res_download_s3_task.get() - logDebug.debug("[DEBUG] log res_download_s3_task {}".format(info)) - - if not bool(info): - logError.error("[ERROR] log res_download_s3_task is empty info") - EC2_ID = "" if EC2_ID is None else EC2_ID - dynamodb.update_process_error(task_id= task_id,identity_id= identity_id,IP=IP,EC2_ID=EC2_ID) - return {"message":"error"} - - length_image = info['images_download'] - batch_input = info['batch_input'] - batch_output = info['batch_output'] - batch_size = info['batch_size'] - - - - # check EC2 get success - if bool(EC2_ID): - insert_KV_EC2(EC2_ID,task_id) - logInfo.info('[INFO] insert_KV_EC2: Task id {} - ec2 {}\n'.format(task_id,EC2_ID)) - - # queue_req - queue_req=Queue.Queue(maxsize=0) - # notice the task is finish and exit loop - quit = Queue.Queue(maxsize=0) - error_message_queue = Queue.Queue() - # save log - log_request_AI = Queue.Queue() - stop_update = False - - def AI_service(): - ai_request_queue = Queue.Queue() - logInfo.debug("[INFO] TASK ID {} , request host {}".format(task_id,host[process_type].format(IP))) - info = { - 'in_queue' : ai_request_queue, - 'augment_code' : augment_code, - 'num_augments_per_image' : num_augments_per_image, - 'id_token' : id_token, - 'project_id' : project_id, - 'project_name' : project_name, - 'process_type': process_type, - 'identity_id' : identity_id, - 'error_message_queue': error_message_queue, - 'log_request_AI': log_request_AI, - 'host': host[process_type].format(IP) - } - - for i in range(batch_size): - w = ThreadRequest(worker=info) - w.setDaemon(True) - w.start() - - for (inp , out) in zip(batch_input,batch_output): - ai_request_queue.put((inp,out)) - ai_request_queue.join() - quit.put({"Done":"test"}) - - - - AI_req_thread = threading.Thread(target=AI_service) - AI_req_thread.daemon = True - AI_req_thread.start() - combined = Queue.Queue(maxsize=0) - - def check_dir(): - gen_dir = os.path.join(current_pwd,'gen_images') - total = length_image* num_augments_per_image if process_type == 'AUGMENT' else length_image - while True: - length = get_number_files(gen_dir) - ## length > 1 : update status RUNNING, check image exist######### - - if length > 0 : - queue_req.put(length) - - if length == total: - break - if stop_update == True : break - time.sleep(3) - - check_dir_thread = threading.Thread(target=check_dir) - check_dir_thread.daemon = True - check_dir_thread.start() - - def listen(q): - while True: - combined.put((q,q.get())) - - t1 = threading.Thread(target=listen,args=(queue_req,)) - t1.daemon =True - t1.start() - t2 = threading.Thread(target=listen,args=(quit,)) - t2.daemon =True - t2.start() - t3 = threading.Thread(target=listen,args=(error_message_queue,)) - t3.daemon =True - t3.start() - t4 = threading.Thread(target=listen,args=(log_request_AI,)) - t4.daemon =True - t4.start() - # count_retries will count failed error from batch_output _request - count_retries = 0 - while True: - which , message = combined.get() - # receive the message from queue quit - if which is quit: - try : - total = length_image* num_augments_per_image if process_type == 'AUGMENT' else length_image - dynamodb.update_process_uploading(task_id,identity_id) - uploading_image({ - 'total_file': total, - 'output_dir': os.path.join(current_pwd,'gen_images'), - 'project_prefix': project_prefix, - 'method_db':method_db, - 'process_type':process_type, - 'list_aug':augment_code, - 'info_request':{ - 'id_token' : id_token, - 'project_id' : project_id, - 'project_name' :project_name, - 'identity_id' :identity_id - }, - }) - stop_update = True - # if a request fail after retry state will reponse status is FINISH_ERROR - if count_retries: - gen_dir = os.path.join(current_pwd,'gen_images') - num_finish = get_number_files(gen_dir) - dynamodb.update_process(task_id = task_id,identity_id=identity_id,num_finish= num_finish, status="FINISH_ERROR") - else: - # complete update - dynamodb.update_finish(task_id,identity_id,total,IP,EC2_ID) - logInfo.info('[INFO] Task ID: {} - {}\n'.format(task_id,"Success")) - if bool(EC2_ID) : - logInfo.info('[INFO] done_task_ec2: Task id {} - ec2 {}\n'.format(task_id,EC2_ID)) - done_task_ec2(EC2_ID,task_id) - except Exception as e : - logError.error('[ERROR] Task ID: {} - Fisnish Task: {}\n'.format(task_id,e)) - return {"Error":e} - return {"complete" :message} - # receive the message from thread request to update Tasks table - elif which is queue_req: - try : - # update RUNNING ######### - # - dynamodb.update_process(task_id,identity_id,message,'RUNNING') - except Exception as e : - dynamodb.update_process_error(task_id=task_id,identity_id=identity_id,IP=IP,EC2_ID=EC2_ID) - logError.error('[ERROR] Task ID: {} - Update Process Task: {}\n'.format(task_id,e)) - return {"Error":e} - # get error from error_message_queue - elif which is error_message_queue: - logError.error('[ERROR] TasK ID: - {} - {}\n'.format(task_id,message)) - if message == 'Error_retry': - count_retries += 1 - logInfo.info("[INFO] error message Error_retry : {} batch_output : {}".format(count_retries,len(batch_output))) - if count_retries < len(batch_output): - continue - stop_update = True - try : - dynamodb.update_process_error(task_id=task_id,identity_id=identity_id,IP=IP,EC2_ID=EC2_ID) - except Exception as e : - logError.error('[ERROR] Write Log Dynamodb error : {}'.format(e)) - if bool(EC2_ID): - logInfo.info('[INFO] done_task_ec2: Task id {} - ec2 {}\n'.format(task_id,EC2_ID)) - done_task_ec2(EC2_ID,task_id) - return {"complete": "error"} - elif which is log_request_AI: - logInfo.info("[INFO] Task ID {} ---- Request AI :{}".format(task_id,message)) - return {"complete" : "Finish"} \ No newline at end of file diff --git a/ai_caller_service_v1/AI_service/kill_worker.sh b/ai_caller_service_v1/AI_service/kill_worker.sh deleted file mode 100644 index d3bda69..0000000 --- a/ai_caller_service_v1/AI_service/kill_worker.sh +++ /dev/null @@ -1,2 +0,0 @@ -kill -9 $(ps aux | grep celery | grep -v grep | awk '{print $2}' | tr '\n' ' ') > /dev/null 2>&1 - diff --git a/ai_caller_service_v1/AI_service/manage.py b/ai_caller_service_v1/AI_service/manage.py deleted file mode 100644 index fab10de..0000000 --- a/ai_caller_service_v1/AI_service/manage.py +++ /dev/null @@ -1,22 +0,0 @@ -#!/usr/bin/env python -"""Django's command-line utility for administrative tasks.""" -import os -import sys - - -def main(): - """Run administrative tasks.""" - os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'AI_service.settings') - try: - from django.core.management import execute_from_command_line - except ImportError as exc: - raise ImportError( - "Couldn't import Django. Are you sure it's installed and " - "available on your PYTHONPATH environment variable? Did you " - "forget to activate a virtual environment?" - ) from exc - execute_from_command_line(sys.argv) - - -if __name__ == '__main__': - main() diff --git a/ai_caller_service_v1/AI_service/requirements.txt b/ai_caller_service_v1/AI_service/requirements.txt deleted file mode 100644 index e2825d7..0000000 --- a/ai_caller_service_v1/AI_service/requirements.txt +++ /dev/null @@ -1,38 +0,0 @@ -amqp==2.6.1 -asgiref==3.4.1 -awscli==1.22.14 -billiard==3.6.4.0 -boto3==1.20.14 -botocore==1.23.14 -celery==4.4.6 -certifi==2021.10.8 -charset-normalizer==2.0.9 -click==8.0.3 -click-didyoumean==0.3.0 -click-plugins==1.1.1 -click-repl==0.2.0 -colorama==0.4.3 -Deprecated==1.2.13 -Django==3.2.9 -django-filter==21.1 -djangorestframework==3.12.4 -docutils==0.15.2 -future==0.18.2 -idna==3.3 -jmespath==0.10.0 -kombu==4.6.10 -prompt-toolkit==3.0.23 -pyasn1==0.4.8 -python-dateutil==2.8.2 -pytz==2021.3 -PyYAML==5.4.1 -redis==4.0.2 -requests==2.26.0 -rsa==4.7.2 -s3transfer==0.5.0 -six==1.16.0 -sqlparse==0.4.2 -urllib3==1.26.7 -vine==1.3.0 -wcwidth==0.2.5 -wrapt==1.13.3 diff --git a/ai_caller_service_v1/AI_service/worker.sh b/ai_caller_service_v1/AI_service/worker.sh deleted file mode 100644 index 4202614..0000000 --- a/ai_caller_service_v1/AI_service/worker.sh +++ /dev/null @@ -1,9 +0,0 @@ -celery -A AI_service.worker worker -l INFO -E -Q low --pool=gevent --concurrency=100 -n worker1@0.0.0.0 & -celery -A AI_service.worker worker -l INFO -E -Q high --concurrency=1 -n worker2@0.0.0.0 & -celery -A AI_service.worker worker -l INFO -E -Q request_ai --pool=gevent --concurrency=100 -n worker3@0.0.0.0 & -celery -A AI_service.worker worker -l INFO -E -Q request_ai --pool=gevent --concurrency=100 -n worker4@0.0.0.0 & -celery -A AI_service.worker worker -l INFO -E -Q download_image --pool=gevent --concurrency=100 -n worker5@0.0.0.0 & -celery -A AI_service.worker worker -l INFO -E -Q download_image --pool=gevent --concurrency=100 -n worker6@0.0.0.0 - - - diff --git a/ai_caller_service_v1/build.sh b/ai_caller_service_v1/build.sh deleted file mode 100644 index 7fe5372..0000000 --- a/ai_caller_service_v1/build.sh +++ /dev/null @@ -1,6 +0,0 @@ -pip install -r AI_service/requirements.txt -wget https://download.redis.io/releases/redis-6.2.6.tar.gz -tar xzf redis-6.2.6.tar.gz -cd redis-6.2.6 -make -src/redis-server \ No newline at end of file diff --git a/ai_caller_service_v1/deploy.sh b/ai_caller_service_v1/deploy.sh deleted file mode 100644 index 03bc3e2..0000000 --- a/ai_caller_service_v1/deploy.sh +++ /dev/null @@ -1,3 +0,0 @@ -cd AI_service -celery -A AI_service.worker worker -l INFO -Q test & -python manage.py runserver 0.0.0.0:443 diff --git a/annotation-app/annotation-service/api-handler-functions/add-class/hdler_add_class.py b/annotation-app/annotation-service/api-handler-functions/add-class/hdler_add_class.py new file mode 100644 index 0000000..69fffda --- /dev/null +++ b/annotation-app/annotation-service/api-handler-functions/add-class/hdler_add_class.py @@ -0,0 +1,59 @@ +import boto3 +import json +import os +import uuid + +from config import * +from response import * +from error_messages import * +from utils import convert_current_date_to_iso8601, aws_get_identity_id, create_unique_id +import const + + +from lambda_base_class import LambdaBaseClass +from boto3.dynamodb.conditions import Key, Attr +from models.annotaition.anno_class_info import AnnoClassInfoModel + + + +class AddClassClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.model_class_info = AnnoClassInfoModel(self.env.TABLE_ANNO_CLASS_INFO) + + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + + self.id_token = body[KEY_NAME_ID_TOKEN] + self.category_id = body["category_id"] + self.ls_class_name = body["ls_class_name"] + + def _check_input_value(self): + return + + def handle(self, event, context): + + ### parse body + self.parser(event) + + ### check identity + identity_id = self.get_identity(self.id_token) + + ### add class name from list + ls_ok, ls_fail = self.model_class_info.add_list_class(self.category_id, self.ls_class_name) + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={ + "ls_name_ok": ls_ok, + 'ls_fail': ls_fail + }, + ) + +@error_response +def lambda_handler(event, context): + return AddClassClass().handle(event, context) \ No newline at end of file diff --git a/annotation-app/annotation-service/api-handler-functions/create-label-category/hdler_create_label_category.py b/annotation-app/annotation-service/api-handler-functions/create-label-category/hdler_create_label_category.py new file mode 100644 index 0000000..cb193b6 --- /dev/null +++ b/annotation-app/annotation-service/api-handler-functions/create-label-category/hdler_create_label_category.py @@ -0,0 +1,63 @@ +import boto3 +import json +import os +import uuid + +from config import * +from response import * +from error_messages import * +from utils import convert_current_date_to_iso8601, aws_get_identity_id, create_unique_id +import const + + +from lambda_base_class import LambdaBaseClass +from boto3.dynamodb.conditions import Key, Attr +from models.annotaition.anno_project_model import AnnoProjectModel +from models.annotaition.anno_label_info_model import AnnoLabelInfoModel + + + +class CreateLabelCategoryClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.client_events = boto3.client('events') + self.label_info_model = AnnoLabelInfoModel(self.env.TABLE_ANNO_LABEL_INFO) + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + + self.id_token = body[KEY_NAME_ID_TOKEN] + self.file_id = body["file_id"] + self.category_name = body["category_name"] + self.category_des = body["category_des"] + + def _check_input_value(self): + return + + def handle(self, event, context): + + ### parse body + self.parser(event) + + ### check identity + identity_id = self.get_identity(self.id_token) + + ### create category id indynamoDB + category_id = create_unique_id() + self.label_info_model.create_new_category(self.file_id, category_id, self.category_name, self.category_des) + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={ + "category_id": category_id, + 'category_name': self.category_name, + 'category_des': self.category_des + }, + ) + +@error_response +def lambda_handler(event, context): + return CreateLabelCategoryClass().handle(event, context) \ No newline at end of file diff --git a/annotation-app/annotation-service/api-handler-functions/get-file-info-n-label/hdler_get_file_info_n_label.py b/annotation-app/annotation-service/api-handler-functions/get-file-info-n-label/hdler_get_file_info_n_label.py new file mode 100644 index 0000000..3d93a78 --- /dev/null +++ b/annotation-app/annotation-service/api-handler-functions/get-file-info-n-label/hdler_get_file_info_n_label.py @@ -0,0 +1,68 @@ +import boto3 +import json +import os +import uuid + +from config import * +from response import * +from error_messages import * +from utils import convert_current_date_to_iso8601, aws_get_identity_id, get_num_prj +import const + + +from lambda_base_class import LambdaBaseClass +from boto3.dynamodb.conditions import Key, Attr +from models.annotaition.anno_data_model import AnnoDataModel +from models.annotaition.anno_label_info_model import AnnoLabelInfoModel + + +class GetFileInfoClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.model_data = AnnoDataModel(self.env.TABLE_ANNO_DATA_ORI) + self.model_label_info = AnnoLabelInfoModel(self.env.TABLE_ANNO_LABEL_INFO) + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + + self.id_token = body[KEY_NAME_ID_TOKEN] + self.project_id = body["project_id"] + self.file_name = body["filename"] + self.category_id = body.get("category_id", "") + + def _check_input_value(self): + return + + def handle(self, event, context): + + ### parse body + self.parser(event) + + ### check identity + identity_id = self.get_identity(self.id_token, self.env.USER_POOL_ID, self.env.IDENTITY_POOL_ID) + + ### get file info including link to s3 of segmentation prelabel + file_info = self.model_data.get_item(self.project_id, self.file_name) + file_id = file_info[AnnoDataModel.FIELD_FILE_ID] + + ### get label json info: label with category and json label + if len(self.category_id)==0: + label_info = self.model_label_info.query_all_category_label(file_id) + else: + label_info = self.model_label_info.get_label_info_of_category(file_id, self.category_id) + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={ + "file_info": file_info, + "label_info": label_info + }, + ) + +@error_response +def lambda_handler(event, context): + + return GetFileInfoClass().handle(event, context) \ No newline at end of file diff --git a/annotation-app/annotation-service/api-handler-functions/list-class/hdler_list_class.py b/annotation-app/annotation-service/api-handler-functions/list-class/hdler_list_class.py new file mode 100644 index 0000000..8bf6ee0 --- /dev/null +++ b/annotation-app/annotation-service/api-handler-functions/list-class/hdler_list_class.py @@ -0,0 +1,57 @@ +import boto3 +import json +import os +import uuid + +from config import * +from response import * +from error_messages import * +from utils import convert_current_date_to_iso8601, aws_get_identity_id, create_unique_id +import const + + +from lambda_base_class import LambdaBaseClass +from boto3.dynamodb.conditions import Key, Attr +from models.annotaition.anno_class_info import AnnoClassInfoModel + + + +class ListClassClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.model_class_info = AnnoClassInfoModel(self.env.TABLE_ANNO_CLASS_INFO) + + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + + self.id_token = body[KEY_NAME_ID_TOKEN] + self.category_id = body["category_id"] + + def _check_input_value(self): + return + + def handle(self, event, context): + + ### parse body + self.parser(event) + + ### check identity + identity_id = self.get_identity(self.id_token) + + ### add class name from list + ls_items = self.model_class_info.query_all_class_of_category(self.category_id) + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={ + "ls_class": ls_items + }, + ) + +@error_response +def lambda_handler(event, context): + return ListClassClass().handle(event, context) \ No newline at end of file diff --git a/annotation-app/annotation-service/api-handler-functions/save-label/hdler_save_label.py b/annotation-app/annotation-service/api-handler-functions/save-label/hdler_save_label.py new file mode 100644 index 0000000..5a3c087 --- /dev/null +++ b/annotation-app/annotation-service/api-handler-functions/save-label/hdler_save_label.py @@ -0,0 +1,59 @@ +import boto3 +import json +import os +import uuid + +from config import * +from response import * +from error_messages import * +from utils import convert_current_date_to_iso8601, aws_get_identity_id, get_num_prj +import const + + +from lambda_base_class import LambdaBaseClass +from boto3.dynamodb.conditions import Key, Attr +from models.annotaition.anno_label_info_model import AnnoLabelInfoModel + + + +class SaveLabelClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.client_events = boto3.client('events') + self.model_label_info = AnnoLabelInfoModel(self.env.TABLE_ANNO_LABEL_INFO) + + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + + self.id_token = body[KEY_NAME_ID_TOKEN] + self.dict_s3_key_label = body["dict_s3_key"] ### contain category_id and s3_key_label + self.file_id = body["file_id"] + + def _check_input_value(self): + return + + def handle(self, event, context): + + ### parse body + self.parser(event) + + ### check identity + identity_id = self.get_identity(self.id_token, self.env.USER_POOL_ID, self.env.IDENTITY_POOL_ID) + + ### update label info for each category of file + for category_id, s3_key_json_label in self.dict_s3_key_label.items(): + self.model_label_info.update_label_for_category(self.file_id, category_id, s3_key_json_label) + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={}, + ) + +@error_response +def lambda_handler(event, context): + + return SaveLabelClass().handle(event, context) \ No newline at end of file diff --git a/annotation-app/annotation-service/template_annotaion_service.yaml b/annotation-app/annotation-service/template_annotaion_service.yaml new file mode 100644 index 0000000..be45525 --- /dev/null +++ b/annotation-app/annotation-service/template_annotaion_service.yaml @@ -0,0 +1,138 @@ +AWSTemplateFormatVersion: '2010-09-09' +Transform: AWS::Serverless-2016-10-31 +Description: > + daita-reference-image-service + + Sample SAM Template for daita-reference-image-service + + +Parameters: + minimumLogLevel: + Type: String + Default: DEBUG + Mode: + Type: String + Default: dev + + StagePara: + Type: String + ApplicationPara: + Type: String + + CommonCodeLayerRef: + Type: String + LambdaRoleArn: + Type: String + + TableAnnoDataOriginalName: + Type: String + TableAnnoProjectSumName: + Type: String + TableAnnoProjectsName: + Type: String + TableLabelInfoName: + Type: String + TableCategoryInfoName: + Type: String + TableClassInfoName: + Type: String + TableAIDefaultClassInfoName: + Type: String + + TableDaitaProjectsName: + Type: String + TableDaitaDataOriginalName: + Type: String + + CognitoUserPoolRef: + Type: String + CognitoIdentityPoolIdRef: + Type: String + + S3AnnoBucketName: + Type: String + S3DaitaBucketName: + Type: String + +Globals: + Function: + Timeout: 800 + Runtime: python3.8 + Architectures: + - x86_64 + Layers: + - !Ref CommonCodeLayerRef + Environment: + Variables: + STAGE: !Ref StagePara + LOGGING: !Ref minimumLogLevel + MODE: !Ref Mode + + TABLE_ANNO_PROJECT_SUMMARY: !Ref TableAnnoProjectSumName + TABLE_ANNO_PROJECT: !Ref TableAnnoProjectsName + TABLE_ANNO_DATA_ORI: !Ref TableAnnoDataOriginalName + TABLE_ANNO_LABEL_INFO: !Ref TableLabelInfoName + TABLE_ANNO_CATEGORY_INFO: !Ref TableCategoryInfoName + TABLE_ANNO_CLASS_INFO: !Ref TableClassInfoName + TABLE_ANNO_AI_DEFAULT_CLASS: !Ref TableAIDefaultClassInfoName + + TABLE_DAITA_PROJECT: !Ref TableDaitaProjectsName + TABLE_DAITA_DATA_ORIGINAL: !Ref TableDaitaDataOriginalName + + COGNITO_USER_POOL: !Ref CognitoUserPoolRef + IDENTITY_POOL: !Ref CognitoIdentityPoolIdRef + + S3_ANNO_BUCKET_NAME: !Ref S3AnnoBucketName + S3_DAITA_BUCKET_NAME: !Ref S3DaitaBucketName + +Resources: + #================ LAMBDA API FUNCTIONS ========================================== + FunctionGetFileInfoNLabel: + Type: AWS::Serverless::Function + Properties: + CodeUri: api-handler-functions/get-file-info-n-label + Handler: hdler_get_file_info_n_label.lambda_handler + Role: !Ref LambdaRoleArn + MemorySize: 256 + + FunctionCreateLabelCategory: + Type: AWS::Serverless::Function + Properties: + CodeUri: api-handler-functions/create-label-category + Handler: hdler_create_label_category.lambda_handler + Role: !Ref LambdaRoleArn + MemorySize: 256 + + FunctionSaveLabel: + Type: AWS::Serverless::Function + Properties: + CodeUri: api-handler-functions/save-label + Handler: hdler_save_label.lambda_handler + Role: !Ref LambdaRoleArn + MemorySize: 256 + + FunctionAddClass: + Type: AWS::Serverless::Function + Properties: + CodeUri: api-handler-functions/add-class + Handler: hdler_add_class.lambda_handler + Role: !Ref LambdaRoleArn + MemorySize: 256 + + + +Outputs: + # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function + # Find out more about other implicit resources you can reference within SAM + # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api + FunctionGetFileInfoNLabelArn: + Description: "FunctionGetFileInfoNLabelArn" + Value: !GetAtt FunctionGetFileInfoNLabel.Arn + FunctionCreateLabelCategoryArn: + Value: !GetAtt FunctionCreateLabelCategory.Arn + FunctionSaveLabelArn: + Value: !GetAtt FunctionSaveLabel.Arn + FunctionAddClassArn: + Value: !GetAtt FunctionAddClass.Arn + + diff --git a/annotation-app/api-service/api_route_define.yaml b/annotation-app/api-service/api_route_define.yaml new file mode 100644 index 0000000..0812808 --- /dev/null +++ b/annotation-app/api-service/api_route_define.yaml @@ -0,0 +1,200 @@ +openapi: "3.0.1" +info: + title: + Fn::Sub: "${StagePara}-${ApplicationPara}-HTTP-API" + version: "2021-04-07" +tags: +- name: "httpapi:createdBy" + x-amazon-apigateway-tag-value: "SAM" +paths: + ###=== for project =====### + ### clone a project from daita to annotation + /annotation/project/clone_from_daita: + post: + responses: + default: + description: "Clone project from daita" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ProjectCloneFunctionArn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + + /annotation/project/get_info: + post: + responses: + default: + description: "Get information of data" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${FunctionGetProjectInfoArn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + + /annotation/project/delete_project: + post: + responses: + default: + description: "Delete Project" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${FuncDeleteProject}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + + /annotation/project/list_data: + post: + responses: + default: + description: "List all data in project" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${FunctionProjectListDataArn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + + /annotation/project/list_project: + post: + responses: + default: + description: "List all data in project" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${FunctionListProjectArn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + + /annotation/project/upload_check: + post: + responses: + default: + description: "List all data in project" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${FunctionUploadCheckArn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + + /annotation/project/upload_update: + post: + responses: + default: + description: "List all data in project" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${FunctionUploadUpdateArn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + + /annotation/project/check_ai_segm_progress: + post: + responses: + default: + description: "Check current progress of ai segmentation" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${FunctionCheckAISegmentationProgressArn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + + /annotation/category/add_class: + post: + responses: + default: + description: "add class anme list to category" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${FunctionAddClassArn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + + ### create category of label task + /annotation/file/create_lable_category: + post: + responses: + default: + description: "Create category of label task" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${FunctionCreateLabelCategoryArn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + + ### get information of file and label + /annotation/file/get_file_info_n_label: + post: + responses: + default: + description: "Clone project from daita" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${FunctionGetFileInfoNLabelArn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + + ### create category of label task + /annotation/file/save_label: + post: + responses: + default: + description: "Save current label of file" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${FunctionSaveLabelArn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + +x-amazon-apigateway-cors: + allowMethods: + - "GET" + - "OPTIONS" + - "POST" + allowHeaders: + - "authorization" + - "content-type" + - "x-amz-date" + - "x-amzm-header" + - "x-api-key" + - "x-apigateway-header" + - "*" + allowOrigins: + - "*" + maxAge: 60 + allowCredentials: false +x-amazon-apigateway-importexport-version: "1.0" \ No newline at end of file diff --git a/annotation-app/api-service/template_api_service.yaml b/annotation-app/api-service/template_api_service.yaml new file mode 100644 index 0000000..2d403f1 --- /dev/null +++ b/annotation-app/api-service/template_api_service.yaml @@ -0,0 +1,78 @@ +AWSTemplateFormatVersion: '2010-09-09' +Transform: AWS::Serverless-2016-10-31 +Description: > + daita-caller-service-app + + Sample SAM Template for daita-caller-service-app + +## The general rule seems to be to use !Sub for in line substitutions and !ref for stand alone text +Parameters: + ApplicationPara: + Type: String + StagePara: + Type: String + + ProjectCloneFunctionArn: + Type: String + FunctionGetFileInfoNLabelArn: + Type: String + FunctionCreateLabelCategoryArn: + Type: String + FunctionGetProjectInfoArn: + Type: String + FunctionSaveLabelArn: + Type: String + FunctionProjectListDataArn: + Type: String + FunctionListProjectArn: + Type: String + FunctionAddClassArn: + Type: String + FuncDeleteProject: + Type: String + + FunctionUploadCheckArn: + Type: String + FunctionUploadUpdateArn: + Type: String + + FunctionCheckAISegmentationProgressArn: + Type: String + +Resources: + #================ ROLES ===================================================== + #-- use this role for apigateway access lambda + ApiGatewayCallLambdaRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Principal: + Service: "apigateway.amazonaws.com" + Action: + - "sts:AssumeRole" + Policies: + - PolicyName: RestApiDirectInvokeLambda + PolicyDocument: + Version: "2012-10-17" + Statement: + Action: + - "lambda:InvokeFunction" + Effect: Allow + Resource: "*" + + AnnotationHttpApi: + Type: AWS::Serverless::HttpApi + Properties: + StageName: !Ref StagePara + DefinitionBody: + Fn::Transform: + Name: AWS::Include + Parameters: + Location: './api_route_define.yaml' + +Outputs: + AnnoHttpApiURL: + Value: !Sub "https://${AnnotationHttpApi}.execute-api.${AWS::Region}.amazonaws.com/${StagePara}" \ No newline at end of file diff --git a/annotation-app/build_annotation.sh b/annotation-app/build_annotation.sh new file mode 100644 index 0000000..2c61d2e --- /dev/null +++ b/annotation-app/build_annotation.sh @@ -0,0 +1,84 @@ +### load config file +. "$1" + +OUTPUT_BUILD_DAITA=$2 +OUTPUT_FE_CONFIG=$3 + +### load output of daita-app +. "$OUTPUT_BUILD_DAITA" + + + +cd annotation-app + +parameters_override="Stage=${ANNOTATION_STAGE} Application=${ANNO_APPLICATION} + S3AnnoBucketName=${ANNO_S3_BUCKET} + CommonCodeLayerRef=${CommonCodeLayerRef} CognitoUserPoolRef=${CognitoUserPoolRef} + CognitoIdentityPoolIdRef=${CognitoIdentityPoolIdRef} + TableDaitaProjectsName=${TableDaitaProjectsName} + TableDaitaDataOriginalName=${TableDaitaDataOriginalName} + TableUserName=${TableUserName} + S3DaitaBucketName=${DAITA_S3_BUCKET} + PublicSubnetOne=${PublicSubnetOne} + PublicSubnetTwo=${PublicSubnetTwo} + ContainerSecurityGroup=${ContainerSecurityGroup} + VPC=${VPC} + VPCEndpointSQSDnsEntries=${VPCEndpointSQSDnsEntries} + EFSFileSystemId=${EFSFileSystemId} + EFSAccessPoint=${EFSAccessPoint} + EFSAccessPointArn=${EFSAccessPointArn} + SendEmailIdentityIDFunction=${SendEmailIdentityIDFunction} + ImageAISegmentationUrl=${IMAGE_AI_SEGMENTATION_URL} + MaxSizeEc2AutoScallEcs=${MAX_SIZE_EC2_AUTOSCALING_ECS}" + +sam build --template-file template_annotation_app.yaml +sam deploy --template-file template_annotation_app.yaml --no-confirm-changeset --disable-rollback \ + --config-env $ANNOTATION_STAGE \ + --resolve-image-repos --resolve-s3 \ + --stack-name "$ANNOTATION_STAGE-$ANNO_APPLICATION-app" \ + --s3-prefix "$ANNOTATION_STAGE-$ANNO_APPLICATION-app" \ + --region $AWS_REGION \ + --parameter-overrides $parameters_override | tee output_anno.txt + +### Read output from template +shopt -s extglob + +declare -A dict_output +filename="output_anno.txt" + +while read line; do + # reading each line + if [[ "$line" =~ "Key".+ ]]; then + + [[ "$line" =~ [[:space:]].+ ]] + a=${BASH_REMATCH[0]} + a=${a##*( )} + a=${a%%*( )} + key=$a + fi + if [[ "$line" =~ "Value".+ ]]; then + [[ "$line" =~ [[:space:]].+ ]] + value=${BASH_REMATCH[0]} + value=${value##*( )} + value=${value%%*( )} + + first_line=$value + is_first_line_value=true + else + if [[ "$line" =~ .*"-------".* ]]; then + echo "skip line" + else + if [ "$is_first_line_value" = true ]; then + final_line=$first_line$line + dict_output[$key]=$final_line + is_first_line_value=false + fi + fi + fi +done < $filename + +ApiAnnoAppUrl=${dict_output["ApiAnnoAppUrl"]} + + +###========= SAVE FE CONFIG =============== +echo "REACT_APP_ANNOTATION_PROJECT_API=$ApiAnnoAppUrl" >> $OUTPUT_FE_CONFIG \ No newline at end of file diff --git a/annotation-app/db-service/db_template.yaml b/annotation-app/db-service/db_template.yaml new file mode 100644 index 0000000..b71a86c --- /dev/null +++ b/annotation-app/db-service/db_template.yaml @@ -0,0 +1,194 @@ +AWSTemplateFormatVersion: '2010-09-09' +Transform: AWS::Serverless-2016-10-31 +Description: > + SAM Template for Nested application resources + +Parameters: + StagePara: + Type: String + ApplicationPara: + Type: String + +Resources: + #================ DYNAMODB ================================================== + + ###==== For Project ======== + ProjectDB: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-${ApplicationPara}-project" + AttributeDefinitions: + - + AttributeName: identity_id + AttributeType: S + - + AttributeName: project_name + AttributeType: S + KeySchema: + - + AttributeName: identity_id + KeyType: HASH + - + AttributeName: project_name + KeyType: RANGE + BillingMode: PAY_PER_REQUEST + StreamSpecification: + StreamViewType: NEW_IMAGE + + DeletedProjectDB: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-${ApplicationPara}-deleted-project" + AttributeDefinitions: + - + AttributeName: identity_id + AttributeType: S + - + AttributeName: project_name + AttributeType: S + KeySchema: + - + AttributeName: identity_id + KeyType: HASH + - + AttributeName: project_name + KeyType: RANGE + BillingMode: PAY_PER_REQUEST + + ProjectSummaryDB: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-${ApplicationPara}-prj-sum-all" + AttributeDefinitions: + - + AttributeName: project_id + AttributeType: S + - + AttributeName: type + AttributeType: S + KeySchema: + - + AttributeName: project_id + KeyType: HASH + - + AttributeName: type + KeyType: RANGE + BillingMode: PAY_PER_REQUEST + + DataOriginalDB: + Type: AWS::DynamoDB::Table + Properties: + AttributeDefinitions: + - + AttributeName: project_id + AttributeType: S + - + AttributeName: filename + AttributeType: S + KeySchema: + - + AttributeName: project_id + KeyType: HASH + - + AttributeName: filename + KeyType: RANGE + TableName: !Sub "${StagePara}-${ApplicationPara}-data-original" + BillingMode: PAY_PER_REQUEST + StreamSpecification: + StreamViewType: NEW_IMAGE + + LabelInfoDB: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-${ApplicationPara}-label-info" + AttributeDefinitions: + - + AttributeName: file_id + AttributeType: S + - + AttributeName: category_id + AttributeType: S + KeySchema: + - + AttributeName: file_id + KeyType: HASH + - + AttributeName: category_id + KeyType: RANGE + BillingMode: PAY_PER_REQUEST + + CategoryInfoDB: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-${ApplicationPara}-category-info" + AttributeDefinitions: + - + AttributeName: project_id + AttributeType: S + - + AttributeName: category_id + AttributeType: S + KeySchema: + - + AttributeName: project_id + KeyType: HASH + - + AttributeName: category_id + KeyType: RANGE + BillingMode: PAY_PER_REQUEST + + ClassInfoDB: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-${ApplicationPara}-class-info" + AttributeDefinitions: + - + AttributeName: category_id + AttributeType: S + - + AttributeName: class_name + AttributeType: S + KeySchema: + - + AttributeName: category_id + KeyType: HASH + - + AttributeName: class_name + KeyType: RANGE + BillingMode: PAY_PER_REQUEST + + + +Outputs: + TableAnnoProjectsName: + Description: "Name of table projects" + Value: !Ref ProjectDB + + TableAnnoProjectSumName: + Description: "Name of table projects" + Value: !Ref ProjectSummaryDB + + TableAnnoDataOriginalName: + Description: "Name of table data original" + Value: !Ref DataOriginalDB + + TableLabelInfoName: + Value: !Ref LabelInfoDB + StreamTableAnnoDataOrginal: + Value: !GetAtt DataOriginalDB.StreamArn + + TableCategoryInfoName: + Value: !Ref CategoryInfoDB + + TableClassInfoName: + Value: !Ref ClassInfoDB + + TableAIDefaultClassInfoName: + Value: const-ai-class-info + + ### contain the config value of some paramters in lambda + TableConfigParametersLambdaName: + Value: config-parameters-lambda + + TableDeletedProjectName: + Value: !Ref DeletedProjectDB \ No newline at end of file diff --git a/annotation-app/ecs-segment-app/api-handler-functions/hdler_stream_data_annotation.py b/annotation-app/ecs-segment-app/api-handler-functions/hdler_stream_data_annotation.py new file mode 100644 index 0000000..bcab554 --- /dev/null +++ b/annotation-app/ecs-segment-app/api-handler-functions/hdler_stream_data_annotation.py @@ -0,0 +1,87 @@ +import boto3 +import json +import os +import uuid +import re +import base64 +from datetime import datetime + +from response import * + +from lambda_base_class import LambdaBaseClass + + + + +class HandleStreamDataOriginAnnotation(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + self.client_events = boto3.client('events') + self.client_step_func= boto3.client('stepfunctions') + self.s3 = boto3.client('s3') + self.sqsResourse = boto3.resource('sqs') + + def handle(self, event, context): + records = event['Records'] + listRecord = [] + print(f'logs :{records}') + for record in records: + if record['eventName'] == 'INSERT': + tempItem = { + 'project_id': record['dynamodb']['Keys']['project_id']['S'], + 'filename' : record['dynamodb']['Keys']['filename']['S'], + 's3_key':record['dynamodb']['NewImage']['s3_key']['S'] + } + listRecord.append(tempItem) + + ### push task to sqs to tracking + input_folder= str(base64.b64encode(str(datetime.now()).encode()).decode("ascii")) + if len(listRecord) == 0: + print('Nothing to generate') + return {"message":"ok"} + + self.client_step_func.start_execution( + stateMachineArn=os.environ["ECS_TASK_ARN"], + input=json.dumps( + { + 'input_folder': input_folder, + 'records': listRecord + } + ) + ) + # queue = self.sqsResourse.get_queue_by_name(QueueName=os.environ['QUEUE']) + # request_queue = {'records' :listRecord } + # request_queue['output_directory'] = os.path.join(input_folder,'output') + # queue.send_message( + # MessageBody=json.dumps(request_queue), + # MessageGroupId="push-task-segement-queue", + # DelaySeconds=0, + # ) + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={ + "ok": "ok" + }, + ) + +@error_response +def lambda_handler(event, context): + return HandleStreamDataOriginAnnotation().handle(event=event,context=context) +# cluster = "segment-ecstask-ServiceApplication-1OPTNXI2VFIW7-ECSCluster-20iFW6ZFwY3Q" +# ecsTask = "arn:aws:ecs:us-east-2:737589818430:task-definition/segment-ecstask-ServiceApplication-1OPTNXI2VFIW7-TaskAISegmenationDefinition-4uPNYxdUAF3n:1" +# ecs = boto3.client('ecs') +# command = ["--input_json_path","data/sample/input.json","--output_folder","data/sample/output"] +# response = ecs.run_task( +# cluster = cluster, +# taskDefinition= ecsTask, +# count=1, +# overrides={ +# 'containerOverrides': [ +# { +# 'name': 'test-ecs-task-ecs-segmentations', +# 'command': command, +# }, +# ] +# } +# ) \ No newline at end of file diff --git a/annotation-app/ecs-segment-app/api-handler-functions/hdler_task_segmentation.py b/annotation-app/ecs-segment-app/api-handler-functions/hdler_task_segmentation.py new file mode 100644 index 0000000..83f031f --- /dev/null +++ b/annotation-app/ecs-segment-app/api-handler-functions/hdler_task_segmentation.py @@ -0,0 +1,65 @@ +import boto3 +import re +import os +import json +import glob + +from response import * + +from lambda_base_class import LambdaBaseClass + + + +class EventTaskQueueSegmentationClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + self.s3 = boto3.client('s3') + dns = (os.environ['SQS_VPC_ENDPOINT']).split(":")[1] + self.sqs = boto3.client('sqs',endpoint_url='https://{}'.format(dns)) + self.sqsResourse = boto3.resource('sqs',endpoint_url='https://{}'.format(dns)) + + def handle(self,event,context): + for record in event['Records']: + body = json.loads(record['body']) + print(body) + OutDir = glob.glob(os.path.join(os.environ['EFSPATH'],body['output_directory'])+'/*') + if len(OutDir) == len(body['output_directory']): + self.sqs.delete_message( + QueueUrl=os.environ['QUEUE'], + ReceiptHandle= record['receiptHandle'] + ) + else: + nbReplay = 0 + if 'sqs-dlq-replay-nb' in record['messageAttributes']: + nbReplay = int(record['messageAttributes']['sqs-dlq-replay-nb']["stringValue"]) + print("nb_replay ",nbReplay) + nbReplay += 1 + if nbReplay > 40: + continue + attributes = record['messageAttributes'] + attributes.update({'sqs-dlq-replay-nb': {'StringValue': str(nbReplay), 'DataType': 'Number'}}) + _sqs_attributes_cleaner(attributes) + self.sqs.send_message( + QueueUrl=os.environ['QUEUE_URL'], + MessageBody=record['body'], + MessageAttributes=record['messageAttributes'], + MessageGroupId=record['attributes']['MessageGroupId'], + MessageDeduplicationId=record['attributes']['MessageDeduplicationId'] + ) + return {} + + +@error_response +def lambda_handler(event, context): + return EventTaskQueueSegmentationClass().handle(event=event,context=context) + +def _sqs_attributes_cleaner(attributes): + d = dict.fromkeys(attributes) + for k in d: + if isinstance(attributes[k], dict): + subd = dict.fromkeys(attributes[k]) + for subk in subd: + if not attributes[k][subk]: + del attributes[k][subk] + else: + attributes[k][''.join(subk[:1].upper() + subk[1:])] = attributes[k].pop(subk) \ No newline at end of file diff --git a/annotation-app/ecs-segment-app/controller.yaml b/annotation-app/ecs-segment-app/controller.yaml new file mode 100644 index 0000000..4929ee3 --- /dev/null +++ b/annotation-app/ecs-segment-app/controller.yaml @@ -0,0 +1,226 @@ +AWSTemplateFormatVersion: '2010-09-09' +Transform: AWS::Serverless-2016-10-31 +Description: > + daita-caller-service-app + Sample SAM Template for daita-caller-service-app +## The general rule seems to be to use !Sub for in line substitutions and !ref for stand alone text +Parameters: + + StagePara: + Type: String + Default: test + + ApplicationPara: + Type: String + Default: ecstask + + AITaskECSClusterArn: + Type: String + + AITaskDefinitionArn: + Type: String + + TableAnnoDataOriginalNameStream: + Type: String + + TableUserName: + Type: String + + CommonCodeLayerRef: + Type: String + + LambdaRoleArn: + Type: String + + SecurityGroup: + Type: String + PublicSubnetOne: + Type: String + PublicSubnetTwo: + Type: String + VPCEndpointSQSDnsEntries: + Type: String + + EFSAccessPointArn: + Type: String + EFSAccessPointRootPath: + Type: String + Default: /mnt/data + + Mode: + Type: String + Default: dev + + minimumLogLevel: + Type: String + Default: DEBUG + + ContainerName: + Type: String + + ContainerMount: + Type: String + + + + TableAnnoDataOriginalName: + Type: String + TableAnnoProjectsName: + Type: String + + CognitoIdentityPoolIdRef: + Type: String + CognitoUserPoolRef: + Type: String + +Globals: + Function: + Timeout: 800 + Runtime: python3.8 + Architectures: + - x86_64 + Layers: + - !Ref CommonCodeLayerRef + Environment: + Variables: + STAGE: !Ref StagePara + MODE: !Ref Mode + LOGGING: !Ref minimumLogLevel + USERPOOL: !Ref CognitoUserPoolRef + IDENTIYPOOL: !Ref CognitoIdentityPoolIdRef + CONTAINER_NAME: !Ref ContainerName + CONTAINER_MOUNT: !Ref ContainerMount + SQS_VPC_ENDPOINT: !Ref VPCEndpointSQSDnsEntries + REGION: !Ref 'AWS::Region' + + +Resources: +############################################################## + LogGroup: + Type: AWS::Logs::LogGroup + Properties: + LogGroupName: !Sub "/aws/vendedlogs/states/${StagePara}-${ApplicationPara}-ecs-segmentation-logs" + RetentionInDays: 7 + + HandleECSTaskStatemachine: + Type: AWS::Serverless::StateMachine + Properties: + Type: STANDARD + Name: !Sub "${StagePara}-${ApplicationPara}-Run-ECS-task-Segment" + Policies: + - LambdaInvokePolicy: + FunctionName: !Ref DownloadS3toEFSFunction + FunctionName: !Ref UploadImage + FunctionName: !Ref SendEmailSegmentationComplete + - Statement: + - Sid: CloudWatchLogsPolicy + Effect: Allow + Action: + - "logs:*" + Resource: "*" + - Sid: CloudWatchEventsFullAccess + Effect: Allow + Action: + - "events:*" + - "ecs:*" + - "iam:PassRole" + - "s3:*" + - "lambda:InvokeFunction" + - "ses:*" + - cognito-identity:* + - cognito-idp:* + Resource: "*" + - Sid: IAMPassRoleForCloudWatchEvents + Effect: Allow + Action: + - "iam:PassRole" + Resource: "arn:aws:iam::*:role/AWS_Events_Invoke_Targets" + Tracing: + Enabled: true + DefinitionUri: ./statemachine/ecs_task.asl.yaml + Logging: + Level: ALL + IncludeExecutionData: true + Destinations: + - CloudWatchLogsLogGroup: + LogGroupArn: !GetAtt LogGroup.Arn + DefinitionSubstitutions: + AITaskECSClusterArn: !Ref AITaskECSClusterArn + AITaskDefinitionArn: !Ref AITaskDefinitionArn + SecurityGroupIds: !Join [",", [!Ref SecurityGroup]] + Subnets: !Join [ ",", [!Ref PublicSubnetOne, !Ref PublicSubnetTwo ] ] + DownloadS3toEFSFunction: !Ref DownloadS3toEFSFunction + UploadImage: !Ref UploadImage + SendEmailSegmentationComplete: !Ref SendEmailSegmentationComplete + +#################Lambda function############################### + StreamDataOriginalAnnotationFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: api-handler-functions + Handler: hdler_stream_data_annotation.lambda_handler + Role: !Ref LambdaRoleArn + MemorySize: 256 + Environment: + Variables: + ECS_TASK_ARN: !GetAtt HandleECSTaskStatemachine.Arn + SendEmailSegmentationComplete: + Type: AWS::Serverless::Function + Properties: + CodeUri: statemachine/functions + Handler: hdler_send_email_to_identityid.lambda_handler + Role: !Ref LambdaRoleArn + Environment: + Variables: + TABLE_USER: !Ref TableUserName + + UploadImage: + Type: AWS::Serverless::Function + Properties: + CodeUri: statemachine/functions + Handler: hdler_updoad_image.lambda_handler + Role: !Ref LambdaRoleArn + VpcConfig: + SecurityGroupIds: + - !Ref SecurityGroup + SubnetIds: + - !Ref PublicSubnetOne + - !Ref PublicSubnetTwo + FileSystemConfigs: + - Arn: !Ref EFSAccessPointArn + LocalMountPath: !Ref EFSAccessPointRootPath + Environment: + Variables: + EFSPATH: !Ref EFSAccessPointRootPath + TABLE_ANNOTATION_ORIGIN: !Ref TableAnnoDataOriginalName + TABLE_ANNOTATION_PROJECT: !Ref TableAnnoProjectsName + # FUNC_SEND_EMAIL_IDENTITY_NAME: !Ref SendEmailIdentityIDFunction + + DownloadS3toEFSFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: statemachine/functions + Handler: hdler_download_image_to_efs.lambda_handler + Role: !Ref LambdaRoleArn + VpcConfig: + SecurityGroupIds: + - !Ref SecurityGroup + SubnetIds: + - !Ref PublicSubnetOne + - !Ref PublicSubnetTwo + FileSystemConfigs: + - Arn: !Ref EFSAccessPointArn + LocalMountPath: !Ref EFSAccessPointRootPath + Environment: + Variables: + EFSPATH: !Ref EFSAccessPointRootPath +##################### Event ################################## + ESMappingDBOriginal: + Type: AWS::Lambda::EventSourceMapping + Properties: + BatchSize: 20 + Enabled: True + EventSourceArn: !Ref TableAnnoDataOriginalNameStream + FunctionName: !Ref StreamDataOriginalAnnotationFunction + StartingPosition: LATEST + MaximumBatchingWindowInSeconds: 120 diff --git a/annotation-app/ecs-segment-app/ecs_segementation.yaml b/annotation-app/ecs-segment-app/ecs_segementation.yaml new file mode 100644 index 0000000..9dcc886 --- /dev/null +++ b/annotation-app/ecs-segment-app/ecs_segementation.yaml @@ -0,0 +1,179 @@ +AWSTemplateFormatVersion: '2010-09-09' + +Parameters: + ECSAMI: + Type: String + Default: ami-09cb4bc2dcc083845 + + ContainerSecurityGroup: + Type: String + + InstanceType: + Type: String + Default: c4.xlarge + AllowedValues: [t2.micro, t2.small, t2.medium, t2.large, m3.medium, m3.large, + m3.xlarge, m3.2xlarge, m4.large, m4.xlarge, m4.2xlarge, m4.4xlarge, m4.10xlarge, + c4.large, c4.xlarge, c4.2xlarge, c4.4xlarge, c4.8xlarge, c3.large, c3.xlarge, + c3.2xlarge, c3.4xlarge, c3.8xlarge, r3.large, r3.xlarge, r3.2xlarge, r3.4xlarge, + r3.8xlarge, i2.xlarge, i2.2xlarge, i2.4xlarge, i2.8xlarge] + ConstraintDescription: Please choose a valid instance type. + + PublicSubnetOne: + Type: String + + PublicSubnetTwo: + Type: String + + ### currently, set directly in template + # DesiredCapacity: + # Type: Number + # Default: '1' + # Description: Number of EC2 instances to launch in your ECS cluster. + + StagePara: + Type: String + + ApplicationPara: + Type: String + + EC2Role: + Type: String + + MaxSizeEc2AutoScallEcs: + Type: String + + ExecuteArn: + Type: String + + ImageUrl: + Type: String + + TaskRole: + Type: String + + EFSAccessPoint: + Type: String + EFSFileSystemId: + Type: String + + ContainerPath: + Type: String + +Resources: + + ECSCluster: + Type: AWS::ECS::Cluster + Properties: + ClusterName: !Sub "${StagePara}-${ApplicationPara}-ECS-Segmentation-Cluster" + + ECSAutoScalingGroup: + Type: AWS::AutoScaling::AutoScalingGroup + DependsOn: ECSCluster + Properties: + AutoScalingGroupName: !Sub "${StagePara}-${ApplicationPara}-Segementation-ECS" + # HealthCheckGracePeriod: 60 + # HealthCheckType: EC2 + VPCZoneIdentifier: + - !Ref PublicSubnetOne + - !Ref PublicSubnetTwo + LaunchConfigurationName: !Ref 'ContainerInstances' + NewInstancesProtectedFromScaleIn: false #if true this block scale in termination completely + MinSize: '0' + MaxSize: !Ref MaxSizeEc2AutoScallEcs + DesiredCapacity: 0 + + EC2InstanceProfile: + Type: AWS::IAM::InstanceProfile + Properties: + Path: / + Roles: [!Ref 'EC2Role'] + + ContainerInstances: + Type: AWS::AutoScaling::LaunchConfiguration + Properties: + ImageId: !Ref ECSAMI + SecurityGroups: [!Ref ContainerSecurityGroup] + InstanceType: !Ref InstanceType + IamInstanceProfile: !Ref 'EC2InstanceProfile' + UserData: + Fn::Base64: !Sub | + #!/bin/bash -xe + echo ECS_CLUSTER=${ECSCluster} >> /etc/ecs/ecs.config + yum install -y aws-cfn-bootstrap + /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource ECSAutoScalingGroup --region ${AWS::Region} + + ECSTaskLogGroup: + Type: AWS::Logs::LogGroup + Properties: + LogGroupName: !Sub "/aws/vendedlogs/states/${StagePara}-MyECSTaskLogGroup-${AWS::StackName}" + RetentionInDays: 7 + + TaskAISegmenationDefinition: + Type: AWS::ECS::TaskDefinition + Properties: + ExecutionRoleArn: !Ref ExecuteArn + TaskRoleArn: !Ref TaskRole + NetworkMode: awsvpc + ContainerDefinitions: + - + Name: !Sub "${StagePara}-${ApplicationPara}-ecs-segmentations" + Image: !Ref ImageUrl + Cpu: 4092 + Memory: 4092 + LogConfiguration: + LogDriver: 'awslogs' + Options: + awslogs-group: !Ref ECSTaskLogGroup + awslogs-region: !Ref 'AWS::Region' + awslogs-create-group: true + awslogs-stream-prefix: !Ref 'ApplicationPara' + MountPoints: + - + SourceVolume: "my-vol" + ContainerPath: !Ref ContainerPath + Volumes: + - + EFSVolumeConfiguration: + AuthorizationConfig: + AccessPointId: !Ref EFSAccessPoint + FilesystemId: !Ref EFSFileSystemId + TransitEncryption: ENABLED # enable this so maybe we don't need to config a access point https://docs.aws.amazon.com/pt_br/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-authorizationconfig.html + Name: "my-vol" + + ###________ CAPACITY CONFIG FOR CLUSTER ______________ + ###### doc: https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-target-tracking.html + CapacityProvider: + Type: AWS::ECS::CapacityProvider + DependsOn: ECSCluster + Properties: + Name: !Sub "${StagePara}-${ApplicationPara}-capacity-provider-segmentation" + AutoScalingGroupProvider: + AutoScalingGroupArn: !Ref ECSAutoScalingGroup + ManagedScaling: + # InstanceWarmupPeriod: 300 + MaximumScalingStepSize: 2 + MinimumScalingStepSize: 1 + Status: ENABLED + TargetCapacity: 100 + # ManagedTerminationProtection: ENABLED + + ClusterCapacityProviderAssociation: + Type: AWS::ECS::ClusterCapacityProviderAssociations + Properties: + Cluster: !Ref ECSCluster + CapacityProviders: + - !Ref CapacityProvider + DefaultCapacityProviderStrategy: + - CapacityProvider: !Ref CapacityProvider + Weight: 1 + +Outputs: + + TaskAISegmenationDefinition: + Value: !Ref TaskAISegmenationDefinition + + ECSCluster: + Value: !Ref ECSCluster + + ContainerName: + Value: !Sub "${StagePara}-${ApplicationPara}-ecs-segmentations" \ No newline at end of file diff --git a/annotation-app/ecs-segment-app/network.yaml b/annotation-app/ecs-segment-app/network.yaml new file mode 100644 index 0000000..d304d44 --- /dev/null +++ b/annotation-app/ecs-segment-app/network.yaml @@ -0,0 +1,252 @@ +AWSTemplateFormatVersion: '2010-09-09' +Description: > + ECS-workers-app + + + +Parameters: + ECSAMI: + Type: String + Default: ami-09cb4bc2dcc083845 + + StagePara: + Type: String + + ApplicationPara: + Type: String + +Mappings: + # Hard values for the subnet masks. These masks define + # the range of internal IP addresses that can be assigned. + # The VPC can have all IP's from 10.0.0.0 to 10.0.255.255 + # There are four subnets which cover the ranges: + # + # 10.0.0.0 - 10.0.0.255 + # 10.0.1.0 - 10.0.1.255 + # 10.0.2.0 - 10.0.2.255 + # 10.0.3.0 - 10.0.3.255 + # + # If you need more IP addresses (perhaps you have so many + # instances that you run out) then you can customize these + # ranges to add more + SubnetConfig: + VPC: + CIDR: '10.0.0.0/16' + PublicOne: + CIDR: '10.0.0.0/24' + PublicTwo: + CIDR: '10.0.1.0/24' + PrivateOne: + CIDR: '10.0.2.0/24' + PrivateTwo: + CIDR: '10.0.3.0/24' + +Resources: + VPC: + Type: AWS::EC2::VPC + Properties: + EnableDnsSupport: true + EnableDnsHostnames: true + CidrBlock: !FindInMap ['SubnetConfig', 'VPC', 'CIDR'] + Tags: + - Key: "Name" + Value: !Sub "${StagePara}_${ApplicationPara}_vpc_ecs_segment" + + PublicSubnetOne: + Type: AWS::EC2::Subnet + Properties: + AvailabilityZone: + Fn::Select: + - 0 + - Fn::GetAZs: {Ref: 'AWS::Region'} + VpcId: !Ref VPC + CidrBlock: !FindInMap ['SubnetConfig', 'PublicOne', 'CIDR'] + MapPublicIpOnLaunch: true + Tags: + - Key: "Name" + Value: !Sub "${StagePara}_${ApplicationPara}_pb_subnet_1" + + # NatGatewayOneAttachment: + # Type: AWS::EC2::EIP + # Properties: + # Domain: vpc + # Tags: + # - Key: "Name" + # Value: !Sub "${StagePara}_${ApplicationPara}_eip_natgw_1_ecs" + + # NatGatewayOne: + # Type: AWS::EC2::NatGateway + # Properties: + # AllocationId: !GetAtt NatGatewayOneAttachment.AllocationId + # SubnetId: !Ref PublicSubnetOne + + PublicSubnetTwo: + Type: AWS::EC2::Subnet + Properties: + AvailabilityZone: + Fn::Select: + - 1 + - Fn::GetAZs: {Ref: 'AWS::Region'} + VpcId: !Ref VPC + CidrBlock: !FindInMap ['SubnetConfig', 'PublicTwo', 'CIDR'] + MapPublicIpOnLaunch: true + Tags: + - Key: "Name" + Value: !Sub "${StagePara}_${ApplicationPara}_pb_subnet_2" + + InternetGateway: + Type: AWS::EC2::InternetGateway + + GatewayAttachment: + Type: AWS::EC2::VPCGatewayAttachment + Properties: + VpcId: !Ref VPC + InternetGatewayId: !Ref InternetGateway + + PublicRouteTable: + Type: AWS::EC2::RouteTable + Properties: + VpcId: !Ref VPC + Tags: + - Key: "Name" + Value: !Sub "${StagePara}_${ApplicationPara}_pb_route_table" + + PublicRoute: + Type: AWS::EC2::Route + Properties: + RouteTableId: !Ref PublicRouteTable + DestinationCidrBlock: '0.0.0.0/0' + GatewayId: !Ref InternetGateway + + PublicSubnetOneRouteTableAssociation: + Type: AWS::EC2::SubnetRouteTableAssociation + Properties: + SubnetId: !Ref PublicSubnetOne + RouteTableId: !Ref PublicRouteTable + + PublicSubnetTwoRouteTableAssociation: + Type: AWS::EC2::SubnetRouteTableAssociation + Properties: + SubnetId: !Ref PublicSubnetTwo + RouteTableId: !Ref PublicRouteTable + + ContainerSecurityGroup: + Type: AWS::EC2::SecurityGroup + Properties: + GroupDescription: Access container + VpcId: !Ref VPC + Tags: + - Key: "Name" + Value: !Sub "${StagePara}_${ApplicationPara}_security_group" + + EcsSecurityGroupIngressFromSelf: + Type: AWS::EC2::SecurityGroupIngress + Properties: + Description: Ingress from container in the same security group + GroupId: !Ref ContainerSecurityGroup + IpProtocol: -1 + SourceSecurityGroupId: !Ref ContainerSecurityGroup + + DynamoDBEndpoint: + Type: AWS::EC2::VPCEndpoint + Properties: + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: "*" + Principal: "*" + Resource: "*" + RouteTableIds: + - !Ref 'PublicRouteTable' + ServiceName: !Sub com.amazonaws.${AWS::Region}.dynamodb + VpcId: !Ref VPC + + S3VPCEndpoint: + Type: AWS::EC2::VPCEndpoint + Properties: + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: "*" + Principal: "*" + Resource: "*" + RouteTableIds: + - !Ref 'PublicRouteTable' + # SecurityGroupIds: + # - !Ref ContainerSecurityGroup + # SubnetIds: + # - !Ref PublicSubnetOne + # - !Ref PublicSubnetTwo + ServiceName: !Sub com.amazonaws.${AWS::Region}.s3 + VpcId: !Ref VPC + + ECRPullImageEndpoint: + Type: AWS::EC2::VPCEndpoint + Properties: + VpcEndpointType: Interface + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: "*" + Principal: "*" + Resource: "*" + ServiceName: !Sub com.amazonaws.${AWS::Region}.ecr.dkr + VpcId: !Ref VPC + EC2Endpoint: + Type: AWS::EC2::VPCEndpoint + Properties: + VpcEndpointType: Interface + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: "*" + Principal: "*" + Resource: "*" + ServiceName: !Sub com.amazonaws.${AWS::Region}.ec2 + VpcId: !Ref VPC + + + SQSEndpoint: + Type: AWS::EC2::VPCEndpoint + Properties: + VpcEndpointType: Interface + VpcId: !Ref VPC + SecurityGroupIds: + - !Ref ContainerSecurityGroup + SubnetIds: + - !Ref PublicSubnetOne + - !Ref PublicSubnetTwo + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: "*" + Principal: "*" + Resource: "*" + ServiceName: !Sub com.amazonaws.${AWS::Region}.sqs + +Outputs: + PublicSubnetOne: + Value: !Ref PublicSubnetOne + + PublicSubnetTwo: + Value: !Ref PublicSubnetTwo + + ContainerSecurityGroup: + Value: !Ref ContainerSecurityGroup + + VPC: + Value: !Ref VPC + + VpcEndointSQS: + Value: !Ref SQSEndpoint + + S3VPCEndpoint: + Value: !Ref S3VPCEndpoint + + VPCSQSEndpointDnsEntries: + Value: !Select [0, !GetAtt SQSEndpoint.DnsEntries] \ No newline at end of file diff --git a/annotation-app/ecs-segment-app/role.yaml b/annotation-app/ecs-segment-app/role.yaml new file mode 100644 index 0000000..89278c4 --- /dev/null +++ b/annotation-app/ecs-segment-app/role.yaml @@ -0,0 +1,229 @@ +AWSTemplateFormatVersion: '2010-09-09' + +Resources: + + ECSTaskExecutionRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Statement: + - Effect: Allow + Principal: + Service: [ecs-tasks.amazonaws.com] + AWS: !Sub "arn:aws:iam::${AWS::AccountId}:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS" + Action: ['sts:AssumeRole'] + Path: / + Policies: + - PolicyName: AmazonECSTaskExecutionRolePolicy + PolicyDocument: + Statement: + - Effect: Allow + Action: + - 'ecr:GetAuthorizationToken' + - 'ecr:BatchCheckLayerAvailability' + - 'ecr:GetDownloadUrlForLayer' + - 'ecr:BatchGetImage' + + - 'logs:*' + - 'iam:PassRole' + Resource: '*' + + ECSRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Statement: + - Effect: Allow + Principal: + Service: [ + "ecs.amazonaws.com", + "lambda.amazonaws.com", + "ecs-tasks.amazonaws.com" + ] + AWS: !Sub "arn:aws:iam::${AWS::AccountId}:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS" + Action: ['sts:AssumeRole'] + Path: / + Policies: + - PolicyName: ecs-service + PolicyDocument: + Statement: + - Effect: Allow + Action: + + - 'ec2:AttachNetworkInterface' + - 'ec2:CreateNetworkInterface' + - 'ec2:CreateNetworkInterfacePermission' + - 'ec2:DeleteNetworkInterface' + - 'ec2:DeleteNetworkInterfacePermission' + - 'ec2:Describe*' + - 'ec2:DetachNetworkInterface' + + - 'elasticloadbalancing:DeregisterInstancesFromLoadBalancer' + - 'elasticloadbalancing:DeregisterTargets' + - 'elasticloadbalancing:Describe*' + - 'elasticloadbalancing:RegisterInstancesWithLoadBalancer' + - 'elasticloadbalancing:RegisterTargets' + + - s3:* + - s3-object-lambda:* + - logs:* + + - 'iam:PassRole' + Resource: '*' + + EC2Role: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Statement: + - Effect: Allow + Principal: + Service: [ec2.amazonaws.com] + Action: ['sts:AssumeRole'] + Path: / + ManagedPolicyArns: + - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role + Policies: + - PolicyName: ecs-service + PolicyDocument: + Statement: + - Effect: Allow + Action: + - 'ecs:CreateCluster' + - 'ecs:DeregisterContainerInstance' + - 'ecs:DiscoverPollEndpoint' + - 'ecs:Poll' + - 'ecs:RegisterContainerInstance' + - 'ecs:StartTelemetrySession' + - 'ecs:Submit*' + - 'logs:CreateLogStream' + - 'logs:PutLogEvents' + - 'ecr:GetAuthorizationToken' + - 'ecr:BatchGetImage' + - 'ecr:GetDownloadUrlForLayer' + + - 'iam:PassRole' + Resource: '*' + + AutoscalingRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Statement: + - Effect: Allow + Principal: + Service: [application-autoscaling.amazonaws.com] + Action: ['sts:AssumeRole'] + Path: / + Policies: + - PolicyName: Service-Autoscaling + PolicyDocument: + Statement: + - Effect: Allow + Action: + - 'application-autoscaling:*' + - 'cloudwatch:*' + - 'ecs:DescribeServices' + - 'ecs:UpdateService' + Resource: '*' + + GeneralLambdaExecutionRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Principal: + Service: "lambda.amazonaws.com" + Action: + - "sts:AssumeRole" + Policies: + - PolicyName: 'SecretsManagerParameterAccess' + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - ssm:GetParam* + - ssm:DescribeParam* + Resource: + - arn:aws:ssm:*:*:parameter/* + - PolicyName: 'CloudwatchPermission' + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - 'logs:CreateLogGroup' + - 'logs:CreateLogStream' + - 'logs:PutLogEvents' + Resource: '*' + - PolicyName: 'CognitoPermission' + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - cognito-identity:* + - cognito-idp:* + Resource: '*' + - PolicyName: 'DynamoDBPermission' + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - dynamodb:* + Resource: "*" + - PolicyName: "OtherServicePermission" + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - events:PutEvents + - s3:Get* + - ecr:* + - elasticfilesystem:* + - states:* + - s3:* + Resource: "*" + + ApiGatewayCallLambdaRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Principal: + Service: "apigateway.amazonaws.com" + Action: + - "sts:AssumeRole" + Policies: + - PolicyName: RestApiDirectInvokeLambda + PolicyDocument: + Version: "2012-10-17" + Statement: + Action: + - "lambda:InvokeFunction" + Effect: Allow + Resource: "*" + +Outputs: + EC2Role: + Value: !Ref EC2Role + + ECSTask: + Value: !Ref ECSTaskExecutionRole + + ECSRole: + Value: !Ref ECSRole + + GeneralLambdaExecutionRole: + Value: !Ref GeneralLambdaExecutionRole \ No newline at end of file diff --git a/annotation-app/ecs-segment-app/statemachine/ecs_task.asl.yaml b/annotation-app/ecs-segment-app/statemachine/ecs_task.asl.yaml new file mode 100644 index 0000000..0985596 --- /dev/null +++ b/annotation-app/ecs-segment-app/statemachine/ecs_task.asl.yaml @@ -0,0 +1,57 @@ +StartAt: DownloadS3toEFSFunction +States: + + DownloadS3toEFSFunction: + Type: Task + Resource: 'arn:aws:states:::lambda:invoke' + InputPath: $ + Parameters: + FunctionName: '${DownloadS3toEFSFunction}' + Payload.$: $ + Next: RunECSTask + + RunECSTask: + Type: Task + Resource: arn:aws:states:::ecs:runTask.sync + Parameters: + Cluster: ${AITaskECSClusterArn} + TaskDefinition: ${AITaskDefinitionArn} + NetworkConfiguration: + AwsvpcConfiguration: + Subnets: !Join [ ${Subnets} ] + SecurityGroups: !Join [ ${SecurityGroupIds} ] + # AssignPublicIp: ENABLED + Overrides: + ContainerOverrides: + - Name.$ : "$.Payload.Name" + Command.$: "$.Payload.Command" + Retry: + - ErrorEquals: + - RetriableCallerServiceError + - ErrorEquals: + - States.ALL + IntervalSeconds: 3 + MaxAttempts: 5 + BackoffRate: 1 + Next: UploadImage + + UploadImage: + Type: Task + Resource: 'arn:aws:states:::lambda:invoke' + InputPath: $ + Parameters: + FunctionName: '${UploadImage}' + Payload: + input_folder.$ : $$.Execution.Input.input_folder + records.$: $$.Execution.Input.records + Next: SendEmailSegmentationComplete + + SendEmailSegmentationComplete: + Type: Task + Resource: 'arn:aws:states:::lambda:invoke' + InputPath: $ + Parameters: + FunctionName: '${SendEmailSegmentationComplete}' + Payload.$: $ + End: true +TimeoutSeconds: 6000 \ No newline at end of file diff --git a/annotation-app/ecs-segment-app/statemachine/functions/hdler_download_image_to_efs.py b/annotation-app/ecs-segment-app/statemachine/functions/hdler_download_image_to_efs.py new file mode 100644 index 0000000..477d555 --- /dev/null +++ b/annotation-app/ecs-segment-app/statemachine/functions/hdler_download_image_to_efs.py @@ -0,0 +1,65 @@ +import boto3 +import re +import os +import json + +from response import * + +from lambda_base_class import LambdaBaseClass +s3 = boto3.client('s3') +def save_image_to_efs(s3_key : str,folder : str): + print(s3_key) + bucket, filename = split(s3_key) + basename = os.path.join(folder,os.path.basename(filename)) + new_image = os.path.join(str(os.environ['EFSPATH']),basename) + s3.download_file(bucket,filename,new_image) + + return basename + +def parse_json_ecs_segmentation(records,input_folder): + newJson = {'images':[]} + + for id ,record in enumerate(records): + newJson['images'].append( + { + "image_path":os.path.join(str(os.environ['CONTAINER_MOUNT']),save_image_to_efs('s3://{}'.format(record['s3_key']),input_folder)), + "image_id": id + }, + ) + + fileinput = os.path.join(input_folder,'input.json') + with open(os.path.join(str(os.environ['EFSPATH']), fileinput),'w') as f: + json.dump(newJson,f) + return fileinput + +def split(uri): + if not 's3' in uri[:2]: + temp = uri.split('/') + bucket = temp[0] + filename = '/'.join([temp[i] for i in range(1,len(temp))]) + else: + match = re.match(r's3:\/\/(.+?)\/(.+)', uri) + bucket = match.group(1) + filename = match.group(2) + return bucket, filename + +class DownloadImageEFSClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def handle(self,event, context): + input_folder = event['input_folder'] + os.makedirs(os.path.join(str(os.environ['EFSPATH']),input_folder),exist_ok=True) + inputJson = parse_json_ecs_segmentation(event['records'],input_folder) + inputJsonContainerVolume = os.path.join(str(os.environ['CONTAINER_MOUNT']),inputJson) + output_folder = os.path.join(input_folder,'output') + os.makedirs(os.path.join(str(os.environ['EFSPATH']),output_folder),exist_ok=True) + return { + "output_directory": output_folder, + "Name": os.environ['CONTAINER_NAME'], + "Command": ["--input_json_path",inputJsonContainerVolume,"--output_folder",os.path.join(str(os.environ['CONTAINER_MOUNT']),output_folder)] + } + +@error_response +def lambda_handler(event, context): + return DownloadImageEFSClass().handle(event=event,context=context) \ No newline at end of file diff --git a/annotation-app/ecs-segment-app/statemachine/functions/hdler_send_email_to_identityid.py b/annotation-app/ecs-segment-app/statemachine/functions/hdler_send_email_to_identityid.py new file mode 100644 index 0000000..6a7ebd5 --- /dev/null +++ b/annotation-app/ecs-segment-app/statemachine/functions/hdler_send_email_to_identityid.py @@ -0,0 +1,103 @@ +import boto3 +import json +import os +import uuid + +from config import * +from response import * +from error_messages import * +from utils import convert_current_date_to_iso8601, aws_get_identity_id, get_num_prj +import const + + +from lambda_base_class import LambdaBaseClass +from boto3.dynamodb.conditions import Key, Attr + +table = boto3.client('dynamodb') +cognito_client = boto3.client('cognito-idp') + +class SendEmailIdentityIDClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + # self.identity_id = body["identity_id"] + # self.message_email = body["message_email"] + # self.message_email_text = body["message_email_text"] + + def _check_input_value(self): + return + + def get_mail_User(self, identity_id): + resq = table.scan(TableName=os.environ['TABLE_USER'], + FilterExpression='#id = :id', + ExpressionAttributeNames= + { + '#id':'identity_id' + } , + ExpressionAttributeValues={ + ':id':{'S':identity_id} + }) + + userInfo = resq['Items'] + + print("User info: ", userInfo) + if len(userInfo) > 0: + ID_User = userInfo[0]['ID']['S'] + response = cognito_client.list_users(UserPoolId = os.environ['USERPOOL'], + AttributesToGet = ['email'], + Filter=f'sub=\"{ID_User}\"' + ) + print("response list cognito user: \n", response) + if len(response['Users']) > 0: + user_cognito = response['Users'][0] + mail = user_cognito['Attributes'][0]['Value'] + return mail + + return None + + def send_mail(self, mail, message_email, message_email_text): + client = boto3.client("ses") + print("send email to: ", mail) + response = client.send_email( + Destination={ + "ToAddresses": [mail], + }, + Message={ + "Body": { + "Html": { + "Charset": "UTF-8", + "Data": message_email, + }, + "Text": { + "Charset": "UTF-8", + "Data": message_email_text, + }, + }, + "Subject": { + "Charset": "UTF-8", + "Data": "Your AI detection results are ready", + }, + }, + Source="DAITA Team ", + ) + + return + + def handle(self, event, context): + print(event) + data = event['Payload']['data'] + for it in data: + email = self.get_mail_User(it['identity_id']) + self.send_mail(email, it['message_email'], it['message_email_text']) + + return + + +@error_response +def lambda_handler(event, context): + + return SendEmailIdentityIDClass().handle(event, context) \ No newline at end of file diff --git a/annotation-app/ecs-segment-app/statemachine/functions/hdler_updoad_image.py b/annotation-app/ecs-segment-app/statemachine/functions/hdler_updoad_image.py new file mode 100644 index 0000000..f2b7062 --- /dev/null +++ b/annotation-app/ecs-segment-app/statemachine/functions/hdler_updoad_image.py @@ -0,0 +1,128 @@ +import boto3 +import re +import os +import json +from response import * + +from lambda_base_class import LambdaBaseClass +from models.annotaition.anno_data_model import AnnoDataModel +from models.annotaition.anno_project_model import AnnoProjectModel +table = boto3.client('dynamodb') +s3 = boto3.client('s3') +model_data = AnnoDataModel(os.environ['TABLE_ANNOTATION_ORIGIN']) +model_project = AnnoProjectModel(os.environ['TABLE_ANNOTATION_PROJECT']) +client = boto3.client('cognito-idp') + +def update_s3_gen(project_id, filename, s3_key_gen): + response = table.update_item( + TableName=os.environ["TABLE_ANNOTATION_ORIGIN"], + Key={ + 'project_id': { + 'S': project_id + }, + 'filename': { + 'S': filename + } + }, + ExpressionAttributeNames={ + '#gen': 's3_key_segm', + }, + ExpressionAttributeValues={ + ':gen':{ + 'S': s3_key_gen + } + }, + UpdateExpression='SET #gen = :gen', + ) + print(f'Response ',response) + +def upload_segmentation_s3(data,s3_key): + dirfilename = os.path.dirname(s3_key) + dirfilename = dirfilename.replace('raw_data','clone_project') + basename = os.path.splitext(os.path.basename(s3_key))[0] + '_segment.json' + filename = os.path.join(dirfilename, basename) + bucket = filename.split('/')[0] + key = '/'.join(filename.split('/')[1:]) + s3.put_object( + Body=data, + Bucket=bucket , + Key= key + ) + return filename + +def send_mail(identity_id, project_name): + message_email = """ +

Dear User,

+

Your AI detection task is completed. Please log into DAITA Platform and go to Annotation to access your results.

+

Best,

+

The DAITA Team

+

---

+

In case you encounter any issues or questions, please contact us at contact@daita.tech.

+ """.format( + project_name + ) + message_email_text = """ + Dear User, + Your AI detection task is completed. Please log into DAITA Platform and go to https://app.daita.tech/annotation/project/{} to access your results + Best, + The DAITA Team + In case you encounter any issues or questions, please contact us at contact@daita.tech. + """.format( + project_name + ) + + print("send email to identityid: ", identity_id) + + # response = invoke_lambda_func(os.environ["FUNC_SEND_EMAIL_IDENTITY_NAME"], + # { + # "identity_id": identity_id, + # "message_email": message_email, + # "message_email_text": message_email_text + # }) + + return { + "identity_id": identity_id, + "message_email": message_email, + "message_email_text": message_email_text + } + +def invoke_lambda_func(function_name, body_info, type_request="RequestResponse"): + lambdaInvokeClient = boto3.client('lambda') + print("invoke function name: ", function_name) + lambdaInvokeReq = lambdaInvokeClient.invoke( + FunctionName=function_name, + Payload=json.dumps({'body': body_info}), + InvocationType=type_request, + ) + + return lambdaInvokeReq['Payload'].read() + +def check_finish(project_id): + total, finish = model_data.query_progress_ai_segm(project_id) + + if total == finish and total != 0: + # get identity id + ls_projectResponse = model_project.find_project_by_project_ID(project_id) + if len(ls_projectResponse)>0: + projectResponse = ls_projectResponse[0] + identity_id = projectResponse['identity_id'] + project_name = projectResponse['project_name'] + return send_mail(identity_id, project_name) + return None + + +@error_response +def lambda_handler(event, context): + output_folder = os.path.join(event['input_folder'],'output') + output_folder = os.path.join(os.environ['EFSPATH'],output_folder) + + print(event['records']) + result = {'data':[]} + for index , it in enumerate(event['records']): + with open(os.path.join(output_folder,str(index)+'.json'),'r') as f: + s3_key = upload_segmentation_s3(f.read(),s3_key=it['s3_key']) + update_s3_gen(it['project_id'], it['filename'],s3_key) + response = check_finish(it['project_id']) + if response != None: + result['data'].append(response) + return result \ No newline at end of file diff --git a/annotation-app/ecs-segment-app/storage.yaml b/annotation-app/ecs-segment-app/storage.yaml new file mode 100644 index 0000000..fda8a26 --- /dev/null +++ b/annotation-app/ecs-segment-app/storage.yaml @@ -0,0 +1,78 @@ +AWSTemplateFormatVersion: '2010-09-09' + +Parameters: + VPC: + Type: String + + PublicSubnetOne: + Type: String + + PublicSubnetTwo: + Type: String + + SecurityGroup: + Type: String + + EFSAccessPointRootPath: + Type: String + Default: /app/data + + StagePara: + Type: String + + ApplicationPara: + Type: String + +Resources: + + EFSFileSystem: + Type: AWS::EFS::FileSystem + # FileSystemTags: + # - Key: "Name" + # Value: !Sub "${StagePara}_${ApplicationPara}_efs_ecs_segmentation" + + MountTargeSubnetOne: + Type: AWS::EFS::MountTarget + Properties: + FileSystemId: !Ref EFSFileSystem + SubnetId: !Ref PublicSubnetOne + SecurityGroups: + - !Ref SecurityGroup + + MountTargeSubnetTwo: + Type: AWS::EFS::MountTarget + Properties: + FileSystemId: !Ref EFSFileSystem + SubnetId: !Ref PublicSubnetTwo + SecurityGroups: + - !Ref SecurityGroup + + AccessPoint: + Type: AWS::EFS::AccessPoint + Properties: + FileSystemId: !Ref EFSFileSystem + PosixUser: + Gid: "1000" + Uid: "1000" + RootDirectory: + Path: !Ref EFSAccessPointRootPath + CreationInfo: + OwnerGid: "1000" + OwnerUid: "1000" + Permissions: "777" + +Outputs: + EFSFileSystem: + Value: !Ref EFSFileSystem + + MountTargeSubnetOne: + Value: !Ref MountTargeSubnetOne + + MountTargeSubnetTwo: + Value: !Ref MountTargeSubnetTwo + + AccessPoint: + Value: !Ref AccessPoint + + AccessPointARN: + Value: !GetAtt AccessPoint.Arn \ No newline at end of file diff --git a/annotation-app/ecs-segment-app/template.yaml b/annotation-app/ecs-segment-app/template.yaml new file mode 100644 index 0000000..7585297 --- /dev/null +++ b/annotation-app/ecs-segment-app/template.yaml @@ -0,0 +1,149 @@ +AWSTemplateFormatVersion: '2010-09-09' +Transform: AWS::Serverless-2016-10-31 +Description: > + SAM Template for Nested application resources + +## The general rule seems to be to use !Sub for in line substitutions and !ref for stand alone text +Parameters: + + StagePara: + Type: String + + ImageAISegmentationUrl: + Type: String + MaxSizeEc2AutoScallEcs: + Type: String + + ApplicationPara: + Type: String + + TableAnnoDataOriginalNameStream: + Type: String + + LambdaRoleArn: + Type: String + + CommonCodeLayerRef: + Type: String + + ContainerPath: + Type: String + Default: /app/data + + TableAnnoDataOriginalName: + Type: String + TableAnnoProjectsName: + Type: String + + ### For ecs network + PublicSubnetOne: + Type: String + PublicSubnetTwo: + Type: String + ContainerSecurityGroup: + Type: String + VPC: + Type: String + VPCEndpointSQSDnsEntries: + Type: String + + ### For infra storeage + EFSFileSystemId: + Type: String + EFSAccessPoint: + Type: String + EFSAccessPointArn: + Type: String + + CognitoIdentityPoolIdRef: + Type: String + CognitoUserPoolRef: + Type: String + + TableUserName: + Type: String + + SendEmailIdentityIDFunction: + Type: String + +Resources: + + + #================ APPLICATIONS ============================================= + RoleApplication: + Type: AWS::Serverless::Application + Properties: + Location: ./role.yaml + + # NetworkLayerApplication: + # Type: AWS::Serverless::Application + # Properties: + # Location: ./network.yaml + # Parameters: + # StagePara: !Ref StagePara + # ApplicationPara: !Ref ApplicationPara + + # StorageLayerApplication: + # Type: AWS::Serverless::Application + # Properties: + # Location: ./storage.yaml + # Parameters: + # VPC: !Ref VPC + # PublicSubnetOne: !Ref PublicSubnetOne + # PublicSubnetTwo: !Ref PublicSubnetTwo + # SecurityGroup: !Ref ContainerSecurityGroup + + # StagePara: !Ref StagePara + # ApplicationPara: !Ref ApplicationPara + + ECSServiceApplication: + Type: AWS::Serverless::Application + Properties: + Location: ./ecs_segementation.yaml + Parameters: + StagePara: !Ref StagePara + ApplicationPara: !Ref ApplicationPara + EC2Role: !GetAtt RoleApplication.Outputs.EC2Role + ExecuteArn: !GetAtt RoleApplication.Outputs.ECSTask + TaskRole: !GetAtt RoleApplication.Outputs.ECSRole + ImageUrl: !Ref ImageAISegmentationUrl + ContainerPath: !Ref ContainerPath + ### storage infra + EFSAccessPoint: !Ref EFSAccessPoint + EFSFileSystemId: !Ref EFSFileSystemId + ### for ecs network infra + PublicSubnetOne: !Ref PublicSubnetOne + PublicSubnetTwo: !Ref PublicSubnetTwo + ContainerSecurityGroup: !Ref ContainerSecurityGroup + MaxSizeEc2AutoScallEcs: !Ref MaxSizeEc2AutoScallEcs + + ECSControllerApplication: + Type: AWS::Serverless::Application + Properties: + Location: ./controller.yaml + Parameters: + StagePara: !Ref StagePara + ApplicationPara: !Ref ApplicationPara + LambdaRoleArn: !Ref LambdaRoleArn + CommonCodeLayerRef: !Ref CommonCodeLayerRef + TableAnnoDataOriginalNameStream: !Ref TableAnnoDataOriginalNameStream + TableAnnoDataOriginalName: !Ref TableAnnoDataOriginalName + AITaskDefinitionArn: !GetAtt ECSServiceApplication.Outputs.TaskAISegmenationDefinition + AITaskECSClusterArn: !GetAtt ECSServiceApplication.Outputs.ECSCluster + ### For network infra + SecurityGroup: !Ref ContainerSecurityGroup + PublicSubnetOne: !Ref PublicSubnetOne + PublicSubnetTwo: !Ref PublicSubnetTwo + VPCEndpointSQSDnsEntries: !Ref VPCEndpointSQSDnsEntries + + ContainerName: !GetAtt ECSServiceApplication.Outputs.ContainerName + ContainerMount: !Ref ContainerPath + + EFSAccessPointArn: !Ref EFSAccessPointArn + TableAnnoProjectsName: !Ref TableAnnoProjectsName + CognitoUserPoolRef: !Ref CognitoUserPoolRef + CognitoIdentityPoolIdRef: !Ref CognitoIdentityPoolIdRef + + TableUserName: !Ref TableUserName + + # SendEmailIdentityIDFunction: !Ref SendEmailIdentityIDFunction \ No newline at end of file diff --git a/annotation-app/project-service/api-handler-functions/check-ai-segmentation/hdler_check_si_segm_progress.py b/annotation-app/project-service/api-handler-functions/check-ai-segmentation/hdler_check_si_segm_progress.py new file mode 100644 index 0000000..fe03d25 --- /dev/null +++ b/annotation-app/project-service/api-handler-functions/check-ai-segmentation/hdler_check_si_segm_progress.py @@ -0,0 +1,60 @@ +import boto3 +import json +import os +import uuid + +from config import * +from response import * +from error_messages import * +from utils import convert_current_date_to_iso8601, aws_get_identity_id, get_num_prj +import const + + +from lambda_base_class import LambdaBaseClass +from boto3.dynamodb.conditions import Key, Attr +from models.annotaition.anno_data_model import AnnoDataModel +from models.annotaition.anno_label_info_model import AnnoLabelInfoModel + + + +class CheckAISegmentProgressClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.model_data = AnnoDataModel(self.env.TABLE_ANNO_DATA_ORI) + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + + self.id_token = body[KEY_NAME_ID_TOKEN] + self.project_id = body["project_id"] + + + def _check_input_value(self): + return + + def handle(self, event, context): + + ### parse body + self.parser(event) + + ### check identity + identity_id = self.get_identity(self.id_token, self.env.USER_POOL_ID, self.env.IDENTITY_POOL_ID) + + total, finish = self.model_data.query_progress_ai_segm(self.project_id) + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={ + 'total': total, + 'finished': finish + }, + ) + + +@error_response +def lambda_handler(event, context): + + return CheckAISegmentProgressClass().handle(event, context) \ No newline at end of file diff --git a/annotation-app/project-service/api-handler-functions/clone-project/hdler_project_clone.py b/annotation-app/project-service/api-handler-functions/clone-project/hdler_project_clone.py new file mode 100644 index 0000000..784fb97 --- /dev/null +++ b/annotation-app/project-service/api-handler-functions/clone-project/hdler_project_clone.py @@ -0,0 +1,141 @@ +import boto3 +import json +import os +import uuid + +from config import * +from response import * +from error_messages import * +from utils import convert_current_date_to_iso8601, aws_get_identity_id, get_num_prj +import const + + +from lambda_base_class import LambdaBaseClass +from boto3.dynamodb.conditions import Key, Attr +from models.annotaition.anno_project_model import AnnoProjectModel +from models.project_model import ProjectModel, ProjectItem +from models.data_model import DataModel, DataItem + + +class ProjectCloneClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.client_events = boto3.client('events') + self.client_step_func = boto3.client('stepfunctions') + self.anno_project_model = AnnoProjectModel(self.env.TABLE_ANNO_PROJECT) + self.daita_project_model = ProjectModel(self.env.TABLE_DAITA_PROJECT) + + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + + self.id_token = body[KEY_NAME_ID_TOKEN] + self.anno_project_name = body["anno_project_name"] + self.daita_project_name = body["daita_project_name"] + self.project_info = body["project_info"] + + def _check_input_value(self): + # ### check number max + # if self.number_random <= 0 or self.number_random >= prebuild_dataset[PrebuildDatasetModel.FIELD_TOTAL_IMAGES]: + # self.number_random = prebuild_dataset[PrebuildDatasetModel.FIELD_TOTAL_IMAGES] + + # ### udpate the link to s3 + # self.s3_key = prebuild_dataset[PrebuildDatasetModel.FIELD_S3_KEY] + # self.visual_name = prebuild_dataset[PrebuildDatasetModel.FIELD_VISUAL_NAME] + + ### + try: + # check length of projectname and project info + if len(self.anno_project_name) > const.MAX_LENGTH_PROJECT_NAME_INFO: + raise Exception(const.MES_LENGTH_OF_PROJECT_NAME) + if len(self.project_info) > const.MAX_LENGTH_PROJECT_DESCRIPTION: + raise Exception(const.MES_LENGTH_OF_PROJECT_INFO) + except Exception as e: + print('Error: ', repr(e)) + raise Exception(repr(e)) + + return + + def handle(self, event, context): + + ### parse body + self.parser(event) + + ### check identity + identity_id = self.get_identity(self.id_token) + + ### check daita_project exist or not + project_rec = self.daita_project_model.get_project_info(identity_id, self.daita_project_name) + if project_rec is None: + raise Exception(MESS_PROJECT_NOT_EXIST.format(self.daita_project_name)) + + ### check limit project + # num_prj=get_num_prj(identity_id) + # if num_prj >= const.MAX_NUM_PRJ_PER_USER: + # raise Exception(const.MES_REACH_LIMIT_NUM_PRJ) + + ### create project on DB + _uuid = uuid.uuid4().hex + project_id = f'{self.anno_project_name}_{_uuid}' + s3_prj_root = f'{self.env.S3_ANNO_BUCKET_NAME}/{identity_id}/{project_id}' + s3_prefix = f'{self.env.S3_ANNO_BUCKET_NAME}/{identity_id}/{project_id}/{const.FOLDER_RAW_DATA_NAME}' + s3_label = f'{self.env.S3_ANNO_BUCKET_NAME}/{identity_id}/{project_id}/{const.FOLDER_LABEL_NAME}' + db_resource = boto3.resource('dynamodb') + try: + gen_status = AnnoProjectModel.VALUE_GEN_STATUS_GENERATING + item = { + AnnoProjectModel.FIELD_PROJECT_ID: project_id, + AnnoProjectModel.FIELD_IDENTITY_ID: identity_id, + AnnoProjectModel.FIELD_PROJECT_NAME: self.anno_project_name, + AnnoProjectModel.FIELD_S3_PREFIX: s3_prefix, + AnnoProjectModel.FIELD_S3_LABEL: s3_label, + AnnoProjectModel.FIELD_S3_PRJ_ROOT: s3_prj_root, + AnnoProjectModel.FIELD_PROJECT_INFO: self.project_info, + AnnoProjectModel.FIELD_CREATED_DATE: convert_current_date_to_iso8601(), + AnnoProjectModel.FIELD_GEN_STATUS: gen_status, + AnnoProjectModel.FIELD_LINKED_PROJECT: project_rec.get_value_w_default(ProjectItem.FIELD_PROJECT_ID, "") + } + condition = Attr(AnnoProjectModel.FIELD_PROJECT_NAME).not_exists() & Attr(AnnoProjectModel.FIELD_IDENTITY_ID).not_exists() + self.anno_project_model.put_item_w_condition(item, condition=condition) + except db_resource.meta.client.exceptions.ConditionalCheckFailedException as e: + print('Error condition: ', e) + raise Exception(MES_DUPLICATE_PROJECT_NAME.format(self.project_name)) + except Exception as e: + print('Error: ', repr(e)) + raise Exception(repr(e)) + + ### call async step function + stepfunction_input = { + "identity_id": identity_id, + "anno_project_id": project_id, + "anno_project_name": self.anno_project_name, + "anno_bucket_name": self.env.S3_ANNO_BUCKET_NAME, + "daita_project_id": project_rec.get_value_w_default(ProjectItem.FIELD_PROJECT_ID, ""), + "s3_prefix_create": s3_prefix, + # "s3_prefix_prebuild": self.s3_key if (f"s3://{self.bucket_name}" not in self.s3_key) else self.s3_key.replace(f"s3://{self.bucket_name}/", "") + } + response = self.client_step_func.start_execution( + stateMachineArn=self.env.SM_CLONE_PROJECT_ARN, + input=json.dumps(stepfunction_input) + ) + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={ + "project_id": project_id, + "s3_prefix": s3_prefix, + "s3_prj_root": s3_prj_root, + "s3_label": s3_label, + "gen_status": gen_status, + "project_name": self.anno_project_name, + "link_daita_prj_id": project_rec.get_value_w_default(ProjectItem.FIELD_PROJECT_ID) + }, + ) + +@error_response +def lambda_handler(event, context): + + return ProjectCloneClass().handle(event, context) \ No newline at end of file diff --git a/annotation-app/project-service/api-handler-functions/delete-project/hdler_delete_project.py b/annotation-app/project-service/api-handler-functions/delete-project/hdler_delete_project.py new file mode 100644 index 0000000..212879c --- /dev/null +++ b/annotation-app/project-service/api-handler-functions/delete-project/hdler_delete_project.py @@ -0,0 +1,55 @@ +import boto3 +import json +import os +import uuid + +from config import * +from response import * +from error_messages import * +from utils import convert_current_date_to_iso8601, aws_get_identity_id, get_num_prj +import const + + +from lambda_base_class import LambdaBaseClass +from models.annotaition.anno_project_model import AnnoProjectModel +from models.annotaition.anno_data_model import AnnoDataModel +from models.annotaition.anno_project_sum_model import AnnoProjectSumModel + + +class DeleteProjectAnnotation(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + self.anno_project_model = AnnoProjectModel(self.env.TABLE_ANNO_PROJECT) + self.model_data = AnnoDataModel(self.env.TABLE_ANNO_DATA_ORI) + self.model_anno_prj_sum = AnnoProjectSumModel(self.env.TABLE_ANNO_PROJECT_SUMMARY) + self.model_anno_deleted_prj = AnnoProjectModel(self.env.TABLE_ANNO_DELETED_PRJ) + + @LambdaBaseClass.parse_body + def parser(self, body): + self.id_token = body["id_token"] + self.project_name = body["project_name"] + self.project_id = body["project_id"] + + def handle(self, event, context): + ### parse body + self.parser(event) + + identity_id = self.get_identity(self.id_token, self.env.USER_POOL_ID, self.env.IDENTITY_POOL_ID) + + item_delete = self.anno_project_model.delete_project(identity_id, self.project_name) + self.model_anno_deleted_prj.put_item_w_condition(item_delete) + self.model_data.delete_project(self.project_id) + self.model_anno_prj_sum.update_deleted_status(self.project_id, AnnoProjectSumModel.VALUE_TYPE_ORIGINAL) + + ### TODO delete in annotation + ### delete category + ### delete class + ### delete label info + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={}, + ) + +def lambda_handler(event, context): + return DeleteProjectAnnotation().handle(event=event, context=context) \ No newline at end of file diff --git a/annotation-app/project-service/api-handler-functions/get-project-info/hdler_get_project_info.py b/annotation-app/project-service/api-handler-functions/get-project-info/hdler_get_project_info.py new file mode 100644 index 0000000..b561aad --- /dev/null +++ b/annotation-app/project-service/api-handler-functions/get-project-info/hdler_get_project_info.py @@ -0,0 +1,81 @@ + + +from lambda_base_class import LambdaBaseClass +import json +import boto3 +import os + +from boto3.dynamodb.conditions import Key, Attr +from utils import convert_response, aws_get_identity_id, dydb_get_project_full +from error_messages import * +from response import * + +from models.annotaition.anno_project_model import AnnoProjectModel +from models.annotaition.anno_project_sum_model import AnnoProjectSumModel +from models.annotaition.anno_class_info import AnnoClassInfoModel + + +class GetProjectInfoClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + self.anno_project_model = AnnoProjectModel(self.env.TABLE_ANNO_PROJECT) + self.anno_project_sum_model = AnnoProjectSumModel(self.env.TABLE_ANNO_PROJECT_SUMMARY) + self.model_class_info = AnnoClassInfoModel(self.env.TABLE_ANNO_CLASS_INFO) + + def parser(self, body): + self.id_token = body["id_token"] + self.project_name = body["project_name"] + + def handle(self, event, context): + self.parser(json.loads(event['body'])) + + identity_id = self.get_identity(self.id_token, self.env.USER_POOL_ID, self.env.IDENTITY_POOL_ID) + + ### get project_id + project_info = self.anno_project_model.get_project_info(identity_id, self.project_name, + [AnnoProjectModel.FIELD_PROJECT_ID, AnnoProjectModel.FIELD_CATEGORY_DEFAULT, AnnoProjectModel.FIELD_S3_PREFIX, + AnnoProjectModel.FIELD_S3_LABEL, AnnoProjectModel.FIELD_GEN_STATUS]) + if project_info is None: + raise Exception(MESS_PROJECT_NOT_EXIST.format(self.project_name)) + project_id = project_info[AnnoProjectModel.FIELD_PROJECT_ID] + + # get info detail of a project + items = self.anno_project_sum_model.query_data_project_id(project_id) + if len(items)>0: + groups = {} + for item in items: + type = item['type'] + groups[type] = { + 'count': int(item['count']), + 'size': int(item['total_size']) + } + else: + groups = None + + gen_status = project_info[AnnoProjectModel.FIELD_GEN_STATUS] + if gen_status == AnnoProjectModel.VALUE_GEN_STATUS_FINISH: + ls_categorys = { + "category_id": project_info.get(AnnoProjectModel.FIELD_CATEGORY_DEFAULT, ""), + "ls_class": self.model_class_info.query_all_class_of_category(project_info[AnnoProjectModel.FIELD_CATEGORY_DEFAULT]) + } + else: + ls_categorys = {} + + print("ls_category", ls_categorys) + return convert_response({'data': { + "identity_id": identity_id, + "project_name": self.project_name, + "project_id": project_id, + "s3_raw_data": project_info[AnnoProjectModel.FIELD_S3_PREFIX], + "s3_label": project_info[AnnoProjectModel.FIELD_S3_LABEL], + "gen_status": project_info[AnnoProjectModel.FIELD_GEN_STATUS], + "ls_category": ls_categorys, + "groups": groups, + }, + "error": False, + "success": True, + "message": None }) + +@error_response +def lambda_handler(event, context): + return GetProjectInfoClass().handle(event=event, context=context) \ No newline at end of file diff --git a/annotation-app/project-service/api-handler-functions/list-data/hdler_list_data.py b/annotation-app/project-service/api-handler-functions/list-data/hdler_list_data.py new file mode 100644 index 0000000..223d4b4 --- /dev/null +++ b/annotation-app/project-service/api-handler-functions/list-data/hdler_list_data.py @@ -0,0 +1,62 @@ +import boto3 +import json +import os +import uuid + +from config import * +from response import * +from error_messages import * +from utils import convert_current_date_to_iso8601, aws_get_identity_id, get_num_prj +import const + + +from lambda_base_class import LambdaBaseClass +from boto3.dynamodb.conditions import Key, Attr +from models.annotaition.anno_data_model import AnnoDataModel +from models.annotaition.anno_label_info_model import AnnoLabelInfoModel + +MAX_NUMBER_LIMIT = 200 + +class ListDataClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.model_data = AnnoDataModel(self.env.TABLE_ANNO_DATA_ORI) + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + + self.id_token = body[KEY_NAME_ID_TOKEN] + self.project_id = body["project_id"] + self.next_token = body["next_token"] + self.num_limit = min(MAX_NUMBER_LIMIT, body.get( + "num_limit", MAX_NUMBER_LIMIT)) + + def _check_input_value(self): + return + + def handle(self, event, context): + + ### parse body + self.parser(event) + + ### check identity + identity_id = self.get_identity(self.id_token, self.env.USER_POOL_ID, self.env.IDENTITY_POOL_ID) + + next_token, query_items = self.model_data.query_data_follow_batch(self.project_id, self.next_token, self.num_limit) + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={ + 'items': query_items, + 'next_token': next_token + }, + ) + + +@error_response +def lambda_handler(event, context): + + return ListDataClass().handle(event, context) \ No newline at end of file diff --git a/annotation-app/project-service/api-handler-functions/list-project/hdler_list_project.py b/annotation-app/project-service/api-handler-functions/list-project/hdler_list_project.py new file mode 100644 index 0000000..52827b0 --- /dev/null +++ b/annotation-app/project-service/api-handler-functions/list-project/hdler_list_project.py @@ -0,0 +1,56 @@ +import boto3 +import json +import os +import uuid + +from config import * +from response import * +from error_messages import * +from utils import convert_current_date_to_iso8601, aws_get_identity_id, get_num_prj +import const + + +from lambda_base_class import LambdaBaseClass +from boto3.dynamodb.conditions import Key, Attr +from models.annotaition.anno_project_model import AnnoProjectModel + +MAX_NUMBER_LIMIT = 200 + +class ListProject(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.model_project = AnnoProjectModel(self.env.TABLE_ANNO_PROJECT) + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + + self.id_token = body[KEY_NAME_ID_TOKEN] + + def _check_input_value(self): + return + + def handle(self, event, context): + + ### parse body + self.parser(event) + + ### check identity + identity_id = self.get_identity(self.id_token, self.env.USER_POOL_ID, self.env.IDENTITY_POOL_ID) + + ls_projects = self.model_project.get_all_project(identity_id) + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={ + 'items': ls_projects + }, + ) + + +@error_response +def lambda_handler(event, context): + + return ListProject().handle(event, context) \ No newline at end of file diff --git a/annotation-app/project-service/api-handler-functions/upload-check/hdler_upload_check.py b/annotation-app/project-service/api-handler-functions/upload-check/hdler_upload_check.py new file mode 100644 index 0000000..f6c4305 --- /dev/null +++ b/annotation-app/project-service/api-handler-functions/upload-check/hdler_upload_check.py @@ -0,0 +1,63 @@ + + +from typing import List +from lambda_base_class import LambdaBaseClass +import json +import boto3 +import os + +from boto3.dynamodb.conditions import Key, Attr +from utils import convert_response, aws_get_identity_id, dydb_get_project_full, convert_current_date_to_iso8601 +from error_messages import * + +from models.annotaition.anno_project_sum_model import AnnoProjectSumModel +from models.annotaition.anno_data_model import AnnoDataModel +from response import * +import const + + +class ProjectUploadUpdateClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + self.model_data = AnnoDataModel(self.env.TABLE_ANNO_DATA_ORI) + self.model_project_sum = AnnoProjectSumModel(self.env.TABLE_ANNO_PROJECT_SUMMARY) + + @LambdaBaseClass.parse_body + def parser(self, body): + self.id_token = body["id_token"] + self.project_id = body["project_id"] + self.ls_filename: List[str] = body["ls_filename"] + + def _check_input_value(self): + self.identity_id = self.get_identity(self.id_token, self.env.USER_POOL_ID, self.env.IDENTITY_POOL_ID) + + # check quantiy of items + MAX_NUMBER_ITEM_QUERY = 200 + if len(self.ls_filename) > MAX_NUMBER_ITEM_QUERY: + raise Exception( + f'The number of items is over {MAX_NUMBER_ITEM_QUERY}') + + + def handle(self, event, context): + ### parse body + self.parser(event) + + # query data from DB + ls_data = self.model_data.get_item_from_list(self.project_id, self.ls_filename) + + # check available image is over the limitation + current_num_data = self.model_project_sum.get_current_number_data_in_prj(self.project_id) + if len(self.ls_filename)-len(ls_data)+current_num_data > const.MAX_NUM_IMAGES_IN_ORIGINAL: + raise (Exception( + f'The number of images should not exceed {const.MAX_NUM_IMAGES_IN_ORIGINAL}!')) + + return convert_response({ + 'data': ls_data, + "error": False, + "success": True, + "message": None + }) + +@error_response +def lambda_handler(event, context): + return ProjectUploadUpdateClass().handle(event=event, context=context) \ No newline at end of file diff --git a/annotation-app/project-service/api-handler-functions/upload-update/hdler_upload_update.py b/annotation-app/project-service/api-handler-functions/upload-update/hdler_upload_update.py new file mode 100644 index 0000000..8ad7406 --- /dev/null +++ b/annotation-app/project-service/api-handler-functions/upload-update/hdler_upload_update.py @@ -0,0 +1,82 @@ + + +from lambda_base_class import LambdaBaseClass +import json +import boto3 +import os + +from boto3.dynamodb.conditions import Key, Attr +from utils import convert_response, aws_get_identity_id, dydb_get_project_full, convert_current_date_to_iso8601 +from error_messages import * +from response import * + +from models.annotaition.anno_project_sum_model import AnnoProjectSumModel +from models.annotaition.anno_data_model import AnnoDataModel + + + +class ProjectUploadUpdateClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + self.model_data = AnnoDataModel(self.env.TABLE_ANNO_DATA_ORI) + self.model_project_sum = AnnoProjectSumModel(self.env.TABLE_ANNO_PROJECT_SUMMARY) + + @LambdaBaseClass.parse_body + def parser(self, body): + self.id_token = body["id_token"] + self.project_id = body['project_id'] + self.project_name = body['project_name'] + self.ls_object_info = body['ls_object_info'] + + def _check_input_value(self): + self.identity_id = self.get_identity(self.id_token, self.env.USER_POOL_ID, self.env.IDENTITY_POOL_ID) + + # check quantiy of items + MAX_NUMBER_ITEM_PUT = 200 + if len(self.ls_object_info) > MAX_NUMBER_ITEM_PUT: + raise Exception( + f'The number of items is over {MAX_NUMBER_ITEM_PUT}') + if len(self.ls_object_info) == 0: + raise Exception('The number of items must not empty') + + def handle(self, event, context): + ### parse body + self.parser(event) + + ### update summary information + total_size = 0 + count = 0 + for object in self.ls_object_info: + size_old = object.get('size_old', 0) + total_size += (object['size']-size_old) + if size_old <= 0: + count += 1 + + ### check number images in original must smaller than a limitation + item = self.model_project_sum.get_item_prj_sum_info(self.project_id, ls_fields_projection=[]) + if item: + current_num_data = item.get(AnnoProjectSumModel.FIELD_NUM_EXIST_DATA, 0) + thumbnail_key = item.get(AnnoProjectSumModel.FIELD_THUM_KEY, None) + thumbnail_filename = item.get(AnnoProjectSumModel.FIELD_THUM_FILENAME, None) + else: + current_num_data = 0 + thumbnail_key = self.ls_object_info[0]["s3_key"] + thumbnail_filename = self.ls_object_info[0]["filename"] + + num_final = current_num_data + len(self.ls_object_info) + + ### put item from ls object + self.model_data.put_item_from_ls_object(self.project_id, self.ls_object_info) + + ### update summary information + self.model_project_sum.update_upload_new_data(self.project_id, total_size=total_size, count = count, num_final=num_final, + thum_filename=thumbnail_filename, thum_s3_key=thumbnail_key) + + return convert_response({'data': {}, + "error": False, + "success": True, + "message": None}) + +@error_response +def lambda_handler(event, context): + return ProjectUploadUpdateClass().handle(event=event, context=context) \ No newline at end of file diff --git a/annotation-app/project-service/statemachine/clone_project_data/functions/hdler_move_s3_data.py b/annotation-app/project-service/statemachine/clone_project_data/functions/hdler_move_s3_data.py new file mode 100644 index 0000000..f64093d --- /dev/null +++ b/annotation-app/project-service/statemachine/clone_project_data/functions/hdler_move_s3_data.py @@ -0,0 +1,104 @@ +import boto3 +import json +import os +import random + +from config import * +from response import * +from error_messages import * +from identity_check import * +from s3_utils import separate_s3_uri + +from system_parameter_store import SystemParameterStore +from lambda_base_class import LambdaBaseClass +from models.data_model import DataModel, DataItem + +from utils import create_unique_id, get_bucket_key_from_s3_uri, split_ls_into_batch + +class MoveS3DataClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.client_events = boto3.client('events') + self.const = SystemParameterStore() + self.s3 = boto3.client('s3') + + self.daita_data_original_model = DataModel(self.env.TABLE_DAITA_DATA_ORIGINAL) + + @LambdaBaseClass.parse_body + def parser(self, body): + + self.logger.debug(f"body in main_parser: {body}") + + self.s3_prefix_created_store = body["s3_prefix_create"] ### s3 of anno, where store image + self.anno_project_id = body["anno_project_id"] + self.anno_project_name = body["anno_project_name"] + self.identity_id = body["identity_id"] + self.daita_project_id = body["daita_project_id"] + + def _check_input_value(self): + pass + + def move_data_s3(self): + """ + Move data original of daita project to anno project + """ + ls_info = [] + + + #get all data in original data of daita project + ls_task_params = [] + ls_data_original_items = self.daita_data_original_model.get_all_data_in_project(self.daita_project_id) + + for obj in ls_data_original_items: + old_key = separate_s3_uri(obj[DataItem.FIELD_S3_KEY], self.env.S3_DAITA_BUCKET_NAME)[1] + old_source = { 'Bucket': self.env.S3_DAITA_BUCKET_NAME, + 'Key': old_key} + new_key = os.path.join(separate_s3_uri(self.s3_prefix_created_store, self.env.S3_ANNO_BUCKET_NAME)[1], old_key.split("/")[-1]) + size = obj[DataItem.FIELD_SIZE] + + ### copy data to new s3 folder + boto3.resource('s3').meta.client.copy(old_source, self.env.S3_ANNO_BUCKET_NAME, new_key) + + ## add to list info + ls_info.append((new_key.split('/')[-1], f"{self.env.S3_ANNO_BUCKET_NAME}/{new_key}", size)) + + return ls_info + + def handle(self, event, context): + + ### parse body + self.parser(event) + + # move data in s3 + ls_info = self.move_data_s3() + + if len(ls_info)>0: + bucket, folder = get_bucket_key_from_s3_uri(self.s3_prefix_created_store) + s3_key_path = os.path.join(folder, f"clone_project/RI_{create_unique_id()}.json") + self.s3.put_object( + Body=json.dumps(ls_info), + Bucket= bucket, + Key= s3_key_path + ) + else: + bucket = None + s3_key_path = None + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={ + "identity_id": self.identity_id, + "anno_project_id": self.anno_project_id, + "anno_project_name": self.anno_project_name, + "s3_key_path": s3_key_path + }, + is_in_stepfunction=True + ) + +def lambda_handler(event, context): + + return MoveS3DataClass().handle(event, context) + + \ No newline at end of file diff --git a/annotation-app/project-service/statemachine/clone_project_data/functions/hdler_update_input_data.py b/annotation-app/project-service/statemachine/clone_project_data/functions/hdler_update_input_data.py new file mode 100644 index 0000000..ebbbf50 --- /dev/null +++ b/annotation-app/project-service/statemachine/clone_project_data/functions/hdler_update_input_data.py @@ -0,0 +1,106 @@ +import boto3 +import json +import os + +from config import * +from response import * +from error_messages import * +from identity_check import * + +from system_parameter_store import SystemParameterStore +from lambda_base_class import LambdaBaseClass +from models.annotaition.anno_data_model import AnnoDataModel +from models.annotaition.anno_project_sum_model import AnnoProjectSumModel +from models.annotaition.anno_project_model import AnnoProjectModel +from models.annotaition.anno_category_info import AnnoCategoryInfoModel +from models.annotaition.anno_class_info import AnnoClassInfoModel +from utils import get_bucket_key_from_s3_uri, split_ls_into_batch, convert_current_date_to_iso8601, create_unique_id + + +class MoveUpdateDataClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.client_events = boto3.client('events') + self.const = SystemParameterStore() + self.s3 = boto3.client('s3') + + self.anno_data_model = AnnoDataModel(self.env.TABLE_ANNO_DATA_ORI) + self.anno_prj_sum_model = AnnoProjectSumModel(self.env.TABLE_ANNO_PROJECT_SUMMARY) + self.anno_project_model = AnnoProjectModel(self.env.TABLE_ANNO_PROJECT) + self.model_anno_category_info = AnnoCategoryInfoModel(self.env.TABLE_ANNO_CATEGORY_INFO) + self.model_class_info = AnnoClassInfoModel(self.env.TABLE_ANNO_CLASS_INFO) + self.model_ai_default_class = AnnoClassInfoModel(self.env.TABLE_ANNO_AI_DEFAULT_CLASS) + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + self.identity_id = body[KEY_NAME_IDENTITY_ID] + self.anno_project_id = body["anno_project_id"] + self.s3_key_path = body["s3_key_path"] + self.anno_project_name = body["anno_project_name"] + + def _check_input_value(self): + pass + + def handle(self, event, context): + + ### parse body + self.parser(event) + + resultS3 = self.s3.get_object(Bucket=self.env.S3_ANNO_BUCKET_NAME, Key=self.s3_key_path) + ls_info = json.loads(resultS3["Body"].read().decode()) + + # update to DB + # create the batch request from input data and summary the information + ls_item_request = [] + total_size = 0 + count = 0 + for object in ls_info: + # update summary information + size_old = 0 + total_size += (object[2]-size_old) + if size_old <= 0: + count += 1 + + type_method = VALUE_TYPE_DATA_ORIGINAL + item_request = { + AnnoDataModel.FIELD_PROJECT_ID: self.anno_project_id, # partition key + AnnoDataModel.FIELD_S3_KEY: object[1], # sort_key + AnnoDataModel.FIELD_FILE_ID: create_unique_id(), + AnnoDataModel.FIELD_FILENAME: object[0], + AnnoDataModel.FIELD_HASH: '', # we use function get it mean that this field is optional in body + AnnoDataModel.FIELD_SIZE: object[2], # size must be in Byte unit + AnnoDataModel.FIELD_IS_ORIGINAL: True, + AnnoDataModel.FIELD_CREATED_TIME: convert_current_date_to_iso8601() + } + ls_item_request.append(item_request) + + ### write data detail in to DB + self.anno_data_model.batch_write(ls_item_request) + + ### update summary information + self.anno_prj_sum_model.update_project_sum(self.anno_project_id, VALUE_TYPE_DATA_ORIGINAL, total_size, count, ls_item_request[0]['s3_key'], ls_item_request[0]['filename']) + + ### create default category + category_id = self.model_anno_category_info.create_new_category(self.anno_project_id, "default", "this category was created by default") + + ### add default AI class to DB + ls_default_items = self.model_ai_default_class.get_all_AI_default_class() + self.model_class_info.add_default_AI_class(category_id, ls_default_items) + + # update generate status + self.anno_project_model.update_project_gen_status_category_default(self.identity_id, self.anno_project_name, AnnoProjectModel.VALUE_GEN_STATUS_FINISH, category_id) + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={}, + is_in_stepfunction=True + ) + +def lambda_handler(event, context): + + return MoveUpdateDataClass().handle(event, context) + + \ No newline at end of file diff --git a/annotation-app/project-service/statemachine/clone_project_data/functions/hdler_update_sumary_db.py b/annotation-app/project-service/statemachine/clone_project_data/functions/hdler_update_sumary_db.py new file mode 100644 index 0000000..176c681 --- /dev/null +++ b/annotation-app/project-service/statemachine/clone_project_data/functions/hdler_update_sumary_db.py @@ -0,0 +1,114 @@ +import boto3 +import json +import os + +from config import * +from response import * +from error_messages import * +from identity_check import * + +from system_parameter_store import SystemParameterStore +from lambda_base_class import LambdaBaseClass +from models.project_model import ProjectModel, ProjectItem +from models.project_sum_model import ProjectSumModel +from utils import get_bucket_key_from_s3_uri, split_ls_into_batch + +db_client = boto3.client('dynamodb') +db_resource = boto3.resource('dynamodb') + +class UpdateSummaryDBClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.client_events = boto3.client('events') + self.const = SystemParameterStore() + self.s3 = boto3.client('s3') + self.project_sum_model = ProjectSumModel(os.environ["TABLE_PROJECT_SUMMARY"]) + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + self.identity_id = body[KEY_NAME_IDENTITY_ID] + self.project_id = body[KEY_NAME_PROJECT_ID] + self.project_name = body["project_name"] + self.total_size = body["total_size"] + self.count = body["count"] + self.thu_key = body["thu_key"] + self.thu_name = body["thu_name"] + + def _check_input_value(self): + pass + + def handle(self, event, context): + + ### parse body + self.parser(event) + + # update summary information + try: + response = db_client.update_item( + TableName=os.environ["TABLE_PROJECT_SUMMARY"], + Key={ + 'project_id': { + 'S': self.project_id + }, + 'type': { + 'S': VALUE_TYPE_DATA_ORIGINAL + } + }, + ExpressionAttributeNames={ + '#SI': 'total_size', + '#COU': 'count', + '#TK': 'thu_key', + '#TN': 'thu_name' + }, + ExpressionAttributeValues={ + ':si': { + 'N': str(self.total_size) + }, + ':cou': { + 'N': str(self.count) + }, + ':tk': { + 'S': self.thu_key + }, + ':tn': { + 'S': self.thu_name + } + }, + UpdateExpression='SET #TK = :tk, #TN = :tn ADD #SI :si, #COU :cou', + ) + print('response_summary: ', response) + except Exception as e: + print('Error: ', repr(e)) + raise Exception(repr(e)) + + # update generate status + try: + table = db_resource.Table(os.environ['TABLE_PROJECT']) + response = table.update_item( + Key={ + 'identity_id': self.identity_id, + 'project_name': self.project_name, + }, + ExpressionAttributeValues={ + ':st': VALUE_STATUS_CREATE_SAMPLE_PRJ_FINISH, + }, + UpdateExpression='SET gen_status = :st' + ) + except Exception as e: + print('Error: ', repr(e)) + raise Exception(repr(e)) + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={}, + is_in_stepfunction=True + ) + +def lambda_handler(event, context): + + return UpdateSummaryDBClass().handle(event, context) + + \ No newline at end of file diff --git a/annotation-app/project-service/statemachine/clone_project_data/sm_clone_project.asl.yaml b/annotation-app/project-service/statemachine/clone_project_data/sm_clone_project.asl.yaml new file mode 100644 index 0000000..d888e38 --- /dev/null +++ b/annotation-app/project-service/statemachine/clone_project_data/sm_clone_project.asl.yaml @@ -0,0 +1,49 @@ +StartAt: GetDataTask +States: + GetDataTask: + Type: Task + Resource: 'arn:aws:states:::lambda:invoke' + InputPath: $ + ResultSelector: + body.$: $.Payload.body.data + ResultPath: $ + OutputPath: $ + Parameters: + FunctionName: "${Arn_FuncMoveS3Data}" + Payload: + body.$: $ + Next: MoveUpdateDataMap + Comment: >- + Check the level of parallelism, split requests into chunks and invoke + lamndas + Retry: + - ErrorEquals: + - RetriableCallerServiceError + IntervalSeconds: 1 + MaxAttempts: 2 + BackoffRate: 1 + + MoveUpdateDataMap: + Type: Task + Resource: 'arn:aws:states:::lambda:invoke' + InputPath: $.body + ResultPath: $ + OutputPath: $.Payload + Parameters: + FunctionName: "${Arn_FuncUpdateInputData}" + Payload: + body.$: $ + End: true + + # TaskUpdateSummaryDB: + # Type: Task + # Resource: 'arn:aws:states:::lambda:invoke' + # InputPath: $.body.data + # OutputPath: $.Payload.body + # Parameters: + # FunctionName: "${Arn_FuncUpdateSumaryDatabase}" + # Payload: + # body.$: $ + # End: true + +TimeoutSeconds: 6000 diff --git a/annotation-app/project-service/template_project_service.yaml b/annotation-app/project-service/template_project_service.yaml new file mode 100644 index 0000000..5193034 --- /dev/null +++ b/annotation-app/project-service/template_project_service.yaml @@ -0,0 +1,274 @@ +AWSTemplateFormatVersion: '2010-09-09' +Transform: AWS::Serverless-2016-10-31 +Description: > + daita-reference-image-service + + Sample SAM Template for daita-reference-image-service + + +Parameters: + minimumLogLevel: + Type: String + Default: DEBUG + Mode: + Type: String + Default: dev + + StagePara: + Type: String + ApplicationPara: + Type: String + + CommonCodeLayerRef: + Type: String + LambdaRoleArn: + Type: String + + TableAnnoDataOriginalName: + Type: String + TableAnnoProjectSumName: + Type: String + TableAnnoProjectsName: + Type: String + TableCategoryInfoName: + Type: String + TableAIDefaultClassInfoName: + Type: String + TableClassInfoName: + Type: String + TableDeletedProjectName: + Type: String + + TableConfigParametersLambdaName: + Type: String + + TableDaitaProjectsName: + Type: String + TableDaitaDataOriginalName: + Type: String + + CognitoUserPoolRef: + Type: String + CognitoIdentityPoolIdRef: + Type: String + + S3AnnoBucketName: + Type: String + S3DaitaBucketName: + Type: String + +Globals: + Function: + Timeout: 800 + Runtime: python3.8 + Architectures: + - x86_64 + Layers: + - !Ref CommonCodeLayerRef + Environment: + Variables: + STAGE: !Ref StagePara + LOGGING: !Ref minimumLogLevel + MODE: !Ref Mode + + TABLE_ANNO_PROJECT_SUMMARY: !Ref TableAnnoProjectSumName + TABLE_ANNO_PROJECT: !Ref TableAnnoProjectsName + TABLE_ANNO_DATA_ORI: !Ref TableAnnoDataOriginalName + TABLE_ANNO_CATEGORY_INFO: !Ref TableCategoryInfoName + TABLE_ANNO_AI_DEFAULT_CLASS: !Ref TableAIDefaultClassInfoName + TABLE_ANNO_CLASS_INFO: !Ref TableClassInfoName + TABLE_ANNO_DELETED_PRJ: !Ref TableDeletedProjectName + + TABLE_DAITA_PROJECT: !Ref TableDaitaProjectsName + TABLE_DAITA_DATA_ORIGINAL: !Ref TableDaitaDataOriginalName + + TABLE_CONFIG_PARA_LAMBDA: !Ref TableConfigParametersLambdaName + + COGNITO_USER_POOL: !Ref CognitoUserPoolRef + IDENTITY_POOL: !Ref CognitoIdentityPoolIdRef + + S3_ANNO_BUCKET_NAME: !Ref S3AnnoBucketName + S3_DAITA_BUCKET_NAME: !Ref S3DaitaBucketName + +Resources: + #================ LAMBDA API FUNCTIONS ========================================== + ProjectCloneFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: api-handler-functions/clone-project + Handler: hdler_project_clone.lambda_handler + Role: !Ref LambdaRoleArn + MemorySize: 256 + Environment: + Variables: + SM_CLONE_PROJECT_ARN: !GetAtt CloneProjectSM.Arn + + FunctionGetProjectInfo: + Type: AWS::Serverless::Function + Properties: + CodeUri: api-handler-functions/get-project-info + Handler: hdler_get_project_info.lambda_handler + Role: !Ref LambdaRoleArn + MemorySize: 256 + + FunctionProjectListData: + Type: AWS::Serverless::Function + Properties: + CodeUri: api-handler-functions/list-data + Handler: hdler_list_data.lambda_handler + Role: !Ref LambdaRoleArn + MemorySize: 256 + + FunctionListProject: + Type: AWS::Serverless::Function + Properties: + CodeUri: api-handler-functions/list-project + Handler: hdler_list_project.lambda_handler + Role: !Ref LambdaRoleArn + MemorySize: 256 + + FunctionUploadCheck: + Type: AWS::Serverless::Function + Properties: + CodeUri: api-handler-functions/upload-check + Handler: hdler_upload_check.lambda_handler + Role: !Ref LambdaRoleArn + MemorySize: 256 + + FunctionUploadUpdate: + Type: AWS::Serverless::Function + Properties: + CodeUri: api-handler-functions/upload-update + Handler: hdler_upload_update.lambda_handler + Role: !Ref LambdaRoleArn + MemorySize: 256 + + FuncDeleteProject: + Type: AWS::Serverless::Function + Properties: + Handler: hdler_delete_project.lambda_handler + CodeUri: api-handler-functions/delete-project + Role: !Ref LambdaRoleArn + MemorySize: 256 + + FunctionCheckAISegmentationProgress: + Type: AWS::Serverless::Function + Properties: + Handler: hdler_check_si_segm_progress.lambda_handler + CodeUri: api-handler-functions/check-ai-segmentation + Role: !Ref LambdaRoleArn + MemorySize: 256 + + + #================ LAMBDA STEP FUNCTIONS ========================================== + FuncMoveS3Data: + Type: AWS::Serverless::Function + Properties: + Timeout: 900 + Handler: hdler_move_s3_data.lambda_handler + CodeUri: statemachine/clone_project_data/functions + Role: !Ref LambdaRoleArn + MemorySize: 256 + + FuncUpdateInputData: + Type: AWS::Serverless::Function + Properties: + Handler: hdler_update_input_data.lambda_handler + CodeUri: statemachine/clone_project_data/functions + Role: !Ref LambdaRoleArn + MemorySize: 256 + + FuncUpdateSumaryDatabase: + Type: AWS::Serverless::Function + Properties: + Handler: hdler_update_sumary_db.lambda_handler + CodeUri: statemachine/clone_project_data/functions + Role: !Ref LambdaRoleArn + MemorySize: 256 + + + #================ PROCESSOR STATE MACHINE =================================== + CloneProjectSMLogGroup: + Type: AWS::Logs::LogGroup + Properties: + LogGroupName: !Sub "/aws/vendedlogs/states/${StagePara}-${ApplicationPara}-CloneProject" + RetentionInDays: 7 + + CloneProjectSM: + Type: AWS::Serverless::StateMachine + Properties: + Type: STANDARD + Name: !Sub "${StagePara}-${ApplicationPara}-CloneProjectSM" + Policies: + - LambdaInvokePolicy: + FunctionName: !Ref FuncMoveS3Data + - LambdaInvokePolicy: + FunctionName: !Ref FuncUpdateInputData + - LambdaInvokePolicy: + FunctionName: !Ref FuncUpdateSumaryDatabase + - Statement: + - Sid: CloudWatchLogsPolicy + Effect: Allow + Action: + - "logs:CreateLogDelivery" + - "logs:GetLogDelivery" + - "logs:UpdateLogDelivery" + - "logs:DeleteLogDelivery" + - "logs:ListLogDeliveries" + - "logs:PutResourcePolicy" + - "logs:DescribeResourcePolicies" + - "logs:DescribeLogGroup" + - "logs:DescribeLogGroups" + Resource: "*" + - Sid: CloudWatchEventsFullAccess + Effect: Allow + Action: + - "events:*" + Resource: "*" + - Sid: IAMPassRoleForCloudWatchEvents + Effect: Allow + Action: + - "iam:PassRole" + Resource: "arn:aws:iam::*:role/AWS_Events_Invoke_Targets" + Tracing: + Enabled: true + DefinitionUri: ./statemachine/clone_project_data/sm_clone_project.asl.yaml + Logging: + Level: ALL + IncludeExecutionData: true + Destinations: + - CloudWatchLogsLogGroup: + LogGroupArn: !GetAtt CloneProjectSMLogGroup.Arn + DefinitionSubstitutions: + Arn_FuncMoveS3Data: !GetAtt FuncMoveS3Data.Arn + Arn_FuncUpdateInputData: !GetAtt FuncUpdateInputData.Arn + Arn_FuncUpdateSumaryDatabase: !GetAtt FuncUpdateSumaryDatabase.Arn + + +Outputs: + # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function + # Find out more about other implicit resources you can reference within SAM + # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api + ProjectCloneFunctionArn: + Description: "ProjectCloneFunctionArn" + Value: !GetAtt ProjectCloneFunction.Arn + + FunctionGetProjectInfoArn: + Value: !GetAtt FunctionGetProjectInfo.Arn + + FunctionProjectListDataArn: + Value: !GetAtt FunctionProjectListData.Arn + + FunctionListProjectArn: + Value: !GetAtt FunctionListProject.Arn + FuncDeleteProject: + Value: !GetAtt FuncDeleteProject.Arn + + FunctionUploadCheckArn: + Value: !GetAtt FunctionUploadCheck.Arn + FunctionUploadUpdateArn: + Value: !GetAtt FunctionUploadUpdate.Arn + + FunctionCheckAISegmentationProgressArn: + Value: !GetAtt FunctionCheckAISegmentationProgress.Arn + diff --git a/annotation-app/samconfig.toml b/annotation-app/samconfig.toml new file mode 100644 index 0000000..6b7ba84 --- /dev/null +++ b/annotation-app/samconfig.toml @@ -0,0 +1,19 @@ +version = 0.1 + +[dev.deploy.parameters] +confirm_changeset = true +capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" +disable_rollback = true +image_repositories = [] + +[prod.deploy.parameters] +confirm_changeset = true +capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" +disable_rollback = true +image_repositories = [] + +[dev1.deploy.parameters] +confirm_changeset = true +capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" +disable_rollback = true +image_repositories = [] \ No newline at end of file diff --git a/annotation-app/template_annotation_app.yaml b/annotation-app/template_annotation_app.yaml new file mode 100644 index 0000000..a70a712 --- /dev/null +++ b/annotation-app/template_annotation_app.yaml @@ -0,0 +1,257 @@ +AWSTemplateFormatVersion: '2010-09-09' +Transform: AWS::Serverless-2016-10-31 +Description: > + SAM Template for Annotation application + +## The general rule seems to be to use !Sub for in line substitutions and !ref for stand alone text +Parameters: + Stage: + Type: String + Application: + Type: String + + S3AnnoBucketName: + Type: String + S3DaitaBucketName: + Type: String + + CommonCodeLayerRef: + Type: String + + CognitoUserPoolRef: + Type: String + CognitoIdentityPoolIdRef: + Type: String + + TableDaitaProjectsName: + Type: String + TableDaitaDataOriginalName: + Type: String + TableUserName: + Type: String + + SendEmailIdentityIDFunction: + Type: String + + ### parameter from infra + PublicSubnetOne: + Type: String + PublicSubnetTwo: + Type: String + ContainerSecurityGroup: + Type: String + VPC: + Type: String + VPCEndpointSQSDnsEntries: + Type: String + + ### paras for infra storage + EFSFileSystemId: + Type: String + EFSAccessPoint: + Type: String + EFSAccessPointArn: + Type: String + + ImageAISegmentationUrl: + Type: String + MaxSizeEc2AutoScallEcs: + Type: String + +Resources: + + #================ ROLES ===================================================== + # lambda role + GeneralLambdaExecutionRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Principal: + Service: "lambda.amazonaws.com" + Action: + - "sts:AssumeRole" + Policies: + - PolicyName: 'SecretsManagerParameterAccess' + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - ssm:GetParam* + - ssm:DescribeParam* + Resource: + - arn:aws:ssm:*:*:parameter/* + - PolicyName: 'CloudwatchPermission' + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - 'logs:CreateLogGroup' + - 'logs:CreateLogStream' + - 'logs:PutLogEvents' + Resource: '*' + - PolicyName: 'CognitoPermission' + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - cognito-identity:* + - cognito-idp:* + Resource: '*' + - PolicyName: 'DynamoDBPermission' + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - dynamodb:* + Resource: "*" + - PolicyName: "OtherServicePermission" + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - events:PutEvents + - s3:Get* + - ecr:* + - elasticfilesystem:* + - states:* + - s3:* + - ec2:* + - sqs:* + - "ses:*" + - cognito-identity:* + - cognito-idp:* + Resource: "*" + + #================ APPLICATIONS ============================================= + APIService: + Type: AWS::Serverless::Application + Properties: + Location: api-service/template_api_service.yaml + Parameters: + StagePara: !Ref Stage + ApplicationPara: !Ref Application + ### function arn + FunctionGetFileInfoNLabelArn: !GetAtt AnnotationService.Outputs.FunctionGetFileInfoNLabelArn + FunctionCreateLabelCategoryArn: !GetAtt AnnotationService.Outputs.FunctionCreateLabelCategoryArn + FunctionSaveLabelArn: !GetAtt AnnotationService.Outputs.FunctionSaveLabelArn + FunctionAddClassArn: !GetAtt AnnotationService.Outputs.FunctionAddClassArn + FunctionGetProjectInfoArn: !GetAtt ProjectService.Outputs.FunctionGetProjectInfoArn + FunctionProjectListDataArn: !GetAtt ProjectService.Outputs.FunctionProjectListDataArn + ProjectCloneFunctionArn: !GetAtt ProjectService.Outputs.ProjectCloneFunctionArn + FunctionListProjectArn: !GetAtt ProjectService.Outputs.FunctionListProjectArn + FuncDeleteProject: !GetAtt ProjectService.Outputs.FuncDeleteProject + FunctionUploadCheckArn: !GetAtt ProjectService.Outputs.FunctionUploadCheckArn + FunctionUploadUpdateArn: !GetAtt ProjectService.Outputs.FunctionUploadUpdateArn + FunctionCheckAISegmentationProgressArn: !GetAtt ProjectService.Outputs.FunctionCheckAISegmentationProgressArn + + DatabaseService: + Type: AWS::Serverless::Application + Properties: + Location: db-service/db_template.yaml + Parameters: + StagePara: !Ref Stage + ApplicationPara: !Ref Application + + ProjectService: + Type: AWS::Serverless::Application + Properties: + Location: project-service/template_project_service.yaml + Parameters: + StagePara: !Ref Stage + ApplicationPara: !Ref Application + ###_____ for common layer and role + CommonCodeLayerRef: !Ref CommonCodeLayerRef + LambdaRoleArn: !GetAtt GeneralLambdaExecutionRole.Arn + ###_____ for authorize + CognitoUserPoolRef: !Ref CognitoUserPoolRef + CognitoIdentityPoolIdRef: !Ref CognitoIdentityPoolIdRef + ###_____ for annotation project table + TableAnnoDataOriginalName: !GetAtt DatabaseService.Outputs.TableAnnoDataOriginalName + TableAnnoProjectSumName: !GetAtt DatabaseService.Outputs.TableAnnoProjectSumName + TableAnnoProjectsName: !GetAtt DatabaseService.Outputs.TableAnnoProjectsName + TableCategoryInfoName: !GetAtt DatabaseService.Outputs.TableCategoryInfoName + TableAIDefaultClassInfoName: !GetAtt DatabaseService.Outputs.TableAIDefaultClassInfoName + TableClassInfoName: !GetAtt DatabaseService.Outputs.TableClassInfoName + TableDeletedProjectName: !GetAtt DatabaseService.Outputs.TableDeletedProjectName + ###_____ for daita project table + TableDaitaProjectsName: !Ref TableDaitaProjectsName + TableDaitaDataOriginalName: !Ref TableDaitaDataOriginalName + ###_____ for config table + TableConfigParametersLambdaName: !GetAtt DatabaseService.Outputs.TableConfigParametersLambdaName + ###_____ for s3 bucket + S3AnnoBucketName: !Ref S3AnnoBucketName + S3DaitaBucketName: !Ref S3DaitaBucketName + + AnnotationService: + Type: AWS::Serverless::Application + Properties: + Location: annotation-service/template_annotaion_service.yaml + Parameters: + StagePara: !Ref Stage + ApplicationPara: !Ref Application + ###_____ for common layer and role + CommonCodeLayerRef: !Ref CommonCodeLayerRef + LambdaRoleArn: !GetAtt GeneralLambdaExecutionRole.Arn + ###_____ for authorize + CognitoUserPoolRef: !Ref CognitoUserPoolRef + CognitoIdentityPoolIdRef: !Ref CognitoIdentityPoolIdRef + ###_____ for annotation project table + TableAnnoDataOriginalName: !GetAtt DatabaseService.Outputs.TableAnnoDataOriginalName + TableAnnoProjectSumName: !GetAtt DatabaseService.Outputs.TableAnnoProjectSumName + TableAnnoProjectsName: !GetAtt DatabaseService.Outputs.TableAnnoProjectsName + TableLabelInfoName: !GetAtt DatabaseService.Outputs.TableLabelInfoName + TableCategoryInfoName: !GetAtt DatabaseService.Outputs.TableCategoryInfoName + TableClassInfoName: !GetAtt DatabaseService.Outputs.TableClassInfoName + TableAIDefaultClassInfoName: !GetAtt DatabaseService.Outputs.TableAIDefaultClassInfoName + ###_____ for daita project table + TableDaitaProjectsName: !Ref TableDaitaProjectsName + TableDaitaDataOriginalName: !Ref TableDaitaDataOriginalName + ###_____ for s3 bucket + S3AnnoBucketName: !Ref S3AnnoBucketName + S3DaitaBucketName: !Ref S3DaitaBucketName + + ECSSegmentationServiceApp: + Type: AWS::Serverless::Application + Properties: + Location: ecs-segment-app/template.yaml + Parameters: + StagePara: !Ref Stage + ApplicationPara: !Ref Application + TableAnnoDataOriginalNameStream: !GetAtt DatabaseService.Outputs.StreamTableAnnoDataOrginal + CommonCodeLayerRef: !Ref CommonCodeLayerRef + LambdaRoleArn: !GetAtt GeneralLambdaExecutionRole.Arn + TableAnnoDataOriginalName: !GetAtt DatabaseService.Outputs.TableAnnoDataOriginalName + ### for infra network related VPC + PublicSubnetOne: !Ref PublicSubnetOne + PublicSubnetTwo: !Ref PublicSubnetTwo + ContainerSecurityGroup: !Ref ContainerSecurityGroup + VPC: !Ref VPC + VPCEndpointSQSDnsEntries: !Ref VPCEndpointSQSDnsEntries + ### for infra storage + EFSFileSystemId: !Ref EFSFileSystemId + EFSAccessPoint: !Ref EFSAccessPoint + EFSAccessPointArn: !Ref EFSAccessPointArn + TableAnnoProjectsName: !GetAtt DatabaseService.Outputs.TableAnnoProjectsName + CognitoUserPoolRef: !Ref CognitoUserPoolRef + CognitoIdentityPoolIdRef: !Ref CognitoIdentityPoolIdRef + TableUserName: !Ref TableUserName + SendEmailIdentityIDFunction: !Ref SendEmailIdentityIDFunction + ### for ecs + ImageAISegmentationUrl: !Ref ImageAISegmentationUrl + MaxSizeEc2AutoScallEcs: !Ref MaxSizeEc2AutoScallEcs + +Outputs: + ApiAnnoAppUrl: + Value: !GetAtt APIService.Outputs.AnnoHttpApiURL diff --git a/core_service/api_gateway/de_api_project.py b/core_service/api_gateway/de_api_project.py index cf1fabe..981cd3b 100644 --- a/core_service/api_gateway/de_api_project.py +++ b/core_service/api_gateway/de_api_project.py @@ -1,3 +1,4 @@ +import os import boto3 from .api_gateway_service import ApiGatewayToService @@ -12,7 +13,7 @@ def deploy_api_project(ls_lambda_info, general_info): RESOURCE_SENMAIL = 'send-mail' STAGES = general_info['MODE'] - API_GW_ROLE = 'arn:aws:iam::366577564432:role/role_apigw' + API_GW_ROLE = os.environ.get('ROLEGATEWAY','arn:aws:iam::366577564432:role/role_apigw') gateway = ApiGatewayToService(boto3.client('apigateway')) gateway.create_rest_api(REST_API_NAME) project_id = gateway.add_rest_resource(gateway.root_id, RESOURCE_PROJECT) diff --git a/core_service/aws_lambda/lambda_service.py b/core_service/aws_lambda/lambda_service.py index 5732c67..2cd9bb3 100644 --- a/core_service/aws_lambda/lambda_service.py +++ b/core_service/aws_lambda/lambda_service.py @@ -1,3 +1,4 @@ +import os import boto3 import botocore from aws_lambda.utils.utils import create_zip_object @@ -28,7 +29,7 @@ def deploy_lambda_function(self, function_name, ls_files, env_vari, handler, des # KMSKeyArn='', # use as default of Lambda service key MemorySize=memorysize, Publish=True, - Role='arn:aws:iam::366577564432:role/role_lambda', + Role=os.environ.get('ROLEGATEWAY','arn:aws:iam::366577564432:role/role_lambda'), Runtime='python3.8', Tags={ 'DEPARTMENT': 'Assets', diff --git a/core_service/aws_lambda/project/code/credential_login.py b/core_service/aws_lambda/project/code/credential_login.py index 84ee12e..8c7b779 100644 --- a/core_service/aws_lambda/project/code/credential_login.py +++ b/core_service/aws_lambda/project/code/credential_login.py @@ -23,10 +23,11 @@ cog_provider_client = boto3.client('cognito-idp') cog_identity_client = boto3.client('cognito-identity') # endpoint = 'https://devdaitaloginsocial.auth.us-east-2.amazoncognito.com/oauth2/token' -endpoint = 'https://auth.daita.tech/oauth2/token' -client_id = '4cpbb5etp3q7grnnrhrc7irjoa' +endpoint = OAUTHENPOINT +client_id = CLIENTPOOLID def getRedirectURI(): - return 'https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/login_social' + return ENDPPOINTREDIRCTLOGINSOCIALOAUTH + ############################################################################################################################################################# def getMail(user): response = cog_provider_client.list_users( @@ -86,6 +87,7 @@ def Oauth2(code): headers = {"Content-Type": "application/x-www-form-urlencoded"} data = urlencode(params) result = requests.post(endpoint, data=data, headers=headers) + print(result.text) return result ############################################################################################################ def getDisplayName(username): diff --git a/core_service/aws_lambda/project/code/forgot_password.py b/core_service/aws_lambda/project/code/forgot_password.py index 1cb89ee..149d140 100644 --- a/core_service/aws_lambda/project/code/forgot_password.py +++ b/core_service/aws_lambda/project/code/forgot_password.py @@ -16,21 +16,24 @@ cog_identity_client = boto3.client('cognito-identity') RESPONSE_HEADER = { "Access-Control-Allow-Credentials": "true", - "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", + "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", } + + def getMail(user): response = cog_provider_client.list_users( UserPoolId=USERPOOLID ) # info_user = list(filter(lambda x : x['Username'] == user,response['Users'])) - for _ , data in enumerate(response['Users']): + for _, data in enumerate(response['Users']): if data['Username'] == user: - for info in data['Attributes']: + for info in data['Attributes']: if info['Name'] == 'email': return info['Value'] return None + @error_response def lambda_handler(event, context): @@ -62,21 +65,19 @@ def lambda_handler(event, context): break if not is_email_verified: return generate_response( - message= MessageUserVerifyConfirmCode, + message=MessageUserVerifyConfirmCode, headers=RESPONSE_HEADER ) mail = getMail(username) AddTriggerCustomMail({ - 'region':REGION, - 'user':username, - 'mail':mail, - 'subject':'Your email confirmation code' + 'region': REGION, + 'user': username, + 'mail': mail, + 'subject': 'Your email confirmation code', + 'confirm_code_Table': os.environ['TABLE_CONFIRM_CODE'] }) return generate_response( message=MessageForgotPasswordSuccessfully, headers=RESPONSE_HEADER ) - - - \ No newline at end of file diff --git a/core_service/aws_lambda/project/code/invite_friend.py b/core_service/aws_lambda/project/code/invite_friend.py index 23f0606..3937577 100644 --- a/core_service/aws_lambda/project/code/invite_friend.py +++ b/core_service/aws_lambda/project/code/invite_friend.py @@ -22,7 +22,8 @@ def get_email(user): return None, e info_user = list(filter(lambda x: x["Username"] == user, resp["Users"])) if len(info_user): - email = list(filter(lambda x: x["Name"] == "email", info_user[0]["Attributes"])) + email = list( + filter(lambda x: x["Name"] == "email", info_user[0]["Attributes"])) return email[0]["Value"], None return None, None @@ -36,11 +37,11 @@ def lambda_handler(event, context): return convert_response( {"error": True, "success": False, "message": repr(e), "data": None} ) - print(body) email_name, error = get_email(source_user) if error != None: return convert_response( - {"error": True, "success": False, "message": str(error), "data": None} + {"error": True, "success": False, + "message": str(error), "data": None} ) if email_name == None: return convert_response( diff --git a/core_service/aws_lambda/project/code/login.py b/core_service/aws_lambda/project/code/login.py index 2f64c1d..bc97d8f 100644 --- a/core_service/aws_lambda/project/code/login.py +++ b/core_service/aws_lambda/project/code/login.py @@ -64,13 +64,6 @@ def checkEmailVerified(access_token): if it['Name'] == 'email_verified' and it['Value'] == 'false': return False return True - -#################################################################################### - - -def checkEmailVerified(email): - - return True #################################################################################### diff --git a/core_service/aws_lambda/project/code/login_social.py b/core_service/aws_lambda/project/code/login_social.py index 8a27cae..5dc141b 100644 --- a/core_service/aws_lambda/project/code/login_social.py +++ b/core_service/aws_lambda/project/code/login_social.py @@ -31,7 +31,7 @@ def lambda_handler(event, context): except Exception as e: print(e) if 'error_description' in param: - location = 'https://app.daita.tech/' + location =LOCATION headers = {"Location":location, "Access-Control-Allow-Methods": "GET,HEAD,OPTIONS,POST,PUT"} return { "statusCode": 302, diff --git a/core_service/aws_lambda/project/code/presigned_url_s3.py b/core_service/aws_lambda/project/code/presigned_url_s3.py index bf70e07..dbbf219 100644 --- a/core_service/aws_lambda/project/code/presigned_url_s3.py +++ b/core_service/aws_lambda/project/code/presigned_url_s3.py @@ -1,4 +1,3 @@ -from email.mime import base import os import json import boto3 @@ -6,44 +5,83 @@ from response import * from config import * s3 = boto3.client('s3') -bucket = os.environ['BUCKET_S3'] +bucket = os.environ['BUCKET_NAME'] RESPONSE_HEADER = { "Access-Control-Allow-Creentials": "true", - "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", + "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", } -def generate_presigned_url(object_keyname,expired=3600): + + +def invokeUploadUpdateFunc(info): + lambdaInvokeClient = boto3.client('lambda') + lambdaInvokeReq = lambdaInvokeClient.invoke( + FunctionName='staging-project-upload-update', + Payload=json.dumps({'body': info}), + InvocationType="RequestResponse", + ) + print(lambdaInvokeReq['Payload'].read()) + + +def invokeUploadCheck(info): + lambdaInvokeClient = boto3.client('lambda') + lambdaInvokeReq = lambdaInvokeClient.invoke( + FunctionName='staging-project-upload-check', + Payload=json.dumps({'body': info}), + InvocationType="RequestResponse", + ) + print(lambdaInvokeReq['Payload'].read()) + + +def generate_presigned_url(object_keyname, expired=3600): reponse = s3.generate_presigned_post( - Bucket= bucket, + Bucket=bucket, Key=object_keyname, ExpiresIn=expired ) return reponse - @error_response def lambda_handler(event, context): - + try: body = json.loads(event['body']) filenames = body['filenames'] - identity = body['identity'] - project_id = body['project_id'] + daita_token = body['daita_token'] + ls_object_info = body['ls_object_info'] except Exception as e: print(e) return generate_response( message=MessageAuthenFailed, data={}, headers=RESPONSE_HEADER, - error = True) - folder = os.path.join(identity,project_id) + error=True) + folder = os.path.join(identity, project_id) data = {} for it in filenames: basename = os.path.basename(it) - data[basename] = generate_presigned_url(object_keyname=os.path.join(folder,basename)) - + data[basename] = generate_presigned_url( + object_keyname=os.path.join(folder, basename)) + ls_filename = [] + for objectS3 in ls_object_info: + objectS3['s3_key'] = os.path.join( + bucket, os.path.join(folder, objectS3['filename'])) + ls_filename.append(objectS3['filename']) + invokeUploadUpdateFunc(json.dumps({ + "id_token": id_token, + "project_id": project_id, + "project_name": project_name, + "ls_object_info": ls_object_info + })) + invokeUploadCheck(json.dumps( + { + "id_token": id_token, + "ls_filename": ls_filename, + "project_id": project_id + } + )) return generate_response( - message="Generate presign Url S3 successfully", - data=data, - headers=RESPONSE_HEADER - ) \ No newline at end of file + message="Generate presign Url S3 successfully", + data=data, + headers=RESPONSE_HEADER + ) diff --git a/core_service/aws_lambda/project/code/project_info.py b/core_service/aws_lambda/project/code/project_info.py index 02b7367..fe0cd43 100644 --- a/core_service/aws_lambda/project/code/project_info.py +++ b/core_service/aws_lambda/project/code/project_info.py @@ -115,9 +115,9 @@ def lambda_handler(event, context): ## get task of generation ls_tasks = get_running_task(os.environ['T_TASKS'], db_resource, ls_tasks, identity_id, res_projectid) ls_tasks = get_running_task("down_tasks", db_resource, ls_tasks, identity_id, res_projectid) - ls_tasks = get_running_task("dev-healthcheck-tasks", db_resource, ls_tasks, identity_id, res_projectid, "HEALTHCHECK") - ls_tasks = get_running_task("dev-dataflow-task", db_resource, ls_tasks, identity_id, res_projectid) - ls_tasks = get_running_task("dev-reference-image-tasks", db_resource, ls_tasks, identity_id, res_projectid) + ls_tasks = get_running_task("devdaitabeapp-healthcheck-tasks", db_resource, ls_tasks, identity_id, res_projectid, "HEALTHCHECK") + ls_tasks = get_running_task("devdaitabeapp-dataflow-task", db_resource, ls_tasks, identity_id, res_projectid) + ls_tasks = get_running_task("devdaitabeapp-reference-image-tasks", db_resource, ls_tasks, identity_id, res_projectid) return convert_response({'data': { "identity_id": identity_id, diff --git a/core_service/aws_lambda/project/code/slack_webhook_feedback.py b/core_service/aws_lambda/project/code/slack_webhook_feedback.py index 6a822dd..231336b 100644 --- a/core_service/aws_lambda/project/code/slack_webhook_feedback.py +++ b/core_service/aws_lambda/project/code/slack_webhook_feedback.py @@ -1,4 +1,5 @@ import requests +import re import json import pytz import os @@ -7,19 +8,53 @@ from http import HTTPStatus import os import uuid +import slack_sdk import boto3 import cognitojwt from error import * from response import * from config import * +import tempfile cog_provider_client = boto3.client('cognito-idp') cog_identity_client = boto3.client('cognito-identity') RESPONSE_HEADER = { "Access-Control-Allow-Creentials": "true", - "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", + "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", } ############################################################################################################ +s3 = boto3.client('s3') + + +def validFileImage(filename): + return (os.path.splitext(filename)[1]).lower() in ['.jpeg', '.png', '.jpg'] + + +def split(uri): + if not 's3' in uri[:2]: + temp = uri.split('/') + bucket = temp[0] + filename = '/'.join([temp[i] for i in range(1, len(temp))]) + else: + match = re.match(r's3:\/\/(.+?)\/(.+)', uri) + bucket = match.group(1) + filename = match.group(2) + return bucket, filename + + +def postMessageWithFiles(message, fileList, channel): + SLACK_TOKEN = OAUTH2BOT + client = slack_sdk.WebClient(token=SLACK_TOKEN) + for file in fileList: + upload = client.files_upload(file=file, filename=file) + message = message+"<"+upload['file']['permalink']+"| >" + outP = client.chat_postMessage( + channel=channel, + text=message + ) + print(outP) + + def getDisplayName(username): response = cog_provider_client.admin_get_user( UserPoolId=USERPOOLID, @@ -32,15 +67,16 @@ def getDisplayName(username): name = "" if "name" in user_attributes: name = user_attributes["name"] - elif "given_name" in user_attributes or \ - "family_name" in user_attributes: + elif "given_name" in user_attributes or "family_name" in user_attributes: name = f"{user_attributes.pop('given_name', '')} {user_attributes.pop('family_name', '')}" else: name = user_attributes["email"] return name #################################################################################################### -def claimsToken(jwt_token,field): + + +def claimsToken(jwt_token, field): """ Validate JWT claims & retrieve user identifier """ @@ -55,45 +91,60 @@ def claimsToken(jwt_token,field): return verified_claims.get(field) + class Feedback(object): def __init__(self): - self.db_client = boto3.resource('dynamodb',region_name=REGION) + self.db_client = boto3.resource('dynamodb', region_name=REGION) self.TBL = "feedback" - def CreateItem(self,info): - self.db_client.Table(self.TBL).put_item(Item={ - "ID":info["ID"], - "name":info["name"], - "content":info["content"], - "created_time":info["created_time"], - }) - - def CheckKeyIsExist(self,ID): + + def CreateItem(self, info): + self.db_client.Table(self.TBL).put_item(Item={ + "ID": info["ID"], + "name": info["name"], + "content": info["content"], + "images": info["images"], + "created_time": info["created_time"], + }) + + def CheckKeyIsExist(self, ID): response = self.db_client.Table(self.TBL).get_item(Key={ - "ID":ID + "ID": ID }) if 'Item' in response: return True return False + @error_response def lambda_handler(event, context): headers = event['headers']['Authorization'] authorization_header = headers - token = authorization_header.replace('Bearer ','') + token = authorization_header.replace('Bearer ', '') feedbackDB = Feedback() if not len(authorization_header): raise Exception(MessageMissingAuthorizationHeader) - + try: body = json.loads(event['body']) text = body['text'] + images = body['images'] except Exception as e: print(e) raise Exception(MessageUnmarshalInputJson) + if len(text) > 750: + raise Exception(MessageErrorFeedbackLimitword) + if not isinstance(images, list): + raise Exception(MessageErrorFeedbackInvalidType) + + if len(images) > 3: + raise Exception(MessageErrorFeedbackLimitImages) + for it in images: + if not validFileImage(it): + raise Exception(MessageErrorInvalidExtension) try: - username = claimsToken(token,'username') + username = claimsToken(token, 'username') except Exception as e: raise e info = {} @@ -104,32 +155,34 @@ def lambda_handler(event, context): datetimeString = datetimeUTC.strftime('%Y:%m:%d %H:%M:%S %Z %z') if not feedbackDB.CheckKeyIsExist(key): info = { - "ID":key, - "name":username, - "content":text, + "ID": key, + "name": username, + "content": text, + "images": images, "created_time": datetimeString } feedbackDB.CreateItem(info) break - message = "Username: {}\n Time: {}\n Content: {}".format(getDisplayName(info['name']),info['created_time'],info['content']) - payload ={"channel":CHANNELWEBHOOK, - "username":getDisplayName(username), - "text":message, - "icon_emoji":":ghost:"} - - req = requests.post( - WEBHOOK,json=payload - ) - - - if req.status_code != 200: - return generate_response( - message=MessageSendFeedbackFailed, - data={}, - headers=RESPONSE_HEADER - ) + message = "Username: {}\n Time: {}\n Content: {}".format( + getDisplayName(info['name']), info['created_time'], info['content']) + fileList = [] + dir = tempfile.TemporaryDirectory(dir='/tmp') + dirname = dir.name + try: + for it in images: + bucket, filename = split(it) + resultS3 = s3.get_object(Bucket=bucket, Key=filename) + tmpfile = os.path.join(dirname, os.path.basename(filename)) + fileList.append(tmpfile) + with open(tmpfile, 'wb') as file: + file.write(resultS3['Body'].read()) + except Exception as e: + print(e) + raise Exception(e) + postMessageWithFiles(message, fileList, CHANNELWEBHOOK) + dir.cleanup() return generate_response( - message=MessageSendFeedbackSuccessfully, - data={}, - headers=RESPONSE_HEADER - ) \ No newline at end of file + message=MessageSendFeedbackSuccessfully, + data={}, + headers=RESPONSE_HEADER + ) diff --git a/core_service/aws_lambda/project/common/config.py b/core_service/aws_lambda/project/common/config.py index 42ee085..ff23b48 100644 --- a/core_service/aws_lambda/project/common/config.py +++ b/core_service/aws_lambda/project/common/config.py @@ -1,7 +1,20 @@ -CLIENTPOOLID = "4cpbb5etp3q7grnnrhrc7irjoa" -USERPOOLID = "us-east-2_ZbwpnYN4g" +import os +config_env = { + 'CLIENTPOOLID': {'dev': '7v8h65t0d3elscfqll090acf9h', 'staging': '4cpbb5etp3q7grnnrhrc7irjoa'}, + 'USERPOOLID': {'dev': 'us-east-2_6Sc8AZij7', 'staging': 'us-east-2_ZbwpnYN4g'}, + 'IDENTITYPOOLID': {'dev': 'us-east-2:639788f0-a9b0-460d-9f50-23bbe5bc7140', 'staging': 'us-east-2:fa0b76bc-01fa-4bb8-b7cf-a5000954aafb'}, + 'LOCATION': {'dev': "https://dev.daita.tech/", 'staging': 'https://app.daita.tech/'}, + 'ENDPPOINTREDIRCTLOGINSOCIALOAUTH': {'dev': 'https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/auth/login_social', 'staging': 'https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/login_social'}, + 'OAUTHENPOINT': {'dev': 'https://authdev.daita.tech/oauth2/token', 'staging': 'https://auth.daita.tech/oauth2/token'}, + 'WEBHOOK': {'dev': 'https://hooks.slack.com/services/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', 'staging': 'https://hooks.slack.com/services/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'}, + 'CHANNELWEBHOOK': {'dev': '#feedback-daita', 'staging': '#user-feedback'}, + 'OAUTH2OFBOT': {'dev': 'xoxb-2966360820768-3760970933602-MoApe9duMpoO5KAa6HaCUzzY'} +} +OAUTH2BOT = config_env['OAUTH2OFBOT'][os.environ['MODE']] +CLIENTPOOLID = config_env['CLIENTPOOLID'][os.environ['MODE']] +USERPOOLID = config_env['USERPOOLID'][os.environ['MODE']] REGION = "us-east-2" -IDENTITYPOOLID = "us-east-2:fa0b76bc-01fa-4bb8-b7cf-a5000954aafb" +IDENTITYPOOLID = config_env['IDENTITYPOOLID'][os.environ['MODE']] SITEKEYGOOGLE = "6LcqEGMeAAAAAAEDnBue7fwR4pmvNO7JKWkHtAjl" SECRETKEYGOOGLE = "6LcqEGMeAAAAAOiJAMcg1NNfj6eA62gQPLJAtQMt" ENDPOINTCAPTCHAVERIFY = "https://www.google.com/recaptcha/api/siteverify" @@ -9,7 +22,10 @@ # Change these if used with GitHub Enterprise (see below) GITHUB_API_URL = "https://api.github.com" GITHUB_LOGIN_URL = "https://github.com" -WEBHOOK="https://hooks.slack.com/services/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" -CHANNELWEBHOOK="#user-feedback" -AWS_ACC_ID ='366577564432' -STS_ARN = 'arn:aws:iam::366577564432:role/stscognito' \ No newline at end of file +WEBHOOK = config_env['WEBHOOK'][os.environ['MODE']] +CHANNELWEBHOOK = config_env['CHANNELWEBHOOK'][os.environ['MODE']] +AWS_ACC_ID = '366577564432' +STS_ARN = 'arn:aws:iam::366577564432:role/stscognito' +LOCATION = config_env['LOCATION'][os.environ['MODE']] +ENDPPOINTREDIRCTLOGINSOCIALOAUTH = config_env['ENDPPOINTREDIRCTLOGINSOCIALOAUTH'][os.environ['MODE']] +OAUTHENPOINT = config_env['OAUTHENPOINT'][os.environ['MODE']] diff --git a/core_service/aws_lambda/project/common/error.py b/core_service/aws_lambda/project/common/error.py index 94ce7a5..13c497e 100644 --- a/core_service/aws_lambda/project/common/error.py +++ b/core_service/aws_lambda/project/common/error.py @@ -31,6 +31,10 @@ MessageLoginMailNotExist = "Email does not exist." MessageAnotherUserIsLoginBefore = "You are already logged in on another device." MessageErrorUserdoesnotlogin = "Current user did not login to the application!" -MessageLogoutSuccessfully = "User log out successful." +MessageLogoutSuccessfully = "User log out successful." MessageErrorCredential = "Failed to retrieve credentials." MessageSuccessfullyCredential = "Successfully retrieved credential." +MessageErrorFeedbackInvalidType = "Please check format json!" +MessageErrorFeedbackLimitword = "Word limit for feedback!" +MessageErrorFeedbackLimitImages = "Attachment file count limit!" +MessageErrorInvalidExtension = "Invalid extension of file attachment!" \ No newline at end of file diff --git a/core_service/aws_lambda/project/de_lambda.py b/core_service/aws_lambda/project/de_lambda.py index 12a5638..6d6dc7b 100644 --- a/core_service/aws_lambda/project/de_lambda.py +++ b/core_service/aws_lambda/project/de_lambda.py @@ -14,10 +14,5 @@ def deploy_lambda(general_info): ls_lambda_val = [] ls_lambda_val += deploy_lambda_project(general_info, lambda_service) - # ls_lambda_val += deploy_lambda_generate(general_info, lambda_service) - # ls_lambda_val += deploy_lambda_balancer(general_info, lambda_service) - ls_lambda_val += deploy_lambda_auth(general_info, lambda_service) ls_lambda_val += deploy_lambda_webhook(general_info, lambda_service) - ls_lambda_val += deploy_lambda_send_mail(general_info, lambda_service) - ls_lambda_val += deploy_lambda_s3(general_info, lambda_service) return ls_lambda_val diff --git a/core_service/aws_lambda/project/de_lambda_auth_funcs.py b/core_service/aws_lambda/project/de_lambda_auth_funcs.py index 743225f..bc62d50 100644 --- a/core_service/aws_lambda/project/de_lambda_auth_funcs.py +++ b/core_service/aws_lambda/project/de_lambda_auth_funcs.py @@ -19,7 +19,8 @@ def deploy_lambda_auth(general_info, lambda_service): ], { 'USER_POOL_ID': general_info['USER_POOL_ID'], - 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'] + 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], + 'MODE': general_info['MODE'] }, 'logout.lambda_handler', 'staging: logout') @@ -38,7 +39,8 @@ def deploy_lambda_auth(general_info, lambda_service): { 'USER_POOL_ID': general_info['USER_POOL_ID'], 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], - 'IS_ENABLE_KMS': general_info['IS_ENABLE_KMS'] + 'IS_ENABLE_KMS': general_info['IS_ENABLE_KMS'], + 'MODE': general_info['MODE'] }, 'login.lambda_handler', @@ -56,7 +58,8 @@ def deploy_lambda_auth(general_info, lambda_service): ], { 'USER_POOL_ID': general_info['USER_POOL_ID'], - 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'] + 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], + 'MODE': general_info['MODE'] }, 'register.lambda_handler', 'staging: register') @@ -76,7 +79,8 @@ def deploy_lambda_auth(general_info, lambda_service): { 'USER_POOL_ID': general_info['USER_POOL_ID'], 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], - 'IS_ENABLE_KMS': general_info['IS_ENABLE_KMS'] + 'IS_ENABLE_KMS': general_info['IS_ENABLE_KMS'], + 'MODE': general_info['MODE'] }, 'login_social.lambda_handler', 'staging: login_social') @@ -95,7 +99,8 @@ def deploy_lambda_auth(general_info, lambda_service): { 'USER_POOL_ID': general_info['USER_POOL_ID'], 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], - 'IS_ENABLE_KMS': general_info['IS_ENABLE_KMS'] + 'IS_ENABLE_KMS': general_info['IS_ENABLE_KMS'], + 'MODE': general_info['MODE'] }, 'credential_login.lambda_handler', 'staging: credential_login') @@ -111,7 +116,8 @@ def deploy_lambda_auth(general_info, lambda_service): ], { 'USER_POOL_ID': general_info['USER_POOL_ID'], - 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'] + 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], + 'MODE': general_info['MODE'] }, 'template_mail.lambda_handler', 'staging: template_mail') @@ -126,7 +132,8 @@ def deploy_lambda_auth(general_info, lambda_service): ], { 'USER_POOL_ID': general_info['USER_POOL_ID'], - 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'] + 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], + 'MODE': general_info['MODE'] }, 'resend_confirmcode.lambda_handler', 'staging: template_mail') @@ -141,7 +148,8 @@ def deploy_lambda_auth(general_info, lambda_service): ], { 'USER_POOL_ID': general_info['USER_POOL_ID'], - 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'] + 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], + 'MODE': general_info['MODE'] }, 'auth_confirm.lambda_handler', 'staging: auth_confirm') @@ -160,7 +168,8 @@ def deploy_lambda_auth(general_info, lambda_service): ], { 'USER_POOL_ID': general_info['USER_POOL_ID'], - 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'] + 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], + 'MODE': general_info['MODE'] }, 'forgot_password.lambda_handler', 'staging: forgot password') @@ -179,7 +188,8 @@ def deploy_lambda_auth(general_info, lambda_service): ], { 'USER_POOL_ID': general_info['USER_POOL_ID'], - 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'] + 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], + 'MODE': general_info['MODE'] }, 'confirm_code_forgot_password.lambda_handler', 'staging: confirm code after request forgot password') @@ -198,7 +208,8 @@ def deploy_lambda_auth(general_info, lambda_service): ], { 'USER_POOL_ID': general_info['USER_POOL_ID'], - 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'] + 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], + 'MODE': general_info['MODE'] }, 'login_refresh_token.lambda_handler', 'staging: Get refresh Token') @@ -215,7 +226,9 @@ def deploy_lambda_auth(general_info, lambda_service): PROJECT_DIR.joinpath( "packages"), ], - env_vari={}, + env_vari={ + 'MODE': general_info['MODE'] + }, handler='github_openid_token_wrapper.lambda_handler', description='staging: Wrapper for Github token api to comply with Cognito OpenID') add_lambda_info_to_list(ls_lambda_val, lambda_uri, @@ -231,7 +244,9 @@ def deploy_lambda_auth(general_info, lambda_service): PROJECT_DIR.joinpath( "packages"), ], - env_vari={}, + env_vari={ + 'MODE': general_info['MODE'] + }, handler='github_openid_userinfo_wrapper.lambda_handler', description='staging: Wrapper for Github userinfo api to comply with Cognito OpenID') add_lambda_info_to_list(ls_lambda_val, lambda_uri, diff --git a/core_service/aws_lambda/project/de_lambda_mail_funcs.py b/core_service/aws_lambda/project/de_lambda_mail_funcs.py index 33d020c..da4e355 100644 --- a/core_service/aws_lambda/project/de_lambda_mail_funcs.py +++ b/core_service/aws_lambda/project/de_lambda_mail_funcs.py @@ -15,7 +15,8 @@ def deploy_lambda_send_mail(general_info, lambda_service): ], { 'USER_POOL_ID' : general_info['USER_POOL_ID'], - 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'] + 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], + 'MODE': general_info['MODE'] }, 'invite_friend.lambda_handler', 'staging: reference-email') @@ -26,7 +27,8 @@ def deploy_lambda_send_mail(general_info, lambda_service): ], { 'USER_POOL_ID' : general_info['USER_POOL_ID'], - 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'] + 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], + 'MODE': general_info['MODE'] }, 'cognito_send_mail.lambda_handler', 'staging: cognito_send_mail') diff --git a/core_service/aws_lambda/project/de_lambda_s3_funcs.py b/core_service/aws_lambda/project/de_lambda_s3_funcs.py index 4ddc8f1..12dbf3f 100644 --- a/core_service/aws_lambda/project/de_lambda_s3_funcs.py +++ b/core_service/aws_lambda/project/de_lambda_s3_funcs.py @@ -13,7 +13,8 @@ def deploy_lambda_s3(general_info, lambda_service): ], { 'USER_POOL_ID' : general_info['USER_POOL_ID'], - 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'] + 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], + 'MODE': general_info['MODE'] }, 'presigned_url_s3.py.lambda_handler', 'staging: presigned_url_s3.py') diff --git a/core_service/aws_lambda/project/de_lambda_webhook.py b/core_service/aws_lambda/project/de_lambda_webhook.py index 6f091a5..add1440 100644 --- a/core_service/aws_lambda/project/de_lambda_webhook.py +++ b/core_service/aws_lambda/project/de_lambda_webhook.py @@ -4,20 +4,25 @@ PROJECT_DIR = Path(__file__).parent CODE_DIR = PROJECT_DIR.joinpath("code") + def deploy_lambda_webhook(general_info, lambda_service): - ls_lambda_val = [] - + ls_lambda_val = [] + lambda_uri, lambda_version = lambda_service.deploy_lambda_function(f'slack_webhook_feedback', - [ CODE_DIR.joinpath("slack_webhook_feedback.py"), - PROJECT_DIR.joinpath("common"), - PROJECT_DIR.joinpath("packages") - ], - { - 'USER_POOL_ID' : general_info['USER_POOL_ID'], - 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'] - }, - 'slack_webhook_feedback.lambda_handler', - 'staging: slack_webhook_feedback') - add_lambda_info_to_list(ls_lambda_val, lambda_uri, lambda_version, 'webhook', 'client-feedback') + [CODE_DIR.joinpath("slack_webhook_feedback.py"), + PROJECT_DIR.joinpath( + "common"), + PROJECT_DIR.joinpath( + "packages") + ], + { + 'USER_POOL_ID': general_info['USER_POOL_ID'], + 'IDENTITY_POOL_ID': general_info['IDENTITY_POOL_ID'], + 'MODE': general_info['MODE'] + }, + 'slack_webhook_feedback.lambda_handler', + 'staging: slack_webhook_feedback') + add_lambda_info_to_list(ls_lambda_val, lambda_uri, + lambda_version, 'webhook', 'client-feedback') - return ls_lambda_val \ No newline at end of file + return ls_lambda_val diff --git a/core_service/aws_lambda/project/packages/slack/__init__.py b/core_service/aws_lambda/project/packages/slack/__init__.py new file mode 100644 index 0000000..bad3414 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/__init__.py @@ -0,0 +1,14 @@ +import logging +from logging import NullHandler +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.web/webhook/rtm") + +from slack_sdk.rtm import RTMClient # noqa +from slack_sdk.web.async_client import AsyncWebClient # noqa +from slack_sdk.web.legacy_client import LegacyWebClient as WebClient # noqa +from slack_sdk.webhook.async_client import AsyncWebhookClient # noqa +from slack_sdk.webhook.client import WebhookClient # noqa + +# Set default logging handler to avoid "No handler found" warnings. +logging.getLogger(__name__).addHandler(NullHandler()) diff --git a/core_service/aws_lambda/project/packages/slack/deprecation.py b/core_service/aws_lambda/project/packages/slack/deprecation.py new file mode 100644 index 0000000..6352e66 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/deprecation.py @@ -0,0 +1,14 @@ +import os +import warnings + + +def show_message(old: str, new: str) -> None: + skip_deprecation = os.environ.get("SLACKCLIENT_SKIP_DEPRECATION") # for unit tests etc. + if skip_deprecation: + return + + message = ( + f"{old} package is deprecated. Please use {new} package instead. " + "For more info, go to https://slack.dev/python-slack-sdk/v3-migration/" + ) + warnings.warn(message) diff --git a/core_service/aws_lambda/project/packages/slack/errors.py b/core_service/aws_lambda/project/packages/slack/errors.py new file mode 100644 index 0000000..7d5f130 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/errors.py @@ -0,0 +1,10 @@ +from slack_sdk.errors import BotUserAccessError # noqa +from slack_sdk.errors import SlackApiError # noqa +from slack_sdk.errors import SlackClientError # noqa +from slack_sdk.errors import SlackClientNotConnectedError # noqa +from slack_sdk.errors import SlackObjectFormationError # noqa +from slack_sdk.errors import SlackRequestError # noqa + +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.errors") diff --git a/ai_caller_service_v1/AI_service/app/migrations/__init__.py b/core_service/aws_lambda/project/packages/slack/py.typed similarity index 100% rename from ai_caller_service_v1/AI_service/app/migrations/__init__.py rename to core_service/aws_lambda/project/packages/slack/py.typed diff --git a/core_service/aws_lambda/project/packages/slack/rtm/__init__.py b/core_service/aws_lambda/project/packages/slack/rtm/__init__.py new file mode 100644 index 0000000..1d3d1a0 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/rtm/__init__.py @@ -0,0 +1,6 @@ +from slack_sdk.rtm import RTMClient # noqa +from slack_sdk.web.legacy_client import LegacyWebClient as WebClient # noqa + +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.web/rtm") diff --git a/core_service/aws_lambda/project/packages/slack/rtm/client.py b/core_service/aws_lambda/project/packages/slack/rtm/client.py new file mode 100644 index 0000000..cff4d0b --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/rtm/client.py @@ -0,0 +1,6 @@ +from slack_sdk.rtm import RTMClient # noqa +from slack_sdk.web.legacy_client import LegacyWebClient as WebClient # noqa + +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.rtm.client") diff --git a/core_service/aws_lambda/project/packages/slack/signature/__init__.py b/core_service/aws_lambda/project/packages/slack/signature/__init__.py new file mode 100644 index 0000000..cf3fefc --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/signature/__init__.py @@ -0,0 +1,5 @@ +from slack_sdk.signature import SignatureVerifier # noqa + +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.signature") diff --git a/core_service/aws_lambda/project/packages/slack/signature/verifier.py b/core_service/aws_lambda/project/packages/slack/signature/verifier.py new file mode 100644 index 0000000..f5923ce --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/signature/verifier.py @@ -0,0 +1,71 @@ +import hashlib +import hmac +from time import time +from typing import Dict, Optional, Union + + +class Clock: + @staticmethod + def now() -> float: + return time() + + +class SignatureVerifier: + def __init__(self, signing_secret: str, clock: Clock = Clock()): + """Slack request signature verifier + + Slack signs its requests using a secret that's unique to your app. + With the help of signing secrets, your app can more confidently verify + whether requests from us are authentic. + https://api.slack.com/authentication/verifying-requests-from-slack + """ + self.signing_secret = signing_secret + self.clock = clock + + def is_valid_request( + self, + body: Union[str, bytes], + headers: Dict[str, str], + ) -> bool: + """Verifies if the given signature is valid""" + if headers is None: + return False + normalized_headers = {k.lower(): v for k, v in headers.items()} + return self.is_valid( + body=body, + timestamp=normalized_headers.get("x-slack-request-timestamp", None), + signature=normalized_headers.get("x-slack-signature", None), + ) + + def is_valid( + self, + body: Union[str, bytes], + timestamp: str, + signature: str, + ) -> bool: + """Verifies if the given signature is valid""" + if timestamp is None or signature is None: + return False + + if abs(self.clock.now() - int(timestamp)) > 60 * 5: + return False + + calculated_signature = self.generate_signature(timestamp=timestamp, body=body) + if calculated_signature is None: + return False + return hmac.compare_digest(calculated_signature, signature) + + def generate_signature(self, *, timestamp: str, body: Union[str, bytes]) -> Optional[str]: + """Generates a signature""" + if timestamp is None: + return None + if body is None: + body = "" + if isinstance(body, bytes): + body = body.decode("utf-8") + + format_req = str.encode(f"v0:{timestamp}:{body}") + encoded_secret = str.encode(self.signing_secret) + request_hash = hmac.new(encoded_secret, format_req, hashlib.sha256).hexdigest() + calculated_signature = f"v0={request_hash}" + return calculated_signature diff --git a/core_service/aws_lambda/project/packages/slack/version.py b/core_service/aws_lambda/project/packages/slack/version.py new file mode 100644 index 0000000..a51188c --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/version.py @@ -0,0 +1 @@ +from slack_sdk.version import __version__ # noqa diff --git a/core_service/aws_lambda/project/packages/slack/web/__init__.py b/core_service/aws_lambda/project/packages/slack/web/__init__.py new file mode 100644 index 0000000..f61f4b6 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/__init__.py @@ -0,0 +1,11 @@ +import slack_sdk.version as slack_version # noqa +from slack import deprecation +from slack_sdk.web.async_client import AsyncSlackResponse # noqa +from slack_sdk.web.async_client import AsyncWebClient # noqa +from slack_sdk.web.internal_utils import _to_0_or_1_if_bool # noqa +from slack_sdk.web.internal_utils import convert_bool_to_0_or_1 # noqa +from slack_sdk.web.internal_utils import get_user_agent # noqa +from slack_sdk.web.legacy_client import LegacyWebClient as WebClient # noqa +from slack_sdk.web.slack_response import SlackResponse # noqa + +deprecation.show_message(__name__, "slack_sdk.web") diff --git a/core_service/aws_lambda/project/packages/slack/web/async_base_client.py b/core_service/aws_lambda/project/packages/slack/web/async_base_client.py new file mode 100644 index 0000000..9347bac --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/async_base_client.py @@ -0,0 +1,165 @@ +import logging +from ssl import SSLContext +from typing import Optional, Union, Dict + +import aiohttp +from aiohttp import FormData + +from slack.web import convert_bool_to_0_or_1, get_user_agent +from slack.web.async_internal_utils import ( + _build_req_args, + _get_url, + _files_to_data, + _request_with_session, +) +from slack.web.async_slack_response import AsyncSlackResponse +from slack.web.deprecation import show_2020_01_deprecation + + +class AsyncBaseClient: + BASE_URL = "https://www.slack.com/api/" + + def __init__( + self, + token: Optional[str] = None, + base_url: str = BASE_URL, + timeout: int = 30, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + session: Optional[aiohttp.ClientSession] = None, + trust_env_in_session: bool = False, + headers: Optional[dict] = None, + user_agent_prefix: Optional[str] = None, + user_agent_suffix: Optional[str] = None, + ): + self.token = None if token is None else token.strip() + self.base_url = base_url + self.timeout = timeout + self.ssl = ssl + self.proxy = proxy + self.session = session + # https://github.com/slackapi/python-slack-sdk/issues/738 + self.trust_env_in_session = trust_env_in_session + self.headers = headers or {} + self.headers["User-Agent"] = get_user_agent(user_agent_prefix, user_agent_suffix) + self._logger = logging.getLogger(__name__) + + async def api_call( # skipcq: PYL-R1710 + self, + api_method: str, + *, + http_verb: str = "POST", + files: dict = None, + data: Union[dict, FormData] = None, + params: dict = None, + json: dict = None, # skipcq: PYL-W0621 + headers: dict = None, + auth: dict = None, + ) -> AsyncSlackResponse: + """Create a request and execute the API call to Slack. + + Args: + api_method (str): The target Slack API method. + e.g. 'chat.postMessage' + http_verb (str): HTTP Verb. e.g. 'POST' + files (dict): Files to multipart upload. + e.g. {image OR file: file_object OR file_path} + data: The body to attach to the request. If a dictionary is + provided, form-encoding will take place. + e.g. {'key1': 'value1', 'key2': 'value2'} + params (dict): The URL parameters to append to the URL. + e.g. {'key1': 'value1', 'key2': 'value2'} + json (dict): JSON for the body to attach to the request + (if files or data is not specified). + e.g. {'key1': 'value1', 'key2': 'value2'} + headers (dict): Additional request headers + auth (dict): A dictionary that consists of client_id and client_secret + + Returns: + (AsyncSlackResponse) + The server's response to an HTTP request. Data + from the response can be accessed like a dict. + If the response included 'next_cursor' it can + be iterated on to execute subsequent requests. + + Raises: + SlackApiError: The following Slack API call failed: + 'chat.postMessage'. + SlackRequestError: Json data can only be submitted as + POST requests. + """ + + api_url = _get_url(self.base_url, api_method) + headers = headers or {} + headers.update(self.headers) + + req_args = _build_req_args( + token=self.token, + http_verb=http_verb, + files=files, + data=data, + params=params, + json=json, # skipcq: PYL-W0621 + headers=headers, + auth=auth, + ssl=self.ssl, + proxy=self.proxy, + ) + + show_2020_01_deprecation(api_method) + + return await self._send( + http_verb=http_verb, + api_url=api_url, + req_args=req_args, + ) + + async def _send(self, http_verb: str, api_url: str, req_args: dict) -> AsyncSlackResponse: + """Sends the request out for transmission. + + Args: + http_verb (str): The HTTP verb. e.g. 'GET' or 'POST'. + api_url (str): The Slack API url. e.g. 'https://slack.com/api/chat.postMessage' + req_args (dict): The request arguments to be attached to the request. + e.g. + { + json: { + 'attachments': [{"pretext": "pre-hello", "text": "text-world"}], + 'channel': '#random' + } + } + Returns: + The response parsed into a AsyncSlackResponse object. + """ + open_files = _files_to_data(req_args) + try: + if "params" in req_args: + # True/False -> "1"/"0" + req_args["params"] = convert_bool_to_0_or_1(req_args["params"]) + + res = await self._request(http_verb=http_verb, api_url=api_url, req_args=req_args) + finally: + for f in open_files: + f.close() + + data = { + "client": self, + "http_verb": http_verb, + "api_url": api_url, + "req_args": req_args, + } + return AsyncSlackResponse(**{**data, **res}).validate() + + async def _request(self, *, http_verb, api_url, req_args) -> Dict[str, any]: + """Submit the HTTP request with the running session or a new session. + Returns: + A dictionary of the response data. + """ + return await _request_with_session( + current_session=self.session, + timeout=self.timeout, + logger=self._logger, + http_verb=http_verb, + api_url=api_url, + req_args=req_args, + ) diff --git a/core_service/aws_lambda/project/packages/slack/web/async_client.py b/core_service/aws_lambda/project/packages/slack/web/async_client.py new file mode 100644 index 0000000..06aeae2 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/async_client.py @@ -0,0 +1,15 @@ +# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +# +# *** DO NOT EDIT THIS FILE *** +# +# 1) Modify slack/web/client.py +# 2) Run `python setup.py validate` +# +# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +from slack import deprecation +from slack_sdk.web.legacy_client import LegacyWebClient as WebClient # noqa +from slack_sdk.web.async_client import AsyncWebClient # noqa +from slack_sdk.web.async_client import AsyncSlackResponse # noqa + +deprecation.show_message(__name__, "slack_sdk.web.client") diff --git a/core_service/aws_lambda/project/packages/slack/web/async_internal_utils.py b/core_service/aws_lambda/project/packages/slack/web/async_internal_utils.py new file mode 100644 index 0000000..7929157 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/async_internal_utils.py @@ -0,0 +1,199 @@ +import asyncio +import json +from asyncio import AbstractEventLoop +from logging import Logger +from ssl import SSLContext +from typing import Union, Optional, BinaryIO, List, Dict +from urllib.parse import urljoin + +import aiohttp +from aiohttp import FormData, BasicAuth, ClientSession + +from slack.errors import SlackRequestError, SlackApiError +from slack.web import get_user_agent + + +def _get_event_loop() -> AbstractEventLoop: + """Retrieves the event loop or creates a new one.""" + try: + return asyncio.get_event_loop() + except RuntimeError: + loop = asyncio.new_event_loop() + asyncio.set_event_loop(loop) + return loop + + +def _get_url(base_url: str, api_method: str) -> str: + """Joins the base Slack URL and an API method to form an absolute URL. + + Args: + base_url (str): The base URL + api_method (str): The Slack Web API method. e.g. 'chat.postMessage' + + Returns: + The absolute API URL. + e.g. 'https://www.slack.com/api/chat.postMessage' + """ + return urljoin(base_url, api_method) + + +def _get_headers( + *, + headers: dict, + token: Optional[str], + has_json: bool, + has_files: bool, + request_specific_headers: Optional[dict], +) -> Dict[str, str]: + """Constructs the headers need for a request. + Args: + has_json (bool): Whether or not the request has json. + has_files (bool): Whether or not the request has files. + request_specific_headers (dict): Additional headers specified by the user for a specific request. + + Returns: + The headers dictionary. + e.g. { + 'Content-Type': 'application/json;charset=utf-8', + 'Authorization': 'Bearer xoxb-1234-1243', + 'User-Agent': 'Python/3.6.8 slack/2.1.0 Darwin/17.7.0' + } + """ + final_headers = { + "User-Agent": get_user_agent(), + "Content-Type": "application/x-www-form-urlencoded", + } + + if token: + final_headers.update({"Authorization": "Bearer {}".format(token)}) + if headers is None: + headers = {} + + # Merge headers specified at client initialization. + final_headers.update(headers) + + # Merge headers specified for a specific request. e.g. oauth.access + if request_specific_headers: + final_headers.update(request_specific_headers) + + if has_json: + final_headers.update({"Content-Type": "application/json;charset=utf-8"}) + + if has_files: + # These are set automatically by the aiohttp library. + final_headers.pop("Content-Type", None) + + return final_headers + + +def _build_req_args( + *, + token: Optional[str], + http_verb: str, + files: dict, + data: Union[dict, FormData], + params: dict, + json: dict, # skipcq: PYL-W0621 + headers: dict, + auth: dict, + ssl: Optional[SSLContext], + proxy: Optional[str], +) -> dict: + has_json = json is not None + has_files = files is not None + if has_json and http_verb != "POST": + msg = "Json data can only be submitted as POST requests. GET requests should use the 'params' argument." + raise SlackRequestError(msg) + + if auth: + auth = BasicAuth(auth["client_id"], auth["client_secret"]) + + if data is not None and isinstance(data, dict): + data = {k: v for k, v in data.items() if v is not None} + if files is not None and isinstance(files, dict): + files = {k: v for k, v in files.items() if v is not None} + if params is not None and isinstance(params, dict): + params = {k: v for k, v in params.items() if v is not None} + + token: Optional[str] = token + if params is not None and "token" in params: + token = params.pop("token") + if json is not None and "token" in json: + token = json.pop("token") + req_args = { + "headers": _get_headers( + headers=headers, + token=token, + has_json=has_json, + has_files=has_files, + request_specific_headers=headers, + ), + "data": data, + "files": files, + "params": params, + "json": json, + "ssl": ssl, + "proxy": proxy, + "auth": auth, + } + return req_args + + +def _files_to_data(req_args: dict) -> List[BinaryIO]: + open_files = [] + files = req_args.pop("files", None) + if files is not None: + for k, v in files.items(): + if isinstance(v, str): + f = open(v.encode("utf-8", "ignore"), "rb") + open_files.append(f) + req_args["data"].update({k: f}) + else: + req_args["data"].update({k: v}) + return open_files + + +async def _request_with_session( + *, + current_session: Optional[ClientSession], + timeout: int, + logger: Logger, + http_verb: str, + api_url: str, + req_args: dict, +) -> Dict[str, any]: + """Submit the HTTP request with the running session or a new session. + Returns: + A dictionary of the response data. + """ + session = None + use_running_session = current_session and not current_session.closed + if use_running_session: + session = current_session + else: + session = aiohttp.ClientSession( + timeout=aiohttp.ClientTimeout(total=timeout), + auth=req_args.pop("auth", None), + ) + + response = None + try: + async with session.request(http_verb, api_url, **req_args) as res: + data = {} + try: + data = await res.json() + except aiohttp.ContentTypeError: + logger.debug(f"No response data returned from the following API call: {api_url}.") + except json.decoder.JSONDecodeError as e: + message = f"Failed to parse the response body: {str(e)}" + raise SlackApiError(message, res) + + response = { + "data": data, + "headers": res.headers, + "status_code": res.status, + } + finally: + if not use_running_session: + await session.close() + return response diff --git a/core_service/aws_lambda/project/packages/slack/web/async_slack_response.py b/core_service/aws_lambda/project/packages/slack/web/async_slack_response.py new file mode 100644 index 0000000..150bc51 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/async_slack_response.py @@ -0,0 +1,187 @@ +"""A Python module for interacting and consuming responses from Slack.""" + +import logging + +import slack.errors as e +from slack.web.internal_utils import _next_cursor_is_present + + +class AsyncSlackResponse: + """An iterable container of response data. + + Attributes: + data (dict): The json-encoded content of the response. Along + with the headers and status code information. + + Methods: + validate: Check if the response from Slack was successful. + get: Retrieves any key from the response data. + next: Retrieves the next portion of results, + if 'next_cursor' is present. + + Example: + ```python + import os + import slack + + client = slack.AsyncWebClient(token=os.environ['SLACK_API_TOKEN']) + + response1 = await client.auth_revoke(test='true') + assert not response1['revoked'] + + response2 = await client.auth_test() + assert response2.get('ok', False) + + users = [] + async for page in await client.users_list(limit=2): + users = users + page['members'] + ``` + + Note: + Some responses return collections of information + like channel and user lists. If they do it's likely + that you'll only receive a portion of results. This + object allows you to iterate over the response which + makes subsequent API requests until your code hits + 'break' or there are no more results to be found. + + Any attributes or methods prefixed with _underscores are + intended to be "private" internal use only. They may be changed or + removed at anytime. + """ + + def __init__( + self, + *, + client, # AsyncWebClient + http_verb: str, + api_url: str, + req_args: dict, + data: dict, + headers: dict, + status_code: int, + ): + self.http_verb = http_verb + self.api_url = api_url + self.req_args = req_args + self.data = data + self.headers = headers + self.status_code = status_code + self._initial_data = data + self._iteration = None # for __iter__ & __next__ + self._client = client + self._logger = logging.getLogger(__name__) + + def __str__(self): + """Return the Response data if object is converted to a string.""" + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + return f"{self.data}" + + def __contains__(self, key: str) -> bool: + return self.get(key) is not None + + def __getitem__(self, key): + """Retrieves any key from the data store. + + Note: + This is implemented so users can reference the + SlackResponse object like a dictionary. + e.g. response["ok"] + + Returns: + The value from data or None. + """ + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + if self.data is None: + raise ValueError("As the response.data is empty, this operation is unsupported") + return self.data.get(key, None) + + def __aiter__(self): + """Enables the ability to iterate over the response. + It's required async-for the iterator protocol. + + Note: + This enables Slack cursor-based pagination. + + Returns: + (AsyncSlackResponse) self + """ + self._iteration = 0 + self.data = self._initial_data + return self + + async def __anext__(self): + """Retrieves the next portion of results, if 'next_cursor' is present. + + Note: + Some responses return collections of information + like channel and user lists. If they do it's likely + that you'll only receive a portion of results. This + method allows you to iterate over the response until + your code hits 'break' or there are no more results + to be found. + + Returns: + (AsyncSlackResponse) self + With the new response data now attached to this object. + + Raises: + SlackApiError: If the request to the Slack API failed. + StopAsyncIteration: If 'next_cursor' is not present or empty. + """ + self._iteration += 1 + if self._iteration == 1: + return self + if _next_cursor_is_present(self.data): # skipcq: PYL-R1705 + params = self.req_args.get("params", {}) + if params is None: + params = {} + params.update({"cursor": self.data["response_metadata"]["next_cursor"]}) + self.req_args.update({"params": params}) + + response = await self._client._request( # skipcq: PYL-W0212 + http_verb=self.http_verb, + api_url=self.api_url, + req_args=self.req_args, + ) + + self.data = response["data"] + self.headers = response["headers"] + self.status_code = response["status_code"] + return self.validate() + else: + raise StopAsyncIteration + + def get(self, key, default=None): + """Retrieves any key from the response data. + + Note: + This is implemented so users can reference the + SlackResponse object like a dictionary. + e.g. response.get("ok", False) + + Returns: + The value from data or the specified default. + """ + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + if self.data is None: + return None + return self.data.get(key, default) + + def validate(self): + """Check if the response from Slack was successful. + + Returns: + (AsyncSlackResponse) + This method returns it's own object. e.g. 'self' + + Raises: + SlackApiError: The request to the Slack API failed. + """ + if self.status_code == 200 and self.data and self.data.get("ok", False): + return self + msg = "The request to the Slack API failed." + raise e.SlackApiError(message=msg, response=self) diff --git a/core_service/aws_lambda/project/packages/slack/web/base_client.py b/core_service/aws_lambda/project/packages/slack/web/base_client.py new file mode 100644 index 0000000..a5b4356 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/base_client.py @@ -0,0 +1,497 @@ +"""A Python module for interacting with Slack's Web API.""" + +import asyncio +import copy +import hashlib +import hmac +import io +import json +import logging +import mimetypes +import urllib +import uuid +import warnings +from http.client import HTTPResponse +from ssl import SSLContext +from typing import BinaryIO, Dict, List +from typing import Optional, Union +from urllib.error import HTTPError +from urllib.parse import urlencode +from urllib.request import Request, urlopen, OpenerDirector, ProxyHandler, HTTPSHandler + +import aiohttp +from aiohttp import FormData, BasicAuth + +import slack.errors as err +from slack.errors import SlackRequestError +from slack.web import convert_bool_to_0_or_1, get_user_agent +from slack.web.async_internal_utils import ( + _get_event_loop, + _build_req_args, + _get_url, + _files_to_data, + _request_with_session, +) +from slack.web.deprecation import show_2020_01_deprecation +from slack.web.slack_response import SlackResponse + + +class BaseClient: + BASE_URL = "https://www.slack.com/api/" + + def __init__( + self, + token: Optional[str] = None, + base_url: str = BASE_URL, + timeout: int = 30, + loop: Optional[asyncio.AbstractEventLoop] = None, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + run_async: bool = False, + use_sync_aiohttp: bool = False, + session: Optional[aiohttp.ClientSession] = None, + headers: Optional[dict] = None, + user_agent_prefix: Optional[str] = None, + user_agent_suffix: Optional[str] = None, + ): + self.token = None if token is None else token.strip() + self.base_url = base_url + self.timeout = timeout + self.ssl = ssl + self.proxy = proxy + self.run_async = run_async + self.use_sync_aiohttp = use_sync_aiohttp + self.session = session + self.headers = headers or {} + self.headers["User-Agent"] = get_user_agent(user_agent_prefix, user_agent_suffix) + self._logger = logging.getLogger(__name__) + self._event_loop = loop + + def api_call( # skipcq: PYL-R1710 + self, + api_method: str, + *, + http_verb: str = "POST", + files: dict = None, + data: Union[dict, FormData] = None, + params: dict = None, + json: dict = None, # skipcq: PYL-W0621 + headers: dict = None, + auth: dict = None, + ) -> Union[asyncio.Future, SlackResponse]: + """Create a request and execute the API call to Slack. + + Args: + api_method (str): The target Slack API method. + e.g. 'chat.postMessage' + http_verb (str): HTTP Verb. e.g. 'POST' + files (dict): Files to multipart upload. + e.g. {image OR file: file_object OR file_path} + data: The body to attach to the request. If a dictionary is + provided, form-encoding will take place. + e.g. {'key1': 'value1', 'key2': 'value2'} + params (dict): The URL parameters to append to the URL. + e.g. {'key1': 'value1', 'key2': 'value2'} + json (dict): JSON for the body to attach to the request + (if files or data is not specified). + e.g. {'key1': 'value1', 'key2': 'value2'} + headers (dict): Additional request headers + auth (dict): A dictionary that consists of client_id and client_secret + + Returns: + (SlackResponse) + The server's response to an HTTP request. Data + from the response can be accessed like a dict. + If the response included 'next_cursor' it can + be iterated on to execute subsequent requests. + + Raises: + SlackApiError: The following Slack API call failed: + 'chat.postMessage'. + SlackRequestError: Json data can only be submitted as + POST requests. + """ + + api_url = _get_url(self.base_url, api_method) + headers = headers or {} + headers.update(self.headers) + + req_args = _build_req_args( + token=self.token, + http_verb=http_verb, + files=files, + data=data, + params=params, + json=json, # skipcq: PYL-W0621 + headers=headers, + auth=auth, + ssl=self.ssl, + proxy=self.proxy, + ) + + show_2020_01_deprecation(api_method) + + if self.run_async or self.use_sync_aiohttp: + if self._event_loop is None: + self._event_loop = _get_event_loop() + + future = asyncio.ensure_future( + self._send(http_verb=http_verb, api_url=api_url, req_args=req_args), + loop=self._event_loop, + ) + if self.run_async: + return future + if self.use_sync_aiohttp: + # Using this is no longer recommended - just keep this for backward-compatibility + return self._event_loop.run_until_complete(future) + else: + return self._sync_send(api_url=api_url, req_args=req_args) + + # ================================================================= + # aiohttp based async WebClient + # ================================================================= + + async def _send(self, http_verb: str, api_url: str, req_args: dict) -> SlackResponse: + """Sends the request out for transmission. + + Args: + http_verb (str): The HTTP verb. e.g. 'GET' or 'POST'. + api_url (str): The Slack API url. e.g. 'https://slack.com/api/chat.postMessage' + req_args (dict): The request arguments to be attached to the request. + e.g. + { + json: { + 'attachments': [{"pretext": "pre-hello", "text": "text-world"}], + 'channel': '#random' + } + } + Returns: + The response parsed into a SlackResponse object. + """ + open_files = _files_to_data(req_args) + try: + if "params" in req_args: + # True/False -> "1"/"0" + req_args["params"] = convert_bool_to_0_or_1(req_args["params"]) + + res = await self._request(http_verb=http_verb, api_url=api_url, req_args=req_args) + finally: + for f in open_files: + f.close() + + data = { + "client": self, + "http_verb": http_verb, + "api_url": api_url, + "req_args": req_args, + "use_sync_aiohttp": self.use_sync_aiohttp, + } + return SlackResponse(**{**data, **res}).validate() + + async def _request(self, *, http_verb, api_url, req_args) -> Dict[str, any]: + """Submit the HTTP request with the running session or a new session. + Returns: + A dictionary of the response data. + """ + return await _request_with_session( + current_session=self.session, + timeout=self.timeout, + logger=self._logger, + http_verb=http_verb, + api_url=api_url, + req_args=req_args, + ) + + # ================================================================= + # urllib based WebClient + # ================================================================= + + def _sync_send(self, api_url, req_args) -> SlackResponse: + params = req_args["params"] if "params" in req_args else None + data = req_args["data"] if "data" in req_args else None + files = req_args["files"] if "files" in req_args else None + _json = req_args["json"] if "json" in req_args else None + headers = req_args["headers"] if "headers" in req_args else None + token = params.get("token") if params and "token" in params else None + auth = req_args["auth"] if "auth" in req_args else None # Basic Auth for oauth.v2.access / oauth.access + if auth is not None: + if isinstance(auth, BasicAuth): + headers["Authorization"] = auth.encode() + elif isinstance(auth, str): + headers["Authorization"] = auth + else: + self._logger.warning(f"As the auth: {auth}: {type(auth)} is unsupported, skipped") + + body_params = {} + if params: + body_params.update(params) + if data: + body_params.update(data) + + return self._urllib_api_call( + token=token, + url=api_url, + query_params={}, + body_params=body_params, + files=files, + json_body=_json, + additional_headers=headers, + ) + + def _request_for_pagination(self, api_url, req_args) -> Dict[str, any]: + """This method is supposed to be used only for SlackResponse pagination + + You can paginate using Python's for iterator as below: + + for response in client.conversations_list(limit=100): + # do something with each response here + """ + response = self._perform_urllib_http_request(url=api_url, args=req_args) + return { + "status_code": int(response["status"]), + "headers": dict(response["headers"]), + "data": json.loads(response["body"]), + } + + def _urllib_api_call( + self, + *, + token: str = None, + url: str, + query_params: Dict[str, str] = {}, + json_body: Dict = {}, + body_params: Dict[str, str] = {}, + files: Dict[str, io.BytesIO] = {}, + additional_headers: Dict[str, str] = {}, + ) -> SlackResponse: + files_to_close: List[BinaryIO] = [] + try: + # True/False -> "1"/"0" + query_params = convert_bool_to_0_or_1(query_params) + body_params = convert_bool_to_0_or_1(body_params) + + if self._logger.level <= logging.DEBUG: + + def convert_params(values: dict) -> dict: + if not values or not isinstance(values, dict): + return {} + return {k: ("(bytes)" if isinstance(v, bytes) else v) for k, v in values.items()} + + headers = {k: "(redacted)" if k.lower() == "authorization" else v for k, v in additional_headers.items()} + self._logger.debug( + f"Sending a request - url: {url}, " + f"query_params: {convert_params(query_params)}, " + f"body_params: {convert_params(body_params)}, " + f"files: {convert_params(files)}, " + f"json_body: {json_body}, " + f"headers: {headers}" + ) + + request_data = {} + if files is not None and isinstance(files, dict) and len(files) > 0: + if body_params: + for k, v in body_params.items(): + request_data.update({k: v}) + + for k, v in files.items(): + if isinstance(v, str): + f: BinaryIO = open(v.encode("utf-8", "ignore"), "rb") + files_to_close.append(f) + request_data.update({k: f}) + elif isinstance(v, (bytearray, bytes)): + request_data.update({k: io.BytesIO(v)}) + else: + request_data.update({k: v}) + + request_headers = self._build_urllib_request_headers( + token=token or self.token, + has_json=json is not None, + has_files=files is not None, + additional_headers=additional_headers, + ) + request_args = { + "headers": request_headers, + "data": request_data, + "params": body_params, + "files": files, + "json": json_body, + } + if query_params: + q = urlencode(query_params) + url = f"{url}&{q}" if "?" in url else f"{url}?{q}" + + response = self._perform_urllib_http_request(url=url, args=request_args) + if response.get("body"): + try: + response_body_data: dict = json.loads(response["body"]) + except json.decoder.JSONDecodeError as e: + message = f"Failed to parse the response body: {str(e)}" + raise err.SlackApiError(message, response) + else: + response_body_data: dict = None + + if query_params: + all_params = copy.copy(body_params) + all_params.update(query_params) + else: + all_params = body_params + request_args["params"] = all_params # for backward-compatibility + + return SlackResponse( + client=self, + http_verb="POST", # you can use POST method for all the Web APIs + api_url=url, + req_args=request_args, + data=response_body_data, + headers=dict(response["headers"]), + status_code=response["status"], + use_sync_aiohttp=False, + ).validate() + finally: + for f in files_to_close: + if not f.closed: + f.close() + + def _perform_urllib_http_request(self, *, url: str, args: Dict[str, Dict[str, any]]) -> Dict[str, any]: + headers = args["headers"] + if args["json"]: + body = json.dumps(args["json"]) + headers["Content-Type"] = "application/json;charset=utf-8" + elif args["data"]: + boundary = f"--------------{uuid.uuid4()}" + sep_boundary = b"\r\n--" + boundary.encode("ascii") + end_boundary = sep_boundary + b"--\r\n" + body = io.BytesIO() + data = args["data"] + for key, value in data.items(): + readable = getattr(value, "readable", None) + if readable and value.readable(): + filename = "Uploaded file" + name_attr = getattr(value, "name", None) + if name_attr: + filename = name_attr.decode("utf-8") if isinstance(name_attr, bytes) else name_attr + if "filename" in data: + filename = data["filename"] + mimetype = mimetypes.guess_type(filename)[0] or "application/octet-stream" + title = ( + f'\r\nContent-Disposition: form-data; name="{key}"; filename="{filename}"\r\n' + + f"Content-Type: {mimetype}\r\n" + ) + value = value.read() + else: + title = f'\r\nContent-Disposition: form-data; name="{key}"\r\n' + value = str(value).encode("utf-8") + body.write(sep_boundary) + body.write(title.encode("utf-8")) + body.write(b"\r\n") + body.write(value) + + body.write(end_boundary) + body = body.getvalue() + headers["Content-Type"] = f"multipart/form-data; boundary={boundary}" + headers["Content-Length"] = len(body) + elif args["params"]: + body = urlencode(args["params"]) + headers["Content-Type"] = "application/x-www-form-urlencoded" + else: + body = None + + if isinstance(body, str): + body = body.encode("utf-8") + + # NOTE: Intentionally ignore the `http_verb` here + # Slack APIs accepts any API method requests with POST methods + try: + # urllib not only opens http:// or https:// URLs, but also ftp:// and file://. + # With this it might be possible to open local files on the executing machine + # which might be a security risk if the URL to open can be manipulated by an external user. + # (BAN-B310) + if url.lower().startswith("http"): + req = Request(method="POST", url=url, data=body, headers=headers) + opener: Optional[OpenerDirector] = None + if self.proxy is not None: + if isinstance(self.proxy, str): + opener = urllib.request.build_opener( + ProxyHandler({"http": self.proxy, "https": self.proxy}), + HTTPSHandler(context=self.ssl), + ) + else: + raise SlackRequestError(f"Invalid proxy detected: {self.proxy} must be a str value") + + # NOTE: BAN-B310 is already checked above + resp: Optional[HTTPResponse] = None + if opener: + resp = opener.open(req, timeout=self.timeout) # skipcq: BAN-B310 + else: + resp = urlopen(req, context=self.ssl, timeout=self.timeout) # skipcq: BAN-B310 + charset = resp.headers.get_content_charset() or "utf-8" + body: str = resp.read().decode(charset) # read the response body here + return {"status": resp.code, "headers": resp.headers, "body": body} + raise SlackRequestError(f"Invalid URL detected: {url}") + except HTTPError as e: + resp = {"status": e.code, "headers": e.headers} + if e.code == 429: + # for compatibility with aiohttp + resp["headers"]["Retry-After"] = resp["headers"]["retry-after"] + + charset = e.headers.get_content_charset() or "utf-8" + body: str = e.read().decode(charset) # read the response body here + resp["body"] = body + return resp + + except Exception as err: + self._logger.error(f"Failed to send a request to Slack API server: {err}") + raise err + + def _build_urllib_request_headers( + self, token: str, has_json: bool, has_files: bool, additional_headers: dict + ) -> Dict[str, str]: + headers = {"Content-Type": "application/x-www-form-urlencoded"} + headers.update(self.headers) + if token: + headers.update({"Authorization": "Bearer {}".format(token)}) + if additional_headers: + headers.update(additional_headers) + if has_json: + headers.update({"Content-Type": "application/json;charset=utf-8"}) + if has_files: + # will be set afterwards + headers.pop("Content-Type", None) + return headers + + # ================================================================= + + @staticmethod + def validate_slack_signature(*, signing_secret: str, data: str, timestamp: str, signature: str) -> bool: + """ + Slack creates a unique string for your app and shares it with you. Verify + requests from Slack with confidence by verifying signatures using your + signing secret. + + On each HTTP request that Slack sends, we add an X-Slack-Signature HTTP + header. The signature is created by combining the signing secret with the + body of the request we're sending using a standard HMAC-SHA256 keyed hash. + + https://api.slack.com/docs/verifying-requests-from-slack#how_to_make_a_request_signature_in_4_easy_steps__an_overview + + Args: + signing_secret: Your application's signing secret, available in the + Slack API dashboard + data: The raw body of the incoming request - no headers, just the body. + timestamp: from the 'X-Slack-Request-Timestamp' header + signature: from the 'X-Slack-Signature' header - the calculated signature + should match this. + + Returns: + True if signatures matches + """ + warnings.warn( + "As this method is deprecated since slackclient 2.6.0, " + "use `from slack.signature import SignatureVerifier` instead", + DeprecationWarning, + ) + format_req = str.encode(f"v0:{timestamp}:{data}") + encoded_secret = str.encode(signing_secret) + request_hash = hmac.new(encoded_secret, format_req, hashlib.sha256).hexdigest() + calculated_signature = f"v0={request_hash}" + return hmac.compare_digest(calculated_signature, signature) diff --git a/core_service/aws_lambda/project/packages/slack/web/classes/__init__.py b/core_service/aws_lambda/project/packages/slack/web/classes/__init__.py new file mode 100644 index 0000000..58311a6 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/classes/__init__.py @@ -0,0 +1,10 @@ +from slack_sdk.models import BaseObject # noqa +from slack_sdk.models import JsonObject # noqa +from slack_sdk.models import JsonValidator # noqa +from slack_sdk.models import EnumValidator # noqa +from slack_sdk.models import extract_json # noqa +from slack_sdk.models import show_unknown_key_warning # noqa + +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.models") diff --git a/core_service/aws_lambda/project/packages/slack/web/classes/actions.py b/core_service/aws_lambda/project/packages/slack/web/classes/actions.py new file mode 100644 index 0000000..6f2ded5 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/classes/actions.py @@ -0,0 +1,13 @@ +from slack_sdk.models.attachments import AbstractActionSelector # noqa +from slack_sdk.models.attachments import Action # noqa +from slack_sdk.models.attachments import ActionButton # noqa +from slack_sdk.models.attachments import ActionChannelSelector # noqa +from slack_sdk.models.attachments import ActionConversationSelector # noqa +from slack_sdk.models.attachments import ActionExternalSelector # noqa +from slack_sdk.models.attachments import ActionLinkButton # noqa +from slack_sdk.models.attachments import ActionUserSelector # noqa +from slack_sdk.models.dialogs import ActionStaticSelector # noqa + +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.models.attachments/dialogs") diff --git a/core_service/aws_lambda/project/packages/slack/web/classes/attachments.py b/core_service/aws_lambda/project/packages/slack/web/classes/attachments.py new file mode 100644 index 0000000..3f85c40 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/classes/attachments.py @@ -0,0 +1,9 @@ +from slack_sdk.models.attachments import Attachment # noqa +from slack_sdk.models.attachments import AttachmentField # noqa +from slack_sdk.models.attachments import BlockAttachment # noqa +from slack_sdk.models.attachments import InteractiveAttachment # noqa +from slack_sdk.models.attachments import SeededColors # noqa + +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.models.attachments") diff --git a/core_service/aws_lambda/project/packages/slack/web/classes/blocks.py b/core_service/aws_lambda/project/packages/slack/web/classes/blocks.py new file mode 100644 index 0000000..6ad4fc7 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/classes/blocks.py @@ -0,0 +1,13 @@ +from slack import deprecation +from slack_sdk.models.blocks import ActionsBlock # noqa +from slack_sdk.models.blocks import Block # noqa +from slack_sdk.models.blocks import CallBlock # noqa +from slack_sdk.models.blocks import ContextBlock # noqa +from slack_sdk.models.blocks import DividerBlock # noqa +from slack_sdk.models.blocks import FileBlock # noqa +from slack_sdk.models.blocks import HeaderBlock # noqa +from slack_sdk.models.blocks import ImageBlock # noqa +from slack_sdk.models.blocks import InputBlock # noqa +from slack_sdk.models.blocks import SectionBlock # noqa + +deprecation.show_message(__name__, "slack_sdk.models.blocks") diff --git a/core_service/aws_lambda/project/packages/slack/web/classes/dialog_elements.py b/core_service/aws_lambda/project/packages/slack/web/classes/dialog_elements.py new file mode 100644 index 0000000..00b3126 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/classes/dialog_elements.py @@ -0,0 +1,14 @@ +from slack_sdk.models.dialogs import AbstractDialogSelector # noqa +from slack_sdk.models.dialogs import DialogChannelSelector # noqa +from slack_sdk.models.dialogs import DialogConversationSelector # noqa +from slack_sdk.models.dialogs import DialogExternalSelector # noqa +from slack_sdk.models.dialogs import DialogStaticSelector # noqa +from slack_sdk.models.dialogs import DialogTextArea # noqa +from slack_sdk.models.dialogs import DialogTextComponent # noqa +from slack_sdk.models.dialogs import DialogTextField # noqa +from slack_sdk.models.dialogs import DialogUserSelector # noqa +from slack_sdk.models.dialogs import TextElementSubtypes # noqa + +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.models.blocks") diff --git a/core_service/aws_lambda/project/packages/slack/web/classes/dialogs.py b/core_service/aws_lambda/project/packages/slack/web/classes/dialogs.py new file mode 100644 index 0000000..7e6d1cb --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/classes/dialogs.py @@ -0,0 +1,5 @@ +from slack_sdk.models.dialogs import DialogBuilder # noqa + +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.models.dialogs") diff --git a/core_service/aws_lambda/project/packages/slack/web/classes/elements.py b/core_service/aws_lambda/project/packages/slack/web/classes/elements.py new file mode 100644 index 0000000..ce2671f --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/classes/elements.py @@ -0,0 +1,27 @@ +from slack_sdk.models.blocks import BlockElement # noqa +from slack_sdk.models.blocks import ButtonElement # noqa +from slack_sdk.models.blocks import ChannelMultiSelectElement # noqa +from slack_sdk.models.blocks import ChannelSelectElement # noqa +from slack_sdk.models.blocks import CheckboxesElement # noqa +from slack_sdk.models.blocks import ConversationFilter # noqa +from slack_sdk.models.blocks import ConversationMultiSelectElement # noqa +from slack_sdk.models.blocks import ConversationSelectElement # noqa +from slack_sdk.models.blocks import DatePickerElement # noqa +from slack_sdk.models.blocks import ExternalDataMultiSelectElement # noqa +from slack_sdk.models.blocks import ExternalDataSelectElement # noqa +from slack_sdk.models.blocks import ImageElement # noqa +from slack_sdk.models.blocks import InputInteractiveElement # noqa +from slack_sdk.models.blocks import InteractiveElement # noqa +from slack_sdk.models.blocks import LinkButtonElement # noqa +from slack_sdk.models.blocks import OverflowMenuElement # noqa +from slack_sdk.models.blocks import PlainTextInputElement # noqa +from slack_sdk.models.blocks import RadioButtonsElement # noqa +from slack_sdk.models.blocks import SelectElement # noqa +from slack_sdk.models.blocks import StaticMultiSelectElement # noqa +from slack_sdk.models.blocks import StaticSelectElement # noqa +from slack_sdk.models.blocks import UserMultiSelectElement # noqa +from slack_sdk.models.blocks import UserSelectElement # noqa + +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.models.blocks") diff --git a/core_service/aws_lambda/project/packages/slack/web/classes/interactions.py b/core_service/aws_lambda/project/packages/slack/web/classes/interactions.py new file mode 100644 index 0000000..f4d871e --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/classes/interactions.py @@ -0,0 +1,137 @@ +import json +from typing import List, NamedTuple + +from . import BaseObject + + +class IDNamePair(NamedTuple): + """Simple type used to help with unpacking event data""" + + id: str + name: str + + +class InteractiveEvent(BaseObject): + response_url: str + user: IDNamePair + team: IDNamePair + channel: IDNamePair + + raw_event: dict + + def __init__(self, event: dict): + self.raw_event = event + self.response_url = event["response_url"] + + +class MessageInteractiveEvent(InteractiveEvent): + event_type: str + message_ts: str + trigger_id: str + action_id: str + block_id: str + message: dict + + def __init__(self, event: dict): + """ + Convenience class to parse an interactive message payload from the events API + + Args: + event: the raw event dictionary + """ + super().__init__(event) + self.user = IDNamePair(event["user"]["id"], event["user"]["username"]) + self.team: IDNamePair = IDNamePair(event["team"]["id"], event["team"]["domain"]) + self.channel: IDNamePair = IDNamePair(event["channel"]["id"], event["channel"]["name"]) + self.event_type = event["type"] + self.message_ts = event["message"]["ts"] + self.trigger_id = event["trigger_id"] + # actions payload is an array, but will only have one item (the action + # actually interacted with) + action = event["actions"][0] + self.action_id = action["action_id"] + self.block_id = action["block_id"] + if action.get("selected_option"): + self.value = action["selected_option"]["value"] + else: + self.value = action["value"] + self.message = event["message"] + + +class DialogInteractiveEvent(InteractiveEvent): + event_type: str + submission: dict + state: dict + + def __init__(self, event: dict): + """ + Convenience class to parse a dialog interaction payload from the events API + + Args: + event: the raw event dictionary + """ + super().__init__(event) + self.user = IDNamePair(event["user"]["id"], event["user"]["name"]) + self.team = IDNamePair(event["team"]["id"], event["team"]["domain"]) + self.channel = IDNamePair(event["channel"]["id"], event["channel"]["name"]) + self.callback_id = event["callback_id"] + self.event_type = event["type"] + self.submission = event["submission"] + if event["state"]: + self.state = json.loads(event["state"]) + else: + self.state = {} + + def require_any(self, requirements: List[str]) -> dict: + """ + Convenience method to construct the 'errors' response to send directly back to + the invoking HTTP request + + Args: + requirements: List of required dialog components, by name + """ + if any(self.submission.get(requirement, "") for requirement in requirements): # skipcq: PYL-R1705 + return {} + else: + errors = [] + for key in self.submission: + error_text = "At least one value is required" + errors.append({"name": key, "error": error_text}) + return {"errors": errors} + + +class SlashCommandInteractiveEvent(InteractiveEvent): + trigger_id: str + command: str + text: str + + def __init__(self, event: dict): + """ + Convenience class to parse a slash command payload from the events API + + Args: + event: the raw event dictionary + """ + super().__init__(event) + self.user = IDNamePair(event["user_id"], event["user_name"]) + self.channel = IDNamePair(event["channel_id"], event["channel_name"]) + self.team = IDNamePair(event["team_id"], event["team_domain"]) + self.trigger_id = event["trigger_id"] + self.command = event["command"] + self.text = event["text"] + + @staticmethod + def create_reply(message, ephemeral=False) -> dict: + """ + Create a reply suitable to send directly back to the invoking HTTP request + + Args: + message: Text to send + ephemeral: Whether the response should be limited to a single user, or to + broadcast the reply (_and_ the user's original invocation) to the + channel publicly + """ + if ephemeral: # skipcq: PYL-R1705 + return {"text": message, "response_type": "ephemeral"} + else: + return {"text": message, "response_type": "in_channel"} diff --git a/core_service/aws_lambda/project/packages/slack/web/classes/messages.py b/core_service/aws_lambda/project/packages/slack/web/classes/messages.py new file mode 100644 index 0000000..37b1dca --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/classes/messages.py @@ -0,0 +1 @@ +from slack_sdk.models.messages.message import Message # noqa diff --git a/core_service/aws_lambda/project/packages/slack/web/classes/objects.py b/core_service/aws_lambda/project/packages/slack/web/classes/objects.py new file mode 100644 index 0000000..6025aa0 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/classes/objects.py @@ -0,0 +1,19 @@ +from slack_sdk.models.blocks import ButtonStyles # noqa +from slack_sdk.models.blocks import ConfirmObject # noqa +from slack_sdk.models.blocks import DynamicSelectElementTypes # noqa +from slack_sdk.models.blocks import MarkdownTextObject # noqa +from slack_sdk.models.blocks import Option # noqa +from slack_sdk.models.blocks import OptionGroup # noqa +from slack_sdk.models.blocks import PlainTextObject # noqa +from slack_sdk.models.blocks import TextObject # noqa +from slack_sdk.models.messages import ChannelLink # noqa +from slack_sdk.models.messages import DateLink # noqa +from slack_sdk.models.messages import EveryoneLink # noqa +from slack_sdk.models.messages import HereLink # noqa +from slack_sdk.models.messages import Link # noqa +from slack_sdk.models.messages import ObjectLink # noqa + + +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.models.blocks/messages") diff --git a/core_service/aws_lambda/project/packages/slack/web/classes/views.py b/core_service/aws_lambda/project/packages/slack/web/classes/views.py new file mode 100644 index 0000000..a23f3ed --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/classes/views.py @@ -0,0 +1,7 @@ +from slack_sdk.models.views import View # noqa +from slack_sdk.models.views import ViewState # noqa +from slack_sdk.models.views import ViewStateValue # noqa + +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.models.views") diff --git a/core_service/aws_lambda/project/packages/slack/web/client.py b/core_service/aws_lambda/project/packages/slack/web/client.py new file mode 100644 index 0000000..82900bf --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/client.py @@ -0,0 +1,4 @@ +from slack import deprecation +from slack_sdk.web.legacy_client import LegacyWebClient as WebClient # noqa + +deprecation.show_message(__name__, "slack_sdk.web.client") diff --git a/core_service/aws_lambda/project/packages/slack/web/deprecation.py b/core_service/aws_lambda/project/packages/slack/web/deprecation.py new file mode 100644 index 0000000..5ce5f06 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/deprecation.py @@ -0,0 +1,30 @@ +import os +import warnings + +# https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api +deprecated_method_prefixes_2020_01 = [ + "channels.", + "groups.", + "im.", + "mpim.", + "admin.conversations.whitelist.", +] + + +def show_2020_01_deprecation(method_name: str): + """Prints a warning if the given method is deprecated""" + + skip_deprecation = os.environ.get("SLACKCLIENT_SKIP_DEPRECATION") # for unit tests etc. + if skip_deprecation: + return + if not method_name: + return + + matched_prefixes = [prefix for prefix in deprecated_method_prefixes_2020_01 if method_name.startswith(prefix)] + if len(matched_prefixes) > 0: + message = ( + f"{method_name} is deprecated. Please use the Conversations API instead. " + "For more info, go to " + "https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api" + ) + warnings.warn(message) diff --git a/core_service/aws_lambda/project/packages/slack/web/internal_utils.py b/core_service/aws_lambda/project/packages/slack/web/internal_utils.py new file mode 100644 index 0000000..67ce3d3 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/internal_utils.py @@ -0,0 +1,52 @@ +import json +from typing import Union, Dict, List + +from slack.errors import SlackRequestError +from slack.web.classes.attachments import Attachment +from slack.web.classes.blocks import Block + + +def _parse_web_class_objects(kwargs) -> None: + def to_dict(obj: Union[Dict, Block, Attachment]): + if isinstance(obj, Block): + return obj.to_dict() + if isinstance(obj, Attachment): + return obj.to_dict() + return obj + + blocks = kwargs.get("blocks", None) + if blocks is not None and isinstance(blocks, list): + dict_blocks = [to_dict(b) for b in blocks] + kwargs.update({"blocks": dict_blocks}) + + attachments = kwargs.get("attachments", None) + if attachments is not None and isinstance(attachments, list): + dict_attachments = [to_dict(a) for a in attachments] + kwargs.update({"attachments": dict_attachments}) + + +def _update_call_participants(kwargs, users: Union[str, List[Dict[str, str]]]) -> None: + if users is None: + return + + if isinstance(users, list): + kwargs.update({"users": json.dumps(users)}) + elif isinstance(users, str): + kwargs.update({"users": users}) + else: + raise SlackRequestError("users must be either str or List[Dict[str, str]]") + + +def _next_cursor_is_present(data) -> bool: + """Determine if the response contains 'next_cursor' + and 'next_cursor' is not empty. + + Returns: + A boolean value. + """ + present = ( + "response_metadata" in data + and "next_cursor" in data["response_metadata"] + and data["response_metadata"]["next_cursor"] != "" + ) + return present diff --git a/core_service/aws_lambda/project/packages/slack/web/slack_response.py b/core_service/aws_lambda/project/packages/slack/web/slack_response.py new file mode 100644 index 0000000..a6c9cde --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/web/slack_response.py @@ -0,0 +1,6 @@ +from slack import deprecation +from slack_sdk.web.legacy_slack_response import ( # noqa + LegacySlackResponse as SlackResponse, +) + +deprecation.show_message(__name__, "slack_sdk.web.slack_response") diff --git a/core_service/aws_lambda/project/packages/slack/webhook/__init__.py b/core_service/aws_lambda/project/packages/slack/webhook/__init__.py new file mode 100644 index 0000000..6b03c46 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/webhook/__init__.py @@ -0,0 +1,7 @@ +from slack_sdk.webhook.webhook_response import WebhookResponse # noqa +from slack_sdk.webhook.client import WebhookClient # noqa +from slack_sdk.webhook.async_client import AsyncWebhookClient # noqa + +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.webhook") diff --git a/core_service/aws_lambda/project/packages/slack/webhook/async_client.py b/core_service/aws_lambda/project/packages/slack/webhook/async_client.py new file mode 100644 index 0000000..31310c9 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/webhook/async_client.py @@ -0,0 +1,119 @@ +import json +import logging +from ssl import SSLContext +from typing import Dict, Union, List, Optional + +import aiohttp +from aiohttp import BasicAuth, ClientSession + +from slack.errors import SlackApiError +from .internal_utils import _debug_log_response, _build_request_headers, _build_body +from .webhook_response import WebhookResponse +from ..web.classes.attachments import Attachment +from ..web.classes.blocks import Block + + +class AsyncWebhookClient: + logger = logging.getLogger(__name__) + + def __init__( + self, + url: str, + timeout: int = 30, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + session: Optional[ClientSession] = None, + trust_env_in_session: bool = False, + auth: Optional[BasicAuth] = None, + default_headers: Optional[Dict[str, str]] = None, + ): + self.url = url + self.timeout = timeout + self.ssl = ssl + self.proxy = proxy + self.trust_env_in_session = trust_env_in_session + self.session = session + self.auth = auth + self.default_headers = default_headers if default_headers else {} + + async def send( + self, + *, + text: Optional[str] = None, + attachments: Optional[List[Union[Dict[str, any], Attachment]]] = None, + blocks: Optional[List[Union[Dict[str, any], Block]]] = None, + response_type: Optional[str] = None, + headers: Optional[Dict[str, str]] = None, + ) -> WebhookResponse: + """Performs a Slack API request and returns the result. + + Args: + text: The text message (even when having blocks, setting this as well is recommended as it works as fallback) + attachments: A collection of attachments + blocks: A collection of Block Kit UI components + response_type: The type of message (either 'in_channel' or 'ephemeral') + headers: Request headers to append only for this request + Returns: + Webhook response + """ + return await self.send_dict( + body={ + "text": text, + "attachments": attachments, + "blocks": blocks, + "response_type": response_type, + }, + headers=headers, + ) + + async def send_dict(self, body: Dict[str, any], headers: Optional[Dict[str, str]] = None) -> WebhookResponse: + return await self._perform_http_request( + body=_build_body(body), + headers=_build_request_headers(self.default_headers, headers), + ) + + async def _perform_http_request(self, *, body: Dict[str, any], headers: Dict[str, str]) -> WebhookResponse: + body = json.dumps(body) + headers["Content-Type"] = "application/json;charset=utf-8" + + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"Sending a request - url: {self.url}, body: {body}, headers: {headers}") + session: Optional[ClientSession] = None + use_running_session = self.session and not self.session.closed + if use_running_session: + session = self.session + else: + session = aiohttp.ClientSession( + timeout=aiohttp.ClientTimeout(total=self.timeout), + auth=self.auth, + trust_env=self.trust_env_in_session, + ) + + try: + request_kwargs = { + "headers": headers, + "data": body, + "ssl": self.ssl, + "proxy": self.proxy, + } + async with session.request("POST", self.url, **request_kwargs) as res: + response_body = {} + try: + response_body = await res.text() + except aiohttp.ContentTypeError: + self._logger.debug(f"No response data returned from the following API call: {self.url}.") + except json.decoder.JSONDecodeError as e: + message = f"Failed to parse the response body: {str(e)}" + raise SlackApiError(message, res) + + resp = WebhookResponse( + url=self.url, + status_code=res.status, + body=response_body, + headers=res.headers, + ) + _debug_log_response(self.logger, resp) + return resp + finally: + if not use_running_session: + await session.close() diff --git a/core_service/aws_lambda/project/packages/slack/webhook/client.py b/core_service/aws_lambda/project/packages/slack/webhook/client.py new file mode 100644 index 0000000..7d39c03 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/webhook/client.py @@ -0,0 +1,116 @@ +import json +import logging +import urllib +from http.client import HTTPResponse +from ssl import SSLContext +from typing import Dict, Union, List, Optional +from urllib.error import HTTPError +from urllib.request import Request, urlopen, OpenerDirector, ProxyHandler, HTTPSHandler + +from slack.errors import SlackRequestError +from .internal_utils import _build_body, _build_request_headers, _debug_log_response +from .webhook_response import WebhookResponse +from ..web.classes.attachments import Attachment +from ..web.classes.blocks import Block + + +class WebhookClient: + logger = logging.getLogger(__name__) + + def __init__( + self, + url: str, + timeout: int = 30, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + default_headers: Optional[Dict[str, str]] = None, + ): + self.url = url + self.timeout = timeout + self.ssl = ssl + self.proxy = proxy + self.default_headers = default_headers if default_headers else {} + + def send( + self, + *, + text: Optional[str] = None, + attachments: Optional[List[Union[Dict[str, any], Attachment]]] = None, + blocks: Optional[List[Union[Dict[str, any], Block]]] = None, + response_type: Optional[str] = None, + headers: Optional[Dict[str, str]] = None, + ) -> WebhookResponse: + return self.send_dict( + body={ + "text": text, + "attachments": attachments, + "blocks": blocks, + "response_type": response_type, + }, + headers=headers, + ) + + def send_dict(self, body: Dict[str, any], headers: Optional[Dict[str, str]] = None) -> WebhookResponse: + return self._perform_http_request( + body=_build_body(body), + headers=_build_request_headers(self.default_headers, headers), + ) + + def _perform_http_request(self, *, body: Dict[str, any], headers: Dict[str, str]) -> WebhookResponse: + body = json.dumps(body) + headers["Content-Type"] = "application/json;charset=utf-8" + + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"Sending a request - url: {self.url}, body: {body}, headers: {headers}") + try: + url = self.url + opener: Optional[OpenerDirector] = None + # for security (BAN-B310) + if url.lower().startswith("http"): + req = Request(method="POST", url=url, data=body.encode("utf-8"), headers=headers) + if self.proxy is not None: + if isinstance(self.proxy, str): + opener = urllib.request.build_opener( + ProxyHandler({"http": self.proxy, "https": self.proxy}), + HTTPSHandler(context=self.ssl), + ) + else: + raise SlackRequestError(f"Invalid proxy detected: {self.proxy} must be a str value") + else: + raise SlackRequestError(f"Invalid URL detected: {url}") + + # NOTE: BAN-B310 is already checked above + resp: Optional[HTTPResponse] = None + if opener: + resp = opener.open(req, timeout=self.timeout) # skipcq: BAN-B310 + else: + resp = urlopen(req, context=self.ssl, timeout=self.timeout) # skipcq: BAN-B310 + charset: str = resp.headers.get_content_charset() or "utf-8" + response_body: str = resp.read().decode(charset) + resp = WebhookResponse( + url=url, + status_code=resp.status, + body=response_body, + headers=resp.headers, + ) + _debug_log_response(self.logger, resp) + return resp + + except HTTPError as e: + charset = e.headers.get_content_charset() or "utf-8" + body: str = e.read().decode(charset) # read the response body here + resp = WebhookResponse( + url=url, + status_code=e.code, + body=body, + headers=e.headers, + ) + if e.code == 429: + # for backward-compatibility with WebClient (v.2.5.0 or older) + resp.headers["Retry-After"] = resp.headers["retry-after"] + _debug_log_response(self.logger, resp) + return resp + + except Exception as err: + self.logger.error(f"Failed to send a request to Slack API server: {err}") + raise err diff --git a/core_service/aws_lambda/project/packages/slack/webhook/internal_utils.py b/core_service/aws_lambda/project/packages/slack/webhook/internal_utils.py new file mode 100644 index 0000000..ea21e88 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/webhook/internal_utils.py @@ -0,0 +1,40 @@ +import logging +from typing import Optional, Dict + +from slack.web import get_user_agent, convert_bool_to_0_or_1 +from slack.web.internal_utils import _parse_web_class_objects +from slack.webhook import WebhookResponse + + +def _build_body(original_body: Dict[str, any]) -> Dict[str, any]: + body = {k: v for k, v in original_body.items() if v is not None} + body = convert_bool_to_0_or_1(body) + _parse_web_class_objects(body) + return body + + +def _build_request_headers( + default_headers: Dict[str, str], + additional_headers: Optional[Dict[str, str]], +) -> Dict[str, str]: + if additional_headers is None: + return {} + + request_headers = { + "User-Agent": get_user_agent(), + "Content-Type": "application/json;charset=utf-8", + } + request_headers.update(default_headers) + if additional_headers: + request_headers.update(additional_headers) + return request_headers + + +def _debug_log_response(logger, resp: WebhookResponse) -> None: + if logger.level <= logging.DEBUG: + logger.debug( + "Received the following response - " + f"status: {resp.status_code}, " + f"headers: {(dict(resp.headers))}, " + f"body: {resp.body}" + ) diff --git a/core_service/aws_lambda/project/packages/slack/webhook/webhook_response.py b/core_service/aws_lambda/project/packages/slack/webhook/webhook_response.py new file mode 100644 index 0000000..5da6b80 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack/webhook/webhook_response.py @@ -0,0 +1,13 @@ +class WebhookResponse: + def __init__( + self, + *, + url: str, + status_code: int, + body: str, + headers: dict, + ): + self.api_url = url + self.status_code = status_code + self.body = body + self.headers = headers diff --git a/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/INSTALLER b/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/INSTALLER new file mode 100644 index 0000000..a1b589e --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/INSTALLER @@ -0,0 +1 @@ +pip diff --git a/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/LICENSE b/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/LICENSE new file mode 100644 index 0000000..a638457 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/LICENSE @@ -0,0 +1,21 @@ +The MIT License (MIT) + +Copyright (c) 2015- Slack Technologies, LLC + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. diff --git a/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/METADATA b/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/METADATA new file mode 100644 index 0000000..2dcc26b --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/METADATA @@ -0,0 +1,350 @@ +Metadata-Version: 2.1 +Name: slack-sdk +Version: 3.17.2 +Summary: The Slack API Platform SDK for Python +Home-page: https://github.com/slackapi/python-slack-sdk +Author: Slack Technologies, LLC +Author-email: opensource@slack.com +License: MIT +Keywords: slack slack-api web-api slack-rtm websocket chat chatbot chatops +Platform: UNKNOWN +Classifier: Development Status :: 5 - Production/Stable +Classifier: Intended Audience :: Developers +Classifier: Topic :: Communications :: Chat +Classifier: Topic :: System :: Networking +Classifier: Topic :: Office/Business +Classifier: License :: OSI Approved :: MIT License +Classifier: Programming Language :: Python +Classifier: Programming Language :: Python :: 3 +Classifier: Programming Language :: Python :: 3.6 +Classifier: Programming Language :: Python :: 3.7 +Classifier: Programming Language :: Python :: 3.8 +Classifier: Programming Language :: Python :: 3.9 +Classifier: Programming Language :: Python :: 3.10 +Requires-Python: >=3.6.0 +Description-Content-Type: text/markdown +License-File: LICENSE +Provides-Extra: optional +Requires-Dist: aiodns (>1.0) ; extra == 'optional' +Requires-Dist: aiohttp (<4,>=3.7.3) ; extra == 'optional' +Requires-Dist: boto3 (<=2) ; extra == 'optional' +Requires-Dist: SQLAlchemy (<2,>=1) ; extra == 'optional' +Requires-Dist: websockets (<11,>=10) ; extra == 'optional' +Requires-Dist: websocket-client (<2,>=1) ; extra == 'optional' +Provides-Extra: testing +Requires-Dist: pytest (<7,>=6.2.5) ; extra == 'testing' +Requires-Dist: pytest-asyncio (<1) ; extra == 'testing' +Requires-Dist: Flask-Sockets (<1,>=0.2) ; extra == 'testing' +Requires-Dist: Flask (<2,>=1) ; extra == 'testing' +Requires-Dist: Werkzeug (<2) ; extra == 'testing' +Requires-Dist: itsdangerous (==1.1.0) ; extra == 'testing' +Requires-Dist: Jinja2 (==3.0.3) ; extra == 'testing' +Requires-Dist: pytest-cov (<3,>=2) ; extra == 'testing' +Requires-Dist: codecov (<3,>=2) ; extra == 'testing' +Requires-Dist: flake8 (<5,>=4) ; extra == 'testing' +Requires-Dist: black (==22.3.0) ; extra == 'testing' +Requires-Dist: click (==8.0.4) ; extra == 'testing' +Requires-Dist: psutil (<6,>=5) ; extra == 'testing' +Requires-Dist: databases (>=0.5) ; extra == 'testing' +Requires-Dist: boto3 (<=2) ; extra == 'testing' +Requires-Dist: moto (<4,>=3) ; extra == 'testing' + +# Python Slack SDK + +The Slack platform offers several APIs to build apps. Each Slack API delivers part of the capabilities from the platform, so that you can pick just those that fit for your needs. This SDK offers a corresponding package for each of Slack’s APIs. They are small and powerful when used independently, and work seamlessly when used together, too. + +**Comprehensive documentation on using the Slack Python can be found at [https://slack.dev/python-slack-sdk/](https://slack.dev/python-slack-sdk/)** + +[![pypi package][pypi-image]][pypi-url] +[![Build Status][build-image]][build-url] +[![Python Version][python-version]][pypi-url] +[![codecov][codecov-image]][codecov-url] +[![contact][contact-image]][contact-url] + +Whether you're building a custom app for your team, or integrating a third party service into your Slack workflows, Slack Developer Kit for Python allows you to leverage the flexibility of Python to get your project up and running as quickly as possible. + +The **Python Slack SDK** allows interaction with: + +- `slack_sdk.web`: for calling the [Web API methods][api-methods] +- `slack_sdk.webhook`: for utilizing the [Incoming Webhooks](https://api.slack.com/messaging/webhooks) and [`response_url`s in payloads](https://api.slack.com/interactivity/handling#message_responses) +- `slack_sdk.signature`: for [verifying incoming requests from the Slack API server](https://api.slack.com/authentication/verifying-requests-from-slack) +- `slack_sdk.socket_mode`: for receiving and sending messages over [Socket Mode](https://api.slack.com/socket-mode) connections +- `slack_sdk.audit_logs`: for utilizing [Audit Logs APIs](https://api.slack.com/admins/audit-logs) +- `slack_sdk.scim`: for utilizing [SCIM APIs](https://api.slack.com/admins/scim) +- `slack_sdk.oauth`: for implementing the [Slack OAuth flow](https://api.slack.com/authentication/oauth-v2) +- `slack_sdk.models`: for constructing [Block Kit](https://api.slack.com/block-kit) UI components using easy-to-use builders +- `slack_sdk.rtm`: for utilizing the [RTM API][rtm-docs] + +If you want to use our [Events API][events-docs] and Interactivity features, please check the [Bolt for Python][bolt-python] library. Details on the Tokens and Authentication can be found in our [Auth Guide](https://slack.dev/python-slack-sdk/installation/). + +## slackclient is in maintenance mode + +Are you looking for [slackclient](https://pypi.org/project/slackclient/)? The website is live [here](https://slack.dev/python-slackclient/) just like before. However, the slackclient project is in maintenance mode now and this [`slack_sdk`](https://pypi.org/project/slack-sdk/) is the successor. If you have time to make a migration to slack_sdk v3, please follow [our migration guide](https://slack.dev/python-slack-sdk/v3-migration/) to ensure your app continues working after updating. + +## Table of contents + +* [Requirements](#requirements) +* [Installation](#installation) +* [Getting started tutorial](#getting-started-tutorial) +* [Basic Usage of the Web Client](#basic-usage-of-the-web-client) + * [Sending a message to Slack](#sending-a-message-to-slack) + * [Uploading files to Slack](#uploading-files-to-slack) +* [Async usage](#async-usage) + * [WebClient as a script](#asyncwebclient-in-a-script) + * [WebClient in a framework](#asyncwebclient-in-a-framework) +* [Advanced Options](#advanced-options) + * [SSL](#ssl) + * [Proxy](#proxy) + * [DNS performance](#dns-performance) + * [Example](#example) +* [Migrating from v1](#migrating-from-v1) +* [Support](#support) +* [Development](#development) + +### Requirements + +--- + +This library requires Python 3.6 and above. If you require Python 2, please use our [SlackClient - v1.x][slackclientv1]. If you're unsure how to check what version of Python you're on, you can check it using the following: + +> **Note:** You may need to use `python3` before your commands to ensure you use the correct Python path. e.g. `python3 --version` + +```bash +python --version + +-- or -- + +python3 --version +``` + +### Installation + +We recommend using [PyPI][pypi] to install the Slack Developer Kit for Python. + +```bash +$ pip install slack_sdk +``` + +### Getting started tutorial + +--- + +We've created this [tutorial](https://github.com/slackapi/python-slack-sdk/tree/main/tutorial) to build a basic Slack app in less than 10 minutes. It requires some general programming knowledge, and Python basics. It focuses on the interacting with Slack's Web and RTM API. Use it to give you an idea of how to use this SDK. + +**[Read the tutorial to get started!](https://github.com/slackapi/python-slack-sdk/tree/main/tutorial)** + +### Basic Usage of the Web Client + +--- + +Slack provide a Web API that gives you the ability to build applications that interact with Slack in a variety of ways. This Development Kit is a module based wrapper that makes interaction with that API easier. We have a basic example here with some of the more common uses but a full list of the available methods are available [here][api-methods]. More detailed examples can be found in [our guide](https://slack.dev/python-slack-sdk/web/). + +#### Sending a message to Slack + +One of the most common use-cases is sending a message to Slack. If you want to send a message as your app, or as a user, this method can do both. In our examples, we specify the channel name, however it is recommended to use the `channel_id` where possible. Also, if your app's bot user is not in a channel yet, invite the bot user before running the code snippet (or add `chat:write.public` to Bot Token Scopes for posting in any public channels). + +```python +import os +from slack_sdk import WebClient +from slack_sdk.errors import SlackApiError + +client = WebClient(token=os.environ['SLACK_BOT_TOKEN']) + +try: + response = client.chat_postMessage(channel='#random', text="Hello world!") + assert response["message"]["text"] == "Hello world!" +except SlackApiError as e: + # You will get a SlackApiError if "ok" is False + assert e.response["ok"] is False + assert e.response["error"] # str like 'invalid_auth', 'channel_not_found' + print(f"Got an error: {e.response['error']}") +``` + +Here we also ensure that the response back from Slack is a successful one and that the message is the one we sent by using the `assert` statement. + +#### Uploading files to Slack + +We've changed the process for uploading files to Slack to be much easier and straight forward. You can now just include a path to the file directly in the API call and upload it that way. You can find the details on this api call [here][files.upload] + +```python +import os +from slack_sdk import WebClient +from slack_sdk.errors import SlackApiError + +client = WebClient(token=os.environ['SLACK_BOT_TOKEN']) + +try: + filepath="./tmp.txt" + response = client.files_upload(channels='#random', file=filepath) + assert response["file"] # the uploaded file +except SlackApiError as e: + # You will get a SlackApiError if "ok" is False + assert e.response["ok"] is False + assert e.response["error"] # str like 'invalid_auth', 'channel_not_found' + print(f"Got an error: {e.response['error']}") +``` + +### Async usage + +`AsyncWebClient` in this SDK requires [AIOHttp][aiohttp] under the hood for asynchronous requests. + +#### AsyncWebClient in a script + +```python +import asyncio +import os +from slack_sdk.web.async_client import AsyncWebClient +from slack_sdk.errors import SlackApiError + +client = AsyncWebClient(token=os.environ['SLACK_BOT_TOKEN']) + +async def post_message(): + try: + response = await client.chat_postMessage(channel='#random', text="Hello world!") + assert response["message"]["text"] == "Hello world!" + except SlackApiError as e: + assert e.response["ok"] is False + assert e.response["error"] # str like 'invalid_auth', 'channel_not_found' + print(f"Got an error: {e.response['error']}") + +asyncio.run(post_message()) +``` + +#### AsyncWebClient in a framework + +If you are using a framework invoking the asyncio event loop like : sanic/jupyter notebook/etc. + +```python +import os +from slack_sdk.web.async_client import AsyncWebClient +from slack_sdk.errors import SlackApiError + +client = AsyncWebClient(token=os.environ['SLACK_BOT_TOKEN']) +# Define this as an async function +async def send_to_slack(channel, text): + try: + # Don't forget to have await as the client returns asyncio.Future + response = await client.chat_postMessage(channel=channel, text=text) + assert response["message"]["text"] == text + except SlackApiError as e: + assert e.response["ok"] is False + assert e.response["error"] # str like 'invalid_auth', 'channel_not_found' + raise e + +from aiohttp import web + +async def handle_requests(request: web.Request) -> web.Response: + text = 'Hello World!' + if 'text' in request.query: + text = "\t".join(request.query.getall("text")) + try: + await send_to_slack(channel="#random", text=text) + return web.json_response(data={'message': 'Done!'}) + except SlackApiError as e: + return web.json_response(data={'message': f"Failed due to {e.response['error']}"}) + + +if __name__ == "__main__": + app = web.Application() + app.add_routes([web.get("/", handle_requests)]) + # e.g., http://localhost:3000/?text=foo&text=bar + web.run_app(app, host="0.0.0.0", port=3000) +``` + +### Advanced Options + +#### SSL + +You can provide a custom SSL context or disable verification by passing the `ssl` option, supported by both the RTM and the Web client. + +For async requests, see the [AIOHttp SSL documentation](https://docs.aiohttp.org/en/stable/client_advanced.html#ssl-control-for-tcp-sockets). + +For sync requests, see the [urllib SSL documentation](https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen). + +#### Proxy + +A proxy is supported when making async requests, pass the `proxy` option, supported by both the RTM and the Web client. + +For async requests, see [AIOHttp Proxy documentation](https://docs.aiohttp.org/en/stable/client_advanced.html#proxy-support). + +For sync requests, setting either `HTTPS_PROXY` env variable or the `proxy` option works. + +#### DNS performance + +Using the async client and looking for a performance boost? Installing the optional dependencies (aiodns) may help speed up DNS resolving by the client. We've included it as an extra called "optional": +```bash +$ pip install slack_sdk[optional] +``` + +#### Example + +```python +import os +from slack_sdk import WebClient +from ssl import SSLContext + +sslcert = SSLContext() +# pip3 install proxy.py +# proxy --port 9000 --log-level d +proxyinfo = "http://localhost:9000" + +client = WebClient( + token=os.environ['SLACK_BOT_TOKEN'], + ssl=sslcert, + proxy=proxyinfo +) +response = client.chat_postMessage(channel="#random", text="Hello World!") +print(response) +``` + +### Migrating from v2 + +If you're migrating from slackclient v2.x of slack_sdk to v3.x, Please follow our migration guide to ensure your app continues working after updating. + +**[Check out the Migration Guide here!](https://slack.dev/python-slack-sdk/v3-migration/)** + +### Migrating from v1 + +If you're migrating from v1.x of slackclient to v2.x, Please follow our migration guide to ensure your app continues working after updating. + +**[Check out the Migration Guide here!](https://github.com/slackapi/python-slack-sdk/wiki/Migrating-to-2.x)** + +### Support + +--- + +If you get stuck, we’re here to help. The following are the best ways to get assistance working through your issue: + +Use our [Github Issue Tracker][gh-issues] for reporting bugs or requesting features. +Visit the [Slack Community][slack-community] for getting help using Slack Developer Kit for Python or just generally bond with your fellow Slack developers. + +### Contributing + +We welcome contributions from everyone! Please check out our +[Contributor's Guide](.github/contributing.md) for how to contribute in a +helpful and collaborative way. + + + +[pypi-image]: https://badge.fury.io/py/slack-sdk.svg +[pypi-url]: https://pypi.org/project/slack-sdk/ +[python-version]: https://img.shields.io/pypi/pyversions/slack-sdk.svg +[build-image]: https://github.com/slackapi/python-slack-sdk/workflows/CI%20Build/badge.svg +[build-url]: https://github.com/slackapi/python-slack-sdk/actions?query=workflow%3A%22CI+Build%22 +[codecov-image]: https://codecov.io/gh/slackapi/python-slack-sdk/branch/main/graph/badge.svg +[codecov-url]: https://codecov.io/gh/slackapi/python-slack-sdk +[contact-image]: https://img.shields.io/badge/contact-support-green.svg +[contact-url]: https://slack.com/support +[slackclientv1]: https://github.com/slackapi/python-slackclient/tree/v1 +[api-methods]: https://api.slack.com/methods +[rtm-docs]: https://api.slack.com/rtm +[events-docs]: https://api.slack.com/events-api +[bolt-python]: https://github.com/slackapi/bolt-python +[pypi]: https://pypi.org/ +[gh-issues]: https://github.com/slackapi/python-slack-sdk/issues +[slack-community]: https://slackcommunity.com/ +[files.upload]: https://api.slack.com/methods/files.upload +[aiohttp]: https://aiohttp.readthedocs.io/ +[urllib]: https://docs.python.org/3/library/urllib.request.html + + diff --git a/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/RECORD b/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/RECORD new file mode 100644 index 0000000..4050f71 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/RECORD @@ -0,0 +1,303 @@ +slack/__init__.py,sha256=ycePP2KcG1gdKf4Av3d91O3GUiYsY6IwsTbzGa3e9sU,578 +slack/__pycache__/__init__.cpython-38.pyc,, +slack/__pycache__/deprecation.cpython-38.pyc,, +slack/__pycache__/errors.cpython-38.pyc,, +slack/__pycache__/version.cpython-38.pyc,, +slack/deprecation.py,sha256=8eN7EPxyvHhU_lOrEeq6grZ_D1mXWH5-BySqdNVU_lQ,412 +slack/errors.py,sha256=gEfl27CA2EjLPvWGnXx10LFeOd05t60-blEfXmRWCmo,432 +slack/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 +slack/rtm/__init__.py,sha256=FICjIqtwtqXae15cy5QsLh-XGKb3gKkM6yj4mmCF8e4,209 +slack/rtm/__pycache__/__init__.cpython-38.pyc,, +slack/rtm/__pycache__/client.cpython-38.pyc,, +slack/rtm/client.py,sha256=1hX89r-IXNdH1LsdzwHxtGT1wxLXxHe72_DgYoXMp5k,212 +slack/signature/__init__.py,sha256=uT2XuitoIker5EGunWxzkkrqX-YRpU2joagBA89wne4,148 +slack/signature/__pycache__/__init__.cpython-38.pyc,, +slack/signature/__pycache__/verifier.cpython-38.pyc,, +slack/signature/verifier.py,sha256=4uHNn59hRICUaAjv8R6xfYMknMCqVRCV8pEF8QHm1ws,2359 +slack/version.py,sha256=JuQy7nSSXf6fTX_r1klZY2SC1OeyZxgQuZVHInmVhs0,50 +slack/web/__init__.py,sha256=-voth4ZdPVhuch64VKTZc8cADOOjf5P31izp8yUBeWY,605 +slack/web/__pycache__/__init__.cpython-38.pyc,, +slack/web/__pycache__/async_base_client.cpython-38.pyc,, +slack/web/__pycache__/async_client.cpython-38.pyc,, +slack/web/__pycache__/async_internal_utils.cpython-38.pyc,, +slack/web/__pycache__/async_slack_response.cpython-38.pyc,, +slack/web/__pycache__/base_client.cpython-38.pyc,, +slack/web/__pycache__/client.cpython-38.pyc,, +slack/web/__pycache__/deprecation.cpython-38.pyc,, +slack/web/__pycache__/internal_utils.cpython-38.pyc,, +slack/web/__pycache__/slack_response.cpython-38.pyc,, +slack/web/async_base_client.py,sha256=MB0Kw-mLmI0Lzafe0feXkR02_wq1aPt5uSjiL61pPGQ,5796 +slack/web/async_client.py,sha256=B2Bvzvy3ciMWrp0OLsLuaGzcTRA0cyJy7pms7pTNWNA,519 +slack/web/async_internal_utils.py,sha256=NZGgcyLnO_WW_XXj0X6Pve_amJidWl1nkpnPaAmJrUI,6042 +slack/web/async_slack_response.py,sha256=rq9Tmu4w8ZBD6n5NlCzLPf8miUqmdbbEBik-VFTTCXM,6227 +slack/web/base_client.py,sha256=n7uyCwIrR__mA-GYPT3FytwOPdlNXnwh2Cfx7Bgcox4,20247 +slack/web/classes/__init__.py,sha256=Lqwn4-Ge6HhcwePlnYnHhG8qlTJzc_FBo-abbF-fPes,397 +slack/web/classes/__pycache__/__init__.cpython-38.pyc,, +slack/web/classes/__pycache__/actions.cpython-38.pyc,, +slack/web/classes/__pycache__/attachments.cpython-38.pyc,, +slack/web/classes/__pycache__/blocks.cpython-38.pyc,, +slack/web/classes/__pycache__/dialog_elements.cpython-38.pyc,, +slack/web/classes/__pycache__/dialogs.cpython-38.pyc,, +slack/web/classes/__pycache__/elements.cpython-38.pyc,, +slack/web/classes/__pycache__/interactions.cpython-38.pyc,, +slack/web/classes/__pycache__/messages.cpython-38.pyc,, +slack/web/classes/__pycache__/objects.cpython-38.pyc,, +slack/web/classes/__pycache__/views.cpython-38.pyc,, +slack/web/classes/actions.py,sha256=d4YuoHRWHVJubhz9eVSt-2QCea-4LrfoThPmJlxLypA,716 +slack/web/classes/attachments.py,sha256=IYmq1aC_PHPNjQFRBh19pQJuW8ao_q7ynSyY9mhimjI,422 +slack/web/classes/blocks.py,sha256=SrSK2UTKK1QOxdZCmw4vZxEYSRe9FnutSDa_BiP-y08,645 +slack/web/classes/dialog_elements.py,sha256=HeGH2BTKov5lihAgOHP3RvIaLzA9xipiSdnR7z6wKns,750 +slack/web/classes/dialogs.py,sha256=MdK5w6BDsmbCL3Jo5o7RuGuTDl74PyUBe-PA3zQ0nM8,154 +slack/web/classes/elements.py,sha256=O_KxvhX5xYfo5kG3DzdxGwX8q0PXdPHTbk40xEwwMGk,1585 +slack/web/classes/interactions.py,sha256=BKLtLAYg9HpycDcTVXT6woI07m9pFw-S7bICXmpkafk,4495 +slack/web/classes/messages.py,sha256=U5R37o7rAiG5ts-udsk3LrD9CikeltQ0_7WFOVQ3HU0,62 +slack/web/classes/objects.py,sha256=40-HX2mdll7QxGMrggPpPOzSDzoGlZnW1kwN0HcQr5U,909 +slack/web/classes/views.py,sha256=lnlW6jSdsb2mpKCcWSojB_iZTguSbOOEyu-3xLizlrY,252 +slack/web/client.py,sha256=mIl9nESQ-b1ajMc38xVaCzfUroDtbdD_chj-8-1yCqQ,167 +slack/web/deprecation.py,sha256=Vfa_vp8q71fstbHVrQto8JIZ_2Vw9hrhOWe1DkQKtpE,974 +slack/web/internal_utils.py,sha256=U9NGsSeYGJTxVBl9Y78RaXI3oDguypNzOdWqUCV-hpI,1635 +slack/web/slack_response.py,sha256=InbuTpPSDM7me7S1wDi6-QRDFYtV_P83RSkLgbPPBO8,200 +slack/webhook/__init__.py,sha256=rVgUiHx_NpbaKOmbZT1gObpf3NJvkwTeMCJz1LchICI,288 +slack/webhook/__pycache__/__init__.cpython-38.pyc,, +slack/webhook/__pycache__/async_client.cpython-38.pyc,, +slack/webhook/__pycache__/client.cpython-38.pyc,, +slack/webhook/__pycache__/internal_utils.cpython-38.pyc,, +slack/webhook/__pycache__/webhook_response.cpython-38.pyc,, +slack/webhook/async_client.py,sha256=1GxwkWaq3-mU7_XyJ6SL2CT_vN1AbRN3oqdX8umJD_o,4471 +slack/webhook/client.py,sha256=NV8RhQf6A99F4Ft1_hlCf1f6v-sYqf3JBtVx1nZCB_I,4537 +slack/webhook/internal_utils.py,sha256=QJ_rrHi4zFefmWsm4VkofJQnQcRuNFqqP9-6POHC3KU,1223 +slack/webhook/webhook_response.py,sha256=MFi7RTPOmw8NaqCf5WIYeTASu0grdb4XCsMHlu3X830,281 +slack_sdk-3.17.2.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4 +slack_sdk-3.17.2.dist-info/LICENSE,sha256=CI4LW3VJqjtVNFZh1wNv4FarRyBEvIsXz_WIZCiNlSI,1091 +slack_sdk-3.17.2.dist-info/METADATA,sha256=5PYJd0vdckXMo3IyM3m6UB5_cfqwjBEMCP1qHng7onw,15380 +slack_sdk-3.17.2.dist-info/RECORD,, +slack_sdk-3.17.2.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 +slack_sdk-3.17.2.dist-info/WHEEL,sha256=z9j0xAa_JmUKMpmz72K0ZGALSM_n-wQVmGbleXx2VHg,110 +slack_sdk-3.17.2.dist-info/top_level.txt,sha256=XeKOHW4p6qjRLdY5mtrQgeg_VwSkQOqCDklTnoi7qP8,16 +slack_sdk/__init__.py,sha256=70MDjsxBDegUQgwEUlBbxbR0dSRQg3uaa6BduTuCbDs,1458 +slack_sdk/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/__pycache__/aiohttp_version_checker.cpython-38.pyc,, +slack_sdk/__pycache__/proxy_env_variable_loader.cpython-38.pyc,, +slack_sdk/__pycache__/version.cpython-38.pyc,, +slack_sdk/aiohttp_version_checker.py,sha256=IyMSH1eEhTFL7NuysnbtRXlldndNsZo0F3GPV2raRu8,928 +slack_sdk/audit_logs/__init__.py,sha256=JIz7l1YFyQ8Yg1AaF0KAkFOM9Shovjv6C4VtD0jlL0o,326 +slack_sdk/audit_logs/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/audit_logs/__pycache__/async_client.cpython-38.pyc,, +slack_sdk/audit_logs/async_client.py,sha256=4IoCDD7EjWH5kj_WW9CIr-hw0Bvkqn69SfmGi3lxOOY,93 +slack_sdk/audit_logs/v1/__init__.py,sha256=LOTT4J7Q_y2SOExJqXtATWRv_x1PrLq09sz_4oTNBDY,181 +slack_sdk/audit_logs/v1/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/audit_logs/v1/__pycache__/async_client.cpython-38.pyc,, +slack_sdk/audit_logs/v1/__pycache__/client.cpython-38.pyc,, +slack_sdk/audit_logs/v1/__pycache__/internal_utils.cpython-38.pyc,, +slack_sdk/audit_logs/v1/__pycache__/logs.cpython-38.pyc,, +slack_sdk/audit_logs/v1/__pycache__/response.cpython-38.pyc,, +slack_sdk/audit_logs/v1/async_client.py,sha256=X3LEhxDSdZdhib49rFQeC_ziFphdO7iXJt3sFwUofiY,14275 +slack_sdk/audit_logs/v1/client.py,sha256=kOo-AmgOjzlCYgGzjV1-N4q4skEPj_ndac8suUlqW6U,14570 +slack_sdk/audit_logs/v1/internal_utils.py,sha256=hTJsnp4iy28_ofZqsiJ9zsg01oE5qFeJkP9iuwXHo_U,1342 +slack_sdk/audit_logs/v1/logs.py,sha256=74hV6S4BMZ_ERZpOQxP_JDezzdFuwA7rfAURlMQqcdU,21900 +slack_sdk/audit_logs/v1/response.py,sha256=rRUCAAtLUGM2GMM6WKvGPG9jd4was_3JGBQQMYrQPFU,942 +slack_sdk/errors/__init__.py,sha256=8Yj0JW7ZGTx4RnyPjw5_SlB0A6tgTgxWIeUXPD4zTxY,1908 +slack_sdk/errors/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/http_retry/__init__.py,sha256=R4tuI1PJ5SSV-swyobBvbHd5YGB-Kk45wUcma_EvGvg,1256 +slack_sdk/http_retry/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/http_retry/__pycache__/async_handler.cpython-38.pyc,, +slack_sdk/http_retry/__pycache__/builtin_async_handlers.cpython-38.pyc,, +slack_sdk/http_retry/__pycache__/builtin_handlers.cpython-38.pyc,, +slack_sdk/http_retry/__pycache__/builtin_interval_calculators.cpython-38.pyc,, +slack_sdk/http_retry/__pycache__/handler.cpython-38.pyc,, +slack_sdk/http_retry/__pycache__/interval_calculator.cpython-38.pyc,, +slack_sdk/http_retry/__pycache__/jitter.cpython-38.pyc,, +slack_sdk/http_retry/__pycache__/request.cpython-38.pyc,, +slack_sdk/http_retry/__pycache__/response.cpython-38.pyc,, +slack_sdk/http_retry/__pycache__/state.cpython-38.pyc,, +slack_sdk/http_retry/async_handler.py,sha256=-1QzBMOyqflE4O30eEzzxREiJ3hi2NxI1y7yUWdUU6o,2657 +slack_sdk/http_retry/builtin_async_handlers.py,sha256=DvdNnZSZ5zI_TuYQ4jJW-Zseefy7PKd2Hp9d0XC9BwA,2982 +slack_sdk/http_retry/builtin_handlers.py,sha256=3Vc8eKkY_tjtofrhJzK1HuDhRDfoW6L2Aa8psj1d8SQ,2894 +slack_sdk/http_retry/builtin_interval_calculators.py,sha256=HINMCJbp33aPD9M44Jx_uW3HojyOAmHfP9S0OpWRRow,1622 +slack_sdk/http_retry/handler.py,sha256=6ieu5qWL8LTUnwlsg2AK3qx2zkZchcMgBcTCSB8gU08,2465 +slack_sdk/http_retry/interval_calculator.py,sha256=CqyC4syTimf_GGeMwWp3vDHUt-fXcbVqD66RNOlptjw,450 +slack_sdk/http_retry/jitter.py,sha256=rfhXMxF4gi0jQmnScUFFBKZb7C_8S4S6x42U7RtELuM,597 +slack_sdk/http_retry/request.py,sha256=R6_-rIboZqjb-HCXh9DG8WNzvEYoU8f_HZsKCzd8V3M,1035 +slack_sdk/http_retry/response.py,sha256=cVUdQ8fVkpZSsFkYr8kve6rCA6d9TrEC-kRYOxPVopA,639 +slack_sdk/http_retry/state.py,sha256=8IefdBkkl8OAibzEOjwImYKDfeshvNelDFdUp7JgunA,570 +slack_sdk/models/__init__.py,sha256=bpDcA85hdTl7zuEh3meqM_SsaW0XwEPVaWSwFWi0V7g,2044 +slack_sdk/models/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/models/__pycache__/basic_objects.cpython-38.pyc,, +slack_sdk/models/__pycache__/dialoags.cpython-38.pyc,, +slack_sdk/models/attachments/__init__.py,sha256=B8tysgNyjq4eKrgy8adyHtxPWLZnxPYtfHThr7peIJU,24563 +slack_sdk/models/attachments/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/models/basic_objects.py,sha256=JU9ERKbsTXa8KQrAZuu9IFn8yP_iHVTHKdY15oFCCew,3907 +slack_sdk/models/blocks/__init__.py,sha256=uN8d4bLP9uwWLBcE0MxZSTM0Q6k_eqnCQfQaFgMRrzw,3119 +slack_sdk/models/blocks/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/models/blocks/__pycache__/basic_components.cpython-38.pyc,, +slack_sdk/models/blocks/__pycache__/block_elements.cpython-38.pyc,, +slack_sdk/models/blocks/__pycache__/blocks.cpython-38.pyc,, +slack_sdk/models/blocks/basic_components.py,sha256=4T0Y8hMxqTGdNplFN5WhumuRkQGWGDxoqnf6BosLZOc,21079 +slack_sdk/models/blocks/block_elements.py,sha256=WcknQ8PcbxibBY6eCTtrYu-bga_g_6kgTxIPzvtcyHo,68918 +slack_sdk/models/blocks/blocks.py,sha256=ad4CLqcD6yDcFrUmLYiPxGGTq6z4qk8mbFB7WE_6zFk,20044 +slack_sdk/models/dialoags.py,sha256=73zMn27CsgTFlBXBDqhcC3_Ptjtw_ZkTlrUknUcYxps,1034 +slack_sdk/models/dialogs/__init__.py,sha256=kgvEpalFYopnjP_3ZfDB9U97apnZl7htf-GKLvoPvHE,32896 +slack_sdk/models/dialogs/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/models/messages/__init__.py,sha256=sF8sY58XkiIz2QITtzsrG-Id_XqQzAYeCr8C4sQyyBE,2768 +slack_sdk/models/messages/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/models/messages/__pycache__/message.cpython-38.pyc,, +slack_sdk/models/messages/message.py,sha256=upPeWEzQsDy1xzJ_0kZj9CcJwwA3WtNwGBkl0ZqwKtc,2935 +slack_sdk/models/metadata/__init__.py,sha256=4MtwV-zbimggweGsV_U8zd--4AJheNKL5vtyZVUIzsw,630 +slack_sdk/models/metadata/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/models/views/__init__.py,sha256=QkQ_CyjNLgJoF5xljE8nn_XrEZTaJIEDZFwUzh4iirw,8915 +slack_sdk/models/views/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/__init__.py,sha256=Gi-uNiQMWKW1B8V0ba2ZLSsg5hWbbxLL-hQhCw6rbVU,611 +slack_sdk/oauth/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/authorize_url_generator/__init__.py,sha256=f-ND4FU_5TUqNqiJA8lSGkNXdBzyVY3o_jPUCMSM3us,2043 +slack_sdk/oauth/authorize_url_generator/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/installation_store/__init__.py,sha256=RHcQzwrmw7OZP1bn7zcuBoODouQaC6UNr23uMnEKLaA,228 +slack_sdk/oauth/installation_store/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/installation_store/__pycache__/async_cacheable_installation_store.cpython-38.pyc,, +slack_sdk/oauth/installation_store/__pycache__/async_installation_store.cpython-38.pyc,, +slack_sdk/oauth/installation_store/__pycache__/cacheable_installation_store.cpython-38.pyc,, +slack_sdk/oauth/installation_store/__pycache__/installation_store.cpython-38.pyc,, +slack_sdk/oauth/installation_store/__pycache__/internals.cpython-38.pyc,, +slack_sdk/oauth/installation_store/amazon_s3/__init__.py,sha256=EqPl32xgQXKzU_dIOJ-suoEbOQVHeJ5FheGpssZAOuk,13764 +slack_sdk/oauth/installation_store/amazon_s3/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/installation_store/async_cacheable_installation_store.py,sha256=D9uxIVoYqb76ODsePe-t8iuXJoMuRZUrqGhcxhrWrnQ,4847 +slack_sdk/oauth/installation_store/async_installation_store.py,sha256=knSyewbv599SKVIMe5k8Ou3Fyysoy8FSKwvFaO3R14U,2973 +slack_sdk/oauth/installation_store/cacheable_installation_store.py,sha256=3fr0LX-v_2ppB7trG8SFi2YWumwVWqvRHyDQcks9rkY,4642 +slack_sdk/oauth/installation_store/file/__init__.py,sha256=yEBfOT6xr-R26RZyRu1TcHVHsK76czjYDVlLAAQjtX8,9464 +slack_sdk/oauth/installation_store/file/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/installation_store/installation_store.py,sha256=7PpahfVvoERl2y6N9AU95gmQQ-xdtP9X5YGFHXUSx-A,2891 +slack_sdk/oauth/installation_store/internals.py,sha256=8wQgaH10aTMSqIzu8njgquTfR6xbwM2RcgznSP_ctWg,943 +slack_sdk/oauth/installation_store/models/__init__.py,sha256=M0FjdbdJ_jci1dboQQ2C3EEL_4QgbATSLy6PMxxagyI,106 +slack_sdk/oauth/installation_store/models/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/installation_store/models/__pycache__/bot.cpython-38.pyc,, +slack_sdk/oauth/installation_store/models/__pycache__/installation.cpython-38.pyc,, +slack_sdk/oauth/installation_store/models/bot.py,sha256=g7AwpAIrdYhnVxqVdgXr-lKLlfX7cQku1jCJVKmIxoA,5005 +slack_sdk/oauth/installation_store/models/installation.py,sha256=Pucwnf1YrbJz4Z6Mjm9AA_VIuo0Xucpbf4fByDg98r8,9610 +slack_sdk/oauth/installation_store/sqlalchemy/__init__.py,sha256=qK1bTeism3Ou_DTPxhgcx_q3qzu-66UFg3D8qY3q4Ow,14019 +slack_sdk/oauth/installation_store/sqlalchemy/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/installation_store/sqlite3/__init__.py,sha256=yqlGQiJWcvRH7PG3CubrC3GqMxKB39vjEvv0WDS9CN0,22355 +slack_sdk/oauth/installation_store/sqlite3/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/redirect_uri_page_renderer/__init__.py,sha256=6S011hKo5KeKb9yIF3q0rR3E3jzPA1lgwfT7Rf0h-xY,1927 +slack_sdk/oauth/redirect_uri_page_renderer/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/state_store/__init__.py,sha256=meCPZyohte2-yFkdKQX6rI7laa_wQrXfWRHdp2rh79o,310 +slack_sdk/oauth/state_store/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/state_store/__pycache__/async_state_store.cpython-38.pyc,, +slack_sdk/oauth/state_store/__pycache__/state_store.cpython-38.pyc,, +slack_sdk/oauth/state_store/amazon_s3/__init__.py,sha256=D02HecKzCIOz0UYsV4spGewXlVtEHjPS7_ec7d0JH0E,2248 +slack_sdk/oauth/state_store/amazon_s3/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/state_store/async_state_store.py,sha256=HeMBX-pQfGEcfUBiU9VjjP7TRUsgd_JXDgR555DBMyE,325 +slack_sdk/oauth/state_store/file/__init__.py,sha256=ZrTrp8vWI4yUmTJ7P2cOwQGkljXRk4n2DjBYDZaMXh0,2220 +slack_sdk/oauth/state_store/file/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/state_store/sqlalchemy/__init__.py,sha256=POdUW09FeRmpt4v0TFyHcp6aNTBNfPA7NCh-uTBJrTU,2608 +slack_sdk/oauth/state_store/sqlalchemy/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/state_store/sqlite3/__init__.py,sha256=uI2NvZDscaQ_L6nJ1Hz8pKzxh2biU7m7p5kAxYukFek,3367 +slack_sdk/oauth/state_store/sqlite3/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/state_store/state_store.py,sha256=4dW8MTVP1AY1wB8oTO8ftkOXCYw1HVu1ILAK-J2FtCE,296 +slack_sdk/oauth/state_utils/__init__.py,sha256=1XHxXu6thQxJTUbEjRJsewm59GzAv7bosoYfy1WdoDE,1464 +slack_sdk/oauth/state_utils/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/token_rotation/__init__.py,sha256=DKXgsrOjzpcey7aymeGyUZu_ZwoqDcpHeOYvVX6vpOc,69 +slack_sdk/oauth/token_rotation/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/oauth/token_rotation/__pycache__/async_rotator.cpython-38.pyc,, +slack_sdk/oauth/token_rotation/__pycache__/rotator.cpython-38.pyc,, +slack_sdk/oauth/token_rotation/async_rotator.py,sha256=yqRMiVdQmxsVsYL_rZOY9shuyhYm8akr-3DwwuHqzjY,5369 +slack_sdk/oauth/token_rotation/rotator.py,sha256=7HCVXm3eFebEO4dz444mPUVoIVEVEdza1Cnpg3tLHOI,5192 +slack_sdk/proxy_env_variable_loader.py,sha256=UbV7v_RUyvOsIFCRZpDsuF73wkEZpWwwlSJiT4QBr_c,821 +slack_sdk/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 +slack_sdk/rtm/__init__.py,sha256=RTQidpVVRjHMwh2utrpXRFqHPKp5dVifwccejqTzhP8,23608 +slack_sdk/rtm/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/rtm/v2/__init__.py,sha256=yXnDq1BUw_-Ont0KTYs9qofesOOfEOIjRpET01pYZpY,71 +slack_sdk/rtm/v2/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/rtm_v2/__init__.py,sha256=sDi_As6MLxV0h65Mtfl9zvrazlx3EBMb3O-OJTUK_oc,16049 +slack_sdk/rtm_v2/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/scim/__init__.py,sha256=bG4eW2flXJRVbwMMmj9RNfR7F6Zkngbga76WDzrpUFc,719 +slack_sdk/scim/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/scim/__pycache__/async_client.cpython-38.pyc,, +slack_sdk/scim/async_client.py,sha256=P3k_G_Bb_h_zMeiWB76tjhDDFWfE0CBV6ykoM48DHV4,83 +slack_sdk/scim/v1/__init__.py,sha256=oNq_Xu8MY2Os16JtNp5uvI_9TJ6k67TWRb314_YthqU,283 +slack_sdk/scim/v1/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/scim/v1/__pycache__/async_client.cpython-38.pyc,, +slack_sdk/scim/v1/__pycache__/client.cpython-38.pyc,, +slack_sdk/scim/v1/__pycache__/default_arg.cpython-38.pyc,, +slack_sdk/scim/v1/__pycache__/group.cpython-38.pyc,, +slack_sdk/scim/v1/__pycache__/internal_utils.cpython-38.pyc,, +slack_sdk/scim/v1/__pycache__/response.cpython-38.pyc,, +slack_sdk/scim/v1/__pycache__/types.cpython-38.pyc,, +slack_sdk/scim/v1/__pycache__/user.cpython-38.pyc,, +slack_sdk/scim/v1/async_client.py,sha256=s-Que6X6cD4rc1opriHCC9QpsDyid-DZr6uDWkudn2Q,14971 +slack_sdk/scim/v1/client.py,sha256=w1xYLZtp7xqVabfuba9TyKXYjt0LqxR4hGz-QuRlNGo,15812 +slack_sdk/scim/v1/default_arg.py,sha256=THlEpdO44hwb9WXlm91BIWfErrK7jRB8WKuYXcKQT2c,53 +slack_sdk/scim/v1/group.py,sha256=NG_JuhM_Yv5a4urYASoMdC2vHiJO631s8tLCCKEF5cU,2474 +slack_sdk/scim/v1/internal_utils.py,sha256=8kEEQcjGF6-xGEKFtqHPPXOBthPAuSLQ0Acagj8lb7Y,4919 +slack_sdk/scim/v1/response.py,sha256=O0L_Wo5aoVyNhVGNxuJAQSieHV_n50_x78zOwCRJaZ4,7805 +slack_sdk/scim/v1/types.py,sha256=3K9n3iekV_HNbHzdlfXMffSorcxN90XK-l92u8d9Eqs,798 +slack_sdk/scim/v1/user.py,sha256=AyDCEL4Ccn6WlTRdGMMFgr4GMDqWZHj42H7n9asHvuQ,7868 +slack_sdk/signature/__init__.py,sha256=nyvA5lQF1_1WMUIzUC9o63cnECO0A4kgonJtG3xCtfQ,2405 +slack_sdk/signature/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/socket_mode/__init__.py,sha256=__J2av98KYvLDi2uvGC7AprV6ue4wiwpeatDjXeX1ys,358 +slack_sdk/socket_mode/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/socket_mode/__pycache__/async_client.cpython-38.pyc,, +slack_sdk/socket_mode/__pycache__/async_listeners.cpython-38.pyc,, +slack_sdk/socket_mode/__pycache__/client.cpython-38.pyc,, +slack_sdk/socket_mode/__pycache__/interval_runner.cpython-38.pyc,, +slack_sdk/socket_mode/__pycache__/listeners.cpython-38.pyc,, +slack_sdk/socket_mode/__pycache__/request.cpython-38.pyc,, +slack_sdk/socket_mode/__pycache__/response.cpython-38.pyc,, +slack_sdk/socket_mode/aiohttp/__init__.py,sha256=U5r6wkr4GgSLSzutAibjC0wEQ_9ncQHVjbkVajKFEis,21141 +slack_sdk/socket_mode/aiohttp/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/socket_mode/async_client.py,sha256=n9pQqFrhKVKWdQM7fWVy9PPv3_5nuJ6y_qHl6gDHYJA,6838 +slack_sdk/socket_mode/async_listeners.py,sha256=AzczZ_vN8hbW7ywegqJjrlKwfVPoHCboNyVp91oclpc,612 +slack_sdk/socket_mode/builtin/__init__.py,sha256=EiGtPx6ouhhOhLZ9PjA1XvK_u9KvTjVsupBJB4ZSM2k,76 +slack_sdk/socket_mode/builtin/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/socket_mode/builtin/__pycache__/client.cpython-38.pyc,, +slack_sdk/socket_mode/builtin/__pycache__/connection.cpython-38.pyc,, +slack_sdk/socket_mode/builtin/__pycache__/frame_header.cpython-38.pyc,, +slack_sdk/socket_mode/builtin/__pycache__/internals.cpython-38.pyc,, +slack_sdk/socket_mode/builtin/client.py,sha256=grAPYfG24WCXnkdkEvoQYwIHD-IcxID2FcZqTiwIRjs,12399 +slack_sdk/socket_mode/builtin/connection.py,sha256=e0BQwCl98MX_u5y8QPLtJlGcQXfW-25_Jo93I9Q3iB4,21241 +slack_sdk/socket_mode/builtin/frame_header.py,sha256=CxYuxsrSubYckypbIruUFkZpRj8iOyZMI2p3Qjz4wdQ,1066 +slack_sdk/socket_mode/builtin/internals.py,sha256=ynXO0lyadUTE7UOeWEkQTItaNxDlVbvfHJcM-v7N-yc,14390 +slack_sdk/socket_mode/client.py,sha256=8Ej_pkXD9tW2VimZq0TTlMcnGe0pNf9hOOl5wCAIudc,5995 +slack_sdk/socket_mode/interval_runner.py,sha256=l0NPPldyx6Ac_FFginDVaea7lBOm-th4X-SLiPZyBYw,907 +slack_sdk/socket_mode/listeners.py,sha256=OgMS9NmdVbkzexKd3L_ytU8l3nofz-MDqPPCAhgZZ6A,527 +slack_sdk/socket_mode/request.py,sha256=SQfJXxf0Au_W24AqJ3dqwBY56AZY7hU02hTp4fPv9qY,1991 +slack_sdk/socket_mode/response.py,sha256=Vga-5Yy6HgH4i5v_31v-ix3vQjGu5SjmRR9L0a-D8CY,880 +slack_sdk/socket_mode/websocket_client/__init__.py,sha256=UtflHfCqbITAs3Ilhi3pRISYYBrYnt5iL7ZI4Em0e8w,10523 +slack_sdk/socket_mode/websocket_client/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/socket_mode/websockets/__init__.py,sha256=4no6ZUhNuehM69aCpwek-MzuV5BlX88pJl0Z6ZSh6bE,11144 +slack_sdk/socket_mode/websockets/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/version.py,sha256=LjZnS37uyQaPB20m45QiCTNyqsjTTARPc8P6iYyfLpI,93 +slack_sdk/web/__init__.py,sha256=enoydG9T3JTKCfyn1h-GMEdwRZpW_nfhS1bPWsUcmJI,277 +slack_sdk/web/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/web/__pycache__/async_base_client.cpython-38.pyc,, +slack_sdk/web/__pycache__/async_client.cpython-38.pyc,, +slack_sdk/web/__pycache__/async_internal_utils.cpython-38.pyc,, +slack_sdk/web/__pycache__/async_slack_response.cpython-38.pyc,, +slack_sdk/web/__pycache__/base_client.cpython-38.pyc,, +slack_sdk/web/__pycache__/client.cpython-38.pyc,, +slack_sdk/web/__pycache__/deprecation.cpython-38.pyc,, +slack_sdk/web/__pycache__/internal_utils.cpython-38.pyc,, +slack_sdk/web/__pycache__/legacy_base_client.cpython-38.pyc,, +slack_sdk/web/__pycache__/legacy_client.cpython-38.pyc,, +slack_sdk/web/__pycache__/legacy_slack_response.cpython-38.pyc,, +slack_sdk/web/__pycache__/slack_response.cpython-38.pyc,, +slack_sdk/web/async_base_client.py,sha256=8zuuQDar27fXQCZuuD44ELeH1qOd3Zd79fKlqJtAICQ,8548 +slack_sdk/web/async_client.py,sha256=wREBxK8NHMoepH-QtdJhmdBRO-EoRAcmw4niFb9Adac,150473 +slack_sdk/web/async_internal_utils.py,sha256=QtRKE_O1ksGsvoj9oyFLSWcBgVygicDsU5yYewXAOTE,8479 +slack_sdk/web/async_slack_response.py,sha256=UOgpW04um9v-1GygPuO2UiKHtCgOFe4i_7EZ1HAmtmA,6625 +slack_sdk/web/base_client.py,sha256=-3VMrBkNh_TshXGIwyHew-KYz_S-LSQkisc_zAIitlQ,25161 +slack_sdk/web/client.py,sha256=j9b3fs09uB7z86N2j3Vi6lktJltnZbfjT9Mzk2oHe1A,145944 +slack_sdk/web/deprecation.py,sha256=Vfa_vp8q71fstbHVrQto8JIZ_2Vw9hrhOWe1DkQKtpE,974 +slack_sdk/web/internal_utils.py,sha256=KD-dafHPYPc4575O0N2OAKmCJM0UqFV1yWDoXqwRODY,10892 +slack_sdk/web/legacy_base_client.py,sha256=y9xnhBuWHJ5P-G85w9rTPc7YB37NOCbM1oCVKtKBU_A,24077 +slack_sdk/web/legacy_client.py,sha256=gdS_40DMwyHvu3lpdlUNrOhcZXYwEyRqOTjifhU-Zz0,150004 +slack_sdk/web/legacy_slack_response.py,sha256=Ua9WOZv6P_luKrd7DenZkelpxHh6v95Wu93KyRVEK9w,7705 +slack_sdk/web/slack_response.py,sha256=1m5NkoKCX-sUIlpBv-862SrstdUQQSttUhxbhRIWEMQ,6488 +slack_sdk/webhook/__init__.py,sha256=ETx2P__LZfGQCbX-HVIISl5RRPCPdQQPSyH3GZBnSl4,313 +slack_sdk/webhook/__pycache__/__init__.cpython-38.pyc,, +slack_sdk/webhook/__pycache__/async_client.cpython-38.pyc,, +slack_sdk/webhook/__pycache__/client.cpython-38.pyc,, +slack_sdk/webhook/__pycache__/internal_utils.cpython-38.pyc,, +slack_sdk/webhook/__pycache__/webhook_response.cpython-38.pyc,, +slack_sdk/webhook/async_client.py,sha256=TtgR_duRIb23UDdbgbOHSy49ovfPL1e4lPMMsxM3w3M,11274 +slack_sdk/webhook/client.py,sha256=QiG6x4qPPu3sGTOzKTBkz_8PxAKcHS_zw6sNa3lj5bY,12001 +slack_sdk/webhook/internal_utils.py,sha256=8k9CNXRvPBuNXfwT5LrWKR3TB90bMSWNesESGVFoecI,1351 +slack_sdk/webhook/webhook_response.py,sha256=5u9-3fzsHAaxMICdBD_RCfdz6Stegmg4U8QEGYJagYU,322 diff --git a/daita-app/core-service/functions/handlers/delete_images/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/REQUESTED similarity index 100% rename from daita-app/core-service/functions/handlers/delete_images/__init__.py rename to core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/REQUESTED diff --git a/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/WHEEL b/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/WHEEL new file mode 100644 index 0000000..0b18a28 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/WHEEL @@ -0,0 +1,6 @@ +Wheel-Version: 1.0 +Generator: bdist_wheel (0.37.1) +Root-Is-Purelib: true +Tag: py2-none-any +Tag: py3-none-any + diff --git a/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/top_level.txt b/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/top_level.txt new file mode 100644 index 0000000..a3ba6d6 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk-3.17.2.dist-info/top_level.txt @@ -0,0 +1,2 @@ +slack +slack_sdk diff --git a/core_service/aws_lambda/project/packages/slack_sdk/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/__init__.py new file mode 100644 index 0000000..cb92879 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/__init__.py @@ -0,0 +1,53 @@ +""" +* The SDK website: https://slack.dev/python-slack-sdk/ +* PyPI package: https://pypi.org/project/slack-sdk/ + +Here is the list of key modules in this SDK: + +#### Web API Client + +* Web API client: `slack_sdk.web.client` +* asyncio-based Web API client: `slack_sdk.web.async_client` + +#### Webhook / response_url Client + +* Webhook client: `slack_sdk.webhook.client` +* asyncio-based Webhook client: `slack_sdk.webhook.async_client` + +#### Socket Mode Client + +* The built-in Socket Mode client: `slack_sdk.socket_mode.builtin.client` +* [aiohttp](https://pypi.org/project/aiohttp/) based client: `slack_sdk.socket_mode.aiohttp` +* [websocket_client](https://pypi.org/project/websocket-client/) based client: `slack_sdk.socket_mode.websocket_client` +* [websockets](https://pypi.org/project/websockets/) based client: `slack_sdk.socket_mode.websockets` + +#### OAuth + +* `slack_sdk.oauth.installation_store.installation_store` +* `slack_sdk.oauth.state_store` + +#### Audit Logs API Client + +* `slack_sdk.audit_logs.v1.client` +* `slack_sdk.audit_logs.v1.async_client` + +#### SCIM API Client + +* `slack_sdk.scim.v1.client` +* `slack_sdk.scim.v1.async_client` + +""" +import logging +from logging import NullHandler + +# from .rtm import RTMClient +from .web import WebClient +from .webhook import WebhookClient + +__all__ = [ + "WebClient", + "WebhookClient", +] + +# Set default logging handler to avoid "No handler found" warnings. +logging.getLogger(__name__).addHandler(NullHandler()) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/aiohttp_version_checker.py b/core_service/aws_lambda/project/packages/slack_sdk/aiohttp_version_checker.py new file mode 100644 index 0000000..9f4d70f --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/aiohttp_version_checker.py @@ -0,0 +1,23 @@ +"""Internal module for checking aiohttp compatibility of async modules""" +import logging +from typing import Callable + + +def _print_warning_log(message: str) -> None: + logging.getLogger(__name__).warning(message) + + +def validate_aiohttp_version( + aiohttp_version: str, + print_warning: Callable[[str], None] = _print_warning_log, +): + if aiohttp_version is not None: + elements = aiohttp_version.split(".") + if len(elements) >= 3: + # patch version can be a non-numeric value + major, minor, patch = int(elements[0]), int(elements[1]), elements[2] + if major <= 2 or (major == 3 and (minor == 6 or (minor == 7 and patch == "0"))): + print_warning( + "We highly recommend upgrading aiohttp to 3.7.3 or higher versions." + "An older version of the library may not work with the Slack server-side in the future." + ) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/__init__.py new file mode 100644 index 0000000..0460ad2 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/__init__.py @@ -0,0 +1,11 @@ +"""Audit Logs API is a set of APIs for monitoring what’s happening in your Enterprise Grid organization. + +Refer to https://slack.dev/python-slack-sdk/audit-logs/ for details. +""" +from .v1.client import AuditLogsClient +from .v1.response import AuditLogsResponse + +__all__ = [ + "AuditLogsClient", + "AuditLogsResponse", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/async_client.py b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/async_client.py new file mode 100644 index 0000000..3e606fb --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/async_client.py @@ -0,0 +1,5 @@ +from .v1.async_client import AsyncAuditLogsClient + +__all__ = [ + "AsyncAuditLogsClient", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/__init__.py new file mode 100644 index 0000000..24677fe --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/__init__.py @@ -0,0 +1,4 @@ +"""Audit Logs API is a set of APIs for monitoring what’s happening in your Enterprise Grid organization. + +Refer to https://slack.dev/python-slack-sdk/audit-logs/ for details. +""" diff --git a/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/async_client.py b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/async_client.py new file mode 100644 index 0000000..8d82909 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/async_client.py @@ -0,0 +1,356 @@ +"""Audit Logs API is a set of APIs for monitoring what’s happening in your Enterprise Grid organization. + +Refer to https://slack.dev/python-slack-sdk/audit-logs/ for details. +""" +import json +import logging +from ssl import SSLContext +from typing import Any, List +from typing import Dict, Optional + +import aiohttp +from aiohttp import BasicAuth, ClientSession + +from slack_sdk.errors import SlackApiError +from .internal_utils import ( + _build_request_headers, + _debug_log_response, + get_user_agent, +) +from .response import AuditLogsResponse +from slack_sdk.http_retry.async_handler import AsyncRetryHandler +from slack_sdk.http_retry.builtin_async_handlers import async_default_handlers +from slack_sdk.http_retry.request import HttpRequest as RetryHttpRequest +from slack_sdk.http_retry.response import HttpResponse as RetryHttpResponse +from slack_sdk.http_retry.state import RetryState +from ...proxy_env_variable_loader import load_http_proxy_from_env + + +class AsyncAuditLogsClient: + BASE_URL = "https://api.slack.com/audit/v1/" + + token: str + timeout: int + ssl: Optional[SSLContext] + proxy: Optional[str] + base_url: str + session: Optional[ClientSession] + trust_env_in_session: bool + auth: Optional[BasicAuth] + default_headers: Dict[str, str] + logger: logging.Logger + retry_handlers: List[AsyncRetryHandler] + + def __init__( + self, + token: str, + timeout: int = 30, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + base_url: str = BASE_URL, + session: Optional[ClientSession] = None, + trust_env_in_session: bool = False, + auth: Optional[BasicAuth] = None, + default_headers: Optional[Dict[str, str]] = None, + user_agent_prefix: Optional[str] = None, + user_agent_suffix: Optional[str] = None, + logger: Optional[logging.Logger] = None, + retry_handlers: Optional[List[AsyncRetryHandler]] = None, + ): + """API client for Audit Logs API + See https://api.slack.com/admins/audit-logs for more details + + Args: + token: An admin user's token, which starts with `xoxp-` + timeout: Request timeout (in seconds) + ssl: `ssl.SSLContext` to use for requests + proxy: Proxy URL (e.g., `localhost:9000`, `http://localhost:9000`) + base_url: The base URL for API calls + session: `aiohttp.ClientSession` instance + trust_env_in_session: True/False for `aiohttp.ClientSession` + auth: Basic auth info for `aiohttp.ClientSession` + default_headers: Request headers to add to all requests + user_agent_prefix: Prefix for User-Agent header value + user_agent_suffix: Suffix for User-Agent header value + logger: Custom logger + retry_handlers: Retry handlers + """ + self.token = token + self.timeout = timeout + self.ssl = ssl + self.proxy = proxy + self.base_url = base_url + self.session = session + self.trust_env_in_session = trust_env_in_session + self.auth = auth + self.default_headers = default_headers if default_headers else {} + self.default_headers["User-Agent"] = get_user_agent(user_agent_prefix, user_agent_suffix) + self.logger = logger if logger is not None else logging.getLogger(__name__) + self.retry_handlers = retry_handlers if retry_handlers is not None else async_default_handlers() + + if self.proxy is None or len(self.proxy.strip()) == 0: + env_variable = load_http_proxy_from_env(self.logger) + if env_variable is not None: + self.proxy = env_variable + + async def schemas( + self, + *, + query_params: Optional[Dict[str, Any]] = None, + headers: Optional[Dict[str, str]] = None, + ) -> AuditLogsResponse: + """Returns information about the kind of objects which the Audit Logs API + returns as a list of all objects and a short description. + Authentication not required. + + Args: + query_params: Set any values if you want to add query params + headers: Additional request headers + Returns: + API response + """ + return await self.api_call( + path="schemas", + query_params=query_params, + headers=headers, + ) + + async def actions( + self, + *, + query_params: Optional[Dict[str, Any]] = None, + headers: Optional[Dict[str, str]] = None, + ) -> AuditLogsResponse: + """Returns information about the kind of actions that the Audit Logs API + returns as a list of all actions and a short description of each. + Authentication not required. + + Args: + query_params: Set any values if you want to add query params + headers: Additional request headers + + Returns: + API response + """ + return await self.api_call( + path="actions", + query_params=query_params, + headers=headers, + ) + + async def logs( + self, + *, + latest: Optional[int] = None, + oldest: Optional[int] = None, + limit: Optional[int] = None, + action: Optional[str] = None, + actor: Optional[str] = None, + entity: Optional[str] = None, + cursor: Optional[str] = None, + additional_query_params: Optional[Dict[str, Any]] = None, + headers: Optional[Dict[str, str]] = None, + ) -> AuditLogsResponse: + """This is the primary endpoint for retrieving actual audit events from your organization. + It will return a list of actions that have occurred on the installed workspace or grid organization. + Authentication required. + + The following filters can be applied in order to narrow the range of actions returned. + Filters are added as query string parameters and can be combined together. + Multiple filter parameters are additive (a boolean AND) and are separated + with an ampersand (&) in the query string. Filtering is entirely optional. + + Args: + latest: Unix timestamp of the most recent audit event to include (inclusive). + oldest: Unix timestamp of the least recent audit event to include (inclusive). + Data is not available prior to March 2018. + limit: Number of results to optimistically return, maximum 9999. + action: Name of the action. + actor: User ID who initiated the action. + entity: ID of the target entity of the action (such as a channel, workspace, organization, file). + cursor: The next page cursor of pagination + additional_query_params: Add anything else if you need to use the ones this library does not support + headers: Additional request headers + + Returns: + API response + """ + query_params = { + "latest": latest, + "oldest": oldest, + "limit": limit, + "action": action, + "actor": actor, + "entity": entity, + "cursor": cursor, + } + if additional_query_params is not None: + query_params.update(additional_query_params) + query_params = {k: v for k, v in query_params.items() if v is not None} + return await self.api_call( + path="logs", + query_params=query_params, + headers=headers, + ) + + async def api_call( + self, + *, + http_verb: str = "GET", + path: str, + query_params: Optional[Dict[str, Any]] = None, + body_params: Optional[Dict[str, Any]] = None, + headers: Optional[Dict[str, str]] = None, + ) -> AuditLogsResponse: + url = f"{self.base_url}{path}" + return await self._perform_http_request( + http_verb=http_verb, + url=url, + query_params=query_params, + body_params=body_params, + headers=_build_request_headers( + token=self.token, + default_headers=self.default_headers, + additional_headers=headers, + ), + ) + + async def _perform_http_request( + self, + *, + http_verb: str, + url: str, + query_params: Optional[Dict[str, Any]], + body_params: Optional[Dict[str, Any]], + headers: Dict[str, str], + ) -> AuditLogsResponse: + if body_params is not None: + body_params = json.dumps(body_params) + headers["Content-Type"] = "application/json;charset=utf-8" + + session: Optional[ClientSession] = None + use_running_session = self.session and not self.session.closed + if use_running_session: + session = self.session + else: + session = aiohttp.ClientSession( + timeout=aiohttp.ClientTimeout(total=self.timeout), + auth=self.auth, + trust_env=self.trust_env_in_session, + ) + + last_error = None + resp: Optional[AuditLogsResponse] = None + try: + request_kwargs = { + "headers": headers, + "params": query_params, + "data": body_params, + "ssl": self.ssl, + "proxy": self.proxy, + } + retry_request = RetryHttpRequest( + method=http_verb, + url=url, + headers=headers, + body_params=body_params, + ) + + retry_state = RetryState() + counter_for_safety = 0 + while counter_for_safety < 100: + counter_for_safety += 1 + # If this is a retry, the next try started here. We can reset the flag. + retry_state.next_attempt_requested = False + retry_response: Optional[RetryHttpResponse] = None + response_body = "" + + if self.logger.level <= logging.DEBUG: + headers_for_logging = { + k: "(redacted)" if k.lower() == "authorization" else v for k, v in headers.items() + } + self.logger.debug( + f"Sending a request - " + f"url: {url}, " + f"params: {query_params}, " + f"body: {body_params}, " + f"headers: {headers_for_logging}" + ) + + try: + async with session.request(http_verb, url, **request_kwargs) as res: + try: + response_body = await res.text() + retry_response = RetryHttpResponse( + status_code=res.status, + headers=res.headers, + data=response_body.encode("utf-8") if response_body is not None else None, + ) + except aiohttp.ContentTypeError: + self.logger.debug(f"No response data returned from the following API call: {url}.") + except json.decoder.JSONDecodeError as e: + message = f"Failed to parse the response body: {str(e)}" + raise SlackApiError(message, res) + + if res.status == 429: + for handler in self.retry_handlers: + if await handler.can_retry_async( + state=retry_state, + request=retry_request, + response=retry_response, + ): + if self.logger.level <= logging.DEBUG: + self.logger.info( + f"A retry handler found: {type(handler).__name__} " + f"for {http_verb} {url} - rate_limited" + ) + await handler.prepare_for_next_attempt_async( + state=retry_state, + request=retry_request, + response=retry_response, + ) + break + + if retry_state.next_attempt_requested is False: + resp = AuditLogsResponse( + url=url, + status_code=res.status, + raw_body=response_body, + headers=res.headers, + ) + _debug_log_response(self.logger, resp) + return resp + + except Exception as e: + last_error = e + for handler in self.retry_handlers: + if await handler.can_retry_async( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ): + if self.logger.level <= logging.DEBUG: + self.logger.info( + f"A retry handler found: {type(handler).__name__} " f"for {http_verb} {url} - {e}" + ) + await handler.prepare_for_next_attempt_async( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ) + break + + if retry_state.next_attempt_requested is False: + raise last_error + + if resp is not None: + return resp + raise last_error + + finally: + if not use_running_session: + await session.close() + + return resp diff --git a/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/client.py b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/client.py new file mode 100644 index 0000000..27e7cce --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/client.py @@ -0,0 +1,362 @@ +"""Audit Logs API is a set of APIs for monitoring what’s happening in your Enterprise Grid organization. + +Refer to https://slack.dev/python-slack-sdk/audit-logs/ for details. +""" +import json +import logging +import urllib +from http.client import HTTPResponse +from ssl import SSLContext +from typing import Dict, Optional, List, Any +from urllib.error import HTTPError +from urllib.request import Request, urlopen, OpenerDirector, ProxyHandler, HTTPSHandler + +from slack_sdk.errors import SlackRequestError +from .internal_utils import ( + _build_query, + _build_request_headers, + _debug_log_response, + get_user_agent, +) +from .response import AuditLogsResponse +from slack_sdk.http_retry import default_retry_handlers +from slack_sdk.http_retry.handler import RetryHandler +from slack_sdk.http_retry.request import HttpRequest as RetryHttpRequest +from slack_sdk.http_retry.response import HttpResponse as RetryHttpResponse +from slack_sdk.http_retry.state import RetryState +from ...proxy_env_variable_loader import load_http_proxy_from_env + + +class AuditLogsClient: + BASE_URL = "https://api.slack.com/audit/v1/" + + token: str + timeout: int + ssl: Optional[SSLContext] + proxy: Optional[str] + base_url: str + default_headers: Dict[str, str] + logger: logging.Logger + retry_handlers: List[RetryHandler] + + def __init__( + self, + token: str, + timeout: int = 30, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + base_url: str = BASE_URL, + default_headers: Optional[Dict[str, str]] = None, + user_agent_prefix: Optional[str] = None, + user_agent_suffix: Optional[str] = None, + logger: Optional[logging.Logger] = None, + retry_handlers: Optional[List[RetryHandler]] = None, + ): + """API client for Audit Logs API + See https://api.slack.com/admins/audit-logs for more details + + Args: + token: An admin user's token, which starts with `xoxp-` + timeout: Request timeout (in seconds) + ssl: `ssl.SSLContext` to use for requests + proxy: Proxy URL (e.g., `localhost:9000`, `http://localhost:9000`) + base_url: The base URL for API calls + default_headers: Request headers to add to all requests + user_agent_prefix: Prefix for User-Agent header value + user_agent_suffix: Suffix for User-Agent header value + logger: Custom logger + retry_handlers: Retry handlers + """ + self.token = token + self.timeout = timeout + self.ssl = ssl + self.proxy = proxy + self.base_url = base_url + self.default_headers = default_headers if default_headers else {} + self.default_headers["User-Agent"] = get_user_agent(user_agent_prefix, user_agent_suffix) + self.logger = logger if logger is not None else logging.getLogger(__name__) + self.retry_handlers = retry_handlers if retry_handlers is not None else default_retry_handlers() + + if self.proxy is None or len(self.proxy.strip()) == 0: + env_variable = load_http_proxy_from_env(self.logger) + if env_variable is not None: + self.proxy = env_variable + + def schemas( + self, + *, + query_params: Optional[Dict[str, Any]] = None, + headers: Optional[Dict[str, str]] = None, + ) -> AuditLogsResponse: + """Returns information about the kind of objects which the Audit Logs API + returns as a list of all objects and a short description. + Authentication not required. + + Args: + query_params: Set any values if you want to add query params + headers: Additional request headers + Returns: + API response + """ + return self.api_call( + path="schemas", + query_params=query_params, + headers=headers, + ) + + def actions( + self, + *, + query_params: Optional[Dict[str, Any]] = None, + headers: Optional[Dict[str, str]] = None, + ) -> AuditLogsResponse: + """Returns information about the kind of actions that the Audit Logs API + returns as a list of all actions and a short description of each. + Authentication not required. + + Args: + query_params: Set any values if you want to add query params + headers: Additional request headers + + Returns: + API response + """ + return self.api_call( + path="actions", + query_params=query_params, + headers=headers, + ) + + def logs( + self, + *, + latest: Optional[int] = None, + oldest: Optional[int] = None, + limit: Optional[int] = None, + action: Optional[str] = None, + actor: Optional[str] = None, + entity: Optional[str] = None, + cursor: Optional[str] = None, + additional_query_params: Optional[Dict[str, Any]] = None, + headers: Optional[Dict[str, str]] = None, + ) -> AuditLogsResponse: + """This is the primary endpoint for retrieving actual audit events from your organization. + It will return a list of actions that have occurred on the installed workspace or grid organization. + Authentication required. + + The following filters can be applied in order to narrow the range of actions returned. + Filters are added as query string parameters and can be combined together. + Multiple filter parameters are additive (a boolean AND) and are separated + with an ampersand (&) in the query string. Filtering is entirely optional. + + Args: + latest: Unix timestamp of the most recent audit event to include (inclusive). + oldest: Unix timestamp of the least recent audit event to include (inclusive). + Data is not available prior to March 2018. + limit: Number of results to optimistically return, maximum 9999. + action: Name of the action. + actor: User ID who initiated the action. + entity: ID of the target entity of the action (such as a channel, workspace, organization, file). + cursor: The next page cursor of pagination + additional_query_params: Add anything else if you need to use the ones this library does not support + headers: Additional request headers + + Returns: + API response + """ + query_params = { + "latest": latest, + "oldest": oldest, + "limit": limit, + "action": action, + "actor": actor, + "entity": entity, + "cursor": cursor, + } + if additional_query_params is not None: + query_params.update(additional_query_params) + query_params = {k: v for k, v in query_params.items() if v is not None} + return self.api_call( + path="logs", + query_params=query_params, + headers=headers, + ) + + def api_call( + self, + *, + http_verb: str = "GET", + path: str, + query_params: Optional[Dict[str, Any]] = None, + body_params: Optional[Dict[str, Any]] = None, + headers: Optional[Dict[str, str]] = None, + ) -> AuditLogsResponse: + """Performs a Slack API request and returns the result.""" + url = f"{self.base_url}{path}" + query = _build_query(query_params) + if len(query) > 0: + url += f"?{query}" + + return self._perform_http_request( + http_verb=http_verb, + url=url, + body=body_params, + headers=_build_request_headers( + token=self.token, + default_headers=self.default_headers, + additional_headers=headers, + ), + ) + + def _perform_http_request( + self, + *, + http_verb: str = "GET", + url: str, + body: Optional[Dict[str, Any]] = None, + headers: Dict[str, str], + ) -> AuditLogsResponse: + if body is not None: + body = json.dumps(body) + headers["Content-Type"] = "application/json;charset=utf-8" + + if self.logger.level <= logging.DEBUG: + headers_for_logging = {k: "(redacted)" if k.lower() == "authorization" else v for k, v in headers.items()} + self.logger.debug(f"Sending a request - url: {url}, body: {body}, headers: {headers_for_logging}") + + # NOTE: Intentionally ignore the `http_verb` here + # Slack APIs accepts any API method requests with POST methods + req = Request( + method=http_verb, + url=url, + data=body.encode("utf-8") if body is not None else None, + headers=headers, + ) + resp = None + last_error = None + + retry_state = RetryState() + counter_for_safety = 0 + while counter_for_safety < 100: + counter_for_safety += 1 + # If this is a retry, the next try started here. We can reset the flag. + retry_state.next_attempt_requested = False + + try: + resp = self._perform_http_request_internal(url, req) + # The resp is a 200 OK response + return resp + + except HTTPError as e: + # read the response body here + charset = e.headers.get_content_charset() or "utf-8" + response_body: str = e.read().decode(charset) + # As adding new values to HTTPError#headers can be ignored, building a new dict object here + response_headers = dict(e.headers.items()) + resp = AuditLogsResponse( + url=url, + status_code=e.code, + raw_body=response_body, + headers=response_headers, + ) + if e.code == 429: + # for backward-compatibility with WebClient (v.2.5.0 or older) + if "retry-after" not in resp.headers and "Retry-After" in resp.headers: + resp.headers["retry-after"] = resp.headers["Retry-After"] + if "Retry-After" not in resp.headers and "retry-after" in resp.headers: + resp.headers["Retry-After"] = resp.headers["retry-after"] + _debug_log_response(self.logger, resp) + + # Try to find a retry handler for this error + retry_request = RetryHttpRequest.from_urllib_http_request(req) + retry_response = RetryHttpResponse( + status_code=e.code, + headers={k: [v] for k, v in e.headers.items()}, + data=response_body.encode("utf-8") if response_body is not None else None, + ) + for handler in self.retry_handlers: + if handler.can_retry( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ): + if self.logger.level <= logging.DEBUG: + self.logger.info( + f"A retry handler found: {type(handler).__name__} for {req.method} {req.full_url} - {e}" + ) + handler.prepare_for_next_attempt( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ) + break + + if retry_state.next_attempt_requested is False: + return resp + + except Exception as err: + last_error = err + self.logger.error(f"Failed to send a request to Slack API server: {err}") + + # Try to find a retry handler for this error + retry_request = RetryHttpRequest.from_urllib_http_request(req) + for handler in self.retry_handlers: + if handler.can_retry( + state=retry_state, + request=retry_request, + response=None, + error=err, + ): + if self.logger.level <= logging.DEBUG: + self.logger.info( + f"A retry handler found: {type(handler).__name__} for {req.method} {req.full_url} - {err}" + ) + handler.prepare_for_next_attempt( + state=retry_state, + request=retry_request, + response=None, + error=err, + ) + self.logger.info(f"Going to retry the same request: {req.method} {req.full_url}") + break + + if retry_state.next_attempt_requested is False: + raise err + + if resp is not None: + return resp + raise last_error + + def _perform_http_request_internal(self, url: str, req: Request) -> AuditLogsResponse: + opener: Optional[OpenerDirector] = None + # for security (BAN-B310) + if url.lower().startswith("http"): + if self.proxy is not None: + if isinstance(self.proxy, str): + opener = urllib.request.build_opener( + ProxyHandler({"http": self.proxy, "https": self.proxy}), + HTTPSHandler(context=self.ssl), + ) + else: + raise SlackRequestError(f"Invalid proxy detected: {self.proxy} must be a str value") + else: + raise SlackRequestError(f"Invalid URL detected: {url}") + + # NOTE: BAN-B310 is already checked above + http_resp: Optional[HTTPResponse] = None + if opener: + http_resp = opener.open(req, timeout=self.timeout) # skipcq: BAN-B310 + else: + http_resp = urlopen(req, context=self.ssl, timeout=self.timeout) # skipcq: BAN-B310 + charset: str = http_resp.headers.get_content_charset() or "utf-8" + response_body: str = http_resp.read().decode(charset) + resp = AuditLogsResponse( + url=url, + status_code=http_resp.status, + raw_body=response_body, + headers=http_resp.headers, + ) + _debug_log_response(self.logger, resp) + return resp diff --git a/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/internal_utils.py b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/internal_utils.py new file mode 100644 index 0000000..c4521c7 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/internal_utils.py @@ -0,0 +1,40 @@ +import logging +from typing import Optional, Dict, Any +from urllib.parse import quote + +from slack_sdk.web.internal_utils import get_user_agent +from .response import AuditLogsResponse + + +def _build_query(params: Optional[Dict[str, Any]]) -> str: + if params is not None and len(params) > 0: + return "&".join({f"{quote(str(k))}={quote(str(v))}" for k, v in params.items() if v is not None}) + return "" + + +def _build_request_headers( + token: str, + default_headers: Dict[str, str], + additional_headers: Optional[Dict[str, str]], +) -> Dict[str, str]: + request_headers = { + "Content-Type": "application/json;charset=utf-8", + "Authorization": f"Bearer {token}", + } + if default_headers is None or "User-Agent" not in default_headers: + request_headers["User-Agent"] = get_user_agent() + if default_headers is not None: + request_headers.update(default_headers) + if additional_headers is not None: + request_headers.update(additional_headers) + return request_headers + + +def _debug_log_response(logger, resp: AuditLogsResponse) -> None: + if logger.level <= logging.DEBUG: + logger.debug( + "Received the following response - " + f"status: {resp.status_code}, " + f"headers: {(dict(resp.headers))}, " + f"body: {resp.raw_body}" + ) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/logs.py b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/logs.py new file mode 100644 index 0000000..ee74365 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/logs.py @@ -0,0 +1,640 @@ +from typing import Optional, List, Union, Any, Dict + + +class App: + id: Optional[str] + name: Optional[str] + is_distributed: Optional[bool] + is_directory_approved: Optional[bool] + is_workflow_app: Optional[bool] + scopes: Optional[List[str]] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + id: Optional[str] = None, + name: Optional[str] = None, + is_distributed: Optional[bool] = None, + is_directory_approved: Optional[bool] = None, + is_workflow_app: Optional[bool] = None, + scopes: Optional[List[str]] = None, + **kwargs, + ) -> None: + self.id = id + self.name = name + self.is_distributed = is_distributed + self.is_directory_approved = is_directory_approved + self.is_workflow_app = is_workflow_app + self.scopes = scopes + self.unknown_fields = kwargs + + +class User: + id: Optional[str] + name: Optional[str] + email: Optional[str] + team: Optional[str] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + id: Optional[str] = None, + name: Optional[str] = None, + email: Optional[str] = None, + team: Optional[str] = None, + **kwargs, + ) -> None: + self.id = id + self.name = name + self.email = email + self.team = team + self.unknown_fields = kwargs + + +class Actor: + type: Optional[str] + user: Optional[User] + unknown_fields: Dict[str, Any] + + def __init__( + self, + type: Optional[str] = None, + user: Optional[Union[User, Dict[str, Any]]] = None, + **kwargs, + ) -> None: + self.type = type + self.user = User(**user) if isinstance(user, dict) else user + self.unknown_fields = kwargs + + +class Location: + type: Optional[str] + id: Optional[str] + name: Optional[str] + domain: Optional[str] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + type: Optional[str] = None, + id: Optional[str] = None, + name: Optional[str] = None, + domain: Optional[str] = None, + **kwargs, + ) -> None: + self.type = type + self.id = id + self.name = name + self.domain = domain + self.unknown_fields = kwargs + + +class Context: + location: Optional[Location] + ua: Optional[str] + ip_address: Optional[str] + session_id: Optional[str] + app: Optional[App] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + location: Optional[Union[Location, Dict[str, Any]]] = None, + ua: Optional[str] = None, + ip_address: Optional[str] = None, + session_id: Optional[str] = None, + app: Optional[Union[App, Dict[str, Any]]] = None, + **kwargs, + ) -> None: + self.location = Location(**location) if isinstance(location, dict) else location + self.ua = ua + self.ip_address = ip_address + self.session_id = session_id + self.app = App(**app) if isinstance(app, dict) else app + self.unknown_fields = kwargs + + +class RetentionPolicy: + type: Optional[str] + duration_days: Optional[int] + + def __init__( + self, + *, + type: Optional[str] = None, + duration_days: Optional[int] = None, + **kwargs, + ) -> None: + self.type = type + self.duration_days = duration_days + self.unknown_fields = kwargs + + +class ConversationPref: + type: Optional[List[str]] + user: Optional[List[str]] + + def __init__( + self, + *, + type: Optional[List[str]] = None, + user: Optional[List[str]] = None, + **kwargs, + ) -> None: + self.type = type + self.user = user + self.unknown_fields = kwargs + + +class FeatureEnablement: + enabled: Optional[bool] + + def __init__( + self, + *, + enabled: Optional[bool] = None, + **kwargs, + ) -> None: + self.enabled = enabled + self.unknown_fields = kwargs + + +class Details: + name: Optional[str] + new_value: Optional[Union[str, List[str], Dict[str, Any]]] + previous_value: Optional[Union[str, List[str], Dict[str, Any]]] + expires_on: Optional[int] + mobile_only: Optional[bool] + web_only: Optional[bool] + non_sso_only: Optional[bool] + type: Optional[str] + is_workflow: Optional[bool] + inviter: Optional[User] + kicker: Optional[User] + shared_to: Optional[str] + reason: Optional[str] + origin_team: Optional[str] + target_team: Optional[str] + is_internal_integration: Optional[bool] + cleared_resolution: Optional[str] + app_owner_id: Optional[str] + bot_scopes: Optional[List[str]] + new_scopes: Optional[List[str]] + previous_scopes: Optional[List[str]] + granular_bot_token: Optional[bool] + scopes: Optional[List[str]] + scopes_bot: Optional[List[str]] + resolution: Optional[str] + app_previously_resolved: Optional[bool] + admin_app_id: Optional[str] + bot_id: Optional[str] + installer_user_id: Optional[str] + approver_id: Optional[str] + approval_type: Optional[str] + app_previously_approved: Optional[bool] + old_scopes: Optional[List[str]] + channels: Optional[List[str]] + permissions: Optional[List[Dict[str, Any]]] + new_version_id: Optional[str] + trigger: Optional[str] + export_type: Optional[str] + export_start_ts: Optional[str] + export_end_ts: Optional[str] + barrier_id: Optional[str] + primary_usergroup_id: Optional[str] + barriered_from_usergroup_ids: Optional[List[str]] + restricted_subjects: Optional[List[str]] + duration: Optional[int] + desktop_app_browser_quit: Optional[bool] + invite_id: Optional[str] + external_organization_id: Optional[str] + external_organization_name: Optional[str] + external_user_id: Optional[str] + external_user_email: Optional[str] + channel_id: Optional[str] + added_team_id: Optional[str] + unknown_fields: Dict[str, Any] + is_token_rotation_enabled_app: Optional[bool] + old_retention_policy: Optional[RetentionPolicy] + new_retention_policy: Optional[RetentionPolicy] + who_can_post: Optional[ConversationPref] + can_thread: Optional[ConversationPref] + is_external_limited: Optional[bool] + exporting_team_id: Optional[int] + session_search_start: Optional[int] + deprecation_search_end: Optional[int] + is_error: Optional[bool] + creator: Optional[str] + team: Optional[str] + app_id: Optional[str] + enable_at_here: Optional[FeatureEnablement] + enable_at_channel: Optional[FeatureEnablement] + can_huddle: Optional[FeatureEnablement] + + def __init__( + self, + *, + name: Optional[str] = None, + new_value: Optional[Union[str, List[str], Dict[str, Any]]] = None, + previous_value: Optional[Union[str, List[str], Dict[str, Any]]] = None, + expires_on: Optional[int] = None, + mobile_only: Optional[bool] = None, + web_only: Optional[bool] = None, + non_sso_only: Optional[bool] = None, + type: Optional[str] = None, + is_workflow: Optional[bool] = None, + inviter: Optional[Union[Dict[str, Any], User]] = None, + kicker: Optional[Union[Dict[str, Any], User]] = None, + shared_to: Optional[str] = None, + reason: Optional[str] = None, + origin_team: Optional[str] = None, + target_team: Optional[str] = None, + is_internal_integration: Optional[bool] = None, + cleared_resolution: Optional[str] = None, + app_owner_id: Optional[str] = None, + bot_scopes: Optional[List[str]] = None, + new_scopes: Optional[List[str]] = None, + previous_scopes: Optional[List[str]] = None, + granular_bot_token: Optional[bool] = None, + scopes: Optional[List[str]] = None, + scopes_bot: Optional[List[str]] = None, + resolution: Optional[str] = None, + app_previously_resolved: Optional[bool] = None, + admin_app_id: Optional[str] = None, + bot_id: Optional[str] = None, + installer_user_id: Optional[str] = None, + approver_id: Optional[str] = None, + approval_type: Optional[str] = None, + app_previously_approved: Optional[bool] = None, + old_scopes: Optional[List[str]] = None, + channels: Optional[List[str]] = None, + permissions: Optional[List[Dict[str, Any]]] = None, + new_version_id: Optional[str] = None, + trigger: Optional[str] = None, + export_type: Optional[str] = None, + export_start_ts: Optional[str] = None, + export_end_ts: Optional[str] = None, + barrier_id: Optional[str] = None, + primary_usergroup_id: Optional[str] = None, + barriered_from_usergroup_ids: Optional[List[str]] = None, + restricted_subjects: Optional[List[str]] = None, + duration: Optional[int] = None, + desktop_app_browser_quit: Optional[bool] = None, + invite_id: Optional[str] = None, + external_organization_id: Optional[str] = None, + external_organization_name: Optional[str] = None, + external_user_id: Optional[str] = None, + external_user_email: Optional[str] = None, + channel_id: Optional[str] = None, + added_team_id: Optional[str] = None, + is_token_rotation_enabled_app: Optional[bool] = None, + old_retention_policy: Optional[Union[Dict[str, Any], RetentionPolicy]] = None, + new_retention_policy: Optional[Union[Dict[str, Any], RetentionPolicy]] = None, + who_can_post: Optional[Union[Dict[str, List[str]], ConversationPref]] = None, + can_thread: Optional[Union[Dict[str, List[str]], ConversationPref]] = None, + is_external_limited: Optional[bool] = None, + exporting_team_id: Optional[int] = None, + session_search_start: Optional[int] = None, + deprecation_search_end: Optional[int] = None, + is_error: Optional[bool] = None, + creator: Optional[str] = None, + team: Optional[str] = None, + app_id: Optional[str] = None, + enable_at_here: Optional[Union[Dict[str, Any], FeatureEnablement]] = None, + enable_at_channel: Optional[Union[Dict[str, Any], FeatureEnablement]] = None, + can_huddle: Optional[Union[Dict[str, Any], FeatureEnablement]] = None, + **kwargs, + ) -> None: + self.name = name + self.new_value = new_value + self.previous_value = previous_value + self.expires_on = expires_on + self.mobile_only = mobile_only + self.web_only = web_only + self.non_sso_only = non_sso_only + self.type = type + self.is_workflow = is_workflow + self.inviter = inviter if inviter is None or isinstance(inviter, User) else User(**inviter) + self.kicker = kicker if kicker is None or isinstance(kicker, User) else User(**kicker) + self.shared_to = shared_to + self.reason = reason + self.origin_team = origin_team + self.target_team = target_team + self.is_internal_integration = is_internal_integration + self.cleared_resolution = cleared_resolution + self.app_owner_id = app_owner_id + self.bot_scopes = bot_scopes + self.new_scopes = new_scopes + self.previous_scopes = previous_scopes + self.granular_bot_token = granular_bot_token + self.scopes = scopes + self.scopes_bot = scopes_bot + self.resolution = resolution + self.app_previously_resolved = app_previously_resolved + self.admin_app_id = admin_app_id + self.bot_id = bot_id + self.unknown_fields = kwargs + self.installer_user_id = installer_user_id + self.approver_id = approver_id + self.approval_type = approval_type + self.app_previously_approved = app_previously_approved + self.old_scopes = old_scopes + self.channels = channels + self.permissions = permissions + self.new_version_id = new_version_id + self.trigger = trigger + self.export_type = export_type + self.export_start_ts = export_start_ts + self.export_end_ts = export_end_ts + self.barrier_id = barrier_id + self.primary_usergroup_id = primary_usergroup_id + self.barriered_from_usergroup_ids = barriered_from_usergroup_ids + self.restricted_subjects = restricted_subjects + self.duration = duration + self.desktop_app_browser_quit = desktop_app_browser_quit + self.invite_id = invite_id + self.external_organization_id = external_organization_id + self.external_organization_name = external_organization_name + self.external_user_id = external_user_id + self.external_user_email = external_user_email + self.channel_id = channel_id + self.added_team_id = added_team_id + self.is_token_rotation_enabled_app = is_token_rotation_enabled_app + self.old_retention_policy = ( + old_retention_policy + if old_retention_policy is None or isinstance(old_retention_policy, RetentionPolicy) + else RetentionPolicy(**old_retention_policy) + ) + self.new_retention_policy = ( + new_retention_policy + if new_retention_policy is None or isinstance(new_retention_policy, RetentionPolicy) + else RetentionPolicy(**new_retention_policy) + ) + self.who_can_post = ( + who_can_post + if who_can_post is None or isinstance(who_can_post, ConversationPref) + else ConversationPref(**who_can_post) + ) + self.can_thread = ( + can_thread if can_thread is None or isinstance(can_thread, ConversationPref) else ConversationPref(**can_thread) + ) + self.is_external_limited = is_external_limited + self.exporting_team_id = exporting_team_id + self.session_search_start = session_search_start + self.deprecation_search_end = deprecation_search_end + self.is_error = is_error + self.creator = creator + self.team = team + self.app_id = app_id + self.enable_at_here = ( + enable_at_here + if enable_at_here is None or isinstance(enable_at_here, FeatureEnablement) + else FeatureEnablement(**enable_at_here) + ) + self.enable_at_channel = ( + enable_at_channel + if enable_at_channel is None or isinstance(enable_at_channel, FeatureEnablement) + else FeatureEnablement(**enable_at_channel) + ) + self.can_huddle = ( + can_huddle + if can_huddle is None or isinstance(can_huddle, FeatureEnablement) + else FeatureEnablement(**can_huddle) + ) + + +class Channel: + id: Optional[str] + privacy: Optional[str] + name: Optional[str] + is_shared: Optional[bool] + is_org_shared: Optional[bool] + teams_shared_with: Optional[List[str]] + original_connected_channel_id: Optional[str] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + id: Optional[str] = None, + privacy: Optional[str] = None, + name: Optional[str] = None, + is_shared: Optional[bool] = None, + is_org_shared: Optional[bool] = None, + teams_shared_with: Optional[List[str]] = None, + original_connected_channel_id: Optional[str] = None, + **kwargs, + ) -> None: + self.id = id + self.privacy = privacy + self.name = name + self.is_shared = is_shared + self.is_org_shared = is_org_shared + self.teams_shared_with = teams_shared_with + self.original_connected_channel_id = original_connected_channel_id + self.unknown_fields = kwargs + + +class File: + id: Optional[str] + name: Optional[str] + filetype: Optional[str] + title: Optional[str] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + id: Optional[str] = None, + name: Optional[str] = None, + filetype: Optional[str] = None, + title: Optional[str] = None, + **kwargs, + ) -> None: + self.id = id + self.name = name + self.filetype = filetype + self.title = title + self.unknown_fields = kwargs + + +class Usergroup: + id: Optional[str] + name: Optional[str] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + id: Optional[str] = None, + name: Optional[str] = None, + **kwargs, + ) -> None: + self.id = id + self.name = name + self.unknown_fields = kwargs + + +class Workflow: + id: Optional[str] + name: Optional[str] + domain: Optional[str] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + id: Optional[str] = None, + name: Optional[str] = None, + domain: Optional[str] = None, + **kwargs, + ) -> None: + self.id = id + self.name = name + self.domain = domain + self.unknown_fields = kwargs + + +class InformationBarrier: + id: Optional[str] + primary_usergroup: Optional[str] + barriered_from_usergroups: Optional[List[str]] + restricted_subjects: Optional[List[str]] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + id: Optional[str] = None, + primary_usergroup: Optional[str] = None, + barriered_from_usergroups: Optional[List[str]] = None, + restricted_subjects: Optional[List[str]] = None, + **kwargs, + ) -> None: + self.id = id + self.primary_usergroup = primary_usergroup + self.barriered_from_usergroups = barriered_from_usergroups + self.restricted_subjects = restricted_subjects + self.unknown_fields = kwargs + + +class Entity: + type: Optional[str] + user: Optional[User] + workspace: Optional[Location] + enterprise: Optional[Location] + channel: Optional[Channel] + file: Optional[File] + app: Optional[App] + usergroup: Optional[Usergroup] + workflow: Optional[Workflow] + barrier: Optional[InformationBarrier] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + type: Optional[str] = None, + user: Optional[Union[User, Dict[str, Any]]] = None, + workspace: Optional[Union[Location, Dict[str, Any]]] = None, + enterprise: Optional[Union[Location, Dict[str, Any]]] = None, + channel: Optional[Union[Channel, Dict[str, Any]]] = None, + file: Optional[Union[File, Dict[str, Any]]] = None, + app: Optional[Union[App, Dict[str, Any]]] = None, + usergroup: Optional[Union[Usergroup, Dict[str, Any]]] = None, + workflow: Optional[Union[Workflow, Dict[str, Any]]] = None, + barrier: Optional[Union[InformationBarrier, Dict[str, Any]]] = None, + **kwargs, + ) -> None: + self.type = type + self.user = User(**user) if isinstance(user, dict) else user + self.workspace = Location(**workspace) if isinstance(workspace, dict) else workspace + self.enterprise = Location(**enterprise) if isinstance(enterprise, dict) else enterprise + self.channel = Channel(**channel) if isinstance(channel, dict) else channel + self.file = File(**file) if isinstance(file, dict) else file + self.app = App(**app) if isinstance(app, dict) else app + self.usergroup = Usergroup(**usergroup) if isinstance(usergroup, dict) else usergroup + self.workflow = Workflow(**workflow) if isinstance(workflow, dict) else workflow + self.barrier = InformationBarrier(**barrier) if isinstance(barrier, dict) else barrier + self.unknown_fields = kwargs + + +class Entry: + id: Optional[str] + date_create: Optional[int] + action: Optional[str] + actor: Optional[Actor] + entity: Optional[Entity] + context: Optional[Context] + details: Optional[Details] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + id: Optional[str] = None, + date_create: Optional[int] = None, + action: Optional[str] = None, + actor: Optional[Union[Actor, Dict[str, Any]]] = None, + entity: Optional[Union[Entity, Dict[str, Any]]] = None, + context: Optional[Union[Context, Dict[str, Any]]] = None, + details: Optional[Union[Details, Dict[str, Any]]] = None, + **kwargs, + ) -> None: + self.id = id + self.date_create = date_create + self.action = action + self.actor = Actor(**actor) if isinstance(actor, dict) else actor + self.entity = Entity(**entity) if isinstance(entity, dict) else entity + self.context = Context(**context) if isinstance(context, dict) else context + self.details = Details(**details) if isinstance(details, dict) else details + self.unknown_fields = kwargs + + +class ResponseMetadata: + next_cursor: Optional[str] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + next_cursor: Optional[str] = None, + **kwargs, + ) -> None: + self.next_cursor = next_cursor + self.unknown_fields = kwargs + + +class LogsResponse: + entries: Optional[List[Entry]] + response_metadata: Optional[ResponseMetadata] + ok: Optional[bool] + error: Optional[str] + needed: Optional[str] + provided: Optional[str] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + entries: Optional[List[Union[Entry, Dict[str, Any]]]] = None, + response_metadata: Optional[Union[ResponseMetadata, Dict[str, Any]]] = None, + ok: Optional[bool] = None, + error: Optional[str] = None, + needed: Optional[str] = None, + provided: Optional[str] = None, + **kwargs, + ) -> None: + self.entries = [Entry(**e) if isinstance(e, dict) else e for e in entries] + self.response_metadata = ( + ResponseMetadata(**response_metadata) if isinstance(response_metadata, dict) else response_metadata + ) + self.ok = ok + self.error = error + self.needed = needed + self.provided = provided + self.unknown_fields = kwargs diff --git a/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/response.py b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/response.py new file mode 100644 index 0000000..9fe66d0 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/audit_logs/v1/response.py @@ -0,0 +1,34 @@ +import json +from typing import Dict, Any, Optional + +from slack_sdk.audit_logs.v1.logs import LogsResponse + + +# TODO: Unlike WebClient's responses, this class has not yet provided __iter__ method +class AuditLogsResponse: + url: str + status_code: int + headers: Dict[str, Any] + raw_body: Optional[str] + body: Optional[Dict[str, Any]] + typed_body: Optional[LogsResponse] + + @property + def typed_body(self) -> Optional[LogsResponse]: # type: ignore + if self.body is None: + return None + return LogsResponse(**self.body) + + def __init__( + self, + *, + url: str, + status_code: int, + raw_body: Optional[str], + headers: dict, + ): + self.url = url + self.status_code = status_code + self.headers = headers + self.raw_body = raw_body + self.body = json.loads(raw_body) if raw_body is not None and raw_body.startswith("{") else None diff --git a/core_service/aws_lambda/project/packages/slack_sdk/errors/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/errors/__init__.py new file mode 100644 index 0000000..51b9a04 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/errors/__init__.py @@ -0,0 +1,58 @@ +"""Errors that can be raised by this SDK""" + + +class SlackClientError(Exception): + """Base class for Client errors""" + + +class BotUserAccessError(SlackClientError): + """Error raised when an 'xoxb-*' token is + being used for a Slack API method that only accepts 'xoxp-*' tokens. + """ + + +class SlackRequestError(SlackClientError): + """Error raised when there's a problem with the request that's being submitted.""" + + +class SlackApiError(SlackClientError): + """Error raised when Slack does not send the expected response. + + Attributes: + response (SlackResponse): The SlackResponse object containing all of the data sent back from the API. + + Note: + The message (str) passed into the exception is used when + a user converts the exception to a str. + i.e. str(SlackApiError("This text will be sent as a string.")) + """ + + def __init__(self, message, response): + msg = f"{message}\nThe server responded with: {response}" + self.response = response + super(SlackApiError, self).__init__(msg) + + +class SlackTokenRotationError(SlackClientError): + """Error raised when the oauth.v2.access call for token rotation fails""" + + api_error: SlackApiError + + def __init__(self, api_error: SlackApiError): + self.api_error = api_error + + +class SlackClientNotConnectedError(SlackClientError): + """Error raised when attempting to send messages over the websocket when the + connection is closed.""" + + +class SlackObjectFormationError(SlackClientError): + """Error raised when a constructed object is not valid/malformed""" + + +class SlackClientConfigurationError(SlackClientError): + """Error raised because of invalid configuration on the client side: + * when attempting to send messages over the websocket when the connection is closed. + * when external system (e.g., Amazon S3) configuration / credentials are not correct + """ diff --git a/core_service/aws_lambda/project/packages/slack_sdk/http_retry/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/__init__.py new file mode 100644 index 0000000..b7116da --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/__init__.py @@ -0,0 +1,48 @@ +from typing import List + +from .handler import RetryHandler +from .builtin_handlers import ( + ConnectionErrorRetryHandler, + RateLimitErrorRetryHandler, +) +from .interval_calculator import RetryIntervalCalculator +from .builtin_interval_calculators import ( + FixedValueRetryIntervalCalculator, + BackoffRetryIntervalCalculator, +) +from .jitter import Jitter +from .request import HttpRequest +from .response import HttpResponse +from .state import RetryState + +connect_error_retry_handler = ConnectionErrorRetryHandler() +rate_limit_error_retry_handler = RateLimitErrorRetryHandler() + + +def default_retry_handlers() -> List[RetryHandler]: + return [connect_error_retry_handler] + + +def all_builtin_retry_handlers() -> List[RetryHandler]: + return [ + connect_error_retry_handler, + rate_limit_error_retry_handler, + ] + + +__all__ = [ + "RetryHandler", + "ConnectionErrorRetryHandler", + "RateLimitErrorRetryHandler", + "RetryIntervalCalculator", + "FixedValueRetryIntervalCalculator", + "BackoffRetryIntervalCalculator", + "Jitter", + "HttpRequest", + "HttpResponse", + "RetryState", + "connect_error_retry_handler", + "rate_limit_error_retry_handler", + "default_retry_handlers", + "all_builtin_retry_handlers", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/http_retry/async_handler.py b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/async_handler.py new file mode 100644 index 0000000..ed9f611 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/async_handler.py @@ -0,0 +1,89 @@ +"""asyncio compatible RetryHandler interface. +You can pass an array of handlers to customize retry logics in supported API clients. +""" + +import asyncio +from typing import Optional + +from slack_sdk.http_retry.state import RetryState +from slack_sdk.http_retry.request import HttpRequest +from slack_sdk.http_retry.response import HttpResponse +from slack_sdk.http_retry.interval_calculator import RetryIntervalCalculator +from slack_sdk.http_retry.builtin_interval_calculators import ( + BackoffRetryIntervalCalculator, +) + +default_interval_calculator = BackoffRetryIntervalCalculator() + + +class AsyncRetryHandler: + """asyncio compatible RetryHandler interface. + You can pass an array of handlers to customize retry logics in supported API clients. + """ + + max_retry_count: int + interval_calculator: RetryIntervalCalculator + + def __init__( + self, + max_retry_count: int = 1, + interval_calculator: RetryIntervalCalculator = default_interval_calculator, + ): + """RetryHandler interface. + + Args: + max_retry_count: The maximum times to do retries + interval_calculator: Pass an interval calculator for customizing the logic + """ + self.max_retry_count = max_retry_count + self.interval_calculator = interval_calculator + + async def can_retry_async( + self, + *, + state: RetryState, + request: HttpRequest, + response: Optional[HttpResponse] = None, + error: Optional[Exception] = None, + ) -> bool: + if state.current_attempt >= self.max_retry_count: + return False + return await self._can_retry_async( + state=state, + request=request, + response=response, + error=error, + ) + + async def _can_retry_async( + self, + *, + state: RetryState, + request: HttpRequest, + response: Optional[HttpResponse] = None, + error: Optional[Exception] = None, + ) -> bool: + raise NotImplementedError() + + async def prepare_for_next_attempt_async( + self, + *, + state: RetryState, + request: HttpRequest, + response: Optional[HttpResponse] = None, + error: Optional[Exception] = None, + ) -> None: + state.next_attempt_requested = True + duration = self.interval_calculator.calculate_sleep_duration(state.current_attempt) + await asyncio.sleep(duration) + state.increment_current_attempt() + + +__all__ = [ + "RetryState", + "HttpRequest", + "HttpResponse", + "RetryIntervalCalculator", + "BackoffRetryIntervalCalculator", + "default_interval_calculator", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/http_retry/builtin_async_handlers.py b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/builtin_async_handlers.py new file mode 100644 index 0000000..9993f5e --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/builtin_async_handlers.py @@ -0,0 +1,90 @@ +import asyncio +import random +from typing import Optional, List, Type + +from aiohttp import ServerDisconnectedError, ServerConnectionError, ClientOSError + +from slack_sdk.http_retry.async_handler import AsyncRetryHandler +from slack_sdk.http_retry.interval_calculator import RetryIntervalCalculator +from slack_sdk.http_retry.state import RetryState +from slack_sdk.http_retry.request import HttpRequest +from slack_sdk.http_retry.response import HttpResponse +from slack_sdk.http_retry.handler import default_interval_calculator + + +class AsyncConnectionErrorRetryHandler(AsyncRetryHandler): + """RetryHandler that does retries for connectivity issues.""" + + def __init__( + self, + max_retry_count: int = 1, + interval_calculator: RetryIntervalCalculator = default_interval_calculator, + error_types: List[Type[Exception]] = [ + ServerConnectionError, + ServerDisconnectedError, + # ClientOSError: [Errno 104] Connection reset by peer + ClientOSError, + ], + ): + super().__init__(max_retry_count, interval_calculator) + self.error_types_to_do_retries = error_types + + async def _can_retry_async( + self, + *, + state: RetryState, + request: HttpRequest, + response: Optional[HttpResponse] = None, + error: Optional[Exception] = None, + ) -> bool: + if error is None: + return False + + for error_type in self.error_types_to_do_retries: + if isinstance(error, error_type): + return True + return False + + +class AsyncRateLimitErrorRetryHandler(AsyncRetryHandler): + """RetryHandler that does retries for rate limited errors.""" + + async def _can_retry_async( + self, + *, + state: RetryState, + request: HttpRequest, + response: Optional[HttpResponse], + error: Optional[Exception], + ) -> bool: + return response is not None and response.status_code == 429 + + async def prepare_for_next_attempt_async( + self, + *, + state: RetryState, + request: HttpRequest, + response: Optional[HttpResponse] = None, + error: Optional[Exception] = None, + ) -> None: + if response is None: + raise error + + state.next_attempt_requested = True + retry_after_header_name: Optional[str] = None + for k in response.headers.keys(): + if k.lower() == "retry-after": + retry_after_header_name = k + break + duration = 1 + if retry_after_header_name is None: + # This situation usually does not arise. Just in case. + duration += random.random() + else: + duration = int(response.headers.get(retry_after_header_name)[0]) + random.random() + await asyncio.sleep(duration) + state.increment_current_attempt() + + +def async_default_handlers() -> List[AsyncRetryHandler]: + return [AsyncConnectionErrorRetryHandler()] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/http_retry/builtin_handlers.py b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/builtin_handlers.py new file mode 100644 index 0000000..d8ac13d --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/builtin_handlers.py @@ -0,0 +1,89 @@ +import random +import time +from http.client import RemoteDisconnected +from typing import Optional, List, Type +from urllib.error import URLError + +from slack_sdk.http_retry.interval_calculator import RetryIntervalCalculator +from slack_sdk.http_retry.state import RetryState +from slack_sdk.http_retry.request import HttpRequest +from slack_sdk.http_retry.response import HttpResponse +from slack_sdk.http_retry.handler import RetryHandler, default_interval_calculator + + +class ConnectionErrorRetryHandler(RetryHandler): + """RetryHandler that does retries for connectivity issues.""" + + def __init__( + self, + max_retry_count: int = 1, + interval_calculator: RetryIntervalCalculator = default_interval_calculator, + error_types: List[Type[Exception]] = [ + # To cover URLError: + URLError, + ConnectionResetError, + RemoteDisconnected, + ], + ): + super().__init__(max_retry_count, interval_calculator) + self.error_types_to_do_retries = error_types + + def _can_retry( + self, + *, + state: RetryState, + request: HttpRequest, + response: Optional[HttpResponse] = None, + error: Optional[Exception] = None, + ) -> bool: + if error is None: + return False + + if isinstance(error, URLError): + if response is not None: + return False # status 40x + + for error_type in self.error_types_to_do_retries: + if isinstance(error, error_type): + return True + return False + + +class RateLimitErrorRetryHandler(RetryHandler): + """RetryHandler that does retries for rate limited errors.""" + + def _can_retry( + self, + *, + state: RetryState, + request: HttpRequest, + response: Optional[HttpResponse] = None, + error: Optional[Exception] = None, + ) -> bool: + return response is not None and response.status_code == 429 + + def prepare_for_next_attempt( + self, + *, + state: RetryState, + request: HttpRequest, + response: Optional[HttpResponse] = None, + error: Optional[Exception] = None, + ) -> None: + if response is None: + raise error + + state.next_attempt_requested = True + retry_after_header_name: Optional[str] = None + for k in response.headers.keys(): + if k.lower() == "retry-after": + retry_after_header_name = k + break + duration = 1 + if retry_after_header_name is None: + # This situation usually does not arise. Just in case. + duration += random.random() + else: + duration = int(response.headers.get(retry_after_header_name)[0]) + random.random() + time.sleep(duration) + state.increment_current_attempt() diff --git a/core_service/aws_lambda/project/packages/slack_sdk/http_retry/builtin_interval_calculators.py b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/builtin_interval_calculators.py new file mode 100644 index 0000000..6354171 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/builtin_interval_calculators.py @@ -0,0 +1,44 @@ +from typing import Optional +from .jitter import Jitter, RandomJitter +from .interval_calculator import RetryIntervalCalculator + + +class FixedValueRetryIntervalCalculator(RetryIntervalCalculator): + """Retry interval calculator that uses a fixed value.""" + + fixed_interval: float + + def __init__(self, fixed_internal: float = 0.5): + """Retry interval calculator that uses a fixed value. + + Args: + fixed_internal: The fixed interval seconds + """ + self.fixed_interval = fixed_internal + + def calculate_sleep_duration(self, current_attempt: int) -> float: + return self.fixed_interval + + +class BackoffRetryIntervalCalculator(RetryIntervalCalculator): + """Retry interval calculator that calculates in the manner of Exponential Backoff And Jitter + see also: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ + """ + + backoff_factor: float + jitter: Jitter + + def __init__(self, backoff_factor: float = 0.5, jitter: Optional[Jitter] = None): + """Retry interval calculator that calculates in the manner of Exponential Backoff And Jitter + + Args: + backoff_factor: The factor for the backoff interval calculation + jitter: The jitter logic implementation + """ + self.backoff_factor = backoff_factor + self.jitter = jitter if jitter is not None else RandomJitter() + + def calculate_sleep_duration(self, current_attempt: int) -> float: + interval = self.backoff_factor * (2 ** (current_attempt)) + sleep_duration = self.jitter.recalculate(interval) + return sleep_duration diff --git a/core_service/aws_lambda/project/packages/slack_sdk/http_retry/handler.py b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/handler.py new file mode 100644 index 0000000..7c8aa46 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/handler.py @@ -0,0 +1,80 @@ +"""RetryHandler interface. +You can pass an array of handlers to customize retry logics in supported API clients. +""" + +import time +from typing import Optional + +from slack_sdk.http_retry.state import RetryState +from slack_sdk.http_retry.request import HttpRequest +from slack_sdk.http_retry.response import HttpResponse +from slack_sdk.http_retry.interval_calculator import RetryIntervalCalculator +from slack_sdk.http_retry.builtin_interval_calculators import ( + BackoffRetryIntervalCalculator, +) + +default_interval_calculator = BackoffRetryIntervalCalculator() + + +# Note that you cannot add aiohttp to this class as the external dependency is optional +class RetryHandler: + """RetryHandler interface. + You can pass an array of handlers to customize retry logics in supported API clients. + """ + + max_retry_count: int + interval_calculator: RetryIntervalCalculator + + def __init__( + self, + max_retry_count: int = 1, + interval_calculator: RetryIntervalCalculator = default_interval_calculator, + ): + """RetryHandler interface. + + Args: + max_retry_count: The maximum times to do retries + interval_calculator: Pass an interval calculator for customizing the logic + """ + self.max_retry_count = max_retry_count + self.interval_calculator = interval_calculator + + def can_retry( + self, + *, + state: RetryState, + request: HttpRequest, + response: Optional[HttpResponse] = None, + error: Optional[Exception] = None, + ) -> bool: + if state.current_attempt >= self.max_retry_count: + return False + return self._can_retry( + state=state, + request=request, + response=response, + error=error, + ) + + def _can_retry( + self, + *, + state: RetryState, + request: HttpRequest, + response: Optional[HttpResponse] = None, + error: Optional[Exception] = None, + ) -> bool: + raise NotImplementedError() + + def prepare_for_next_attempt( + self, + *, + state: RetryState, + request: HttpRequest, + response: Optional[HttpResponse] = None, + error: Optional[Exception] = None, + ) -> None: + state.next_attempt_requested = True + duration = self.interval_calculator.calculate_sleep_duration(state.current_attempt) + time.sleep(duration) + state.increment_current_attempt() diff --git a/core_service/aws_lambda/project/packages/slack_sdk/http_retry/interval_calculator.py b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/interval_calculator.py new file mode 100644 index 0000000..3911dd3 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/interval_calculator.py @@ -0,0 +1,12 @@ +class RetryIntervalCalculator: + """Retry interval calculator interface.""" + + def calculate_sleep_duration(self, current_attempt: int) -> float: + """Calculates an interval duration in seconds. + + Args: + current_attempt: the number of the current attempt (zero-origin; 0 means no retries are done so far) + Returns: + calculated interval duration in seconds + """ + raise NotImplementedError() diff --git a/core_service/aws_lambda/project/packages/slack_sdk/http_retry/jitter.py b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/jitter.py new file mode 100644 index 0000000..852eaac --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/jitter.py @@ -0,0 +1,24 @@ +import random + + +class Jitter: + """Jitter interface""" + + def recalculate(self, duration: float) -> float: + """Recalculate the given duration. + see also: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ + + Args: + duration: the duration in seconds + + Returns: + A new duration that the jitter amount is added + """ + raise NotImplementedError() + + +class RandomJitter(Jitter): + """Random jitter implementation""" + + def recalculate(self, duration: float) -> float: + return duration + random.random() diff --git a/core_service/aws_lambda/project/packages/slack_sdk/http_retry/request.py b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/request.py new file mode 100644 index 0000000..76db4a4 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/request.py @@ -0,0 +1,36 @@ +from typing import Dict, Optional, List, Union, Any +from urllib.request import Request + + +class HttpRequest: + """HTTP request representation""" + + method: str + url: str + headers: Dict[str, Union[str, List[str]]] + body_params: Optional[Dict[str, Any]] + data: Optional[bytes] + + def __init__( + self, + *, + method: str, + url: str, + headers: Dict[str, Union[str, List[str]]], + body_params: Optional[Dict[str, Any]] = None, + data: Optional[bytes] = None, + ): + self.method = method + self.url = url + self.headers = {k: v if isinstance(v, list) else [v] for k, v in headers.items()} + self.body_params = body_params + self.data = data + + @classmethod + def from_urllib_http_request(cls, req: Request) -> "HttpRequest": + return HttpRequest( + method=req.method, + url=req.full_url, + headers={k: v if isinstance(v, list) else [v] for k, v in req.headers.items()}, + data=req.data, + ) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/http_retry/response.py b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/response.py new file mode 100644 index 0000000..2826810 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/response.py @@ -0,0 +1,23 @@ +from typing import Dict, Optional, List, Union, Any + + +class HttpResponse: + """HTTP response representation""" + + status_code: int + headers: Dict[str, List[str]] + body: Optional[Dict[str, Any]] + data: Optional[bytes] + + def __init__( + self, + *, + status_code: Union[int, str], + headers: Dict[str, Union[str, List[str]]], + body: Optional[Dict[str, Any]] = None, + data: Optional[bytes] = None, + ): + self.status_code = int(status_code) + self.headers = {k: v if isinstance(v, list) else [v] for k, v in headers.items()} + self.body = body + self.data = data diff --git a/core_service/aws_lambda/project/packages/slack_sdk/http_retry/state.py b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/state.py new file mode 100644 index 0000000..4217eb5 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/http_retry/state.py @@ -0,0 +1,21 @@ +from typing import Optional, Any, Dict + + +class RetryState: + next_attempt_requested: bool + current_attempt: int # zero-origin + custom_values: Optional[Dict[str, Any]] + + def __init__( + self, + *, + current_attempt: int = 0, + custom_values: Optional[Dict[str, Any]] = None, + ): + self.next_attempt_requested = False + self.current_attempt = current_attempt + self.custom_values = custom_values + + def increment_current_attempt(self) -> int: + self.current_attempt += 1 + return self.current_attempt diff --git a/core_service/aws_lambda/project/packages/slack_sdk/models/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/models/__init__.py new file mode 100644 index 0000000..f3e9e34 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/models/__init__.py @@ -0,0 +1,58 @@ +"""Classes for constructing Slack-specific data structure""" + +import logging +from typing import Union, Dict, Any, Sequence, List + +from .basic_objects import BaseObject +from .basic_objects import EnumValidator +from .basic_objects import JsonObject +from .basic_objects import JsonValidator + + +# NOTE: used only for legacy components - don't use this for Block Kit +def extract_json( + item_or_items: Union[JsonObject, Sequence[JsonObject]], *format_args +) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]: # type: ignore + """ + Given a sequence (or single item), attempt to call the to_dict() method on each + item and return a plain list. If item is not the expected type, return it + unmodified, in case it's already a plain dict or some other user created class. + + Args: + item_or_items: item(s) to go through + format_args: Any formatting specifiers to pass into the object's to_dict + method + """ + try: + return [ # type: ignore + elem.to_dict(*format_args) if isinstance(elem, JsonObject) else elem for elem in item_or_items + ] + except TypeError: # not iterable, so try returning it as a single item + return ( # type: ignore + item_or_items.to_dict(*format_args) if isinstance(item_or_items, JsonObject) else item_or_items + ) + + +def show_unknown_key_warning(name: Union[str, object], others: dict): + if "type" in others: + others.pop("type") + if len(others) > 0: + keys = ", ".join(others.keys()) + logger = logging.getLogger(__name__) + if isinstance(name, object): + name = name.__class__.__name__ + logger.debug( + f"!!! {name}'s constructor args ({keys}) were ignored." + f"If they should be supported by this library, report this issue to the project :bow: " + f"https://github.com/slackapi/python-slack-sdk/issues" + ) + + +__all__ = [ + "BaseObject", + "EnumValidator", + "JsonObject", + "JsonValidator", + "extract_json", + "show_unknown_key_warning", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/models/attachments/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/models/attachments/__init__.py new file mode 100644 index 0000000..6a40df9 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/models/attachments/__init__.py @@ -0,0 +1,588 @@ +import re +from abc import ABCMeta, abstractmethod +from typing import List, Optional, Set, Sequence + +from slack_sdk.models import extract_json +from slack_sdk.models.basic_objects import ( + EnumValidator, + JsonObject, + JsonValidator, +) +from slack_sdk.models.blocks import ( + Block, + Option, + ConfirmObject, + ButtonStyles, + DynamicSelectElementTypes, +) + + +class Action(JsonObject): + """Action in attachments + https://api.slack.com/messaging/composing/layouts#attachments + https://api.slack.com/legacy/interactive-message-field-guide#message_action_fields + """ + + attributes = {"name", "text", "url"} + + def __init__( + self, + *, + text: str, + subtype: str, + name: Optional[str] = None, + url: Optional[str] = None, + ): + self.name = name + self.url = url + self.text = text + self.subtype = subtype + + @JsonValidator("name or url attribute is required") + def name_or_url_present(self): + return self.name is not None or self.url is not None + + def to_dict(self) -> dict: # skipcq: PYL-W0221 + json = super().to_dict() + json["type"] = self.subtype + return json + + +class ActionButton(Action): + @property + def attributes(self): + return super().attributes.union({"style", "value"}) + + value_max_length = 2000 + + def __init__( + self, + *, + name: str, + text: str, + value: str, + confirm: Optional[ConfirmObject] = None, + style: Optional[str] = None, + ): + """Simple button for use inside attachments + + https://api.slack.com/legacy/message-buttons + + Args: + name: Name this specific action. The name will be returned to your + Action URL along with the message's callback_id when this action is + invoked. Use it to identify this particular response path. + text: The user-facing label for the message button or menu + representing this action. Cannot contain markup. + value: Provide a string identifying this specific action. It will be + sent to your Action URL along with the name and attachment's + callback_id . If providing multiple actions with the same name, value + can be strategically used to differentiate intent. Cannot exceed 2000 + characters. + confirm: a ConfirmObject that will appear in a dialog to confirm + user's choice. + style: Leave blank to indicate that this is an ordinary button. Use + "primary" or "danger" to mark important buttons. + """ + super().__init__(name=name, text=text, subtype="button") + self.value = value + self.confirm = confirm + self.style = style + + @JsonValidator(f"value attribute cannot exceed {value_max_length} characters") + def value_length(self): + return len(self.value) <= self.value_max_length + + @EnumValidator("style", ButtonStyles) + def style_valid(self): + return self.style is None or self.style in ButtonStyles + + def to_dict(self) -> dict: + json = super().to_dict() + if self.confirm is not None: + json["confirm"] = extract_json(self.confirm, "action") + return json + + +class ActionLinkButton(Action): + def __init__(self, *, text: str, url: str): + """A simple interactive button that just opens a URL + + https://api.slack.com/messaging/composing/layouts#attachments + + Args: + text: text to display on the button, eg 'Click Me!" + url: the URL to open + """ + super().__init__(text=text, url=url, subtype="button") + + +class AbstractActionSelector(Action, metaclass=ABCMeta): + DataSourceTypes = DynamicSelectElementTypes.union({"external", "static"}) + + attributes = {"data_source", "name", "text", "type"} + + @property + @abstractmethod + def data_source(self) -> str: + pass + + def __init__(self, *, name: str, text: str, selected_option: Optional[Option] = None): + super().__init__(text=text, name=name, subtype="select") + self.selected_option = selected_option + + @EnumValidator("data_source", DataSourceTypes) + def data_source_valid(self): + return self.data_source in self.DataSourceTypes + + def to_dict(self) -> dict: + json = super().to_dict() + if self.selected_option is not None: + # this is a special case for ExternalActionSelectElement - in that case, + # you pass the initial value of the selector as a selected_options array + json["selected_options"] = extract_json([self.selected_option], "action") + return json + + +class ActionUserSelector(AbstractActionSelector): + data_source = "users" + + def __init__(self, name: str, text: str, selected_user: Optional[Option] = None): + """Automatically populate the selector with a list of users in the workspace. + + https://api.slack.com/legacy/message-menus#allow_users_to_select_from_a_list_of_members + + Args: + name: Name this specific action. The name will be returned to your + Action URL along with the message's callback_id when this action is + invoked. Use it to identify this particular response path. + text: The user-facing label for the message button or menu + representing this action. Cannot contain markup. + selected_user: An Option object to pre-select as the default + value. + """ + super().__init__(name=name, text=text, selected_option=selected_user) + + +class ActionChannelSelector(AbstractActionSelector): + data_source = "channels" + + def __init__(self, name: str, text: str, selected_channel: Optional[Option] = None): + """ + Automatically populate the selector with a list of public channels in the + workspace. + + https://api.slack.com/legacy/message-menus#let_users_choose_one_of_their_workspace_s_channels + + Args: + name: Name this specific action. The name will be returned to your + Action URL along with the message's callback_id when this action is + invoked. Use it to identify this particular response path. + text: The user-facing label for the message button or menu + representing this action. Cannot contain markup. + selected_channel: An Option object to pre-select as the default + value. + """ + super().__init__(name=name, text=text, selected_option=selected_channel) + + +class ActionConversationSelector(AbstractActionSelector): + data_source = "conversations" + + def __init__(self, name: str, text: str, selected_conversation: Optional[Option] = None): + """ + Automatically populate the selector with a list of conversations they have in + the workspace. + + https://api.slack.com/legacy/message-menus#let_users_choose_one_of_their_conversations + + Args: + name: Name this specific action. The name will be returned to your + Action URL along with the message's callback_id when this action is + invoked. Use it to identify this particular response path. + text: The user-facing label for the message button or menu + representing this action. Cannot contain markup. + selected_conversation: An Option object to pre-select as the default + value. + """ + super().__init__(name=name, text=text, selected_option=selected_conversation) + + +class ActionExternalSelector(AbstractActionSelector): + data_source = "external" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"min_query_length"}) + + def __init__( + self, + *, + name: str, + text: str, + selected_option: Optional[Option] = None, + min_query_length: Optional[int] = None, + ): + """ + Populate a message select menu from your own application dynamically. + + https://api.slack.com/legacy/message-menus#populate_message_menus_dynamically + + Args: + name: Name this specific action. The name will be returned to your + Action URL along with the message's callback_id when this action is + invoked. Use it to identify this particular response path. + text: The user-facing label for the message button or menu + representing this action. Cannot contain markup. + selected_option: An Option object to pre-select as the default + value. + min_query_length: Specify the number of characters that must be typed + by a user into a dynamic select menu before dispatching to the app. + """ + super().__init__(name=name, text=text, selected_option=selected_option) + self.min_query_length = min_query_length + + +SeededColors = {"danger", "good", "warning"} + + +class AttachmentField(JsonObject): + attributes = {"short", "title", "value"} + + def __init__( + self, + *, + title: Optional[str] = None, + value: Optional[str] = None, + short: bool = True, + ): + self.title = title + self.value = value + self.short = short + + +class Attachment(JsonObject): + attributes = { + "author_icon", + "author_link", + "author_name", + "color", + "fallback", + "fields", + "footer", + "footer_icon", + "image_url", + "pretext", + "text", + "thumb_url", + "title", + "title_link", + "ts", + } + + fields: Sequence[AttachmentField] + + MarkdownFields = {"fields", "pretext", "text"} + + footer_max_length = 300 + + def __init__( + self, + *, + text: str, + fallback: Optional[str] = None, + fields: Optional[Sequence[AttachmentField]] = None, + color: Optional[str] = None, + markdown_in: Optional[Sequence[str]] = None, + title: Optional[str] = None, + title_link: Optional[str] = None, + pretext: Optional[str] = None, + author_name: Optional[str] = None, + author_link: Optional[str] = None, + author_icon: Optional[str] = None, + image_url: Optional[str] = None, + thumb_url: Optional[str] = None, + footer: Optional[str] = None, + footer_icon: Optional[str] = None, + ts: Optional[int] = None, + ): + """ + A supplemental object that will display after the rest of the message. + Considered legacy - recommended replacement is to use message blocks instead. + + https://api.slack.com/reference/messaging/attachments#fields + + Args: + text: The main body text of the attachment. It can be formatted as + plain text, or with markdown by including it in the markdown_in + parameter. The content will automatically collapse if it contains 700+ + characters or 5+ linebreaks, and will display a "Show more..." link to + expand the content. + fallback: A plain text summary of the attachment used in clients that + don't show formatted text (eg. IRC, mobile notifications). + fields: An array of AttachmentField objects that get displayed in a + table-like way. For best results, include no more than 2-3 field + objects. + color: Changes the color of the border on the left side of this attachment + from the default gray. Can either be one of "good" (green), "warning" + (yellow), "danger" (red), or any hex color code (eg. #439FE0) + markdown_in: An array of field names that should be formatted by + markdown syntax - allowed values: "pretext", "text", "fields" + title: Large title text near the top of the attachment. + title_link: A valid URL that turns the title text into a hyperlink. + pretext: Text that appears above the message attachment block. It can + be formatted as plain text, or with markdown by including it in the + markdown_in parameter. + author_name: Small text used to display the author's name. + author_link: A valid URL that will hyperlink the author_name text. + Will only work if author_name is present. + author_icon: A valid URL that displays a small 16px by 16px image to + the left of the author_name text. Will only work if author_name is + present. + image_url: A valid URL to an image file that will be displayed at the + bottom of the attachment. We support GIF, JPEG, PNG, and BMP formats. + Large images will be resized to a maximum width of 360px or a maximum + height of 500px, while still maintaining the original aspect ratio. + Cannot be used with thumb_url. + thumb_url: A valid URL to an image file that will be displayed as a + thumbnail on the right side of a message attachment. We currently + support the following formats: GIF, JPEG, PNG, and BMP. The thumbnail's + longest dimension will be scaled down to 75px while maintaining the + aspect ratio of the image. The filesize of the image must also be less + than 500 KB. For best results, please use images that are already 75px + by 75px. + footer: Some brief text to help contextualize and identify an + attachment. Limited to 300 characters, and may be truncated further when + displayed to users in environments with limited screen real estate. + footer_icon: A valid URL to an image file that will be displayed + beside the footer text. Will only work if footer is present. We'll + render what you provide at 16px by 16px. It's best to use an image that + is similarly sized. + ts: An integer Unix timestamp that is used to related your attachment + to a specific time. The attachment will display the additional timestamp + value as part of the attachment's footer. Your message's timestamp will + be displayed in varying ways, depending on how far in the past or future + it is, relative to the present. Form factors, like mobile versus + desktop may also transform its rendered appearance. + """ + self.text = text + self.title = title + self.fallback = fallback + self.pretext = pretext + self.title_link = title_link + self.color = color + self.author_name = author_name + self.author_link = author_link + self.author_icon = author_icon + self.image_url = image_url + self.thumb_url = thumb_url + self.footer = footer + self.footer_icon = footer_icon + self.ts = ts + self.fields = fields or [] + self.markdown_in = markdown_in or [] + + @JsonValidator(f"footer attribute cannot exceed {footer_max_length} characters") + def footer_length(self) -> bool: + return self.footer is None or len(self.footer) <= self.footer_max_length + + @JsonValidator("ts attribute cannot be present if footer attribute is absent") + def ts_without_footer(self) -> bool: + return self.ts is None or self.footer is not None + + @EnumValidator("markdown_in", MarkdownFields) + def markdown_in_valid(self): + return not self.markdown_in or all(e in self.MarkdownFields for e in self.markdown_in) + + @JsonValidator("color attribute must be 'good', 'warning', 'danger', or a hex color code") + def color_valid(self) -> bool: + return ( + self.color is None + or self.color in SeededColors + or re.match("^#(?:[0-9A-F]{2}){3}$", self.color, re.IGNORECASE) is not None + ) + + @JsonValidator("image_url attribute cannot be present if thumb_url is populated") + def image_url_and_thumb_url_populated(self) -> bool: + return self.image_url is None or self.thumb_url is None + + @JsonValidator("name must be present if link is present") + def author_link_without_author_name(self) -> bool: + return self.author_link is None or self.author_name is not None + + @JsonValidator("icon must be present if link is present") + def author_link_without_author_icon(self) -> bool: + return self.author_link is None or self.author_icon is not None + + def to_dict(self) -> dict: # skipcq: PYL-W0221 + json = super().to_dict() + if self.fields is not None: + json["fields"] = extract_json(self.fields) + if self.markdown_in: + json["mrkdwn_in"] = self.markdown_in + return json + + +class BlockAttachment(Attachment): + blocks: List[Block] + + @property + def attributes(self): + return super().attributes.union({"blocks", "color"}) + + def __init__( + self, + *, + blocks: Sequence[Block], + color: Optional[str] = None, + fallback: Optional[str] = None, + ): + """ + A bridge between legacy attachments and Block Kit formatting - pass a list of + Block objects directly to this attachment. + + https://api.slack.com/reference/messaging/attachments#fields + + Args: + blocks: a sequence of Block objects + color: Changes the color of the border on the left side of this + attachment from the default gray. Can either be one of "good" (green), + "warning" (yellow), "danger" (red), or any hex color code (eg. #439FE0) + fallback: fallback text + """ + super().__init__(text="", fallback=fallback, color=color) + self.blocks = list(blocks) + + @JsonValidator("fields attribute cannot be populated on BlockAttachment") + def fields_attribute_absent(self) -> bool: + return not self.fields + + def to_dict(self) -> dict: + json = super().to_dict() + json.update({"blocks": extract_json(self.blocks)}) + del json["fields"] # cannot supply fields and blocks at the same time + return json + + +class InteractiveAttachment(Attachment): + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"callback_id"}) + + actions_max_length = 5 + + def __init__( + self, + *, + actions: Sequence[Action], + callback_id: str, + text: str, + fallback: Optional[str] = None, + fields: Optional[Sequence[AttachmentField]] = None, + color: Optional[str] = None, + markdown_in: Optional[Sequence[str]] = None, + title: Optional[str] = None, + title_link: Optional[str] = None, + pretext: Optional[str] = None, + author_name: Optional[str] = None, + author_link: Optional[str] = None, + author_icon: Optional[str] = None, + image_url: Optional[str] = None, + thumb_url: Optional[str] = None, + footer: Optional[str] = None, + footer_icon: Optional[str] = None, + ts: Optional[int] = None, + ): + """ + An Attachment, but designed to contain interactive Actions + Considered legacy - recommended replacement is to use message blocks instead. + + https://api.slack.com/legacy/interactive-message-field-guide#attachment_fields + https://api.slack.com/reference/messaging/attachments#fields + + Args: + actions: A collection of Action objects to include in the attachment. + Cannot exceed 5 elements. + callback_id: The ID used to identify this attachment. Will be part of the + payload sent back to your application. + text: The main body text of the attachment. It can be formatted as + plain text, or with markdown by including it in the markdown_in + parameter. The content will automatically collapse if it contains 700+ + characters or 5+ linebreaks, and will display a "Show more..." link to + expand the content. + fallback: A plain text summary of the attachment used in clients that + don't show formatted text (eg. IRC, mobile notifications). + fields: An array of AttachmentField objects that get displayed in a + table-like way. For best results, include no more than 2-3 field + objects. + color: Changes the color of the border on the left side of this attachment + from the default gray. Can either be one of "good" (green), "warning" + (yellow), "danger" (red), or any hex color code (eg. #439FE0) + markdown_in: An array of field names that should be formatted by + markdown syntax - allowed values: "pretext", "text", "fields" + title: Large title text near the top of the attachment. + title_link: A valid URL that turns the title text into a hyperlink. + pretext: Text that appears above the message attachment block. It can + be formatted as plain text, or with markdown by including it in the + markdown_in parameter. + author_name: Small text used to display the author's name. + author_link: A valid URL that will hyperlink the author_name text. + Will only work if author_name is present. + author_icon: A valid URL that displays a small 16px by 16px image to + the left of the author_name text. Will only work if author_name is + present. + image_url: A valid URL to an image file that will be displayed at the + bottom of the attachment. We support GIF, JPEG, PNG, and BMP formats. + Large images will be resized to a maximum width of 360px or a maximum + height of 500px, while still maintaining the original aspect ratio. + Cannot be used with thumb_url. + thumb_url: A valid URL to an image file that will be displayed as a + thumbnail on the right side of a message attachment. We currently + support the following formats: GIF, JPEG, PNG, and BMP. The thumbnail's + longest dimension will be scaled down to 75px while maintaining the + aspect ratio of the image. The filesize of the image must also be less + than 500 KB. For best results, please use images that are already 75px + by 75px. + footer: Some brief text to help contextualize and identify an + attachment. Limited to 300 characters, and may be truncated further when + displayed to users in environments with limited screen real estate. + footer_icon: A valid URL to an image file that will be displayed + beside the footer text. Will only work if footer is present. We'll + render what you provide at 16px by 16px. It's best to use an image that + is similarly sized. + ts: An integer Unix timestamp that is used to related your attachment + to a specific time. The attachment will display the additional timestamp + value as part of the attachment's footer. Your message's timestamp will + be displayed in varying ways, depending on how far in the past or future + it is, relative to the present. Form factors, like mobile versus + desktop may also transform its rendered appearance. + """ + super().__init__( + text=text, + title=title, + fallback=fallback, + fields=fields, + pretext=pretext, + title_link=title_link, + color=color, + author_name=author_name, + author_link=author_link, + author_icon=author_icon, + image_url=image_url, + thumb_url=thumb_url, + footer=footer, + footer_icon=footer_icon, + ts=ts, + markdown_in=markdown_in, + ) + self.callback_id = callback_id + self.actions = actions or [] + + @JsonValidator(f"actions attribute cannot exceed {actions_max_length} elements") + def actions_length(self) -> bool: + return len(self.actions) <= self.actions_max_length + + def to_dict(self) -> dict: + json = super().to_dict() + json["actions"] = extract_json(self.actions) + return json diff --git a/core_service/aws_lambda/project/packages/slack_sdk/models/basic_objects.py b/core_service/aws_lambda/project/packages/slack_sdk/models/basic_objects.py new file mode 100644 index 0000000..232ec21 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/models/basic_objects.py @@ -0,0 +1,113 @@ +from abc import ABCMeta, abstractmethod +from functools import wraps +from typing import Callable, Iterable, Set, Union, Any, Tuple + +from slack_sdk.errors import SlackObjectFormationError + + +class BaseObject: + """The base class for all model objects in this module""" + + def __str__(self): + return f"" + + +class JsonObject(BaseObject, metaclass=ABCMeta): + """The base class for JSON serializable class objects""" + + @property + @abstractmethod + def attributes(self) -> Set[str]: + """Provide a set of attributes of this object that will make up its JSON structure""" + return set() + + def validate_json(self) -> None: + """ + Raises: + SlackObjectFormationError if the object was not valid + """ + for attribute in (func for func in dir(self) if not func.startswith("__")): + method = getattr(self, attribute, None) + if callable(method) and hasattr(method, "validator"): + method() + + def get_non_null_attributes(self) -> dict: + """ + Construct a dictionary out of non-null keys (from attributes property) + present on this object + """ + + def to_dict_compatible(value: Union[dict, list, object, Tuple]) -> Union[dict, list, Any]: + if isinstance(value, (list, Tuple)): # skipcq: PYL-R1705 + return [to_dict_compatible(v) for v in value] + else: + to_dict = getattr(value, "to_dict", None) + if to_dict and callable(to_dict): # skipcq: PYL-R1705 + return {k: to_dict_compatible(v) for k, v in value.to_dict().items()} # type: ignore + else: + return value + + def is_not_empty(self, key: str) -> bool: + value = getattr(self, key, None) + if value is None: + return False + has_len = getattr(value, "__len__", None) is not None + if has_len: # skipcq: PYL-R1705 + return len(value) > 0 + else: + return value is not None + + return { + key: to_dict_compatible(getattr(self, key, None)) for key in sorted(self.attributes) if is_not_empty(self, key) + } + + def to_dict(self, *args) -> dict: + """ + Extract this object as a JSON-compatible, Slack-API-valid dictionary + + Args: + *args: Any specific formatting args (rare; generally not required) + + Raises: + SlackObjectFormationError if the object was not valid + """ + self.validate_json() + return self.get_non_null_attributes() + + def __repr__(self): + dict_value = self.get_non_null_attributes() + if dict_value: # skipcq: PYL-R1705 + return f"" + else: + return self.__str__() + + def __eq__(self, other: Any) -> bool: + if not isinstance(other, JsonObject): + return False + return self.to_dict() == other.to_dict() + + +class JsonValidator: + def __init__(self, message: str): + """ + Decorate a method on a class to mark it as a JSON validator. Validation + functions should return true if valid, false if not. + + Args: + message: Message to be attached to the thrown SlackObjectFormationError + """ + self.message = message + + def __call__(self, func: Callable) -> Callable[..., None]: + @wraps(func) + def wrapped_f(*args, **kwargs): + if not func(*args, **kwargs): + raise SlackObjectFormationError(self.message) + + wrapped_f.validator = True + return wrapped_f + + +class EnumValidator(JsonValidator): + def __init__(self, attribute: str, enum: Iterable[str]): + super().__init__(f"{attribute} attribute must be one of the following values: " f"{', '.join(enum)}") diff --git a/core_service/aws_lambda/project/packages/slack_sdk/models/blocks/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/models/blocks/__init__.py new file mode 100644 index 0000000..deea314 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/models/blocks/__init__.py @@ -0,0 +1,95 @@ +"""Block Kit data model objects + +To learn more about Block Kit, please check the following resources and tools: + +* https://api.slack.com/block-kit +* https://api.slack.com/reference/block-kit/blocks +* https://app.slack.com/block-kit-builder +""" +from .basic_components import ButtonStyles +from .basic_components import ConfirmObject +from .basic_components import DynamicSelectElementTypes +from .basic_components import MarkdownTextObject +from .basic_components import Option +from .basic_components import OptionGroup +from .basic_components import PlainTextObject +from .basic_components import TextObject +from .block_elements import BlockElement +from .block_elements import ButtonElement +from .block_elements import ChannelMultiSelectElement +from .block_elements import ChannelSelectElement +from .block_elements import CheckboxesElement +from .block_elements import ConversationFilter +from .block_elements import ConversationMultiSelectElement +from .block_elements import ConversationSelectElement +from .block_elements import DatePickerElement +from .block_elements import TimePickerElement +from .block_elements import ExternalDataMultiSelectElement +from .block_elements import ExternalDataSelectElement +from .block_elements import ImageElement +from .block_elements import InputInteractiveElement +from .block_elements import InteractiveElement +from .block_elements import LinkButtonElement +from .block_elements import OverflowMenuElement +from .block_elements import PlainTextInputElement +from .block_elements import RadioButtonsElement +from .block_elements import SelectElement +from .block_elements import StaticMultiSelectElement +from .block_elements import StaticSelectElement +from .block_elements import UserMultiSelectElement +from .block_elements import UserSelectElement +from .blocks import ActionsBlock +from .blocks import Block +from .blocks import CallBlock +from .blocks import ContextBlock +from .blocks import DividerBlock +from .blocks import FileBlock +from .blocks import HeaderBlock +from .blocks import ImageBlock +from .blocks import InputBlock +from .blocks import SectionBlock + +__all__ = [ + "ButtonStyles", + "ConfirmObject", + "DynamicSelectElementTypes", + "MarkdownTextObject", + "Option", + "OptionGroup", + "PlainTextObject", + "TextObject", + "BlockElement", + "ButtonElement", + "ChannelMultiSelectElement", + "ChannelSelectElement", + "CheckboxesElement", + "ConversationFilter", + "ConversationMultiSelectElement", + "ConversationSelectElement", + "DatePickerElement", + "TimePickerElement", + "ExternalDataMultiSelectElement", + "ExternalDataSelectElement", + "ImageElement", + "InputInteractiveElement", + "InteractiveElement", + "LinkButtonElement", + "OverflowMenuElement", + "PlainTextInputElement", + "RadioButtonsElement", + "SelectElement", + "StaticMultiSelectElement", + "StaticSelectElement", + "UserMultiSelectElement", + "UserSelectElement", + "ActionsBlock", + "Block", + "CallBlock", + "ContextBlock", + "DividerBlock", + "FileBlock", + "HeaderBlock", + "ImageBlock", + "InputBlock", + "SectionBlock", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/models/blocks/basic_components.py b/core_service/aws_lambda/project/packages/slack_sdk/models/blocks/basic_components.py new file mode 100644 index 0000000..1e708ff --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/models/blocks/basic_components.py @@ -0,0 +1,526 @@ +import copy +import logging +import warnings +from typing import List, Optional, Set, Union, Sequence, Dict, Any + +from slack_sdk.models import show_unknown_key_warning +from slack_sdk.models.basic_objects import ( + JsonObject, + JsonValidator, +) +from slack_sdk.models.messages import Link + +ButtonStyles = {"danger", "primary"} +DynamicSelectElementTypes = {"channels", "conversations", "users"} + + +class TextObject(JsonObject): + """The interface for text objects (types: plain_text, mrkdwn)""" + + attributes = {"text", "type", "emoji"} + logger = logging.getLogger(__name__) + + def _subtype_warning(self): # skipcq: PYL-R0201 + warnings.warn( + "subtype is deprecated since slackclient 2.6.0, use type instead", + DeprecationWarning, + ) + + @property + def subtype(self) -> Optional[str]: + return self.type + + @classmethod + def parse( + cls, + text: Union[str, Dict[str, Any], "TextObject"], + default_type: str = "mrkdwn", + ) -> Optional["TextObject"]: + if not text: # skipcq: PYL-R1705 + return None + elif isinstance(text, str): + if default_type == PlainTextObject.type: # skipcq: PYL-R1705 + return PlainTextObject.from_str(text) + else: + return MarkdownTextObject.from_str(text) + elif isinstance(text, dict): + d = copy.copy(text) + t = d.pop("type") + if t == PlainTextObject.type: # skipcq: PYL-R1705 + return PlainTextObject(**d) + else: + return MarkdownTextObject(**d) + elif isinstance(text, TextObject): + return text + else: + cls.logger.warning(f"Unknown type ({type(text)}) detected when parsing a TextObject") + return None + + def __init__( + self, + text: str, + type: Optional[str] = None, # skipcq: PYL-W0622 + subtype: Optional[str] = None, + emoji: Optional[bool] = None, + **kwargs, + ): + """Super class for new text "objects" used in Block kit""" + if subtype: + self._subtype_warning() + + self.text = text + self.type = type if type else subtype + self.emoji = emoji + + +class PlainTextObject(TextObject): + """plain_text typed text object""" + + type = "plain_text" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"emoji"}) + + def __init__(self, *, text: str, emoji: Optional[bool] = None): + """A plain text object, meaning markdown characters will not be parsed as + formatting information. + https://api.slack.com/reference/block-kit/composition-objects#text + + Args: + text (required): The text for the block. This field accepts any of the standard text formatting markup + when type is mrkdwn. + emoji: Indicates whether emojis in a text field should be escaped into the colon emoji format. + This field is only usable when type is plain_text. + """ + super().__init__(text=text, type=self.type) + self.emoji = emoji + + @staticmethod + def from_str(text: str) -> "PlainTextObject": + return PlainTextObject(text=text, emoji=True) + + @staticmethod + def direct_from_string(text: str) -> Dict[str, Any]: + """Transforms a string into the required object shape to act as a PlainTextObject""" + return PlainTextObject.from_str(text).to_dict() + + +class MarkdownTextObject(TextObject): + """mrkdwn typed text object""" + + type = "mrkdwn" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"verbatim"}) + + def __init__(self, *, text: str, verbatim: Optional[bool] = None): + """A Markdown text object, meaning markdown characters will be parsed as + formatting information. + https://api.slack.com/reference/block-kit/composition-objects#text + + Args: + text (required): The text for the block. This field accepts any of the standard text formatting markup + when type is mrkdwn. + verbatim: When set to false (as is default) URLs will be auto-converted into links, + conversation names will be link-ified, and certain mentions will be automatically parsed. + Using a value of true will skip any preprocessing of this nature, + although you can still include manual parsing strings. This field is only usable when type is mrkdwn. + """ + super().__init__(text=text, type=self.type) + self.verbatim = verbatim + + @staticmethod + def from_str(text: str) -> "MarkdownTextObject": + """Transforms a string into the required object shape to act as a MarkdownTextObject""" + return MarkdownTextObject(text=text) + + @staticmethod + def direct_from_string(text: str) -> Dict[str, Any]: + """Transforms a string into the required object shape to act as a MarkdownTextObject""" + return MarkdownTextObject.from_str(text).to_dict() + + @staticmethod + def from_link(link: Link, title: str = "") -> "MarkdownTextObject": + """ + Transform a Link object directly into the required object shape + to act as a MarkdownTextObject + """ + if title: + title = f": {title}" + return MarkdownTextObject(text=f"{link}{title}") + + @staticmethod + def direct_from_link(link: Link, title: str = "") -> Dict[str, Any]: + """ + Transform a Link object directly into the required object shape + to act as a MarkdownTextObject + """ + return MarkdownTextObject.from_link(link, title).to_dict() + + +class Option(JsonObject): + """Option object used in dialogs, legacy message actions (interactivity in attachments), + and blocks. JSON must be retrieved with an explicit option_type - the Slack API has + different required formats in different situations + """ + + attributes = {} # no attributes because to_dict has unique implementations + logger = logging.getLogger(__name__) + + label_max_length = 75 + value_max_length = 75 + + def __init__( + self, + *, + value: str, + label: Optional[str] = None, + text: Optional[Union[str, Dict[str, Any], TextObject]] = None, # Block Kit + description: Optional[Union[str, Dict[str, Any], TextObject]] = None, + url: Optional[str] = None, + **others: Dict[str, Any], + ): + """ + An object that represents a single selectable item in a block element ( + SelectElement, OverflowMenuElement) or dialog element + (StaticDialogSelectElement) + + Blocks: + https://api.slack.com/reference/block-kit/composition-objects#option + + Dialogs: + https://api.slack.com/dialogs#select_elements + + Legacy interactive attachments: + https://api.slack.com/legacy/interactive-message-field-guide#option_fields + + Args: + label: A short, user-facing string to label this option to users. + Cannot exceed 75 characters. + value: A short string that identifies this particular option to your + application. It will be part of the payload when this option is selected + . Cannot exceed 75 characters. + description: A user-facing string that provides more details about + this option. Only supported in legacy message actions, not in blocks or + dialogs. + """ + if text: + # For better compatibility with Block Kit ("mrkdwn" does not work for it), + # we've changed the default text object type to plain_text since version 3.10.0 + self._text: Optional[TextObject] = TextObject.parse( + text=text, # "text" here can be either a str or a TextObject + default_type=PlainTextObject.type, + ) + self._label: Optional[str] = None + else: + self._text: Optional[TextObject] = None + self._label: Optional[str] = label + + # for backward-compatibility with version 2.0-2.5, the following fields return str values + self.text: Optional[str] = self._text.text if self._text else None + self.label: Optional[str] = self._label + + self.value: str = value + + # for backward-compatibility with version 2.0-2.5, the following fields return str values + if isinstance(description, str): + self.description = description + self._block_description = PlainTextObject.from_str(description) + elif isinstance(description, dict): + self.description = description["text"] + self._block_description = TextObject.parse(description) + elif isinstance(description, TextObject): + self.description = description.text + self._block_description = description + else: + self.description = None + self._block_description = None + + # A URL to load in the user's browser when the option is clicked. + # The url attribute is only available in overflow menus. + # Maximum length for this field is 3000 characters. + # If you're using url, you'll still receive an interaction payload + # and will need to send an acknowledgement response. + self.url: Optional[str] = url + show_unknown_key_warning(self, others) + + @JsonValidator(f"label attribute cannot exceed {label_max_length} characters") + def _validate_label_length(self) -> bool: + return self._label is None or len(self._label) <= self.label_max_length + + @JsonValidator(f"text attribute cannot exceed {label_max_length} characters") + def _validate_text_length(self) -> bool: + return self._text is None or self._text.text is None or len(self._text.text) <= self.label_max_length + + @JsonValidator(f"value attribute cannot exceed {value_max_length} characters") + def _validate_value_length(self) -> bool: + return len(self.value) <= self.value_max_length + + @classmethod + def parse_all(cls, options: Optional[Sequence[Union[Dict[str, Any], "Option"]]]) -> Optional[List["Option"]]: + if options is None: + return None + option_objects: List[Option] = [] + for o in options: + if isinstance(o, dict): + d = copy.copy(o) + option_objects.append(Option(**d)) + elif isinstance(o, Option): + option_objects.append(o) + else: + cls.logger.warning(f"Unknown option object detected and skipped ({o})") + return option_objects + + def to_dict(self, option_type: str = "block") -> Dict[str, Any]: # skipcq: PYL-W0221 + """ + Different parent classes must call this with a valid value from OptionTypes - + either "dialog", "action", or "block", so that JSON is returned in the + correct shape. + """ + self.validate_json() + if option_type == "dialog": # skipcq: PYL-R1705 + return {"label": self.label, "value": self.value} + elif option_type == "action" or option_type == "attachment": + # "action" can be confusing but it means a legacy message action in attachments + # we don't remove the type name for backward compatibility though + json = {"text": self.label, "value": self.value} + if self.description is not None: + json["description"] = self.description + return json + else: # if option_type == "block"; this should be the most common case + text: TextObject = self._text or PlainTextObject.from_str(self.label) + json: Dict[str, Any] = { + "text": text.to_dict(), + "value": self.value, + } + if self._block_description: + json["description"] = self._block_description.to_dict() + if self.url: + json["url"] = self.url + return json + + @staticmethod + def from_single_value(value_and_label: str): + """Creates a simple Option instance with the same value and label""" + return Option(value=value_and_label, label=value_and_label) + + +class OptionGroup(JsonObject): + """ + JSON must be retrieved with an explicit option_type - the Slack API has + different required formats in different situations + """ + + attributes = {} # no attributes because to_dict has unique implementations + label_max_length = 75 + options_max_length = 100 + logger = logging.getLogger(__name__) + + def __init__( + self, + *, + label: Optional[Union[str, Dict[str, Any], TextObject]] = None, + options: Sequence[Union[Dict[str, Any], Option]], + **others: Dict[str, Any], + ): + """ + Create a group of Option objects - pass in a label (that will be part of the + UI) and a list of Option objects. + + Blocks: + https://api.slack.com/reference/block-kit/composition-objects#option-group + + Dialogs: + https://api.slack.com/dialogs#select_elements + + Legacy interactive attachments: + https://api.slack.com/legacy/interactive-message-field-guide#option_groups_to_place_within_message_menu_actions + + Args: + label: Text to display at the top of this group of options. + options: A list of no more than 100 Option objects. + """ # noqa prevent flake8 blowing up on the long URL + # default_type=PlainTextObject.type is for backward-compatibility + self._label: Optional[TextObject] = TextObject.parse(label, default_type=PlainTextObject.type) + self.label: Optional[str] = self._label.text if self._label else None + self.options = Option.parse_all(options) # compatible with version 2.5 + show_unknown_key_warning(self, others) + + @JsonValidator(f"label attribute cannot exceed {label_max_length} characters") + def _validate_label_length(self): + return self.label is None or len(self.label) <= self.label_max_length + + @JsonValidator(f"options attribute cannot exceed {options_max_length} elements") + def _validate_options_length(self): + return self.options is None or len(self.options) <= self.options_max_length + + @classmethod + def parse_all( + cls, option_groups: Optional[Sequence[Union[Dict[str, Any], "OptionGroup"]]] + ) -> Optional[List["OptionGroup"]]: + if option_groups is None: + return None + option_group_objects = [] + for o in option_groups: + if isinstance(o, dict): + d = copy.copy(o) + option_group_objects.append(OptionGroup(**d)) + elif isinstance(o, OptionGroup): + option_group_objects.append(o) + else: + cls.logger.warning(f"Unknown option group object detected and skipped ({o})") + return option_group_objects + + def to_dict(self, option_type: str = "block") -> Dict[str, Any]: # skipcq: PYL-W0221 + self.validate_json() + dict_options = [o.to_dict(option_type) for o in self.options] + if option_type == "dialog": # skipcq: PYL-R1705 + return { + "label": self.label, + "options": dict_options, + } + elif option_type == "action": + return { + "text": self.label, + "options": dict_options, + } + else: # if option_type == "block"; this should be the most common case + dict_label: Dict[str, Any] = self._label.to_dict() + return { + "label": dict_label, + "options": dict_options, + } + + +class ConfirmObject(JsonObject): + attributes = {} # no attributes because to_dict has unique implementations + + title_max_length = 100 + text_max_length = 300 + confirm_max_length = 30 + deny_max_length = 30 + + @classmethod + def parse(cls, confirm: Union["ConfirmObject", Dict[str, Any]]): + if confirm: + if isinstance(confirm, ConfirmObject): # skipcq: PYL-R1705 + return confirm + elif isinstance(confirm, dict): + return ConfirmObject(**confirm) + else: + # Not yet implemented: show some warning here + return None + return None + + def __init__( + self, + *, + title: Union[str, Dict[str, Any], PlainTextObject], + text: Union[str, Dict[str, Any], TextObject], + confirm: Union[str, Dict[str, Any], PlainTextObject] = "Yes", + deny: Union[str, Dict[str, Any], PlainTextObject] = "No", + style: Optional[str] = None, + ): + """ + An object that defines a dialog that provides a confirmation step to any + interactive element. This dialog will ask the user to confirm their action by + offering a confirm and deny button. + https://api.slack.com/reference/block-kit/composition-objects#confirm + """ + self._title = TextObject.parse(title, default_type=PlainTextObject.type) + self._text = TextObject.parse(text, default_type=MarkdownTextObject.type) + self._confirm = TextObject.parse(confirm, default_type=PlainTextObject.type) + self._deny = TextObject.parse(deny, default_type=PlainTextObject.type) + self._style = style + + # for backward-compatibility with version 2.0-2.5, the following fields return str values + self.title = self._title.text if self._title else None + self.text = self._text.text if self._text else None + self.confirm = self._confirm.text if self._confirm else None + self.deny = self._deny.text if self._deny else None + self.style = self._style + + @JsonValidator(f"title attribute cannot exceed {title_max_length} characters") + def title_length(self) -> bool: + return self._title is None or len(self._title.text) <= self.title_max_length + + @JsonValidator(f"text attribute cannot exceed {text_max_length} characters") + def text_length(self) -> bool: + return self._text is None or len(self._text.text) <= self.text_max_length + + @JsonValidator(f"confirm attribute cannot exceed {confirm_max_length} characters") + def confirm_length(self) -> bool: + return self._confirm is None or len(self._confirm.text) <= self.confirm_max_length + + @JsonValidator(f"deny attribute cannot exceed {deny_max_length} characters") + def deny_length(self) -> bool: + return self._deny is None or len(self._deny.text) <= self.deny_max_length + + @JsonValidator('style for confirm must be either "primary" or "danger"') + def _validate_confirm_style(self) -> bool: + return self._style is None or self._style in ["primary", "danger"] + + def to_dict(self, option_type: str = "block") -> Dict[str, Any]: # skipcq: PYL-W0221 + if option_type == "action": # skipcq: PYL-R1705 + # deliberately skipping JSON validators here - can't find documentation + # on actual limits here + json = { + "ok_text": self._confirm.text if self._confirm and self._confirm.text != "Yes" else "Okay", + "dismiss_text": self._deny.text if self._deny and self._deny.text != "No" else "Cancel", + } + if self._title: + json["title"] = self._title.text + if self._text: + json["text"] = self._text.text + return json + + else: + self.validate_json() + json = {} + if self._title: + json["title"] = self._title.to_dict() + if self._text: + json["text"] = self._text.to_dict() + if self._confirm: + json["confirm"] = self._confirm.to_dict() + if self._deny: + json["deny"] = self._deny.to_dict() + if self._style: + json["style"] = self._style + return json + + +class DispatchActionConfig(JsonObject): + attributes = {"trigger_actions_on"} + + @classmethod + def parse(cls, config: Union["DispatchActionConfig", Dict[str, Any]]): + if config: + if isinstance(config, DispatchActionConfig): # skipcq: PYL-R1705 + return config + elif isinstance(config, dict): + return DispatchActionConfig(**config) + else: + # Not yet implemented: show some warning here + return None + return None + + def __init__( + self, + *, + trigger_actions_on: Optional[List[Any]] = None, + ): + """ + Determines when a plain-text input element will return a block_actions interaction payload. + https://api.slack.com/reference/block-kit/composition-objects#dispatch_action_config + """ + self._trigger_actions_on = trigger_actions_on or [] + + def to_dict(self) -> Dict[str, Any]: # skipcq: PYL-W0221 + self.validate_json() + json = {} + if self._trigger_actions_on: + json["trigger_actions_on"] = self._trigger_actions_on + return json diff --git a/core_service/aws_lambda/project/packages/slack_sdk/models/blocks/block_elements.py b/core_service/aws_lambda/project/packages/slack_sdk/models/blocks/block_elements.py new file mode 100644 index 0000000..890d835 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/models/blocks/block_elements.py @@ -0,0 +1,1482 @@ +import copy +import logging +import re +import warnings +from abc import ABCMeta +from typing import List, Optional, Set, Union, Sequence + +from slack_sdk.models import show_unknown_key_warning +from slack_sdk.models.basic_objects import ( + JsonObject, + JsonValidator, + EnumValidator, +) +from .basic_components import ButtonStyles +from .basic_components import ConfirmObject +from .basic_components import DispatchActionConfig +from .basic_components import MarkdownTextObject +from .basic_components import Option +from .basic_components import OptionGroup +from .basic_components import PlainTextObject +from .basic_components import TextObject + + +# ------------------------------------------------- +# Block Elements +# ------------------------------------------------- + + +class BlockElement(JsonObject, metaclass=ABCMeta): + """Block Elements are things that exists inside of your Blocks. + https://api.slack.com/reference/block-kit/block-elements + """ + + attributes = {"type"} + logger = logging.getLogger(__name__) + + def _subtype_warning(self): # skipcq: PYL-R0201 + warnings.warn( + "subtype is deprecated since slackclient 2.6.0, use type instead", + DeprecationWarning, + ) + + @property + def subtype(self) -> Optional[str]: + return self.type + + def __init__( + self, + *, + type: Optional[str] = None, # skipcq: PYL-W0622 + subtype: Optional[str] = None, + **others: dict, + ): + if subtype: + self._subtype_warning() + self.type = type if type else subtype + show_unknown_key_warning(self, others) + + @classmethod + def parse(cls, block_element: Union[dict, "BlockElement"]) -> Optional[Union["BlockElement", TextObject]]: + if block_element is None: # skipcq: PYL-R1705 + return None + elif isinstance(block_element, dict): + if "type" in block_element: + d = copy.copy(block_element) + t = d.pop("type") + if t == PlainTextObject.type: # skipcq: PYL-R1705 + return PlainTextObject(**d) + elif t == MarkdownTextObject.type: + return MarkdownTextObject(**d) + elif t == ImageElement.type: + return ImageElement(**d) + elif t == ButtonElement.type: + return ButtonElement(**d) + elif t == StaticSelectElement.type: + return StaticSelectElement(**d) + elif t == StaticMultiSelectElement.type: + return StaticMultiSelectElement(**d) + elif t == ExternalDataSelectElement.type: + return ExternalDataSelectElement(**d) + elif t == ExternalDataMultiSelectElement.type: + return ExternalDataMultiSelectElement(**d) + elif t == UserSelectElement.type: + return UserSelectElement(**d) + elif t == UserMultiSelectElement.type: + return UserMultiSelectElement(**d) + elif t == ConversationSelectElement.type: + return ConversationSelectElement(**d) + elif t == ConversationMultiSelectElement.type: + return ConversationMultiSelectElement(**d) + elif t == ChannelSelectElement.type: + return ChannelSelectElement(**d) + elif t == ChannelMultiSelectElement.type: + return ChannelMultiSelectElement(**d) + elif t == PlainTextInputElement.type: + return PlainTextInputElement(**d) + elif t == RadioButtonsElement.type: + return RadioButtonsElement(**d) + elif t == CheckboxesElement.type: + return CheckboxesElement(**d) + elif t == OverflowMenuElement.type: + return OverflowMenuElement(**d) + elif t == DatePickerElement.type: + return DatePickerElement(**d) + elif t == TimePickerElement.type: + return TimePickerElement(**d) + else: + cls.logger.warning(f"Unknown element detected and skipped ({block_element})") + return None + else: + cls.logger.warning(f"Unknown element detected and skipped ({block_element})") + return None + elif isinstance(block_element, (TextObject, BlockElement)): + return block_element + else: + cls.logger.warning(f"Unknown element detected and skipped ({block_element})") + return None + + @classmethod + def parse_all(cls, block_elements: Sequence[Union[dict, "BlockElement"]]) -> List["BlockElement"]: + return [cls.parse(e) for e in block_elements or []] + + +# ------------------------------------------------- +# Interactive Block Elements +# ------------------------------------------------- + +# This is a base class +class InteractiveElement(BlockElement): + action_id_max_length = 255 + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"alt_text", "action_id"}) + + def __init__( + self, + *, + action_id: Optional[str] = None, + type: Optional[str] = None, # skipcq: PYL-W0622 + subtype: Optional[str] = None, + **others: dict, + ): + """An interactive block element. + + We generally recommend using the concrete subclasses for better supports of available properties. + """ + if subtype: + self._subtype_warning() + super().__init__(type=type or subtype) + + # Note that we don't intentionally have show_unknown_key_warning for the unknown key warnings here. + # It's fine to pass any kwargs to the held dict here although the class does not do any validation. + # show_unknown_key_warning(self, others) + + self.action_id = action_id + + @JsonValidator(f"action_id attribute cannot exceed {action_id_max_length} characters") + def _validate_action_id_length(self) -> bool: + return self.action_id is None or len(self.action_id) <= self.action_id_max_length + + +# This is a base class +class InputInteractiveElement(InteractiveElement, metaclass=ABCMeta): + placeholder_max_length = 150 + + attributes = {"type", "action_id", "placeholder", "confirm", "focus_on_load"} + + @property + def subtype(self) -> Optional[str]: + return self.type + + def __init__( + self, + *, + action_id: Optional[str] = None, + placeholder: Optional[Union[str, TextObject]] = None, + type: Optional[str] = None, # skipcq: PYL-W0622 + subtype: Optional[str] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """InteractiveElement that is usable in input blocks + + We generally recommend using the concrete subclasses for better supports of available properties. + """ + if subtype: + self._subtype_warning() + super().__init__(action_id=action_id, type=type or subtype) + + # Note that we don't intentionally have show_unknown_key_warning for the unknown key warnings here. + # It's fine to pass any kwargs to the held dict here although the class does not do any validation. + # show_unknown_key_warning(self, others) + + self.placeholder = TextObject.parse(placeholder) + self.confirm = ConfirmObject.parse(confirm) + self.focus_on_load = focus_on_load + + @JsonValidator(f"placeholder attribute cannot exceed {placeholder_max_length} characters") + def _validate_placeholder_length(self) -> bool: + return ( + self.placeholder is None + or self.placeholder.text is None + or len(self.placeholder.text) <= self.placeholder_max_length + ) + + +# ------------------------------------------------- +# Button +# ------------------------------------------------- + + +class ButtonElement(InteractiveElement): + type = "button" + text_max_length = 75 + url_max_length = 3000 + value_max_length = 2000 + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"text", "url", "value", "style", "confirm", "accessibility_label"}) + + def __init__( + self, + *, + text: Union[str, dict, TextObject], + action_id: Optional[str] = None, + url: Optional[str] = None, + value: Optional[str] = None, + style: Optional[str] = None, # primary, danger + confirm: Optional[Union[dict, ConfirmObject]] = None, + accessibility_label: Optional[str] = None, + **others: dict, + ): + """An interactive element that inserts a button. The button can be a trigger for + anything from opening a simple link to starting a complex workflow. + https://api.slack.com/reference/block-kit/block-elements#button + + Args: + text (required): A text object that defines the button's text. + Can only be of type: plain_text. + Maximum length for the text in this field is 75 characters. + action_id (required): An identifier for this action. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + url: A URL to load in the user's browser when the button is clicked. + Maximum length for this field is 3000 characters. + If you're using url, you'll still receive an interaction payload + and will need to send an acknowledgement response. + value: The value to send along with the interaction payload. + Maximum length for this field is 2000 characters. + style: Decorates buttons with alternative visual color schemes. Use this option with restraint. + "primary" gives buttons a green outline and text, ideal for affirmation or confirmation actions. + "primary" should only be used for one button within a set. + "danger" gives buttons a red outline and text, and should be used when the action is destructive. + Use "danger" even more sparingly than "primary". + If you don't include this field, the default button style will be used. + confirm: A confirm object that defines an optional confirmation dialog after the button is clicked. + accessibility_label: A label for longer descriptive text about a button element. + This label will be read out by screen readers instead of the button text object. + Maximum length for this field is 75 characters. + """ + super().__init__(action_id=action_id, type=self.type) + show_unknown_key_warning(self, others) + + # NOTE: default_type=PlainTextObject.type here is only for backward-compatibility with version 2.5.0 + self.text = TextObject.parse(text, default_type=PlainTextObject.type) + self.url = url + self.value = value + self.style = style + self.confirm = ConfirmObject.parse(confirm) + self.accessibility_label = accessibility_label + + @JsonValidator(f"text attribute cannot exceed {text_max_length} characters") + def _validate_text_length(self) -> bool: + return self.text is None or self.text.text is None or len(self.text.text) <= self.text_max_length + + @JsonValidator(f"url attribute cannot exceed {url_max_length} characters") + def _validate_url_length(self) -> bool: + return self.url is None or len(self.url) <= self.url_max_length + + @JsonValidator(f"value attribute cannot exceed {value_max_length} characters") + def _validate_value_length(self) -> bool: + return self.value is None or len(self.value) <= self.value_max_length + + @EnumValidator("style", ButtonStyles) + def _validate_style_valid(self): + return self.style is None or self.style in ButtonStyles + + @JsonValidator(f"accessibility_label attribute cannot exceed {text_max_length} characters") + def _validate_accessibility_label_length(self) -> bool: + return self.accessibility_label is None or len(self.accessibility_label) <= self.text_max_length + + +class LinkButtonElement(ButtonElement): + def __init__( + self, + *, + text: Union[str, dict, PlainTextObject], + url: str, + action_id: Optional[str] = None, + style: Optional[str] = None, + **others: dict, + ): + """A simple button that simply opens a given URL. You will still receive an + interaction payload and will need to send an acknowledgement response. + This is a helper class that makes creating links simpler. + https://api.slack.com/reference/block-kit/block-elements#button + + Args: + text (required): A text object that defines the button's text. + Can only be of type: plain_text. + Maximum length for the text in this field is 75 characters. + url (required): A URL to load in the user's browser when the button is clicked. + Maximum length for this field is 3000 characters. + If you're using url, you'll still receive an interaction payload + and will need to send an acknowledgement response. + action_id (required): An identifier for this action. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + style: Decorates buttons with alternative visual color schemes. Use this option with restraint. + "primary" gives buttons a green outline and text, ideal for affirmation or confirmation actions. + "primary" should only be used for one button within a set. + "danger" gives buttons a red outline and text, and should be used when the action is destructive. + Use "danger" even more sparingly than "primary". + If you don't include this field, the default button style will be used. + """ + super().__init__( + # NOTE: value must be always absent + text=text, + url=url, + action_id=action_id, + value=None, + style=style, + ) + show_unknown_key_warning(self, others) + + +# ------------------------------------------------- +# Checkboxes +# ------------------------------------------------- + + +class CheckboxesElement(InputInteractiveElement): + type = "checkboxes" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"options", "initial_options"}) + + def __init__( + self, + *, + action_id: Optional[str] = None, + options: Optional[Sequence[Union[dict, Option]]] = None, + initial_options: Optional[Sequence[Union[dict, Option]]] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """A checkbox group that allows a user to choose multiple items from a list of possible options. + https://api.slack.com/reference/block-kit/block-elements#checkboxes + + Args: + action_id (required): An identifier for the action triggered when the checkbox group is changed. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + options (required): An array of option objects. A maximum of 10 options are allowed. + initial_options: An array of option objects that exactly matches one or more of the options. + These options will be selected when the checkbox group initially loads. + confirm: A confirm object that defines an optional confirmation dialog that appears + after clicking one of the checkboxes in this element. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.options = Option.parse_all(options) + self.initial_options = Option.parse_all(initial_options) + + +# ------------------------------------------------- +# DatePicker +# ------------------------------------------------- + + +class DatePickerElement(InputInteractiveElement): + type = "datepicker" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"initial_date"}) + + def __init__( + self, + *, + action_id: Optional[str] = None, + placeholder: Optional[Union[str, dict, TextObject]] = None, + initial_date: Optional[str] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """ + An element which lets users easily select a date from a calendar style UI. + Date picker elements can be used inside of SectionBlocks and ActionsBlocks. + https://api.slack.com/reference/block-kit/block-elements#datepicker + + Args: + action_id (required): An identifier for the action triggered when a menu option is selected. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + placeholder: A plain_text only text object that defines the placeholder text shown on the datepicker. + Maximum length for the text in this field is 150 characters. + initial_date: The initial date that is selected when the element is loaded. + This should be in the format YYYY-MM-DD. + confirm: A confirm object that defines an optional confirmation dialog + that appears after a date is selected. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + placeholder=TextObject.parse(placeholder, PlainTextObject.type), + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.initial_date = initial_date + + @JsonValidator("initial_date attribute must be in format 'YYYY-MM-DD'") + def _validate_initial_date_valid(self) -> bool: + return ( + self.initial_date is None + or re.match(r"\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])", self.initial_date) is not None + ) + + +# ------------------------------------------------- +# TimePicker +# ------------------------------------------------- + + +class TimePickerElement(InputInteractiveElement): + type = "timepicker" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"initial_time"}) + + def __init__( + self, + *, + action_id: Optional[str] = None, + placeholder: Optional[Union[str, dict, TextObject]] = None, + initial_time: Optional[str] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """ + An element which allows selection of a time of day. + On desktop clients, this time picker will take the form of a dropdown list + with free-text entry for precise choices. + On mobile clients, the time picker will use native time picker UIs. + https://api.slack.com/reference/block-kit/block-elements#timepicker + + Args: + action_id (required): An identifier for the action triggered when a time is selected. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + placeholder: A plain_text only text object that defines the placeholder text shown on the timepicker. + Maximum length for the text in this field is 150 characters. + initial_time: The initial time that is selected when the element is loaded. + This should be in the format HH:mm, where HH is the 24-hour format of an hour (00 to 23) + and mm is minutes with leading zeros (00 to 59), for example 22:25 for 10:25pm. + confirm: A confirm object that defines an optional confirmation dialog + that appears after a time is selected. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + placeholder=TextObject.parse(placeholder, PlainTextObject.type), + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.initial_time = initial_time + + @JsonValidator("initial_time attribute must be in format 'HH:mm'") + def _validate_initial_time_valid(self) -> bool: + return self.initial_time is None or re.match(r"([0-1][0-9]|2[0-3]):([0-5][0-9])", self.initial_time) is not None + + +# ------------------------------------------------- +# Image +# ------------------------------------------------- + + +class ImageElement(BlockElement): + type = "image" + image_url_max_length = 3000 + alt_text_max_length = 2000 + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"alt_text", "image_url"}) + + def __init__( + self, + *, + image_url: Optional[str] = None, + alt_text: Optional[str] = None, + **others: dict, + ): + """An element to insert an image - this element can be used in section and + context blocks only. If you want a block with only an image in it, + you're looking for the image block. + https://api.slack.com/reference/block-kit/block-elements#image + + Args: + image_url (required): The URL of the image to be displayed. + alt_text (required): A plain-text summary of the image. This should not contain any markup. + """ + super().__init__(type=self.type) + show_unknown_key_warning(self, others) + + self.image_url = image_url + self.alt_text = alt_text + + @JsonValidator(f"image_url attribute cannot exceed {image_url_max_length} characters") + def _validate_image_url_length(self) -> bool: + return len(self.image_url) <= self.image_url_max_length + + @JsonValidator(f"alt_text attribute cannot exceed {alt_text_max_length} characters") + def _validate_alt_text_length(self) -> bool: + return len(self.alt_text) <= self.alt_text_max_length + + +# ------------------------------------------------- +# Static Select +# ------------------------------------------------- + + +class StaticSelectElement(InputInteractiveElement): + type = "static_select" + options_max_length = 100 + option_groups_max_length = 100 + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"options", "option_groups", "initial_option"}) + + def __init__( + self, + *, + placeholder: Optional[Union[str, dict, TextObject]] = None, + action_id: Optional[str] = None, + options: Optional[Sequence[Union[dict, Option]]] = None, + option_groups: Optional[Sequence[Union[dict, OptionGroup]]] = None, + initial_option: Optional[Union[dict, Option]] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """This is the simplest form of select menu, with a static list of options passed in when defining the element. + https://api.slack.com/reference/block-kit/block-elements#static_select + + Args: + placeholder (required): A plain_text only text object that defines the placeholder text shown on the menu. + Maximum length for the text in this field is 150 characters. + action_id (required): An identifier for the action triggered when a menu option is selected. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + options (either options or option_groups is required): An array of option objects. + Maximum number of options is 100. + If option_groups is specified, this field should not be. + option_groups (either options or option_groups is required): An array of option group objects. + Maximum number of option groups is 100. + If options is specified, this field should not be. + initial_option: A single option that exactly matches one of the options or option_groups. + This option will be selected when the menu initially loads. + confirm: A confirm object that defines an optional confirmation dialog + that appears after a menu item is selected. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + placeholder=TextObject.parse(placeholder, PlainTextObject.type), + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.options = options + self.option_groups = option_groups + self.initial_option = initial_option + + @JsonValidator(f"options attribute cannot exceed {options_max_length} elements") + def _validate_options_length(self) -> bool: + return self.options is None or len(self.options) <= self.options_max_length + + @JsonValidator(f"option_groups attribute cannot exceed {option_groups_max_length} elements") + def _validate_option_groups_length(self) -> bool: + return self.option_groups is None or len(self.option_groups) <= self.option_groups_max_length + + @JsonValidator("options and option_groups cannot both be specified") + def _validate_options_and_option_groups_both_specified(self) -> bool: + return not (self.options is not None and self.option_groups is not None) + + @JsonValidator("options or option_groups must be specified") + def _validate_neither_options_or_option_groups_is_specified(self) -> bool: + return self.options is not None or self.option_groups is not None + + +class StaticMultiSelectElement(InputInteractiveElement): + type = "multi_static_select" + options_max_length = 100 + option_groups_max_length = 100 + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"options", "option_groups", "initial_options", "max_selected_items"}) + + def __init__( + self, + *, + placeholder: Optional[Union[str, dict, TextObject]] = None, + action_id: Optional[str] = None, + options: Optional[Sequence[Option]] = None, + option_groups: Optional[Sequence[OptionGroup]] = None, + initial_options: Optional[Sequence[Option]] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + max_selected_items: Optional[int] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """ + This is the simplest form of select menu, with a static list of options passed in when defining the element. + https://api.slack.com/reference/block-kit/block-elements#static_multi_select + + Args: + placeholder (required): A plain_text only text object that defines the placeholder text shown on the menu. + Maximum length for the text in this field is 150 characters. + action_id (required): An identifier for the action triggered when a menu option is selected. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + options (either options or option_groups is required): An array of option objects. + Maximum number of options is 100. + If option_groups is specified, this field should not be. + option_groups (either options or option_groups is required): An array of option group objects. + Maximum number of option groups is 100. + If options is specified, this field should not be. + initial_options: An array of option objects that exactly match one or more of the options + within options or option_groups. These options will be selected when the menu initially loads. + confirm: A confirm object that defines an optional confirmation dialog + that appears before the multi-select choices are submitted. + max_selected_items: Specifies the maximum number of items that can be selected in the menu. + Minimum number is 1. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + placeholder=TextObject.parse(placeholder, PlainTextObject.type), + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.options = Option.parse_all(options) + self.option_groups = OptionGroup.parse_all(option_groups) + self.initial_options = Option.parse_all(initial_options) + self.max_selected_items = max_selected_items + + @JsonValidator(f"options attribute cannot exceed {options_max_length} elements") + def _validate_options_length(self) -> bool: + return self.options is None or len(self.options) <= self.options_max_length + + @JsonValidator(f"option_groups attribute cannot exceed {option_groups_max_length} elements") + def _validate_option_groups_length(self) -> bool: + return self.option_groups is None or len(self.option_groups) <= self.option_groups_max_length + + @JsonValidator("options and option_groups cannot both be specified") + def _validate_options_and_option_groups_both_specified(self) -> bool: + return self.options is None or self.option_groups is None + + @JsonValidator("options or option_groups must be specified") + def _validate_neither_options_or_option_groups_is_specified(self) -> bool: + return self.options is not None or self.option_groups is not None + + +# SelectElement will be deprecated in version 3, use StaticSelectElement instead +class SelectElement(InputInteractiveElement): + type = "static_select" + options_max_length = 100 + option_groups_max_length = 100 + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"options", "option_groups", "initial_option"}) + + def __init__( + self, + *, + action_id: Optional[str] = None, + placeholder: Optional[str] = None, + options: Optional[Sequence[Option]] = None, + option_groups: Optional[Sequence[OptionGroup]] = None, + initial_option: Optional[Option] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """This is the simplest form of select menu, with a static list of options passed in when defining the element. + https://api.slack.com/reference/block-kit/block-elements#static_select + + Args: + action_id (required): An identifier for the action triggered when a menu option is selected. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + placeholder (required): A plain_text only text object that defines the placeholder text shown on the menu. + Maximum length for the text in this field is 150 characters. + options (either options or option_groups is required): An array of option objects. + Maximum number of options is 100. + If option_groups is specified, this field should not be. + option_groups (either options or option_groups is required): An array of option group objects. + Maximum number of option groups is 100. + If options is specified, this field should not be. + initial_option: A single option that exactly matches one of the options or option_groups. + This option will be selected when the menu initially loads. + confirm: A confirm object that defines an optional confirmation dialog + that appears after a menu item is selected. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + placeholder=TextObject.parse(placeholder, PlainTextObject.type), + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.options = options + self.option_groups = option_groups + self.initial_option = initial_option + + @JsonValidator(f"options attribute cannot exceed {options_max_length} elements") + def _validate_options_length(self) -> bool: + return self.options is None or len(self.options) <= self.options_max_length + + @JsonValidator(f"option_groups attribute cannot exceed {option_groups_max_length} elements") + def _validate_option_groups_length(self) -> bool: + return self.option_groups is None or len(self.option_groups) <= self.option_groups_max_length + + @JsonValidator("options and option_groups cannot both be specified") + def _validate_options_and_option_groups_both_specified(self) -> bool: + return not (self.options is not None and self.option_groups is not None) + + @JsonValidator("options or option_groups must be specified") + def _validate_neither_options_or_option_groups_is_specified(self) -> bool: + return self.options is not None or self.option_groups is not None + + +# ------------------------------------------------- +# External Data Source Select +# ------------------------------------------------- + + +class ExternalDataSelectElement(InputInteractiveElement): + type = "external_select" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"min_query_length", "initial_option"}) + + def __init__( + self, + *, + action_id: Optional[str] = None, + placeholder: Optional[Union[str, TextObject]] = None, + initial_option: Union[Optional[Option], Optional[OptionGroup]] = None, + min_query_length: Optional[int] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """ + This select menu will load its options from an external data source, allowing + for a dynamic list of options. + https://api.slack.com/reference/block-kit/block-elements#external_select + + Args: + action_id (required): An identifier for the action triggered when a menu option is selected. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + placeholder (required): A plain_text only text object that defines the placeholder text shown on the menu. + Maximum length for the text in this field is 150 characters. + initial_option: A single option that exactly matches one of the options + within the options or option_groups loaded from the external data source. + This option will be selected when the menu initially loads. + min_query_length: When the typeahead field is used, a request will be sent on every character change. + If you prefer fewer requests or more fully ideated queries, + use the min_query_length attribute to tell Slack + the fewest number of typed characters required before dispatch. + The default value is 3. + confirm: A confirm object that defines an optional confirmation dialog + that appears after a menu item is selected. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + placeholder=TextObject.parse(placeholder, PlainTextObject.type), + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.min_query_length = min_query_length + self.initial_option = initial_option + + +class ExternalDataMultiSelectElement(InputInteractiveElement): + type = "multi_external_select" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"min_query_length", "initial_options", "max_selected_items"}) + + def __init__( + self, + *, + placeholder: Optional[Union[str, dict, TextObject]] = None, + action_id: Optional[str] = None, + min_query_length: Optional[int] = None, + initial_options: Optional[Sequence[Union[dict, Option]]] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + max_selected_items: Optional[int] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """ + This select menu will load its options from an external data source, allowing + for a dynamic list of options. + https://api.slack.com/reference/block-kit/block-elements#external_multi_select + + Args: + placeholder (required): A plain_text only text object that defines the placeholder text shown on the menu. + Maximum length for the text in this field is 150 characters. + action_id (required): An identifier for the action triggered when a menu option is selected. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + min_query_length: When the typeahead field is used, a request will be sent on every character change. + If you prefer fewer requests or more fully ideated queries, + use the min_query_length attribute to tell Slack + the fewest number of typed characters required before dispatch. + The default value is 3 + initial_options: An array of option objects that exactly match one or more of the options + within options or option_groups. These options will be selected when the menu initially loads. + confirm: A confirm object that defines an optional confirmation dialog that appears + before the multi-select choices are submitted. + max_selected_items: Specifies the maximum number of items that can be selected in the menu. + Minimum number is 1. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + placeholder=TextObject.parse(placeholder, PlainTextObject.type), + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.min_query_length = min_query_length + self.initial_options = Option.parse_all(initial_options) + self.max_selected_items = max_selected_items + + +# ------------------------------------------------- +# Users Select +# ------------------------------------------------- + + +class UserSelectElement(InputInteractiveElement): + type = "users_select" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"initial_user"}) + + def __init__( + self, + *, + placeholder: Optional[Union[str, dict, TextObject]] = None, + action_id: Optional[str] = None, + initial_user: Optional[str] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """ + This select menu will populate its options with a list of Slack users visible to + the current user in the active workspace. + https://api.slack.com/reference/block-kit/block-elements#users_select + + Args: + placeholder (required): A plain_text only text object that defines the placeholder text shown on the menu. + Maximum length for the text in this field is 150 characters. + action_id (required): An identifier for the action triggered when a menu option is selected. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + initial_user: The user ID of any valid user to be pre-selected when the menu loads. + confirm: A confirm object that defines an optional confirmation dialog + that appears after a menu item is selected. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + placeholder=TextObject.parse(placeholder, PlainTextObject.type), + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.initial_user = initial_user + + +class UserMultiSelectElement(InputInteractiveElement): + type = "multi_users_select" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"initial_users", "max_selected_items"}) + + def __init__( + self, + *, + action_id: Optional[str] = None, + placeholder: Optional[Union[str, dict, TextObject]] = None, + initial_users: Optional[Sequence[str]] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + max_selected_items: Optional[int] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """ + This select menu will populate its options with a list of Slack users visible to + the current user in the active workspace. + https://api.slack.com/reference/block-kit/block-elements#users_multi_select + + Args: + action_id (required): An identifier for the action triggered when a menu option is selected. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + placeholder (required): A plain_text only text object that defines the placeholder text shown on the menu. + Maximum length for the text in this field is 150 characters. + initial_users: An array of user IDs of any valid users to be pre-selected when the menu loads. + confirm: A confirm object that defines an optional confirmation dialog that appears + before the multi-select choices are submitted. + max_selected_items: Specifies the maximum number of items that can be selected in the menu. + Minimum number is 1. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + placeholder=TextObject.parse(placeholder, PlainTextObject.type), + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.initial_users = initial_users + self.max_selected_items = max_selected_items + + +# ------------------------------------------------- +# Conversations Select +# ------------------------------------------------- + + +class ConversationFilter(JsonObject): + attributes = {"include", "exclude_bot_users", "exclude_external_shared_channels"} + logger = logging.getLogger(__name__) + + def __init__( + self, + *, + include: Optional[Sequence[str]] = None, + exclude_bot_users: Optional[bool] = None, + exclude_external_shared_channels: Optional[bool] = None, + ): + """Provides a way to filter the list of options in a conversations select menu + or conversations multi-select menu. + https://api.slack.com/reference/block-kit/composition-objects#filter_conversations + + Args: + include: Indicates which type of conversations should be included in the list. + When this field is provided, any conversations that do not match will be excluded. + You should provide an array of strings from the following options: + "im", "mpim", "private", and "public". The array cannot be empty. + exclude_bot_users: Indicates whether to exclude bot users from conversation lists. Defaults to false. + exclude_external_shared_channels: Indicates whether to exclude external shared channels + from conversation lists. Defaults to false. + """ + self.include = include + self.exclude_bot_users = exclude_bot_users + self.exclude_external_shared_channels = exclude_external_shared_channels + + @classmethod + def parse(cls, filter: Union[dict, "ConversationFilter"]): # skipcq: PYL-W0622 + if filter is None: # skipcq: PYL-R1705 + return None + elif isinstance(filter, ConversationFilter): + return filter + elif isinstance(filter, dict): + d = copy.copy(filter) + return ConversationFilter(**d) + else: + cls.logger.warning(f"Unknown conversation filter object detected and skipped ({filter})") + return None + + +class ConversationSelectElement(InputInteractiveElement): + type = "conversations_select" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union( + { + "initial_conversation", + "response_url_enabled", + "filter", + "default_to_current_conversation", + } + ) + + def __init__( + self, + *, + placeholder: Optional[Union[str, dict, TextObject]] = None, + action_id: Optional[str] = None, + initial_conversation: Optional[str] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + response_url_enabled: Optional[bool] = None, + default_to_current_conversation: Optional[bool] = None, + filter: Optional[ConversationFilter] = None, # skipcq: PYL-W0622 + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """ + This select menu will populate its options with a list of public and private + channels, DMs, and MPIMs visible to the current user in the active workspace. + https://api.slack.com/reference/block-kit/block-elements#conversation_select + + Args: + placeholder (required): A plain_text only text object that defines the placeholder text shown on the menu. + Maximum length for the text in this field is 150 characters. + action_id (required): An identifier for the action triggered when a menu option is selected. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + initial_conversation: The ID of any valid conversation to be pre-selected when the menu loads. + If default_to_current_conversation is also supplied, initial_conversation will take precedence. + confirm: A confirm object that defines an optional confirmation dialog + that appears after a menu item is selected. + response_url_enabled: This field only works with menus in input blocks in modals. + When set to true, the view_submission payload from the menu's parent view will contain a response_url. + This response_url can be used for message responses. The target conversation for the message + will be determined by the value of this select menu. + default_to_current_conversation: Pre-populates the select menu with the conversation + that the user was viewing when they opened the modal, if available. Default is false. + filter: A filter object that reduces the list of available conversations using the specified criteria. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + placeholder=TextObject.parse(placeholder, PlainTextObject.type), + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.initial_conversation = initial_conversation + self.response_url_enabled = response_url_enabled + self.default_to_current_conversation = default_to_current_conversation + self.filter = filter + + +class ConversationMultiSelectElement(InputInteractiveElement): + type = "multi_conversations_select" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union( + { + "initial_conversations", + "max_selected_items", + "default_to_current_conversation", + "filter", + } + ) + + def __init__( + self, + *, + placeholder: Optional[Union[str, dict, TextObject]] = None, + action_id: Optional[str] = None, + initial_conversations: Optional[Sequence[str]] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + max_selected_items: Optional[int] = None, + default_to_current_conversation: Optional[bool] = None, + filter: Optional[Union[dict, ConversationFilter]] = None, # skipcq: PYL-W0622 + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """ + This multi-select menu will populate its options with a list of public and private channels, + DMs, and MPIMs visible to the current user in the active workspace. + https://api.slack.com/reference/block-kit/block-elements#conversation_multi_select + + Args: + placeholder (required): A plain_text only text object that defines the placeholder text shown on the menu. + Maximum length for the text in this field is 150 characters. + action_id (required): An identifier for the action triggered when a menu option is selected. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + initial_conversations: An array of one or more IDs of any valid conversations to be pre-selected + when the menu loads. If default_to_current_conversation is also supplied, + initial_conversations will be ignored. + confirm: A confirm object that defines an optional confirmation dialog that appears + before the multi-select choices are submitted. + max_selected_items: Specifies the maximum number of items that can be selected in the menu. + Minimum number is 1. + default_to_current_conversation: Pre-populates the select menu with the conversation that + the user was viewing when they opened the modal, if available. Default is false. + filter: A filter object that reduces the list of available conversations using the specified criteria. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + placeholder=TextObject.parse(placeholder, PlainTextObject.type), + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.initial_conversations = initial_conversations + self.max_selected_items = max_selected_items + self.default_to_current_conversation = default_to_current_conversation + self.filter = ConversationFilter.parse(filter) + + +# ------------------------------------------------- +# Channels Select +# ------------------------------------------------- + + +class ChannelSelectElement(InputInteractiveElement): + type = "channels_select" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"initial_channel", "response_url_enabled"}) + + def __init__( + self, + *, + placeholder: Optional[Union[str, dict, TextObject]] = None, + action_id: Optional[str] = None, + initial_channel: Optional[str] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + response_url_enabled: Optional[bool] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """ + This select menu will populate its options with a list of public channels + visible to the current user in the active workspace. + https://api.slack.com/reference/block-kit/block-elements#channel_select + + Args: + placeholder (required): A plain_text only text object that defines the placeholder text shown on the menu. + Maximum length for the text in this field is 150 characters. + action_id (required): An identifier for the action triggered when a menu option is selected. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + initial_channel: The ID of any valid public channel to be pre-selected when the menu loads. + confirm: A confirm object that defines an optional confirmation dialog that appears + after a menu item is selected. + response_url_enabled: This field only works with menus in input blocks in modals. + When set to true, the view_submission payload from the menu's parent view will contain a response_url. + This response_url can be used for message responses. + The target channel for the message will be determined by the value of this select menu + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + placeholder=TextObject.parse(placeholder, PlainTextObject.type), + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.initial_channel = initial_channel + self.response_url_enabled = response_url_enabled + + +class ChannelMultiSelectElement(InputInteractiveElement): + type = "multi_channels_select" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"initial_channels", "max_selected_items"}) + + def __init__( + self, + *, + placeholder: Optional[Union[str, dict, TextObject]] = None, + action_id: Optional[str] = None, + initial_channels: Optional[Sequence[str]] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + max_selected_items: Optional[int] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """ + This multi-select menu will populate its options with a list of public channels visible + to the current user in the active workspace. + https://api.slack.com/reference/block-kit/block-elements#channel_multi_select + + Args: + placeholder (required): A plain_text only text object that defines the placeholder text shown on the menu. + Maximum length for the text in this field is 150 characters. + action_id (required): An identifier for the action triggered when a menu option is selected. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + initial_channels: An array of one or more IDs of any valid public channel + to be pre-selected when the menu loads. + confirm: A confirm object that defines an optional confirmation dialog that appears + before the multi-select choices are submitted. + max_selected_items: Specifies the maximum number of items that can be selected in the menu. + Minimum number is 1. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + placeholder=TextObject.parse(placeholder, PlainTextObject.type), + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.initial_channels = initial_channels + self.max_selected_items = max_selected_items + + +# ------------------------------------------------- +# Input Elements +# ------------------------------------------------- + + +class PlainTextInputElement(InputInteractiveElement): + type = "plain_text_input" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union( + { + "initial_value", + "multiline", + "min_length", + "max_length", + "dispatch_action_config", + } + ) + + def __init__( + self, + *, + action_id: Optional[str] = None, + placeholder: Optional[Union[str, dict, TextObject]] = None, + initial_value: Optional[str] = None, + multiline: Optional[bool] = None, + min_length: Optional[int] = None, + max_length: Optional[int] = None, + dispatch_action_config: Optional[Union[dict, DispatchActionConfig]] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """ + A plain-text input, similar to the HTML tag, creates a field + where a user can enter freeform data. It can appear as a single-line + field or a larger textarea using the multiline flag. Plain-text input + elements can be used inside of SectionBlocks and ActionsBlocks. + https://api.slack.com/reference/block-kit/block-elements#input + + Args: + action_id (required): An identifier for the input value when the parent modal is submitted. + You can use this when you receive a view_submission payload to identify the value of the input element. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + placeholder: A plain_text only text object that defines the placeholder text shown + in the plain-text input. Maximum length for the text in this field is 150 characters. + initial_value: The initial value in the plain-text input when it is loaded. + multiline: Indicates whether the input will be a single line (false) or a larger textarea (true). + Defaults to false. + min_length: The minimum length of input that the user must provide. If the user provides less, + they will receive an error. Maximum value is 3000. + max_length: The maximum length of input that the user can provide. If the user provides more, + they will receive an error. + dispatch_action_config: A dispatch configuration object that determines when + during text input the element returns a block_actions payload. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + placeholder=TextObject.parse(placeholder, PlainTextObject.type), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.initial_value = initial_value + self.multiline = multiline + self.min_length = min_length + self.max_length = max_length + self.dispatch_action_config = dispatch_action_config + + +# ------------------------------------------------- +# Radio Buttons Select +# ------------------------------------------------- + + +class RadioButtonsElement(InputInteractiveElement): + type = "radio_buttons" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"options", "initial_option"}) + + def __init__( + self, + *, + action_id: Optional[str] = None, + options: Optional[Sequence[Union[dict, Option]]] = None, + initial_option: Optional[Union[dict, Option]] = None, + confirm: Optional[Union[dict, ConfirmObject]] = None, + focus_on_load: Optional[bool] = None, + **others: dict, + ): + """A radio button group that allows a user to choose one item from a list of possible options. + https://api.slack.com/reference/block-kit/block-elements#radio + + Args: + action_id (required): An identifier for the action triggered when the radio button group is changed. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + options (required): An array of option objects. A maximum of 10 options are allowed. + initial_option: An option object that exactly matches one of the options. + This option will be selected when the radio button group initially loads. + confirm: A confirm object that defines an optional confirmation dialog that appears + after clicking one of the radio buttons in this element. + focus_on_load: Indicates whether the element will be set to auto focus within the view object. + Only one element can be set to true. Defaults to false. + """ + super().__init__( + type=self.type, + action_id=action_id, + confirm=ConfirmObject.parse(confirm), + focus_on_load=focus_on_load, + ) + show_unknown_key_warning(self, others) + + self.options = options + self.initial_option = initial_option + + +# ------------------------------------------------- +# Overflow Menu Select +# ------------------------------------------------- + + +class OverflowMenuElement(InteractiveElement): + type = "overflow" + options_min_length = 1 + options_max_length = 5 + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"confirm", "options"}) + + def __init__( + self, + *, + action_id: Optional[str] = None, + options: Sequence[Option], + confirm: Optional[Union[dict, ConfirmObject]] = None, + **others: dict, + ): + """ + This is like a cross between a button and a select menu - when a user clicks + on this overflow button, they will be presented with a list of options to + choose from. Unlike the select menu, there is no typeahead field, and the + button always appears with an ellipsis ("…") rather than customisable text. + + As such, it is usually used if you want a more compact layout than a select + menu, or to supply a list of less visually important actions after a row of + buttons. You can also specify simple URL links as overflow menu options, + instead of actions. + + https://api.slack.com/reference/block-kit/block-elements#overflow + + Args: + action_id (required): An identifier for the action triggered when a menu option is selected. + You can use this when you receive an interaction payload to identify the source of the action. + Should be unique among all other action_ids in the containing block. + Maximum length for this field is 255 characters. + options (required): An array of option objects to display in the menu. + Maximum number of options is 5, minimum is 1. + confirm: A confirm object that defines an optional confirmation dialog that appears + after a menu item is selected. + """ + super().__init__(action_id=action_id, type=self.type) + show_unknown_key_warning(self, others) + + self.options = options + self.confirm = ConfirmObject.parse(confirm) + + @JsonValidator(f"options attribute must have between {options_min_length} " f"and {options_max_length} items") + def _validate_options_length(self) -> bool: + return self.options_min_length <= len(self.options) <= self.options_max_length diff --git a/core_service/aws_lambda/project/packages/slack_sdk/models/blocks/blocks.py b/core_service/aws_lambda/project/packages/slack_sdk/models/blocks/blocks.py new file mode 100644 index 0000000..fcbc502 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/models/blocks/blocks.py @@ -0,0 +1,494 @@ +import copy +import logging +import warnings +from typing import Dict, Sequence, Optional, Set, Union, Any, List + +from slack_sdk.models import show_unknown_key_warning +from slack_sdk.models.basic_objects import ( + JsonObject, + JsonValidator, +) +from .basic_components import MarkdownTextObject +from .basic_components import PlainTextObject +from .basic_components import TextObject +from .block_elements import BlockElement +from .block_elements import ImageElement +from .block_elements import InputInteractiveElement +from .block_elements import InteractiveElement + + +# ------------------------------------------------- +# Base Classes +# ------------------------------------------------- + + +class Block(JsonObject): + """Blocks are a series of components that can be combined + to create visually rich and compellingly interactive messages. + https://api.slack.com/reference/block-kit/blocks + """ + + attributes = {"block_id", "type"} + block_id_max_length = 255 + logger = logging.getLogger(__name__) + + def _subtype_warning(self): # skipcq: PYL-R0201 + warnings.warn( + "subtype is deprecated since slackclient 2.6.0, use type instead", + DeprecationWarning, + ) + + @property + def subtype(self) -> Optional[str]: + return self.type + + def __init__( + self, + *, + type: Optional[str] = None, # skipcq: PYL-W0622 + subtype: Optional[str] = None, # deprecated + block_id: Optional[str] = None, + ): + if subtype: + self._subtype_warning() + self.type = type if type else subtype + self.block_id = block_id + self.color = None + + @JsonValidator(f"block_id cannot exceed {block_id_max_length} characters") + def _validate_block_id_length(self): + return self.block_id is None or len(self.block_id) <= self.block_id_max_length + + @classmethod + def parse(cls, block: Union[dict, "Block"]) -> Optional["Block"]: + if block is None: # skipcq: PYL-R1705 + return None + elif isinstance(block, Block): + return block + else: + if "type" in block: + type = block["type"] # skipcq: PYL-W0622 + if type == SectionBlock.type: # skipcq: PYL-R1705 + return SectionBlock(**block) + elif type == DividerBlock.type: + return DividerBlock(**block) + elif type == ImageBlock.type: + return ImageBlock(**block) + elif type == ActionsBlock.type: + return ActionsBlock(**block) + elif type == ContextBlock.type: + return ContextBlock(**block) + elif type == InputBlock.type: + return InputBlock(**block) + elif type == FileBlock.type: + return FileBlock(**block) + elif type == CallBlock.type: + return CallBlock(**block) + elif type == HeaderBlock.type: + return HeaderBlock(**block) + else: + cls.logger.warning(f"Unknown block detected and skipped ({block})") + return None + else: + cls.logger.warning(f"Unknown block detected and skipped ({block})") + return None + + @classmethod + def parse_all(cls, blocks: Optional[Sequence[Union[dict, "Block"]]]) -> List["Block"]: + return [cls.parse(b) for b in blocks or []] + + +# ------------------------------------------------- +# Block Classes +# ------------------------------------------------- + + +class SectionBlock(Block): + type = "section" + fields_max_length = 10 + text_max_length = 3000 + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"text", "fields", "accessory"}) + + def __init__( + self, + *, + block_id: Optional[str] = None, + text: Optional[Union[str, dict, TextObject]] = None, + fields: Optional[Sequence[Union[str, dict, TextObject]]] = None, + accessory: Optional[Union[dict, BlockElement]] = None, + **others: dict, + ): + """A section is one of the most flexible blocks available. + https://api.slack.com/reference/block-kit/blocks#section + + Args: + block_id (required): A string acting as a unique identifier for a block. + If not specified, one will be generated. + You can use this block_id when you receive an interaction payload to identify the source of the action. + Maximum length for this field is 255 characters. + block_id should be unique for each message and each iteration of a message. + If a message is updated, use a new block_id. + text (preferred): The text for the block, in the form of a text object. + Maximum length for the text in this field is 3000 characters. + This field is not required if a valid array of fields objects is provided instead. + fields (required if no text is provided): Required if no text is provided. + An array of text objects. Any text objects included with fields will be rendered + in a compact format that allows for 2 columns of side-by-side text. + Maximum number of items is 10. Maximum length for the text in each item is 2000 characters. + accessory: One of the available element objects. + """ + super().__init__(type=self.type, block_id=block_id) + show_unknown_key_warning(self, others) + + self.text = TextObject.parse(text) + field_objects = [] + for f in fields or []: + if isinstance(f, str): + field_objects.append(MarkdownTextObject.from_str(f)) + elif isinstance(f, TextObject): + field_objects.append(f) + elif isinstance(f, dict) and "type" in f: + d = copy.copy(f) + t = d.pop("type") + if t == MarkdownTextObject.type: + field_objects.append(MarkdownTextObject(**d)) + else: + field_objects.append(PlainTextObject(**d)) + else: + self.logger.warning(f"Unsupported filed detected and skipped {f}") + self.fields = field_objects + self.accessory = BlockElement.parse(accessory) + + @JsonValidator("text or fields attribute must be specified") + def _validate_text_or_fields_populated(self): + return self.text is not None or self.fields + + @JsonValidator(f"fields attribute cannot exceed {fields_max_length} items") + def _validate_fields_length(self): + return self.fields is None or len(self.fields) <= self.fields_max_length + + @JsonValidator(f"text attribute cannot exceed {text_max_length} characters") + def _validate_alt_text_length(self): + return self.text is None or len(self.text.text) <= self.text_max_length + + +class DividerBlock(Block): + type = "divider" + + def __init__( + self, + *, + block_id: Optional[str] = None, + **others: dict, + ): + """A content divider, like an
, to split up different blocks inside of a message. + https://api.slack.com/reference/block-kit/blocks#divider + + Args: + block_id: A string acting as a unique identifier for a block. If not specified, one will be generated. + You can use this block_id when you receive an interaction payload to identify the source of the action. + Maximum length for this field is 255 characters. + block_id should be unique for each message and each iteration of a message. + If a message is updated, use a new block_id. + """ + super().__init__(type=self.type, block_id=block_id) + show_unknown_key_warning(self, others) + + +class ImageBlock(Block): + type = "image" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"alt_text", "image_url", "title"}) + + image_url_max_length = 3000 + alt_text_max_length = 2000 + title_max_length = 2000 + + def __init__( + self, + *, + image_url: str, + alt_text: str, + title: Optional[Union[str, dict, TextObject]] = None, + block_id: Optional[str] = None, + **others: dict, + ): + """A simple image block, designed to make those cat photos really pop. + https://api.slack.com/reference/block-kit/blocks#image + + Args: + image_url (required): The URL of the image to be displayed. + Maximum length for this field is 3000 characters. + alt_text (required): A plain-text summary of the image. This should not contain any markup. + Maximum length for this field is 2000 characters. + title: An optional title for the image in the form of a text object that can only be of type: plain_text. + Maximum length for the text in this field is 2000 characters. + block_id: A string acting as a unique identifier for a block. If not specified, one will be generated. + Maximum length for this field is 255 characters. + block_id should be unique for each message and each iteration of a message. + If a message is updated, use a new block_id. + """ + super().__init__(type=self.type, block_id=block_id) + show_unknown_key_warning(self, others) + + self.image_url = image_url + self.alt_text = alt_text + self.title = TextObject.parse(title) + + @JsonValidator(f"image_url attribute cannot exceed {image_url_max_length} characters") + def _validate_image_url_length(self): + return len(self.image_url) <= self.image_url_max_length + + @JsonValidator(f"alt_text attribute cannot exceed {alt_text_max_length} characters") + def _validate_alt_text_length(self): + return len(self.alt_text) <= self.alt_text_max_length + + @JsonValidator(f"title attribute cannot exceed {title_max_length} characters") + def _validate_title_length(self): + return self.title is None or self.title.text is None or len(self.title.text) <= self.title_max_length + + +class ActionsBlock(Block): + type = "actions" + elements_max_length = 5 + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"elements"}) + + def __init__( + self, + *, + elements: Sequence[Union[dict, InteractiveElement]], + block_id: Optional[str] = None, + **others: dict, + ): + """A block that is used to hold interactive elements. + https://api.slack.com/reference/block-kit/blocks#actions + + Args: + elements (required): An array of interactive element objects - buttons, select menus, overflow menus, + or date pickers. There is a maximum of 5 elements in each action block. + block_id: A string acting as a unique identifier for a block. + If not specified, a block_id will be generated. + You can use this block_id when you receive an interaction payload to identify the source of the action. + Maximum length for this field is 255 characters. + block_id should be unique for each message and each iteration of a message. + If a message is updated, use a new block_id. + """ + super().__init__(type=self.type, block_id=block_id) + show_unknown_key_warning(self, others) + + self.elements = BlockElement.parse_all(elements) + + @JsonValidator(f"elements attribute cannot exceed {elements_max_length} elements") + def _validate_elements_length(self): + return self.elements is None or len(self.elements) <= self.elements_max_length + + +class ContextBlock(Block): + type = "context" + elements_max_length = 10 + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"elements"}) + + def __init__( + self, + *, + elements: Sequence[Union[dict, ImageElement, TextObject]], + block_id: Optional[str] = None, + **others: dict, + ): + """Displays message context, which can include both images and text. + https://api.slack.com/reference/block-kit/blocks#context + + Args: + elements (required): An array of image elements and text objects. Maximum number of items is 10. + block_id: A string acting as a unique identifier for a block. If not specified, one will be generated. + Maximum length for this field is 255 characters. + block_id should be unique for each message and each iteration of a message. + If a message is updated, use a new block_id. + """ + super().__init__(type=self.type, block_id=block_id) + show_unknown_key_warning(self, others) + + self.elements = BlockElement.parse_all(elements) + + @JsonValidator(f"elements attribute cannot exceed {elements_max_length} elements") + def _validate_elements_length(self): + return self.elements is None or len(self.elements) <= self.elements_max_length + + +class InputBlock(Block): + type = "input" + label_max_length = 2000 + hint_max_length = 2000 + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"label", "hint", "element", "optional", "dispatch_action"}) + + def __init__( + self, + *, + label: Union[str, dict, PlainTextObject], + element: Union[str, dict, InputInteractiveElement], + block_id: Optional[str] = None, + hint: Optional[Union[str, dict, PlainTextObject]] = None, + dispatch_action: Optional[bool] = None, + optional: Optional[bool] = None, + **others: dict, + ): + """A block that collects information from users - it can hold a plain-text input element, + a select menu element, a multi-select menu element, or a datepicker. + https://api.slack.com/reference/block-kit/blocks#input + + Args: + label (required): A label that appears above an input element in the form of a text object + that must have type of plain_text. Maximum length for the text in this field is 2000 characters. + element (required): An plain-text input element, a checkbox element, a radio button element, + a select menu element, a multi-select menu element, or a datepicker. + block_id: A string acting as a unique identifier for a block. If not specified, one will be generated. + Maximum length for this field is 255 characters. + block_id should be unique for each message or view and each iteration of a message or view. + If a message or view is updated, use a new block_id. + hint: An optional hint that appears below an input element in a lighter grey. + It must be a text object with a type of plain_text. + Maximum length for the text in this field is 2000 characters. + dispatch_action: A boolean that indicates whether or not the use of elements in this block + should dispatch a block_actions payload. Defaults to false. + optional: A boolean that indicates whether the input element may be empty when a user submits the modal. + Defaults to false. + """ + super().__init__(type=self.type, block_id=block_id) + show_unknown_key_warning(self, others) + + self.label = TextObject.parse(label, default_type=PlainTextObject.type) + self.element = BlockElement.parse(element) + self.hint = TextObject.parse(hint, default_type=PlainTextObject.type) + self.dispatch_action = dispatch_action + self.optional = optional + + @JsonValidator(f"label attribute cannot exceed {label_max_length} characters") + def _validate_label_length(self): + return self.label is None or self.label.text is None or len(self.label.text) <= self.label_max_length + + @JsonValidator(f"hint attribute cannot exceed {hint_max_length} characters") + def _validate_hint_length(self): + return self.hint is None or self.hint.text is None or len(self.hint.text) <= self.label_max_length + + @JsonValidator( + ( + "element attribute must be a string, select element, multi-select element, " + "or a datepicker. (Sub-classes of InputInteractiveElement)" + ) + ) + def _validate_element_type(self): + return self.element is None or isinstance(self.element, (str, InputInteractiveElement)) + + +class FileBlock(Block): + type = "file" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"external_id", "source"}) + + def __init__( + self, + *, + external_id: str, + source: str = "remote", + block_id: Optional[str] = None, + **others: dict, + ): + """Displays a remote file. + https://api.slack.com/reference/block-kit/blocks#file + + Args: + external_id (required): The external unique ID for this file. + source (required): At the moment, source will always be remote for a remote file. + block_id: A string acting as a unique identifier for a block. If not specified, one will be generated. + Maximum length for this field is 255 characters. + block_id should be unique for each message and each iteration of a message. + If a message is updated, use a new block_id. + """ + super().__init__(type=self.type, block_id=block_id) + show_unknown_key_warning(self, others) + + self.external_id = external_id + self.source = source + + +class CallBlock(Block): + type = "call" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"call_id", "api_decoration_available", "call"}) + + def __init__( + self, + *, + call_id: str, + api_decoration_available: Optional[bool] = None, + call: Optional[Dict[str, Dict[str, Any]]] = None, + block_id: Optional[str] = None, + **others: dict, + ): + """Displays a call information + https://api.slack.com/reference/block-kit/blocks#call + """ + super().__init__(type=self.type, block_id=block_id) + show_unknown_key_warning(self, others) + + self.call_id = call_id + self.api_decoration_available = api_decoration_available + self.call = call + + +class HeaderBlock(Block): + type = "header" + text_max_length = 150 + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"text"}) + + def __init__( + self, + *, + block_id: Optional[str] = None, + text: Optional[Union[str, dict, TextObject]] = None, + **others: dict, + ): + """A header is a plain-text block that displays in a larger, bold font. + https://api.slack.com/reference/block-kit/blocks#header + + Args: + block_id: A string acting as a unique identifier for a block. If not specified, one will be generated. + Maximum length for this field is 255 characters. + block_id should be unique for each message and each iteration of a message. + If a message is updated, use a new block_id. + text (required): The text for the block, in the form of a plain_text text object. + Maximum length for the text in this field is 150 characters. + """ + super().__init__(type=self.type, block_id=block_id) + show_unknown_key_warning(self, others) + + self.text = TextObject.parse(text, default_type=PlainTextObject.type) + + @JsonValidator("text attribute must be specified") + def _validate_text(self): + return self.text is not None + + @JsonValidator(f"text attribute cannot exceed {text_max_length} characters") + def _validate_alt_text_length(self): + return self.text is None or len(self.text.text) <= self.text_max_length diff --git a/core_service/aws_lambda/project/packages/slack_sdk/models/dialoags.py b/core_service/aws_lambda/project/packages/slack_sdk/models/dialoags.py new file mode 100644 index 0000000..1adf7b5 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/models/dialoags.py @@ -0,0 +1,29 @@ +from slack_sdk.models.dialogs import AbstractDialogSelector +from slack_sdk.models.dialogs import DialogChannelSelector +from slack_sdk.models.dialogs import DialogConversationSelector +from slack_sdk.models.dialogs import DialogExternalSelector +from slack_sdk.models.dialogs import DialogStaticSelector +from slack_sdk.models.dialogs import DialogTextArea +from slack_sdk.models.dialogs import DialogTextComponent +from slack_sdk.models.dialogs import DialogTextField +from slack_sdk.models.dialogs import DialogUserSelector +from slack_sdk.models.dialogs import TextElementSubtypes +from slack_sdk.models.dialogs import DialogBuilder + +from slack import deprecation + +deprecation.show_message(__name__, "slack_sdk.models.dialogs") + +__all__ = [ + "AbstractDialogSelector", + "DialogChannelSelector", + "DialogConversationSelector", + "DialogExternalSelector", + "DialogStaticSelector", + "DialogTextArea", + "DialogTextComponent", + "DialogTextField", + "DialogUserSelector", + "TextElementSubtypes", + "DialogBuilder", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/models/dialogs/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/models/dialogs/__init__.py new file mode 100644 index 0000000..43bba25 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/models/dialogs/__init__.py @@ -0,0 +1,921 @@ +from abc import ABCMeta, abstractmethod +from json import dumps +from typing import List, Optional, Union, Set, Sequence + +from slack_sdk.models import extract_json +from slack_sdk.models.attachments import AbstractActionSelector +from slack_sdk.models.basic_objects import EnumValidator, JsonObject, JsonValidator +from slack_sdk.models.blocks import Option, OptionGroup, DynamicSelectElementTypes + +TextElementSubtypes = {"email", "number", "tel", "url"} + + +class DialogTextComponent(JsonObject, metaclass=ABCMeta): + attributes = { + "hint", + "label", + "max_length", + "min_length", + "name", + "optional", + "placeholder", + "subtype", + "type", + "value", + } + + name_max_length = 300 + label_max_length = 48 + placeholder_max_length = 150 + hint_max_length = 150 + + @property + @abstractmethod + def type(self): + pass + + @property + @abstractmethod + def max_value_length(self): + pass + + def __init__( + self, + *, + name: str, + label: str, + optional: bool = False, + placeholder: Optional[str] = None, + hint: Optional[str] = None, + value: Optional[str] = None, + min_length: int = 0, + max_length: Optional[int] = None, + subtype: Optional[str] = None, + ): + self.name = name + self.label = label + self.optional = optional + self.placeholder = placeholder + self.hint = hint + self.value = value + self.min_length = min_length + self.max_length = max_length or self.max_value_length + self.subtype = subtype + + @JsonValidator(f"name attribute cannot exceed {name_max_length} characters") + def name_length(self) -> bool: + return len(self.name) < self.name_max_length + + @JsonValidator(f"label attribute cannot exceed {label_max_length} characters") + def label_length(self) -> bool: + return len(self.label) < self.label_max_length + + @JsonValidator(f"placeholder attribute cannot exceed {placeholder_max_length} characters") + def placeholder_length(self) -> bool: + return self.placeholder is None or len(self.placeholder) < self.placeholder_max_length + + @JsonValidator(f"hint attribute cannot exceed {hint_max_length} characters") + def hint_length(self) -> bool: + return self.hint is None or len(self.hint) < self.hint_max_length + + @JsonValidator("value attribute exceeded bounds") + def value_length(self) -> bool: + return self.value is None or len(self.value) < self.max_value_length + + @JsonValidator("min_length attribute must be greater than or equal to 0") + def min_length_above_zero(self) -> bool: + return self.min_length is None or self.min_length >= 0 + + @JsonValidator("min_length attribute exceed bounds") + def min_length_length(self) -> bool: + return self.min_length is None or self.min_length <= self.max_value_length + + @JsonValidator("min_length attribute must be less than max value attribute") + def min_length_below_max_length(self) -> bool: + return self.min_length is None or self.min_length < self.max_length + + @JsonValidator("max_length attribute must be greater than or equal to 0") + def max_length_above_zero(self) -> bool: + return self.max_length is None or self.max_length > 0 + + @JsonValidator("max_length attribute exceeded bounds") + def max_length_length(self) -> bool: + return self.max_length is None or self.max_length <= self.max_value_length + + @EnumValidator("subtype", TextElementSubtypes) + def subtype_valid(self) -> bool: + return self.subtype is None or self.subtype in TextElementSubtypes + + +class DialogTextField(DialogTextComponent): + """ + Text elements are single-line plain text fields. + + https://api.slack.com/dialogs#text_elements + """ + + type = "text" + max_value_length = 150 + + +class DialogTextArea(DialogTextComponent): + """ + A textarea is a multi-line plain text editing control. You've likely encountered + these on the world wide web. Use this element if you want a relatively long + answer from users. The element UI provides a remaining character count to the + max_length you have set or the default, 3000. + + https://api.slack.com/dialogs#textarea_elements + """ + + type = "textarea" + max_value_length = 3000 + + +class AbstractDialogSelector(JsonObject, metaclass=ABCMeta): + DataSourceTypes = DynamicSelectElementTypes.union({"external", "static"}) + + attributes = {"data_source", "label", "name", "optional", "placeholder", "type"} + + name_max_length = 300 + label_max_length = 48 + placeholder_max_length = 150 + + @property + @abstractmethod + def data_source(self) -> str: + pass + + def __init__( + self, + *, + name: str, + label: str, + optional: bool = False, + value: Optional[Union[Option, str]] = None, + placeholder: Optional[str] = None, + ): + self.name = name + self.label = label + self.optional = optional + self.value = value + self.placeholder = placeholder + self.type = "select" + + @JsonValidator(f"name attribute cannot exceed {name_max_length} characters") + def name_length(self) -> bool: + return len(self.name) < self.name_max_length + + @JsonValidator(f"label attribute cannot exceed {label_max_length} characters") + def label_length(self) -> bool: + return len(self.label) < self.label_max_length + + @JsonValidator(f"placeholder attribute cannot exceed {placeholder_max_length} characters") + def placeholder_length(self) -> bool: + return self.placeholder is None or len(self.placeholder) < self.placeholder_max_length + + @EnumValidator("data_source", DataSourceTypes) + def data_source_valid(self) -> bool: + return self.data_source in self.DataSourceTypes + + def to_dict(self) -> dict: # skipcq: PYL-W0221 + json = super().to_dict() + if self.data_source == "external": + if isinstance(self.value, Option): + json["selected_options"] = extract_json([self.value], "dialog") + elif self.value is not None: + json["selected_options"] = Option.from_single_value(self.value) + else: + if isinstance(self.value, Option): + json["value"] = self.value.value + elif self.value is not None: + json["value"] = self.value + return json + + +class DialogStaticSelector(AbstractDialogSelector): + """ + Use the select element for multiple choice selections allowing users to pick a + single item from a list. True to web roots, this selection is displayed as a + dropdown menu. + + https://api.slack.com/dialogs#select_elements + """ + + data_source = "static" + + options_max_length = 100 + + def __init__( + self, + *, + name: str, + label: str, + options: Union[Sequence[Option], Sequence[OptionGroup]], + optional: bool = False, + value: Optional[Union[Option, str]] = None, + placeholder: Optional[str] = None, + ): + """ + Use the select element for multiple choice selections allowing users to pick + a single item from a list. True to web roots, this selection is displayed as + a dropdown menu. + + A select element may contain up to 100 selections, provided as a list of + Option or OptionGroup objects + + https://api.slack.com/dialogs#attributes_select_elements + + Args: + name: Name of form element. Required. No more than 300 characters. + label: Label displayed to user. Required. No more than 48 characters. + options: A list of up to 100 Option or OptionGroup objects. Object + types cannot be mixed. + optional: Provide true when the form element is not required. By + default, form elements are required. + value: Provide a default selected value. + placeholder: A string displayed as needed to help guide users in + completing the element. 150 character maximum. + """ + super().__init__( + name=name, + label=label, + optional=optional, + value=value, + placeholder=placeholder, + ) + self.options = options + + @JsonValidator(f"options attribute cannot exceed {options_max_length} items") + def options_length(self) -> bool: + return len(self.options) < self.options_max_length + + def to_dict(self) -> dict: + json = super().to_dict() + if isinstance(self.options[0], OptionGroup): + json["option_groups"] = extract_json(self.options, "dialog") + else: + json["options"] = extract_json(self.options, "dialog") + return json + + +class DialogUserSelector(AbstractDialogSelector): + data_source = "users" + + def __init__( # skipcq: PYL-W0235 + self, + *, + name: str, + label: str, + optional: bool = False, + value: Optional[str] = None, + placeholder: Optional[str] = None, + ): + """ + Now you can easily populate a select menu with a list of users. For example, + when you are creating a bug tracking app, you want to include a field for an + assignee. Slack pre-populates the user list in client-side, so your app + doesn't need access to a related OAuth scope. + + https://api.slack.com/dialogs#dynamic_select_elements_users + + Args: + name: Name of form element. Required. No more than 300 characters. + label: Label displayed to user. Required. No more than 48 characters. + optional: Provide true when the form element is not required. By + default, form elements are required. + value: Provide a default selected value. + placeholder: A string displayed as needed to help guide users in + completing the element. 150 character maximum. + """ + super().__init__( + name=name, + label=label, + optional=optional, + value=value, + placeholder=placeholder, + ) + + +class DialogChannelSelector(AbstractDialogSelector): + data_source = "channels" + + def __init__( # skipcq: PYL-W0235 + self, + *, + name: str, + label: str, + optional: bool = False, + value: Optional[str] = None, + placeholder: Optional[str] = None, + ): + """ + You can also provide a select menu with a list of channels. Specify your + data_source as channels to limit only to public channels + + https://api.slack.com/dialogs#dynamic_select_elements_channels_conversations + + Args: + name: Name of form element. Required. No more than 300 characters. + label: Label displayed to user. Required. No more than 48 characters. + optional: Provide true when the form element is not required. By + default, form elements are required. + value: Provide a default selected value. + placeholder: A string displayed as needed to help guide users in + completing the element. 150 character maximum. + """ + super().__init__( + name=name, + label=label, + optional=optional, + value=value, + placeholder=placeholder, + ) + + +class DialogConversationSelector(AbstractDialogSelector): + data_source = "conversations" + + def __init__( # skipcq: PYL-W0235 + self, + *, + name: str, + label: str, + optional: bool = False, + value: Optional[str] = None, + placeholder: Optional[str] = None, + ): + """ + You can also provide a select menu with a list of conversations - including + private channels, direct messages, MPIMs, and whatever else we consider a + conversation-like thing. + + https://api.slack.com/dialogs#dynamic_select_elements_channels_conversations + + Args: + name: Name of form element. Required. No more than 300 characters. + label: Label displayed to user. Required. No more than 48 characters. + optional: Provide true when the form element is not required. By + default, form elements are required. + value: Provide a default selected value. + placeholder: A string displayed as needed to help guide users in + completing the element. 150 character maximum. + """ + super().__init__( + name=name, + label=label, + optional=optional, + value=value, + placeholder=placeholder, + ) + + +class DialogExternalSelector(AbstractDialogSelector): + data_source = "external" + + @property + def attributes(self) -> Set[str]: + return super().attributes.union({"min_query_length"}) + + def __init__( + self, + *, + name: str, + label: str, + value: Optional[Option] = None, + min_query_length: Optional[int] = None, + optional: Optional[bool] = False, + placeholder: Optional[str] = None, + ): + """ + Use the select element for multiple choice selections allowing users to pick + a single item from a list. True to web roots, this selection is displayed as + a dropdown menu. + + A list of options can be loaded from an external URL and used in your dialog + menus. + + https://api.slack.com/dialogs#dynamic_select_elements_external + + Args: + name: Name of form element. Required. No more than 300 characters. + label: Label displayed to user. Required. No more than 48 characters. + min_query_length: Specify the number of characters that must be typed + by a user into a dynamic select menu before dispatching to the app. + optional: Provide true when the form element is not required. By + default, form elements are required. + value: Provide a default selected value. This should be a single + Option or OptionGroup that exactly matches one that will be returned + from your external endpoint. + placeholder: A string displayed as needed to help guide users in + completing the element. 150 character maximum. + """ + super().__init__( + name=name, + label=label, + value=value, + optional=optional, + placeholder=placeholder, + ) + self.min_query_length = min_query_length + + +class DialogBuilder(JsonObject): + attributes = {} # no attributes because to_dict has unique implementation + + _callback_id: Optional[str] + _elements: List[Union[DialogTextComponent, AbstractDialogSelector]] + _submit_label: Optional[str] + _notify_on_cancel: bool + _state: Optional[str] + + title_max_length = 24 + submit_label_max_length = 24 + elements_max_length = 10 + state_max_length = 3000 + + def __init__(self): + """ + Create a DialogBuilder to more easily construct the JSON required to submit a + dialog to Slack + """ + self._title = None + self._callback_id = None + self._elements = [] + self._submit_label = None + self._notify_on_cancel = False + self._state = None + + def title(self, title: str) -> "DialogBuilder": + """ + Specify a title for this dialog + + Args: + title: must not exceed 24 characters + """ + self._title = title + return self + + def state(self, state: Union[dict, str]) -> "DialogBuilder": + """ + Pass state into this dialog - dictionaries will be automatically formatted to + JSON + + Args: + state: Extra state information that you need to pass from this dialog + back to your application on submission + """ + if isinstance(state, dict): + self._state = dumps(state) + else: + self._state = state + return self + + def callback_id(self, callback_id: str) -> "DialogBuilder": + """ + Specify a callback ID for this dialog, which your application will then + receive upon dialog submission + + Args: + callback_id: a string identifying this particular dialog + """ + self._callback_id = callback_id + return self + + def submit_label(self, label: str) -> "DialogBuilder": + """ + The label to use on the 'Submit' button on the dialog. Defaults to 'Submit' + if not specified. + + Args: + label: must not exceed 24 characters, and must be a single word (no + spaces) + """ + self._submit_label = label + return self + + def notify_on_cancel(self, notify: bool) -> "DialogBuilder": + """ + Whether this dialog should send a request to your application even if the + user cancels their interaction. Defaults to False. + + Args: + notify: Set to True to indicate that your application should receive a + request even if the user cancels interaction with the dialog. + """ + self._notify_on_cancel = notify + return self + + def text_field( + self, + *, + name: str, + label: str, + optional: bool = False, + placeholder: Optional[str] = None, + hint: Optional[str] = None, + value: Optional[str] = None, + min_length: int = 0, + max_length: int = 150, + subtype: Optional[str] = None, + ) -> "DialogBuilder": + """ + Text elements are single-line plain text fields. + + https://api.slack.com/dialogs#attributes_text_elements + + Args: + name: Name of form element. Required. No more than 300 characters. + label: Label displayed to user. Required. 48 character maximum. + optional: Provide true when the form element is not required. By + default, form elements are required. + placeholder: A string displayed as needed to help guide users in + completing the element. 150 character maximum. + hint: Helpful text provided to assist users in answering a question. + Up to 150 characters. + value: A default value for this field. Up to 150 characters. + min_length: Minimum input length allowed for element. Up to 150 + characters. Defaults to 0. + max_length: Maximum input length allowed for element. Up to 150 + characters. Defaults to 150. + subtype: A subtype for this text input. Accepts email, number, tel, + or url. In some form factors, optimized input is provided for this + subtype. + """ + self._elements.append( + DialogTextField( + name=name, + label=label, + optional=optional, + placeholder=placeholder, + hint=hint, + value=value, + min_length=min_length, + max_length=max_length, + subtype=subtype, + ) + ) + return self + + def text_area( + self, + *, + name: str, + label: str, + optional: bool = False, + placeholder: Optional[str] = None, + hint: Optional[str] = None, + value: Optional[str] = None, + min_length: int = 0, + max_length: int = 3000, + subtype: Optional[str] = None, + ) -> "DialogBuilder": + """ + A textarea is a multi-line plain text editing control. You've likely + encountered these on the world wide web. Use this element if you want a + relatively long answer from users. The element UI provides a remaining + character count to the max_length you have set or the default, + 3000. + + https://api.slack.com/dialogs#attributes_textarea_elements + + Args: + name: Name of form element. Required. No more than 300 characters. + label: Label displayed to user. Required. 48 character maximum. + optional: Provide true when the form element is not required. By + default, form elements are required. + placeholder: A string displayed as needed to help guide users in + completing the element. 150 character maximum. + hint: Helpful text provided to assist users in answering a question. + Up to 150 characters. + value: A default value for this field. Up to 3000 characters. + min_length: Minimum input length allowed for element. 1-3000 + characters. Defaults to 0. + max_length: Maximum input length allowed for element. 0-3000 + characters. Defaults to 3000. + subtype: A subtype for this text input. Accepts email, number, tel, + or url. In some form factors, optimized input is provided for this + subtype. + """ + self._elements.append( + DialogTextArea( + name=name, + label=label, + optional=optional, + placeholder=placeholder, + hint=hint, + value=value, + min_length=min_length, + max_length=max_length, + subtype=subtype, + ) + ) + return self + + def static_selector( + self, + *, + name: str, + label: str, + options: Union[Sequence[Option], Sequence[OptionGroup]], + optional: bool = False, + value: Optional[str] = None, + placeholder: Optional[str] = None, + ) -> "DialogBuilder": + """ + Use the select element for multiple choice selections allowing users to pick + a single item from a list. True to web roots, this selection is displayed as + a dropdown menu. + + A select element may contain up to 100 selections, provided as a list of + Option or OptionGroup objects + + https://api.slack.com/dialogs#attributes_select_elements + + Args: + name: Name of form element. Required. No more than 300 characters. + label: Label displayed to user. Required. No more than 48 characters. + options: A list of up to 100 Option or OptionGroup objects. Object + types cannot be mixed. + optional: Provide true when the form element is not required. By + default, form elements are required. + value: Provide a default selected value. + placeholder: A string displayed as needed to help guide users in + completing the element. 150 character maximum. + """ + self._elements.append( + DialogStaticSelector( + name=name, + label=label, + options=options, + optional=optional, + value=value, + placeholder=placeholder, + ) + ) + return self + + def external_selector( + self, + *, + name: str, + label: str, + optional: bool = False, + value: Optional[Option] = None, + placeholder: Optional[str] = None, + min_query_length: Optional[int] = None, + ) -> "DialogBuilder": + """ + Use the select element for multiple choice selections allowing users to pick + a single item from a list. True to web roots, this selection is displayed as + a dropdown menu. + + A list of options can be loaded from an external URL and used in your dialog + menus. + + https://api.slack.com/dialogs#dynamic_select_elements_external + + Args: + name: Name of form element. Required. No more than 300 characters. + label: Label displayed to user. Required. No more than 48 characters. + min_query_length: Specify the number of characters that must be + typed by a user into a dynamic select menu before dispatching to your + application. + optional: Provide true when the form element is not required. By + default, form elements are required. + value: Provide a default selected value. This should be a single + Option or OptionGroup that exactly matches one that will be returned + from your external endpoint. + placeholder: A string displayed as needed to help guide users in + completing the element. 150 character maximum. + """ + self._elements.append( + DialogExternalSelector( + name=name, + label=label, + optional=optional, + value=value, + placeholder=placeholder, + min_query_length=min_query_length, + ) + ) + return self + + def user_selector( + self, + *, + name: str, + label: str, + optional: bool = False, + value: Optional[str] = None, + placeholder: Optional[str] = None, + ) -> "DialogBuilder": + """ + Now you can easily populate a select menu with a list of users. For example, + when you are creating a bug tracking app, you want to include a field for an + assignee. Slack pre-populates the user list in client-side, so your app + doesn't need access to a related OAuth scope. + + https://api.slack.com/dialogs#dynamic_select_elements_users + + Args: + name: Name of form element. Required. No more than 300 characters. + label: Label displayed to user. Required. No more than 48 characters. + optional: Provide true when the form element is not required. By + default, form elements are required. + value: Provide a default selected value. + placeholder: A string displayed as needed to help guide users in + completing the element. 150 character maximum. + """ + self._elements.append( + DialogUserSelector( + name=name, + label=label, + optional=optional, + value=value, + placeholder=placeholder, + ) + ) + return self + + def channel_selector( + self, + *, + name: str, + label: str, + optional: bool = False, + value: Optional[str] = None, + placeholder: Optional[str] = None, + ) -> "DialogBuilder": + """ + You can also provide a select menu with a list of channels. Specify your + data_source as channels to limit only to public channels + + https://api.slack.com/dialogs#dynamic_select_elements_channels_conversations + + Args: + name: Name of form element. Required. No more than 300 characters. + label: Label displayed to user. Required. No more than 48 characters. + optional: Provide true when the form element is not required. By + default, form elements are required. + value: Provide a default selected value. + placeholder: A string displayed as needed to help guide users in + completing the element. 150 character maximum. + """ + self._elements.append( + DialogChannelSelector( + name=name, + label=label, + optional=optional, + value=value, + placeholder=placeholder, + ) + ) + return self + + def conversation_selector( + self, + *, + name: str, + label: str, + optional: bool = False, + value: Optional[str] = None, + placeholder: Optional[str] = None, + ) -> "DialogBuilder": + """ + You can also provide a select menu with a list of conversations - including + private channels, direct messages, MPIMs, and whatever else we consider a + conversation-like thing. + + https://api.slack.com/dialogs#dynamic_select_elements_channels_conversations + + Args: + name: Name of form element. Required. No more than 300 characters. + label: Label displayed to user. Required. No more than 48 characters. + optional: Provide true when the form element is not required. By + default, form elements are required. + value: Provide a default selected value. + placeholder: A string displayed as needed to help guide users in + completing the element. 150 character maximum. + """ + self._elements.append( + DialogConversationSelector( + name=name, + label=label, + optional=optional, + value=value, + placeholder=placeholder, + ) + ) + return self + + @JsonValidator("title attribute is required") + def title_present(self) -> bool: + return self._title is not None + + @JsonValidator(f"title attribute cannot exceed {title_max_length} characters") + def title_length(self) -> bool: + return self._title is not None and len(self._title) <= self.title_max_length + + @JsonValidator("callback_id attribute is required") + def callback_id_present(self) -> bool: + return self._callback_id is not None + + @JsonValidator(f"dialogs must contain between 1 and {elements_max_length} elements") + def elements_length(self) -> bool: + return 0 < len(self._elements) <= self.elements_max_length + + @JsonValidator(f"submit_label cannot exceed {submit_label_max_length} characters") + def submit_label_length(self) -> bool: + return self._submit_label is None or len(self._submit_label) <= self.submit_label_max_length + + @JsonValidator("submit_label can only be one word") + def submit_label_valid(self) -> bool: + return self._submit_label is None or " " not in self._submit_label + + @JsonValidator(f"state cannot exceed {state_max_length} characters") + def state_length(self) -> bool: + return not self._state or len(self._state) <= self.state_max_length + + def to_dict(self) -> dict: # skipcq: PYL-W0221 + self.validate_json() + json = { + "title": self._title, + "callback_id": self._callback_id, + "elements": extract_json(self._elements), + "notify_on_cancel": self._notify_on_cancel, + } + if self._submit_label is not None: + json["submit_label"] = self._submit_label + if self._state is not None: + json["state"] = self._state + return json + + +class ActionStaticSelector(AbstractActionSelector): + """ + Use the select element for multiple choice selections allowing users to pick a + single item from a list. True to web roots, this selection is displayed as a + dropdown menu. + + https://api.slack.com/dialogs#select_elements + """ + + data_source = "static" + + options_max_length = 100 + + def __init__( + self, + *, + name: str, + text: str, + options: Sequence[Union[Option, OptionGroup]], + selected_option: Optional[Option] = None, + ): + """ + Help users make clear, concise decisions by providing a menu of options + within messages. + + https://api.slack.com/docs/message-menus + + Args: + name: Name this specific action. The name will be returned to your + Action URL along with the message's callback_id when this action is + invoked. Use it to identify this particular response path. + text: The user-facing label for the message button or menu + representing this action. Cannot contain markup. + options: A list of no mre than 100 Option or OptionGroup objects + selected_option: An Option object to pre-select as the default + value. + """ + super().__init__(name=name, text=text, selected_option=selected_option) + self.options = options + + @JsonValidator(f"options attribute cannot exceed {options_max_length} items") + def options_length(self) -> bool: + return len(self.options) < self.options_max_length + + def to_dict(self) -> dict: + json = super().to_dict() + if isinstance(self.options[0], OptionGroup): + json["option_groups"] = extract_json(self.options, "action") + else: + json["options"] = extract_json(self.options, "action") + return json + + +__all__ = [ + "TextElementSubtypes", + "AbstractDialogSelector", + "DialogChannelSelector", + "DialogConversationSelector", + "DialogExternalSelector", + "DialogStaticSelector", + "DialogTextArea", + "DialogTextComponent", + "DialogTextField", + "DialogUserSelector", + "TextElementSubtypes", + "DialogBuilder", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/models/messages/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/models/messages/__init__.py new file mode 100644 index 0000000..facbff8 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/models/messages/__init__.py @@ -0,0 +1,85 @@ +from datetime import datetime # type: ignore +from typing import Optional, Union + +from slack_sdk.models.basic_objects import BaseObject + + +class Link(BaseObject): + def __init__(self, *, url: str, text: str): + """Base class used to generate links in Slack's not-quite Markdown, not quite HTML syntax + https://api.slack.com/reference/surfaces/formatting#linking_to_urls + """ + self.url = url + self.text = text + + def __str__(self): + if self.text: + separator = "|" + else: + separator = "" + return f"<{self.url}{separator}{self.text}>" + + +class DateLink(Link): + def __init__( + self, + *, + date: Union[datetime, int], + date_format: str, + fallback: str, + link: Optional[str] = None, + ): + """Text containing a date or time should display that date in the local timezone of the person seeing the text. + https://api.slack.com/reference/surfaces/formatting#date-formatting + """ + if isinstance(date, datetime): + epoch = int(date.timestamp()) # type: ignore + else: + epoch = date + if link is not None: + link = f"^{link}" + else: + link = "" + super().__init__(url=f"!date^{epoch}^{date_format}{link}", text=fallback) + + +class ObjectLink(Link): + prefix_mapping = { + "C": "#", # channel + "G": "#", # group message + "U": "@", # user + "W": "@", # workspace user (enterprise) + "B": "@", # bot user + "S": "!subteam^", # user groups, originally known as subteams + } + + def __init__(self, *, object_id: str, text: str = ""): + """Convenience class to create links to specific object types + https://api.slack.com/reference/surfaces/formatting#linking-channels + """ + prefix = self.prefix_mapping.get(object_id[0].upper(), "@") + super().__init__(url=f"{prefix}{object_id}", text=text) + + +class ChannelLink(Link): + def __init__(self): + """Represents an @channel link, which notifies everyone present in this channel. + https://api.slack.com/reference/surfaces/formatting + """ + super().__init__(url="!channel", text="channel") + + +class HereLink(Link): + def __init__(self): + """Represents an @here link, which notifies all online users of this channel. + https://api.slack.com/reference/surfaces/formatting + """ + super().__init__(url="!here", text="here") + + +class EveryoneLink(Link): + def __init__(self): + """Represents an @everyone link, which notifies all users of this workspace. + https://api.slack.com/reference/surfaces/formatting + """ + super().__init__(url="!everyone", text="everyone") diff --git a/core_service/aws_lambda/project/packages/slack_sdk/models/messages/message.py b/core_service/aws_lambda/project/packages/slack_sdk/models/messages/message.py new file mode 100644 index 0000000..6d92ebe --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/models/messages/message.py @@ -0,0 +1,77 @@ +import logging +import os +import warnings +from typing import Optional, Sequence + +from slack_sdk.models import extract_json +from slack_sdk.models.attachments import Attachment +from slack_sdk.models.basic_objects import ( + JsonObject, + JsonValidator, +) +from slack_sdk.models.blocks import Block + +LOGGER = logging.getLogger(__name__) + +skip_warn = os.environ.get("SLACKCLIENT_SKIP_DEPRECATION") # for unit tests etc. +if not skip_warn: + message = "This class is no longer actively maintained. " "Please use a dict object for building message data instead." + warnings.warn(message) + + +class Message(JsonObject): + attributes = {"text"} + + attachments_max_length = 100 + + def __init__( + self, + *, + text: str, + attachments: Optional[Sequence[Attachment]] = None, + blocks: Optional[Sequence[Block]] = None, + markdown: bool = True, + ): + """ + Create a message. + + https://api.slack.com/messaging/composing#message-structure + + Args: + text: Plain or Slack Markdown-like text to display in the message. + attachments: A list of Attachment objects to display after the rest of + the message's content. More than 20 is not recommended, but the actual + limit is 100 + blocks: A list of Block objects to attach to this message. If + specified, the 'text' property is ignored (more specifically, it's used + as a fallback on clients that can't render blocks) + markdown: Whether to parse markdown into formatting such as + bold/italics, or leave text completely unmodified. + """ + self.text = text + self.attachments = attachments or [] + self.blocks = blocks or [] + self.markdown = markdown + + @JsonValidator(f"attachments attribute cannot exceed {attachments_max_length} items") + def attachments_length(self): + return self.attachments is None or len(self.attachments) <= self.attachments_max_length + + def to_dict(self) -> dict: # skipcq: PYL-W0221 + json = super().to_dict() + if len(self.text) > 40000: + LOGGER.error("Messages over 40,000 characters are automatically truncated by Slack") + # The following limitation used to be true in the past. + # As of Feb 2021, having both is recommended + # ----------------- + # if self.text and self.blocks: + # # Slack doesn't render the text property if there are blocks, so: + # LOGGER.info(q + # "text attribute is treated as fallback text if blocks are attached to " + # "a message - insert text as a new SectionBlock if you want it to be " + # "displayed " + # ) + json["attachments"] = extract_json(self.attachments) + json["blocks"] = extract_json(self.blocks) + json["mrkdwn"] = self.markdown + return json diff --git a/core_service/aws_lambda/project/packages/slack_sdk/models/metadata/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/models/metadata/__init__.py new file mode 100644 index 0000000..5a1b7a9 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/models/metadata/__init__.py @@ -0,0 +1,30 @@ +from typing import Dict, Any +from slack_sdk.models.basic_objects import JsonObject + + +class Metadata(JsonObject): + """Message metadata + + https://api.slack.com/metadata + """ + + attributes = { + "event_type", + "event_payload", + } + + def __init__( + self, + event_type: str, + event_payload: Dict[str, Any], + **kwargs, + ): + self.event_type = event_type + self.event_payload = event_payload + self.additional_attributes = kwargs + + def __str__(self): + return str(self.get_non_null_attributes()) + + def __repr__(self): + return self.__str__() diff --git a/core_service/aws_lambda/project/packages/slack_sdk/models/views/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/models/views/__init__.py new file mode 100644 index 0000000..4b42907 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/models/views/__init__.py @@ -0,0 +1,232 @@ +import copy +import logging +from typing import Optional, Union, Dict, Sequence + +from slack_sdk.models.basic_objects import JsonObject, JsonValidator +from slack_sdk.models.blocks import Block, TextObject, PlainTextObject, Option + + +class View(JsonObject): + """View object for modals and Home tabs. + + https://api.slack.com/reference/surfaces/views + """ + + types = ["modal", "home", "workflow_step"] + + attributes = { + "type", + "id", + "callback_id", + "external_id", + "team_id", + "bot_id", + "app_id", + "root_view_id", + "previous_view_id", + "title", + "submit", + "close", + "blocks", + "private_metadata", + "state", + "hash", + "clear_on_close", + "notify_on_close", + } + + def __init__( + self, + # "modal", "home", and "workflow_step" + type: str, # skipcq: PYL-W0622 + id: Optional[str] = None, # skipcq: PYL-W0622 + callback_id: Optional[str] = None, + external_id: Optional[str] = None, + team_id: Optional[str] = None, + bot_id: Optional[str] = None, + app_id: Optional[str] = None, + root_view_id: Optional[str] = None, + previous_view_id: Optional[str] = None, + title: Optional[Union[str, dict, PlainTextObject]] = None, + submit: Optional[Union[str, dict, PlainTextObject]] = None, + close: Optional[Union[str, dict, PlainTextObject]] = None, + blocks: Optional[Sequence[Union[dict, Block]]] = None, + private_metadata: Optional[str] = None, + state: Optional[Union[dict, "ViewState"]] = None, + hash: Optional[str] = None, # skipcq: PYL-W0622 + clear_on_close: Optional[bool] = None, + notify_on_close: Optional[bool] = None, + **kwargs, + ): + self.type = type + self.id = id + self.callback_id = callback_id + self.external_id = external_id + self.team_id = team_id + self.bot_id = bot_id + self.app_id = app_id + self.root_view_id = root_view_id + self.previous_view_id = previous_view_id + self.title = TextObject.parse(title, default_type=PlainTextObject.type) + self.submit = TextObject.parse(submit, default_type=PlainTextObject.type) + self.close = TextObject.parse(close, default_type=PlainTextObject.type) + self.blocks = Block.parse_all(blocks) + self.private_metadata = private_metadata + self.state = state + if self.state is not None and isinstance(self.state, dict): + self.state = ViewState(**self.state) + self.hash = hash + self.clear_on_close = clear_on_close + self.notify_on_close = notify_on_close + self.additional_attributes = kwargs + + title_max_length = 24 + blocks_max_length = 100 + close_max_length = 24 + submit_max_length = 24 + private_metadata_max_length = 3000 + callback_id_max_length: int = 255 + + @JsonValidator('type must be either "modal", "home" or "workflow_step"') + def _validate_type(self): + return self.type is not None and self.type in self.types + + @JsonValidator(f"title must be between 1 and {title_max_length} characters") + def _validate_title_length(self): + return self.title is None or 1 <= len(self.title.text) <= self.title_max_length + + @JsonValidator(f"views must contain between 1 and {blocks_max_length} blocks") + def _validate_blocks_length(self): + return self.blocks is None or 0 < len(self.blocks) <= self.blocks_max_length + + @JsonValidator("home view cannot have submit and close") + def _validate_home_tab_structure(self): + return self.type != "home" or (self.type == "home" and self.close is None and self.submit is None) + + @JsonValidator(f"close cannot exceed {close_max_length} characters") + def _validate_close_length(self): + return self.close is None or len(self.close.text) <= self.close_max_length + + @JsonValidator(f"submit cannot exceed {submit_max_length} characters") + def _validate_submit_length(self): + return self.submit is None or len(self.submit.text) <= int(self.submit_max_length) + + @JsonValidator(f"private_metadata cannot exceed {private_metadata_max_length} characters") + def _validate_private_metadata_max_length(self): + return self.private_metadata is None or len(self.private_metadata) <= self.private_metadata_max_length + + @JsonValidator(f"callback_id cannot exceed {callback_id_max_length} characters") + def _validate_callback_id_max_length(self): + return self.callback_id is None or len(self.callback_id) <= self.callback_id_max_length + + def __str__(self): + return str(self.get_non_null_attributes()) + + def __repr__(self): + return self.__str__() + + +class ViewState(JsonObject): + attributes = {"values"} + logger = logging.getLogger(__name__) + + @classmethod + def _show_warning_about_unknown(cls, value): + c = value.__class__ + name = ".".join([c.__module__, c.__name__]) + cls.logger.warning(f"Unknown type for view.state.values detected ({name}) and ViewState skipped to add it") + + def __init__( + self, + *, + values: Dict[str, Dict[str, Union[dict, "ViewStateValue"]]], + ): + value_objects: Dict[str, Dict[str, ViewStateValue]] = {} + new_state_values = copy.copy(values) + if isinstance(new_state_values, dict): # just in case + for block_id, actions in new_state_values.items(): + if actions is None: # skipcq: PYL-R1724 + continue + elif isinstance(actions, dict): + new_actions: Dict[str, Union[ViewStateValue, dict]] = copy.copy(actions) + for action_id, v in actions.items(): + if isinstance(v, dict): + d = copy.copy(v) + value_object = ViewStateValue(**d) + elif isinstance(v, ViewStateValue): + value_object = v + else: + self._show_warning_about_unknown(v) + continue + new_actions[action_id] = value_object + value_objects[block_id] = new_actions + else: + self._show_warning_about_unknown(v) + self.values = value_objects + + def to_dict(self, *args) -> Dict[str, Dict[str, Dict[str, dict]]]: # type: ignore + self.validate_json() + if self.values is not None: + dict_values: Dict[str, Dict[str, dict]] = {} + for block_id, actions in self.values.items(): + if actions: + dict_value: Dict[str, dict] = { + action_id: value.to_dict() # type: ignore + for action_id, value in actions.items() # type: ignore + } + dict_values[block_id] = dict_value + return {"values": dict_values} # type: ignore + else: + return {} + + +class ViewStateValue(JsonObject): + attributes = { + "type", + "value", + "selected_date", + "selected_conversation", + "selected_channel", + "selected_user", + "selected_option", + "selected_conversations", + "selected_channels", + "selected_users", + "selected_options", + } + + def __init__( + self, + *, + type: Optional[str] = None, # skipcq: PYL-W0622 + value: Optional[str] = None, + selected_date: Optional[str] = None, + selected_conversation: Optional[str] = None, + selected_channel: Optional[str] = None, + selected_user: Optional[str] = None, + selected_option: Optional[str] = None, + selected_conversations: Optional[Sequence[str]] = None, + selected_channels: Optional[Sequence[str]] = None, + selected_users: Optional[Sequence[str]] = None, + selected_options: Optional[Sequence[Union[dict, Option]]] = None, + ): + self.type = type + self.value = value + self.selected_date = selected_date + self.selected_conversation = selected_conversation + self.selected_channel = selected_channel + self.selected_user = selected_user + self.selected_option = selected_option + self.selected_conversations = selected_conversations + self.selected_channels = selected_channels + self.selected_users = selected_users + + if isinstance(selected_options, list): + self.selected_options = [] + for option in selected_options: + if isinstance(option, Option): + self.selected_options.append(option) + elif isinstance(option, dict): + self.selected_options.append(Option(**option)) + else: + self.selected_options = selected_options diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/__init__.py new file mode 100644 index 0000000..c16cd49 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/__init__.py @@ -0,0 +1,19 @@ +"""Modules for implementing the Slack OAuth flow + +https://slack.dev/python-slack-sdk/oauth/ +""" +from .authorize_url_generator import AuthorizeUrlGenerator +from .authorize_url_generator import OpenIDConnectAuthorizeUrlGenerator +from .installation_store import InstallationStore +from .redirect_uri_page_renderer import RedirectUriPageRenderer +from .state_store import OAuthStateStore +from .state_utils import OAuthStateUtils + +__all__ = [ + "AuthorizeUrlGenerator", + "OpenIDConnectAuthorizeUrlGenerator", + "InstallationStore", + "RedirectUriPageRenderer", + "OAuthStateStore", + "OAuthStateUtils", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/authorize_url_generator/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/authorize_url_generator/__init__.py new file mode 100644 index 0000000..3ceecf2 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/authorize_url_generator/__init__.py @@ -0,0 +1,63 @@ +from typing import Optional, Sequence + + +class AuthorizeUrlGenerator: + def __init__( + self, + *, + client_id: str, + redirect_uri: Optional[str] = None, + scopes: Optional[Sequence[str]] = None, + user_scopes: Optional[Sequence[str]] = None, + authorization_url: str = "https://slack.com/oauth/v2/authorize", + ): + self.client_id = client_id + self.redirect_uri = redirect_uri + self.scopes = scopes + self.user_scopes = user_scopes + self.authorization_url = authorization_url + + def generate(self, state: str) -> str: + scopes = ",".join(self.scopes) if self.scopes else "" + user_scopes = ",".join(self.user_scopes) if self.user_scopes else "" + url = ( + f"{self.authorization_url}?" + f"state={state}&" + f"client_id={self.client_id}&" + f"scope={scopes}&" + f"user_scope={user_scopes}" + ) + if self.redirect_uri is not None: + url += f"&redirect_uri={self.redirect_uri}" + return url + + +class OpenIDConnectAuthorizeUrlGenerator: + """Refer to https://openid.net/specs/openid-connect-core-1_0.html""" + + def __init__( + self, + *, + client_id: str, + redirect_uri: str, + scopes: Optional[Sequence[str]] = None, + authorization_url: str = "https://slack.com/openid/connect/authorize", + ): + self.client_id = client_id + self.redirect_uri = redirect_uri + self.scopes = scopes + self.authorization_url = authorization_url + + def generate(self, state: str, nonce: Optional[str] = None) -> str: + scopes = ",".join(self.scopes) if self.scopes else "" + url = ( + f"{self.authorization_url}?" + "response_type=code&" + f"state={state}&" + f"client_id={self.client_id}&" + f"scope={scopes}&" + f"redirect_uri={self.redirect_uri}" + ) + if nonce is not None: + url += f"&nonce={nonce}" + return url diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/__init__.py new file mode 100644 index 0000000..b93ebca --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/__init__.py @@ -0,0 +1,10 @@ +from .file import FileInstallationStore +from .installation_store import InstallationStore +from .models import Bot, Installation + +__all__ = [ + "FileInstallationStore", + "InstallationStore", + "Bot", + "Installation", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/amazon_s3/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/amazon_s3/__init__.py new file mode 100644 index 0000000..49da3d9 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/amazon_s3/__init__.py @@ -0,0 +1,344 @@ +import json +import logging +from logging import Logger +from typing import Optional + +from botocore.client import BaseClient + +from slack_sdk.errors import SlackClientConfigurationError +from slack_sdk.oauth.installation_store.async_installation_store import ( + AsyncInstallationStore, +) +from slack_sdk.oauth.installation_store.installation_store import InstallationStore +from slack_sdk.oauth.installation_store.models.bot import Bot +from slack_sdk.oauth.installation_store.models.installation import Installation + + +class AmazonS3InstallationStore(InstallationStore, AsyncInstallationStore): + def __init__( + self, + *, + s3_client: BaseClient, + bucket_name: str, + client_id: str, + historical_data_enabled: bool = True, + logger: Logger = logging.getLogger(__name__), + ): + self.s3_client = s3_client + self.bucket_name = bucket_name + self.historical_data_enabled = historical_data_enabled + self.client_id = client_id + self._logger = logger + + @property + def logger(self) -> Logger: + if self._logger is None: + self._logger = logging.getLogger(__name__) + return self._logger + + async def async_save(self, installation: Installation): + return self.save(installation) + + async def async_save_bot(self, bot: Bot): + return self.save_bot(bot) + + def save(self, installation: Installation): + none = "none" + e_id = installation.enterprise_id or none + t_id = installation.team_id or none + workspace_path = f"{self.client_id}/{e_id}-{t_id}" + + self.save_bot(installation.to_bot()) + + if self.historical_data_enabled: + history_version: str = str(installation.installed_at) + + # per workspace + entity: str = json.dumps(installation.__dict__) + response = self.s3_client.put_object( + Bucket=self.bucket_name, + Body=entity, + Key=f"{workspace_path}/installer-latest", + ) + self.logger.debug(f"S3 put_object response: {response}") + response = self.s3_client.put_object( + Bucket=self.bucket_name, + Body=entity, + Key=f"{workspace_path}/installer-{history_version}", + ) + self.logger.debug(f"S3 put_object response: {response}") + + # per workspace per user + u_id = installation.user_id or none + entity: str = json.dumps(installation.__dict__) + response = self.s3_client.put_object( + Bucket=self.bucket_name, + Body=entity, + Key=f"{workspace_path}/installer-{u_id}-latest", + ) + self.logger.debug(f"S3 put_object response: {response}") + response = self.s3_client.put_object( + Bucket=self.bucket_name, + Body=entity, + Key=f"{workspace_path}/installer-{u_id}-{history_version}", + ) + self.logger.debug(f"S3 put_object response: {response}") + + else: + # per workspace + entity: str = json.dumps(installation.__dict__) + response = self.s3_client.put_object( + Bucket=self.bucket_name, + Body=entity, + Key=f"{workspace_path}/installer-latest", + ) + self.logger.debug(f"S3 put_object response: {response}") + + # per workspace per user + u_id = installation.user_id or none + entity: str = json.dumps(installation.__dict__) + response = self.s3_client.put_object( + Bucket=self.bucket_name, + Body=entity, + Key=f"{workspace_path}/installer-{u_id}-latest", + ) + self.logger.debug(f"S3 put_object response: {response}") + + def save_bot(self, bot: Bot): + none = "none" + e_id = bot.enterprise_id or none + t_id = bot.team_id or none + workspace_path = f"{self.client_id}/{e_id}-{t_id}" + + if self.historical_data_enabled: + history_version: str = str(bot.installed_at) + entity: str = json.dumps(bot.__dict__) + response = self.s3_client.put_object( + Bucket=self.bucket_name, + Body=entity, + Key=f"{workspace_path}/bot-latest", + ) + self.logger.debug(f"S3 put_object response: {response}") + response = self.s3_client.put_object( + Bucket=self.bucket_name, + Body=entity, + Key=f"{workspace_path}/bot-{history_version}", + ) + self.logger.debug(f"S3 put_object response: {response}") + + else: + entity: str = json.dumps(bot.__dict__) + response = self.s3_client.put_object( + Bucket=self.bucket_name, + Body=entity, + Key=f"{workspace_path}/bot-latest", + ) + self.logger.debug(f"S3 put_object response: {response}") + + async def async_find_bot( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Bot]: + return self.find_bot( + enterprise_id=enterprise_id, + team_id=team_id, + is_enterprise_install=is_enterprise_install, + ) + + def find_bot( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Bot]: + none = "none" + e_id = enterprise_id or none + t_id = team_id or none + if is_enterprise_install: + t_id = none + workspace_path = f"{self.client_id}/{e_id}-{t_id}" + try: + fetch_response = self.s3_client.get_object( + Bucket=self.bucket_name, + Key=f"{workspace_path}/bot-latest", + ) + self.logger.debug(f"S3 get_object response: {fetch_response}") + body = fetch_response["Body"].read().decode("utf-8") + data = json.loads(body) + return Bot(**data) + except Exception as e: # skipcq: PYL-W0703 + message = f"Failed to find bot installation data for enterprise: {e_id}, team: {t_id}: {e}" + self.logger.warning(message) + return None + + async def async_find_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Installation]: + return self.find_installation( + enterprise_id=enterprise_id, + team_id=team_id, + user_id=user_id, + is_enterprise_install=is_enterprise_install, + ) + + def find_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Installation]: + none = "none" + e_id = enterprise_id or none + t_id = team_id or none + if is_enterprise_install: + t_id = none + workspace_path = f"{self.client_id}/{e_id}-{t_id}" + try: + key = f"{workspace_path}/installer-{user_id}-latest" if user_id else f"{workspace_path}/installer-latest" + fetch_response = self.s3_client.get_object( + Bucket=self.bucket_name, + Key=key, + ) + self.logger.debug(f"S3 get_object response: {fetch_response}") + body = fetch_response["Body"].read().decode("utf-8") + data = json.loads(body) + installation = Installation(**data) + + if installation is not None and user_id is not None: + # Retrieve the latest bot token, just in case + # See also: https://github.com/slackapi/bolt-python/issues/664 + latest_bot_installation = self.find_installation( + enterprise_id=enterprise_id, + team_id=team_id, + is_enterprise_install=is_enterprise_install, + ) + if latest_bot_installation is not None and installation.bot_token != latest_bot_installation.bot_token: + # NOTE: this logic is based on the assumption that every single installation has bot scopes + # If you need to installation patterns without bot scopes in the same S3 bucket, + # please fork this code and implement your own logic. + installation.bot_id = latest_bot_installation.bot_id + installation.bot_user_id = latest_bot_installation.bot_user_id + installation.bot_token = latest_bot_installation.bot_token + installation.bot_scopes = latest_bot_installation.bot_scopes + installation.bot_refresh_token = latest_bot_installation.bot_refresh_token + installation.bot_token_expires_at = latest_bot_installation.bot_token_expires_at + + return installation + + except Exception as e: # skipcq: PYL-W0703 + message = f"Failed to find an installation data for enterprise: {e_id}, team: {t_id}: {e}" + self.logger.warning(message) + return None + + async def async_delete_bot(self, *, enterprise_id: Optional[str], team_id: Optional[str]) -> None: + return self.delete_bot( + enterprise_id=enterprise_id, + team_id=team_id, + ) + + def delete_bot(self, *, enterprise_id: Optional[str], team_id: Optional[str]) -> None: + none = "none" + e_id = enterprise_id or none + t_id = team_id or none + workspace_path = f"{self.client_id}/{e_id}-{t_id}" + objects = self.s3_client.list_objects( + Bucket=self.bucket_name, + Prefix=f"{workspace_path}/bot-", + ) + for content in objects.get("Contents", []): + key = content.get("Key") + if key is not None: + self.logger.info(f"Going to delete bot installation ({key})") + try: + self.s3_client.delete_object( + Bucket=self.bucket_name, + Key=content.get("Key"), + ) + except Exception as e: # skipcq: PYL-W0703 + message = f"Failed to find bot installation data for enterprise: {e_id}, team: {t_id}: {e}" + raise SlackClientConfigurationError(message) + + async def async_delete_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + ) -> None: + return self.delete_installation( + enterprise_id=enterprise_id, + team_id=team_id, + user_id=user_id, + ) + + def delete_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + ) -> None: + none = "none" + e_id = enterprise_id or none + t_id = team_id or none + workspace_path = f"{self.client_id}/{e_id}-{t_id}" + objects = self.s3_client.list_objects( + Bucket=self.bucket_name, + Prefix=f"{workspace_path}/installer-{user_id or ''}", + ) + deleted_keys = [] + for content in objects.get("Contents", []): + key = content.get("Key") + if key is not None: + self.logger.info(f"Going to delete installation ({key})") + try: + self.s3_client.delete_object( + Bucket=self.bucket_name, + Key=key, + ) + deleted_keys.append(key) + except Exception as e: # skipcq: PYL-W0703 + message = f"Failed to find bot installation data for enterprise: {e_id}, team: {t_id}: {e}" + raise SlackClientConfigurationError(message) + + try: + no_user_id_key = key.replace(f"-{user_id}", "") + if not no_user_id_key.endswith("installer-latest"): + self.s3_client.delete_object( + Bucket=self.bucket_name, + Key=no_user_id_key, + ) + deleted_keys.append(no_user_id_key) + except Exception as e: # skipcq: PYL-W0703 + message = f"Failed to find bot installation data for enterprise: {e_id}, team: {t_id}: {e}" + raise SlackClientConfigurationError(message) + + # Check the remaining installation data + objects = self.s3_client.list_objects( + Bucket=self.bucket_name, + Prefix=f"{workspace_path}/installer-", + MaxKeys=10, # the small number would be enough for this purpose + ) + keys = [c.get("Key") for c in objects.get("Contents", []) if c.get("Key") not in deleted_keys] + # If only installer-latest remains, we should delete the one as well + if len(keys) == 1 and keys[0].endswith("installer-latest"): + content = objects.get("Contents", [])[0] + try: + self.s3_client.delete_object( + Bucket=self.bucket_name, + Key=content.get("Key"), + ) + except Exception as e: # skipcq: PYL-W0703 + message = f"Failed to find bot installation data for enterprise: {e_id}, team: {t_id}: {e}" + raise SlackClientConfigurationError(message) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/async_cacheable_installation_store.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/async_cacheable_installation_store.py new file mode 100644 index 0000000..a5d2e7b --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/async_cacheable_installation_store.py @@ -0,0 +1,136 @@ +from logging import Logger +from typing import Optional, Dict + +from slack_sdk.oauth.installation_store import Bot, Installation +from slack_sdk.oauth.installation_store.async_installation_store import ( + AsyncInstallationStore, +) + + +class AsyncCacheableInstallationStore(AsyncInstallationStore): + underlying: AsyncInstallationStore + cached_bots: Dict[str, Bot] # type: ignore + cached_installations: Dict[str, Installation] # type: ignore + + def __init__(self, installation_store: AsyncInstallationStore): + """A simple memory cache wrapper for any installation stores. + + Args: + installation_store: The installation store to wrap + """ + self.underlying = installation_store + self.cached_bots = {} + self.cached_installations = {} + + @property + def logger(self) -> Logger: + return self.underlying.logger + + async def async_save(self, installation: Installation): # type: ignore + # Invalidate cache data for update operations + key = f"{installation.enterprise_id or ''}-{installation.team_id or ''}" + if key in self.cached_bots: + self.cached_bots.pop(key) + key = f"{installation.enterprise_id or ''}-{installation.team_id or ''}-{installation.user_id or ''}" + if key in self.cached_installations: + self.cached_installations.pop(key) + return await self.underlying.async_save(installation) + + async def async_save_bot(self, bot: Bot): # type: ignore + # Invalidate cache data for update operations + key = f"{bot.enterprise_id or ''}-{bot.team_id or ''}" + if key in self.cached_bots: + self.cached_bots.pop(key) + return await self.underlying.async_save_bot(bot) + + async def async_find_bot( # type: ignore + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Bot]: + if is_enterprise_install or team_id is None: + team_id = "" + key = f"{enterprise_id or ''}-{team_id or ''}" + if key in self.cached_bots: + return self.cached_bots[key] + bot = await self.underlying.async_find_bot( + enterprise_id=enterprise_id, + team_id=team_id, + is_enterprise_install=is_enterprise_install, + ) + if bot: + self.cached_bots[key] = bot + return bot + + async def async_find_installation( # type: ignore + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Installation]: + if is_enterprise_install or team_id is None: + team_id = "" + key = f"{enterprise_id or ''}-{team_id or ''}-{user_id or ''}" + if key in self.cached_installations: + return self.cached_installations[key] + installation = await self.underlying.async_find_installation( + enterprise_id=enterprise_id, + team_id=team_id, + user_id=user_id, + is_enterprise_install=is_enterprise_install, + ) + if installation: + self.cached_installations[key] = installation + return installation + + async def async_delete_bot( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + ) -> None: + await self.underlying.async_delete_bot( + enterprise_id=enterprise_id, + team_id=team_id, + ) + key = f"{enterprise_id or ''}-{team_id or ''}" + self.cached_bots.pop(key) + + async def async_delete_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + ) -> None: + await self.underlying.async_delete_installation( + enterprise_id=enterprise_id, + team_id=team_id, + user_id=user_id, + ) + key_prefix = f"{enterprise_id or ''}-{team_id or ''}" + for key in list(self.cached_installations.keys()): + if key.startswith(key_prefix): + self.cached_installations.pop(key) + + async def async_delete_all( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + ): + await self.underlying.async_delete_all( + enterprise_id=enterprise_id, + team_id=team_id, + ) + key_prefix = f"{enterprise_id or ''}-{team_id or ''}" + for key in list(self.cached_bots.keys()): + if key.startswith(key_prefix): + self.cached_bots.pop(key) + for key in list(self.cached_installations.keys()): + if key.startswith(key_prefix): + self.cached_installations.pop(key) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/async_installation_store.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/async_installation_store.py new file mode 100644 index 0000000..f8b76b8 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/async_installation_store.py @@ -0,0 +1,92 @@ +from logging import Logger +from typing import Optional + +from .models.bot import Bot +from .models.installation import Installation + + +class AsyncInstallationStore: + """The installation store interface for asyncio-based apps. + + The minimum required methods are: + + * async_save(installation) + * async_find_installation(enterprise_id, team_id, user_id, is_enterprise_install) + + If you would like to properly handle app uninstallations and token revocations, + the following methods should be implemented. + + * async_delete_installation(enterprise_id, team_id, user_id) + * async_delete_all(enterprise_id, team_id) + + If your app needs only bot scope installations, the simpler way to implement would be: + + * async_save(installation) + * async_find_bot(enterprise_id, team_id, is_enterprise_install) + * async_delete_bot(enterprise_id, team_id) + * async_delete_all(enterprise_id, team_id) + """ + + @property + def logger(self) -> Logger: + raise NotImplementedError() + + async def async_save(self, installation: Installation): + """Saves an installation data""" + raise NotImplementedError() + + async def async_save_bot(self, bot: Bot): + """Saves a bot installation data""" + raise NotImplementedError() + + async def async_find_bot( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Bot]: + """Finds a bot scope installation per workspace / org""" + raise NotImplementedError() + + async def async_find_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Installation]: + """Finds a relevant installation for the given IDs. + If the user_id is absent, this method may return the latest installation in the workspace / org. + """ + raise NotImplementedError() + + async def async_delete_bot( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + ) -> None: + """Deletes a bot scope installation per workspace / org""" + raise NotImplementedError() + + async def async_delete_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + ) -> None: + """Deletes an installation that matches the given IDs""" + raise NotImplementedError() + + async def async_delete_all( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + ): + """Deletes all installation data for the given workspace / org""" + await self.async_delete_bot(enterprise_id=enterprise_id, team_id=team_id) + await self.async_delete_installation(enterprise_id=enterprise_id, team_id=team_id) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/cacheable_installation_store.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/cacheable_installation_store.py new file mode 100644 index 0000000..8b93c1a --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/cacheable_installation_store.py @@ -0,0 +1,136 @@ +from logging import Logger +from typing import Optional, Dict + +from slack_sdk.oauth import InstallationStore +from slack_sdk.oauth.installation_store import Bot, Installation + + +class CacheableInstallationStore(InstallationStore): + underlying: InstallationStore + cached_bots: Dict[str, Bot] # type: ignore + cached_installations: Dict[str, Installation] # type: ignore + + def __init__(self, installation_store: InstallationStore): + """A simple memory cache wrapper for any installation stores. + + Args: + installation_store: The installation store to wrap + """ + self.underlying = installation_store + self.cached_bots = {} + self.cached_installations = {} + + @property + def logger(self) -> Logger: + return self.underlying.logger + + def save(self, installation: Installation): # type: ignore + # Invalidate cache data for update operations + key = f"{installation.enterprise_id or ''}-{installation.team_id or ''}" + if key in self.cached_bots: + self.cached_bots.pop(key) + key = f"{installation.enterprise_id or ''}-{installation.team_id or ''}-{installation.user_id or ''}" + if key in self.cached_installations: + self.cached_installations.pop(key) + + return self.underlying.save(installation) + + def save_bot(self, bot: Bot): # type: ignore + # Invalidate cache data for update operations + key = f"{bot.enterprise_id or ''}-{bot.team_id or ''}" + if key in self.cached_bots: + self.cached_bots.pop(key) + return self.underlying.save_bot(bot) + + def find_bot( # type: ignore + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Bot]: + if is_enterprise_install or team_id is None: + team_id = "" + key = f"{enterprise_id or ''}-{team_id or ''}" + if key in self.cached_bots: + return self.cached_bots[key] + bot = self.underlying.find_bot( + enterprise_id=enterprise_id, + team_id=team_id, + is_enterprise_install=is_enterprise_install, + ) + if bot: + self.cached_bots[key] = bot + return bot + + def find_installation( # type: ignore + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Installation]: + if is_enterprise_install or team_id is None: + team_id = "" + key = f"{enterprise_id or ''}-{team_id or ''}-{user_id or ''}" + if key in self.cached_installations: + return self.cached_installations[key] + installation = self.underlying.find_installation( + enterprise_id=enterprise_id, + team_id=team_id, + user_id=user_id, + is_enterprise_install=is_enterprise_install, + ) + if installation: + self.cached_installations[key] = installation + return installation + + def delete_bot( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + ) -> None: + self.underlying.delete_bot( + enterprise_id=enterprise_id, + team_id=team_id, + ) + key = f"{enterprise_id or ''}-{team_id or ''}" + if key in self.cached_bots: + self.cached_bots.pop(key) + + def delete_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + ) -> None: + self.underlying.delete_installation( + enterprise_id=enterprise_id, + team_id=team_id, + user_id=user_id, + ) + key_prefix = f"{enterprise_id or ''}-{team_id or ''}" + for key in list(self.cached_installations.keys()): + if key.startswith(key_prefix): + self.cached_installations.pop(key) + + def delete_all( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + ): + self.underlying.delete_all( + enterprise_id=enterprise_id, + team_id=team_id, + ) + key_prefix = f"{enterprise_id or ''}-{team_id or ''}" + for key in list(self.cached_bots.keys()): + if key.startswith(key_prefix): + self.cached_bots.pop(key) + for key in list(self.cached_installations.keys()): + if key.startswith(key_prefix): + self.cached_installations.pop(key) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/file/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/file/__init__.py new file mode 100644 index 0000000..beb18c3 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/file/__init__.py @@ -0,0 +1,245 @@ +import glob +import json +import logging +import os +from logging import Logger +from pathlib import Path +from typing import Optional, Union + +from slack_sdk.oauth.installation_store.async_installation_store import ( + AsyncInstallationStore, +) +from slack_sdk.oauth.installation_store.installation_store import InstallationStore +from slack_sdk.oauth.installation_store.models.bot import Bot +from slack_sdk.oauth.installation_store.models.installation import Installation + + +class FileInstallationStore(InstallationStore, AsyncInstallationStore): + def __init__( + self, + *, + base_dir: str = str(Path.home()) + "/.bolt-app-installation", + historical_data_enabled: bool = True, + client_id: Optional[str] = None, + logger: Logger = logging.getLogger(__name__), + ): + self.base_dir = base_dir + self.historical_data_enabled = historical_data_enabled + self.client_id = client_id + if self.client_id is not None: + self.base_dir = f"{self.base_dir}/{self.client_id}" + self._logger = logger + + @property + def logger(self) -> Logger: + if self._logger is None: + self._logger = logging.getLogger(__name__) + return self._logger + + async def async_save(self, installation: Installation): + return self.save(installation) + + async def async_save_bot(self, bot: Bot): + return self.save_bot(bot) + + def save(self, installation: Installation): + none = "none" + e_id = installation.enterprise_id or none + t_id = installation.team_id or none + team_installation_dir = f"{self.base_dir}/{e_id}-{t_id}" + self._mkdir(team_installation_dir) + + self.save_bot(installation.to_bot()) + + if self.historical_data_enabled: + history_version: str = str(installation.installed_at) + + # per workspace + entity: str = json.dumps(installation.__dict__) + with open(f"{team_installation_dir}/installer-latest", "w") as f: + f.write(entity) + with open(f"{team_installation_dir}/installer-{history_version}", "w") as f: + f.write(entity) + + # per workspace per user + u_id = installation.user_id or none + entity: str = json.dumps(installation.__dict__) + with open(f"{team_installation_dir}/installer-{u_id}-latest", "w") as f: + f.write(entity) + with open(f"{team_installation_dir}/installer-{u_id}-{history_version}", "w") as f: + f.write(entity) + + else: + u_id = installation.user_id or none + installer_filepath = f"{team_installation_dir}/installer-{u_id}-latest" + with open(installer_filepath, "w") as f: + entity: str = json.dumps(installation.__dict__) + f.write(entity) + + def save_bot(self, bot: Bot): + none = "none" + e_id = bot.enterprise_id or none + t_id = bot.team_id or none + team_installation_dir = f"{self.base_dir}/{e_id}-{t_id}" + self._mkdir(team_installation_dir) + + if self.historical_data_enabled: + history_version: str = str(bot.installed_at) + + entity: str = json.dumps(bot.__dict__) + with open(f"{team_installation_dir}/bot-latest", "w") as f: + f.write(entity) + with open(f"{team_installation_dir}/bot-{history_version}", "w") as f: + f.write(entity) + else: + with open(f"{team_installation_dir}/bot-latest", "w") as f: + entity: str = json.dumps(bot.__dict__) + f.write(entity) + + async def async_find_bot( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Bot]: + return self.find_bot( + enterprise_id=enterprise_id, + team_id=team_id, + is_enterprise_install=is_enterprise_install, + ) + + def find_bot( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Bot]: + none = "none" + e_id = enterprise_id or none + t_id = team_id or none + if is_enterprise_install: + t_id = none + bot_filepath = f"{self.base_dir}/{e_id}-{t_id}/bot-latest" + try: + with open(bot_filepath) as f: + data = json.loads(f.read()) + return Bot(**data) + except FileNotFoundError as e: + message = f"Installation data missing for enterprise: {e_id}, team: {t_id}: {e}" + self.logger.debug(message) + return None + + async def async_find_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Installation]: + return self.find_installation( + enterprise_id=enterprise_id, + team_id=team_id, + user_id=user_id, + is_enterprise_install=is_enterprise_install, + ) + + def find_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Installation]: + none = "none" + e_id = enterprise_id or none + t_id = team_id or none + if is_enterprise_install: + t_id = none + installation_filepath = f"{self.base_dir}/{e_id}-{t_id}/installer-latest" + if user_id is not None: + installation_filepath = f"{self.base_dir}/{e_id}-{t_id}/installer-{user_id}-latest" + + try: + installation: Optional[Installation] = None + with open(installation_filepath) as f: + data = json.loads(f.read()) + installation = Installation(**data) + + if installation is not None and user_id is not None: + # Retrieve the latest bot token, just in case + # See also: https://github.com/slackapi/bolt-python/issues/664 + latest_bot_installation = self.find_installation( + enterprise_id=enterprise_id, + team_id=team_id, + is_enterprise_install=is_enterprise_install, + ) + if latest_bot_installation is not None and installation.bot_token != latest_bot_installation.bot_token: + # NOTE: this logic is based on the assumption that every single installation has bot scopes + # If you need to installation patterns without bot scopes in the same S3 bucket, + # please fork this code and implement your own logic. + installation.bot_id = latest_bot_installation.bot_id + installation.bot_user_id = latest_bot_installation.bot_user_id + installation.bot_token = latest_bot_installation.bot_token + installation.bot_scopes = latest_bot_installation.bot_scopes + installation.bot_refresh_token = latest_bot_installation.bot_refresh_token + installation.bot_token_expires_at = latest_bot_installation.bot_token_expires_at + + return installation + + except FileNotFoundError as e: + message = f"Installation data missing for enterprise: {e_id}, team: {t_id}: {e}" + self.logger.debug(message) + return None + + async def async_delete_bot(self, *, enterprise_id: Optional[str], team_id: Optional[str]) -> None: + return self.delete_bot(enterprise_id=enterprise_id, team_id=team_id) + + def delete_bot(self, *, enterprise_id: Optional[str], team_id: Optional[str]) -> None: + none = "none" + e_id = enterprise_id or none + t_id = team_id or none + filepath_glob = f"{self.base_dir}/{e_id}-{t_id}/bot-*" + self._delete_by_glob(e_id, t_id, filepath_glob) + + async def async_delete_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + ) -> None: + return self.delete_installation(enterprise_id=enterprise_id, team_id=team_id, user_id=user_id) + + def delete_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + ) -> None: + none = "none" + e_id = enterprise_id or none + t_id = team_id or none + if user_id is not None: + filepath_glob = f"{self.base_dir}/{e_id}-{t_id}/installer-{user_id}-*" + else: + filepath_glob = f"{self.base_dir}/{e_id}-{t_id}/installer-*" + self._delete_by_glob(e_id, t_id, filepath_glob) + + def _delete_by_glob(self, e_id: str, t_id: str, filepath_glob: str): + for filepath in glob.glob(filepath_glob): + try: + os.remove(filepath) + except FileNotFoundError as e: + message = f"Failed to delete installation data for enterprise: {e_id}, team: {t_id}: {e}" + self.logger.warning(message) + + @staticmethod + def _mkdir(path: Union[str, Path]): + if isinstance(path, str): + path = Path(path) + path.mkdir(parents=True, exist_ok=True) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/installation_store.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/installation_store.py new file mode 100644 index 0000000..8003ba4 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/installation_store.py @@ -0,0 +1,96 @@ +"""Slack installation data store + +Refer to https://slack.dev/python-slack-sdk/oauth/ for details. +""" +from logging import Logger +from typing import Optional + +from .models.bot import Bot +from .models.installation import Installation + + +class InstallationStore: + """The installation store interface. + + The minimum required methods are: + + * save(installation) + * find_installation(enterprise_id, team_id, user_id, is_enterprise_install) + + If you would like to properly handle app uninstallations and token revocations, + the following methods should be implemented. + + * delete_installation(enterprise_id, team_id, user_id) + * delete_all(enterprise_id, team_id) + + If your app needs only bot scope installations, the simpler way to implement would be: + + * save(installation) + * find_bot(enterprise_id, team_id, is_enterprise_install) + * delete_bot(enterprise_id, team_id) + * delete_all(enterprise_id, team_id) + """ + + @property + def logger(self) -> Logger: + raise NotImplementedError() + + def save(self, installation: Installation): + """Saves an installation data""" + raise NotImplementedError() + + def save_bot(self, bot: Bot): + """Saves a bot installation data""" + raise NotImplementedError() + + def find_bot( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Bot]: + """Finds a bot scope installation per workspace / org""" + raise NotImplementedError() + + def find_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Installation]: + """Finds a relevant installation for the given IDs. + If the user_id is absent, this method may return the latest installation in the workspace / org. + """ + raise NotImplementedError() + + def delete_bot( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + ) -> None: + """Deletes a bot scope installation per workspace / org""" + raise NotImplementedError() + + def delete_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + ) -> None: + """Deletes an installation that matches the given IDs""" + raise NotImplementedError() + + def delete_all( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + ): + """Deletes all installation data for the given workspace / org""" + self.delete_bot(enterprise_id=enterprise_id, team_id=team_id) + self.delete_installation(enterprise_id=enterprise_id, team_id=team_id) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/internals.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/internals.py new file mode 100644 index 0000000..f0b31d3 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/internals.py @@ -0,0 +1,32 @@ +import platform +import datetime + +(major, minor, patch) = platform.python_version_tuple() +is_python_3_6: bool = int(major) == 3 and int(minor) >= 6 + +utc_timezone = datetime.timezone.utc + + +def _from_iso_format_to_datetime(iso_datetime_str: str) -> datetime.datetime: + if is_python_3_6: + elements = iso_datetime_str.split(" ") + ymd = elements[0].split("-") + hms = elements[1].split(":") + return datetime.datetime( + int(ymd[0]), + int(ymd[1]), + int(ymd[2]), + int(hms[0]), + int(hms[1]), + int(hms[2]), + 0, + utc_timezone, + ) + else: + if "+" not in iso_datetime_str: + iso_datetime_str += "+00:00" + return datetime.datetime.fromisoformat(iso_datetime_str) + + +def _from_iso_format_to_unix_timestamp(iso_datetime_str: str) -> float: + return _from_iso_format_to_datetime(iso_datetime_str).timestamp() diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/models/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/models/__init__.py new file mode 100644 index 0000000..d65b147 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/models/__init__.py @@ -0,0 +1,7 @@ +from .bot import Bot +from .installation import Installation + +__all__ = [ + "Bot", + "Installation", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/models/bot.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/models/bot.py new file mode 100644 index 0000000..36be386 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/models/bot.py @@ -0,0 +1,126 @@ +import re +from datetime import datetime # type: ignore +from time import time +from typing import Optional, Union, Dict, Any, Sequence + +from slack_sdk.oauth.installation_store.internals import ( + _from_iso_format_to_unix_timestamp, +) + + +class Bot: + app_id: Optional[str] + enterprise_id: Optional[str] + enterprise_name: Optional[str] + team_id: Optional[str] + team_name: Optional[str] + bot_token: str + bot_id: str + bot_user_id: str + bot_scopes: Sequence[str] + # only when token rotation is enabled + bot_refresh_token: Optional[str] + # only when token rotation is enabled + bot_token_expires_at: Optional[int] + is_enterprise_install: bool + installed_at: float + + custom_values: Dict[str, Any] + + def __init__( + self, + *, + app_id: Optional[str] = None, + # org / workspace + enterprise_id: Optional[str] = None, + enterprise_name: Optional[str] = None, + team_id: Optional[str] = None, + team_name: Optional[str] = None, + # bot + bot_token: str, + bot_id: str, + bot_user_id: str, + bot_scopes: Union[str, Sequence[str]] = "", + # only when token rotation is enabled + bot_refresh_token: Optional[str] = None, + # only when token rotation is enabled + bot_token_expires_in: Optional[int] = None, + # only for duplicating this object + # only when token rotation is enabled + bot_token_expires_at: Optional[Union[int, datetime, str]] = None, + is_enterprise_install: Optional[bool] = False, + # timestamps + # The expected value type is float but the internals handle other types too + # for str values, we support only ISO datetime format. + installed_at: Union[float, datetime, str], + # custom values + custom_values: Optional[Dict[str, Any]] = None, + ): + self.app_id = app_id + self.enterprise_id = enterprise_id + self.enterprise_name = enterprise_name + self.team_id = team_id + self.team_name = team_name + + self.bot_token = bot_token + self.bot_id = bot_id + self.bot_user_id = bot_user_id + if isinstance(bot_scopes, str): + self.bot_scopes = bot_scopes.split(",") if len(bot_scopes) > 0 else [] + else: + self.bot_scopes = bot_scopes + self.bot_refresh_token = bot_refresh_token + if bot_token_expires_at is not None: + if type(bot_token_expires_at) == datetime: + self.bot_token_expires_at = int(bot_token_expires_at.timestamp()) # type: ignore + elif type(bot_token_expires_at) == str and not re.match("^\\d+$", bot_token_expires_at): + self.bot_token_expires_at = int(_from_iso_format_to_unix_timestamp(bot_token_expires_at)) + else: + self.bot_token_expires_at = int(bot_token_expires_at) + elif bot_token_expires_in is not None: + self.bot_token_expires_at = int(time()) + bot_token_expires_in + else: + self.bot_token_expires_at = None + self.is_enterprise_install = is_enterprise_install or False + + if type(installed_at) == float: + self.installed_at = installed_at # type: ignore + elif type(installed_at) == datetime: + self.installed_at = installed_at.timestamp() # type: ignore + elif type(installed_at) == str: + if re.match("^\\d+.\\d+$", installed_at): + self.installed_at = float(installed_at) + else: + self.installed_at = _from_iso_format_to_unix_timestamp(installed_at) + else: + raise ValueError(f"Unsupported data format for installed_at {installed_at}") + + self.custom_values = custom_values if custom_values is not None else {} + + def set_custom_value(self, name: str, value: Any): + self.custom_values[name] = value + + def get_custom_value(self, name: str) -> Optional[Any]: + return self.custom_values.get(name) + + def to_dict(self) -> Dict[str, Any]: + standard_values = { + "app_id": self.app_id, + "enterprise_id": self.enterprise_id, + "enterprise_name": self.enterprise_name, + "team_id": self.team_id, + "team_name": self.team_name, + "bot_token": self.bot_token, + "bot_id": self.bot_id, + "bot_user_id": self.bot_user_id, + "bot_scopes": ",".join(self.bot_scopes) if self.bot_scopes else None, + "bot_refresh_token": self.bot_refresh_token, + "bot_token_expires_at": datetime.utcfromtimestamp(self.bot_token_expires_at) + if self.bot_token_expires_at is not None + else None, + "is_enterprise_install": self.is_enterprise_install, + "installed_at": datetime.utcfromtimestamp(self.installed_at), + } + # prioritize standard_values over custom_values + # when the same keys exist in both + return {**self.custom_values, **standard_values} diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/models/installation.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/models/installation.py new file mode 100644 index 0000000..e55cf2b --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/models/installation.py @@ -0,0 +1,217 @@ +import re +from datetime import datetime # type: ignore +from time import time +from typing import Optional, Union, Dict, Any, Sequence + +from slack_sdk.oauth.installation_store.internals import ( + _from_iso_format_to_unix_timestamp, +) +from slack_sdk.oauth.installation_store.models.bot import Bot + + +class Installation: + app_id: Optional[str] + enterprise_id: Optional[str] + enterprise_name: Optional[str] + enterprise_url: Optional[str] + team_id: Optional[str] + team_name: Optional[str] + bot_token: Optional[str] + bot_id: Optional[str] + bot_user_id: Optional[str] + bot_scopes: Optional[Sequence[str]] + bot_refresh_token: Optional[str] # only when token rotation is enabled + # only when token rotation is enabled + # Unix time (seconds): only when token rotation is enabled + bot_token_expires_at: Optional[int] + user_id: str + user_token: Optional[str] + user_scopes: Optional[Sequence[str]] + user_refresh_token: Optional[str] # only when token rotation is enabled + # Unix time (seconds): only when token rotation is enabled + user_token_expires_at: Optional[int] + incoming_webhook_url: Optional[str] + incoming_webhook_channel: Optional[str] + incoming_webhook_channel_id: Optional[str] + incoming_webhook_configuration_url: Optional[str] + is_enterprise_install: bool + token_type: Optional[str] + installed_at: float + + custom_values: Dict[str, Any] + + def __init__( + self, + *, + app_id: Optional[str] = None, + # org / workspace + enterprise_id: Optional[str] = None, + enterprise_name: Optional[str] = None, + enterprise_url: Optional[str] = None, + team_id: Optional[str] = None, + team_name: Optional[str] = None, + # bot + bot_token: Optional[str] = None, + bot_id: Optional[str] = None, + bot_user_id: Optional[str] = None, + bot_scopes: Union[str, Sequence[str]] = "", + bot_refresh_token: Optional[str] = None, # only when token rotation is enabled + # only when token rotation is enabled + bot_token_expires_in: Optional[int] = None, + # only for duplicating this object + # only when token rotation is enabled + bot_token_expires_at: Optional[Union[int, datetime, str]] = None, + # installer + user_id: str, + user_token: Optional[str] = None, + user_scopes: Union[str, Sequence[str]] = "", + user_refresh_token: Optional[str] = None, # only when token rotation is enabled + # only when token rotation is enabled + user_token_expires_in: Optional[int] = None, + # only for duplicating this object + # only when token rotation is enabled + user_token_expires_at: Optional[Union[int, datetime, str]] = None, + # incoming webhook + incoming_webhook_url: Optional[str] = None, + incoming_webhook_channel: Optional[str] = None, + incoming_webhook_channel_id: Optional[str] = None, + incoming_webhook_configuration_url: Optional[str] = None, + # org app + is_enterprise_install: Optional[bool] = False, + token_type: Optional[str] = None, + # timestamps + # The expected value type is float but the internals handle other types too + # for str values, we supports only ISO datetime format. + installed_at: Optional[Union[float, datetime, str]] = None, + # custom values + custom_values: Optional[Dict[str, Any]] = None, + ): + self.app_id = app_id + self.enterprise_id = enterprise_id + self.enterprise_name = enterprise_name + self.enterprise_url = enterprise_url + self.team_id = team_id + self.team_name = team_name + self.bot_token = bot_token + self.bot_id = bot_id + self.bot_user_id = bot_user_id + if isinstance(bot_scopes, str): + self.bot_scopes = bot_scopes.split(",") if len(bot_scopes) > 0 else [] + else: + self.bot_scopes = bot_scopes + self.bot_refresh_token = bot_refresh_token + if bot_token_expires_at is not None: + if type(bot_token_expires_at) == datetime: + ts: float = bot_token_expires_at.timestamp() # type: ignore + self.bot_token_expires_at = int(ts) + elif type(bot_token_expires_at) == str and not re.match("^\\d+$", bot_token_expires_at): + self.bot_token_expires_at = int(_from_iso_format_to_unix_timestamp(bot_token_expires_at)) + else: + self.bot_token_expires_at = bot_token_expires_at # type: ignore + elif bot_token_expires_in is not None: + self.bot_token_expires_at = int(time()) + bot_token_expires_in + else: + self.bot_token_expires_at = None + + self.user_id = user_id + self.user_token = user_token + if isinstance(user_scopes, str): + self.user_scopes = user_scopes.split(",") if len(user_scopes) > 0 else [] + else: + self.user_scopes = user_scopes + self.user_refresh_token = user_refresh_token + if user_token_expires_at is not None: + if type(user_token_expires_at) == datetime: + ts: float = user_token_expires_at.timestamp() # type: ignore + self.user_token_expires_at = int(ts) + elif type(user_token_expires_at) == str and not re.match("^\\d+$", user_token_expires_at): + self.user_token_expires_at = int(_from_iso_format_to_unix_timestamp(user_token_expires_at)) + else: + self.user_token_expires_at = user_token_expires_at # type: ignore + elif user_token_expires_in is not None: + self.user_token_expires_at = int(time()) + user_token_expires_in + else: + self.user_token_expires_at = None + + self.incoming_webhook_url = incoming_webhook_url + self.incoming_webhook_channel = incoming_webhook_channel + self.incoming_webhook_channel_id = incoming_webhook_channel_id + self.incoming_webhook_configuration_url = incoming_webhook_configuration_url + + self.is_enterprise_install = is_enterprise_install or False + self.token_type = token_type + + if installed_at is None: + self.installed_at = datetime.now().timestamp() + elif type(installed_at) == float: + self.installed_at = installed_at # type: ignore + elif type(installed_at) == datetime: + self.installed_at = installed_at.timestamp() # type: ignore + elif type(installed_at) == str: + if re.match("^\\d+.\\d+$", installed_at): + self.installed_at = float(installed_at) + else: + self.installed_at = _from_iso_format_to_unix_timestamp(installed_at) + else: + raise ValueError(f"Unsupported data format for installed_at {installed_at}") + + self.custom_values = custom_values if custom_values is not None else {} + + def to_bot(self) -> Bot: + return Bot( + app_id=self.app_id, + enterprise_id=self.enterprise_id, + enterprise_name=self.enterprise_name, + team_id=self.team_id, + team_name=self.team_name, + bot_token=self.bot_token, + bot_id=self.bot_id, + bot_user_id=self.bot_user_id, + bot_scopes=self.bot_scopes, + bot_refresh_token=self.bot_refresh_token, + bot_token_expires_at=self.bot_token_expires_at, + is_enterprise_install=self.is_enterprise_install, + installed_at=self.installed_at, + custom_values=self.custom_values, + ) + + def set_custom_value(self, name: str, value: Any): + self.custom_values[name] = value + + def get_custom_value(self, name: str) -> Optional[Any]: + return self.custom_values.get(name) + + def to_dict(self) -> Dict[str, Any]: + standard_values = { + "app_id": self.app_id, + "enterprise_id": self.enterprise_id, + "enterprise_name": self.enterprise_name, + "enterprise_url": self.enterprise_url, + "team_id": self.team_id, + "team_name": self.team_name, + "bot_token": self.bot_token, + "bot_id": self.bot_id, + "bot_user_id": self.bot_user_id, + "bot_scopes": ",".join(self.bot_scopes) if self.bot_scopes else None, + "bot_refresh_token": self.bot_refresh_token, + "bot_token_expires_at": datetime.utcfromtimestamp(self.bot_token_expires_at) + if self.bot_token_expires_at is not None + else None, + "user_id": self.user_id, + "user_token": self.user_token, + "user_scopes": ",".join(self.user_scopes) if self.user_scopes else None, + "user_refresh_token": self.user_refresh_token, + "user_token_expires_at": datetime.utcfromtimestamp(self.user_token_expires_at) + if self.user_token_expires_at is not None + else None, + "incoming_webhook_url": self.incoming_webhook_url, + "incoming_webhook_channel": self.incoming_webhook_channel, + "incoming_webhook_channel_id": self.incoming_webhook_channel_id, + "incoming_webhook_configuration_url": self.incoming_webhook_configuration_url, + "is_enterprise_install": self.is_enterprise_install, + "token_type": self.token_type, + "installed_at": datetime.utcfromtimestamp(self.installed_at), + } + # prioritize standard_values over custom_values + # when the same keys exist in both + return {**self.custom_values, **standard_values} diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/sqlalchemy/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/sqlalchemy/__init__.py new file mode 100644 index 0000000..71e35c3 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/sqlalchemy/__init__.py @@ -0,0 +1,356 @@ +import logging +from logging import Logger +from typing import Optional + +import sqlalchemy +from sqlalchemy import ( + Table, + Column, + Integer, + String, + DateTime, + Index, + and_, + desc, + MetaData, +) +from sqlalchemy.engine import Engine +from sqlalchemy.sql.sqltypes import Boolean + +from slack_sdk.oauth.installation_store.installation_store import InstallationStore +from slack_sdk.oauth.installation_store.models.bot import Bot +from slack_sdk.oauth.installation_store.models.installation import Installation + + +class SQLAlchemyInstallationStore(InstallationStore): + default_bots_table_name: str = "slack_bots" + default_installations_table_name: str = "slack_installations" + + client_id: str + engine: Engine + metadata: MetaData + installations: Table + + @classmethod + def build_installations_table(cls, metadata: MetaData, table_name: str) -> Table: + return sqlalchemy.Table( + table_name, + metadata, + Column("id", Integer, primary_key=True, autoincrement=True), + Column("client_id", String(32), nullable=False), + Column("app_id", String(32), nullable=False), + Column("enterprise_id", String(32)), + Column("enterprise_name", String(200)), + Column("enterprise_url", String(200)), + Column("team_id", String(32)), + Column("team_name", String(200)), + Column("bot_token", String(200)), + Column("bot_id", String(32)), + Column("bot_user_id", String(32)), + Column("bot_scopes", String(1000)), + Column("bot_refresh_token", String(200)), # added in v3.8.0 + Column("bot_token_expires_at", DateTime), # added in v3.8.0 + Column("user_id", String(32), nullable=False), + Column("user_token", String(200)), + Column("user_scopes", String(1000)), + Column("user_refresh_token", String(200)), # added in v3.8.0 + Column("user_token_expires_at", DateTime), # added in v3.8.0 + Column("incoming_webhook_url", String(200)), + Column("incoming_webhook_channel", String(200)), + Column("incoming_webhook_channel_id", String(200)), + Column("incoming_webhook_configuration_url", String(200)), + Column("is_enterprise_install", Boolean, default=False, nullable=False), + Column("token_type", String(32)), + Column( + "installed_at", + DateTime, + nullable=False, + default=sqlalchemy.sql.func.now(), # type: ignore + ), + Index( + f"{table_name}_idx", + "client_id", + "enterprise_id", + "team_id", + "user_id", + "installed_at", + ), + ) + + @classmethod + def build_bots_table(cls, metadata: MetaData, table_name: str) -> Table: + return Table( + table_name, + metadata, + Column("id", Integer, primary_key=True, autoincrement=True), + Column("client_id", String(32), nullable=False), + Column("app_id", String(32), nullable=False), + Column("enterprise_id", String(32)), + Column("enterprise_name", String(200)), + Column("team_id", String(32)), + Column("team_name", String(200)), + Column("bot_token", String(200)), + Column("bot_id", String(32)), + Column("bot_user_id", String(32)), + Column("bot_scopes", String(1000)), + Column("bot_refresh_token", String(200)), # added in v3.8.0 + Column("bot_token_expires_at", DateTime), # added in v3.8.0 + Column("is_enterprise_install", Boolean, default=False, nullable=False), + Column( + "installed_at", + DateTime, + nullable=False, + default=sqlalchemy.sql.func.now(), # type: ignore + ), + Index( + f"{table_name}_idx", + "client_id", + "enterprise_id", + "team_id", + "installed_at", + ), + ) + + def __init__( + self, + client_id: str, + engine: Engine, + bots_table_name: str = default_bots_table_name, + installations_table_name: str = default_installations_table_name, + logger: Logger = logging.getLogger(__name__), + ): + self.metadata = sqlalchemy.MetaData() + self.bots = self.build_bots_table(metadata=self.metadata, table_name=bots_table_name) + self.installations = self.build_installations_table(metadata=self.metadata, table_name=installations_table_name) + self.client_id = client_id + self._logger = logger + self.engine = engine + + def create_tables(self): + self.metadata.create_all(self.engine) + + @property + def logger(self) -> Logger: + return self._logger + + def save(self, installation: Installation): + with self.engine.begin() as conn: + i = installation.to_dict() + i["client_id"] = self.client_id + + i_column = self.installations.c + installations_rows = conn.execute( + sqlalchemy.select([i_column.id]) + .where( + and_( + i_column.client_id == self.client_id, + i_column.enterprise_id == installation.enterprise_id, + i_column.team_id == installation.team_id, + i_column.installed_at == i.get("installed_at"), + ) + ) + .limit(1) + ) + installations_row_id: Optional[str] = None + for row in installations_rows: + installations_row_id = row["id"] + if installations_row_id is None: + conn.execute(self.installations.insert(), i) + else: + update_statement = self.installations.update().where(i_column.id == installations_row_id).values(**i) + conn.execute(update_statement, i) + + # bots + self.save_bot(installation.to_bot()) + + def save_bot(self, bot: Bot): + with self.engine.begin() as conn: + # bots + b = bot.to_dict() + b["client_id"] = self.client_id + + b_column = self.bots.c + bots_rows = conn.execute( + sqlalchemy.select([b_column.id]) + .where( + and_( + b_column.client_id == self.client_id, + b_column.enterprise_id == bot.enterprise_id, + b_column.team_id == bot.team_id, + b_column.installed_at == b.get("installed_at"), + ) + ) + .limit(1) + ) + bots_row_id: Optional[str] = None + for row in bots_rows: + bots_row_id = row["id"] + if bots_row_id is None: + conn.execute(self.bots.insert(), b) + else: + update_statement = self.bots.update().where(b_column.id == bots_row_id).values(**b) + conn.execute(update_statement, b) + + def find_bot( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Bot]: + if is_enterprise_install or team_id is None: + team_id = None + + c = self.bots.c + query = ( + self.bots.select() + .where( + and_( + c.client_id == self.client_id, + c.enterprise_id == enterprise_id, + c.team_id == team_id, + ) + ) + .order_by(desc(c.installed_at)) + .limit(1) + ) + + with self.engine.connect() as conn: + result: object = conn.execute(query) + for row in result: # type: ignore + return Bot( + app_id=row["app_id"], + enterprise_id=row["enterprise_id"], + enterprise_name=row["enterprise_name"], + team_id=row["team_id"], + team_name=row["team_name"], + bot_token=row["bot_token"], + bot_id=row["bot_id"], + bot_user_id=row["bot_user_id"], + bot_scopes=row["bot_scopes"], + bot_refresh_token=row["bot_refresh_token"], + bot_token_expires_at=row["bot_token_expires_at"], + is_enterprise_install=row["is_enterprise_install"], + installed_at=row["installed_at"], + ) + return None + + def find_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Installation]: + if is_enterprise_install or team_id is None: + team_id = None + + c = self.installations.c + where_clause = and_(c.enterprise_id == enterprise_id, c.team_id == team_id) + if user_id is not None: + where_clause = and_( + c.client_id == self.client_id, + c.enterprise_id == enterprise_id, + c.team_id == team_id, + c.user_id == user_id, + ) + + query = self.installations.select().where(where_clause).order_by(desc(c.installed_at)).limit(1) + + installation: Optional[Installation] = None + with self.engine.connect() as conn: + result: object = conn.execute(query) + for row in result: # type: ignore + installation = Installation( + app_id=row["app_id"], + enterprise_id=row["enterprise_id"], + enterprise_name=row["enterprise_name"], + enterprise_url=row["enterprise_url"], + team_id=row["team_id"], + team_name=row["team_name"], + bot_token=row["bot_token"], + bot_id=row["bot_id"], + bot_user_id=row["bot_user_id"], + bot_scopes=row["bot_scopes"], + bot_refresh_token=row["bot_refresh_token"], + bot_token_expires_at=row["bot_token_expires_at"], + user_id=row["user_id"], + user_token=row["user_token"], + user_scopes=row["user_scopes"], + user_refresh_token=row["user_refresh_token"], + user_token_expires_at=row["user_token_expires_at"], + # Only the incoming webhook issued in the latest installation is set in this logic + incoming_webhook_url=row["incoming_webhook_url"], + incoming_webhook_channel=row["incoming_webhook_channel"], + incoming_webhook_channel_id=row["incoming_webhook_channel_id"], + incoming_webhook_configuration_url=row["incoming_webhook_configuration_url"], + is_enterprise_install=row["is_enterprise_install"], + token_type=row["token_type"], + installed_at=row["installed_at"], + ) + + if user_id is not None and installation is not None: + # Retrieve the latest bot token, just in case + # See also: https://github.com/slackapi/bolt-python/issues/664 + where_clause = and_( + c.client_id == self.client_id, + c.enterprise_id == enterprise_id, + c.team_id == team_id, + c.bot_token.is_not(None), # the latest one that has a bot token + ) + query = self.installations.select().where(where_clause).order_by(desc(c.installed_at)).limit(1) + with self.engine.connect() as conn: + result: object = conn.execute(query) + for row in result: # type: ignore + installation.bot_token = row["bot_token"] + installation.bot_id = row["bot_id"] + installation.bot_user_id = row["bot_user_id"] + installation.bot_scopes = row["bot_scopes"] + installation.bot_refresh_token = row["bot_refresh_token"] + installation.bot_token_expires_at = row["bot_token_expires_at"] + + return installation + + def delete_bot(self, *, enterprise_id: Optional[str], team_id: Optional[str]) -> None: + table = self.bots + c = table.c + with self.engine.begin() as conn: + deletion = table.delete().where( + and_( + c.client_id == self.client_id, + c.enterprise_id == enterprise_id, + c.team_id == team_id, + ) + ) + conn.execute(deletion) + + def delete_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + ) -> None: + table = self.installations + c = table.c + with self.engine.begin() as conn: + if user_id is not None: + deletion = table.delete().where( + and_( + c.client_id == self.client_id, + c.enterprise_id == enterprise_id, + c.team_id == team_id, + c.user_id == user_id, + ) + ) + conn.execute(deletion) + else: + deletion = table.delete().where( + and_( + c.client_id == self.client_id, + c.enterprise_id == enterprise_id, + c.team_id == team_id, + ) + ) + conn.execute(deletion) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/sqlite3/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/sqlite3/__init__.py new file mode 100644 index 0000000..75fcd30 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/installation_store/sqlite3/__init__.py @@ -0,0 +1,611 @@ +import logging +import sqlite3 +from logging import Logger +from sqlite3 import Connection +from typing import Optional + +from slack_sdk.oauth.installation_store.async_installation_store import ( + AsyncInstallationStore, +) +from slack_sdk.oauth.installation_store.installation_store import InstallationStore +from slack_sdk.oauth.installation_store.models.bot import Bot +from slack_sdk.oauth.installation_store.models.installation import Installation + + +class SQLite3InstallationStore(InstallationStore, AsyncInstallationStore): + def __init__( + self, + *, + database: str, + client_id: str, + logger: Logger = logging.getLogger(__name__), + ): + self.database = database + self.client_id = client_id + self.init_called = False + self._logger = logger + + @property + def logger(self) -> Logger: + if self._logger is None: + self._logger = logging.getLogger(__name__) + return self._logger + + def init(self): + try: + with sqlite3.connect(database=self.database) as conn: + cur = conn.execute("select count(1) from slack_installations;") + row_num = cur.fetchone()[0] + self.logger.debug(f"{row_num} installations are stored in {self.database}") + except Exception: # skipcq: PYL-W0703 + self.create_tables() + self.init_called = True + + def connect(self) -> Connection: + if not self.init_called: + self.init() + return sqlite3.connect(database=self.database) + + def create_tables(self): + with sqlite3.connect(database=self.database) as conn: + conn.execute( + """ + create table slack_installations ( + id integer primary key autoincrement, + client_id text not null, + app_id text not null, + enterprise_id text not null default '', + enterprise_name text, + enterprise_url text, + team_id text not null default '', + team_name text, + bot_token text not null, + bot_id text not null, + bot_user_id text not null, + bot_scopes text, + bot_refresh_token text, -- since v3.8 + bot_token_expires_at datetime, -- since v3.8 + user_id text not null, + user_token text, + user_scopes text, + user_refresh_token text, -- since v3.8 + user_token_expires_at datetime, -- since v3.8 + incoming_webhook_url text, + incoming_webhook_channel text, + incoming_webhook_channel_id text, + incoming_webhook_configuration_url text, + is_enterprise_install boolean not null default 0, + token_type text, + installed_at datetime not null default current_timestamp + ); + """ + ) + conn.execute( + """ + create index slack_installations_idx on slack_installations ( + client_id, + enterprise_id, + team_id, + user_id, + installed_at + ); + """ + ) + conn.execute( + """ + create table slack_bots ( + id integer primary key autoincrement, + client_id text not null, + app_id text not null, + enterprise_id text not null default '', + enterprise_name text, + team_id text not null default '', + team_name text, + bot_token text not null, + bot_id text not null, + bot_user_id text not null, + bot_scopes text, + bot_refresh_token text, -- since v3.8 + bot_token_expires_at datetime, -- since v3.8 + is_enterprise_install boolean not null default 0, + installed_at datetime not null default current_timestamp + ); + """ + ) + conn.execute( + """ + create index slack_bots_idx on slack_bots ( + client_id, + enterprise_id, + team_id, + installed_at + ); + """ + ) + self.logger.debug(f"Tables have been created (database: {self.database})") + conn.commit() + + async def async_save(self, installation: Installation): + return self.save(installation) + + async def async_save_bot(self, bot: Bot): + return self.save_bot(bot) + + def save(self, installation: Installation): + with self.connect() as conn: + conn.execute( + """ + insert into slack_installations ( + client_id, + app_id, + enterprise_id, + enterprise_name, + enterprise_url, + team_id, + team_name, + bot_token, + bot_id, + bot_user_id, + bot_scopes, + bot_refresh_token, -- since v3.8 + bot_token_expires_at, -- since v3.8 + user_id, + user_token, + user_scopes, + user_refresh_token, -- since v3.8 + user_token_expires_at, -- since v3.8 + incoming_webhook_url, + incoming_webhook_channel, + incoming_webhook_channel_id, + incoming_webhook_configuration_url, + is_enterprise_install, + token_type + ) + values + ( + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ? + ); + """, + [ + self.client_id, + installation.app_id, + installation.enterprise_id or "", + installation.enterprise_name, + installation.enterprise_url, + installation.team_id or "", + installation.team_name, + installation.bot_token, + installation.bot_id, + installation.bot_user_id, + ",".join(installation.bot_scopes), + installation.bot_refresh_token, + installation.bot_token_expires_at, + installation.user_id, + installation.user_token, + ",".join(installation.user_scopes) if installation.user_scopes else None, + installation.user_refresh_token, + installation.user_token_expires_at, + installation.incoming_webhook_url, + installation.incoming_webhook_channel, + installation.incoming_webhook_channel_id, + installation.incoming_webhook_configuration_url, + 1 if installation.is_enterprise_install else 0, + installation.token_type, + ], + ) + self.logger.debug( + f"New rows in slack_bots and slack_installations have been created (database: {self.database})" + ) + conn.commit() + + self.save_bot(installation.to_bot()) + + def save_bot(self, bot: Bot): + with self.connect() as conn: + conn.execute( + """ + insert into slack_bots ( + client_id, + app_id, + enterprise_id, + enterprise_name, + team_id, + team_name, + bot_token, + bot_id, + bot_user_id, + bot_scopes, + bot_refresh_token, -- since v3.8 + bot_token_expires_at, -- since v3.8 + is_enterprise_install + ) + values + ( + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ?, + ? + ); + """, + [ + self.client_id, + bot.app_id, + bot.enterprise_id or "", + bot.enterprise_name, + bot.team_id or "", + bot.team_name, + bot.bot_token, + bot.bot_id, + bot.bot_user_id, + ",".join(bot.bot_scopes), + bot.bot_refresh_token, + bot.bot_token_expires_at, + bot.is_enterprise_install, + ], + ) + conn.commit() + + async def async_find_bot( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Bot]: + return self.find_bot( + enterprise_id=enterprise_id, + team_id=team_id, + is_enterprise_install=is_enterprise_install, + ) + + def find_bot( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Bot]: + if is_enterprise_install or team_id is None: + team_id = "" + + try: + with self.connect() as conn: + cur = conn.execute( + """ + select + app_id, + enterprise_id, + enterprise_name, + team_id, + team_name, + bot_token, + bot_id, + bot_user_id, + bot_scopes, + bot_refresh_token, -- since v3.8 + bot_token_expires_at, -- since v3.8 + is_enterprise_install, + installed_at + from + slack_bots + where + client_id = ? + and + enterprise_id = ? + and + team_id = ? + order by installed_at desc + limit 1 + """, + [self.client_id, enterprise_id or "", team_id or ""], + ) + row = cur.fetchone() + result = "found" if row and len(row) > 0 else "not found" + self.logger.debug(f"find_bot's query result: {result} (database: {self.database})") + if row and len(row) > 0: + bot = Bot( + app_id=row[0], + enterprise_id=row[1], + enterprise_name=row[2], + team_id=row[3], + team_name=row[4], + bot_token=row[5], + bot_id=row[6], + bot_user_id=row[7], + bot_scopes=row[8], + bot_refresh_token=row[9], + bot_token_expires_at=row[10], + is_enterprise_install=row[11], + installed_at=row[12], + ) + return bot + return None + + except Exception as e: # skipcq: PYL-W0703 + message = f"Failed to find bot installation data for enterprise: {enterprise_id}, team: {team_id}: {e}" + if self.logger.level <= logging.DEBUG: + self.logger.exception(message) + else: + self.logger.warning(message) + return None + + async def async_find_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Installation]: + return self.find_installation( + enterprise_id=enterprise_id, + team_id=team_id, + user_id=user_id, + is_enterprise_install=is_enterprise_install, + ) + + def find_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + is_enterprise_install: Optional[bool] = False, + ) -> Optional[Installation]: + if is_enterprise_install or team_id is None: + team_id = "" + + try: + with self.connect() as conn: + row = None + columns = """ + app_id, + enterprise_id, + enterprise_name, + enterprise_url, + team_id, + team_name, + bot_token, + bot_id, + bot_user_id, + bot_scopes, + bot_refresh_token, -- since v3.8 + bot_token_expires_at, -- since v3.8 + user_id, + user_token, + user_scopes, + user_refresh_token, -- since v3.8 + user_token_expires_at, -- since v3.8 + incoming_webhook_url, + incoming_webhook_channel, + incoming_webhook_channel_id, + incoming_webhook_configuration_url, + is_enterprise_install, + token_type, + installed_at + """ + if user_id is None: + cur = conn.execute( + f""" + select + {columns} + from + slack_installations + where + client_id = ? + and + enterprise_id = ? + and + team_id = ? + order by installed_at desc + limit 1 + """, + [self.client_id, enterprise_id or "", team_id], + ) + row = cur.fetchone() + else: + cur = conn.execute( + f""" + select + {columns} + from + slack_installations + where + client_id = ? + and + enterprise_id = ? + and + team_id = ? + and + user_id = ? + order by installed_at desc + limit 1 + """, + [self.client_id, enterprise_id or "", team_id, user_id], + ) + row = cur.fetchone() + + if row is None: + return None + + result = "found" if row and len(row) > 0 else "not found" + self.logger.debug(f"find_installation's query result: {result} (database: {self.database})") + if row and len(row) > 0: + installation = Installation( + app_id=row[0], + enterprise_id=row[1], + enterprise_name=row[2], + enterprise_url=row[3], + team_id=row[4], + team_name=row[5], + bot_token=row[6], + bot_id=row[7], + bot_user_id=row[8], + bot_scopes=row[9], + bot_refresh_token=row[10], + bot_token_expires_at=row[11], + user_id=row[12], + user_token=row[13], + user_scopes=row[14], + user_refresh_token=row[15], + user_token_expires_at=row[16], + incoming_webhook_url=row[17], + incoming_webhook_channel=row[18], + incoming_webhook_channel_id=row[19], + incoming_webhook_configuration_url=row[20], + is_enterprise_install=row[21], + token_type=row[22], + installed_at=row[23], + ) + + if user_id is not None: + # Retrieve the latest bot token, just in case + # See also: https://github.com/slackapi/bolt-python/issues/664 + cur = conn.execute( + """ + select + bot_token, + bot_id, + bot_user_id, + bot_scopes, + bot_refresh_token, + bot_token_expires_at + from + slack_installations + where + client_id = ? + and + enterprise_id = ? + and + team_id = ? + and + bot_token is not null + order by installed_at desc + limit 1 + """, + [self.client_id, enterprise_id or "", team_id], + ) + row = cur.fetchone() + installation.bot_token = row[0] + installation.bot_id = row[1] + installation.bot_user_id = row[2] + installation.bot_scopes = row[3] + installation.bot_refresh_token = row[4] + installation.bot_token_expires_at = row[5] + + return installation + return None + + except Exception as e: # skipcq: PYL-W0703 + message = f"Failed to find an installation data for enterprise: {enterprise_id}, team: {team_id}: {e}" + if self.logger.level <= logging.DEBUG: + self.logger.exception(message) + else: + self.logger.warning(message) + return None + + def delete_bot(self, *, enterprise_id: Optional[str], team_id: Optional[str]) -> None: + try: + with self.connect() as conn: + conn.execute( + """ + delete + from + slack_bots + where + client_id = ? + and + enterprise_id = ? + and + team_id = ? + """, + [self.client_id, enterprise_id or "", team_id or ""], + ) + conn.commit() + except Exception as e: # skipcq: PYL-W0703 + message = f"Failed to delete bot installation data for enterprise: {enterprise_id}, team: {team_id}: {e}" + if self.logger.level <= logging.DEBUG: + self.logger.exception(message) + else: + self.logger.warning(message) + + def delete_installation( + self, + *, + enterprise_id: Optional[str], + team_id: Optional[str], + user_id: Optional[str] = None, + ) -> None: + try: + with self.connect() as conn: + if user_id is None: + conn.execute( + """ + delete + from + slack_installations + where + client_id = ? + and + enterprise_id = ? + and + team_id = ? + """, + [self.client_id, enterprise_id or "", team_id], + ) + else: + conn.execute( + """ + delete + from + slack_installations + where + client_id = ? + and + enterprise_id = ? + and + team_id = ? + and + user_id = ? + """, + [self.client_id, enterprise_id or "", team_id, user_id], + ) + conn.commit() + except Exception as e: # skipcq: PYL-W0703 + message = f"Failed to delete installation data for enterprise: {enterprise_id}, team: {team_id}: {e}" + if self.logger.level <= logging.DEBUG: + self.logger.exception(message) + else: + self.logger.warning(message) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/redirect_uri_page_renderer/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/redirect_uri_page_renderer/__init__.py new file mode 100644 index 0000000..87b943a --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/redirect_uri_page_renderer/__init__.py @@ -0,0 +1,71 @@ +from typing import Optional + + +class RedirectUriPageRenderer: + def __init__( + self, + *, + install_path: str, + redirect_uri_path: str, + success_url: Optional[str] = None, + failure_url: Optional[str] = None, + ): + self.install_path = install_path + self.redirect_uri_path = redirect_uri_path + self.success_url = success_url + self.failure_url = failure_url + + def render_success_page( + self, + app_id: str, + team_id: Optional[str], + is_enterprise_install: Optional[bool] = None, + enterprise_url: Optional[str] = None, + ) -> str: + url = self.success_url + if url is None: + if is_enterprise_install is True and enterprise_url is not None and app_id is not None: + url = f"{enterprise_url}manage/organization/apps/profile/{app_id}/workspaces/add" + elif team_id is None or app_id is None: + url = "slack://open" + else: + url = f"slack://app?team={team_id}&id={app_id}" + browser_url = f"https://app.slack.com/client/{team_id}" + + return f""" + + + + + + +

Thank you!

+

Redirecting to the Slack App... click here. If you use the browser version of Slack, click this link instead.

+ + +""" # noqa: E501 + + def render_failure_page(self, reason: str) -> str: + return f""" + + + + + +

Oops, Something Went Wrong!

+

Please try again from here or contact the app owner (reason: {reason})

+ + +""" diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/__init__.py new file mode 100644 index 0000000..9c96c2e --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/__init__.py @@ -0,0 +1,12 @@ +"""OAuth state parameter data store + +Refer to https://slack.dev/python-slack-sdk/oauth/ for details. +""" +# from .amazon_s3_state_store import AmazonS3OAuthStateStore +from .file import FileOAuthStateStore +from .state_store import OAuthStateStore + +__all__ = [ + "FileOAuthStateStore", + "OAuthStateStore", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/amazon_s3/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/amazon_s3/__init__.py new file mode 100644 index 0000000..a954e27 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/amazon_s3/__init__.py @@ -0,0 +1,69 @@ +import logging +import time +from logging import Logger +from uuid import uuid4 + +from botocore.client import BaseClient + +from ..async_state_store import AsyncOAuthStateStore +from ..state_store import OAuthStateStore + + +class AmazonS3OAuthStateStore(OAuthStateStore, AsyncOAuthStateStore): + def __init__( + self, + *, + s3_client: BaseClient, + bucket_name: str, + expiration_seconds: int, + logger: Logger = logging.getLogger(__name__), + ): + self.s3_client = s3_client + self.bucket_name = bucket_name + self.expiration_seconds = expiration_seconds + self._logger = logger + + @property + def logger(self) -> Logger: + if self._logger is None: + self._logger = logging.getLogger(__name__) + return self._logger + + async def async_issue(self, *args, **kwargs) -> str: + return self.issue(*args, **kwargs) + + async def async_consume(self, state: str) -> bool: + return self.consume(state) + + def issue(self, *args, **kwargs) -> str: + state = str(uuid4()) + response = self.s3_client.put_object( + Bucket=self.bucket_name, + Body=str(time.time()), + Key=state, + ) + self.logger.debug(f"S3 put_object response: {response}") + return state + + def consume(self, state: str) -> bool: + try: + fetch_response = self.s3_client.get_object( + Bucket=self.bucket_name, + Key=state, + ) + self.logger.debug(f"S3 get_object response: {fetch_response}") + body = fetch_response["Body"].read().decode("utf-8") + created = float(body) + expiration = created + self.expiration_seconds + still_valid: bool = time.time() < expiration + + deletion_response = self.s3_client.delete_object( + Bucket=self.bucket_name, + Key=state, + ) + self.logger.debug(f"S3 delete_object response: {deletion_response}") + return still_valid + except Exception as e: # skipcq: PYL-W0703 + message = f"Failed to find any persistent data for state: {state} - {e}" + self.logger.warning(message) + return False diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/async_state_store.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/async_state_store.py new file mode 100644 index 0000000..22e883b --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/async_state_store.py @@ -0,0 +1,13 @@ +from logging import Logger + + +class AsyncOAuthStateStore: + @property + def logger(self) -> Logger: + raise NotImplementedError() + + async def async_issue(self, *args, **kwargs) -> str: + raise NotImplementedError() + + async def async_consume(self, state: str) -> bool: + raise NotImplementedError() diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/file/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/file/__init__.py new file mode 100644 index 0000000..ffd619c --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/file/__init__.py @@ -0,0 +1,71 @@ +import logging +import os +import time +from logging import Logger +from pathlib import Path +from typing import Union, Optional +from uuid import uuid4 + +from ..async_state_store import AsyncOAuthStateStore +from ..state_store import OAuthStateStore + + +class FileOAuthStateStore(OAuthStateStore, AsyncOAuthStateStore): + def __init__( + self, + *, + expiration_seconds: int, + base_dir: str = str(Path.home()) + "/.bolt-app-oauth-state", + client_id: Optional[str] = None, + logger: Logger = logging.getLogger(__name__), + ): + self.expiration_seconds = expiration_seconds + + self.base_dir = base_dir + self.client_id = client_id + if self.client_id is not None: + self.base_dir = f"{self.base_dir}/{self.client_id}" + self._logger = logger + + @property + def logger(self) -> Logger: + if self._logger is None: + self._logger = logging.getLogger(__name__) + return self._logger + + async def async_issue(self, *args, **kwargs) -> str: + return self.issue(*args, **kwargs) + + async def async_consume(self, state: str) -> bool: + return self.consume(state) + + def issue(self, *args, **kwargs) -> str: + state = str(uuid4()) + self._mkdir(self.base_dir) + filepath = f"{self.base_dir}/{state}" + with open(filepath, "w") as f: + content = str(time.time()) + f.write(content) + return state + + def consume(self, state: str) -> bool: + filepath = f"{self.base_dir}/{state}" + try: + with open(filepath) as f: + created = float(f.read()) + expiration = created + self.expiration_seconds + still_valid: bool = time.time() < expiration + + os.remove(filepath) # consume the file by deleting it + return still_valid + + except FileNotFoundError as e: + message = f"Failed to find any persistent data for state: {state} - {e}" + self.logger.warning(message) + return False + + @staticmethod + def _mkdir(path: Union[str, Path]): + if isinstance(path, str): + path = Path(path) + path.mkdir(parents=True, exist_ok=True) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/sqlalchemy/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/sqlalchemy/__init__.py new file mode 100644 index 0000000..6f4a61f --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/sqlalchemy/__init__.py @@ -0,0 +1,75 @@ +import logging +import time +from datetime import datetime # type: ignore +from logging import Logger +from uuid import uuid4 + +from ..state_store import OAuthStateStore +import sqlalchemy +from sqlalchemy import Table, Column, Integer, String, DateTime, and_, MetaData +from sqlalchemy.engine import Engine + + +class SQLAlchemyOAuthStateStore(OAuthStateStore): + default_table_name: str = "slack_oauth_states" + + expiration_seconds: int + engine: Engine + metadata: MetaData + oauth_states: Table + + @classmethod + def build_oauth_states_table(cls, metadata: MetaData, table_name: str) -> Table: + return sqlalchemy.Table( + table_name, + metadata, + metadata, + Column("id", Integer, primary_key=True, autoincrement=True), + Column("state", String(200), nullable=False), + Column("expire_at", DateTime, nullable=False), + ) + + def __init__( + self, + expiration_seconds: int, + engine: Engine, + logger: Logger = logging.getLogger(__name__), + table_name: str = default_table_name, + ): + self.expiration_seconds = expiration_seconds + self._logger = logger + self.engine = engine + self.metadata = MetaData() + self.oauth_states = self.build_oauth_states_table(self.metadata, table_name) + + @property + def logger(self) -> Logger: + if self._logger is None: + self._logger = logging.getLogger(__name__) + return self._logger + + def issue(self, *args, **kwargs) -> str: + state: str = str(uuid4()) + now = datetime.utcfromtimestamp(time.time() + self.expiration_seconds) + with self.engine.begin() as conn: + conn.execute( + self.oauth_states.insert(), + {"state": state, "expire_at": now}, + ) + return state + + def consume(self, state: str) -> bool: + try: + with self.engine.begin() as conn: + c = self.oauth_states.c + query = self.oauth_states.select().where(and_(c.state == state, c.expire_at > datetime.utcnow())) + result = conn.execute(query) + for row in result: + self.logger.debug(f"consume's query result: {row}") + conn.execute(self.oauth_states.delete().where(c.id == row["id"])) + return True + return False + except Exception as e: # skipcq: PYL-W0703 + message = f"Failed to find any persistent data for state: {state} - {e}" + self.logger.warning(message) + return False diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/sqlite3/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/sqlite3/__init__.py new file mode 100644 index 0000000..8775c6e --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/sqlite3/__init__.py @@ -0,0 +1,96 @@ +import logging +import sqlite3 +import time +from logging import Logger +from sqlite3 import Connection +from uuid import uuid4 + +from ..async_state_store import AsyncOAuthStateStore +from ..state_store import OAuthStateStore + + +class SQLite3OAuthStateStore(OAuthStateStore, AsyncOAuthStateStore): + def __init__( + self, + *, + database: str, + expiration_seconds: int, + logger: Logger = logging.getLogger(__name__), + ): + self.database = database + self.expiration_seconds = expiration_seconds + self.init_called = False + self._logger = logger + + @property + def logger(self) -> Logger: + if self._logger is None: + self._logger = logging.getLogger(__name__) + return self._logger + + def init(self): + try: + with sqlite3.connect(database=self.database) as conn: + cur = conn.execute("select count(1) from oauth_states;") + row_num = cur.fetchone()[0] + self.logger.debug(f"{row_num} oauth states are stored in {self.database}") + except Exception: # skipcq: PYL-W0703 + self.create_tables() + self.init_called = True + + def connect(self) -> Connection: + if not self.init_called: + self.init() + return sqlite3.connect(database=self.database) + + def create_tables(self): + with sqlite3.connect(database=self.database) as conn: + conn.execute( + """ + create table oauth_states ( + id integer primary key autoincrement, + state text not null, + expire_at datetime not null + ); + """ + ) + self.logger.debug(f"Tables have been created (database: {self.database})") + conn.commit() + + async def async_issue(self, *args, **kwargs) -> str: + return self.issue(*args, **kwargs) + + async def async_consume(self, state: str) -> bool: + return self.consume(state) + + def issue(self, *args, **kwargs) -> str: + state: str = str(uuid4()) + with self.connect() as conn: + parameters = [ + state, + time.time() + self.expiration_seconds, + ] + conn.execute("insert into oauth_states (state, expire_at) values (?, ?);", parameters) + self.logger.debug(f"issue's insertion result: {parameters} (database: {self.database})") + conn.commit() + return state + + def consume(self, state: str) -> bool: + try: + with self.connect() as conn: + cur = conn.execute( + "select id, state from oauth_states where state = ? and expire_at > ?;", + [state, time.time()], + ) + row = cur.fetchone() + self.logger.debug(f"consume's query result: {row} (database: {self.database})") + if row and len(row) > 0: + id = row[0] # skipcq: PYL-W0622 + conn.execute("delete from oauth_states where id = ?;", [id]) + conn.commit() + return True + return False + except Exception as e: # skipcq: PYL-W0703 + message = f"Failed to find any persistent data for state: {state} - {e}" + self.logger.warning(message) + return False diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/state_store.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/state_store.py new file mode 100644 index 0000000..78fba7b --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_store/state_store.py @@ -0,0 +1,13 @@ +from logging import Logger + + +class OAuthStateStore: + @property + def logger(self) -> Logger: + raise NotImplementedError() + + def issue(self, *args, **kwargs) -> str: + raise NotImplementedError() + + def consume(self, state: str) -> bool: + raise NotImplementedError() diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_utils/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_utils/__init__.py new file mode 100644 index 0000000..c4f821f --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/state_utils/__init__.py @@ -0,0 +1,41 @@ +from typing import Optional, Dict, Sequence, Union + + +class OAuthStateUtils: + cookie_name: str + expiration_seconds: int + + default_cookie_name: str = "slack-app-oauth-state" + default_expiration_seconds: int = 60 * 10 # 10 minutes + + def __init__( + self, + *, + cookie_name: str = default_cookie_name, + expiration_seconds: int = default_expiration_seconds, + ): + self.cookie_name = cookie_name + self.expiration_seconds = expiration_seconds + + def build_set_cookie_for_new_state(self, state: str) -> str: + return f"{self.cookie_name}={state}; " "Secure; " "HttpOnly; " "Path=/; " f"Max-Age={self.expiration_seconds}" + + def build_set_cookie_for_deletion(self) -> str: + return f"{self.cookie_name}=deleted; " "Secure; " "HttpOnly; " "Path=/; " "Expires=Thu, 01 Jan 1970 00:00:00 GMT" + + def is_valid_browser( + self, + state: Optional[str], + request_headers: Dict[str, Union[str, Sequence[str]]], + ) -> bool: + if state is None or request_headers is None or request_headers.get("cookie", None) is None: + return False + cookies = request_headers["cookie"] + if isinstance(cookies, str): + cookies = [cookies] + for cookie in cookies: + values = cookie.split(";") + for value in values: + if value.strip() == f"{self.cookie_name}={state}": + return True + return False diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/token_rotation/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/token_rotation/__init__.py new file mode 100644 index 0000000..5915afe --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/token_rotation/__init__.py @@ -0,0 +1,5 @@ +from .rotator import TokenRotator + +__all__ = [ + "TokenRotator", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/token_rotation/async_rotator.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/token_rotation/async_rotator.py new file mode 100644 index 0000000..c26e31b --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/token_rotation/async_rotator.py @@ -0,0 +1,142 @@ +from time import time +from typing import Optional + +from slack_sdk.errors import SlackApiError, SlackTokenRotationError +from slack_sdk.web.async_client import AsyncWebClient +from slack_sdk.oauth.installation_store import Installation, Bot + + +class AsyncTokenRotator: + client: AsyncWebClient + client_id: str + client_secret: str + + def __init__( + self, + *, + client_id: str, + client_secret: str, + client: Optional[AsyncWebClient] = None, + ): + self.client = client if client is not None else AsyncWebClient(token=None) + self.client_id = client_id + self.client_secret = client_secret + + async def perform_token_rotation( # type: ignore + self, + *, + installation: Installation, + minutes_before_expiration: int = 120, # 2 hours by default + ) -> Optional[Installation]: # type: ignore + """Performs token rotation if the underlying tokens (bot / user) are expired / expiring. + + Args: + installation: the current installation data + minutes_before_expiration: the minutes before the token expiration + + Returns: + None if no rotation is necessary for now. + """ + + # TODO: make the following two calls in parallel for better performance + + # bot + rotated_bot: Optional[Bot] = await self.perform_bot_token_rotation( # type: ignore + bot=installation.to_bot(), + minutes_before_expiration=minutes_before_expiration, + ) + + # user + rotated_installation = await self.perform_user_token_rotation( + installation=installation, + minutes_before_expiration=minutes_before_expiration, + ) + + if rotated_bot is not None: + if rotated_installation is None: + rotated_installation = Installation(**installation.to_dict()) # type: ignore + rotated_installation.bot_token = rotated_bot.bot_token + rotated_installation.bot_refresh_token = rotated_bot.bot_refresh_token + rotated_installation.bot_token_expires_at = rotated_bot.bot_token_expires_at + + return rotated_installation # type: ignore + + async def perform_bot_token_rotation( # type: ignore + self, + *, + bot: Bot, + minutes_before_expiration: int = 120, # 2 hours by default + ) -> Optional[Bot]: + """Performs bot token rotation if the underlying bot token is expired / expiring. + + Args: + bot: the current bot installation data + minutes_before_expiration: the minutes before the token expiration + + Returns: + None if no rotation is necessary for now. + """ + if bot.bot_token_expires_at is None: + return None + if bot.bot_token_expires_at > time() + minutes_before_expiration * 60: + return None + + try: + refresh_response = await self.client.oauth_v2_access( + client_id=self.client_id, + client_secret=self.client_secret, + grant_type="refresh_token", + refresh_token=bot.bot_refresh_token, + ) + # TODO: error handling + + if refresh_response.get("token_type") != "bot": + return None + + refreshed_bot = Bot(**bot.to_dict()) # type: ignore + refreshed_bot.bot_token = refresh_response.get("access_token") + refreshed_bot.bot_refresh_token = refresh_response.get("refresh_token") + refreshed_bot.bot_token_expires_at = int(time()) + int(refresh_response.get("expires_in")) + return refreshed_bot + + except SlackApiError as e: + raise SlackTokenRotationError(e) + + async def perform_user_token_rotation( # type: ignore + self, + *, + installation: Installation, + minutes_before_expiration: int = 120, # 2 hours by default + ) -> Optional[Bot]: # type: ignore + """Performs user token rotation if the underlying user token is expired / expiring. + + Args: + installation: the current installation data + minutes_before_expiration: the minutes before the token expiration + + Returns: + None if no rotation is necessary for now. + """ + if installation.user_token_expires_at is None: + return None + if installation.user_token_expires_at > time() + minutes_before_expiration * 60: + return None + + try: + refresh_response = await self.client.oauth_v2_access( + client_id=self.client_id, + client_secret=self.client_secret, + grant_type="refresh_token", + refresh_token=installation.user_refresh_token, + ) + if refresh_response.get("token_type") != "user": + return None + + refreshed_installation = Installation(**installation.to_dict()) # type: ignore + refreshed_installation.user_token = refresh_response.get("access_token") + refreshed_installation.user_refresh_token = refresh_response.get("refresh_token") + refreshed_installation.user_token_expires_at = int(time()) + int(refresh_response.get("expires_in")) + return refreshed_installation # type: ignore + + except SlackApiError as e: + raise SlackTokenRotationError(e) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/oauth/token_rotation/rotator.py b/core_service/aws_lambda/project/packages/slack_sdk/oauth/token_rotation/rotator.py new file mode 100644 index 0000000..d3327ce --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/oauth/token_rotation/rotator.py @@ -0,0 +1,135 @@ +from time import time +from typing import Optional + +from slack_sdk.errors import SlackApiError, SlackTokenRotationError +from slack_sdk.web import WebClient +from slack_sdk.oauth.installation_store import Installation, Bot + + +class TokenRotator: + client: WebClient + client_id: str + client_secret: str + + def __init__(self, *, client_id: str, client_secret: str, client: Optional[WebClient] = None): + self.client = client if client is not None else WebClient(token=None) + self.client_id = client_id + self.client_secret = client_secret + + def perform_token_rotation( # type: ignore + self, + *, + installation: Installation, + minutes_before_expiration: int = 120, # 2 hours by default + ) -> Optional[Installation]: + """Performs token rotation if the underlying tokens (bot / user) are expired / expiring. + + Args: + installation: the current installation data + minutes_before_expiration: the minutes before the token expiration + + Returns: + None if no rotation is necessary for now. + """ + + # TODO: make the following two calls in parallel for better performance + + # bot + rotated_bot: Optional[Bot] = self.perform_bot_token_rotation( # type: ignore + bot=installation.to_bot(), + minutes_before_expiration=minutes_before_expiration, + ) + + # user + rotated_installation: Optional[Installation] = self.perform_user_token_rotation( # type: ignore + installation=installation, + minutes_before_expiration=minutes_before_expiration, + ) + + if rotated_bot is not None: + if rotated_installation is None: + rotated_installation = Installation(**installation.to_dict()) # type: ignore + rotated_installation.bot_token = rotated_bot.bot_token + rotated_installation.bot_refresh_token = rotated_bot.bot_refresh_token + rotated_installation.bot_token_expires_at = rotated_bot.bot_token_expires_at + + return rotated_installation + + def perform_bot_token_rotation( # type: ignore + self, + *, + bot: Bot, + minutes_before_expiration: int = 120, # 2 hours by default + ) -> Optional[Bot]: + """Performs bot token rotation if the underlying bot token is expired / expiring. + + Args: + bot: the current bot installation data + minutes_before_expiration: the minutes before the token expiration + + Returns: + None if no rotation is necessary for now. + """ + if bot.bot_token_expires_at is None: + return None + if bot.bot_token_expires_at > time() + minutes_before_expiration * 60: + return None + + try: + refresh_response = self.client.oauth_v2_access( + client_id=self.client_id, + client_secret=self.client_secret, + grant_type="refresh_token", + refresh_token=bot.bot_refresh_token, + ) + if refresh_response.get("token_type") != "bot": + return None + + refreshed_bot = Bot(**bot.to_dict()) # type: ignore + refreshed_bot.bot_token = refresh_response.get("access_token") + refreshed_bot.bot_refresh_token = refresh_response.get("refresh_token") + refreshed_bot.bot_token_expires_at = int(time()) + int(refresh_response.get("expires_in")) + return refreshed_bot + + except SlackApiError as e: + raise SlackTokenRotationError(e) + + def perform_user_token_rotation( # type: ignore + self, + *, + installation: Installation, + minutes_before_expiration: int = 120, # 2 hours by default + ) -> Optional[Installation]: + """Performs user token rotation if the underlying user token is expired / expiring. + + Args: + installation: the current installation data + minutes_before_expiration: the minutes before the token expiration + + Returns: + None if no rotation is necessary for now. + """ + if installation.user_token_expires_at is None: + return None + if installation.user_token_expires_at > time() + minutes_before_expiration * 60: + return None + + try: + refresh_response = self.client.oauth_v2_access( + client_id=self.client_id, + client_secret=self.client_secret, + grant_type="refresh_token", + refresh_token=installation.user_refresh_token, + ) + + if refresh_response.get("token_type") != "user": + return None + + refreshed_installation = Installation(**installation.to_dict()) # type: ignore + refreshed_installation.user_token = refresh_response.get("access_token") + refreshed_installation.user_refresh_token = refresh_response.get("refresh_token") + refreshed_installation.user_token_expires_at = int(time()) + int(refresh_response.get("expires_in")) + return refreshed_installation + + except SlackApiError as e: + raise SlackTokenRotationError(e) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/proxy_env_variable_loader.py b/core_service/aws_lambda/project/packages/slack_sdk/proxy_env_variable_loader.py new file mode 100644 index 0000000..8484369 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/proxy_env_variable_loader.py @@ -0,0 +1,24 @@ +"""Internal module for loading proxy-related env variables""" +import logging +import os +from typing import Optional + +_default_logger = logging.getLogger(__name__) + + +def load_http_proxy_from_env(logger: logging.Logger = _default_logger) -> Optional[str]: + proxy_url = ( + os.environ.get("HTTPS_PROXY") + or os.environ.get("https_proxy") + or os.environ.get("HTTP_PROXY") + or os.environ.get("http_proxy") + ) + if proxy_url is None: + return None + if len(proxy_url.strip()) == 0: + # If the value is an empty string, the intention should be unsetting it + logger.debug("The Slack SDK ignored the proxy env variable as an empty value is set.") + return None + + logger.debug(f"HTTP proxy URL has been loaded from an env variable: {proxy_url}") + return proxy_url diff --git a/daita-app/core-service/functions/handlers/delete_project/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/py.typed similarity index 100% rename from daita-app/core-service/functions/handlers/delete_project/__init__.py rename to core_service/aws_lambda/project/packages/slack_sdk/py.typed diff --git a/core_service/aws_lambda/project/packages/slack_sdk/rtm/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/rtm/__init__.py new file mode 100644 index 0000000..dcfb110 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/rtm/__init__.py @@ -0,0 +1,572 @@ +"""A Python module for interacting with Slack's RTM API.""" + +import asyncio +import collections +import inspect +import logging +import os +import random +import signal +from asyncio import Future +from ssl import SSLContext +from threading import current_thread, main_thread +from typing import Any, Union, Sequence +from typing import Optional, Callable, DefaultDict + +import aiohttp + +import slack_sdk.errors as client_err +from slack_sdk.aiohttp_version_checker import validate_aiohttp_version +from slack_sdk.web.legacy_client import LegacyWebClient as WebClient + + +validate_aiohttp_version(aiohttp.__version__) + + +class RTMClient(object): # skipcq: PYL-R0205 + """An RTMClient allows apps to communicate with the Slack Platform's RTM API. + + The event-driven architecture of this client allows you to simply + link callbacks to their corresponding events. When an event occurs + this client executes your callback while passing along any + information it receives. + + Attributes: + token (str): A string specifying an xoxp or xoxb token. + run_async (bool): A boolean specifying if the client should + be run in async mode. Default is False. + auto_reconnect (bool): When true the client will automatically + reconnect when (not manually) disconnected. Default is True. + ssl (SSLContext): To use SSL support, pass an SSLContext object here. + Default is None. + proxy (str): To use proxy support, pass the string of the proxy server. + e.g. "http://proxy.com" + Authentication credentials can be passed in proxy URL. + e.g. "http://user:pass@some.proxy.com" + Default is None. + timeout (int): The amount of seconds the session should wait before timing out. + Default is 30. + base_url (str): The base url for all HTTP requests. + Note: This is only used in the WebClient. + Default is "https://www.slack.com/api/". + connect_method (str): An string specifying if the client + will connect with `rtm.connect` or `rtm.start`. + Default is `rtm.connect`. + ping_interval (int): automatically send "ping" command every + specified period of seconds. If set to 0, do not send automatically. + Default is 30. + loop (AbstractEventLoop): An event loop provided by asyncio. + If None is specified we attempt to use the current loop + with `get_event_loop`. Default is None. + + Methods: + ping: Sends a ping message over the websocket to Slack. + typing: Sends a typing indicator to the specified channel. + on: Stores and links callbacks to websocket and Slack events. + run_on: Decorator that stores and links callbacks to websocket and Slack events. + start: Starts an RTM Session with Slack. + stop: Closes the websocket connection and ensures it won't reconnect. + + Example: + ```python + import os + from slack import RTMClient + + @RTMClient.run_on(event="message") + def say_hello(**payload): + data = payload['data'] + web_client = payload['web_client'] + if 'Hello' in data['text']: + channel_id = data['channel'] + thread_ts = data['ts'] + user = data['user'] + + web_client.chat_postMessage( + channel=channel_id, + text=f"Hi <@{user}>!", + thread_ts=thread_ts + ) + + slack_token = os.environ["SLACK_API_TOKEN"] + rtm_client = RTMClient(token=slack_token) + rtm_client.start() + ``` + + Note: + The initial state returned when establishing an RTM connection will + be available as the data in payload for the 'open' event. This data is not and + will not be stored on the RTM Client. + + Any attributes or methods prefixed with _underscores are + intended to be "private" internal use only. They may be changed or + removed at anytime. + """ + + _callbacks: DefaultDict = collections.defaultdict(list) + + def __init__( + self, + *, + token: str, + run_async: Optional[bool] = False, + auto_reconnect: Optional[bool] = True, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + timeout: Optional[int] = 30, + base_url: Optional[str] = WebClient.BASE_URL, + connect_method: Optional[str] = None, + ping_interval: Optional[int] = 30, + loop: Optional[asyncio.AbstractEventLoop] = None, + headers: Optional[dict] = {}, + ): + self.token = token.strip() + self.run_async = run_async + self.auto_reconnect = auto_reconnect + self.ssl = ssl + self.proxy = proxy + self.timeout = timeout + self.base_url = base_url + self.connect_method = connect_method + self.ping_interval = ping_interval + self.headers = headers + self._event_loop = loop or asyncio.get_event_loop() + self._web_client = None + self._websocket = None + self._session = None + self._logger = logging.getLogger(__name__) + self._last_message_id = 0 + self._connection_attempts = 0 + self._stopped = False + self._web_client = WebClient( + token=self.token, + base_url=self.base_url, + timeout=self.timeout, + ssl=self.ssl, + proxy=self.proxy, + run_async=self.run_async, + loop=self._event_loop, + session=self._session, + headers=self.headers, + ) + + @staticmethod + def run_on(*, event: str): + """A decorator to store and link a callback to an event.""" + + def decorator(callback): + RTMClient.on(event=event, callback=callback) + return callback + + return decorator + + @classmethod + def on(cls, *, event: str, callback: Callable): + """Stores and links the callback(s) to the event. + + Args: + event (str): A string that specifies a Slack or websocket event. + e.g. 'channel_joined' or 'open' + callback (Callable): Any object or a list of objects that can be called. + e.g. or + [,] + + Raises: + SlackClientError: The specified callback is not callable. + SlackClientError: The callback must accept keyword arguments (**kwargs). + """ + if isinstance(callback, list): + for cb in callback: + cls._validate_callback(cb) + previous_callbacks = cls._callbacks[event] + cls._callbacks[event] = list(set(previous_callbacks + callback)) + else: + cls._validate_callback(callback) + cls._callbacks[event].append(callback) + + def start(self) -> Union[asyncio.Future, Any]: + """Starts an RTM Session with Slack. + + Makes an authenticated call to Slack's RTM API to retrieve + a websocket URL and then connects to the message server. + As events stream-in we run any associated callbacks stored + on the client. + + If 'auto_reconnect' is specified we + retrieve a new url and reconnect any time the connection + is lost unintentionally or an exception is thrown. + + Raises: + SlackApiError: Unable to retrieve RTM URL from Slack. + """ + # Not yet implemented: Add Windows support for graceful shutdowns. + if os.name != "nt" and current_thread() == main_thread(): + signals = (signal.SIGHUP, signal.SIGTERM, signal.SIGINT) + for s in signals: + self._event_loop.add_signal_handler(s, self.stop) + + future: Future[Any] = asyncio.ensure_future(self._connect_and_read(), loop=self._event_loop) + + if self.run_async: + return future + return self._event_loop.run_until_complete(future) + + def stop(self): + """Closes the websocket connection and ensures it won't reconnect. + + If your application outputs the following errors, + call #async_stop() instead and await for the completion on your application side. + + asyncio/base_events.py:641: RuntimeWarning: + coroutine 'ClientWebSocketResponse.close' was never awaited self._ready.clear() + """ + self._logger.debug("The Slack RTMClient is shutting down.") + self._stopped = True + self._close_websocket() + + async def async_stop(self): + """Closes the websocket connection and ensures it won't reconnect.""" + self._logger.debug("The Slack RTMClient is shutting down.") + remaining_futures = self._close_websocket() + for future in remaining_futures: + await future + self._stopped = True + + def send_over_websocket(self, *, payload: dict): + """Sends a message to Slack over the WebSocket connection. + + Note: + The RTM API only supports posting simple messages formatted using + our default message formatting mode. It does not support + attachments or other message formatting modes. For this reason + we recommend users send messages via the Web API methods. + e.g. web_client.chat_postMessage() + + If the message "id" is not specified in the payload, it'll be added. + + Args: + payload (dict): The message to send over the wesocket. + e.g. + { + "id": 1, + "type": "typing", + "channel": "C024BE91L" + } + + Raises: + SlackClientNotConnectedError: Websocket connection is closed. + """ + return asyncio.ensure_future(self._send_json(payload), loop=self._event_loop) + + async def _send_json(self, payload): + if self._websocket is None or self._event_loop is None: + raise client_err.SlackClientNotConnectedError("Websocket connection is closed.") + if "id" not in payload: + payload["id"] = self._next_msg_id() + + return await self._websocket.send_json(payload) + + async def ping(self): + """Sends a ping message over the websocket to Slack. + + Not all web browsers support the WebSocket ping spec, + so the RTM protocol also supports ping/pong messages. + + Raises: + SlackClientNotConnectedError: Websocket connection is closed. + """ + payload = {"id": self._next_msg_id(), "type": "ping"} + await self._send_json(payload=payload) + + async def typing(self, *, channel: str): + """Sends a typing indicator to the specified channel. + + This indicates that this app is currently + writing a message to send to a channel. + + Args: + channel (str): The channel id. e.g. 'C024BE91L' + + Raises: + SlackClientNotConnectedError: Websocket connection is closed. + """ + payload = {"id": self._next_msg_id(), "type": "typing", "channel": channel} + await self._send_json(payload=payload) + + @staticmethod + def _validate_callback(callback): + """Checks if the specified callback is callable and accepts a kwargs param. + + Args: + callback (obj): Any object or a list of objects that can be called. + e.g. + + Raises: + SlackClientError: The specified callback is not callable. + SlackClientError: The callback must accept keyword arguments (**kwargs). + """ + + cb_name = callback.__name__ if hasattr(callback, "__name__") else callback + if not callable(callback): + msg = "The specified callback '{}' is not callable.".format(cb_name) + raise client_err.SlackClientError(msg) + callback_params = inspect.signature(callback).parameters.values() + if not any(param for param in callback_params if param.kind == param.VAR_KEYWORD): + msg = "The callback '{}' must accept keyword arguments (**kwargs).".format(cb_name) + raise client_err.SlackClientError(msg) + + def _next_msg_id(self): + """Retrieves the next message id. + + When sending messages to Slack every event should + have a unique (for that connection) positive integer ID. + + Returns: + An integer representing the message id. e.g. 98 + """ + self._last_message_id += 1 + return self._last_message_id + + async def _connect_and_read(self): + """Retrieves the WS url and connects to Slack's RTM API. + + Makes an authenticated call to Slack's Web API to retrieve + a websocket URL. Then connects to the message server and + reads event messages as they come in. + + If 'auto_reconnect' is specified we + retrieve a new url and reconnect any time the connection + is lost unintentionally or an exception is thrown. + + Raises: + SlackApiError: Unable to retrieve RTM URL from Slack. + websockets.exceptions: Errors thrown by the 'websockets' library. + """ + while not self._stopped: + try: + self._connection_attempts += 1 + async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=self.timeout)) as session: + self._session = session + url, data = await self._retrieve_websocket_info() + async with session.ws_connect( + url, + heartbeat=self.ping_interval, + ssl=self.ssl, + proxy=self.proxy, + ) as websocket: + self._logger.debug("The Websocket connection has been opened.") + self._websocket = websocket + await self._dispatch_event(event="open", data=data) + await self._read_messages() + # The websocket has been disconnected, or self._stopped is True + if not self._stopped and not self.auto_reconnect: + self._logger.warning("Not reconnecting the Websocket because auto_reconnect is False") + return + # No need to wait exponentially here, since the connection was + # established OK, but timed out, or was closed remotely + except ( + client_err.SlackClientNotConnectedError, + client_err.SlackApiError, + # Not yet implemented: Catch websocket exceptions thrown by aiohttp. + ) as exception: + await self._dispatch_event(event="error", data=exception) + error_code = exception.response.get("error", None) if hasattr(exception, "response") else None + if ( + self.auto_reconnect + and not self._stopped + and error_code != "invalid_auth" # "invalid_auth" is unrecoverable + ): + await self._wait_exponentially(exception) + continue + self._logger.exception("The Websocket encountered an error. Closing the connection...") + self._close_websocket() + raise + + async def _read_messages(self): + """Process messages received on the WebSocket connection.""" + while not self._stopped and self._websocket is not None: + try: + # Wait for a message to be received, but timeout after a second so that + # we can check if the socket has been closed, or if self._stopped is + # True + message = await self._websocket.receive(timeout=1) + except asyncio.TimeoutError: + if not self._websocket.closed: + # We didn't receive a message within the timeout interval, but + # aiohttp hasn't closed the socket, so ping responses must still be + # returning + continue + self._logger.warning( + "Websocket was closed (%s).", + self._websocket.close_code if self._websocket else "", + ) + await self._dispatch_event( + event="error", + data=self._websocket.exception() if self._websocket else "", + ) + self._websocket = None + await self._dispatch_event(event="close") + return + + if message.type == aiohttp.WSMsgType.TEXT: + try: + payload = message.json() + event = payload.pop("type", "Unknown") + await self._dispatch_event(event, data=payload) + except Exception as err: # skipcq: PYL-W0703 + data = message.data if message else message + self._logger.info(f"Caught a raised exception ({err}) while dispatching a TEXT message ({data})") + # Raised exceptions here happen in users' code and were just unhandled. + # As they're not intended for closing current WebSocket connection, + # this exception should not be propagated to higher level (#_connect_and_read()). + continue + elif message.type == aiohttp.WSMsgType.ERROR: + self._logger.error("Received an error on the websocket: %r", message) + await self._dispatch_event(event="error", data=message) + elif message.type in ( + aiohttp.WSMsgType.CLOSE, + aiohttp.WSMsgType.CLOSING, + aiohttp.WSMsgType.CLOSED, + ): + self._logger.warning("Websocket was closed.") + self._websocket = None + await self._dispatch_event(event="close") + else: + self._logger.debug("Received unhandled message type: %r", message) + + async def _dispatch_event(self, event, data=None): + """Dispatches the event and executes any associated callbacks. + + Note: To prevent the app from crashing due to callback errors. We + catch all exceptions and send all data to the logger. + + Args: + event (str): The type of event. e.g. 'bot_added' + data (dict): The data Slack sent. e.g. + { + "type": "bot_added", + "bot": { + "id": "B024BE7LH", + "app_id": "A4H1JB4AZ", + "name": "hugbot" + } + } + """ + if self._logger.level <= logging.DEBUG: + self._logger.debug("Received an event: '%s' - %s", event, data) + for callback in self._callbacks[event]: + self._logger.debug( + "Running %s callbacks for event: '%s'", + len(self._callbacks[event]), + event, + ) + try: + if self._stopped and event not in ["close", "error"]: + # Don't run callbacks if client was stopped unless they're + # close/error callbacks. + break + + if inspect.iscoroutinefunction(callback): + await callback(rtm_client=self, web_client=self._web_client, data=data) + else: + if self.run_async is True: + raise client_err.SlackRequestError( + f'The callback "{callback.__name__}" is NOT a coroutine. ' + "Running such with run_async=True is unsupported. " + "Consider adding async/await to the method " + "or going with run_async=False if your app is not really non-blocking." + ) + payload = { + "rtm_client": self, + "web_client": self._web_client, + "data": data, + } + callback(**payload) + except Exception as err: + name = callback.__name__ + module = callback.__module__ + msg = f"When calling '#{name}()' in the '{module}' module the following error was raised: {err}" + self._logger.error(msg) + raise + + async def _retrieve_websocket_info(self): + """Retrieves the WebSocket info from Slack. + + Returns: + A tuple of websocket information. + e.g. + ( + "wss://...", + { + "self": {"id": "U01234ABC","name": "robotoverlord"}, + "team": { + "domain": "exampledomain", + "id": "T123450FP", + "name": "ExampleName" + } + } + ) + + Raises: + SlackApiError: Unable to retrieve RTM URL from Slack. + """ + if self._web_client is None: + self._web_client = WebClient( + token=self.token, + base_url=self.base_url, + timeout=self.timeout, + ssl=self.ssl, + proxy=self.proxy, + run_async=True, + loop=self._event_loop, + session=self._session, + headers=self.headers, + ) + self._logger.debug("Retrieving websocket info.") + use_rtm_start = self.connect_method in ["rtm.start", "rtm_start"] + if self.run_async: + if use_rtm_start: + resp = await self._web_client.rtm_start() + else: + resp = await self._web_client.rtm_connect() + else: + if use_rtm_start: + resp = self._web_client.rtm_start() + else: + resp = self._web_client.rtm_connect() + + url = resp.get("url") # type: ignore + if url is None: + msg = "Unable to retrieve RTM URL from Slack." + raise client_err.SlackApiError(message=msg, response=resp) + return url, resp.data # type: ignore + + async def _wait_exponentially(self, exception, max_wait_time=300): + """Wait exponentially longer for each connection attempt. + + Calculate the number of seconds to wait and then add + a random number of milliseconds to avoid coincidental + synchronized client retries. Wait up to the maximum amount + of wait time specified via 'max_wait_time'. However, + if Slack returned how long to wait use that. + """ + if hasattr(exception, "response"): + wait_time = exception.response.get("headers", {}).get( + "Retry-After", + min((2**self._connection_attempts) + random.random(), max_wait_time), + ) + self._logger.debug("Waiting %s seconds before reconnecting.", wait_time) + await asyncio.sleep(float(wait_time)) + + def _close_websocket(self) -> Sequence[Future]: + """Closes the websocket connection.""" + futures = [] + close_method = getattr(self._websocket, "close", None) + if callable(close_method): + future = asyncio.ensure_future( # skipcq: PYL-E1102 + close_method(), loop=self._event_loop # skipcq: PYL-E1102 + ) # skipcq: PYL-E1102 + futures.append(future) + self._websocket = None + event_f = asyncio.ensure_future(self._dispatch_event(event="close"), loop=self._event_loop) + futures.append(event_f) + return futures diff --git a/core_service/aws_lambda/project/packages/slack_sdk/rtm/v2/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/rtm/v2/__init__.py new file mode 100644 index 0000000..3ddf051 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/rtm/v2/__init__.py @@ -0,0 +1,5 @@ +from slack_sdk.rtm_v2 import RTMClient + +__all__ = [ + "RTMClient", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/rtm_v2/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/rtm_v2/__init__.py new file mode 100644 index 0000000..21f96e3 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/rtm_v2/__init__.py @@ -0,0 +1,391 @@ +"""A Python module for interacting with Slack's RTM API.""" +import inspect +import json +import logging +import time +from concurrent.futures.thread import ThreadPoolExecutor +from logging import Logger +from queue import Queue, Empty +from ssl import SSLContext +from threading import Lock, Event +from typing import Optional, Callable, List, Union + +from slack_sdk.errors import SlackApiError, SlackClientError +from slack_sdk.proxy_env_variable_loader import load_http_proxy_from_env +from slack_sdk.socket_mode.builtin.connection import Connection, ConnectionState +from slack_sdk.socket_mode.interval_runner import IntervalRunner +from slack_sdk.web import WebClient + + +class RTMClient: + token: Optional[str] + bot_id: Optional[str] + default_auto_reconnect_enabled: bool + auto_reconnect_enabled: bool + ssl: Optional[SSLContext] + proxy: Optional[str] + timeout: int + base_url: str + ping_interval: int + logger: Logger + web_client: WebClient + + current_session: Optional[Connection] + current_session_state: Optional[ConnectionState] + wss_uri: Optional[str] + + message_queue: Queue + message_listeners: List[Callable[["RTMClient", dict], None]] + message_processor: IntervalRunner + message_workers: ThreadPoolExecutor + + closed: bool + connect_operation_lock: Lock + + on_message_listeners: List[Callable[[str], None]] + on_error_listeners: List[Callable[[Exception], None]] + on_close_listeners: List[Callable[[int, Optional[str]], None]] + + def __init__( + self, + *, + token: Optional[str] = None, + web_client: Optional[WebClient] = None, + auto_reconnect_enabled: bool = True, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + timeout: int = 30, + base_url: str = WebClient.BASE_URL, + headers: Optional[dict] = None, + ping_interval: int = 5, + concurrency: int = 10, + logger: Optional[logging.Logger] = None, + on_message_listeners: Optional[List[Callable[[str], None]]] = None, + on_error_listeners: Optional[List[Callable[[Exception], None]]] = None, + on_close_listeners: Optional[List[Callable[[int, Optional[str]], None]]] = None, + trace_enabled: bool = False, + all_message_trace_enabled: bool = False, + ping_pong_trace_enabled: bool = False, + ): + self.token = token.strip() if token is not None else None + self.bot_id = None + self.default_auto_reconnect_enabled = auto_reconnect_enabled + # You may want temporarily turn off the auto_reconnect as necessary + self.auto_reconnect_enabled = self.default_auto_reconnect_enabled + self.ssl = ssl + self.proxy = proxy + self.timeout = timeout + self.base_url = base_url + self.headers = headers + self.ping_interval = ping_interval + self.logger = logger or logging.getLogger(__name__) + if self.proxy is None or len(self.proxy.strip()) == 0: + env_variable = load_http_proxy_from_env(self.logger) + if env_variable is not None: + self.proxy = env_variable + + self.web_client = web_client or WebClient( + token=self.token, + base_url=self.base_url, + timeout=self.timeout, + ssl=self.ssl, + proxy=self.proxy, + headers=self.headers, + logger=logger, + ) + + self.on_message_listeners = on_message_listeners or [] + + self.on_error_listeners = on_error_listeners or [] + self.on_close_listeners = on_close_listeners or [] + + self.trace_enabled = trace_enabled + self.all_message_trace_enabled = all_message_trace_enabled + self.ping_pong_trace_enabled = ping_pong_trace_enabled + + self.message_queue = Queue() + + def goodbye_listener(_self, event: dict): + if event.get("type") == "goodbye": + message = "Got a goodbye message. Reconnecting to the server ..." + self.logger.info(message) + self.connect_to_new_endpoint(force=True) + + self.message_listeners = [goodbye_listener] + self.socket_mode_request_listeners = [] + + self.current_session = None + self.current_session_state = ConnectionState() + self.current_session_runner = IntervalRunner(self._run_current_session, 0.1).start() + self.wss_uri = None + + self.current_app_monitor_started = False + self.current_app_monitor = IntervalRunner( + self._monitor_current_session, + self.ping_interval, + ) + + self.closed = False + self.connect_operation_lock = Lock() + + self.message_processor = IntervalRunner(self.process_messages, 0.001).start() + self.message_workers = ThreadPoolExecutor(max_workers=concurrency) + + # -------------------------------------------------------------- + # Decorator to register listeners + # -------------------------------------------------------------- + + def on(self, event_type: str) -> Callable: + """Registers a new event listener. + + Args: + event_type: str representing an event's type (e.g., message, reaction_added) + """ + + def __call__(*args, **kwargs): + func = args[0] + if func is not None: + if isinstance(func, Callable): + name = ( + func.__name__ + if hasattr(func, "__name__") + else f"{func.__class__.__module__}.{func.__class__.__name__}" + ) + inspect_result: inspect.FullArgSpec = inspect.getfullargspec(func) + if inspect_result is not None and len(inspect_result.args) != 2: + actual_args = ", ".join(inspect_result.args) + error = f"The listener '{name}' must accept two args: client, event (actual: {actual_args})" + raise SlackClientError(error) + + def new_message_listener(_self, event: dict): + actual_event_type = event.get("type") + if event.get("bot_id") == self.bot_id: + # SKip the events generated by this bot user + return + # https://github.com/slackapi/python-slack-sdk/issues/533 + if event_type == "*" or (actual_event_type is not None and actual_event_type == event_type): + func(_self, event) + + self.message_listeners.append(new_message_listener) + else: + error = f"The listener '{func}' is not a Callable (actual: {type(func).__name__})" + raise SlackClientError(error) + # Not to cause modification to the decorated method + return func + + return __call__ + + # -------------------------------------------------------------- + # Connections + # -------------------------------------------------------------- + + def is_connected(self) -> bool: + """Returns True if this client is connected.""" + return self.current_session is not None and self.current_session.is_active() + + def issue_new_wss_url(self) -> str: + """Acquires a new WSS URL using rtm.connect API method""" + try: + api_response = self.web_client.rtm_connect() + return api_response["url"] + except SlackApiError as e: + if e.response["error"] == "ratelimited": + delay = int(e.response.headers.get("Retry-After", "30")) # Tier1 + self.logger.info(f"Rate limited. Retrying in {delay} seconds...") + time.sleep(delay) + # Retry to issue a new WSS URL + return self.issue_new_wss_url() + else: + # other errors + self.logger.error(f"Failed to retrieve WSS URL: {e}") + raise e + + def connect_to_new_endpoint(self, force: bool = False): + """Acquires a new WSS URL and tries to connect to the endpoint.""" + with self.connect_operation_lock: + if force or not self.is_connected(): + self.logger.info("Connecting to a new endpoint...") + self.wss_uri = self.issue_new_wss_url() + self.connect() + self.logger.info("Connected to a new endpoint...") + + def connect(self): + """Starts talking to the RTM server through a WebSocket connection""" + if self.bot_id is None: + self.bot_id = self.web_client.auth_test()["bot_id"] + + old_session: Optional[Connection] = self.current_session + old_current_session_state: ConnectionState = self.current_session_state + + if self.wss_uri is None: + self.wss_uri = self.issue_new_wss_url() + + current_session = Connection( + url=self.wss_uri, + logger=self.logger, + ping_interval=self.ping_interval, + trace_enabled=self.trace_enabled, + all_message_trace_enabled=self.all_message_trace_enabled, + ping_pong_trace_enabled=self.ping_pong_trace_enabled, + receive_buffer_size=1024, + proxy=self.proxy, + on_message_listener=self.run_all_message_listeners, + on_error_listener=self.run_all_error_listeners, + on_close_listener=self.run_all_close_listeners, + connection_type_name="RTM", + ) + current_session.connect() + + if old_current_session_state is not None: + old_current_session_state.terminated = True + if old_session is not None: + old_session.close() + + self.current_session = current_session + self.current_session_state = ConnectionState() + self.auto_reconnect_enabled = self.default_auto_reconnect_enabled + + if not self.current_app_monitor_started: + self.current_app_monitor_started = True + self.current_app_monitor.start() + + self.logger.info(f"A new session has been established (session id: {self.session_id()})") + + def disconnect(self): + """Disconnects the current session.""" + self.current_session.disconnect() + + def close(self) -> None: + """ + Closes this instance and cleans up underlying resources. + After calling this method, this instance is no longer usable. + """ + self.closed = True + self.disconnect() + self.current_session.close() + + def start(self) -> None: + """Establishes an RTM connection and blocks the current thread.""" + self.connect() + Event().wait() + + def send(self, payload: Union[dict, str]) -> None: + if payload is None: + return + if self.current_session is None or not self.current_session.is_active(): + raise SlackClientError("The RTM client is not connected to the Slack servers") + if isinstance(payload, str): + self.current_session.send(payload) + else: + self.current_session.send(json.dumps(payload)) + + # -------------------------------------------------------------- + # WS Message Processor + # -------------------------------------------------------------- + + def enqueue_message(self, message: str): + self.message_queue.put(message) + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"A new message enqueued (current queue size: {self.message_queue.qsize()})") + + def process_message(self): + try: + raw_message = self.message_queue.get(timeout=1) + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"A message dequeued (current queue size: {self.message_queue.qsize()})") + + if raw_message is not None: + message: dict = {} + if raw_message.startswith("{"): + message = json.loads(raw_message) + + def _run_message_listeners(): + self.run_message_listeners(message) + + self.message_workers.submit(_run_message_listeners) + except Empty: + pass + + def process_messages(self) -> None: + while not self.closed: + try: + self.process_message() + except Exception as e: + self.logger.exception(f"Failed to process a message: {e}") + + def run_message_listeners(self, message: dict) -> None: + type = message.get("type") + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"Message processing started (type: {type})") + try: + for listener in self.message_listeners: + try: + listener(self, message) + except Exception as e: + self.logger.exception(f"Failed to run a message listener: {e}") + except Exception as e: + self.logger.exception(f"Failed to run message listeners: {e}") + finally: + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"Message processing completed (type: {type})") + + # -------------------------------------------------------------- + # Internals + # -------------------------------------------------------------- + + def session_id(self) -> Optional[str]: + if self.current_session is not None: + return self.current_session.session_id + return None + + def run_all_message_listeners(self, message: str): + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"on_message invoked: (message: {message})") + self.enqueue_message(message) + for listener in self.on_message_listeners: + listener(message) + + def run_all_error_listeners(self, error: Exception): + self.logger.exception( + f"on_error invoked (session id: {self.session_id()}, " f"error: {type(error).__name__}, message: {error})" + ) + for listener in self.on_error_listeners: + listener(error) + + def run_all_close_listeners(self, code: int, reason: Optional[str] = None): + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"on_close invoked (session id: {self.session_id()})") + if self.auto_reconnect_enabled: + self.logger.info("Received CLOSE event. Going to reconnect... " f"(session id: {self.session_id()})") + self.connect_to_new_endpoint() + for listener in self.on_close_listeners: + listener(code, reason) + + def _run_current_session(self): + if self.current_session is not None and self.current_session.is_active(): + session_id = self.session_id() + try: + self.logger.info("Starting to receive messages from a new connection" f" (session id: {session_id})") + self.current_session_state.terminated = False + self.current_session.run_until_completion(self.current_session_state) + self.logger.info("Stopped receiving messages from a connection" f" (session id: {session_id})") + except Exception as e: + self.logger.exception( + "Failed to start or stop the current session" f" (session id: {session_id}, error: {e})" + ) + + def _monitor_current_session(self): + if self.current_app_monitor_started: + try: + self.current_session.check_state() + + if self.auto_reconnect_enabled and (self.current_session is None or not self.current_session.is_active()): + self.logger.info( + "The session seems to be already closed. Going to reconnect... " f"(session id: {self.session_id()})" + ) + self.connect_to_new_endpoint() + except Exception as e: + self.logger.error( + "Failed to check the current session or reconnect to the server " + f"(session id: {self.session_id()}, error: {type(e).__name__}, message: {e})" + ) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/scim/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/scim/__init__.py new file mode 100644 index 0000000..b9db365 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/scim/__init__.py @@ -0,0 +1,23 @@ +"""SCIM API is a set of APIs for provisioning and managing user accounts and groups. +SCIM is used by Single Sign-On (SSO) services and identity providers to manage people across a variety of tools, +including Slack. + +Refer to https://slack.dev/python-slack-sdk/scim/ for details. +""" +from .v1.client import SCIMClient +from .v1.response import SCIMResponse +from .v1.response import SearchUsersResponse, ReadUserResponse +from .v1.response import SearchGroupsResponse, ReadGroupResponse +from .v1.user import User +from .v1.group import Group + +__all__ = [ + "SCIMClient", + "SCIMResponse", + "SearchUsersResponse", + "ReadUserResponse", + "SearchGroupsResponse", + "ReadGroupResponse", + "User", + "Group", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/scim/async_client.py b/core_service/aws_lambda/project/packages/slack_sdk/scim/async_client.py new file mode 100644 index 0000000..17dd330 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/scim/async_client.py @@ -0,0 +1,5 @@ +from .v1.async_client import AsyncSCIMClient + +__all__ = [ + "AsyncSCIMClient", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/__init__.py new file mode 100644 index 0000000..e49bcdd --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/__init__.py @@ -0,0 +1,6 @@ +"""SCIM API is a set of APIs for provisioning and managing user accounts and groups. +SCIM is used by Single Sign-On (SSO) services and identity providers to manage people across a variety of tools, +including Slack. + +Refer to https://slack.dev/python-slack-sdk/scim/ for details. +""" diff --git a/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/async_client.py b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/async_client.py new file mode 100644 index 0000000..0ab220b --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/async_client.py @@ -0,0 +1,398 @@ +import json +import logging +from ssl import SSLContext +from typing import Any, Union, List +from typing import Dict, Optional +from urllib.parse import quote + +import aiohttp +from aiohttp import BasicAuth, ClientSession + +from .internal_utils import ( + _build_request_headers, + _debug_log_response, + get_user_agent, + _to_dict_without_not_given, + _build_query, +) +from .response import ( + SCIMResponse, + SearchUsersResponse, + ReadUserResponse, + SearchGroupsResponse, + ReadGroupResponse, + UserCreateResponse, + UserPatchResponse, + UserUpdateResponse, + UserDeleteResponse, + GroupCreateResponse, + GroupPatchResponse, + GroupUpdateResponse, + GroupDeleteResponse, +) +from .user import User +from .group import Group +from ...proxy_env_variable_loader import load_http_proxy_from_env + +from slack_sdk.http_retry.async_handler import AsyncRetryHandler +from slack_sdk.http_retry.builtin_async_handlers import async_default_handlers +from slack_sdk.http_retry.request import HttpRequest as RetryHttpRequest +from slack_sdk.http_retry.response import HttpResponse as RetryHttpResponse +from slack_sdk.http_retry.state import RetryState + + +class AsyncSCIMClient: + BASE_URL = "https://api.slack.com/scim/v1/" + + token: str + timeout: int + ssl: Optional[SSLContext] + proxy: Optional[str] + base_url: str + session: Optional[ClientSession] + trust_env_in_session: bool + auth: Optional[BasicAuth] + default_headers: Dict[str, str] + logger: logging.Logger + retry_handlers: List[AsyncRetryHandler] + + def __init__( + self, + token: str, + timeout: int = 30, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + base_url: str = BASE_URL, + session: Optional[ClientSession] = None, + trust_env_in_session: bool = False, + auth: Optional[BasicAuth] = None, + default_headers: Optional[Dict[str, str]] = None, + user_agent_prefix: Optional[str] = None, + user_agent_suffix: Optional[str] = None, + logger: Optional[logging.Logger] = None, + retry_handlers: Optional[List[AsyncRetryHandler]] = None, + ): + """API client for SCIM API + See https://api.slack.com/scim for more details + + Args: + token: An admin user's token, which starts with `xoxp-` + timeout: Request timeout (in seconds) + ssl: `ssl.SSLContext` to use for requests + proxy: Proxy URL (e.g., `localhost:9000`, `http://localhost:9000`) + base_url: The base URL for API calls + session: `aiohttp.ClientSession` instance + trust_env_in_session: True/False for `aiohttp.ClientSession` + auth: Basic auth info for `aiohttp.ClientSession` + default_headers: Request headers to add to all requests + user_agent_prefix: Prefix for User-Agent header value + user_agent_suffix: Suffix for User-Agent header value + logger: Custom logger + retry_handlers: Retry handlers + """ + self.token = token + self.timeout = timeout + self.ssl = ssl + self.proxy = proxy + self.base_url = base_url + self.session = session + self.trust_env_in_session = trust_env_in_session + self.auth = auth + self.default_headers = default_headers if default_headers else {} + self.default_headers["User-Agent"] = get_user_agent(user_agent_prefix, user_agent_suffix) + self.logger = logger if logger is not None else logging.getLogger(__name__) + self.retry_handlers = retry_handlers if retry_handlers is not None else async_default_handlers() + + if self.proxy is None or len(self.proxy.strip()) == 0: + env_variable = load_http_proxy_from_env(self.logger) + if env_variable is not None: + self.proxy = env_variable + + # ------------------------- + # Users + # ------------------------- + + async def search_users( + self, + *, + # Pagination required as of August 30, 2019. + count: int, + start_index: int, + filter: Optional[str] = None, + ) -> SearchUsersResponse: + return SearchUsersResponse( + await self.api_call( + http_verb="GET", + path="Users", + query_params={ + "filter": filter, + "count": count, + "startIndex": start_index, + }, + ) + ) + + async def read_user(self, id: str) -> ReadUserResponse: + return ReadUserResponse(await self.api_call(http_verb="GET", path=f"Users/{quote(id)}")) + + async def create_user(self, user: Union[Dict[str, Any], User]) -> UserCreateResponse: + return UserCreateResponse( + await self.api_call( + http_verb="POST", + path="Users", + body_params=user.to_dict() if isinstance(user, User) else _to_dict_without_not_given(user), + ) + ) + + async def patch_user(self, id: str, partial_user: Union[Dict[str, Any], User]) -> UserPatchResponse: + return UserPatchResponse( + await self.api_call( + http_verb="PATCH", + path=f"Users/{quote(id)}", + body_params=partial_user.to_dict() + if isinstance(partial_user, User) + else _to_dict_without_not_given(partial_user), + ) + ) + + async def update_user(self, user: Union[Dict[str, Any], User]) -> UserUpdateResponse: + user_id = user.id if isinstance(user, User) else user["id"] + return UserUpdateResponse( + await self.api_call( + http_verb="PUT", + path=f"Users/{quote(user_id)}", + body_params=user.to_dict() if isinstance(user, User) else _to_dict_without_not_given(user), + ) + ) + + async def delete_user(self, id: str) -> UserDeleteResponse: + return UserDeleteResponse( + await self.api_call( + http_verb="DELETE", + path=f"Users/{quote(id)}", + ) + ) + + # ------------------------- + # Groups + # ------------------------- + + async def search_groups( + self, + *, + # Pagination required as of August 30, 2019. + count: int, + start_index: int, + filter: Optional[str] = None, + ) -> SearchGroupsResponse: + return SearchGroupsResponse( + await self.api_call( + http_verb="GET", + path="Groups", + query_params={ + "filter": filter, + "count": count, + "startIndex": start_index, + }, + ) + ) + + async def read_group(self, id: str) -> ReadGroupResponse: + return ReadGroupResponse(await self.api_call(http_verb="GET", path=f"Groups/{quote(id)}")) + + async def create_group(self, group: Union[Dict[str, Any], Group]) -> GroupCreateResponse: + return GroupCreateResponse( + await self.api_call( + http_verb="POST", + path="Groups", + body_params=group.to_dict() if isinstance(group, Group) else _to_dict_without_not_given(group), + ) + ) + + async def patch_group(self, id: str, partial_group: Union[Dict[str, Any], Group]) -> GroupPatchResponse: + return GroupPatchResponse( + await self.api_call( + http_verb="PATCH", + path=f"Groups/{quote(id)}", + body_params=partial_group.to_dict() + if isinstance(partial_group, Group) + else _to_dict_without_not_given(partial_group), + ) + ) + + async def update_group(self, group: Union[Dict[str, Any], Group]) -> GroupUpdateResponse: + group_id = group.id if isinstance(group, Group) else group["id"] + return GroupUpdateResponse( + await self.api_call( + http_verb="PUT", + path=f"Groups/{quote(group_id)}", + body_params=group.to_dict() if isinstance(group, Group) else _to_dict_without_not_given(group), + ) + ) + + async def delete_group(self, id: str) -> GroupDeleteResponse: + return GroupDeleteResponse( + await self.api_call( + http_verb="DELETE", + path=f"Groups/{quote(id)}", + ) + ) + + # ------------------------- + + async def api_call( + self, + *, + http_verb: str, + path: str, + query_params: Optional[Dict[str, Any]] = None, + body_params: Optional[Dict[str, Any]] = None, + headers: Optional[Dict[str, str]] = None, + ) -> SCIMResponse: + url = f"{self.base_url}{path}" + query = _build_query(query_params) + if len(query) > 0: + url += f"?{query}" + return await self._perform_http_request( + http_verb=http_verb, + url=url, + body_params=body_params, + headers=_build_request_headers( + token=self.token, + default_headers=self.default_headers, + additional_headers=headers, + ), + ) + + async def _perform_http_request( + self, + *, + http_verb: str, + url: str, + body_params: Optional[Dict[str, Any]], + headers: Dict[str, str], + ) -> SCIMResponse: + if body_params is not None: + if body_params.get("schemas") is None: + body_params["schemas"] = ["urn:scim:schemas:core:1.0"] + body_params = json.dumps(body_params) + headers["Content-Type"] = "application/json;charset=utf-8" + + session: Optional[ClientSession] = None + use_running_session = self.session and not self.session.closed + if use_running_session: + session = self.session + else: + session = aiohttp.ClientSession( + timeout=aiohttp.ClientTimeout(total=self.timeout), + auth=self.auth, + trust_env=self.trust_env_in_session, + ) + + last_error: Optional[Exception] = None + resp: Optional[SCIMResponse] = None + try: + request_kwargs = { + "headers": headers, + "data": body_params, + "ssl": self.ssl, + "proxy": self.proxy, + } + retry_request = RetryHttpRequest( + method=http_verb, + url=url, + headers=headers, + body_params=body_params, + ) + + retry_state = RetryState() + counter_for_safety = 0 + while counter_for_safety < 100: + counter_for_safety += 1 + # If this is a retry, the next try started here. We can reset the flag. + retry_state.next_attempt_requested = False + retry_response: Optional[RetryHttpResponse] = None + response_body = "" + + if self.logger.level <= logging.DEBUG: + headers_for_logging = { + k: "(redacted)" if k.lower() == "authorization" else v for k, v in headers.items() + } + self.logger.debug( + f"Sending a request - url: {url}, params: {body_params}, headers: {headers_for_logging}" + ) + + try: + async with session.request(http_verb, url, **request_kwargs) as res: + try: + response_body = await res.text() + retry_response = RetryHttpResponse( + status_code=res.status, + headers=res.headers, + data=response_body.encode("utf-8") if response_body is not None else None, + ) + except aiohttp.ContentTypeError: + self.logger.debug(f"No response data returned from the following API call: {url}.") + + if res.status == 429: + for handler in self.retry_handlers: + if await handler.can_retry_async( + state=retry_state, + request=retry_request, + response=retry_response, + ): + if self.logger.level <= logging.DEBUG: + self.logger.info( + f"A retry handler found: {type(handler).__name__} " + f"for {http_verb} {url} - rate_limited" + ) + await handler.prepare_for_next_attempt_async( + state=retry_state, + request=retry_request, + response=retry_response, + ) + break + + if retry_state.next_attempt_requested is False: + resp = SCIMResponse( + url=url, + status_code=res.status, + raw_body=response_body, + headers=res.headers, + ) + _debug_log_response(self.logger, resp) + return resp + + except Exception as e: + last_error = e + for handler in self.retry_handlers: + if await handler.can_retry_async( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ): + if self.logger.level <= logging.DEBUG: + self.logger.info( + f"A retry handler found: {type(handler).__name__} " f"for {http_verb} {url} - {e}" + ) + await handler.prepare_for_next_attempt_async( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ) + break + + if retry_state.next_attempt_requested is False: + raise last_error + + if resp is not None: + return resp + raise last_error + + finally: + if not use_running_session: + await session.close() + + return resp diff --git a/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/client.py b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/client.py new file mode 100644 index 0000000..450c02d --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/client.py @@ -0,0 +1,418 @@ +"""SCIM API is a set of APIs for provisioning and managing user accounts and groups. +SCIM is used by Single Sign-On (SSO) services and identity providers to manage people across a variety of tools, +including Slack. + +Refer to https://slack.dev/python-slack-sdk/scim/ for details. +""" +import json +import logging +import urllib +from http.client import HTTPResponse +from ssl import SSLContext +from typing import Dict, Optional, Union, Any, List +from urllib.error import HTTPError +from urllib.parse import quote +from urllib.request import Request, urlopen, OpenerDirector, ProxyHandler, HTTPSHandler + +from slack_sdk.errors import SlackRequestError +from .internal_utils import ( + _build_query, + _build_request_headers, + _debug_log_response, + get_user_agent, + _to_dict_without_not_given, +) +from .response import ( + SCIMResponse, + SearchUsersResponse, + ReadUserResponse, + SearchGroupsResponse, + ReadGroupResponse, + UserCreateResponse, + UserPatchResponse, + UserUpdateResponse, + UserDeleteResponse, + GroupCreateResponse, + GroupPatchResponse, + GroupUpdateResponse, + GroupDeleteResponse, +) +from .user import User +from .group import Group + +from slack_sdk.http_retry import default_retry_handlers +from slack_sdk.http_retry.handler import RetryHandler +from slack_sdk.http_retry.request import HttpRequest as RetryHttpRequest +from slack_sdk.http_retry.response import HttpResponse as RetryHttpResponse +from slack_sdk.http_retry.state import RetryState + +from ...proxy_env_variable_loader import load_http_proxy_from_env + + +class SCIMClient: + BASE_URL = "https://api.slack.com/scim/v1/" + + token: str + timeout: int + ssl: Optional[SSLContext] + proxy: Optional[str] + base_url: str + default_headers: Dict[str, str] + logger: logging.Logger + retry_handlers: List[RetryHandler] + + def __init__( + self, + token: str, + timeout: int = 30, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + base_url: str = BASE_URL, + default_headers: Optional[Dict[str, str]] = None, + user_agent_prefix: Optional[str] = None, + user_agent_suffix: Optional[str] = None, + logger: Optional[logging.Logger] = None, + retry_handlers: Optional[List[RetryHandler]] = None, + ): + """API client for SCIM API + See https://api.slack.com/scim for more details + + Args: + token: An admin user's token, which starts with `xoxp-` + timeout: Request timeout (in seconds) + ssl: `ssl.SSLContext` to use for requests + proxy: Proxy URL (e.g., `localhost:9000`, `http://localhost:9000`) + base_url: The base URL for API calls + default_headers: Request headers to add to all requests + user_agent_prefix: Prefix for User-Agent header value + user_agent_suffix: Suffix for User-Agent header value + logger: Custom logger + retry_handlers: Retry handlers + """ + self.token = token + self.timeout = timeout + self.ssl = ssl + self.proxy = proxy + self.base_url = base_url + self.default_headers = default_headers if default_headers else {} + self.default_headers["User-Agent"] = get_user_agent(user_agent_prefix, user_agent_suffix) + self.logger = logger if logger is not None else logging.getLogger(__name__) + self.retry_handlers = retry_handlers if retry_handlers is not None else default_retry_handlers() + + if self.proxy is None or len(self.proxy.strip()) == 0: + env_variable = load_http_proxy_from_env(self.logger) + if env_variable is not None: + self.proxy = env_variable + + # ------------------------- + # Users + # ------------------------- + + def search_users( + self, + *, + # Pagination required as of August 30, 2019. + count: int, + start_index: int, + filter: Optional[str] = None, + ) -> SearchUsersResponse: + return SearchUsersResponse( + self.api_call( + http_verb="GET", + path="Users", + query_params={ + "filter": filter, + "count": count, + "startIndex": start_index, + }, + ) + ) + + def read_user(self, id: str) -> ReadUserResponse: + return ReadUserResponse(self.api_call(http_verb="GET", path=f"Users/{quote(id)}")) + + def create_user(self, user: Union[Dict[str, Any], User]) -> UserCreateResponse: + return UserCreateResponse( + self.api_call( + http_verb="POST", + path="Users", + body_params=user.to_dict() if isinstance(user, User) else _to_dict_without_not_given(user), + ) + ) + + def patch_user(self, id: str, partial_user: Union[Dict[str, Any], User]) -> UserPatchResponse: + return UserPatchResponse( + self.api_call( + http_verb="PATCH", + path=f"Users/{quote(id)}", + body_params=partial_user.to_dict() + if isinstance(partial_user, User) + else _to_dict_without_not_given(partial_user), + ) + ) + + def update_user(self, user: Union[Dict[str, Any], User]) -> UserUpdateResponse: + user_id = user.id if isinstance(user, User) else user["id"] + return UserUpdateResponse( + self.api_call( + http_verb="PUT", + path=f"Users/{quote(user_id)}", + body_params=user.to_dict() if isinstance(user, User) else _to_dict_without_not_given(user), + ) + ) + + def delete_user(self, id: str) -> UserDeleteResponse: + return UserDeleteResponse( + self.api_call( + http_verb="DELETE", + path=f"Users/{quote(id)}", + ) + ) + + # ------------------------- + # Groups + # ------------------------- + + def search_groups( + self, + *, + # Pagination required as of August 30, 2019. + count: int, + start_index: int, + filter: Optional[str] = None, + ) -> SearchGroupsResponse: + return SearchGroupsResponse( + self.api_call( + http_verb="GET", + path="Groups", + query_params={ + "filter": filter, + "count": count, + "startIndex": start_index, + }, + ) + ) + + def read_group(self, id: str) -> ReadGroupResponse: + return ReadGroupResponse(self.api_call(http_verb="GET", path=f"Groups/{quote(id)}")) + + def create_group(self, group: Union[Dict[str, Any], Group]) -> GroupCreateResponse: + return GroupCreateResponse( + self.api_call( + http_verb="POST", + path="Groups", + body_params=group.to_dict() if isinstance(group, Group) else _to_dict_without_not_given(group), + ) + ) + + def patch_group(self, id: str, partial_group: Union[Dict[str, Any], Group]) -> GroupPatchResponse: + return GroupPatchResponse( + self.api_call( + http_verb="PATCH", + path=f"Groups/{quote(id)}", + body_params=partial_group.to_dict() + if isinstance(partial_group, Group) + else _to_dict_without_not_given(partial_group), + ) + ) + + def update_group(self, group: Union[Dict[str, Any], Group]) -> GroupUpdateResponse: + group_id = group.id if isinstance(group, Group) else group["id"] + return GroupUpdateResponse( + self.api_call( + http_verb="PUT", + path=f"Groups/{quote(group_id)}", + body_params=group.to_dict() if isinstance(group, Group) else _to_dict_without_not_given(group), + ) + ) + + def delete_group(self, id: str) -> GroupDeleteResponse: + return GroupDeleteResponse( + self.api_call( + http_verb="DELETE", + path=f"Groups/{quote(id)}", + ) + ) + + # ------------------------- + + def api_call( + self, + *, + http_verb: str, + path: str, + query_params: Optional[Dict[str, Any]] = None, + body_params: Optional[Dict[str, Any]] = None, + headers: Optional[Dict[str, str]] = None, + ) -> SCIMResponse: + """Performs a Slack API request and returns the result.""" + url = f"{self.base_url}{path}" + query = _build_query(query_params) + if len(query) > 0: + url += f"?{query}" + + return self._perform_http_request( + http_verb=http_verb, + url=url, + body=body_params, + headers=_build_request_headers( + token=self.token, + default_headers=self.default_headers, + additional_headers=headers, + ), + ) + + def _perform_http_request( + self, + *, + http_verb: str = "GET", + url: str, + body: Optional[Dict[str, Any]] = None, + headers: Dict[str, str], + ) -> SCIMResponse: + if body is not None: + if body.get("schemas") is None: + body["schemas"] = ["urn:scim:schemas:core:1.0"] + body = json.dumps(body) + headers["Content-Type"] = "application/json;charset=utf-8" + + if self.logger.level <= logging.DEBUG: + headers_for_logging = {k: "(redacted)" if k.lower() == "authorization" else v for k, v in headers.items()} + self.logger.debug(f"Sending a request - {http_verb} url: {url}, body: {body}, headers: {headers_for_logging}") + + # NOTE: Intentionally ignore the `http_verb` here + # Slack APIs accepts any API method requests with POST methods + req = Request( + method=http_verb, + url=url, + data=body.encode("utf-8") if body is not None else None, + headers=headers, + ) + resp = None + last_error = None + + retry_state = RetryState() + counter_for_safety = 0 + while counter_for_safety < 100: + counter_for_safety += 1 + # If this is a retry, the next try started here. We can reset the flag. + retry_state.next_attempt_requested = False + + try: + resp = self._perform_http_request_internal(url, req) + # The resp is a 200 OK response + return resp + + except HTTPError as e: + # read the response body here + charset = e.headers.get_content_charset() or "utf-8" + response_body: str = e.read().decode(charset) + # As adding new values to HTTPError#headers can be ignored, building a new dict object here + response_headers = dict(e.headers.items()) + resp = SCIMResponse( + url=url, + status_code=e.code, + raw_body=response_body, + headers=response_headers, + ) + if e.code == 429: + # for backward-compatibility with WebClient (v.2.5.0 or older) + if "retry-after" not in resp.headers and "Retry-After" in resp.headers: + resp.headers["retry-after"] = resp.headers["Retry-After"] + if "Retry-After" not in resp.headers and "retry-after" in resp.headers: + resp.headers["Retry-After"] = resp.headers["retry-after"] + _debug_log_response(self.logger, resp) + + # Try to find a retry handler for this error + retry_request = RetryHttpRequest.from_urllib_http_request(req) + retry_response = RetryHttpResponse( + status_code=e.code, + headers={k: [v] for k, v in e.headers.items()}, + data=response_body.encode("utf-8") if response_body is not None else None, + ) + for handler in self.retry_handlers: + if handler.can_retry( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ): + if self.logger.level <= logging.DEBUG: + self.logger.info( + f"A retry handler found: {type(handler).__name__} for {req.method} {req.full_url} - {e}" + ) + handler.prepare_for_next_attempt( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ) + break + + if retry_state.next_attempt_requested is False: + return resp + + except Exception as err: + last_error = err + self.logger.error(f"Failed to send a request to Slack API server: {err}") + + # Try to find a retry handler for this error + retry_request = RetryHttpRequest.from_urllib_http_request(req) + for handler in self.retry_handlers: + if handler.can_retry( + state=retry_state, + request=retry_request, + response=None, + error=err, + ): + if self.logger.level <= logging.DEBUG: + self.logger.info( + f"A retry handler found: {type(handler).__name__} for {req.method} {req.full_url} - {err}" + ) + handler.prepare_for_next_attempt( + state=retry_state, + request=retry_request, + response=None, + error=err, + ) + self.logger.info(f"Going to retry the same request: {req.method} {req.full_url}") + break + + if retry_state.next_attempt_requested is False: + raise err + + if resp is not None: + return resp + raise last_error + + def _perform_http_request_internal(self, url: str, req: Request) -> SCIMResponse: + opener: Optional[OpenerDirector] = None + # for security (BAN-B310) + if url.lower().startswith("http"): + if self.proxy is not None: + if isinstance(self.proxy, str): + opener = urllib.request.build_opener( + ProxyHandler({"http": self.proxy, "https": self.proxy}), + HTTPSHandler(context=self.ssl), + ) + else: + raise SlackRequestError(f"Invalid proxy detected: {self.proxy} must be a str value") + else: + raise SlackRequestError(f"Invalid URL detected: {url}") + + # NOTE: BAN-B310 is already checked above + http_resp: Optional[HTTPResponse] = None + if opener: + http_resp = opener.open(req, timeout=self.timeout) # skipcq: BAN-B310 + else: + http_resp = urlopen(req, context=self.ssl, timeout=self.timeout) # skipcq: BAN-B310 + charset: str = http_resp.headers.get_content_charset() or "utf-8" + response_body: str = http_resp.read().decode(charset) + resp = SCIMResponse( + url=url, + status_code=http_resp.status, + raw_body=response_body, + headers=http_resp.headers, + ) + _debug_log_response(self.logger, resp) + return resp diff --git a/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/default_arg.py b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/default_arg.py new file mode 100644 index 0000000..52dca49 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/default_arg.py @@ -0,0 +1,5 @@ +class DefaultArg: + pass + + +NotGiven = DefaultArg() diff --git a/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/group.py b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/group.py new file mode 100644 index 0000000..5bee764 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/group.py @@ -0,0 +1,78 @@ +from typing import Optional, List, Union, Dict, Any + +from .default_arg import DefaultArg, NotGiven +from .internal_utils import _to_dict_without_not_given, _is_iterable + + +class GroupMember: + display: Union[Optional[str], DefaultArg] + value: Union[Optional[str], DefaultArg] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + display: Union[Optional[str], DefaultArg] = NotGiven, + value: Union[Optional[str], DefaultArg] = NotGiven, + **kwargs, + ) -> None: + self.display = display + self.value = value + self.unknown_fields = kwargs + + def to_dict(self): + return _to_dict_without_not_given(self) + + +class GroupMeta: + created: Union[Optional[str], DefaultArg] + location: Union[Optional[str], DefaultArg] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + created: Union[Optional[str], DefaultArg] = NotGiven, + location: Union[Optional[str], DefaultArg] = NotGiven, + **kwargs, + ) -> None: + self.created = created + self.location = location + self.unknown_fields = kwargs + + def to_dict(self): + return _to_dict_without_not_given(self) + + +class Group: + display_name: Union[Optional[str], DefaultArg] + id: Union[Optional[str], DefaultArg] + members: Union[Optional[List[GroupMember]], DefaultArg] + meta: Union[Optional[GroupMeta], DefaultArg] + schemas: Union[Optional[List[str]], DefaultArg] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + display_name: Union[Optional[str], DefaultArg] = NotGiven, + id: Union[Optional[str], DefaultArg] = NotGiven, + members: Union[Optional[List[GroupMember]], DefaultArg] = NotGiven, + meta: Union[Optional[GroupMeta], DefaultArg] = NotGiven, + schemas: Union[Optional[List[str]], DefaultArg] = NotGiven, + **kwargs, + ) -> None: + self.display_name = display_name + self.id = id + self.members = ( + [a if isinstance(a, GroupMember) else GroupMember(**a) for a in members] if _is_iterable(members) else members + ) + self.meta = GroupMeta(**meta) if meta is not None and isinstance(meta, dict) else meta + self.schemas = schemas + self.unknown_fields = kwargs + + def to_dict(self): + return _to_dict_without_not_given(self) + + def __repr__(self): + return f"" diff --git a/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/internal_utils.py b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/internal_utils.py new file mode 100644 index 0000000..1ad1b80 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/internal_utils.py @@ -0,0 +1,149 @@ +import copy +import logging +import re +import sys +from typing import Dict, Callable +from typing import Union, Optional, Any +from urllib.parse import quote + +from .default_arg import DefaultArg, NotGiven +from slack_sdk.web.internal_utils import get_user_agent + + +def _build_query(params: Optional[Dict[str, Any]]) -> str: + if params is not None and len(params) > 0: + return "&".join({f"{quote(str(k))}={quote(str(v))}" for k, v in params.items() if v is not None}) + return "" + + +def _is_iterable(obj: Union[Optional[Any], DefaultArg]) -> bool: + return obj is not None and obj is not NotGiven + + +def _to_dict_without_not_given(obj: Any) -> dict: + dict_value = {} + given_dict = obj if isinstance(obj, dict) else vars(obj) + for key, value in given_dict.items(): + if key == "unknown_fields": + if value is not None: + converted = _to_dict_without_not_given(value) + dict_value.update(converted) + continue + + dict_key = _to_camel_case_key(key) + if value is NotGiven: + continue + if isinstance(value, list): + dict_value[dict_key] = [elem.to_dict() if hasattr(elem, "to_dict") else elem for elem in value] + elif isinstance(value, dict): + dict_value[dict_key] = _to_dict_without_not_given(value) + else: + dict_value[dict_key] = value.to_dict() if hasattr(value, "to_dict") else value + return dict_value + + +def _create_copy(original: Any) -> Any: + if sys.version_info.major == 3 and sys.version_info.minor <= 6: + return copy.copy(original) + else: + return copy.deepcopy(original) + + +def _to_camel_case_key(key: str) -> str: + next_to_capital = False + result = "" + for c in key: + if c == "_": + next_to_capital = True + elif next_to_capital: + result += c.upper() + next_to_capital = False + else: + result += c + return result + + +def _to_snake_cased(original: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any]]: + return _convert_dict_keys( + original, + {}, + lambda s: re.sub( + "^_", + "", + "".join(["_" + c.lower() if c.isupper() else c for c in s]), + ), + ) + + +def _to_camel_cased(original: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any]]: + return _convert_dict_keys( + original, + {}, + _to_camel_case_key, + ) + + +def _convert_dict_keys( + original_dict: Optional[Dict[str, Any]], + result_dict: Dict[str, Any], + convert: Callable[[str], str], +) -> Optional[Dict[str, Any]]: + if original_dict is None: + return result_dict + + for original_key, original_value in original_dict.items(): + new_key = convert(original_key) + if isinstance(original_value, dict): + result_dict[new_key] = {} + new_value = _convert_dict_keys(original_value, result_dict[new_key], convert) + result_dict[new_key] = new_value + elif isinstance(original_value, list): + result_dict[new_key] = [] + is_dict = len(original_value) > 0 and isinstance(original_value[0], dict) + for element in original_value: + if is_dict: + if isinstance(element, dict): + new_element = {} + for elem_key, elem_value in element.items(): + new_element[convert(elem_key)] = ( + _convert_dict_keys(elem_value, {}, convert) + if isinstance(elem_value, dict) + else _create_copy(elem_value) + ) + result_dict[new_key].append(new_element) + else: + result_dict[new_key].append(_create_copy(original_value)) + else: + result_dict[new_key] = _create_copy(original_value) + return result_dict + + +def _build_request_headers( + token: str, + default_headers: Dict[str, str], + additional_headers: Optional[Dict[str, str]], +) -> Dict[str, str]: + request_headers = { + "Content-Type": "application/json;charset=utf-8", + "Authorization": f"Bearer {token}", + } + if default_headers is None or "User-Agent" not in default_headers: + request_headers["User-Agent"] = get_user_agent() + if default_headers is not None: + request_headers.update(default_headers) + if additional_headers is not None: + request_headers.update(additional_headers) + return request_headers + + +def _debug_log_response( # type: ignore + logger, + resp: "SCIMResponse", # noqa: F821 +) -> None: + if logger.level <= logging.DEBUG: + logger.debug( + "Received the following response - " + f"status: {resp.status_code}, " + f"headers: {(dict(resp.headers))}, " + f"body: {resp.raw_body}" + ) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/response.py b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/response.py new file mode 100644 index 0000000..3631de3 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/response.py @@ -0,0 +1,263 @@ +import json +from typing import Dict, Any, List, Optional + +from slack_sdk.scim.v1.group import Group +from slack_sdk.scim.v1.internal_utils import _to_snake_cased +from slack_sdk.scim.v1.user import User + + +class Errors: + code: int + description: str + + def __init__(self, code: int, description: str) -> None: + self.code = code + self.description = description + + def to_dict(self) -> dict: + return {"code": self.code, "description": self.description} + + +class SCIMResponse: + url: str + status_code: int + headers: Dict[str, Any] + raw_body: Optional[str] + body: Optional[Dict[str, Any]] + snake_cased_body: Optional[Dict[str, Any]] + + errors: Optional[Errors] + + @property + def snake_cased_body(self) -> Optional[Dict[str, Any]]: # type: ignore + if self._snake_cased_body is None: + self._snake_cased_body = _to_snake_cased(self.body) + return self._snake_cased_body + + @property + def errors(self) -> Optional[Errors]: # type: ignore + errors = self.snake_cased_body.get("errors") + if errors is None: + return None + return Errors(**errors) + + def __init__( + self, + *, + url: str, + status_code: int, + raw_body: Optional[str], + headers: dict, + ): + self.url = url + self.status_code = status_code + self.headers = headers + self.raw_body = raw_body + self.body = json.loads(raw_body) if raw_body is not None and raw_body.startswith("{") else None + self._snake_cased_body = None # build this when it's accessed for the first time + + def __repr__(self): + dict_value = {} + for key, value in vars(self).items(): + dict_value[key] = value.to_dict() if hasattr(value, "to_dict") else value + + if dict_value: # skipcq: PYL-R1705 + return f"" + else: + return self.__str__() + + +# --------------------------------- +# Users +# --------------------------------- + + +class SearchUsersResponse(SCIMResponse): + users: List[User] + + @property + def users(self) -> List[User]: # type: ignore + return [User(**r) for r in self.snake_cased_body.get("resources")] + + def __init__(self, underlying: SCIMResponse): + self.underlying = underlying + self.url = underlying.url + self.status_code = underlying.status_code + self.headers = underlying.headers + self.raw_body = underlying.raw_body + self.body = underlying.body + self._snake_cased_body = None + + +class ReadUserResponse(SCIMResponse): + user: User + + @property + def user(self) -> User: # type: ignore + return User(**self.snake_cased_body) + + def __init__(self, underlying: SCIMResponse): + self.underlying = underlying + self.url = underlying.url + self.status_code = underlying.status_code + self.headers = underlying.headers + self.raw_body = underlying.raw_body + self.body = underlying.body + self._snake_cased_body = None + + +class UserCreateResponse(SCIMResponse): + user: User + + @property + def user(self) -> User: # type: ignore + return User(**self.snake_cased_body) + + def __init__(self, underlying: SCIMResponse): + self.underlying = underlying + self.url = underlying.url + self.status_code = underlying.status_code + self.headers = underlying.headers + self.raw_body = underlying.raw_body + self.body = underlying.body + self._snake_cased_body = None + + +class UserPatchResponse(SCIMResponse): + user: User + + @property + def user(self) -> User: # type: ignore + return User(**self.snake_cased_body) + + def __init__(self, underlying: SCIMResponse): + self.underlying = underlying + self.url = underlying.url + self.status_code = underlying.status_code + self.headers = underlying.headers + self.raw_body = underlying.raw_body + self.body = underlying.body + self._snake_cased_body = None + + +class UserUpdateResponse(SCIMResponse): + user: User + + @property + def user(self) -> User: # type: ignore + return User(**self.snake_cased_body) + + def __init__(self, underlying: SCIMResponse): + self.underlying = underlying + self.url = underlying.url + self.status_code = underlying.status_code + self.headers = underlying.headers + self.raw_body = underlying.raw_body + self.body = underlying.body + self._snake_cased_body = None + + +class UserDeleteResponse(SCIMResponse): + def __init__(self, underlying: SCIMResponse): + self.underlying = underlying + self.url = underlying.url + self.status_code = underlying.status_code + self.headers = underlying.headers + self.raw_body = underlying.raw_body + self.body = underlying.body + self._snake_cased_body = None + + +# --------------------------------- +# Groups +# --------------------------------- + + +class SearchGroupsResponse(SCIMResponse): + groups: List[Group] + + @property + def groups(self) -> List[Group]: # type: ignore + return [Group(**r) for r in self.snake_cased_body.get("resources")] + + def __init__(self, underlying: SCIMResponse): + self.underlying = underlying + self.url = underlying.url + self.status_code = underlying.status_code + self.headers = underlying.headers + self.raw_body = underlying.raw_body + self.body = underlying.body + self._snake_cased_body = None + + +class ReadGroupResponse(SCIMResponse): + group: Group + + @property + def group(self) -> Group: # type: ignore + return Group(**self.snake_cased_body) + + def __init__(self, underlying: SCIMResponse): + self.underlying = underlying + self.url = underlying.url + self.status_code = underlying.status_code + self.headers = underlying.headers + self.raw_body = underlying.raw_body + self.body = underlying.body + self._snake_cased_body = None + + +class GroupCreateResponse(SCIMResponse): + group: Group + + @property + def group(self) -> Group: # type: ignore + return Group(**self.snake_cased_body) + + def __init__(self, underlying: SCIMResponse): + self.underlying = underlying + self.url = underlying.url + self.status_code = underlying.status_code + self.headers = underlying.headers + self.raw_body = underlying.raw_body + self.body = underlying.body + self._snake_cased_body = None + + +class GroupPatchResponse(SCIMResponse): + def __init__(self, underlying: SCIMResponse): + self.underlying = underlying + self.url = underlying.url + self.status_code = underlying.status_code + self.headers = underlying.headers + self.raw_body = underlying.raw_body + self.body = underlying.body + self._snake_cased_body = None + + +class GroupUpdateResponse(SCIMResponse): + group: Group + + @property + def group(self) -> Group: # type: ignore + return Group(**self.snake_cased_body) + + def __init__(self, underlying: SCIMResponse): + self.underlying = underlying + self.url = underlying.url + self.status_code = underlying.status_code + self.headers = underlying.headers + self.raw_body = underlying.raw_body + self.body = underlying.body + self._snake_cased_body = None + + +class GroupDeleteResponse(SCIMResponse): + def __init__(self, underlying: SCIMResponse): + self.underlying = underlying + self.url = underlying.url + self.status_code = underlying.status_code + self.headers = underlying.headers + self.raw_body = underlying.raw_body + self.body = underlying.body + self._snake_cased_body = None diff --git a/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/types.py b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/types.py new file mode 100644 index 0000000..db9c744 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/types.py @@ -0,0 +1,27 @@ +from typing import Optional, Union, Dict, Any + +from .default_arg import DefaultArg, NotGiven +from .internal_utils import _to_dict_without_not_given + + +class TypeAndValue: + primary: Union[Optional[bool], DefaultArg] + type: Union[Optional[str], DefaultArg] + value: Union[Optional[str], DefaultArg] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + primary: Union[Optional[bool], DefaultArg] = NotGiven, + type: Union[Optional[str], DefaultArg] = NotGiven, + value: Union[Optional[str], DefaultArg] = NotGiven, + **kwargs, + ) -> None: + self.primary = primary + self.type = type + self.value = value + self.unknown_fields = kwargs + + def to_dict(self) -> dict: + return _to_dict_without_not_given(self) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/user.py b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/user.py new file mode 100644 index 0000000..9b2d4ff --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/scim/v1/user.py @@ -0,0 +1,214 @@ +from typing import Optional, Any, List, Dict, Union + +from .default_arg import DefaultArg, NotGiven +from .internal_utils import _to_dict_without_not_given, _is_iterable +from .types import TypeAndValue + + +class UserAddress: + country: Union[Optional[str], DefaultArg] + locality: Union[Optional[str], DefaultArg] + postal_code: Union[Optional[str], DefaultArg] + primary: Union[Optional[bool], DefaultArg] + region: Union[Optional[str], DefaultArg] + street_address: Union[Optional[str], DefaultArg] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + country: Union[Optional[str], DefaultArg] = NotGiven, + locality: Union[Optional[str], DefaultArg] = NotGiven, + postal_code: Union[Optional[str], DefaultArg] = NotGiven, + primary: Union[Optional[bool], DefaultArg] = NotGiven, + region: Union[Optional[str], DefaultArg] = NotGiven, + street_address: Union[Optional[str], DefaultArg] = NotGiven, + **kwargs, + ) -> None: + self.country = country + self.locality = locality + self.postal_code = postal_code + self.primary = primary + self.region = region + self.street_address = street_address + self.unknown_fields = kwargs + + def to_dict(self) -> dict: + return _to_dict_without_not_given(self) + + +class UserEmail(TypeAndValue): + pass + + +class UserPhoneNumber(TypeAndValue): + pass + + +class UserRole(TypeAndValue): + pass + + +class UserGroup: + display: Union[Optional[str], DefaultArg] + value: Union[Optional[str], DefaultArg] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + display: Union[Optional[str], DefaultArg] = NotGiven, + value: Union[Optional[str], DefaultArg] = NotGiven, + **kwargs, + ) -> None: + self.display = display + self.value = value + self.unknown_fields = kwargs + + def to_dict(self) -> dict: + return _to_dict_without_not_given(self) + + +class UserMeta: + created: Union[Optional[str], DefaultArg] + location: Union[Optional[str], DefaultArg] + unknown_fields: Dict[str, Any] + + def __init__( + self, + created: Union[Optional[str], DefaultArg] = NotGiven, + location: Union[Optional[str], DefaultArg] = NotGiven, + **kwargs, + ) -> None: + self.created = created + self.location = location + self.unknown_fields = kwargs + + def to_dict(self) -> dict: + return _to_dict_without_not_given(self) + + +class UserName: + family_name: Union[Optional[str], DefaultArg] + given_name: Union[Optional[str], DefaultArg] + unknown_fields: Dict[str, Any] + + def __init__( + self, + family_name: Union[Optional[str], DefaultArg] = NotGiven, + given_name: Union[Optional[str], DefaultArg] = NotGiven, + **kwargs, + ) -> None: + self.family_name = family_name + self.given_name = given_name + self.unknown_fields = kwargs + + def to_dict(self) -> dict: + return _to_dict_without_not_given(self) + + +class UserPhoto: + type: Union[Optional[str], DefaultArg] + value: Union[Optional[str], DefaultArg] + unknown_fields: Dict[str, Any] + + def __init__( + self, + type: Union[Optional[str], DefaultArg] = NotGiven, + value: Union[Optional[str], DefaultArg] = NotGiven, + **kwargs, + ) -> None: + self.type = type + self.value = value + self.unknown_fields = kwargs + + def to_dict(self) -> dict: + return _to_dict_without_not_given(self) + + +class User: + active: Union[Optional[bool], DefaultArg] + addresses: Union[Optional[List[UserAddress]], DefaultArg] + display_name: Union[Optional[str], DefaultArg] + emails: Union[Optional[List[TypeAndValue]], DefaultArg] + external_id: Union[Optional[str], DefaultArg] + groups: Union[Optional[List[UserGroup]], DefaultArg] + id: Union[Optional[str], DefaultArg] + meta: Union[Optional[UserMeta], DefaultArg] + name: Union[Optional[UserName], DefaultArg] + nick_name: Union[Optional[str], DefaultArg] + phone_numbers: Union[Optional[List[TypeAndValue]], DefaultArg] + photos: Union[Optional[List[UserPhoto]], DefaultArg] + profile_url: Union[Optional[str], DefaultArg] + roles: Union[Optional[List[TypeAndValue]], DefaultArg] + schemas: Union[Optional[List[str]], DefaultArg] + timezone: Union[Optional[str], DefaultArg] + title: Union[Optional[str], DefaultArg] + user_name: Union[Optional[str], DefaultArg] + unknown_fields: Dict[str, Any] + + def __init__( + self, + *, + active: Union[Optional[bool], DefaultArg] = NotGiven, + addresses: Union[Optional[List[Union[UserAddress, Dict[str, Any]]]], DefaultArg] = NotGiven, + display_name: Union[Optional[str], DefaultArg] = NotGiven, + emails: Union[Optional[List[Union[TypeAndValue, Dict[str, Any]]]], DefaultArg] = NotGiven, + external_id: Union[Optional[str], DefaultArg] = NotGiven, + groups: Union[Optional[List[Union[UserGroup, Dict[str, Any]]]], DefaultArg] = NotGiven, + id: Union[Optional[str], DefaultArg] = NotGiven, + meta: Union[Optional[Union[UserMeta, Dict[str, Any]]], DefaultArg] = NotGiven, + name: Union[Optional[Union[UserName, Dict[str, Any]]], DefaultArg] = NotGiven, + nick_name: Union[Optional[str], DefaultArg] = NotGiven, + phone_numbers: Union[Optional[List[Union[TypeAndValue, Dict[str, Any]]]], DefaultArg] = NotGiven, + photos: Union[Optional[List[Union[UserPhoto, Dict[str, Any]]]], DefaultArg] = NotGiven, + profile_url: Union[Optional[str], DefaultArg] = NotGiven, + roles: Union[Optional[List[Union[TypeAndValue, Dict[str, Any]]]], DefaultArg] = NotGiven, + schemas: Union[Optional[List[str]], DefaultArg] = NotGiven, + timezone: Union[Optional[str], DefaultArg] = NotGiven, + title: Union[Optional[str], DefaultArg] = NotGiven, + user_name: Union[Optional[str], DefaultArg] = NotGiven, + **kwargs, + ) -> None: + self.active = active + self.addresses = ( # type: ignore + [a if isinstance(a, UserAddress) else UserAddress(**a) for a in addresses] + if _is_iterable(addresses) + else addresses + ) + self.display_name = display_name + self.emails = ( # type: ignore + [a if isinstance(a, TypeAndValue) else TypeAndValue(**a) for a in emails] if _is_iterable(emails) else emails + ) + self.external_id = external_id + self.groups = ( # type: ignore + [a if isinstance(a, UserGroup) else UserGroup(**a) for a in groups] if _is_iterable(groups) else groups + ) + self.id = id + self.meta = UserMeta(**meta) if meta is not None and isinstance(meta, dict) else meta # type: ignore + self.name = UserName(**name) if name is not None and isinstance(name, dict) else name # type: ignore + self.nick_name = nick_name + self.phone_numbers = ( # type: ignore + [a if isinstance(a, TypeAndValue) else TypeAndValue(**a) for a in phone_numbers] + if _is_iterable(phone_numbers) + else phone_numbers + ) + self.photos = ( # type: ignore + [a if isinstance(a, UserPhoto) else UserPhoto(**a) for a in photos] if _is_iterable(photos) else photos + ) + self.profile_url = profile_url + self.roles = ( # type: ignore + [a if isinstance(a, TypeAndValue) else TypeAndValue(**a) for a in roles] if _is_iterable(roles) else roles + ) + self.schemas = schemas + self.timezone = timezone + self.title = title + self.user_name = user_name + + self.unknown_fields = kwargs + + def to_dict(self): + return _to_dict_without_not_given(self) + + def __repr__(self): + return f"" diff --git a/core_service/aws_lambda/project/packages/slack_sdk/signature/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/signature/__init__.py new file mode 100644 index 0000000..77b8b85 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/signature/__init__.py @@ -0,0 +1,71 @@ +"""Slack request signature verifier""" +import hashlib +import hmac +from time import time +from typing import Dict, Optional, Union + + +class Clock: + def now(self) -> float: # skipcq: PYL-R0201 + return time() + + +class SignatureVerifier: + def __init__(self, signing_secret: str, clock: Clock = Clock()): + """Slack request signature verifier + + Slack signs its requests using a secret that's unique to your app. + With the help of signing secrets, your app can more confidently verify + whether requests from us are authentic. + https://api.slack.com/authentication/verifying-requests-from-slack + """ + self.signing_secret = signing_secret + self.clock = clock + + def is_valid_request( + self, + body: Union[str, bytes], + headers: Dict[str, str], + ) -> bool: + """Verifies if the given signature is valid""" + if headers is None: + return False + normalized_headers = {k.lower(): v for k, v in headers.items()} + return self.is_valid( + body=body, + timestamp=normalized_headers.get("x-slack-request-timestamp", None), + signature=normalized_headers.get("x-slack-signature", None), + ) + + def is_valid( + self, + body: Union[str, bytes], + timestamp: str, + signature: str, + ) -> bool: + """Verifies if the given signature is valid""" + if timestamp is None or signature is None: + return False + + if abs(self.clock.now() - int(timestamp)) > 60 * 5: + return False + + calculated_signature = self.generate_signature(timestamp=timestamp, body=body) + if calculated_signature is None: + return False + return hmac.compare_digest(calculated_signature, signature) + + def generate_signature(self, *, timestamp: str, body: Union[str, bytes]) -> Optional[str]: + """Generates a signature""" + if timestamp is None: + return None + if body is None: + body = "" + if isinstance(body, bytes): + body = body.decode("utf-8") + + format_req = str.encode(f"v0:{timestamp}:{body}") + encoded_secret = str.encode(self.signing_secret) + request_hash = hmac.new(encoded_secret, format_req, hashlib.sha256).hexdigest() + calculated_signature = f"v0={request_hash}" + return calculated_signature diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/__init__.py new file mode 100644 index 0000000..4859abf --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/__init__.py @@ -0,0 +1,11 @@ +"""Socket Mode is a method of connecting your app to Slack’s APIs using WebSockets instead of HTTP. +You can use slack_sdk.socket_mode.SocketModeClient for managing Socket Mode connections +and performing interactions with Slack. + +https://api.slack.com/apis/connections/socket +""" +from .builtin import SocketModeClient + +__all__ = [ + "SocketModeClient", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/aiohttp/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/aiohttp/__init__.py new file mode 100644 index 0000000..caaaa97 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/aiohttp/__init__.py @@ -0,0 +1,439 @@ +"""aiohttp based Socket Mode client + +* https://api.slack.com/apis/connections/socket +* https://slack.dev/python-slack-sdk/socket-mode/ +* https://pypi.org/project/aiohttp/ + +""" +import asyncio +import logging +import time +from asyncio import Future, Lock +from asyncio import Queue +from logging import Logger +from typing import Union, Optional, List, Callable, Awaitable + +import aiohttp +from aiohttp import ClientWebSocketResponse, WSMessage, WSMsgType, ClientConnectionError + +from slack_sdk.proxy_env_variable_loader import load_http_proxy_from_env +from slack_sdk.socket_mode.async_client import AsyncBaseSocketModeClient +from slack_sdk.socket_mode.async_listeners import ( + AsyncWebSocketMessageListener, + AsyncSocketModeRequestListener, +) +from slack_sdk.socket_mode.request import SocketModeRequest +from slack_sdk.web.async_client import AsyncWebClient + + +class SocketModeClient(AsyncBaseSocketModeClient): + logger: Logger + web_client: AsyncWebClient + app_token: str + wss_uri: Optional[str] + auto_reconnect_enabled: bool + message_queue: Queue + message_listeners: List[ + Union[ + AsyncWebSocketMessageListener, + Callable[["AsyncBaseSocketModeClient", dict, Optional[str]], Awaitable[None]], + ] + ] + socket_mode_request_listeners: List[ + Union[ + AsyncSocketModeRequestListener, + Callable[["AsyncBaseSocketModeClient", SocketModeRequest], Awaitable[None]], + ] + ] + + message_receiver: Optional[Future] + message_processor: Future + + proxy: Optional[str] + ping_interval: float + trace_enabled: bool + + last_ping_pong_time: Optional[float] + current_session: Optional[ClientWebSocketResponse] + current_session_monitor: Optional[Future] + + auto_reconnect_enabled: bool + default_auto_reconnect_enabled: bool + closed: bool + stale: bool + connect_operation_lock: Lock + + on_message_listeners: List[Callable[[WSMessage], Awaitable[None]]] + on_error_listeners: List[Callable[[WSMessage], Awaitable[None]]] + on_close_listeners: List[Callable[[WSMessage], Awaitable[None]]] + + def __init__( + self, + app_token: str, + logger: Optional[Logger] = None, + web_client: Optional[AsyncWebClient] = None, + proxy: Optional[str] = None, + auto_reconnect_enabled: bool = True, + ping_interval: float = 5, + trace_enabled: bool = False, + on_message_listeners: Optional[List[Callable[[WSMessage], Awaitable[None]]]] = None, + on_error_listeners: Optional[List[Callable[[WSMessage], Awaitable[None]]]] = None, + on_close_listeners: Optional[List[Callable[[WSMessage], Awaitable[None]]]] = None, + ): + """Socket Mode client + + Args: + app_token: App-level token + logger: Custom logger + web_client: Web API client + auto_reconnect_enabled: True if automatic reconnection is enabled (default: True) + ping_interval: interval for ping-pong with Slack servers (seconds) + trace_enabled: True if more verbose logs to see what's happening under the hood + proxy: the HTTP proxy URL + on_message_listeners: listener functions for on_message + on_error_listeners: listener functions for on_error + on_close_listeners: listener functions for on_close + """ + self.app_token = app_token + self.logger = logger or logging.getLogger(__name__) + self.web_client = web_client or AsyncWebClient() + self.closed = False + self.stale = False + self.connect_operation_lock = Lock() + self.proxy = proxy + if self.proxy is None or len(self.proxy.strip()) == 0: + env_variable = load_http_proxy_from_env(self.logger) + if env_variable is not None: + self.proxy = env_variable + + self.default_auto_reconnect_enabled = auto_reconnect_enabled + self.auto_reconnect_enabled = self.default_auto_reconnect_enabled + self.ping_interval = ping_interval + self.trace_enabled = trace_enabled + self.last_ping_pong_time = None + + self.wss_uri = None + self.message_queue = Queue() + self.message_listeners = [] + self.socket_mode_request_listeners = [] + self.current_session = None + self.current_session_monitor = None + + # https://docs.aiohttp.org/en/stable/client_reference.html + # Unless you are connecting to a large, unknown number of different servers + # over the lifetime of your application, + # it is suggested you use a single session for the lifetime of your application + # to benefit from connection pooling. + self.aiohttp_client_session = aiohttp.ClientSession() + + self.on_message_listeners = on_message_listeners or [] + self.on_error_listeners = on_error_listeners or [] + self.on_close_listeners = on_close_listeners or [] + + self.message_receiver = None + self.message_processor = asyncio.ensure_future(self.process_messages()) + + async def monitor_current_session(self) -> None: + # In the asyncio runtime, accessing a shared object (self.current_session here) from + # multiple tasks can cause race conditions and errors. + # To avoid such, we access only the session that is active when this loop starts. + session: ClientWebSocketResponse = self.current_session + session_id: str = self.build_session_id(session) + + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"A new monitor_current_session() execution loop for {session_id} started") + try: + logging_interval = 100 + counter_for_logging = 0 + + while not self.closed: + if session != self.current_session: + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"The monitor_current_session task for {session_id} is now cancelled") + break + try: + if self.trace_enabled and self.logger.level <= logging.DEBUG: + # The logging here is for detailed investigation on potential issues in this client. + # If you don't see this log for a while, it means that + # this receive_messages execution is no longer working for some reason. + counter_for_logging += 1 + if counter_for_logging >= logging_interval: + counter_for_logging = 0 + log_message = ( + "#monitor_current_session method has been verifying if this session is active " + f"(session: {session_id}, logging interval: {logging_interval})" + ) + self.logger.debug(log_message) + + await asyncio.sleep(self.ping_interval) + + if session is not None and session.closed is False: + t = time.time() + if self.last_ping_pong_time is None: + self.last_ping_pong_time = float(t) + try: + await session.ping(f"sdk-ping-pong:{t}") + except Exception as e: + # The ping() method can fail for some reason. + # To establish a new connection even in this scenario, + # we ignore the exception here. + self.logger.warning(f"Failed to send a ping message ({session_id}): {e}") + + if self.auto_reconnect_enabled: + should_reconnect = False + if session is None or session.closed: + self.logger.info(f"The session ({session_id}) seems to be already closed. Reconnecting...") + should_reconnect = True + + if await self.is_ping_pong_failing(): + disconnected_seconds = int(time.time() - self.last_ping_pong_time) + self.logger.info( + f"The session ({session_id}) seems to be stale. Reconnecting..." + f" reason: disconnected for {disconnected_seconds}+ seconds)" + ) + self.stale = True + self.last_ping_pong_time = None + should_reconnect = True + + if should_reconnect is True or not await self.is_connected(): + await self.connect_to_new_endpoint() + + except Exception as e: + self.logger.error( + f"Failed to check the current session ({session_id}) or reconnect to the server " + f"(error: {type(e).__name__}, message: {e})" + ) + except asyncio.CancelledError: + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"The monitor_current_session task for {session_id} is now cancelled") + raise + + async def receive_messages(self) -> None: + # In the asyncio runtime, accessing a shared object (self.current_session here) from + # multiple tasks can cause race conditions and errors. + # To avoid such, we access only the session that is active when this loop starts. + session = self.current_session + session_id = self.build_session_id(session) + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"A new receive_messages() execution loop with {session_id} started") + try: + consecutive_error_count = 0 + logging_interval = 100 + counter_for_logging = 0 + + while not self.closed: + if session != self.current_session: + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"The running receive_messages task for {session_id} is now cancelled") + break + try: + message: WSMessage = await session.receive() + # just in case, checking if the value is not None + if message is not None: + if self.logger.level <= logging.DEBUG: + # The following logging prints every single received message + # except empty message data ones. + type = WSMsgType(message.type) + message_type = type.name if type is not None else message.type + message_data = message.data + if isinstance(message_data, bytes): + message_data = message_data.decode("utf-8") + if len(message_data) > 0: + # To skip the empty message that Slack server-side often sends + self.logger.debug( + f"Received message " + f"(type: {message_type}, " + f"data: {message_data}, " + f"extra: {message.extra}, " + f"session: {session_id})" + ) + + if self.trace_enabled: + # The logging here is for detailed trouble shooting of potential issues in this client. + # If you don't see this log for a while, it can mean that + # this receive_messages execution is no longer working for some reason. + counter_for_logging += 1 + if counter_for_logging >= logging_interval: + counter_for_logging = 0 + log_message = ( + "#receive_messages method has been working without any issues " + f"(session: {session_id}, logging interval: {logging_interval})" + ) + self.logger.debug(log_message) + + if message.type == WSMsgType.TEXT: + message_data = message.data + await self.enqueue_message(message_data) + for listener in self.on_message_listeners: + await listener(message) + elif message.type == WSMsgType.CLOSE: + if self.auto_reconnect_enabled: + self.logger.info(f"Received CLOSE event from {session_id}. Reconnecting...") + await self.connect_to_new_endpoint() + for listener in self.on_close_listeners: + await listener(message) + elif message.type == WSMsgType.ERROR: + for listener in self.on_error_listeners: + await listener(message) + elif message.type == WSMsgType.CLOSED: + await asyncio.sleep(self.ping_interval) + continue + elif message.type == WSMsgType.PING: + await session.pong(message.data) + continue + elif message.type == WSMsgType.PONG: + if message.data is not None: + str_message_data = message.data.decode("utf-8") + elements = str_message_data.split(":") + if len(elements) == 2 and elements[0] == "sdk-ping-pong": + try: + self.last_ping_pong_time = float(elements[1]) + except Exception as e: + self.logger.warning( + f"Failed to parse the last_ping_pong_time value from {str_message_data}" + f" - error : {e}, session: {session_id}" + ) + continue + + consecutive_error_count = 0 + + except Exception as e: + consecutive_error_count += 1 + self.logger.error(f"Failed to receive or enqueue a message: {type(e).__name__}, {e} ({session_id})") + if isinstance(e, ClientConnectionError): + await asyncio.sleep(self.ping_interval) + else: + await asyncio.sleep(consecutive_error_count) + except asyncio.CancelledError: + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"The running receive_messages task for {session_id} is now cancelled") + raise + + async def is_ping_pong_failing(self) -> bool: + if self.last_ping_pong_time is None: + return False + disconnected_seconds = int(time.time() - self.last_ping_pong_time) + return disconnected_seconds >= (self.ping_interval * 4) + + async def is_connected(self) -> bool: + connected: bool = ( + not self.closed + and not self.stale + and self.current_session is not None + and not self.current_session.closed + and not await self.is_ping_pong_failing() + ) + if self.logger.level <= logging.DEBUG and connected is False: + # Prints more detailed information about the inactive connection + is_ping_pong_failing = await self.is_ping_pong_failing() + session_id = await self.session_id() + self.logger.debug( + "Inactive connection detected (" + f"session_id: {session_id}, " + f"closed: {self.closed}, " + f"stale: {self.stale}, " + f"current_session.closed: {self.current_session.closed}, " + f"is_ping_pong_failing: {is_ping_pong_failing}" + ")" + ) + return connected + + async def session_id(self) -> str: + return self.build_session_id(self.current_session) + + async def connect(self): + old_session: Optional[ClientWebSocketResponse] = None if self.current_session is None else self.current_session + if self.wss_uri is None: + # If the underlying WSS URL does not exist, + # acquiring a new active WSS URL from the server-side first + self.wss_uri = await self.issue_new_wss_url() + + self.current_session = await self.aiohttp_client_session.ws_connect( + self.wss_uri, + autoping=False, + heartbeat=self.ping_interval, + proxy=self.proxy, + ) + session_id: str = await self.session_id() + self.auto_reconnect_enabled = self.default_auto_reconnect_enabled + self.stale = False + self.logger.info(f"A new session ({session_id}) has been established") + + # The first ping from the new connection + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"Sending a ping message with the newly established connection ({session_id})...") + t = time.time() + await self.current_session.ping(f"sdk-ping-pong:{t}") + + if self.current_session_monitor is not None: + self.current_session_monitor.cancel() + + self.current_session_monitor = asyncio.ensure_future(self.monitor_current_session()) + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"A new monitor_current_session() executor has been recreated for {session_id}") + + if self.message_receiver is not None: + self.message_receiver.cancel() + + self.message_receiver = asyncio.ensure_future(self.receive_messages()) + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"A new receive_messages() executor has been recreated for {session_id}") + + if old_session is not None: + await old_session.close() + old_session_id = self.build_session_id(old_session) + self.logger.info(f"The old session ({old_session_id}) has been abandoned") + + async def disconnect(self): + if self.current_session is not None: + await self.current_session.close() + session_id = await self.session_id() + self.logger.info(f"The current session ({session_id}) has been abandoned by disconnect() method call") + + async def send_message(self, message: str): + session_id = await self.session_id() + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"Sending a message: {message} from session: {session_id}") + try: + await self.current_session.send_str(message) + except ConnectionError as e: + # We rarely get this exception while replacing the underlying WebSocket connections. + # We can do one more try here as the self.current_session should be ready now. + if self.logger.level <= logging.DEBUG: + self.logger.debug( + f"Failed to send a message (error: {e}, message: {message}, session: {session_id})" + " as the underlying connection was replaced. Retrying the same request only one time..." + ) + # Although acquiring self.connect_operation_lock also for the first method call is the safest way, + # we avoid synchronizing a lot for better performance. That's why we are doing a retry here. + try: + await self.connect_operation_lock.acquire() + if await self.is_connected(): + await self.current_session.send_str(message) + else: + self.logger.warning( + f"The current session ({session_id}) is no longer active. " "Failed to send a message" + ) + raise e + finally: + if self.connect_operation_lock.locked() is True: + self.connect_operation_lock.release() + + async def close(self): + self.closed = True + self.auto_reconnect_enabled = False + await self.disconnect() + if self.message_processor is not None: + self.message_processor.cancel() + if self.current_session_monitor is not None: + self.current_session_monitor.cancel() + if self.message_receiver is not None: + self.message_receiver.cancel() + if self.aiohttp_client_session is not None: + await self.aiohttp_client_session.close() + + @classmethod + def build_session_id(cls, session: ClientWebSocketResponse) -> str: + if session is None: + return "" + return "s_" + str(hash(session)) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/async_client.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/async_client.py new file mode 100644 index 0000000..7fc0d67 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/async_client.py @@ -0,0 +1,168 @@ +import asyncio +import json +import logging +from asyncio import Queue, Lock +from asyncio.futures import Future +from logging import Logger +from typing import Dict, Union, Any, Optional, List, Callable, Awaitable + +from slack_sdk.errors import SlackApiError +from slack_sdk.socket_mode.async_listeners import ( + AsyncWebSocketMessageListener, + AsyncSocketModeRequestListener, +) +from slack_sdk.socket_mode.request import SocketModeRequest +from slack_sdk.socket_mode.response import SocketModeResponse +from slack_sdk.web.async_client import AsyncWebClient + + +class AsyncBaseSocketModeClient: + logger: Logger + web_client: AsyncWebClient + app_token: str + wss_uri: str + auto_reconnect_enabled: bool + trace_enabled: bool + closed: bool + connect_operation_lock: Lock + + message_queue: Queue + message_listeners: List[ + Union[ + AsyncWebSocketMessageListener, + Callable[["AsyncBaseSocketModeClient", dict, Optional[str]], Awaitable[None]], + ] + ] + socket_mode_request_listeners: List[ + Union[ + AsyncSocketModeRequestListener, + Callable[["AsyncBaseSocketModeClient", SocketModeRequest], Awaitable[None]], + ] + ] + + async def issue_new_wss_url(self) -> str: + try: + response = await self.web_client.apps_connections_open(app_token=self.app_token) + return response["url"] + except SlackApiError as e: + if e.response["error"] == "ratelimited": + # NOTE: ratelimited errors rarely occur with this endpoint + delay = int(e.response.headers.get("Retry-After", "30")) # Tier1 + self.logger.info(f"Rate limited. Retrying in {delay} seconds...") + await asyncio.sleep(delay) + # Retry to issue a new WSS URL + return await self.issue_new_wss_url() + else: + # other errors + self.logger.error(f"Failed to retrieve WSS URL: {e}") + raise e + + async def is_connected(self) -> bool: + return False + + async def session_id(self) -> str: + return "" + + async def connect(self): + raise NotImplementedError() + + async def disconnect(self): + raise NotImplementedError() + + async def connect_to_new_endpoint(self, force: bool = False): + session_id = await self.session_id() + try: + await self.connect_operation_lock.acquire() + if self.trace_enabled: + self.logger.debug(f"For reconnection, the connect_operation_lock was acquired (session: {session_id})") + if force or not await self.is_connected(): + self.wss_uri = await self.issue_new_wss_url() + await self.connect() + finally: + if self.connect_operation_lock.locked() is True: + self.connect_operation_lock.release() + if self.trace_enabled: + self.logger.debug(f"The connect_operation_lock for reconnection was released (session: {session_id})") + + async def close(self): + self.closed = True + await self.disconnect() + + async def send_message(self, message: str): + raise NotImplementedError() + + async def send_socket_mode_response(self, response: Union[Dict[str, Any], SocketModeResponse]): + if isinstance(response, SocketModeResponse): + await self.send_message(json.dumps(response.to_dict())) + else: + await self.send_message(json.dumps(response)) + + async def enqueue_message(self, message: str): + await self.message_queue.put(message) + if self.logger.level <= logging.DEBUG: + queue_size = self.message_queue.qsize() + session_id = await self.session_id() + self.logger.debug(f"A new message enqueued (current queue size: {queue_size}, session: {session_id})") + + async def process_messages(self): + session_id = await self.session_id() + try: + while not self.closed: + try: + await self.process_message() + except asyncio.CancelledError: + # if self.closed is True, the connection is already closed + # In this case, we can ignore the exception here + if not self.closed: + raise + except Exception as e: + self.logger.exception(f"Failed to process a message: {e}, session: {session_id}") + except asyncio.CancelledError: + if self.trace_enabled: + self.logger.debug(f"The running process_messages task for {session_id} is now cancelled") + raise + + async def process_message(self): + raw_message = await self.message_queue.get() + if raw_message is not None: + message: dict = {} + if raw_message.startswith("{"): + message = json.loads(raw_message) + _: Future[None] = asyncio.ensure_future(self.run_message_listeners(message, raw_message)) + + async def run_message_listeners(self, message: dict, raw_message: str) -> None: + session_id = await self.session_id() + type, envelope_id = message.get("type"), message.get("envelope_id") + if self.logger.level <= logging.DEBUG: + self.logger.debug( + f"Message processing started (type: {type}, envelope_id: {envelope_id}, session: {session_id})" + ) + try: + if message.get("type") == "disconnect": + await self.connect_to_new_endpoint(force=True) + return + + for listener in self.message_listeners: + try: + await listener(self, message, raw_message) # type: ignore + except Exception as e: + self.logger.exception(f"Failed to run a message listener: {e}, session: {session_id}") + + if len(self.socket_mode_request_listeners) > 0: + request = SocketModeRequest.from_dict(message) + if request is not None: + for listener in self.socket_mode_request_listeners: + try: + await listener(self, request) # type: ignore + except Exception as e: + self.logger.exception(f"Failed to run a request listener: {e}, session: {session_id}") + except Exception as e: + self.logger.exception(f"Failed to run message listeners: {e}, session: {session_id}") + finally: + if self.logger.level <= logging.DEBUG: + self.logger.debug( + f"Message processing completed (" + f"type: {type}, " + f"envelope_id: {envelope_id}, " + f"session: {session_id})" + ) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/async_listeners.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/async_listeners.py new file mode 100644 index 0000000..c21a3c4 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/async_listeners.py @@ -0,0 +1,20 @@ +from typing import Optional, Callable + +from slack_sdk.socket_mode.request import SocketModeRequest + + +class AsyncWebSocketMessageListener(Callable): + async def __call__( # type: ignore + client: "AsyncBaseSocketModeClient", # noqa: F821 + message: dict, + raw_message: Optional[str] = None, + ): # noqa: F821 + raise NotImplementedError() + + +class AsyncSocketModeRequestListener(Callable): + async def __call__( # type: ignore + client: "AsyncBaseSocketModeClient", # noqa: F821 + request: SocketModeRequest, + ): # noqa: F821 + raise NotImplementedError() diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/__init__.py new file mode 100644 index 0000000..c49d4c9 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/__init__.py @@ -0,0 +1,5 @@ +from .client import SocketModeClient + +__all__ = [ + "SocketModeClient", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/client.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/client.py new file mode 100644 index 0000000..79f875d --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/client.py @@ -0,0 +1,289 @@ +"""The built-in Socket Mode client + +* https://api.slack.com/apis/connections/socket +* https://slack.dev/python-slack-sdk/socket-mode/ + +""" +import logging +from concurrent.futures.thread import ThreadPoolExecutor +from logging import Logger +from queue import Queue +from threading import Lock +from typing import Union, Optional, List, Callable, Dict + +from slack_sdk.socket_mode.client import BaseSocketModeClient +from slack_sdk.socket_mode.listeners import ( + WebSocketMessageListener, + SocketModeRequestListener, +) +from slack_sdk.socket_mode.request import SocketModeRequest +from slack_sdk.web import WebClient +from .connection import Connection, ConnectionState +from ..interval_runner import IntervalRunner +from ...errors import SlackClientConfigurationError, SlackClientNotConnectedError +from ...proxy_env_variable_loader import load_http_proxy_from_env + + +class SocketModeClient(BaseSocketModeClient): + logger: Logger + web_client: WebClient + app_token: str + wss_uri: Optional[str] + message_queue: Queue + message_listeners: List[ + Union[ + WebSocketMessageListener, + Callable[["BaseSocketModeClient", dict, Optional[str]], None], + ] + ] + socket_mode_request_listeners: List[ + Union[ + SocketModeRequestListener, + Callable[["BaseSocketModeClient", SocketModeRequest], None], + ] + ] + + current_session: Optional[Connection] + current_session_state: ConnectionState + current_session_runner: IntervalRunner + + current_app_monitor: IntervalRunner + current_app_monitor_started: bool + + message_processor: IntervalRunner + message_workers: ThreadPoolExecutor + + auto_reconnect_enabled: bool + default_auto_reconnect_enabled: bool + trace_enabled: bool + receive_buffer_size: int # bytes size + + connect_operation_lock: Lock + + on_message_listeners: List[Callable[[str], None]] + on_error_listeners: List[Callable[[Exception], None]] + on_close_listeners: List[Callable[[int, Optional[str]], None]] + + def __init__( + self, + app_token: str, + logger: Optional[Logger] = None, + web_client: Optional[WebClient] = None, + auto_reconnect_enabled: bool = True, + trace_enabled: bool = False, + all_message_trace_enabled: bool = False, + ping_pong_trace_enabled: bool = False, + ping_interval: float = 5, + receive_buffer_size: int = 1024, + concurrency: int = 10, + proxy: Optional[str] = None, + proxy_headers: Optional[Dict[str, str]] = None, + on_message_listeners: Optional[List[Callable[[str], None]]] = None, + on_error_listeners: Optional[List[Callable[[Exception], None]]] = None, + on_close_listeners: Optional[List[Callable[[int, Optional[str]], None]]] = None, + ): + """Socket Mode client + + Args: + app_token: App-level token + logger: Custom logger + web_client: Web API client + auto_reconnect_enabled: True if automatic reconnection is enabled (default: True) + trace_enabled: True if more detailed debug-logging is enabled (default: False) + all_message_trace_enabled: True if all message dump in debug logs is enabled (default: False) + ping_pong_trace_enabled: True if trace logging for all ping-pong communications is enabled (default: False) + ping_interval: interval for ping-pong with Slack servers (seconds) + receive_buffer_size: the chunk size of a single socket recv operation (default: 1024) + concurrency: the size of thread pool (default: 10) + proxy: the HTTP proxy URL + proxy_headers: additional HTTP header for proxy connection + on_message_listeners: listener functions for on_message + on_error_listeners: listener functions for on_error + on_close_listeners: listener functions for on_close + """ + self.app_token = app_token + self.logger = logger or logging.getLogger(__name__) + self.web_client = web_client or WebClient() + self.default_auto_reconnect_enabled = auto_reconnect_enabled + self.auto_reconnect_enabled = self.default_auto_reconnect_enabled + self.trace_enabled = trace_enabled + self.all_message_trace_enabled = all_message_trace_enabled + self.ping_pong_trace_enabled = ping_pong_trace_enabled + self.ping_interval = ping_interval + self.receive_buffer_size = receive_buffer_size + if self.receive_buffer_size < 16: + raise SlackClientConfigurationError("Too small receive_buffer_size detected.") + + self.wss_uri = None + self.message_queue = Queue() + self.message_listeners = [] + self.socket_mode_request_listeners = [] + + self.current_session = None + self.current_session_state = ConnectionState() + self.current_session_runner = IntervalRunner(self._run_current_session, 0.1).start() + + self.current_app_monitor_started = False + self.current_app_monitor = IntervalRunner(self._monitor_current_session, self.ping_interval) + + self.closed = False + self.connect_operation_lock = Lock() + + self.message_processor = IntervalRunner(self.process_messages, 0.001).start() + self.message_workers = ThreadPoolExecutor(max_workers=concurrency) + + self.proxy = proxy + if self.proxy is None or len(self.proxy.strip()) == 0: + env_variable = load_http_proxy_from_env(self.logger) + if env_variable is not None: + self.proxy = env_variable + self.proxy_headers = proxy_headers + + self.on_message_listeners = on_message_listeners or [] + self.on_error_listeners = on_error_listeners or [] + self.on_close_listeners = on_close_listeners or [] + + def session_id(self) -> Optional[str]: + if self.current_session is not None: + return self.current_session.session_id + return None + + def is_connected(self) -> bool: + return self.current_session is not None and self.current_session.is_active() + + def connect(self) -> None: + old_session: Optional[Connection] = self.current_session + old_current_session_state: ConnectionState = self.current_session_state + + if self.wss_uri is None: + self.wss_uri = self.issue_new_wss_url() + + current_session = Connection( + url=self.wss_uri, + logger=self.logger, + ping_interval=self.ping_interval, + trace_enabled=self.trace_enabled, + all_message_trace_enabled=self.all_message_trace_enabled, + ping_pong_trace_enabled=self.ping_pong_trace_enabled, + receive_buffer_size=self.receive_buffer_size, + proxy=self.proxy, + proxy_headers=self.proxy_headers, + on_message_listener=self._on_message, + on_error_listener=self._on_error, + on_close_listener=self._on_close, + ssl_context=self.web_client.ssl, + ) + current_session.connect() + + if old_current_session_state is not None: + old_current_session_state.terminated = True + if old_session is not None: + old_session.close() + + self.current_session = current_session + self.current_session_state = ConnectionState() + self.auto_reconnect_enabled = self.default_auto_reconnect_enabled + + if not self.current_app_monitor_started: + self.current_app_monitor_started = True + self.current_app_monitor.start() + + self.logger.info(f"A new session has been established (session id: {self.session_id()})") + + def disconnect(self) -> None: + if self.current_session is not None: + self.current_session.close() + + def send_message(self, message: str) -> None: + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"Sending a message (session id: {self.session_id()}, message: {message})") + try: + self.current_session.send(message) + except SlackClientNotConnectedError as e: + # We rarely get this exception while replacing the underlying WebSocket connections. + # We can do one more try here as the self.current_session should be ready now. + if self.logger.level <= logging.DEBUG: + self.logger.debug( + f"Failed to send a message (session id: {self.session_id()}, error: {e}, message: {message})" + " as the underlying connection was replaced. Retrying the same request only one time..." + ) + # Although acquiring self.connect_operation_lock also for the first method call is the safest way, + # we avoid synchronizing a lot for better performance. That's why we are doing a retry here. + with self.connect_operation_lock: + if self.is_connected(): + self.current_session.send(message) + else: + self.logger.warning( + f"The current session (session id: {self.session_id()}) is no longer active. " + "Failed to send a message" + ) + raise e + + def close(self): + self.closed = True + self.auto_reconnect_enabled = False + self.disconnect() + if self.current_app_monitor.is_alive(): + self.current_app_monitor.shutdown() + if self.message_processor.is_alive(): + self.message_processor.shutdown() + self.message_workers.shutdown() + + def _on_message(self, message: str): + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"on_message invoked: (message: {message})") + self.enqueue_message(message) + for listener in self.on_message_listeners: + listener(message) + + def _on_error(self, error: Exception): + error_message = ( + f"on_error invoked (session id: {self.session_id()}, " f"error: {type(error).__name__}, message: {error})" + ) + if self.trace_enabled: + self.logger.exception(error_message) + else: + self.logger.error(error_message) + + for listener in self.on_error_listeners: + listener(error) + + def _on_close(self, code: int, reason: Optional[str] = None): + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"on_close invoked (session id: {self.session_id()})") + if self.auto_reconnect_enabled: + self.logger.info("Received CLOSE event. Reconnecting... " f"(session id: {self.session_id()})") + self.connect_to_new_endpoint() + for listener in self.on_close_listeners: + listener(code, reason) + + def _run_current_session(self): + if self.current_session is not None and self.current_session.is_active(): + session_id = self.session_id() + try: + self.logger.info("Starting to receive messages from a new connection" f" (session id: {session_id})") + self.current_session_state.terminated = False + self.current_session.run_until_completion(self.current_session_state) + self.logger.info("Stopped receiving messages from a connection" f" (session id: {session_id})") + except Exception as e: + error_message = "Failed to start or stop the current session" f" (session id: {session_id}, error: {e})" + if self.trace_enabled: + self.logger.exception(error_message) + else: + self.logger.error(error_message) + + def _monitor_current_session(self): + if self.current_app_monitor_started: + try: + self.current_session.check_state() + + if self.auto_reconnect_enabled and (self.current_session is None or not self.current_session.is_active()): + self.logger.info( + "The session seems to be already closed. Reconnecting... " f"(session id: {self.session_id()})" + ) + self.connect_to_new_endpoint() + except Exception as e: + self.logger.error( + "Failed to check the current session or reconnect to the server " + f"(session id: {self.session_id()}, error: {type(e).__name__}, message: {e})" + ) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/connection.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/connection.py new file mode 100644 index 0000000..dd9eed0 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/connection.py @@ -0,0 +1,452 @@ +import socket +import ssl +import struct +import time +from logging import Logger +from threading import Lock +from typing import Optional, Callable, Union, List, Tuple, Dict +from urllib.parse import urlparse +from uuid import uuid4 + +from slack_sdk.errors import SlackClientNotConnectedError, SlackClientConfigurationError +from .frame_header import FrameHeader +from .internals import ( + _parse_handshake_response, + _validate_sec_websocket_accept, + _generate_sec_websocket_key, + _to_readable_opcode, + _receive_messages, + _build_data_frame_for_sending, + _parse_text_payload, + _establish_new_socket_connection, +) + + +class ConnectionState: + # The flag supposed to be used for telling SocketModeClient + # when this connection is no longer available + terminated: bool + + def __init__(self): + self.terminated = False + + +class Connection: + url: str + logger: Logger + proxy: Optional[str] + proxy_headers: Optional[Dict[str, str]] + + trace_enabled: bool + ping_pong_trace_enabled: bool + last_ping_pong_time: Optional[float] + + session_id: str + sock: Optional[ssl.SSLSocket] + + on_message_listener: Optional[Callable[[str], None]] + on_error_listener: Optional[Callable[[Exception], None]] + on_close_listener: Optional[Callable[[int, Optional[str]], None]] + + def __init__( + self, + url: str, + logger: Logger, + proxy: Optional[str] = None, + proxy_headers: Optional[Dict[str, str]] = None, + ping_interval: float = 5, # seconds + receive_timeout: float = 3, + receive_buffer_size: int = 1024, + trace_enabled: bool = False, + all_message_trace_enabled: bool = False, + ping_pong_trace_enabled: bool = False, + on_message_listener: Optional[Callable[[str], None]] = None, + on_error_listener: Optional[Callable[[Exception], None]] = None, + on_close_listener: Optional[Callable[[int, Optional[str]], None]] = None, + connection_type_name: str = "Socket Mode", + ssl_context: Optional[ssl.SSLContext] = None, + ): + self.url = url + self.logger = logger + self.proxy = proxy + self.proxy_headers = proxy_headers + + self.ping_interval = ping_interval + self.receive_timeout = receive_timeout + self.receive_buffer_size = receive_buffer_size + if self.receive_buffer_size < 16: + raise SlackClientConfigurationError("Too small receive_buffer_size detected.") + + self.session_id = str(uuid4()) + self.trace_enabled = trace_enabled + self.all_message_trace_enabled = all_message_trace_enabled + self.ping_pong_trace_enabled = ping_pong_trace_enabled + self.last_ping_pong_time = None + self.consecutive_check_state_error_count = 0 + self.sock = None + # To avoid ssl.SSLError: [SSL: BAD_LENGTH] bad length + self.sock_receive_lock = Lock() + self.sock_send_lock = Lock() + + self.on_message_listener = on_message_listener + self.on_error_listener = on_error_listener + self.on_close_listener = on_close_listener + self.connection_type_name = connection_type_name + + self.ssl_context = ssl_context + + def connect(self) -> None: + try: + parsed_url = urlparse(self.url.strip()) + hostname: str = parsed_url.hostname + port: int = parsed_url.port or (443 if parsed_url.scheme == "wss" else 80) + if self.trace_enabled: + self.logger.debug( + f"Connecting to the address for handshake: {hostname}:{port} " f"(session id: {self.session_id})" + ) + sock: Union[ssl.SSLSocket, socket] = _establish_new_socket_connection( # type: ignore + session_id=self.session_id, + server_hostname=hostname, + server_port=port, + logger=self.logger, + sock_send_lock=self.sock_send_lock, + receive_timeout=self.receive_timeout, + proxy=self.proxy, + proxy_headers=self.proxy_headers, + trace_enabled=self.trace_enabled, + ssl_context=self.ssl_context, + ) + + # WebSocket handshake + try: + path = f"{parsed_url.path}?{parsed_url.query}" + sec_websocket_key = _generate_sec_websocket_key() + message = f"""GET {path} HTTP/1.1 + Host: {parsed_url.hostname} + Upgrade: websocket + Connection: Upgrade + Sec-WebSocket-Key: {sec_websocket_key} + Sec-WebSocket-Version: 13 + + """ + req: str = "\r\n".join([line.lstrip() for line in message.split("\n")]) + if self.trace_enabled: + self.logger.debug( + f"{self.connection_type_name} handshake request (session id: {self.session_id}):\n{req}" + ) + with self.sock_send_lock: + sock.send(req.encode("utf-8")) + + status, headers, text = _parse_handshake_response(sock) + if self.trace_enabled: + self.logger.debug( + f"{self.connection_type_name} handshake response (session id: {self.session_id}):\n{text}" + ) + # HTTP/1.1 101 Switching Protocols + if status == 101: + if not _validate_sec_websocket_accept(sec_websocket_key, headers): + raise SlackClientNotConnectedError( + f"Invalid response header detected in {self.connection_type_name} handshake response" + f" (session id: {self.session_id})" + ) + # set this successfully connected socket + self.sock = sock + self.ping(f"{self.session_id}:{time.time()}") + else: + message = ( + f"Received an unexpected response for handshake " + f"(status: {status}, response: {text}, session id: {self.session_id})" + ) + self.logger.warning(message) + + except socket.error as e: + code: Optional[int] = None + if e.args and len(e.args) > 1 and isinstance(e.args[0], int): + code = e.args[0] + if code is not None: + error_message = f"Error code: {code} (session id: {self.session_id}, error: {e})" + if self.trace_enabled: + self.logger.exception(error_message) + else: + self.logger.error(error_message) + raise + + except Exception as e: + error_message = f"Failed to establish a connection (session id: {self.session_id}, error: {e})" + if self.trace_enabled: + self.logger.exception(error_message) + else: + self.logger.error(error_message) + + if self.on_error_listener is not None: + self.on_error_listener(e) + + self.disconnect() + + def disconnect(self) -> None: + if self.sock is not None: + with self.sock_send_lock: + with self.sock_receive_lock: + # Synchronize before closing this instance's socket + self.sock.close() + self.sock = None + # After this, all operations using self.sock will be skipped + + self.logger.info(f"The connection has been closed (session id: {self.session_id})") + + def is_active(self) -> bool: + return self.sock is not None + + def close(self) -> None: + self.disconnect() + + def ping(self, payload: Union[str, bytes] = "") -> None: + if self.trace_enabled and self.ping_pong_trace_enabled: + if isinstance(payload, bytes): + payload = payload.decode("utf-8") + self.logger.debug("Sending a ping data frame " f"(session id: {self.session_id}, payload: {payload})") + data = _build_data_frame_for_sending(payload, FrameHeader.OPCODE_PING) + with self.sock_send_lock: + if self.sock is not None: + self.sock.send(data) + else: + if self.ping_pong_trace_enabled: + self.logger.debug("Skipped sending a ping message as the underlying socket is no longer available.") + + def pong(self, payload: Union[str, bytes] = "") -> None: + if self.trace_enabled and self.ping_pong_trace_enabled: + if isinstance(payload, bytes): + payload = payload.decode("utf-8") + self.logger.debug("Sending a pong data frame " f"(session id: {self.session_id}, payload: {payload})") + data = _build_data_frame_for_sending(payload, FrameHeader.OPCODE_PONG) + with self.sock_send_lock: + if self.sock is not None: + self.sock.send(data) + else: + if self.ping_pong_trace_enabled: + self.logger.debug("Skipped sending a pong message as the underlying socket is no longer available.") + + def send(self, payload: str) -> None: + if self.trace_enabled: + if isinstance(payload, bytes): + payload = payload.decode("utf-8") + self.logger.debug("Sending a text data frame " f"(session id: {self.session_id}, payload: {payload})") + data = _build_data_frame_for_sending(payload, FrameHeader.OPCODE_TEXT) + with self.sock_send_lock: + try: + self.sock.send(data) + except Exception as e: + # In most cases, we want to retry this operation with a newly established connection. + # Getting this exception means that this connection has been replaced with a new one + # and it's no longer usable. + # The SocketModeClient implementation can do one retry when it gets this exception. + raise SlackClientNotConnectedError( + f"Failed to send a message as the connection is no longer active " + f"(session_id: {self.session_id}, error: {e})" + ) + + def check_state(self) -> None: + try: + if self.sock is not None: + try: + self.ping(f"{self.session_id}:{time.time()}") + except ssl.SSLZeroReturnError as e: + self.logger.info( + "Unable to send a ping message. Closing the connection..." + f" (session id: {self.session_id}, reason: {e})" + ) + self.disconnect() + return + + if self.last_ping_pong_time is not None: + disconnected_seconds = int(time.time() - self.last_ping_pong_time) + if self.trace_enabled and disconnected_seconds > self.ping_interval: + message = ( + f"{disconnected_seconds} seconds have passed " + f"since this client last received a pong response from the server " + f"(session id: {self.session_id})" + ) + self.logger.debug(message) + + is_stale = disconnected_seconds > self.ping_interval * 4 + if is_stale: + self.logger.info( + "The connection seems to be stale. Disconnecting..." + f" (session id: {self.session_id}," + f" reason: disconnected for {disconnected_seconds}+ seconds)" + ) + self.disconnect() + return + else: + self.logger.debug("This connection is already closed." f" (session id: {self.session_id})") + self.consecutive_check_state_error_count = 0 + except Exception as e: + error_message = ( + "Failed to check the state of sock " + f"(session id: {self.session_id}, error: {type(e).__name__}, message: {e})" + ) + if self.trace_enabled: + self.logger.exception(error_message) + else: + self.logger.error(error_message) + + self.consecutive_check_state_error_count += 1 + if self.consecutive_check_state_error_count >= 5: + self.disconnect() + + def run_until_completion(self, state: ConnectionState) -> None: + repeated_messages = {"payload": 0} + ping_count = 0 + pong_count = 0 + ping_pong_log_summary_size = 1000 + while not state.terminated: + try: + if self.is_active(): + received_messages: List[Tuple[Optional[FrameHeader], bytes]] = _receive_messages( + sock=self.sock, + sock_receive_lock=self.sock_receive_lock, + logger=self.logger, + receive_buffer_size=self.receive_buffer_size, + all_message_trace_enabled=self.all_message_trace_enabled, + ) + for message in received_messages: + header, data = message + + # ----------------- + # trace logging + + if self.trace_enabled is True: + opcode: str = _to_readable_opcode(header.opcode) if header else "-" + payload: str = _parse_text_payload(data, self.logger) + count: Optional[int] = repeated_messages.get(payload) + if count is None: + count = 1 + else: + count += 1 + repeated_messages = {payload: count} + if not self.ping_pong_trace_enabled and header is not None and header.opcode is not None: + if header.opcode == FrameHeader.OPCODE_PING: + ping_count += 1 + if ping_count % ping_pong_log_summary_size == 0: + self.logger.debug( + f"Received {ping_pong_log_summary_size} ping data frame " + f"(session id: {self.session_id})" + ) + ping_count = 0 + if header.opcode == FrameHeader.OPCODE_PONG: + pong_count += 1 + if pong_count % ping_pong_log_summary_size == 0: + self.logger.debug( + f"Received {ping_pong_log_summary_size} pong data frame " + f"(session id: {self.session_id})" + ) + pong_count = 0 + + ping_pong_to_skip = ( + header is not None + and header.opcode is not None + and (header.opcode == FrameHeader.OPCODE_PING or header.opcode == FrameHeader.OPCODE_PONG) + and not self.ping_pong_trace_enabled + ) + if not ping_pong_to_skip and count < 5: + # if so many same payloads came in, the trace logging should be skipped. + # e.g., after receiving "UNAUTHENTICATED: cache_error", many "opcode: -, payload: " + self.logger.debug( + "Received a new data frame " + f"(session id: {self.session_id}, opcode: {opcode}, payload: {payload})" + ) + + if header is None: + # Skip no header message + continue + + # ----------------- + # message with opcode + + if header.opcode == FrameHeader.OPCODE_PING: + self.pong(data) + elif header.opcode == FrameHeader.OPCODE_PONG: + str_message = data.decode("utf-8") + elements = str_message.split(":") + if len(elements) >= 2: + session_id, ping_time = elements[0], elements[1] + if self.session_id == session_id: + try: + self.last_ping_pong_time = float(ping_time) + except Exception as e: + self.logger.debug( + "Failed to parse a pong message " f" (message: {str_message}, error: {e}" + ) + elif header.opcode == FrameHeader.OPCODE_TEXT: + if self.on_message_listener is not None: + text = data.decode("utf-8") + self.on_message_listener(text) + elif header.opcode == FrameHeader.OPCODE_CLOSE: + if self.on_close_listener is not None: + if len(data) >= 2: + (code,) = struct.unpack("!H", data[:2]) + reason = data[2:].decode("utf-8") + self.on_close_listener(code, reason) + else: + self.on_close_listener(1005, "") + self.disconnect() + state.terminated = True + else: + # Just warn logging + opcode = _to_readable_opcode(header.opcode) if header else "-" + payload: Union[bytes, str] = data + if header.opcode != FrameHeader.OPCODE_BINARY: + try: + payload = data.decode("utf-8") if data is not None else "" + except Exception as e: + self.logger.info(f"Failed to convert the data to text {e}") + message = ( + "Received an unsupported data frame " + f"(session id: {self.session_id}, opcode: {opcode}, payload: {payload})" + ) + self.logger.warning(message) + else: + time.sleep(0.2) + except socket.timeout: + time.sleep(0.01) + except OSError as e: + # getting errno.EBADF and the socket is no longer available + if e.errno == 9 and state.terminated: + self.logger.debug( + "The reason why you got [Errno 9] Bad file descriptor here is " "the socket is no longer available." + ) + else: + if self.on_error_listener is not None: + self.on_error_listener(e) + else: + error_message = "Got an OSError while receiving data" f" (session id: {self.session_id}, error: {e})" + if self.trace_enabled: + self.logger.exception(error_message) + else: + self.logger.error(error_message) + + # As this connection no longer works in any way, terminating it + if self.is_active(): + try: + self.disconnect() + except Exception as disconnection_error: + error_message = ( + "Failed to disconnect" f" (session id: {self.session_id}, error: {disconnection_error})" + ) + if self.trace_enabled: + self.logger.exception(error_message) + else: + self.logger.error(error_message) + state.terminated = True + break + except Exception as e: + if self.on_error_listener is not None: + self.on_error_listener(e) + else: + error_message = "Got an exception while receiving data" f" (session id: {self.session_id}, error: {e})" + if self.trace_enabled: + self.logger.exception(error_message) + else: + self.logger.error(error_message) + + state.terminated = True diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/frame_header.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/frame_header.py new file mode 100644 index 0000000..8851b3e --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/frame_header.py @@ -0,0 +1,47 @@ +class FrameHeader: + fin: int + rsv1: int + rsv2: int + rsv3: int + opcode: int + masked: int + length: int + + # Opcode + # https://tools.ietf.org/html/rfc6455#section-5.2 + # Non-control frames + # %x0 denotes a continuation frame + OPCODE_CONTINUATION = 0x0 + # %x1 denotes a text frame + OPCODE_TEXT = 0x1 + # %x2 denotes a binary frame + OPCODE_BINARY = 0x2 + # %x3-7 are reserved for further non-control frames + + # Control frames + # %x8 denotes a connection close + OPCODE_CLOSE = 0x8 + # %x9 denotes a ping + OPCODE_PING = 0x9 + # %xA denotes a pong + OPCODE_PONG = 0xA + + # %xB-F are reserved for further control frames + + def __init__( + self, + opcode: int, + fin: int = 1, + rsv1: int = 0, + rsv2: int = 0, + rsv3: int = 0, + masked: int = 0, + length: int = 0, + ): + self.opcode = opcode + self.fin = fin + self.rsv1 = rsv1 + self.rsv2 = rsv2 + self.rsv3 = rsv3 + self.masked = masked + self.length = length diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/internals.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/internals.py new file mode 100644 index 0000000..6daf6f3 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/builtin/internals.py @@ -0,0 +1,403 @@ +import errno +import hashlib +import itertools +import os +import random +import socket +from socket import socket as Socket +import ssl +import struct +from base64 import encodebytes, b64encode +from hmac import compare_digest +from logging import Logger +from threading import Lock +from typing import Tuple, Optional, Union, List, Callable, Dict +from urllib.parse import urlparse, unquote + +from .frame_header import FrameHeader + + +def _parse_connect_response(sock: Socket) -> Tuple[Optional[int], str]: + status = None + lines = [] + while True: + line = [] + while True: + c = sock.recv(1) + line.append(c) + if c == b"\n": + break + line = b"".join(line).decode("utf-8").strip() + if line is None or len(line) == 0: + break + lines.append(line) + if not status: + status_line = line.split(" ", 2) + status = int(status_line[1]) + return status, "\n".join(lines) + + +def _use_or_create_ssl_context(ssl_context: Optional[ssl.SSLContext] = None): + return ssl_context if ssl_context is not None else ssl.create_default_context() + + +def _establish_new_socket_connection( + session_id: str, + server_hostname: str, + server_port: int, + logger: Logger, + sock_send_lock: Lock, + receive_timeout: float, + proxy: Optional[str], + proxy_headers: Optional[Dict[str, str]], + trace_enabled: bool, + ssl_context: Optional[ssl.SSLContext] = None, +) -> Union[ssl.SSLSocket, Socket]: + + ssl_context = _use_or_create_ssl_context(ssl_context) + + if proxy is not None: + parsed_proxy = urlparse(proxy) + proxy_host, proxy_port = parsed_proxy.hostname, parsed_proxy.port or 80 + sock = socket.create_connection((proxy_host, proxy_port), receive_timeout) + if hasattr(socket, "TCP_NODELAY"): + sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + if hasattr(socket, "SO_KEEPALIVE"): + sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) + message = [f"CONNECT {server_hostname}:{server_port} HTTP/1.0"] + if parsed_proxy.username is not None and parsed_proxy.password is not None: + # In the case where the proxy is "http://{username}:{password}@{hostname}:{port}" + raw_value = f"{unquote(parsed_proxy.username)}:{unquote(parsed_proxy.password)}" + auth = b64encode(raw_value.encode("utf-8")).decode("ascii") + message.append(f"Proxy-Authorization: Basic {auth}") + if proxy_headers is not None: + for k, v in proxy_headers.items(): + message.append(f"{k}: {v}") + message.append("") + message.append("") + req: str = "\r\n".join([line.lstrip() for line in message]) + if trace_enabled: + logger.debug(f"Proxy connect request (session id: {session_id}):\n{req}") + with sock_send_lock: + sock.send(req.encode("utf-8")) + status, text = _parse_connect_response(sock) + if trace_enabled: + log_message = f"Proxy connect response (session id: {session_id}):\n{text}" + logger.debug(log_message) + if status != 200: + raise Exception(f"Failed to connect to the proxy (proxy: {proxy}, connect status code: {status})") + + sock = ssl_context.wrap_socket( + sock, + do_handshake_on_connect=True, + suppress_ragged_eofs=True, + server_hostname=server_hostname, + ) + return sock + + if server_port != 443: + # only for library testing + logger.info(f"Using non-ssl socket to connect ({server_hostname}:{server_port})") + sock = socket.create_connection((server_hostname, server_port), timeout=3) + return sock + + sock = socket.create_connection((server_hostname, server_port), receive_timeout) + sock = ssl_context.wrap_socket( + sock, + do_handshake_on_connect=True, + suppress_ragged_eofs=True, + server_hostname=server_hostname, + ) + return sock + + +def _read_http_response_line(sock: ssl.SSLSocket) -> str: + cs = [] + while True: + c: str = sock.recv(1).decode("utf-8") + if c == "\r": + break + if c != "\n": + cs.append(c) + return "".join(cs) + + +def _parse_handshake_response(sock: ssl.SSLSocket) -> Tuple[Optional[int], dict, str]: + """Parses the handshake response. + + Args: + sock: The current active socket + + Returns: + (http status, headers, whole response as a str) + """ + lines = [] + status = None + headers = {} + while True: + line = _read_http_response_line(sock) + if status is None: + elements = line.split(" ") + if len(elements) > 2: + status = int(elements[1]) + else: + elements = line.split(":") + if len(elements) == 2: + headers[elements[0].strip().lower()] = elements[1].strip() + if line is None or len(line.strip()) == 0: + break + lines.append(line) + text = "\n".join(lines) + return (status, headers, text) + + +def _generate_sec_websocket_key() -> str: + return encodebytes(os.urandom(16)).decode("utf-8").strip() + + +def _validate_sec_websocket_accept(sec_websocket_key: str, headers: dict) -> bool: + v = (sec_websocket_key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11").encode("utf-8") + expected = encodebytes(hashlib.sha1(v).digest()).decode("utf-8").strip() + actual = headers.get("sec-websocket-accept", "").strip() + return compare_digest(expected, actual) + + +def _to_readable_opcode(opcode: int) -> str: + if opcode == FrameHeader.OPCODE_CONTINUATION: + return "continuation" + if opcode == FrameHeader.OPCODE_TEXT: + return "text" + if opcode == FrameHeader.OPCODE_BINARY: + return "binary" + if opcode == FrameHeader.OPCODE_CLOSE: + return "close" + if opcode == FrameHeader.OPCODE_PING: + return "ping" + if opcode == FrameHeader.OPCODE_PONG: + return "pong" + return "-" + + +def _parse_text_payload(data: Optional[bytes], logger: Logger) -> str: + try: + if data is not None and isinstance(data, bytes): + return data.decode("utf-8") + else: + return "" + except UnicodeDecodeError as e: + logger.debug(f"Failed to parse a payload (data: {data}, error: {e})") + return "" + + +def _receive_messages( + sock: ssl.SSLSocket, + sock_receive_lock: Lock, + logger: Logger, + receive_buffer_size: int = 1024, + all_message_trace_enabled: bool = False, +) -> List[Tuple[Optional[FrameHeader], bytes]]: + def receive(specific_buffer_size: Optional[int] = None): + size = specific_buffer_size if specific_buffer_size is not None else receive_buffer_size + with sock_receive_lock: + try: + received_bytes = sock.recv(size) + if all_message_trace_enabled: + if len(received_bytes) > 0: + logger.debug(f"Received bytes: {received_bytes}") + return received_bytes + except OSError as e: + # For Linux/macOS, errno.EBADF is the expected error for bad connections. + # The errno.ENOTSOCK can be sent when running on Windows OS. + if e.errno in (errno.EBADF, errno.ENOTSOCK): + logger.debug("The connection seems to be already closed.") + return bytes() + raise e + + return _fetch_messages( + messages=[], + receive=receive, + remaining_bytes=None, + current_mask_key=None, + current_header=None, + current_data=bytes(), + logger=logger, + ) + + +def _fetch_messages( + messages: List[Tuple[Optional[FrameHeader], bytes]], + receive: Callable[[Optional[int]], bytes], # buffer size + logger: Logger, + remaining_bytes: Optional[bytes] = None, + current_mask_key: Optional[str] = None, + current_header: Optional[FrameHeader] = None, + current_data: Optional[bytes] = None, +) -> List[Tuple[Optional[FrameHeader], bytes]]: + + if remaining_bytes is None: + # Fetch more to complete the current message + remaining_bytes = receive() # type: ignore + + if remaining_bytes is None or len(remaining_bytes) == 0: + # no more bytes + if current_header is not None: + _append_message(messages, current_header, current_data) + return messages + + if current_header is None: + # new message + if len(remaining_bytes) <= 2: + remaining_bytes += receive() # type: ignore + + if remaining_bytes[0] == 10: # \n + if current_data is not None and len(current_data) >= 0: + _append_message(messages, current_header, current_data) + _append_message(messages, None, remaining_bytes[:1]) + remaining_bytes = remaining_bytes[1:] + if len(remaining_bytes) == 0: + return messages + else: + return _fetch_messages( + messages=messages, + receive=receive, + remaining_bytes=remaining_bytes, + logger=logger, + ) + + # https://tools.ietf.org/html/rfc6455#section-5.2 + b1, b2 = remaining_bytes[0], remaining_bytes[1] + + # determine data length and the first index of the data part + current_data_length: int = b2 & 0b01111111 + idx_after_length_part: int = 2 + if current_data_length == 126: + if len(remaining_bytes) < 4: + remaining_bytes += receive(1024) + current_data_length = struct.unpack("!H", bytes(remaining_bytes[2:4]))[0] + idx_after_length_part = 4 + elif current_data_length == 127: + if len(remaining_bytes) < 10: + remaining_bytes += receive(1024) + current_data_length = struct.unpack("!Q", bytes(remaining_bytes[2:10]))[0] + idx_after_length_part = 10 + + current_header = FrameHeader( + fin=b1 & 0b10000000, + rsv1=b1 & 0b01000000, + rsv2=b1 & 0b00100000, + rsv3=b1 & 0b00010000, + opcode=b1 & 0b00001111, + masked=b2 & 0b10000000, + length=current_data_length, + ) + if current_header.masked > 0: + if current_mask_key is None: + idx1, idx2 = idx_after_length_part, idx_after_length_part + 4 + current_mask_key = remaining_bytes[idx1:idx2] + idx_after_length_part += 4 + + start, end = idx_after_length_part, idx_after_length_part + current_data_length + data_to_append = remaining_bytes[start:end] + + current_data = bytes() + if current_header.masked > 0: + for i in range(data_to_append): + mask = current_mask_key[i % 4] + data_to_append[i] ^= mask # type: ignore + current_data += data_to_append + else: + current_data += data_to_append + if len(current_data) == current_data_length: + _append_message(messages, current_header, current_data) + remaining_bytes = remaining_bytes[end:] + if len(remaining_bytes) > 0: + # continue with the remaining data + return _fetch_messages( + messages=messages, + receive=receive, + remaining_bytes=remaining_bytes, + logger=logger, + ) + else: + return messages + elif len(current_data) < current_data_length: + # need more bytes to complete this message + return _fetch_messages( + messages=messages, + receive=receive, + current_mask_key=current_mask_key, + current_header=current_header, + current_data=current_data, + logger=logger, + ) + else: + # This pattern is unexpected but set data with the expected length anyway + _append_message(current_header, current_data[:current_data_length]) # type: ignore + return messages + + # work in progress with the current_header/current_data + if current_header is not None: + length_needed = current_header.length - len(current_data) + if length_needed > len(remaining_bytes): + current_data += remaining_bytes + # need more bytes to complete this message + return _fetch_messages( + messages=messages, + receive=receive, + current_mask_key=current_mask_key, + current_header=current_header, + current_data=current_data, + logger=logger, + ) + else: + current_data += remaining_bytes[:length_needed] + _append_message(messages, current_header, current_data) + remaining_bytes = remaining_bytes[length_needed:] + if len(remaining_bytes) == 0: + return messages + else: + # continue with the remaining data + return _fetch_messages( + messages=messages, + receive=receive, + remaining_bytes=remaining_bytes, + logger=logger, + ) + + return messages + + +def _append_message( + messages: List[Tuple[Optional[FrameHeader], bytes]], + header: Optional[FrameHeader], + data: bytes, +) -> None: + messages.append((header, data)) + + +def _build_data_frame_for_sending( + payload: Union[str, bytes], + opcode: int, + fin: int = 1, + rsv1: int = 0, + rsv2: int = 0, + rsv3: int = 0, + masked: int = 1, +): + b1 = fin << 7 | rsv1 << 6 | rsv2 << 5 | rsv3 << 4 | opcode + header: bytes = bytes([b1]) + + original_payload_data: bytes = payload.encode("utf-8") if isinstance(payload, str) else payload + payload_length = len(original_payload_data) + if payload_length <= 125: + b2 = masked << 7 | payload_length + header += bytes([b2]) + else: + b2 = masked << 7 | 126 + header += struct.pack("!BH", b2, payload_length) + + mask_key: List[int] = random.choices(range(256), k=4) + header += bytes(mask_key) + + payload_data: bytes = bytes(byte ^ mask for byte, mask in zip(original_payload_data, itertools.cycle(mask_key))) + return header + payload_data diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/client.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/client.py new file mode 100644 index 0000000..6340445 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/client.py @@ -0,0 +1,157 @@ +import json +import logging +import time +from queue import Queue, Empty +from concurrent.futures.thread import ThreadPoolExecutor +from logging import Logger +from threading import Lock +from typing import Dict, Union, Any, Optional, List, Callable + +from slack_sdk.errors import SlackApiError +from slack_sdk.socket_mode.interval_runner import IntervalRunner +from slack_sdk.socket_mode.listeners import ( + WebSocketMessageListener, + SocketModeRequestListener, +) +from slack_sdk.socket_mode.request import SocketModeRequest +from slack_sdk.socket_mode.response import SocketModeResponse +from slack_sdk.web import WebClient + + +class BaseSocketModeClient: + logger: Logger + web_client: WebClient + app_token: str + wss_uri: str + message_queue: Queue + message_listeners: List[ + Union[ + WebSocketMessageListener, + Callable[["BaseSocketModeClient", dict, Optional[str]], None], + ] + ] + socket_mode_request_listeners: List[ + Union[ + SocketModeRequestListener, + Callable[["BaseSocketModeClient", SocketModeRequest], None], + ] + ] + + message_processor: IntervalRunner + message_workers: ThreadPoolExecutor + + closed: bool + connect_operation_lock: Lock + + def issue_new_wss_url(self) -> str: + try: + response = self.web_client.apps_connections_open(app_token=self.app_token) + return response["url"] + except SlackApiError as e: + if e.response["error"] == "ratelimited": + # NOTE: ratelimited errors rarely occur with this endpoint + delay = int(e.response.headers.get("Retry-After", "30")) # Tier1 + self.logger.info(f"Rate limited. Retrying in {delay} seconds...") + time.sleep(delay) + # Retry to issue a new WSS URL + return self.issue_new_wss_url() + else: + # other errors + self.logger.error(f"Failed to retrieve WSS URL: {e}") + raise e + + def is_connected(self) -> bool: + return False + + def connect(self) -> None: + raise NotImplementedError() + + def disconnect(self) -> None: + raise NotImplementedError() + + def connect_to_new_endpoint(self, force: bool = False): + try: + self.connect_operation_lock.acquire(blocking=True, timeout=5) + if force or not self.is_connected(): + self.logger.info("Connecting to a new endpoint...") + self.wss_uri = self.issue_new_wss_url() + self.connect() + self.logger.info("Connected to a new endpoint...") + finally: + self.connect_operation_lock.release() + + def close(self) -> None: + self.closed = True + self.disconnect() + + def send_message(self, message: str) -> None: + raise NotImplementedError() + + def send_socket_mode_response(self, response: Union[Dict[str, Any], SocketModeResponse]) -> None: + if isinstance(response, SocketModeResponse): + self.send_message(json.dumps(response.to_dict())) + else: + self.send_message(json.dumps(response)) + + def enqueue_message(self, message: str): + self.message_queue.put(message) + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"A new message enqueued (current queue size: {self.message_queue.qsize()})") + + def process_message(self): + try: + raw_message = self.message_queue.get(timeout=1) + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"A message dequeued (current queue size: {self.message_queue.qsize()})") + + if raw_message is not None: + message: dict = {} + if raw_message.startswith("{"): + message = json.loads(raw_message) + if message.get("type") == "disconnect": + self.connect_to_new_endpoint(force=True) + else: + + def _run_message_listeners(): + self.run_message_listeners(message, raw_message) + + self.message_workers.submit(_run_message_listeners) + except Empty: + pass + + def run_message_listeners(self, message: dict, raw_message: str) -> None: + type, envelope_id = message.get("type"), message.get("envelope_id") + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"Message processing started (type: {type}, envelope_id: {envelope_id})") + try: + # just in case, adding the same logic to reconnect here + if message.get("type") == "disconnect": + self.connect_to_new_endpoint(force=True) + return + + for listener in self.message_listeners: + try: + listener(self, message, raw_message) # type: ignore + except Exception as e: + self.logger.exception(f"Failed to run a message listener: {e}") + + if len(self.socket_mode_request_listeners) > 0: + request = SocketModeRequest.from_dict(message) + if request is not None: + for listener in self.socket_mode_request_listeners: + try: + listener(self, request) # type: ignore + except Exception as e: + self.logger.exception(f"Failed to run a request listener: {e}") + except Exception as e: + self.logger.exception(f"Failed to run message listeners: {e}") + finally: + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"Message processing completed (type: {type}, envelope_id: {envelope_id})") + + def process_messages(self) -> None: + while not self.closed: + try: + self.process_message() + except Exception as e: + self.logger.exception(f"Failed to process a message: {e}") diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/interval_runner.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/interval_runner.py new file mode 100644 index 0000000..2e7f132 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/interval_runner.py @@ -0,0 +1,33 @@ +import threading +from threading import Thread, Event +from typing import Callable + + +class IntervalRunner: + event: Event + thread: Thread + + def __init__(self, target: Callable[[], None], interval_seconds: float = 0.1): + self.event = threading.Event() + self.target = target + self.interval_seconds = interval_seconds + self.thread = threading.Thread(target=self._run) + self.thread.daemon = True + + def _run(self) -> None: + while not self.event.is_set(): + self.target() + self.event.wait(self.interval_seconds) + + def start(self) -> "IntervalRunner": + self.thread.start() + return self + + def is_alive(self) -> bool: + return self.thread is not None and self.thread.is_alive() + + def shutdown(self): + if self.is_alive(): + self.event.set() + self.thread.join() + self.thread = None diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/listeners.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/listeners.py new file mode 100644 index 0000000..3a10834 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/listeners.py @@ -0,0 +1,17 @@ +from typing import Optional + +from slack_sdk.socket_mode.request import SocketModeRequest + + +class WebSocketMessageListener: + def __call__( # type: ignore + client: "BaseSocketModeClient", # noqa: F821 + message: dict, + raw_message: Optional[str] = None, + ): # noqa: F821 + raise NotImplementedError() + + +class SocketModeRequestListener: + def __call__(client: "BaseSocketModeClient", request: SocketModeRequest): # type: ignore # noqa: F821 # noqa: F821 + raise NotImplementedError() diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/request.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/request.py new file mode 100644 index 0000000..6a1d38e --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/request.py @@ -0,0 +1,57 @@ +from typing import Union, Optional + +from slack_sdk.models import JsonObject + + +class SocketModeRequest: + type: str + envelope_id: str + payload: dict + accepts_response_payload: bool + retry_attempt: Optional[int] # events_api + retry_reason: Optional[str] # events_api + + def __init__( + self, + type: str, + envelope_id: str, + payload: Union[dict, JsonObject, str], + accepts_response_payload: Optional[bool] = None, + retry_attempt: Optional[int] = None, + retry_reason: Optional[str] = None, + ): + self.type = type + self.envelope_id = envelope_id + + if isinstance(payload, JsonObject): + self.payload = payload.to_dict() + elif isinstance(payload, dict): + self.payload = payload + elif isinstance(payload, str): + self.payload = {"text": payload} + else: + unexpected_payload_type = type(payload) # type: ignore + raise ValueError(f"Unsupported payload data type ({unexpected_payload_type})") + + self.accepts_response_payload = accepts_response_payload or False + self.retry_attempt = retry_attempt + self.retry_reason = retry_reason + + @classmethod + def from_dict(cls, message: dict) -> Optional["SocketModeRequest"]: + if all(k in message for k in ("type", "envelope_id", "payload")): + return SocketModeRequest( + type=message.get("type"), + envelope_id=message.get("envelope_id"), + payload=message.get("payload"), + accepts_response_payload=message.get("accepts_response_payload") or False, + retry_attempt=message.get("retry_attempt"), + retry_reason=message.get("retry_reason"), + ) + return None + + def to_dict(self) -> dict: # skipcq: PYL-W0221 + d = {"envelope_id": self.envelope_id} + if self.payload is not None: + d["payload"] = self.payload + return d diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/response.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/response.py new file mode 100644 index 0000000..4984af0 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/response.py @@ -0,0 +1,28 @@ +from typing import Union, Optional + +from slack_sdk.models import JsonObject + + +class SocketModeResponse: + envelope_id: str + payload: dict + + def __init__(self, envelope_id: str, payload: Optional[Union[dict, JsonObject, str]] = None): + self.envelope_id = envelope_id + + if payload is None: + self.payload = None + elif isinstance(payload, JsonObject): + self.payload = payload.to_dict() + elif isinstance(payload, dict): + self.payload = payload + elif isinstance(payload, str): + self.payload = {"text": payload} + else: + raise ValueError(f"Unsupported payload data type ({type(payload)})") + + def to_dict(self) -> dict: # skipcq: PYL-W0221 + d = {"envelope_id": self.envelope_id} + if self.payload is not None: + d["payload"] = self.payload + return d diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/websocket_client/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/websocket_client/__init__.py new file mode 100644 index 0000000..851622a --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/websocket_client/__init__.py @@ -0,0 +1,258 @@ +"""websocket-client bassd Socket Mode client + +* https://api.slack.com/apis/connections/socket +* https://slack.dev/python-slack-sdk/socket-mode/ +* https://pypi.org/project/websocket-client/ + +""" +import logging +from concurrent.futures.thread import ThreadPoolExecutor +from logging import Logger +from queue import Queue +from threading import Lock +from typing import Union, Optional, List, Callable, Tuple + +import websocket +from websocket import WebSocketApp, WebSocketException + +from slack_sdk.socket_mode.client import BaseSocketModeClient +from slack_sdk.socket_mode.interval_runner import IntervalRunner +from slack_sdk.socket_mode.listeners import ( + WebSocketMessageListener, + SocketModeRequestListener, +) +from slack_sdk.socket_mode.request import SocketModeRequest +from slack_sdk.web import WebClient + + +class SocketModeClient(BaseSocketModeClient): + logger: Logger + web_client: WebClient + app_token: str + wss_uri: Optional[str] + message_queue: Queue + message_listeners: List[ + Union[ + WebSocketMessageListener, + Callable[["BaseSocketModeClient", dict, Optional[str]], None], + ] + ] + socket_mode_request_listeners: List[ + Union[ + SocketModeRequestListener, + Callable[["BaseSocketModeClient", SocketModeRequest], None], + ] + ] + + current_app_monitor: IntervalRunner + current_app_monitor_started: bool + message_processor: IntervalRunner + message_workers: ThreadPoolExecutor + + current_session: Optional[WebSocketApp] + current_session_runner: IntervalRunner + + auto_reconnect_enabled: bool + default_auto_reconnect_enabled: bool + + close: bool + connect_operation_lock: Lock + + on_open_listeners: List[Callable[[WebSocketApp], None]] + on_message_listeners: List[Callable[[WebSocketApp, str], None]] + on_error_listeners: List[Callable[[WebSocketApp, Exception], None]] + on_close_listeners: List[Callable[[WebSocketApp], None]] + + def __init__( + self, + app_token: str, + logger: Optional[Logger] = None, + web_client: Optional[WebClient] = None, + auto_reconnect_enabled: bool = True, + ping_interval: float = 10, + concurrency: int = 10, + trace_enabled: bool = False, + http_proxy_host: Optional[str] = None, + http_proxy_port: Optional[int] = None, + http_proxy_auth: Optional[Tuple[str, str]] = None, + proxy_type: Optional[str] = None, + on_open_listeners: Optional[List[Callable[[WebSocketApp], None]]] = None, + on_message_listeners: Optional[List[Callable[[WebSocketApp, str], None]]] = None, + on_error_listeners: Optional[List[Callable[[WebSocketApp, Exception], None]]] = None, + on_close_listeners: Optional[List[Callable[[WebSocketApp], None]]] = None, + ): + """ + + Args: + app_token: App-level token + logger: Custom logger + web_client: Web API client + auto_reconnect_enabled: True if automatic reconnection is enabled (default: True) + ping_interval: interval for ping-pong with Slack servers (seconds) + concurrency: the size of thread pool (default: 10) + http_proxy_host: the HTTP proxy host + http_proxy_port: the HTTP proxy port + http_proxy_auth: the HTTP proxy username & password + proxy_type: the HTTP proxy type + on_open_listeners: listener functions for on_open + on_message_listeners: listener functions for on_message + on_error_listeners: listener functions for on_error + on_close_listeners: listener functions for on_close + """ + self.app_token = app_token + self.logger = logger or logging.getLogger(__name__) + self.web_client = web_client or WebClient() + self.default_auto_reconnect_enabled = auto_reconnect_enabled + self.auto_reconnect_enabled = self.default_auto_reconnect_enabled + self.ping_interval = ping_interval + self.wss_uri = None + self.message_queue = Queue() + self.message_listeners = [] + self.socket_mode_request_listeners = [] + + self.current_session = None + self.current_session_runner = IntervalRunner(self._run_current_session, 0.5).start() + + self.current_app_monitor_started = False + self.current_app_monitor = IntervalRunner(self._monitor_current_session, self.ping_interval) + + self.closed = False + self.connect_operation_lock = Lock() + + self.message_processor = IntervalRunner(self.process_messages, 0.001).start() + self.message_workers = ThreadPoolExecutor(max_workers=concurrency) + + # NOTE: only global settings is provided by the library + websocket.enableTrace(trace_enabled) + + self.http_proxy_host = http_proxy_host + self.http_proxy_port = http_proxy_port + self.http_proxy_auth = http_proxy_auth + self.proxy_type = proxy_type + + self.on_open_listeners = on_open_listeners or [] + self.on_message_listeners = on_message_listeners or [] + self.on_error_listeners = on_error_listeners or [] + self.on_close_listeners = on_close_listeners or [] + + def is_connected(self) -> bool: + return self.current_session is not None + + def connect(self) -> None: + def on_open(ws: WebSocketApp): + if self.logger.level <= logging.DEBUG: + self.logger.debug("on_open invoked") + for listener in self.on_open_listeners: + listener(ws) + + def on_message(ws: WebSocketApp, message: str): + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"on_message invoked: (message: {message})") + self.enqueue_message(message) + for listener in self.on_message_listeners: + listener(ws, message) + + def on_error(ws: WebSocketApp, error: Exception): + self.logger.error(f"on_error invoked (error: {type(error).__name__}, message: {error})") + for listener in self.on_error_listeners: + listener(ws, error) + + def on_close( + ws: WebSocketApp, + close_status_code: Optional[int] = None, + close_msg: Optional[str] = None, + ): + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"on_close invoked: (code: {close_status_code}, message: {close_msg})") + if self.auto_reconnect_enabled: + self.logger.info("Received CLOSE event. Reconnecting...") + self.connect_to_new_endpoint() + for listener in self.on_close_listeners: + listener(ws) + + old_session: Optional[WebSocketApp] = self.current_session + + if self.wss_uri is None: + self.wss_uri = self.issue_new_wss_url() + + self.current_session = websocket.WebSocketApp( + self.wss_uri, + on_open=on_open, + on_message=on_message, + on_error=on_error, + on_close=on_close, + ) + self.auto_reconnect_enabled = self.default_auto_reconnect_enabled + + if not self.current_app_monitor_started: + self.current_app_monitor_started = True + self.current_app_monitor.start() + + if old_session is not None: + old_session.close() + + self.logger.info("A new session has been established") + + def disconnect(self) -> None: + if self.current_session is not None: + self.current_session.close() + + def send_message(self, message: str) -> None: + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"Sending a message: {message}") + try: + self.current_session.send(message) + except WebSocketException as e: + # We rarely get this exception while replacing the underlying WebSocket connections. + # We can do one more try here as the self.current_session should be ready now. + if self.logger.level <= logging.DEBUG: + self.logger.debug( + f"Failed to send a message (error: {e}, message: {message})" + " as the underlying connection was replaced. Retrying the same request only one time..." + ) + # Although acquiring self.connect_operation_lock also for the first method call is the safest way, + # we avoid synchronizing a lot for better performance. That's why we are doing a retry here. + with self.connect_operation_lock: + if self.is_connected(): + self.current_session.send(message) + else: + self.logger.warning( # type: ignore + f"The current session (session id: {self.session_id()}) is no longer active. " # type: ignore + "Failed to send a message" + ) + raise e + + def close(self) -> None: # type: ignore + self.closed = True + self.auto_reconnect_enabled = False + self.disconnect() + self.current_app_monitor.shutdown() + self.message_processor.shutdown() + self.message_workers.shutdown() + + def _run_current_session(self): + if self.current_session is not None: + try: + self.logger.info("Starting to receive messages from a new connection") + self.current_session.run_forever( + ping_interval=self.ping_interval, + http_proxy_host=self.http_proxy_host, + http_proxy_port=self.http_proxy_port, + http_proxy_auth=self.http_proxy_auth, + proxy_type=self.proxy_type, + ) + self.logger.info("Stopped receiving messages from a connection") + except Exception as e: + self.logger.exception(f"Failed to start or stop the current session: {e}") + + def _monitor_current_session(self): + if self.current_app_monitor_started: + try: + if self.auto_reconnect_enabled and (self.current_session is None or self.current_session.sock is None): + self.logger.info("The session seems to be already closed. Reconnecting...") + self.connect_to_new_endpoint() + except Exception as e: + self.logger.error( + "Failed to check the current session or reconnect to the server " + f"(error: {type(e).__name__}, message: {e})" + ) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/websockets/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/websockets/__init__.py new file mode 100644 index 0000000..39ed6b3 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/socket_mode/websockets/__init__.py @@ -0,0 +1,252 @@ +"""websockets bassd Socket Mode client + +* https://api.slack.com/apis/connections/socket +* https://slack.dev/python-slack-sdk/socket-mode/ +* https://pypi.org/project/websockets/ + +""" +import asyncio +import logging +from asyncio import Future, Lock +from logging import Logger +from asyncio import Queue +from typing import Union, Optional, List, Callable, Awaitable + +import websockets +from websockets.exceptions import WebSocketException + +# To keep compatibility with websockets 8.x, we use this import over .legacy.client +from websockets import WebSocketClientProtocol + +from slack_sdk.socket_mode.async_client import AsyncBaseSocketModeClient +from slack_sdk.socket_mode.async_listeners import ( + AsyncWebSocketMessageListener, + AsyncSocketModeRequestListener, +) +from slack_sdk.socket_mode.request import SocketModeRequest +from slack_sdk.web.async_client import AsyncWebClient + + +class SocketModeClient(AsyncBaseSocketModeClient): + logger: Logger + web_client: AsyncWebClient + app_token: str + wss_uri: Optional[str] + auto_reconnect_enabled: bool + message_queue: Queue + message_listeners: List[ + Union[ + AsyncWebSocketMessageListener, + Callable[["AsyncBaseSocketModeClient", dict, Optional[str]], Awaitable[None]], + ] + ] + socket_mode_request_listeners: List[ + Union[ + AsyncSocketModeRequestListener, + Callable[["AsyncBaseSocketModeClient", SocketModeRequest], Awaitable[None]], + ] + ] + + message_receiver: Optional[Future] + message_processor: Future + + ping_interval: float + trace_enabled: bool + + current_session: Optional[WebSocketClientProtocol] + current_session_monitor: Optional[Future] + + auto_reconnect_enabled: bool + default_auto_reconnect_enabled: bool + closed: bool + connect_operation_lock: Lock + + def __init__( + self, + app_token: str, + logger: Optional[Logger] = None, + web_client: Optional[AsyncWebClient] = None, + auto_reconnect_enabled: bool = True, + ping_interval: float = 10, + trace_enabled: bool = False, + ): + """Socket Mode client + + Args: + app_token: App-level token + logger: Custom logger + web_client: Web API client + auto_reconnect_enabled: True if automatic reconnection is enabled (default: True) + ping_interval: interval for ping-pong with Slack servers (seconds) + trace_enabled: True if more verbose logs to see what's happening under the hood + """ + self.app_token = app_token + self.logger = logger or logging.getLogger(__name__) + self.web_client = web_client or AsyncWebClient() + self.closed = False + self.connect_operation_lock = Lock() + self.default_auto_reconnect_enabled = auto_reconnect_enabled + self.auto_reconnect_enabled = self.default_auto_reconnect_enabled + self.ping_interval = ping_interval + self.trace_enabled = trace_enabled + self.wss_uri = None + self.message_queue = Queue() + self.message_listeners = [] + self.socket_mode_request_listeners = [] + self.current_session = None + self.current_session_monitor = None + + self.message_receiver = None + self.message_processor = asyncio.ensure_future(self.process_messages()) + + async def monitor_current_session(self) -> None: + # In the asyncio runtime, accessing a shared object (self.current_session here) from + # multiple tasks can cause race conditions and errors. + # To avoid such, we access only the session that is active when this loop starts. + session: WebSocketClientProtocol = self.current_session + session_id: str = await self.session_id() + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"A new monitor_current_session() execution loop for {session_id} started") + try: + while not self.closed: + if session != self.current_session: + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"The monitor_current_session task for {session_id} is now cancelled") + break + await asyncio.sleep(self.ping_interval) + try: + if self.auto_reconnect_enabled and (session is None or session.closed): + self.logger.info(f"The session ({session_id}) seems to be already closed. Reconnecting...") + await self.connect_to_new_endpoint() + except Exception as e: + self.logger.error( + "Failed to check the current session or reconnect to the server " + f"(error: {type(e).__name__}, message: {e}, session: {session_id})" + ) + except asyncio.CancelledError: + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"The monitor_current_session task for {session_id} is now cancelled") + raise + + async def receive_messages(self) -> None: + # In the asyncio runtime, accessing a shared object (self.current_session here) from + # multiple tasks can cause race conditions and errors. + # To avoid such, we access only the session that is active when this loop starts. + session: WebSocketClientProtocol = self.current_session + session_id: str = await self.session_id() + consecutive_error_count = 0 + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"A new receive_messages() execution loop with {session_id} started") + try: + while not self.closed: + if session != self.current_session: + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"The running receive_messages task for {session_id} is now cancelled") + break + try: + message = await session.recv() + if message is not None: + if isinstance(message, bytes): + message = message.decode("utf-8") + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"Received message: {message}, session: {session_id}") + await self.enqueue_message(message) + consecutive_error_count = 0 + except Exception as e: + consecutive_error_count += 1 + self.logger.error( + f"Failed to receive or enqueue a message: {type(e).__name__}, error: {e}, session: {session_id}" + ) + if isinstance(e, websockets.ConnectionClosedError): + await asyncio.sleep(self.ping_interval) + else: + await asyncio.sleep(consecutive_error_count) + except asyncio.CancelledError: + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"The running receive_messages task for {session_id} is now cancelled") + raise + + async def is_connected(self) -> bool: + return not self.closed and self.current_session is not None and not self.current_session.closed + + async def session_id(self) -> str: + return self.build_session_id(self.current_session) + + async def connect(self): + if self.wss_uri is None: + self.wss_uri = await self.issue_new_wss_url() + old_session: Optional[WebSocketClientProtocol] = None if self.current_session is None else self.current_session + # NOTE: websockets does not support proxy settings + self.current_session = await websockets.connect( + uri=self.wss_uri, + ping_interval=self.ping_interval, + ) + session_id = await self.session_id() + self.auto_reconnect_enabled = self.default_auto_reconnect_enabled + self.logger.info(f"A new session ({session_id}) has been established") + + if self.current_session_monitor is not None: + self.current_session_monitor.cancel() + self.current_session_monitor = asyncio.ensure_future(self.monitor_current_session()) + + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"A new monitor_current_session() executor has been recreated for {session_id}") + + if self.message_receiver is not None: + self.message_receiver.cancel() + self.message_receiver = asyncio.ensure_future(self.receive_messages()) + + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"A new receive_messages() executor has been recreated for {session_id}") + + if old_session is not None: + await old_session.close() + old_session_id = self.build_session_id(old_session) + self.logger.info(f"The old session ({old_session_id}) has been abandoned") + + async def disconnect(self): + if self.current_session is not None: + await self.current_session.close() + + async def send_message(self, message: str): + session = self.current_session + session_id = self.build_session_id(session) + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"Sending a message: {message}, session: {session_id}") + try: + await session.send(message) + except WebSocketException as e: + # We rarely get this exception while replacing the underlying WebSocket connections. + # We can do one more try here as the self.current_session should be ready now. + if self.logger.level <= logging.DEBUG: + self.logger.debug( + f"Failed to send a message (error: {e}, message: {message}, session: {session_id})" + " as the underlying connection was replaced. Retrying the same request only one time..." + ) + # Although acquiring self.connect_operation_lock also for the first method call is the safest way, + # we avoid synchronizing a lot for better performance. That's why we are doing a retry here. + try: + if await self.is_connected(): + await self.current_session.send(message) + else: + self.logger.warning(f"The current session ({session_id}) is no longer active. Failed to send a message") + raise e + finally: + if self.connect_operation_lock.locked() is True: + self.connect_operation_lock.release() + + async def close(self): + self.closed = True + self.auto_reconnect_enabled = False + await self.disconnect() + self.message_processor.cancel() + if self.current_session_monitor is not None: + self.current_session_monitor.cancel() + if self.message_receiver is not None: + self.message_receiver.cancel() + + @classmethod + def build_session_id(cls, session: WebSocketClientProtocol) -> str: + if session is None: + return "" + return "s_" + str(hash(session)) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/version.py b/core_service/aws_lambda/project/packages/slack_sdk/version.py new file mode 100644 index 0000000..5ea4f85 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/version.py @@ -0,0 +1,2 @@ +"""Check the latest version at https://pypi.org/project/slack-sdk/""" +__version__ = "3.17.2" diff --git a/core_service/aws_lambda/project/packages/slack_sdk/web/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/web/__init__.py new file mode 100644 index 0000000..a3d5ef2 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/web/__init__.py @@ -0,0 +1,9 @@ +"""The Slack Web API allows you to build applications that interact with Slack +in more complex ways than the integrations we provide out of the box.""" +from .client import WebClient +from .slack_response import SlackResponse + +__all__ = [ + "WebClient", + "SlackResponse", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/web/async_base_client.py b/core_service/aws_lambda/project/packages/slack_sdk/web/async_base_client.py new file mode 100644 index 0000000..935487c --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/web/async_base_client.py @@ -0,0 +1,216 @@ +import logging +from ssl import SSLContext +from typing import Optional, Union, Dict, Any, List + +import aiohttp +from aiohttp import FormData, BasicAuth + +from .async_internal_utils import ( + _files_to_data, + _request_with_session, +) # type: ignore +from .async_slack_response import AsyncSlackResponse +from .deprecation import show_2020_01_deprecation +from .internal_utils import ( + convert_bool_to_0_or_1, + _build_req_args, + _get_url, + get_user_agent, +) +from ..proxy_env_variable_loader import load_http_proxy_from_env + +from slack_sdk.http_retry.builtin_async_handlers import async_default_handlers +from slack_sdk.http_retry.handler import RetryHandler + + +class AsyncBaseClient: + BASE_URL = "https://www.slack.com/api/" + + def __init__( + self, + token: Optional[str] = None, + base_url: str = BASE_URL, + timeout: int = 30, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + session: Optional[aiohttp.ClientSession] = None, + trust_env_in_session: bool = False, + headers: Optional[dict] = None, + user_agent_prefix: Optional[str] = None, + user_agent_suffix: Optional[str] = None, + # for Org-Wide App installation + team_id: Optional[str] = None, + logger: Optional[logging.Logger] = None, + retry_handlers: Optional[List[RetryHandler]] = None, + ): + self.token = None if token is None else token.strip() + """A string specifying an `xoxp-*` or `xoxb-*` token.""" + self.base_url = base_url + """A string representing the Slack API base URL. + Default is `'https://www.slack.com/api/'`.""" + self.timeout = timeout + """The maximum number of seconds the client will wait + to connect and receive a response from Slack. + Default is 30 seconds.""" + self.ssl = ssl + """An [`ssl.SSLContext`](https://docs.python.org/3/library/ssl.html#ssl.SSLContext) + instance, helpful for specifying your own custom + certificate chain.""" + self.proxy = proxy + """String representing a fully-qualified URL to a proxy through which + to route all requests to the Slack API. Even if this parameter + is not specified, if any of the following environment variables are + present, they will be loaded into this parameter: `HTTPS_PROXY`, + `https_proxy`, `HTTP_PROXY` or `http_proxy`.""" + self.session = session + """An [`aiohttp.ClientSession`](https://docs.aiohttp.org/en/stable/client_reference.html#client-session) + to attach to all outgoing requests.""" + # https://github.com/slackapi/python-slack-sdk/issues/738 + self.trust_env_in_session = trust_env_in_session + """Boolean setting whether aiohttp outgoing requests + are allowed to read environment variables. Commonly used in conjunction + with proxy support via the `HTTPS_PROXY`, `https_proxy`, `HTTP_PROXY` and + `http_proxy` environment variables.""" + self.headers = headers or {} + """`dict` representing additional request headers to attach to all requests.""" + self.headers["User-Agent"] = get_user_agent(user_agent_prefix, user_agent_suffix) + self.default_params = {} + if team_id is not None: + self.default_params["team_id"] = team_id + self._logger = logger if logger is not None else logging.getLogger(__name__) + self.retry_handlers = retry_handlers if retry_handlers is not None else async_default_handlers() + + if self.proxy is None or len(self.proxy.strip()) == 0: + env_variable = load_http_proxy_from_env(self._logger) + if env_variable is not None: + self.proxy = env_variable + + async def api_call( # skipcq: PYL-R1710 + self, + api_method: str, + *, + http_verb: str = "POST", + files: Optional[dict] = None, + data: Union[dict, FormData] = None, + params: Optional[dict] = None, + json: Optional[dict] = None, # skipcq: PYL-W0621 + headers: Optional[dict] = None, + auth: Optional[dict] = None, + ) -> AsyncSlackResponse: + """Create a request and execute the API call to Slack. + + Args: + api_method (str): The target Slack API method. + e.g. 'chat.postMessage' + http_verb (str): HTTP Verb. e.g. 'POST' + files (dict): Files to multipart upload. + e.g. {image OR file: file_object OR file_path} + data: The body to attach to the request. If a dictionary is + provided, form-encoding will take place. + e.g. {'key1': 'value1', 'key2': 'value2'} + params (dict): The URL parameters to append to the URL. + e.g. {'key1': 'value1', 'key2': 'value2'} + json (dict): JSON for the body to attach to the request + (if files or data is not specified). + e.g. {'key1': 'value1', 'key2': 'value2'} + headers (dict): Additional request headers + auth (dict): A dictionary that consists of client_id and client_secret + + Returns: + (AsyncSlackResponse) + The server's response to an HTTP request. Data + from the response can be accessed like a dict. + If the response included 'next_cursor' it can + be iterated on to execute subsequent requests. + + Raises: + SlackApiError: The following Slack API call failed: + 'chat.postMessage'. + SlackRequestError: Json data can only be submitted as + POST requests. + """ + + api_url = _get_url(self.base_url, api_method) + if auth is not None: + if isinstance(auth, dict): + auth = BasicAuth(auth["client_id"], auth["client_secret"]) + if isinstance(auth, BasicAuth): + if headers is None: + headers = {} + headers["Authorization"] = auth.encode() + auth = None + + headers = headers or {} + headers.update(self.headers) + req_args = _build_req_args( + token=self.token, + http_verb=http_verb, + files=files, + data=data, + default_params=self.default_params, + params=params, + json=json, # skipcq: PYL-W0621 + headers=headers, + auth=auth, + ssl=self.ssl, + proxy=self.proxy, + ) + + show_2020_01_deprecation(api_method) + + return await self._send( + http_verb=http_verb, + api_url=api_url, + req_args=req_args, + ) + + async def _send(self, http_verb: str, api_url: str, req_args: dict) -> AsyncSlackResponse: + """Sends the request out for transmission. + + Args: + http_verb (str): The HTTP verb. e.g. 'GET' or 'POST'. + api_url (str): The Slack API url. e.g. 'https://slack.com/api/chat.postMessage' + req_args (dict): The request arguments to be attached to the request. + e.g. + { + json: { + 'attachments': [{"pretext": "pre-hello", "text": "text-world"}], + 'channel': '#random' + } + } + Returns: + The response parsed into a AsyncSlackResponse object. + """ + open_files = _files_to_data(req_args) + try: + if "params" in req_args: + # True/False -> "1"/"0" + req_args["params"] = convert_bool_to_0_or_1(req_args["params"]) + + res = await self._request(http_verb=http_verb, api_url=api_url, req_args=req_args) + finally: + for f in open_files: + f.close() + + data = { + "client": self, + "http_verb": http_verb, + "api_url": api_url, + "req_args": req_args, + } + return AsyncSlackResponse(**{**data, **res}).validate() + + async def _request(self, *, http_verb, api_url, req_args) -> Dict[str, Any]: + """Submit the HTTP request with the running session or a new session. + Returns: + A dictionary of the response data. + """ + return await _request_with_session( + current_session=self.session, + timeout=self.timeout, + logger=self._logger, + http_verb=http_verb, + api_url=api_url, + req_args=req_args, + retry_handlers=self.retry_handlers, + ) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/web/async_client.py b/core_service/aws_lambda/project/packages/slack_sdk/web/async_client.py new file mode 100644 index 0000000..8eea408 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/web/async_client.py @@ -0,0 +1,4416 @@ +# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +# +# *** DO NOT EDIT THIS FILE *** +# +# 1) Modify slack_sdk/web/client.py +# 2) Run `python setup.py codegen` +# +# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +"""A Python module for interacting with Slack's Web API.""" +import json +import os +from io import IOBase +from typing import Union, Sequence, Optional, Dict, Tuple, Any, List + +import slack_sdk.errors as e +from slack_sdk.models.views import View +from .async_base_client import AsyncBaseClient, AsyncSlackResponse +from .internal_utils import ( + _parse_web_class_objects, + _update_call_participants, + _warn_if_text_or_attachment_fallback_is_missing, + _remove_none_values, +) +from ..models.attachments import Attachment +from ..models.blocks import Block +from ..models.metadata import Metadata + + +class AsyncWebClient(AsyncBaseClient): + """A WebClient allows apps to communicate with the Slack Platform's Web API. + + https://api.slack.com/methods + + The Slack Web API is an interface for querying information from + and enacting change in a Slack workspace. + + This client handles constructing and sending HTTP requests to Slack + as well as parsing any responses received into a `SlackResponse`. + + Attributes: + token (str): A string specifying an `xoxp-*` or `xoxb-*` token. + base_url (str): A string representing the Slack API base URL. + Default is `'https://www.slack.com/api/'` + timeout (int): The maximum number of seconds the client will wait + to connect and receive a response from Slack. + Default is 30 seconds. + ssl (SSLContext): An [`ssl.SSLContext`][1] instance, helpful for specifying + your own custom certificate chain. + proxy (str): String representing a fully-qualified URL to a proxy through + which to route all requests to the Slack API. Even if this parameter + is not specified, if any of the following environment variables are + present, they will be loaded into this parameter: `HTTPS_PROXY`, + `https_proxy`, `HTTP_PROXY` or `http_proxy`. + headers (dict): Additional request headers to attach to all requests. + + Methods: + `api_call`: Constructs a request and executes the API call to Slack. + + Example of recommended usage: + ```python + import os + from slack_sdk.web.async_client import AsyncWebClient + + client = AsyncWebClient(token=os.environ['SLACK_API_TOKEN']) + response = client.chat_postMessage( + channel='#random', + text="Hello world!") + assert response["ok"] + assert response["message"]["text"] == "Hello world!" + ``` + + Example manually creating an API request: + ```python + import os + from slack_sdk.web.async_client import AsyncWebClient + + client = AsyncWebClient(token=os.environ['SLACK_API_TOKEN']) + response = client.api_call( + api_method='chat.postMessage', + json={'channel': '#random','text': "Hello world!"} + ) + assert response["ok"] + assert response["message"]["text"] == "Hello world!" + ``` + + Note: + Any attributes or methods prefixed with _underscores are + intended to be "private" internal use only. They may be changed or + removed at anytime. + + [1]: https://docs.python.org/3/library/ssl.html#ssl.SSLContext + """ + + async def admin_analytics_getFile( + self, + *, + type: str, + date: Optional[str] = None, + metadata_only: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieve analytics data for a given date, presented as a compressed JSON file + https://api.slack.com/methods/admin.analytics.getFile + """ + kwargs.update({"type": type}) + if date is not None: + kwargs.update({"date": date}) + if metadata_only is not None: + kwargs.update({"metadata_only": metadata_only}) + return await self.api_call("admin.analytics.getFile", params=kwargs) + + async def admin_apps_approve( + self, + *, + app_id: Optional[str] = None, + request_id: Optional[str] = None, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Approve an app for installation on a workspace. + Either app_id or request_id is required. + These IDs can be obtained either directly via the app_requested event, + or by the admin.apps.requests.list method. + https://api.slack.com/methods/admin.apps.approve + """ + if app_id: + kwargs.update({"app_id": app_id}) + elif request_id: + kwargs.update({"request_id": request_id}) + else: + raise e.SlackRequestError("The app_id or request_id argument must be specified.") + + kwargs.update( + { + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return await self.api_call("admin.apps.approve", params=kwargs) + + async def admin_apps_approved_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List approved apps for an org or workspace. + https://api.slack.com/methods/admin.apps.approved.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return await self.api_call("admin.apps.approved.list", http_verb="GET", params=kwargs) + + async def admin_apps_clearResolution( + self, + *, + app_id: str, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Clear an app resolution + https://api.slack.com/methods/admin.apps.clearResolution + """ + kwargs.update( + { + "app_id": app_id, + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return await self.api_call("admin.apps.clearResolution", http_verb="POST", params=kwargs) + + async def admin_apps_requests_cancel( + self, + *, + request_id: str, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List app requests for a team/workspace. + https://api.slack.com/methods/admin.apps.requests.cancel + """ + kwargs.update( + { + "request_id": request_id, + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return await self.api_call("admin.apps.requests.cancel", http_verb="POST", params=kwargs) + + async def admin_apps_requests_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List app requests for a team/workspace. + https://api.slack.com/methods/admin.apps.requests.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + } + ) + return await self.api_call("admin.apps.requests.list", http_verb="GET", params=kwargs) + + async def admin_apps_restrict( + self, + *, + app_id: Optional[str] = None, + request_id: Optional[str] = None, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Restrict an app for installation on a workspace. + Exactly one of the team_id or enterprise_id arguments is required, not both. + Either app_id or request_id is required. These IDs can be obtained either directly + via the app_requested event, or by the admin.apps.requests.list method. + https://api.slack.com/methods/admin.apps.restrict + """ + if app_id: + kwargs.update({"app_id": app_id}) + elif request_id: + kwargs.update({"request_id": request_id}) + else: + raise e.SlackRequestError("The app_id or request_id argument must be specified.") + + kwargs.update( + { + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return await self.api_call("admin.apps.restrict", params=kwargs) + + async def admin_apps_restricted_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List restricted apps for an org or workspace. + https://api.slack.com/methods/admin.apps.restricted.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return await self.api_call("admin.apps.restricted.list", http_verb="GET", params=kwargs) + + async def admin_apps_uninstall( + self, + *, + app_id: str, + enterprise_id: Optional[str] = None, + team_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Uninstall an app from one or many workspaces, or an entire enterprise organization. + With an org-level token, enterprise_id or team_ids is required. + https://api.slack.com/methods/admin.apps.uninstall + """ + kwargs.update({"app_id": app_id}) + if enterprise_id is not None: + kwargs.update({"enterprise_id": enterprise_id}) + if team_ids is not None: + if isinstance(team_ids, (list, Tuple)): + kwargs.update({"team_ids": ",".join(team_ids)}) + else: + kwargs.update({"team_ids": team_ids}) + return await self.api_call("admin.apps.uninstall", http_verb="POST", params=kwargs) + + async def admin_auth_policy_getEntities( + self, + *, + policy_name: str, + cursor: Optional[str] = None, + entity_type: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Fetch all the entities assigned to a particular authentication policy by name. + https://api.slack.com/methods/admin.auth.policy.getEntities + """ + kwargs.update({"policy_name": policy_name}) + if cursor is not None: + kwargs.update({"cursor": cursor}) + if entity_type is not None: + kwargs.update({"entity_type": entity_type}) + if limit is not None: + kwargs.update({"limit": limit}) + return await self.api_call("admin.auth.policy.getEntities", http_verb="POST", params=kwargs) + + async def admin_auth_policy_assignEntities( + self, + *, + entity_ids: Union[str, Sequence[str]], + policy_name: str, + entity_type: str, + **kwargs, + ) -> AsyncSlackResponse: + """Assign entities to a particular authentication policy. + https://api.slack.com/methods/admin.auth.policy.assignEntities + """ + if isinstance(entity_ids, (list, Tuple)): + kwargs.update({"entity_ids": ",".join(entity_ids)}) + else: + kwargs.update({"entity_ids": entity_ids}) + kwargs.update({"policy_name": policy_name}) + kwargs.update({"entity_type": entity_type}) + return await self.api_call("admin.auth.policy.assignEntities", http_verb="POST", params=kwargs) + + async def admin_auth_policy_removeEntities( + self, + *, + entity_ids: Union[str, Sequence[str]], + policy_name: str, + entity_type: str, + **kwargs, + ) -> AsyncSlackResponse: + """Remove specified entities from a specified authentication policy. + https://api.slack.com/methods/admin.auth.policy.removeEntities + """ + if isinstance(entity_ids, (list, Tuple)): + kwargs.update({"entity_ids": ",".join(entity_ids)}) + else: + kwargs.update({"entity_ids": entity_ids}) + kwargs.update({"policy_name": policy_name}) + kwargs.update({"entity_type": entity_type}) + return await self.api_call("admin.auth.policy.removeEntities", http_verb="POST", params=kwargs) + + async def admin_barriers_create( + self, + *, + barriered_from_usergroup_ids: Union[str, Sequence[str]], + primary_usergroup_id: str, + restricted_subjects: Union[str, Sequence[str]], + **kwargs, + ) -> AsyncSlackResponse: + """Create an Information Barrier + https://api.slack.com/methods/admin.barriers.create + """ + kwargs.update({"primary_usergroup_id": primary_usergroup_id}) + if isinstance(barriered_from_usergroup_ids, (list, Tuple)): + kwargs.update({"barriered_from_usergroup_ids": ",".join(barriered_from_usergroup_ids)}) + else: + kwargs.update({"barriered_from_usergroup_ids": barriered_from_usergroup_ids}) + if isinstance(restricted_subjects, (list, Tuple)): + kwargs.update({"restricted_subjects": ",".join(restricted_subjects)}) + else: + kwargs.update({"restricted_subjects": restricted_subjects}) + return await self.api_call("admin.barriers.create", http_verb="POST", params=kwargs) + + async def admin_barriers_delete( + self, + *, + barrier_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Delete an existing Information Barrier + https://api.slack.com/methods/admin.barriers.delete + """ + kwargs.update({"barrier_id": barrier_id}) + return await self.api_call("admin.barriers.delete", http_verb="POST", params=kwargs) + + async def admin_barriers_update( + self, + *, + barrier_id: str, + barriered_from_usergroup_ids: Union[str, Sequence[str]], + primary_usergroup_id: str, + restricted_subjects: Union[str, Sequence[str]], + **kwargs, + ) -> AsyncSlackResponse: + """Update an existing Information Barrier + https://api.slack.com/methods/admin.barriers.update + """ + kwargs.update({"barrier_id": barrier_id, "primary_usergroup_id": primary_usergroup_id}) + if isinstance(barriered_from_usergroup_ids, (list, Tuple)): + kwargs.update({"barriered_from_usergroup_ids": ",".join(barriered_from_usergroup_ids)}) + else: + kwargs.update({"barriered_from_usergroup_ids": barriered_from_usergroup_ids}) + if isinstance(restricted_subjects, (list, Tuple)): + kwargs.update({"restricted_subjects": ",".join(restricted_subjects)}) + else: + kwargs.update({"restricted_subjects": restricted_subjects}) + return await self.api_call("admin.barriers.update", http_verb="POST", params=kwargs) + + async def admin_barriers_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Get all Information Barriers for your organization + https://api.slack.com/methods/admin.barriers.list""" + kwargs.update( + { + "cursor": cursor, + "limit": limit, + } + ) + return await self.api_call("admin.barriers.list", http_verb="GET", params=kwargs) + + async def admin_conversations_create( + self, + *, + is_private: bool, + name: str, + description: Optional[str] = None, + org_wide: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Create a public or private channel-based conversation. + https://api.slack.com/methods/admin.conversations.create + """ + kwargs.update( + { + "is_private": is_private, + "name": name, + "description": description, + "org_wide": org_wide, + "team_id": team_id, + } + ) + return await self.api_call("admin.conversations.create", params=kwargs) + + async def admin_conversations_delete( + self, + *, + channel_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Delete a public or private channel. + https://api.slack.com/methods/admin.conversations.delete + """ + kwargs.update({"channel_id": channel_id}) + return await self.api_call("admin.conversations.delete", params=kwargs) + + async def admin_conversations_invite( + self, + *, + channel_id: str, + user_ids: Union[str, Sequence[str]], + **kwargs, + ) -> AsyncSlackResponse: + """Invite a user to a public or private channel. + https://api.slack.com/methods/admin.conversations.invite + """ + kwargs.update({"channel_id": channel_id}) + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + # NOTE: the endpoint is unable to handle Content-Type: application/json as of Sep 3, 2020. + return await self.api_call("admin.conversations.invite", params=kwargs) + + async def admin_conversations_archive( + self, + *, + channel_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Archive a public or private channel. + https://api.slack.com/methods/admin.conversations.archive + """ + kwargs.update({"channel_id": channel_id}) + return await self.api_call("admin.conversations.archive", params=kwargs) + + async def admin_conversations_unarchive( + self, + *, + channel_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Unarchive a public or private channel. + https://api.slack.com/methods/admin.conversations.archive + """ + kwargs.update({"channel_id": channel_id}) + return await self.api_call("admin.conversations.unarchive", params=kwargs) + + async def admin_conversations_rename( + self, + *, + channel_id: str, + name: str, + **kwargs, + ) -> AsyncSlackResponse: + """Rename a public or private channel. + https://api.slack.com/methods/admin.conversations.rename + """ + kwargs.update({"channel_id": channel_id, "name": name}) + return await self.api_call("admin.conversations.rename", params=kwargs) + + async def admin_conversations_search( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + query: Optional[str] = None, + search_channel_types: Optional[Union[str, Sequence[str]]] = None, + sort: Optional[str] = None, + sort_dir: Optional[str] = None, + team_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Search for public or private channels in an Enterprise organization. + https://api.slack.com/methods/admin.conversations.search + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "query": query, + "sort": sort, + "sort_dir": sort_dir, + } + ) + + if isinstance(search_channel_types, (list, Tuple)): + kwargs.update({"search_channel_types": ",".join(search_channel_types)}) + else: + kwargs.update({"search_channel_types": search_channel_types}) + + if isinstance(team_ids, (list, Tuple)): + kwargs.update({"team_ids": ",".join(team_ids)}) + else: + kwargs.update({"team_ids": team_ids}) + + return await self.api_call("admin.conversations.search", params=kwargs) + + async def admin_conversations_convertToPrivate( + self, + *, + channel_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Convert a public channel to a private channel. + https://api.slack.com/methods/admin.conversations.convertToPrivate + """ + kwargs.update({"channel_id": channel_id}) + return await self.api_call("admin.conversations.convertToPrivate", params=kwargs) + + async def admin_conversations_setConversationPrefs( + self, + *, + channel_id: str, + prefs: Union[str, Dict[str, str]], + **kwargs, + ) -> AsyncSlackResponse: + """Set the posting permissions for a public or private channel. + https://api.slack.com/methods/admin.conversations.setConversationPrefs + """ + kwargs.update({"channel_id": channel_id}) + if isinstance(prefs, dict): + kwargs.update({"prefs": json.dumps(prefs)}) + else: + kwargs.update({"prefs": prefs}) + return await self.api_call("admin.conversations.setConversationPrefs", params=kwargs) + + async def admin_conversations_getConversationPrefs( + self, + *, + channel_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Get conversation preferences for a public or private channel. + https://api.slack.com/methods/admin.conversations.getConversationPrefs + """ + kwargs.update({"channel_id": channel_id}) + return await self.api_call("admin.conversations.getConversationPrefs", params=kwargs) + + async def admin_conversations_disconnectShared( + self, + *, + channel_id: str, + leaving_team_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Disconnect a connected channel from one or more workspaces. + https://api.slack.com/methods/admin.conversations.disconnectShared + """ + kwargs.update({"channel_id": channel_id}) + if isinstance(leaving_team_ids, (list, Tuple)): + kwargs.update({"leaving_team_ids": ",".join(leaving_team_ids)}) + else: + kwargs.update({"leaving_team_ids": leaving_team_ids}) + return await self.api_call("admin.conversations.disconnectShared", params=kwargs) + + async def admin_conversations_ekm_listOriginalConnectedChannelInfo( + self, + *, + channel_ids: Optional[Union[str, Sequence[str]]] = None, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List all disconnected channels—i.e., + channels that were once connected to other workspaces and then disconnected—and + the corresponding original channel IDs for key revocation with EKM. + https://api.slack.com/methods/admin.conversations.ekm.listOriginalConnectedChannelInfo + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + } + ) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + if isinstance(team_ids, (list, Tuple)): + kwargs.update({"team_ids": ",".join(team_ids)}) + else: + kwargs.update({"team_ids": team_ids}) + return await self.api_call("admin.conversations.ekm.listOriginalConnectedChannelInfo", params=kwargs) + + async def admin_conversations_restrictAccess_addGroup( + self, + *, + channel_id: str, + group_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Add an allowlist of IDP groups for accessing a channel. + https://api.slack.com/methods/admin.conversations.restrictAccess.addGroup + """ + kwargs.update( + { + "channel_id": channel_id, + "group_id": group_id, + "team_id": team_id, + } + ) + return await self.api_call( + "admin.conversations.restrictAccess.addGroup", + http_verb="GET", + params=kwargs, + ) + + async def admin_conversations_restrictAccess_listGroups( + self, + *, + channel_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List all IDP Groups linked to a channel. + https://api.slack.com/methods/admin.conversations.restrictAccess.listGroups + """ + kwargs.update( + { + "channel_id": channel_id, + "team_id": team_id, + } + ) + return await self.api_call( + "admin.conversations.restrictAccess.listGroups", + http_verb="GET", + params=kwargs, + ) + + async def admin_conversations_restrictAccess_removeGroup( + self, + *, + channel_id: str, + group_id: str, + team_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Remove a linked IDP group linked from a private channel. + https://api.slack.com/methods/admin.conversations.restrictAccess.removeGroup + """ + kwargs.update( + { + "channel_id": channel_id, + "group_id": group_id, + "team_id": team_id, + } + ) + return await self.api_call( + "admin.conversations.restrictAccess.removeGroup", + http_verb="GET", + params=kwargs, + ) + + async def admin_conversations_setTeams( + self, + *, + channel_id: str, + org_channel: Optional[bool] = None, + target_team_ids: Optional[Union[str, Sequence[str]]] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Set the workspaces in an Enterprise grid org that connect to a public or private channel. + https://api.slack.com/methods/admin.conversations.setTeams + """ + kwargs.update( + { + "channel_id": channel_id, + "org_channel": org_channel, + "team_id": team_id, + } + ) + if isinstance(target_team_ids, (list, Tuple)): + kwargs.update({"target_team_ids": ",".join(target_team_ids)}) + else: + kwargs.update({"target_team_ids": target_team_ids}) + return await self.api_call("admin.conversations.setTeams", params=kwargs) + + async def admin_conversations_getTeams( + self, + *, + channel_id: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Set the workspaces in an Enterprise grid org that connect to a channel. + https://api.slack.com/methods/admin.conversations.getTeams + """ + kwargs.update( + { + "channel_id": channel_id, + "cursor": cursor, + "limit": limit, + } + ) + return await self.api_call("admin.conversations.getTeams", params=kwargs) + + async def admin_conversations_getCustomRetention( + self, + *, + channel_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Get a channel's retention policy + https://api.slack.com/methods/admin.conversations.getCustomRetention + """ + kwargs.update({"channel_id": channel_id}) + return await self.api_call("admin.conversations.getCustomRetention", params=kwargs) + + async def admin_conversations_removeCustomRetention( + self, + *, + channel_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Remove a channel's retention policy + https://api.slack.com/methods/admin.conversations.removeCustomRetention + """ + kwargs.update({"channel_id": channel_id}) + return await self.api_call("admin.conversations.removeCustomRetention", params=kwargs) + + async def admin_conversations_setCustomRetention( + self, + *, + channel_id: str, + duration_days: int, + **kwargs, + ) -> AsyncSlackResponse: + """Set a channel's retention policy + https://api.slack.com/methods/admin.conversations.setCustomRetention + """ + kwargs.update({"channel_id": channel_id, "duration_days": duration_days}) + return await self.api_call("admin.conversations.setCustomRetention", params=kwargs) + + async def admin_emoji_add( + self, + *, + name: str, + url: str, + **kwargs, + ) -> AsyncSlackResponse: + """Add an emoji. + https://api.slack.com/methods/admin.emoji.add + """ + kwargs.update({"name": name, "url": url}) + return await self.api_call("admin.emoji.add", http_verb="GET", params=kwargs) + + async def admin_emoji_addAlias( + self, + *, + alias_for: str, + name: str, + **kwargs, + ) -> AsyncSlackResponse: + """Add an emoji alias. + https://api.slack.com/methods/admin.emoji.addAlias + """ + kwargs.update({"alias_for": alias_for, "name": name}) + return await self.api_call("admin.emoji.addAlias", http_verb="GET", params=kwargs) + + async def admin_emoji_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List emoji for an Enterprise Grid organization. + https://api.slack.com/methods/admin.emoji.list + """ + kwargs.update({"cursor": cursor, "limit": limit}) + return await self.api_call("admin.emoji.list", http_verb="GET", params=kwargs) + + async def admin_emoji_remove( + self, + *, + name: str, + **kwargs, + ) -> AsyncSlackResponse: + """Remove an emoji across an Enterprise Grid organization. + https://api.slack.com/methods/admin.emoji.remove + """ + kwargs.update({"name": name}) + return await self.api_call("admin.emoji.remove", http_verb="GET", params=kwargs) + + async def admin_emoji_rename( + self, + *, + name: str, + new_name: str, + **kwargs, + ) -> AsyncSlackResponse: + """Rename an emoji. + https://api.slack.com/methods/admin.emoji.rename + """ + kwargs.update({"name": name, "new_name": new_name}) + return await self.api_call("admin.emoji.rename", http_verb="GET", params=kwargs) + + async def admin_users_session_reset( + self, + *, + user_id: str, + mobile_only: Optional[bool] = None, + web_only: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Wipes all valid sessions on all devices for a given user. + https://api.slack.com/methods/admin.users.session.reset + """ + kwargs.update( + { + "user_id": user_id, + "mobile_only": mobile_only, + "web_only": web_only, + } + ) + return await self.api_call("admin.users.session.reset", params=kwargs) + + async def admin_users_session_resetBulk( + self, + *, + user_ids: Union[str, Sequence[str]], + mobile_only: Optional[bool] = None, + web_only: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Enqueues an asynchronous job to wipe all valid sessions on all devices for a given list of users + https://api.slack.com/methods/admin.users.session.resetBulk + """ + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + kwargs.update( + { + "mobile_only": mobile_only, + "web_only": web_only, + } + ) + return await self.api_call("admin.users.session.resetBulk", params=kwargs) + + async def admin_users_session_invalidate( + self, + *, + session_id: str, + team_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Invalidate a single session for a user by session_id. + https://api.slack.com/methods/admin.users.session.invalidate + """ + kwargs.update({"session_id": session_id, "team_id": team_id}) + return await self.api_call("admin.users.session.invalidate", params=kwargs) + + async def admin_users_session_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + user_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Lists all active user sessions for an organization + https://api.slack.com/methods/admin.users.session.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + "user_id": user_id, + } + ) + return await self.api_call("admin.users.session.list", params=kwargs) + + async def admin_teams_settings_setDefaultChannels( + self, + *, + team_id: str, + channel_ids: Union[str, Sequence[str]], + **kwargs, + ) -> AsyncSlackResponse: + """Set the default channels of a workspace. + https://api.slack.com/methods/admin.teams.settings.setDefaultChannels + """ + kwargs.update({"team_id": team_id}) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return await self.api_call("admin.teams.settings.setDefaultChannels", http_verb="GET", params=kwargs) + + async def admin_users_session_getSettings( + self, + *, + user_ids: Union[str, Sequence[str]], + **kwargs, + ) -> AsyncSlackResponse: + """Get user-specific session settings—the session duration + and what happens when the client closes—given a list of users. + https://api.slack.com/methods/admin.users.session.getSettings + """ + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + return await self.api_call("admin.users.session.getSettings", params=kwargs) + + async def admin_users_session_setSettings( + self, + *, + user_ids: Union[str, Sequence[str]], + desktop_app_browser_quit: Optional[bool] = None, + duration: Optional[int] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Configure the user-level session settings—the session duration + and what happens when the client closes—for one or more users. + https://api.slack.com/methods/admin.users.session.setSettings + """ + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + kwargs.update( + { + "desktop_app_browser_quit": desktop_app_browser_quit, + "duration": duration, + } + ) + return await self.api_call("admin.users.session.setSettings", params=kwargs) + + async def admin_users_session_clearSettings( + self, + *, + user_ids: Union[str, Sequence[str]], + **kwargs, + ) -> AsyncSlackResponse: + """Clear user-specific session settings—the session duration + and what happens when the client closes—for a list of users. + https://api.slack.com/methods/admin.users.session.clearSettings + """ + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + return await self.api_call("admin.users.session.clearSettings", params=kwargs) + + async def admin_users_unsupportedVersions_export( + self, + *, + date_end_of_support: Optional[Union[str, int]] = None, + date_sessions_started: Optional[Union[str, int]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Ask Slackbot to send you an export listing all workspace members using unsupported software, + presented as a zipped CSV file. + https://api.slack.com/methods/admin.users.unsupportedVersions.export + """ + kwargs.update( + { + "date_end_of_support": date_end_of_support, + "date_sessions_started": date_sessions_started, + } + ) + return await self.api_call("admin.users.unsupportedVersions.export", params=kwargs) + + async def admin_inviteRequests_approve( + self, + *, + invite_request_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Approve a workspace invite request. + https://api.slack.com/methods/admin.inviteRequests.approve + """ + kwargs.update({"invite_request_id": invite_request_id, "team_id": team_id}) + return await self.api_call("admin.inviteRequests.approve", params=kwargs) + + async def admin_inviteRequests_approved_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List all approved workspace invite requests. + https://api.slack.com/methods/admin.inviteRequests.approved.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + } + ) + return await self.api_call("admin.inviteRequests.approved.list", params=kwargs) + + async def admin_inviteRequests_denied_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List all denied workspace invite requests. + https://api.slack.com/methods/admin.inviteRequests.denied.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + } + ) + return await self.api_call("admin.inviteRequests.denied.list", params=kwargs) + + async def admin_inviteRequests_deny( + self, + *, + invite_request_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Deny a workspace invite request. + https://api.slack.com/methods/admin.inviteRequests.deny + """ + kwargs.update({"invite_request_id": invite_request_id, "team_id": team_id}) + return await self.api_call("admin.inviteRequests.deny", params=kwargs) + + async def admin_inviteRequests_list( + self, + **kwargs, + ) -> AsyncSlackResponse: + """List all pending workspace invite requests.""" + return await self.api_call("admin.inviteRequests.list", params=kwargs) + + async def admin_teams_admins_list( + self, + *, + team_id: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List all of the admins on a given workspace. + https://api.slack.com/methods/admin.inviteRequests.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + } + ) + return await self.api_call("admin.teams.admins.list", http_verb="GET", params=kwargs) + + async def admin_teams_create( + self, + *, + team_domain: str, + team_name: str, + team_description: Optional[str] = None, + team_discoverability: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Create an Enterprise team. + https://api.slack.com/methods/admin.teams.create + """ + kwargs.update( + { + "team_domain": team_domain, + "team_name": team_name, + "team_description": team_description, + "team_discoverability": team_discoverability, + } + ) + return await self.api_call("admin.teams.create", params=kwargs) + + async def admin_teams_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List all teams on an Enterprise organization. + https://api.slack.com/methods/admin.teams.list + """ + kwargs.update({"cursor": cursor, "limit": limit}) + return await self.api_call("admin.teams.list", params=kwargs) + + async def admin_teams_owners_list( + self, + *, + team_id: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List all of the admins on a given workspace. + https://api.slack.com/methods/admin.teams.owners.list + """ + kwargs.update({"team_id": team_id, "cursor": cursor, "limit": limit}) + return await self.api_call("admin.teams.owners.list", http_verb="GET", params=kwargs) + + async def admin_teams_settings_info( + self, + *, + team_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Fetch information about settings in a workspace + https://api.slack.com/methods/admin.teams.settings.info + """ + kwargs.update({"team_id": team_id}) + return await self.api_call("admin.teams.settings.info", params=kwargs) + + async def admin_teams_settings_setDescription( + self, + *, + team_id: str, + description: str, + **kwargs, + ) -> AsyncSlackResponse: + """Set the description of a given workspace. + https://api.slack.com/methods/admin.teams.settings.setDescription + """ + kwargs.update({"team_id": team_id, "description": description}) + return await self.api_call("admin.teams.settings.setDescription", params=kwargs) + + async def admin_teams_settings_setDiscoverability( + self, + *, + team_id: str, + discoverability: str, + **kwargs, + ) -> AsyncSlackResponse: + """Sets the icon of a workspace. + https://api.slack.com/methods/admin.teams.settings.setDiscoverability + """ + kwargs.update({"team_id": team_id, "discoverability": discoverability}) + return await self.api_call("admin.teams.settings.setDiscoverability", params=kwargs) + + async def admin_teams_settings_setIcon( + self, + *, + team_id: str, + image_url: str, + **kwargs, + ) -> AsyncSlackResponse: + """Sets the icon of a workspace. + https://api.slack.com/methods/admin.teams.settings.setIcon + """ + kwargs.update({"team_id": team_id, "image_url": image_url}) + return await self.api_call("admin.teams.settings.setIcon", http_verb="GET", params=kwargs) + + async def admin_teams_settings_setName( + self, + *, + team_id: str, + name: str, + **kwargs, + ) -> AsyncSlackResponse: + """Sets the icon of a workspace. + https://api.slack.com/methods/admin.teams.settings.setName + """ + kwargs.update({"team_id": team_id, "name": name}) + return await self.api_call("admin.teams.settings.setName", params=kwargs) + + async def admin_usergroups_addChannels( + self, + *, + channel_ids: Union[str, Sequence[str]], + usergroup_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Add one or more default channels to an IDP group. + https://api.slack.com/methods/admin.usergroups.addChannels + """ + kwargs.update({"team_id": team_id, "usergroup_id": usergroup_id}) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return await self.api_call("admin.usergroups.addChannels", params=kwargs) + + async def admin_usergroups_addTeams( + self, + *, + usergroup_id: str, + team_ids: Union[str, Sequence[str]], + auto_provision: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Associate one or more default workspaces with an organization-wide IDP group. + https://api.slack.com/methods/admin.usergroups.addTeams + """ + kwargs.update({"usergroup_id": usergroup_id, "auto_provision": auto_provision}) + if isinstance(team_ids, (list, Tuple)): + kwargs.update({"team_ids": ",".join(team_ids)}) + else: + kwargs.update({"team_ids": team_ids}) + return await self.api_call("admin.usergroups.addTeams", params=kwargs) + + async def admin_usergroups_listChannels( + self, + *, + usergroup_id: str, + include_num_members: Optional[bool] = None, + team_id: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Add one or more default channels to an IDP group. + https://api.slack.com/methods/admin.usergroups.listChannels + """ + kwargs.update( + { + "usergroup_id": usergroup_id, + "include_num_members": include_num_members, + "team_id": team_id, + } + ) + return await self.api_call("admin.usergroups.listChannels", params=kwargs) + + async def admin_usergroups_removeChannels( + self, + *, + usergroup_id: str, + channel_ids: Union[str, Sequence[str]], + **kwargs, + ) -> AsyncSlackResponse: + """Add one or more default channels to an IDP group. + https://api.slack.com/methods/admin.usergroups.removeChannels + """ + kwargs.update({"usergroup_id": usergroup_id}) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return await self.api_call("admin.usergroups.removeChannels", params=kwargs) + + async def admin_users_assign( + self, + *, + team_id: str, + user_id: str, + channel_ids: Optional[Union[str, Sequence[str]]] = None, + is_restricted: Optional[bool] = None, + is_ultra_restricted: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Add an Enterprise user to a workspace. + https://api.slack.com/methods/admin.users.assign + """ + kwargs.update( + { + "team_id": team_id, + "user_id": user_id, + "is_restricted": is_restricted, + "is_ultra_restricted": is_ultra_restricted, + } + ) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return await self.api_call("admin.users.assign", params=kwargs) + + async def admin_users_invite( + self, + *, + team_id: str, + email: str, + channel_ids: Union[str, Sequence[str]], + custom_message: Optional[str] = None, + email_password_policy_enabled: Optional[bool] = None, + guest_expiration_ts: Optional[Union[str, float]] = None, + is_restricted: Optional[bool] = None, + is_ultra_restricted: Optional[bool] = None, + real_name: Optional[str] = None, + resend: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Invite a user to a workspace. + https://api.slack.com/methods/admin.users.invite + """ + kwargs.update( + { + "team_id": team_id, + "email": email, + "custom_message": custom_message, + "email_password_policy_enabled": email_password_policy_enabled, + "guest_expiration_ts": str(guest_expiration_ts) if guest_expiration_ts is not None else None, + "is_restricted": is_restricted, + "is_ultra_restricted": is_ultra_restricted, + "real_name": real_name, + "resend": resend, + } + ) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return await self.api_call("admin.users.invite", params=kwargs) + + async def admin_users_list( + self, + *, + team_id: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List users on a workspace + https://api.slack.com/methods/admin.users.list + """ + kwargs.update( + { + "team_id": team_id, + "cursor": cursor, + "limit": limit, + } + ) + return await self.api_call("admin.users.list", params=kwargs) + + async def admin_users_remove( + self, + *, + team_id: str, + user_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Remove a user from a workspace. + https://api.slack.com/methods/admin.users.remove + """ + kwargs.update({"team_id": team_id, "user_id": user_id}) + return await self.api_call("admin.users.remove", params=kwargs) + + async def admin_users_setAdmin( + self, + *, + team_id: str, + user_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Set an existing guest, regular user, or owner to be an admin user. + https://api.slack.com/methods/admin.users.setAdmin + """ + kwargs.update({"team_id": team_id, "user_id": user_id}) + return await self.api_call("admin.users.setAdmin", params=kwargs) + + async def admin_users_setExpiration( + self, + *, + expiration_ts: int, + user_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Set an expiration for a guest user. + https://api.slack.com/methods/admin.users.setExpiration + """ + kwargs.update({"expiration_ts": expiration_ts, "team_id": team_id, "user_id": user_id}) + return await self.api_call("admin.users.setExpiration", params=kwargs) + + async def admin_users_setOwner( + self, + *, + team_id: str, + user_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Set an existing guest, regular user, or admin user to be a workspace owner. + https://api.slack.com/methods/admin.users.setOwner + """ + kwargs.update({"team_id": team_id, "user_id": user_id}) + return await self.api_call("admin.users.setOwner", params=kwargs) + + async def admin_users_setRegular( + self, + *, + team_id: str, + user_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Set an existing guest user, admin user, or owner to be a regular user. + https://api.slack.com/methods/admin.users.setRegular + """ + kwargs.update({"team_id": team_id, "user_id": user_id}) + return await self.api_call("admin.users.setRegular", params=kwargs) + + async def api_test( + self, + *, + error: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Checks API calling code. + https://api.slack.com/methods/api.test + """ + kwargs.update({"error": error}) + return await self.api_call("api.test", params=kwargs) + + async def apps_connections_open( + self, + *, + app_token: str, + **kwargs, + ) -> AsyncSlackResponse: + """Generate a temporary Socket Mode WebSocket URL that your app can connect to + in order to receive events and interactive payloads + https://api.slack.com/methods/apps.connections.open + """ + kwargs.update({"token": app_token}) + return await self.api_call("apps.connections.open", http_verb="POST", params=kwargs) + + async def apps_event_authorizations_list( + self, + *, + event_context: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Get a list of authorizations for the given event context. + Each authorization represents an app installation that the event is visible to. + https://api.slack.com/methods/apps.event.authorizations.list + """ + kwargs.update({"event_context": event_context, "cursor": cursor, "limit": limit}) + return await self.api_call("apps.event.authorizations.list", params=kwargs) + + async def apps_uninstall( + self, + *, + client_id: str, + client_secret: str, + **kwargs, + ) -> AsyncSlackResponse: + """Uninstalls your app from a workspace. + https://api.slack.com/methods/apps.uninstall + """ + kwargs.update({"client_id": client_id, "client_secret": client_secret}) + return await self.api_call("apps.uninstall", params=kwargs) + + async def auth_revoke( + self, + *, + test: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Revokes a token. + https://api.slack.com/methods/auth.revoke + """ + kwargs.update({"test": test}) + return await self.api_call("auth.revoke", http_verb="GET", params=kwargs) + + async def auth_test( + self, + **kwargs, + ) -> AsyncSlackResponse: + """Checks authentication & identity. + https://api.slack.com/methods/auth.test + """ + return await self.api_call("auth.test", params=kwargs) + + async def auth_teams_list( + self, + cursor: Optional[str] = None, + limit: Optional[int] = None, + include_icon: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List the workspaces a token can access. + https://api.slack.com/methods/auth.teams.list + """ + kwargs.update({"cursor": cursor, "limit": limit, "include_icon": include_icon}) + return await self.api_call("auth.teams.list", params=kwargs) + + async def bookmarks_add( + self, + *, + channel_id: str, + title: str, + type: str, + emoji: Optional[str] = None, + entity_id: Optional[str] = None, + link: Optional[str] = None, # include when type is 'link' + parent_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Add bookmark to a channel. + https://api.slack.com/methods/bookmarks.add + """ + kwargs.update( + { + "channel_id": channel_id, + "title": title, + "type": type, + "emoji": emoji, + "entity_id": entity_id, + "link": link, + "parent_id": parent_id, + } + ) + return await self.api_call("bookmarks.add", http_verb="POST", params=kwargs) + + async def bookmarks_edit( + self, + *, + bookmark_id: str, + channel_id: str, + emoji: Optional[str] = None, + link: Optional[str] = None, + title: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Edit bookmark. + https://api.slack.com/methods/bookmarks.edit + """ + kwargs.update( + { + "bookmark_id": bookmark_id, + "channel_id": channel_id, + "emoji": emoji, + "link": link, + "title": title, + } + ) + return await self.api_call("bookmarks.edit", http_verb="POST", params=kwargs) + + async def bookmarks_list( + self, + *, + channel_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """List bookmark for the channel. + https://api.slack.com/methods/bookmarks.list + """ + kwargs.update({"channel_id": channel_id}) + return await self.api_call("bookmarks.list", http_verb="POST", params=kwargs) + + async def bookmarks_remove( + self, + *, + bookmark_id: str, + channel_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Remove bookmark from the channel. + https://api.slack.com/methods/bookmarks.remove + """ + kwargs.update({"bookmark_id": bookmark_id, "channel_id": channel_id}) + return await self.api_call("bookmarks.remove", http_verb="POST", params=kwargs) + + async def bots_info( + self, + *, + bot: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Gets information about a bot user. + https://api.slack.com/methods/bots.info + """ + kwargs.update({"bot": bot, "team_id": team_id}) + return await self.api_call("bots.info", http_verb="GET", params=kwargs) + + async def calls_add( + self, + *, + external_unique_id: str, + join_url: str, + created_by: Optional[str] = None, + date_start: Optional[int] = None, + desktop_app_join_url: Optional[str] = None, + external_display_id: Optional[str] = None, + title: Optional[str] = None, + users: Optional[Union[str, Sequence[Dict[str, str]]]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Registers a new Call. + https://api.slack.com/methods/calls.add + """ + kwargs.update( + { + "external_unique_id": external_unique_id, + "join_url": join_url, + "created_by": created_by, + "date_start": date_start, + "desktop_app_join_url": desktop_app_join_url, + "external_display_id": external_display_id, + "title": title, + } + ) + _update_call_participants( # skipcq: PTC-W0039 + kwargs, + users if users is not None else kwargs.get("users"), # skipcq: PTC-W0039 + ) # skipcq: PTC-W0039 + return await self.api_call("calls.add", http_verb="POST", params=kwargs) + + async def calls_end( + self, + *, + id: str, + duration: Optional[int] = None, + **kwargs, + ) -> AsyncSlackResponse: # skipcq: PYL-W0622 + """Ends a Call. + https://api.slack.com/methods/calls.end + """ + kwargs.update({"id": id, "duration": duration}) + return await self.api_call("calls.end", http_verb="POST", params=kwargs) + + async def calls_info( + self, + *, + id: str, + **kwargs, + ) -> AsyncSlackResponse: # skipcq: PYL-W0622 + """Returns information about a Call. + https://api.slack.com/methods/calls.info + """ + kwargs.update({"id": id}) + return await self.api_call("calls.info", http_verb="POST", params=kwargs) + + async def calls_participants_add( + self, + *, + id: str, # skipcq: PYL-W0622 + users: Union[str, Sequence[Dict[str, str]]], + **kwargs, + ) -> AsyncSlackResponse: + """Registers new participants added to a Call. + https://api.slack.com/methods/calls.participants.add + """ + kwargs.update({"id": id}) + _update_call_participants(kwargs, users) + return await self.api_call("calls.participants.add", http_verb="POST", params=kwargs) + + async def calls_participants_remove( + self, + *, + id: str, # skipcq: PYL-W0622 + users: Union[str, Sequence[Dict[str, str]]], + **kwargs, + ) -> AsyncSlackResponse: + """Registers participants removed from a Call. + https://api.slack.com/methods/calls.participants.remove + """ + kwargs.update({"id": id}) + _update_call_participants(kwargs, users) + return await self.api_call("calls.participants.remove", http_verb="POST", params=kwargs) + + async def calls_update( + self, + *, + id: str, + desktop_app_join_url: Optional[str] = None, + join_url: Optional[str] = None, + title: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: # skipcq: PYL-W0622 + """Updates information about a Call. + https://api.slack.com/methods/calls.update + """ + kwargs.update( + { + "id": id, + "desktop_app_join_url": desktop_app_join_url, + "join_url": join_url, + "title": title, + } + ) + return await self.api_call("calls.update", http_verb="POST", params=kwargs) + + # -------------------------- + # Deprecated: channels.* + # You can use conversations.* APIs instead. + # https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api + # -------------------------- + + async def channels_archive( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Archives a channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("channels.archive", json=kwargs) + + async def channels_create( + self, + *, + name: str, + **kwargs, + ) -> AsyncSlackResponse: + """Creates a channel.""" + kwargs.update({"name": name}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("channels.create", json=kwargs) + + async def channels_history( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Fetches history of messages and events from a channel.""" + kwargs.update({"channel": channel}) + return await self.api_call("channels.history", http_verb="GET", params=kwargs) + + async def channels_info( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Gets information about a channel.""" + kwargs.update({"channel": channel}) + return await self.api_call("channels.info", http_verb="GET", params=kwargs) + + async def channels_invite( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> AsyncSlackResponse: + """Invites a user to a channel.""" + kwargs.update({"channel": channel, "user": user}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("channels.invite", json=kwargs) + + async def channels_join( + self, + *, + name: str, + **kwargs, + ) -> AsyncSlackResponse: + """Joins a channel, creating it if needed.""" + kwargs.update({"name": name}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("channels.join", json=kwargs) + + async def channels_kick( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> AsyncSlackResponse: + """Removes a user from a channel.""" + kwargs.update({"channel": channel, "user": user}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("channels.kick", json=kwargs) + + async def channels_leave( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Leaves a channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("channels.leave", json=kwargs) + + async def channels_list( + self, + **kwargs, + ) -> AsyncSlackResponse: + """Lists all channels in a Slack team.""" + return await self.api_call("channels.list", http_verb="GET", params=kwargs) + + async def channels_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> AsyncSlackResponse: + """Sets the read cursor in a channel.""" + kwargs.update({"channel": channel, "ts": ts}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("channels.mark", json=kwargs) + + async def channels_rename( + self, + *, + channel: str, + name: str, + **kwargs, + ) -> AsyncSlackResponse: + """Renames a channel.""" + kwargs.update({"channel": channel, "name": name}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("channels.rename", json=kwargs) + + async def channels_replies( + self, + *, + channel: str, + thread_ts: str, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieve a thread of messages posted to a channel""" + kwargs.update({"channel": channel, "thread_ts": thread_ts}) + return await self.api_call("channels.replies", http_verb="GET", params=kwargs) + + async def channels_setPurpose( + self, + *, + channel: str, + purpose: str, + **kwargs, + ) -> AsyncSlackResponse: + """Sets the purpose for a channel.""" + kwargs.update({"channel": channel, "purpose": purpose}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("channels.setPurpose", json=kwargs) + + async def channels_setTopic( + self, + *, + channel: str, + topic: str, + **kwargs, + ) -> AsyncSlackResponse: + """Sets the topic for a channel.""" + kwargs.update({"channel": channel, "topic": topic}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("channels.setTopic", json=kwargs) + + async def channels_unarchive( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Unarchives a channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("channels.unarchive", json=kwargs) + + # -------------------------- + + async def chat_delete( + self, + *, + channel: str, + ts: str, + as_user: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Deletes a message. + https://api.slack.com/methods/chat.delete + """ + kwargs.update({"channel": channel, "ts": ts, "as_user": as_user}) + return await self.api_call("chat.delete", params=kwargs) + + async def chat_deleteScheduledMessage( + self, + *, + channel: str, + scheduled_message_id: str, + as_user: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Deletes a scheduled message. + https://api.slack.com/methods/chat.deleteScheduledMessage + """ + kwargs.update( + { + "channel": channel, + "scheduled_message_id": scheduled_message_id, + "as_user": as_user, + } + ) + return await self.api_call("chat.deleteScheduledMessage", params=kwargs) + + async def chat_getPermalink( + self, + *, + channel: str, + message_ts: str, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieve a permalink URL for a specific extant message + https://api.slack.com/methods/chat.getPermalink + """ + kwargs.update({"channel": channel, "message_ts": message_ts}) + return await self.api_call("chat.getPermalink", http_verb="GET", params=kwargs) + + async def chat_meMessage( + self, + *, + channel: str, + text: str, + **kwargs, + ) -> AsyncSlackResponse: + """Share a me message into a channel. + https://api.slack.com/methods/chat.meMessage + """ + kwargs.update({"channel": channel, "text": text}) + return await self.api_call("chat.meMessage", params=kwargs) + + async def chat_postEphemeral( + self, + *, + channel: str, + user: str, + text: Optional[str] = None, + as_user: Optional[bool] = None, + attachments: Optional[Sequence[Union[Dict, Attachment]]] = None, + blocks: Optional[Sequence[Union[Dict, Block]]] = None, + thread_ts: Optional[str] = None, + icon_emoji: Optional[str] = None, + icon_url: Optional[str] = None, + link_names: Optional[bool] = None, + username: Optional[str] = None, + parse: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Sends an ephemeral message to a user in a channel. + https://api.slack.com/methods/chat.postEphemeral + """ + kwargs.update( + { + "channel": channel, + "user": user, + "text": text, + "as_user": as_user, + "attachments": attachments, + "blocks": blocks, + "thread_ts": thread_ts, + "icon_emoji": icon_emoji, + "icon_url": icon_url, + "link_names": link_names, + "username": username, + "parse": parse, + } + ) + _parse_web_class_objects(kwargs) + kwargs = _remove_none_values(kwargs) + _warn_if_text_or_attachment_fallback_is_missing("chat.postEphemeral", kwargs) + # NOTE: intentionally using json over params for the API methods using blocks/attachments + return await self.api_call("chat.postEphemeral", json=kwargs) + + async def chat_postMessage( + self, + *, + channel: str, + text: Optional[str] = None, + as_user: Optional[bool] = None, + attachments: Optional[Sequence[Union[Dict, Attachment]]] = None, + blocks: Optional[Sequence[Union[Dict, Block]]] = None, + thread_ts: Optional[str] = None, + reply_broadcast: Optional[bool] = None, + unfurl_links: Optional[bool] = None, + unfurl_media: Optional[bool] = None, + container_id: Optional[str] = None, + file_annotation: Optional[str] = None, + icon_emoji: Optional[str] = None, + icon_url: Optional[str] = None, + mrkdwn: Optional[bool] = None, + link_names: Optional[bool] = None, + username: Optional[str] = None, + parse: Optional[str] = None, # none, full + metadata: Optional[Union[Dict, Metadata]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Sends a message to a channel. + https://api.slack.com/methods/chat.postMessage + """ + kwargs.update( + { + "channel": channel, + "text": text, + "as_user": as_user, + "attachments": attachments, + "blocks": blocks, + "thread_ts": thread_ts, + "reply_broadcast": reply_broadcast, + "unfurl_links": unfurl_links, + "unfurl_media": unfurl_media, + "container_id": container_id, + "file_annotation": file_annotation, + "icon_emoji": icon_emoji, + "icon_url": icon_url, + "mrkdwn": mrkdwn, + "link_names": link_names, + "username": username, + "parse": parse, + "metadata": metadata, + } + ) + _parse_web_class_objects(kwargs) + kwargs = _remove_none_values(kwargs) + _warn_if_text_or_attachment_fallback_is_missing("chat.postMessage", kwargs) + # NOTE: intentionally using json over params for the API methods using blocks/attachments + return await self.api_call("chat.postMessage", json=kwargs) + + async def chat_scheduleMessage( + self, + *, + channel: str, + post_at: Union[str, int], + text: str, + as_user: Optional[bool] = None, + attachments: Optional[Sequence[Union[Dict, Attachment]]] = None, + blocks: Optional[Sequence[Union[Dict, Block]]] = None, + thread_ts: Optional[str] = None, + parse: Optional[str] = None, + reply_broadcast: Optional[bool] = None, + unfurl_links: Optional[bool] = None, + unfurl_media: Optional[bool] = None, + link_names: Optional[bool] = None, + metadata: Optional[Union[Dict, Metadata]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Schedules a message. + https://api.slack.com/methods/chat.scheduleMessage + """ + kwargs.update( + { + "channel": channel, + "post_at": post_at, + "text": text, + "as_user": as_user, + "attachments": attachments, + "blocks": blocks, + "thread_ts": thread_ts, + "reply_broadcast": reply_broadcast, + "parse": parse, + "unfurl_links": unfurl_links, + "unfurl_media": unfurl_media, + "link_names": link_names, + "metadata": metadata, + } + ) + _parse_web_class_objects(kwargs) + kwargs = _remove_none_values(kwargs) + _warn_if_text_or_attachment_fallback_is_missing("chat.scheduleMessage", kwargs) + # NOTE: intentionally using json over params for the API methods using blocks/attachments + return await self.api_call("chat.scheduleMessage", json=kwargs) + + async def chat_unfurl( + self, + *, + channel: str, + ts: str, + unfurls: Dict[str, Dict], + user_auth_blocks: Optional[Sequence[Union[Dict, Block]]] = None, + user_auth_message: Optional[str] = None, + user_auth_required: Optional[bool] = None, + user_auth_url: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Provide custom unfurl behavior for user-posted URLs. + https://api.slack.com/methods/chat.unfurl + """ + kwargs.update( + { + "channel": channel, + "ts": ts, + "unfurls": unfurls, + "user_auth_blocks": user_auth_blocks, + "user_auth_message": user_auth_message, + "user_auth_required": user_auth_required, + "user_auth_url": user_auth_url, + } + ) + kwargs = _remove_none_values(kwargs) + # NOTE: intentionally using json over params for API methods using blocks/attachments + return await self.api_call("chat.unfurl", json=kwargs) + + async def chat_update( + self, + *, + channel: str, + ts: str, + text: Optional[str] = None, + attachments: Optional[Sequence[Union[Dict, Attachment]]] = None, + blocks: Optional[Sequence[Union[Dict, Block]]] = None, + as_user: Optional[bool] = None, + file_ids: Optional[Union[str, Sequence[str]]] = None, + link_names: Optional[bool] = None, + parse: Optional[str] = None, # none, full + reply_broadcast: Optional[bool] = None, + metadata: Optional[Union[Dict, Metadata]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Updates a message in a channel. + https://api.slack.com/methods/chat.update + """ + kwargs.update( + { + "channel": channel, + "ts": ts, + "text": text, + "attachments": attachments, + "blocks": blocks, + "as_user": as_user, + "link_names": link_names, + "parse": parse, + "reply_broadcast": reply_broadcast, + "metadata": metadata, + } + ) + if isinstance(file_ids, (list, Tuple)): + kwargs.update({"file_ids": ",".join(file_ids)}) + else: + kwargs.update({"file_ids": file_ids}) + _parse_web_class_objects(kwargs) + kwargs = _remove_none_values(kwargs) + _warn_if_text_or_attachment_fallback_is_missing("chat.update", kwargs) + # NOTE: intentionally using json over params for API methods using blocks/attachments + return await self.api_call("chat.update", json=kwargs) + + async def chat_scheduledMessages_list( + self, + *, + channel: Optional[str] = None, + cursor: Optional[str] = None, + latest: Optional[str] = None, + limit: Optional[int] = None, + oldest: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Lists all scheduled messages. + https://api.slack.com/methods/chat.scheduledMessages.list + """ + kwargs.update( + { + "channel": channel, + "cursor": cursor, + "latest": latest, + "limit": limit, + "oldest": oldest, + "team_id": team_id, + } + ) + return await self.api_call("chat.scheduledMessages.list", params=kwargs) + + async def conversations_acceptSharedInvite( + self, + *, + channel_name: str, + channel_id: Optional[str] = None, + invite_id: Optional[str] = None, + free_trial_accepted: Optional[bool] = None, + is_private: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Accepts an invitation to a Slack Connect channel. + https://api.slack.com/methods/conversations.acceptSharedInvite + """ + if channel_id is None and invite_id is None: + raise e.SlackRequestError("Either channel_id or invite_id must be provided.") + kwargs.update( + { + "channel_name": channel_name, + "channel_id": channel_id, + "invite_id": invite_id, + "free_trial_accepted": free_trial_accepted, + "is_private": is_private, + "team_id": team_id, + } + ) + return await self.api_call("conversations.acceptSharedInvite", http_verb="POST", params=kwargs) + + async def conversations_approveSharedInvite( + self, + *, + invite_id: str, + target_team: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Approves an invitation to a Slack Connect channel. + https://api.slack.com/methods/conversations.approveSharedInvite + """ + kwargs.update({"invite_id": invite_id, "target_team": target_team}) + return await self.api_call("conversations.approveSharedInvite", http_verb="POST", params=kwargs) + + async def conversations_archive( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Archives a conversation. + https://api.slack.com/methods/conversations.archive + """ + kwargs.update({"channel": channel}) + return await self.api_call("conversations.archive", params=kwargs) + + async def conversations_close( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Closes a direct message or multi-person direct message. + https://api.slack.com/methods/conversations.close + """ + kwargs.update({"channel": channel}) + return await self.api_call("conversations.close", params=kwargs) + + async def conversations_create( + self, + *, + name: str, + is_private: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Initiates a public or private channel-based conversation + https://api.slack.com/methods/conversations.create + """ + kwargs.update({"name": name, "is_private": is_private, "team_id": team_id}) + return await self.api_call("conversations.create", params=kwargs) + + async def conversations_declineSharedInvite( + self, + *, + invite_id: str, + target_team: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Declines a Slack Connect channel invite. + https://api.slack.com/methods/conversations.declineSharedInvite + """ + kwargs.update({"invite_id": invite_id, "target_team": target_team}) + return await self.api_call("conversations.declineSharedInvite", http_verb="GET", params=kwargs) + + async def conversations_history( + self, + *, + channel: str, + cursor: Optional[str] = None, + inclusive: Optional[bool] = None, + include_all_metadata: Optional[bool] = None, + latest: Optional[str] = None, + limit: Optional[int] = None, + oldest: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Fetches a conversation's history of messages and events. + https://api.slack.com/methods/conversations.history + """ + kwargs.update( + { + "channel": channel, + "cursor": cursor, + "inclusive": inclusive, + "include_all_metadata": include_all_metadata, + "limit": limit, + "latest": latest, + "oldest": oldest, + } + ) + return await self.api_call("conversations.history", http_verb="GET", params=kwargs) + + async def conversations_info( + self, + *, + channel: str, + include_locale: Optional[bool] = None, + include_num_members: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieve information about a conversation. + https://api.slack.com/methods/conversations.info + """ + kwargs.update( + { + "channel": channel, + "include_locale": include_locale, + "include_num_members": include_num_members, + } + ) + return await self.api_call("conversations.info", http_verb="GET", params=kwargs) + + async def conversations_invite( + self, + *, + channel: str, + users: Union[str, Sequence[str]], + **kwargs, + ) -> AsyncSlackResponse: + """Invites users to a channel. + https://api.slack.com/methods/conversations.invite + """ + kwargs.update({"channel": channel}) + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + return await self.api_call("conversations.invite", params=kwargs) + + async def conversations_inviteShared( + self, + *, + channel: str, + emails: Optional[Union[str, Sequence[str]]] = None, + user_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Sends an invitation to a Slack Connect channel. + https://api.slack.com/methods/conversations.inviteShared + """ + if emails is None and user_ids is None: + raise e.SlackRequestError("Either emails or user ids must be provided.") + kwargs.update({"channel": channel}) + if isinstance(emails, (list, Tuple)): + kwargs.update({"emails": ",".join(emails)}) + else: + kwargs.update({"emails": emails}) + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + return await self.api_call("conversations.inviteShared", http_verb="GET", params=kwargs) + + async def conversations_join( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Joins an existing conversation. + https://api.slack.com/methods/conversations.join + """ + kwargs.update({"channel": channel}) + return await self.api_call("conversations.join", params=kwargs) + + async def conversations_kick( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> AsyncSlackResponse: + """Removes a user from a conversation. + https://api.slack.com/methods/conversations.kick + """ + kwargs.update({"channel": channel, "user": user}) + return await self.api_call("conversations.kick", params=kwargs) + + async def conversations_leave( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Leaves a conversation. + https://api.slack.com/methods/conversations.leave + """ + kwargs.update({"channel": channel}) + return await self.api_call("conversations.leave", params=kwargs) + + async def conversations_list( + self, + *, + cursor: Optional[str] = None, + exclude_archived: Optional[bool] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + types: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Lists all channels in a Slack team. + https://api.slack.com/methods/conversations.list + """ + kwargs.update( + { + "cursor": cursor, + "exclude_archived": exclude_archived, + "limit": limit, + "team_id": team_id, + } + ) + if isinstance(types, (list, Tuple)): + kwargs.update({"types": ",".join(types)}) + else: + kwargs.update({"types": types}) + return await self.api_call("conversations.list", http_verb="GET", params=kwargs) + + async def conversations_listConnectInvites( + self, + *, + count: Optional[int] = None, + cursor: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List shared channel invites that have been generated + or received but have not yet been approved by all parties. + https://api.slack.com/methods/conversations.listConnectInvites + """ + kwargs.update({"count": count, "cursor": cursor, "team_id": team_id}) + return await self.api_call("conversations.listConnectInvites", params=kwargs) + + async def conversations_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> AsyncSlackResponse: + """Sets the read cursor in a channel. + https://api.slack.com/methods/conversations.mark + """ + kwargs.update({"channel": channel, "ts": ts}) + return await self.api_call("conversations.mark", params=kwargs) + + async def conversations_members( + self, + *, + channel: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieve members of a conversation. + https://api.slack.com/methods/conversations.members + """ + kwargs.update({"channel": channel, "cursor": cursor, "limit": limit}) + return await self.api_call("conversations.members", http_verb="GET", params=kwargs) + + async def conversations_open( + self, + *, + channel: Optional[str] = None, + return_im: Optional[bool] = None, + users: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Opens or resumes a direct message or multi-person direct message. + https://api.slack.com/methods/conversations.open + """ + if channel is None and users is None: + raise e.SlackRequestError("Either channel or users must be provided.") + kwargs.update({"channel": channel, "return_im": return_im}) + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + return await self.api_call("conversations.open", params=kwargs) + + async def conversations_rename( + self, + *, + channel: str, + name: str, + **kwargs, + ) -> AsyncSlackResponse: + """Renames a conversation. + https://api.slack.com/methods/conversations.rename + """ + kwargs.update({"channel": channel, "name": name}) + return await self.api_call("conversations.rename", params=kwargs) + + async def conversations_replies( + self, + *, + channel: str, + ts: str, + cursor: Optional[str] = None, + inclusive: Optional[bool] = None, + latest: Optional[str] = None, + limit: Optional[int] = None, + oldest: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieve a thread of messages posted to a conversation + https://api.slack.com/methods/conversations.replies + """ + kwargs.update( + { + "channel": channel, + "ts": ts, + "cursor": cursor, + "inclusive": inclusive, + "limit": limit, + "latest": latest, + "oldest": oldest, + } + ) + return await self.api_call("conversations.replies", http_verb="GET", params=kwargs) + + async def conversations_setPurpose( + self, + *, + channel: str, + purpose: str, + **kwargs, + ) -> AsyncSlackResponse: + """Sets the purpose for a conversation. + https://api.slack.com/methods/conversations.setPurpose + """ + kwargs.update({"channel": channel, "purpose": purpose}) + return await self.api_call("conversations.setPurpose", params=kwargs) + + async def conversations_setTopic( + self, + *, + channel: str, + topic: str, + **kwargs, + ) -> AsyncSlackResponse: + """Sets the topic for a conversation. + https://api.slack.com/methods/conversations.setTopic + """ + kwargs.update({"channel": channel, "topic": topic}) + return await self.api_call("conversations.setTopic", params=kwargs) + + async def conversations_unarchive( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Reverses conversation archival. + https://api.slack.com/methods/conversations.unarchive + """ + kwargs.update({"channel": channel}) + return await self.api_call("conversations.unarchive", params=kwargs) + + async def dialog_open( + self, + *, + dialog: Dict[str, Any], + trigger_id: str, + **kwargs, + ) -> AsyncSlackResponse: + """Open a dialog with a user. + https://api.slack.com/methods/dialog.open + """ + kwargs.update({"dialog": dialog, "trigger_id": trigger_id}) + kwargs = _remove_none_values(kwargs) + # NOTE: As the dialog can be a dict, this API call works only with json format. + return await self.api_call("dialog.open", json=kwargs) + + async def dnd_endDnd( + self, + **kwargs, + ) -> AsyncSlackResponse: + """Ends the current user's Do Not Disturb session immediately. + https://api.slack.com/methods/dnd.endDnd + """ + return await self.api_call("dnd.endDnd", params=kwargs) + + async def dnd_endSnooze( + self, + **kwargs, + ) -> AsyncSlackResponse: + """Ends the current user's snooze mode immediately. + https://api.slack.com/methods/dnd.endSnooze + """ + return await self.api_call("dnd.endSnooze", params=kwargs) + + async def dnd_info( + self, + *, + team_id: Optional[str] = None, + user: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieves a user's current Do Not Disturb status. + https://api.slack.com/methods/dnd.info + """ + kwargs.update({"team_id": team_id, "user": user}) + return await self.api_call("dnd.info", http_verb="GET", params=kwargs) + + async def dnd_setSnooze( + self, + *, + num_minutes: Union[int, str], + **kwargs, + ) -> AsyncSlackResponse: + """Turns on Do Not Disturb mode for the current user, or changes its duration. + https://api.slack.com/methods/dnd.setSnooze + """ + kwargs.update({"num_minutes": num_minutes}) + return await self.api_call("dnd.setSnooze", http_verb="GET", params=kwargs) + + async def dnd_teamInfo( + self, + users: Union[str, Sequence[str]], + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieves the Do Not Disturb status for users on a team. + https://api.slack.com/methods/dnd.teamInfo + """ + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + kwargs.update({"team_id": team_id}) + return await self.api_call("dnd.teamInfo", http_verb="GET", params=kwargs) + + async def emoji_list( + self, + **kwargs, + ) -> AsyncSlackResponse: + """Lists custom emoji for a team. + https://api.slack.com/methods/emoji.list + """ + return await self.api_call("emoji.list", http_verb="GET", params=kwargs) + + async def files_comments_delete( + self, + *, + file: str, + id: str, + **kwargs, # skipcq: PYL-W0622 + ) -> AsyncSlackResponse: + """Deletes an existing comment on a file. + https://api.slack.com/methods/files.comments.delete + """ + kwargs.update({"file": file, "id": id}) + return await self.api_call("files.comments.delete", params=kwargs) + + async def files_delete( + self, + *, + file: str, + **kwargs, + ) -> AsyncSlackResponse: + """Deletes a file. + https://api.slack.com/methods/files.delete + """ + kwargs.update({"file": file}) + return await self.api_call("files.delete", params=kwargs) + + async def files_info( + self, + *, + file: str, + count: Optional[int] = None, + cursor: Optional[str] = None, + limit: Optional[int] = None, + page: Optional[int] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Gets information about a team file. + https://api.slack.com/methods/files.info + """ + kwargs.update( + { + "file": file, + "count": count, + "cursor": cursor, + "limit": limit, + "page": page, + } + ) + return await self.api_call("files.info", http_verb="GET", params=kwargs) + + async def files_list( + self, + *, + channel: Optional[str] = None, + count: Optional[int] = None, + page: Optional[int] = None, + show_files_hidden_by_limit: Optional[bool] = None, + team_id: Optional[str] = None, + ts_from: Optional[str] = None, + ts_to: Optional[str] = None, + types: Optional[Union[str, Sequence[str]]] = None, + user: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Lists & filters team files. + https://api.slack.com/methods/files.list + """ + kwargs.update( + { + "channel": channel, + "count": count, + "page": page, + "show_files_hidden_by_limit": show_files_hidden_by_limit, + "team_id": team_id, + "ts_from": ts_from, + "ts_to": ts_to, + "user": user, + } + ) + if isinstance(types, (list, Tuple)): + kwargs.update({"types": ",".join(types)}) + else: + kwargs.update({"types": types}) + return await self.api_call("files.list", http_verb="GET", params=kwargs) + + async def files_remote_info( + self, + *, + external_id: Optional[str] = None, + file: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieve information about a remote file added to Slack. + https://api.slack.com/methods/files.remote.info + """ + kwargs.update({"external_id": external_id, "file": file}) + return await self.api_call("files.remote.info", http_verb="GET", params=kwargs) + + async def files_remote_list( + self, + *, + channel: Optional[str] = None, + cursor: Optional[str] = None, + limit: Optional[int] = None, + ts_from: Optional[str] = None, + ts_to: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieve information about a remote file added to Slack. + https://api.slack.com/methods/files.remote.list + """ + kwargs.update( + { + "channel": channel, + "cursor": cursor, + "limit": limit, + "ts_from": ts_from, + "ts_to": ts_to, + } + ) + return await self.api_call("files.remote.list", http_verb="GET", params=kwargs) + + async def files_remote_add( + self, + *, + external_id: str, + external_url: str, + title: str, + filetype: Optional[str] = None, + indexable_file_contents: Optional[Union[str, bytes, IOBase]] = None, + preview_image: Optional[Union[str, bytes, IOBase]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Adds a file from a remote service. + https://api.slack.com/methods/files.remote.add + """ + kwargs.update( + { + "external_id": external_id, + "external_url": external_url, + "title": title, + "filetype": filetype, + } + ) + files = None + # preview_image (file): Preview of the document via multipart/form-data. + if preview_image is not None or indexable_file_contents is not None: + files = { + "preview_image": preview_image, + "indexable_file_contents": indexable_file_contents, + } + + return await self.api_call( + # Intentionally using "POST" method over "GET" here + "files.remote.add", + http_verb="POST", + data=kwargs, + files=files, + ) + + async def files_remote_update( + self, + *, + external_id: Optional[str] = None, + external_url: Optional[str] = None, + file: Optional[str] = None, + title: Optional[str] = None, + filetype: Optional[str] = None, + indexable_file_contents: Optional[str] = None, + preview_image: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Updates an existing remote file. + https://api.slack.com/methods/files.remote.update + """ + kwargs.update( + { + "external_id": external_id, + "external_url": external_url, + "file": file, + "title": title, + "filetype": filetype, + } + ) + files = None + # preview_image (file): Preview of the document via multipart/form-data. + if preview_image is not None or indexable_file_contents is not None: + files = { + "preview_image": preview_image, + "indexable_file_contents": indexable_file_contents, + } + + return await self.api_call( + # Intentionally using "POST" method over "GET" here + "files.remote.update", + http_verb="POST", + data=kwargs, + files=files, + ) + + async def files_remote_remove( + self, + *, + external_id: Optional[str] = None, + file: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Remove a remote file. + https://api.slack.com/methods/files.remote.remove + """ + kwargs.update({"external_id": external_id, "file": file}) + return await self.api_call("files.remote.remove", http_verb="POST", params=kwargs) + + async def files_remote_share( + self, + *, + channels: Union[str, Sequence[str]], + external_id: Optional[str] = None, + file: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Share a remote file into a channel. + https://api.slack.com/methods/files.remote.share + """ + if external_id is None and file is None: + raise e.SlackRequestError("Either external_id or file must be provided.") + if isinstance(channels, (list, Tuple)): + kwargs.update({"channels": ",".join(channels)}) + else: + kwargs.update({"channels": channels}) + kwargs.update({"external_id": external_id, "file": file}) + return await self.api_call("files.remote.share", http_verb="GET", params=kwargs) + + async def files_revokePublicURL( + self, + *, + file: str, + **kwargs, + ) -> AsyncSlackResponse: + """Revokes public/external sharing access for a file + https://api.slack.com/methods/files.revokePublicURL + """ + kwargs.update({"file": file}) + return await self.api_call("files.revokePublicURL", params=kwargs) + + async def files_sharedPublicURL( + self, + *, + file: str, + **kwargs, + ) -> AsyncSlackResponse: + """Enables a file for public/external sharing. + https://api.slack.com/methods/files.sharedPublicURL + """ + kwargs.update({"file": file}) + return await self.api_call("files.sharedPublicURL", params=kwargs) + + async def files_upload( + self, + *, + file: Optional[Union[str, bytes, IOBase]] = None, + content: Optional[str] = None, + filename: Optional[str] = None, + filetype: Optional[str] = None, + initial_comment: Optional[str] = None, + thread_ts: Optional[str] = None, + title: Optional[str] = None, + channels: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Uploads or creates a file. + https://api.slack.com/methods/files.upload + """ + if file is None and content is None: + raise e.SlackRequestError("The file or content argument must be specified.") + if file is not None and content is not None: + raise e.SlackRequestError("You cannot specify both the file and the content argument.") + + if isinstance(channels, (list, Tuple)): + kwargs.update({"channels": ",".join(channels)}) + else: + kwargs.update({"channels": channels}) + kwargs.update( + { + "filename": filename, + "filetype": filetype, + "initial_comment": initial_comment, + "thread_ts": thread_ts, + "title": title, + } + ) + if file: + if kwargs.get("filename") is None and isinstance(file, str): + # use the local filename if filename is missing + if kwargs.get("filename") is None: + kwargs["filename"] = file.split(os.path.sep)[-1] + return await self.api_call("files.upload", files={"file": file}, data=kwargs) + else: + kwargs["content"] = content + return await self.api_call("files.upload", data=kwargs) + + # -------------------------- + # Deprecated: groups.* + # You can use conversations.* APIs instead. + # https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api + # -------------------------- + + async def groups_archive( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Archives a private channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("groups.archive", json=kwargs) + + async def groups_create( + self, + *, + name: str, + **kwargs, + ) -> AsyncSlackResponse: + """Creates a private channel.""" + kwargs.update({"name": name}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("groups.create", json=kwargs) + + async def groups_createChild( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Clones and archives a private channel.""" + kwargs.update({"channel": channel}) + return await self.api_call("groups.createChild", http_verb="GET", params=kwargs) + + async def groups_history( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Fetches history of messages and events from a private channel.""" + kwargs.update({"channel": channel}) + return await self.api_call("groups.history", http_verb="GET", params=kwargs) + + async def groups_info( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Gets information about a private channel.""" + kwargs.update({"channel": channel}) + return await self.api_call("groups.info", http_verb="GET", params=kwargs) + + async def groups_invite( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> AsyncSlackResponse: + """Invites a user to a private channel.""" + kwargs.update({"channel": channel, "user": user}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("groups.invite", json=kwargs) + + async def groups_kick( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> AsyncSlackResponse: + """Removes a user from a private channel.""" + kwargs.update({"channel": channel, "user": user}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("groups.kick", json=kwargs) + + async def groups_leave( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Leaves a private channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("groups.leave", json=kwargs) + + async def groups_list( + self, + **kwargs, + ) -> AsyncSlackResponse: + """Lists private channels that the calling user has access to.""" + return await self.api_call("groups.list", http_verb="GET", params=kwargs) + + async def groups_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> AsyncSlackResponse: + """Sets the read cursor in a private channel.""" + kwargs.update({"channel": channel, "ts": ts}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("groups.mark", json=kwargs) + + async def groups_open( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Opens a private channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("groups.open", json=kwargs) + + async def groups_rename( + self, + *, + channel: str, + name: str, + **kwargs, + ) -> AsyncSlackResponse: + """Renames a private channel.""" + kwargs.update({"channel": channel, "name": name}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("groups.rename", json=kwargs) + + async def groups_replies( + self, + *, + channel: str, + thread_ts: str, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieve a thread of messages posted to a private channel""" + kwargs.update({"channel": channel, "thread_ts": thread_ts}) + return await self.api_call("groups.replies", http_verb="GET", params=kwargs) + + async def groups_setPurpose( + self, + *, + channel: str, + purpose: str, + **kwargs, + ) -> AsyncSlackResponse: + """Sets the purpose for a private channel.""" + kwargs.update({"channel": channel, "purpose": purpose}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("groups.setPurpose", json=kwargs) + + async def groups_setTopic( + self, + *, + channel: str, + topic: str, + **kwargs, + ) -> AsyncSlackResponse: + """Sets the topic for a private channel.""" + kwargs.update({"channel": channel, "topic": topic}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("groups.setTopic", json=kwargs) + + async def groups_unarchive( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Unarchives a private channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("groups.unarchive", json=kwargs) + + # -------------------------- + # Deprecated: im.* + # You can use conversations.* APIs instead. + # https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api + # -------------------------- + + async def im_close( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Close a direct message channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("im.close", json=kwargs) + + async def im_history( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Fetches history of messages and events from direct message channel.""" + kwargs.update({"channel": channel}) + return await self.api_call("im.history", http_verb="GET", params=kwargs) + + async def im_list( + self, + **kwargs, + ) -> AsyncSlackResponse: + """Lists direct message channels for the calling user.""" + return await self.api_call("im.list", http_verb="GET", params=kwargs) + + async def im_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> AsyncSlackResponse: + """Sets the read cursor in a direct message channel.""" + kwargs.update({"channel": channel, "ts": ts}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("im.mark", json=kwargs) + + async def im_open( + self, + *, + user: str, + **kwargs, + ) -> AsyncSlackResponse: + """Opens a direct message channel.""" + kwargs.update({"user": user}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("im.open", json=kwargs) + + async def im_replies( + self, + *, + channel: str, + thread_ts: str, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieve a thread of messages posted to a direct message conversation""" + kwargs.update({"channel": channel, "thread_ts": thread_ts}) + return await self.api_call("im.replies", http_verb="GET", params=kwargs) + + # -------------------------- + + async def migration_exchange( + self, + *, + users: Union[str, Sequence[str]], + team_id: Optional[str] = None, + to_old: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """For Enterprise Grid workspaces, map local user IDs to global user IDs + https://api.slack.com/methods/migration.exchange + """ + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + kwargs.update({"team_id": team_id, "to_old": to_old}) + return await self.api_call("migration.exchange", http_verb="GET", params=kwargs) + + # -------------------------- + # Deprecated: mpim.* + # You can use conversations.* APIs instead. + # https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api + # -------------------------- + + async def mpim_close( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Closes a multiparty direct message channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("mpim.close", json=kwargs) + + async def mpim_history( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Fetches history of messages and events from a multiparty direct message.""" + kwargs.update({"channel": channel}) + return await self.api_call("mpim.history", http_verb="GET", params=kwargs) + + async def mpim_list( + self, + **kwargs, + ) -> AsyncSlackResponse: + """Lists multiparty direct message channels for the calling user.""" + return await self.api_call("mpim.list", http_verb="GET", params=kwargs) + + async def mpim_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> AsyncSlackResponse: + """Sets the read cursor in a multiparty direct message channel.""" + kwargs.update({"channel": channel, "ts": ts}) + kwargs = _remove_none_values(kwargs) + return await self.api_call("mpim.mark", json=kwargs) + + async def mpim_open( + self, + *, + users: Union[str, Sequence[str]], + **kwargs, + ) -> AsyncSlackResponse: + """This method opens a multiparty direct message.""" + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + return await self.api_call("mpim.open", params=kwargs) + + async def mpim_replies( + self, + *, + channel: str, + thread_ts: str, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieve a thread of messages posted to a direct message conversation from a + multiparty direct message. + """ + kwargs.update({"channel": channel, "thread_ts": thread_ts}) + return await self.api_call("mpim.replies", http_verb="GET", params=kwargs) + + # -------------------------- + + async def oauth_v2_access( + self, + *, + client_id: str, + client_secret: str, + # This field is required when processing the OAuth redirect URL requests + # while it's absent for token rotation + code: Optional[str] = None, + redirect_uri: Optional[str] = None, + # This field is required for token rotation + grant_type: Optional[str] = None, + # This field is required for token rotation + refresh_token: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Exchanges a temporary OAuth verifier code for an access token. + https://api.slack.com/methods/oauth.v2.access + """ + if redirect_uri is not None: + kwargs.update({"redirect_uri": redirect_uri}) + if code is not None: + kwargs.update({"code": code}) + if grant_type is not None: + kwargs.update({"grant_type": grant_type}) + if refresh_token is not None: + kwargs.update({"refresh_token": refresh_token}) + return await self.api_call( + "oauth.v2.access", + data=kwargs, + auth={"client_id": client_id, "client_secret": client_secret}, + ) + + async def oauth_access( + self, + *, + client_id: str, + client_secret: str, + code: str, + redirect_uri: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Exchanges a temporary OAuth verifier code for an access token. + https://api.slack.com/methods/oauth.access + """ + if redirect_uri is not None: + kwargs.update({"redirect_uri": redirect_uri}) + kwargs.update({"code": code}) + return await self.api_call( + "oauth.access", + data=kwargs, + auth={"client_id": client_id, "client_secret": client_secret}, + ) + + async def oauth_v2_exchange( + self, + *, + token: str, + client_id: str, + client_secret: str, + **kwargs, + ) -> AsyncSlackResponse: + """Exchanges a legacy access token for a new expiring access token and refresh token + https://api.slack.com/methods/oauth.v2.exchange + """ + kwargs.update({"client_id": client_id, "client_secret": client_secret, "token": token}) + return await self.api_call("oauth.v2.exchange", params=kwargs) + + async def openid_connect_token( + self, + client_id: str, + client_secret: str, + code: Optional[str] = None, + redirect_uri: Optional[str] = None, + grant_type: Optional[str] = None, + refresh_token: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Exchanges a temporary OAuth verifier code for an access token for Sign in with Slack. + https://api.slack.com/methods/openid.connect.token + """ + if redirect_uri is not None: + kwargs.update({"redirect_uri": redirect_uri}) + if code is not None: + kwargs.update({"code": code}) + if grant_type is not None: + kwargs.update({"grant_type": grant_type}) + if refresh_token is not None: + kwargs.update({"refresh_token": refresh_token}) + return await self.api_call( + "openid.connect.token", + data=kwargs, + auth={"client_id": client_id, "client_secret": client_secret}, + ) + + async def openid_connect_userInfo( + self, + **kwargs, + ) -> AsyncSlackResponse: + """Get the identity of a user who has authorized Sign in with Slack. + https://api.slack.com/methods/openid.connect.userInfo + """ + return await self.api_call("openid.connect.userInfo", params=kwargs) + + async def pins_add( + self, + *, + channel: str, + timestamp: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Pins an item to a channel. + https://api.slack.com/methods/pins.add + """ + kwargs.update({"channel": channel, "timestamp": timestamp}) + return await self.api_call("pins.add", params=kwargs) + + async def pins_list( + self, + *, + channel: str, + **kwargs, + ) -> AsyncSlackResponse: + """Lists items pinned to a channel. + https://api.slack.com/methods/pins.list + """ + kwargs.update({"channel": channel}) + return await self.api_call("pins.list", http_verb="GET", params=kwargs) + + async def pins_remove( + self, + *, + channel: str, + timestamp: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Un-pins an item from a channel. + https://api.slack.com/methods/pins.remove + """ + kwargs.update({"channel": channel, "timestamp": timestamp}) + return await self.api_call("pins.remove", params=kwargs) + + async def reactions_add( + self, + *, + channel: str, + name: str, + timestamp: str, + **kwargs, + ) -> AsyncSlackResponse: + """Adds a reaction to an item. + https://api.slack.com/methods/reactions.add + """ + kwargs.update({"channel": channel, "name": name, "timestamp": timestamp}) + return await self.api_call("reactions.add", params=kwargs) + + async def reactions_get( + self, + *, + channel: Optional[str] = None, + file: Optional[str] = None, + file_comment: Optional[str] = None, + full: Optional[bool] = None, + timestamp: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Gets reactions for an item. + https://api.slack.com/methods/reactions.get + """ + kwargs.update( + { + "channel": channel, + "file": file, + "file_comment": file_comment, + "full": full, + "timestamp": timestamp, + } + ) + return await self.api_call("reactions.get", http_verb="GET", params=kwargs) + + async def reactions_list( + self, + *, + count: Optional[int] = None, + cursor: Optional[str] = None, + full: Optional[bool] = None, + limit: Optional[int] = None, + page: Optional[int] = None, + team_id: Optional[str] = None, + user: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Lists reactions made by a user. + https://api.slack.com/methods/reactions.list + """ + kwargs.update( + { + "count": count, + "cursor": cursor, + "full": full, + "limit": limit, + "page": page, + "team_id": team_id, + "user": user, + } + ) + return await self.api_call("reactions.list", http_verb="GET", params=kwargs) + + async def reactions_remove( + self, + *, + name: str, + channel: Optional[str] = None, + file: Optional[str] = None, + file_comment: Optional[str] = None, + timestamp: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Removes a reaction from an item. + https://api.slack.com/methods/reactions.remove + """ + kwargs.update( + { + "name": name, + "channel": channel, + "file": file, + "file_comment": file_comment, + "timestamp": timestamp, + } + ) + return await self.api_call("reactions.remove", params=kwargs) + + async def reminders_add( + self, + *, + text: str, + time: str, + team_id: Optional[str] = None, + user: Optional[str] = None, + recurrence: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Creates a reminder. + https://api.slack.com/methods/reminders.add + """ + kwargs.update( + { + "text": text, + "time": time, + "team_id": team_id, + "user": user, + "recurrence": recurrence, + } + ) + return await self.api_call("reminders.add", params=kwargs) + + async def reminders_complete( + self, + *, + reminder: str, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Marks a reminder as complete. + https://api.slack.com/methods/reminders.complete + """ + kwargs.update({"reminder": reminder, "team_id": team_id}) + return await self.api_call("reminders.complete", params=kwargs) + + async def reminders_delete( + self, + *, + reminder: str, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Deletes a reminder. + https://api.slack.com/methods/reminders.delete + """ + kwargs.update({"reminder": reminder, "team_id": team_id}) + return await self.api_call("reminders.delete", params=kwargs) + + async def reminders_info( + self, + *, + reminder: str, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Gets information about a reminder. + https://api.slack.com/methods/reminders.info + """ + kwargs.update({"reminder": reminder, "team_id": team_id}) + return await self.api_call("reminders.info", http_verb="GET", params=kwargs) + + async def reminders_list( + self, + *, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Lists all reminders created by or for a given user. + https://api.slack.com/methods/reminders.list + """ + kwargs.update({"team_id": team_id}) + return await self.api_call("reminders.list", http_verb="GET", params=kwargs) + + async def rtm_connect( + self, + *, + batch_presence_aware: Optional[bool] = None, + presence_sub: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Starts a Real Time Messaging session. + https://api.slack.com/methods/rtm.connect + """ + kwargs.update({"batch_presence_aware": batch_presence_aware, "presence_sub": presence_sub}) + return await self.api_call("rtm.connect", http_verb="GET", params=kwargs) + + async def rtm_start( + self, + *, + batch_presence_aware: Optional[bool] = None, + include_locale: Optional[bool] = None, + mpim_aware: Optional[bool] = None, + no_latest: Optional[bool] = None, + no_unreads: Optional[bool] = None, + presence_sub: Optional[bool] = None, + simple_latest: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Starts a Real Time Messaging session. + https://api.slack.com/methods/rtm.start + """ + kwargs.update( + { + "batch_presence_aware": batch_presence_aware, + "include_locale": include_locale, + "mpim_aware": mpim_aware, + "no_latest": no_latest, + "no_unreads": no_unreads, + "presence_sub": presence_sub, + "simple_latest": simple_latest, + } + ) + return await self.api_call("rtm.start", http_verb="GET", params=kwargs) + + async def search_all( + self, + *, + query: str, + count: Optional[int] = None, + highlight: Optional[bool] = None, + page: Optional[int] = None, + sort: Optional[str] = None, + sort_dir: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Searches for messages and files matching a query. + https://api.slack.com/methods/search.all + """ + kwargs.update( + { + "query": query, + "count": count, + "highlight": highlight, + "page": page, + "sort": sort, + "sort_dir": sort_dir, + "team_id": team_id, + } + ) + return await self.api_call("search.all", http_verb="GET", params=kwargs) + + async def search_files( + self, + *, + query: str, + count: Optional[int] = None, + highlight: Optional[bool] = None, + page: Optional[int] = None, + sort: Optional[str] = None, + sort_dir: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Searches for files matching a query. + https://api.slack.com/methods/search.files + """ + kwargs.update( + { + "query": query, + "count": count, + "highlight": highlight, + "page": page, + "sort": sort, + "sort_dir": sort_dir, + "team_id": team_id, + } + ) + return await self.api_call("search.files", http_verb="GET", params=kwargs) + + async def search_messages( + self, + *, + query: str, + count: Optional[int] = None, + cursor: Optional[str] = None, + highlight: Optional[bool] = None, + page: Optional[int] = None, + sort: Optional[str] = None, + sort_dir: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Searches for messages matching a query. + https://api.slack.com/methods/search.messages + """ + kwargs.update( + { + "query": query, + "count": count, + "cursor": cursor, + "highlight": highlight, + "page": page, + "sort": sort, + "sort_dir": sort_dir, + "team_id": team_id, + } + ) + return await self.api_call("search.messages", http_verb="GET", params=kwargs) + + async def stars_add( + self, + *, + channel: Optional[str] = None, + file: Optional[str] = None, + file_comment: Optional[str] = None, + timestamp: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Adds a star to an item. + https://api.slack.com/methods/stars.add + """ + kwargs.update( + { + "channel": channel, + "file": file, + "file_comment": file_comment, + "timestamp": timestamp, + } + ) + return await self.api_call("stars.add", params=kwargs) + + async def stars_list( + self, + *, + count: Optional[int] = None, + cursor: Optional[str] = None, + limit: Optional[int] = None, + page: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Lists stars for a user. + https://api.slack.com/methods/stars.list + """ + kwargs.update( + { + "count": count, + "cursor": cursor, + "limit": limit, + "page": page, + "team_id": team_id, + } + ) + return await self.api_call("stars.list", http_verb="GET", params=kwargs) + + async def stars_remove( + self, + *, + channel: Optional[str] = None, + file: Optional[str] = None, + file_comment: Optional[str] = None, + timestamp: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Removes a star from an item. + https://api.slack.com/methods/stars.remove + """ + kwargs.update( + { + "channel": channel, + "file": file, + "file_comment": file_comment, + "timestamp": timestamp, + } + ) + return await self.api_call("stars.remove", params=kwargs) + + async def team_accessLogs( + self, + *, + before: Optional[Union[int, str]] = None, + count: Optional[Union[int, str]] = None, + page: Optional[Union[int, str]] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Gets the access logs for the current team. + https://api.slack.com/methods/team.accessLogs + """ + kwargs.update( + { + "before": before, + "count": count, + "page": page, + "team_id": team_id, + } + ) + return await self.api_call("team.accessLogs", http_verb="GET", params=kwargs) + + async def team_billableInfo( + self, + *, + team_id: Optional[str] = None, + user: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Gets billable users information for the current team. + https://api.slack.com/methods/team.billableInfo + """ + kwargs.update({"team_id": team_id, "user": user}) + return await self.api_call("team.billableInfo", http_verb="GET", params=kwargs) + + async def team_billing_info( + self, + **kwargs, + ) -> AsyncSlackResponse: + """Reads a workspace's billing plan information. + https://api.slack.com/methods/team.billing.info + """ + return await self.api_call("team.billing.info", params=kwargs) + + async def team_info( + self, + *, + team: Optional[str] = None, + domain: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Gets information about the current team. + https://api.slack.com/methods/team.info + """ + kwargs.update({"team": team, "domain": domain}) + return await self.api_call("team.info", http_verb="GET", params=kwargs) + + async def team_integrationLogs( + self, + *, + app_id: Optional[str] = None, + change_type: Optional[str] = None, + count: Optional[Union[int, str]] = None, + page: Optional[Union[int, str]] = None, + service_id: Optional[str] = None, + team_id: Optional[str] = None, + user: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Gets the integration logs for the current team. + https://api.slack.com/methods/team.integrationLogs + """ + kwargs.update( + { + "app_id": app_id, + "change_type": change_type, + "count": count, + "page": page, + "service_id": service_id, + "team_id": team_id, + "user": user, + } + ) + return await self.api_call("team.integrationLogs", http_verb="GET", params=kwargs) + + async def team_profile_get( + self, + *, + visibility: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieve a team's profile. + https://api.slack.com/methods/team.profile.get + """ + kwargs.update({"visibility": visibility}) + return await self.api_call("team.profile.get", http_verb="GET", params=kwargs) + + async def team_preferences_list( + self, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieve a list of a workspace's team preferences. + https://api.slack.com/methods/team.preferences.list + """ + return await self.api_call("team.preferences.list", params=kwargs) + + async def usergroups_create( + self, + *, + name: str, + channels: Optional[Union[str, Sequence[str]]] = None, + description: Optional[str] = None, + handle: Optional[str] = None, + include_count: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Create a User Group + https://api.slack.com/methods/usergroups.create + """ + kwargs.update( + { + "name": name, + "description": description, + "handle": handle, + "include_count": include_count, + "team_id": team_id, + } + ) + if isinstance(channels, (list, Tuple)): + kwargs.update({"channels": ",".join(channels)}) + else: + kwargs.update({"channels": channels}) + return await self.api_call("usergroups.create", params=kwargs) + + async def usergroups_disable( + self, + *, + usergroup: str, + include_count: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Disable an existing User Group + https://api.slack.com/methods/usergroups.disable + """ + kwargs.update({"usergroup": usergroup, "include_count": include_count, "team_id": team_id}) + return await self.api_call("usergroups.disable", params=kwargs) + + async def usergroups_enable( + self, + *, + usergroup: str, + include_count: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Enable a User Group + https://api.slack.com/methods/usergroups.enable + """ + kwargs.update({"usergroup": usergroup, "include_count": include_count, "team_id": team_id}) + return await self.api_call("usergroups.enable", params=kwargs) + + async def usergroups_list( + self, + *, + include_count: Optional[bool] = None, + include_disabled: Optional[bool] = None, + include_users: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List all User Groups for a team + https://api.slack.com/methods/usergroups.list + """ + kwargs.update( + { + "include_count": include_count, + "include_disabled": include_disabled, + "include_users": include_users, + "team_id": team_id, + } + ) + return await self.api_call("usergroups.list", http_verb="GET", params=kwargs) + + async def usergroups_update( + self, + *, + usergroup: str, + channels: Optional[Union[str, Sequence[str]]] = None, + description: Optional[str] = None, + handle: Optional[str] = None, + include_count: Optional[bool] = None, + name: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Update an existing User Group + https://api.slack.com/methods/usergroups.update + """ + kwargs.update( + { + "usergroup": usergroup, + "description": description, + "handle": handle, + "include_count": include_count, + "name": name, + "team_id": team_id, + } + ) + if isinstance(channels, (list, Tuple)): + kwargs.update({"channels": ",".join(channels)}) + else: + kwargs.update({"channels": channels}) + return await self.api_call("usergroups.update", params=kwargs) + + async def usergroups_users_list( + self, + *, + usergroup: str, + include_disabled: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List all users in a User Group + https://api.slack.com/methods/usergroups.users.list + """ + kwargs.update( + { + "usergroup": usergroup, + "include_disabled": include_disabled, + "team_id": team_id, + } + ) + return await self.api_call("usergroups.users.list", http_verb="GET", params=kwargs) + + async def usergroups_users_update( + self, + *, + usergroup: str, + users: Union[str, Sequence[str]], + include_count: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Update the list of users for a User Group + https://api.slack.com/methods/usergroups.users.update + """ + kwargs.update( + { + "usergroup": usergroup, + "include_count": include_count, + "team_id": team_id, + } + ) + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + return await self.api_call("usergroups.users.update", params=kwargs) + + async def users_conversations( + self, + *, + cursor: Optional[str] = None, + exclude_archived: Optional[bool] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + types: Optional[Union[str, Sequence[str]]] = None, + user: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """List conversations the calling user may access. + https://api.slack.com/methods/users.conversations + """ + kwargs.update( + { + "cursor": cursor, + "exclude_archived": exclude_archived, + "limit": limit, + "team_id": team_id, + "user": user, + } + ) + if isinstance(types, (list, Tuple)): + kwargs.update({"types": ",".join(types)}) + else: + kwargs.update({"types": types}) + return await self.api_call("users.conversations", http_verb="GET", params=kwargs) + + async def users_deletePhoto( + self, + **kwargs, + ) -> AsyncSlackResponse: + """Delete the user profile photo + https://api.slack.com/methods/users.deletePhoto + """ + return await self.api_call("users.deletePhoto", http_verb="GET", params=kwargs) + + async def users_getPresence( + self, + *, + user: str, + **kwargs, + ) -> AsyncSlackResponse: + """Gets user presence information. + https://api.slack.com/methods/users.getPresence + """ + kwargs.update({"user": user}) + return await self.api_call("users.getPresence", http_verb="GET", params=kwargs) + + async def users_identity( + self, + **kwargs, + ) -> AsyncSlackResponse: + """Get a user's identity. + https://api.slack.com/methods/users.identity + """ + return await self.api_call("users.identity", http_verb="GET", params=kwargs) + + async def users_info( + self, + *, + user: str, + include_locale: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Gets information about a user. + https://api.slack.com/methods/users.info + """ + kwargs.update({"user": user, "include_locale": include_locale}) + return await self.api_call("users.info", http_verb="GET", params=kwargs) + + async def users_list( + self, + *, + cursor: Optional[str] = None, + include_locale: Optional[bool] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Lists all users in a Slack team. + https://api.slack.com/methods/users.list + """ + kwargs.update( + { + "cursor": cursor, + "include_locale": include_locale, + "limit": limit, + "team_id": team_id, + } + ) + return await self.api_call("users.list", http_verb="GET", params=kwargs) + + async def users_lookupByEmail( + self, + *, + email: str, + **kwargs, + ) -> AsyncSlackResponse: + """Find a user with an email address. + https://api.slack.com/methods/users.lookupByEmail + """ + kwargs.update({"email": email}) + return await self.api_call("users.lookupByEmail", http_verb="GET", params=kwargs) + + async def users_setPhoto( + self, + *, + image: Union[str, IOBase], + crop_w: Optional[Union[int, str]] = None, + crop_x: Optional[Union[int, str]] = None, + crop_y: Optional[Union[int, str]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Set the user profile photo + https://api.slack.com/methods/users.setPhoto + """ + kwargs.update({"crop_w": crop_w, "crop_x": crop_x, "crop_y": crop_y}) + return await self.api_call("users.setPhoto", files={"image": image}, data=kwargs) + + async def users_setPresence( + self, + *, + presence: str, + **kwargs, + ) -> AsyncSlackResponse: + """Manually sets user presence. + https://api.slack.com/methods/users.setPresence + """ + kwargs.update({"presence": presence}) + return await self.api_call("users.setPresence", params=kwargs) + + async def users_profile_get( + self, + *, + user: Optional[str] = None, + include_labels: Optional[bool] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Retrieves a user's profile information. + https://api.slack.com/methods/users.profile.get + """ + kwargs.update({"user": user, "include_labels": include_labels}) + return await self.api_call("users.profile.get", http_verb="GET", params=kwargs) + + async def users_profile_set( + self, + *, + name: Optional[str] = None, + value: Optional[str] = None, + user: Optional[str] = None, + profile: Optional[Dict] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Set the profile information for a user. + https://api.slack.com/methods/users.profile.set + """ + kwargs.update( + { + "name": name, + "profile": profile, + "user": user, + "value": value, + } + ) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "profile" parameter + return await self.api_call("users.profile.set", json=kwargs) + + async def views_open( + self, + *, + trigger_id: str, + view: Union[dict, View], + **kwargs, + ) -> AsyncSlackResponse: + """Open a view for a user. + https://api.slack.com/methods/views.open + See https://api.slack.com/block-kit/surfaces/modals for details. + """ + kwargs.update({"trigger_id": trigger_id}) + if isinstance(view, View): + kwargs.update({"view": view.to_dict()}) + else: + kwargs.update({"view": view}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "view" parameter + return await self.api_call("views.open", json=kwargs) + + async def views_push( + self, + *, + trigger_id: str, + view: Union[dict, View], + **kwargs, + ) -> AsyncSlackResponse: + """Push a view onto the stack of a root view. + Push a new view onto the existing view stack by passing a view + payload and a valid trigger_id generated from an interaction + within the existing modal. + Read the modals documentation (https://api.slack.com/block-kit/surfaces/modals) + to learn more about the lifecycle and intricacies of views. + https://api.slack.com/methods/views.push + """ + kwargs.update({"trigger_id": trigger_id}) + if isinstance(view, View): + kwargs.update({"view": view.to_dict()}) + else: + kwargs.update({"view": view}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "view" parameter + return await self.api_call("views.push", json=kwargs) + + async def views_update( + self, + *, + view: Union[dict, View], + external_id: Optional[str] = None, + view_id: Optional[str] = None, + hash: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Update an existing view. + Update a view by passing a new view definition along with the + view_id returned in views.open or the external_id. + See the modals documentation (https://api.slack.com/block-kit/surfaces/modals#updating_views) + to learn more about updating views and avoiding race conditions with the hash argument. + https://api.slack.com/methods/views.update + """ + if isinstance(view, View): + kwargs.update({"view": view.to_dict()}) + else: + kwargs.update({"view": view}) + if external_id: + kwargs.update({"external_id": external_id}) + elif view_id: + kwargs.update({"view_id": view_id}) + else: + raise e.SlackRequestError("Either view_id or external_id is required.") + kwargs.update({"hash": hash}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "view" parameter + return await self.api_call("views.update", json=kwargs) + + async def views_publish( + self, + *, + user_id: str, + view: Union[dict, View], + hash: Optional[str] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Publish a static view for a User. + Create or update the view that comprises an + app's Home tab (https://api.slack.com/surfaces/tabs) + https://api.slack.com/methods/views.publish + """ + kwargs.update({"user_id": user_id, "hash": hash}) + if isinstance(view, View): + kwargs.update({"view": view.to_dict()}) + else: + kwargs.update({"view": view}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "view" parameter + return await self.api_call("views.publish", json=kwargs) + + async def workflows_stepCompleted( + self, + *, + workflow_step_execute_id: str, + outputs: Optional[dict] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Indicate a successful outcome of a workflow step's execution. + https://api.slack.com/methods/workflows.stepCompleted + """ + kwargs.update({"workflow_step_execute_id": workflow_step_execute_id}) + if outputs is not None: + kwargs.update({"outputs": outputs}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "outputs" parameter + return await self.api_call("workflows.stepCompleted", json=kwargs) + + async def workflows_stepFailed( + self, + *, + workflow_step_execute_id: str, + error: Dict[str, str], + **kwargs, + ) -> AsyncSlackResponse: + """Indicate an unsuccessful outcome of a workflow step's execution. + https://api.slack.com/methods/workflows.stepFailed + """ + kwargs.update( + { + "workflow_step_execute_id": workflow_step_execute_id, + "error": error, + } + ) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "error" parameter + return await self.api_call("workflows.stepFailed", json=kwargs) + + async def workflows_updateStep( + self, + *, + workflow_step_edit_id: str, + inputs: Optional[Dict[str, Any]] = None, + outputs: Optional[List[Dict[str, str]]] = None, + **kwargs, + ) -> AsyncSlackResponse: + """Update the configuration for a workflow extension step. + https://api.slack.com/methods/workflows.updateStep + """ + kwargs.update({"workflow_step_edit_id": workflow_step_edit_id}) + if inputs is not None: + kwargs.update({"inputs": inputs}) + if outputs is not None: + kwargs.update({"outputs": outputs}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "inputs" / "outputs" parameters + return await self.api_call("workflows.updateStep", json=kwargs) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/web/async_internal_utils.py b/core_service/aws_lambda/project/packages/slack_sdk/web/async_internal_utils.py new file mode 100644 index 0000000..19541e7 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/web/async_internal_utils.py @@ -0,0 +1,209 @@ +import asyncio +import json +import logging +from asyncio import AbstractEventLoop +from logging import Logger +from typing import Optional, BinaryIO, Dict, Sequence, Union, List, Any + +import aiohttp +from aiohttp import ClientSession + +from slack_sdk.errors import SlackApiError +from slack_sdk.web.internal_utils import _build_unexpected_body_error_message + +from slack_sdk.http_retry.async_handler import AsyncRetryHandler +from slack_sdk.http_retry.request import HttpRequest as RetryHttpRequest +from slack_sdk.http_retry.response import HttpResponse as RetryHttpResponse +from slack_sdk.http_retry.state import RetryState + + +def _get_event_loop() -> AbstractEventLoop: + """Retrieves the event loop or creates a new one.""" + try: + return asyncio.get_event_loop() + except RuntimeError: + loop = asyncio.new_event_loop() + asyncio.set_event_loop(loop) + return loop + + +def _files_to_data(req_args: dict) -> Sequence[BinaryIO]: + open_files = [] + files = req_args.pop("files", None) + if files is not None: + for k, v in files.items(): + if isinstance(v, str): + f = open(v.encode("utf-8", "ignore"), "rb") + open_files.append(f) + req_args["data"].update({k: f}) + else: + req_args["data"].update({k: v}) + return open_files + + +async def _request_with_session( + *, + current_session: Optional[ClientSession], + timeout: int, + logger: Logger, + http_verb: str, + api_url: str, + req_args: dict, + # set the default to an empty array for legacy clients + retry_handlers: Optional[List[AsyncRetryHandler]] = None, +) -> Dict[str, Any]: + """Submit the HTTP request with the running session or a new session. + Returns: + A dictionary of the response data. + """ + retry_handlers = retry_handlers if retry_handlers is not None else [] + session = None + use_running_session = current_session and not current_session.closed + if use_running_session: + session = current_session + else: + session = aiohttp.ClientSession( + timeout=aiohttp.ClientTimeout(total=timeout), + auth=req_args.pop("auth", None), + ) + + last_error: Optional[Exception] = None + resp: Optional[Dict[str, Any]] = None + try: + retry_request = RetryHttpRequest( + method=http_verb, + url=api_url, + headers=req_args.get("headers", {}), + body_params=req_args.get("params"), + data=req_args.get("data"), + ) + + retry_state = RetryState() + counter_for_safety = 0 + while counter_for_safety < 100: + counter_for_safety += 1 + # If this is a retry, the next try started here. We can reset the flag. + retry_state.next_attempt_requested = False + retry_response: Optional[RetryHttpResponse] = None + + if logger.level <= logging.DEBUG: + + def convert_params(values: dict) -> dict: + if not values or not isinstance(values, dict): + return {} + return {k: ("(bytes)" if isinstance(v, bytes) else v) for k, v in values.items()} + + headers = { + k: "(redacted)" if k.lower() == "authorization" else v for k, v in req_args.get("headers", {}).items() + } + logger.debug( + f"Sending a request - url: {http_verb} {api_url}, " + f"params: {convert_params(req_args.get('params'))}, " + f"files: {convert_params(req_args.get('files'))}, " + f"data: {convert_params(req_args.get('data'))}, " + f"json: {convert_params(req_args.get('json'))}, " + f"proxy: {convert_params(req_args.get('proxy'))}, " + f"headers: {headers}" + ) + + try: + async with session.request(http_verb, api_url, **req_args) as res: + data: Union[dict, bytes] = {} + if res.content_type == "application/gzip": + # admin.analytics.getFile + data = await res.read() + retry_response = RetryHttpResponse( + status_code=res.status, + headers=res.headers, + data=data, + ) + else: + try: + data = await res.json() + retry_response = RetryHttpResponse( + status_code=res.status, + headers=res.headers, + body=data, + ) + except aiohttp.ContentTypeError: + logger.debug(f"No response data returned from the following API call: {api_url}.") + except json.decoder.JSONDecodeError: + try: + body: str = await res.text() + message = _build_unexpected_body_error_message(body) + raise SlackApiError(message, res) + except Exception as e: + raise SlackApiError( + f"Unexpectedly failed to read the response body: {str(e)}", + res, + ) + + if logger.level <= logging.DEBUG: + body = data if isinstance(data, dict) else "(binary)" + logger.debug( + "Received the following response - " + f"status: {res.status}, " + f"headers: {dict(res.headers)}, " + f"body: {body}" + ) + + if res.status == 429: + for handler in retry_handlers: + if await handler.can_retry_async( + state=retry_state, + request=retry_request, + response=retry_response, + ): + if logger.level <= logging.DEBUG: + logger.info( + f"A retry handler found: {type(handler).__name__} " + f"for {http_verb} {api_url} - rate_limited" + ) + await handler.prepare_for_next_attempt_async( + state=retry_state, + request=retry_request, + response=retry_response, + ) + break + + if retry_state.next_attempt_requested is False: + response = { + "data": data, + "headers": res.headers, + "status_code": res.status, + } + return response + + except Exception as e: + last_error = e + for handler in retry_handlers: + if await handler.can_retry_async( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ): + if logger.level <= logging.DEBUG: + logger.info( + f"A retry handler found: {type(handler).__name__} " f"for {http_verb} {api_url} - {e}" + ) + await handler.prepare_for_next_attempt_async( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ) + break + + if retry_state.next_attempt_requested is False: + raise last_error + + if resp is not None: + return resp + raise last_error + + finally: + if not use_running_session: + await session.close() + + return response diff --git a/core_service/aws_lambda/project/packages/slack_sdk/web/async_slack_response.py b/core_service/aws_lambda/project/packages/slack_sdk/web/async_slack_response.py new file mode 100644 index 0000000..868bf21 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/web/async_slack_response.py @@ -0,0 +1,192 @@ +"""A Python module for interacting and consuming responses from Slack.""" + +import logging +from typing import Union + +import slack_sdk.errors as e +from .internal_utils import _next_cursor_is_present + + +class AsyncSlackResponse: + """An iterable container of response data. + + Attributes: + data (dict): The json-encoded content of the response. Along + with the headers and status code information. + + Methods: + validate: Check if the response from Slack was successful. + get: Retrieves any key from the response data. + next: Retrieves the next portion of results, + if 'next_cursor' is present. + + Example: + ```python + import os + import slack + + client = slack.AsyncWebClient(token=os.environ['SLACK_API_TOKEN']) + + response1 = await client.auth_revoke(test='true') + assert not response1['revoked'] + + response2 = await client.auth_test() + assert response2.get('ok', False) + + users = [] + async for page in await client.users_list(limit=2): + users = users + page['members'] + ``` + + Note: + Some responses return collections of information + like channel and user lists. If they do it's likely + that you'll only receive a portion of results. This + object allows you to iterate over the response which + makes subsequent API requests until your code hits + 'break' or there are no more results to be found. + + Any attributes or methods prefixed with _underscores are + intended to be "private" internal use only. They may be changed or + removed at anytime. + """ + + def __init__( + self, + *, + client, # AsyncWebClient + http_verb: str, + api_url: str, + req_args: dict, + data: Union[dict, bytes], # data can be binary data + headers: dict, + status_code: int, + ): + self.http_verb = http_verb + self.api_url = api_url + self.req_args = req_args + self.data = data + self.headers = headers + self.status_code = status_code + self._initial_data = data + self._iteration = None # for __iter__ & __next__ + self._client = client + self._logger = logging.getLogger(__name__) + + def __str__(self): + """Return the Response data if object is converted to a string.""" + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + return f"{self.data}" + + def __contains__(self, key: str) -> bool: + return self.get(key) is not None + + def __getitem__(self, key): + """Retrieves any key from the data store. + + Note: + This is implemented so users can reference the + SlackResponse object like a dictionary. + e.g. response["ok"] + + Returns: + The value from data or None. + """ + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + if self.data is None: + raise ValueError("As the response.data is empty, this operation is unsupported") + return self.data.get(key, None) + + def __aiter__(self): + """Enables the ability to iterate over the response. + It's required async-for the iterator protocol. + + Note: + This enables Slack cursor-based pagination. + + Returns: + (AsyncSlackResponse) self + """ + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + self._iteration = 0 + self.data = self._initial_data + return self + + async def __anext__(self): + """Retrieves the next portion of results, if 'next_cursor' is present. + + Note: + Some responses return collections of information + like channel and user lists. If they do it's likely + that you'll only receive a portion of results. This + method allows you to iterate over the response until + your code hits 'break' or there are no more results + to be found. + + Returns: + (AsyncSlackResponse) self + With the new response data now attached to this object. + + Raises: + SlackApiError: If the request to the Slack API failed. + StopAsyncIteration: If 'next_cursor' is not present or empty. + """ + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + self._iteration += 1 + if self._iteration == 1: + return self + if _next_cursor_is_present(self.data): # skipcq: PYL-R1705 + params = self.req_args.get("params", {}) + if params is None: + params = {} + params.update({"cursor": self.data["response_metadata"]["next_cursor"]}) + self.req_args.update({"params": params}) + + response = await self._client._request( # skipcq: PYL-W0212 + http_verb=self.http_verb, + api_url=self.api_url, + req_args=self.req_args, + ) + + self.data = response["data"] + self.headers = response["headers"] + self.status_code = response["status_code"] + return self.validate() + else: + raise StopAsyncIteration + + def get(self, key, default=None): + """Retrieves any key from the response data. + + Note: + This is implemented so users can reference the + SlackResponse object like a dictionary. + e.g. response.get("ok", False) + + Returns: + The value from data or the specified default. + """ + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + if self.data is None: + return None + return self.data.get(key, default) + + def validate(self): + """Check if the response from Slack was successful. + + Returns: + (AsyncSlackResponse) + This method returns it's own object. e.g. 'self' + + Raises: + SlackApiError: The request to the Slack API failed. + """ + if self.status_code == 200 and self.data and (isinstance(self.data, bytes) or self.data.get("ok", False)): + return self + msg = f"The request to the Slack API failed. (url: {self.api_url})" + raise e.SlackApiError(message=msg, response=self) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/web/base_client.py b/core_service/aws_lambda/project/packages/slack_sdk/web/base_client.py new file mode 100644 index 0000000..0f53295 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/web/base_client.py @@ -0,0 +1,580 @@ +"""A Python module for interacting with Slack's Web API.""" + +import copy +import hashlib +import hmac +import io +import json +import logging +import mimetypes +import urllib +import uuid +import warnings +from base64 import b64encode +from http.client import HTTPResponse +from ssl import SSLContext +from typing import BinaryIO, Dict, List, Any +from typing import Optional, Union +from urllib.error import HTTPError +from urllib.parse import urlencode +from urllib.request import Request, urlopen, OpenerDirector, ProxyHandler, HTTPSHandler + +import slack_sdk.errors as err +from slack_sdk.errors import SlackRequestError +from .deprecation import show_2020_01_deprecation +from .internal_utils import ( + convert_bool_to_0_or_1, + get_user_agent, + _get_url, + _build_req_args, + _build_unexpected_body_error_message, +) +from .slack_response import SlackResponse +from slack_sdk.http_retry import default_retry_handlers +from slack_sdk.http_retry.handler import RetryHandler +from slack_sdk.http_retry.request import HttpRequest as RetryHttpRequest +from slack_sdk.http_retry.response import HttpResponse as RetryHttpResponse +from slack_sdk.http_retry.state import RetryState +from slack_sdk.proxy_env_variable_loader import load_http_proxy_from_env + + +class BaseClient: + BASE_URL = "https://www.slack.com/api/" + + def __init__( + self, + token: Optional[str] = None, + base_url: str = BASE_URL, + timeout: int = 30, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + headers: Optional[dict] = None, + user_agent_prefix: Optional[str] = None, + user_agent_suffix: Optional[str] = None, + # for Org-Wide App installation + team_id: Optional[str] = None, + logger: Optional[logging.Logger] = None, + retry_handlers: Optional[List[RetryHandler]] = None, + ): + self.token = None if token is None else token.strip() + """A string specifying an `xoxp-*` or `xoxb-*` token.""" + self.base_url = base_url + """A string representing the Slack API base URL. + Default is `'https://www.slack.com/api/'`.""" + self.timeout = timeout + """The maximum number of seconds the client will wait + to connect and receive a response from Slack. + Default is 30 seconds.""" + self.ssl = ssl + """An [`ssl.SSLContext`](https://docs.python.org/3/library/ssl.html#ssl.SSLContext) + instance, helpful for specifying your own custom + certificate chain.""" + self.proxy = proxy + """String representing a fully-qualified URL to a proxy through which + to route all requests to the Slack API. Even if this parameter + is not specified, if any of the following environment variables are + present, they will be loaded into this parameter: `HTTPS_PROXY`, + `https_proxy`, `HTTP_PROXY` or `http_proxy`.""" + self.headers = headers or {} + """`dict` representing additional request headers to attach to all requests.""" + self.headers["User-Agent"] = get_user_agent(user_agent_prefix, user_agent_suffix) + self.default_params = {} + if team_id is not None: + self.default_params["team_id"] = team_id + self._logger = logger if logger is not None else logging.getLogger(__name__) + + self.retry_handlers = retry_handlers if retry_handlers is not None else default_retry_handlers() + + if self.proxy is None or len(self.proxy.strip()) == 0: + env_variable = load_http_proxy_from_env(self._logger) + if env_variable is not None: + self.proxy = env_variable + + def api_call( # skipcq: PYL-R1710 + self, + api_method: str, + *, + http_verb: str = "POST", + files: Optional[dict] = None, + data: Optional[dict] = None, + params: Optional[dict] = None, + json: Optional[dict] = None, # skipcq: PYL-W0621 + headers: Optional[dict] = None, + auth: Optional[dict] = None, + ) -> SlackResponse: + """Create a request and execute the API call to Slack. + + Args: + api_method (str): The target Slack API method. + e.g. 'chat.postMessage' + http_verb (str): HTTP Verb. e.g. 'POST' + files (dict): Files to multipart upload. + e.g. {image OR file: file_object OR file_path} + data: The body to attach to the request. If a dictionary is + provided, form-encoding will take place. + e.g. {'key1': 'value1', 'key2': 'value2'} + params (dict): The URL parameters to append to the URL. + e.g. {'key1': 'value1', 'key2': 'value2'} + json (dict): JSON for the body to attach to the request + (if files or data is not specified). + e.g. {'key1': 'value1', 'key2': 'value2'} + headers (dict): Additional request headers + auth (dict): A dictionary that consists of client_id and client_secret + + Returns: + (SlackResponse) + The server's response to an HTTP request. Data + from the response can be accessed like a dict. + If the response included 'next_cursor' it can + be iterated on to execute subsequent requests. + + Raises: + SlackApiError: The following Slack API call failed: + 'chat.postMessage'. + SlackRequestError: Json data can only be submitted as + POST requests. + """ + + api_url = _get_url(self.base_url, api_method) + headers = headers or {} + headers.update(self.headers) + req_args = _build_req_args( + token=self.token, + http_verb=http_verb, + files=files, + data=data, + default_params=self.default_params, + params=params, + json=json, # skipcq: PYL-W0621 + headers=headers, + auth=auth, + ssl=self.ssl, + proxy=self.proxy, + ) + + show_2020_01_deprecation(api_method) + return self._sync_send(api_url=api_url, req_args=req_args) + + # ================================================================= + # urllib based WebClient + # ================================================================= + + def _sync_send(self, api_url, req_args) -> SlackResponse: + params = req_args["params"] if "params" in req_args else None + data = req_args["data"] if "data" in req_args else None + files = req_args["files"] if "files" in req_args else None + _json = req_args["json"] if "json" in req_args else None + headers = req_args["headers"] if "headers" in req_args else None + token = params.get("token") if params and "token" in params else None + auth = req_args["auth"] if "auth" in req_args else None # Basic Auth for oauth.v2.access / oauth.access + if auth is not None: + headers = {} + if isinstance(auth, str): + headers["Authorization"] = auth + elif isinstance(auth, dict): + client_id, client_secret = auth["client_id"], auth["client_secret"] + value = b64encode(f"{client_id}:{client_secret}".encode("utf-8")).decode("ascii") + headers["Authorization"] = f"Basic {value}" + else: + self._logger.warning(f"As the auth: {auth}: {type(auth)} is unsupported, skipped") + + body_params = {} + if params: + body_params.update(params) + if data: + body_params.update(data) + + return self._urllib_api_call( + token=token, + url=api_url, + query_params={}, + body_params=body_params, + files=files, + json_body=_json, + additional_headers=headers, + ) + + def _request_for_pagination(self, api_url: str, req_args: Dict[str, Dict[str, Any]]) -> Dict[str, Any]: + """This method is supposed to be used only for SlackResponse pagination + + You can paginate using Python's for iterator as below: + + for response in client.conversations_list(limit=100): + # do something with each response here + """ + response = self._perform_urllib_http_request(url=api_url, args=req_args) + return { + "status_code": int(response["status"]), + "headers": dict(response["headers"]), + "data": json.loads(response["body"]), + } + + def _urllib_api_call( + self, + *, + token: Optional[str] = None, + url: str, + query_params: Dict[str, str], + json_body: Dict, + body_params: Dict[str, str], + files: Dict[str, io.BytesIO], + additional_headers: Dict[str, str], + ) -> SlackResponse: + """Performs a Slack API request and returns the result. + + Args: + token: Slack API Token (either bot token or user token) + url: Complete URL (e.g., https://www.slack.com/api/chat.postMessage) + query_params: Query string + json_body: JSON data structure (it's still a dict at this point), + if you give this argument, body_params and files will be skipped + body_params: Form body params + files: Files to upload + additional_headers: Request headers to append + + Returns: + API response + """ + files_to_close: List[BinaryIO] = [] + try: + # True/False -> "1"/"0" + query_params = convert_bool_to_0_or_1(query_params) + body_params = convert_bool_to_0_or_1(body_params) + + if self._logger.level <= logging.DEBUG: + + def convert_params(values: dict) -> dict: + if not values or not isinstance(values, dict): + return {} + return {k: ("(bytes)" if isinstance(v, bytes) else v) for k, v in values.items()} + + headers = {k: "(redacted)" if k.lower() == "authorization" else v for k, v in additional_headers.items()} + self._logger.debug( + f"Sending a request - url: {url}, " + f"query_params: {convert_params(query_params)}, " + f"body_params: {convert_params(body_params)}, " + f"files: {convert_params(files)}, " + f"json_body: {json_body}, " + f"headers: {headers}" + ) + + request_data = {} + if files is not None and isinstance(files, dict) and len(files) > 0: + if body_params: + for k, v in body_params.items(): + request_data.update({k: v}) + + for k, v in files.items(): + if isinstance(v, str): + f: BinaryIO = open(v.encode("utf-8", "ignore"), "rb") + files_to_close.append(f) + request_data.update({k: f}) + elif isinstance(v, (bytearray, bytes)): + request_data.update({k: io.BytesIO(v)}) + else: + request_data.update({k: v}) + + request_headers = self._build_urllib_request_headers( + token=token or self.token, + has_json=json is not None, + has_files=files is not None, + additional_headers=additional_headers, + ) + request_args = { + "headers": request_headers, + "data": request_data, + "params": body_params, + "files": files, + "json": json_body, + } + if query_params: + q = urlencode(query_params) + url = f"{url}&{q}" if "?" in url else f"{url}?{q}" + + response = self._perform_urllib_http_request(url=url, args=request_args) + response_body = response.get("body", None) # skipcq: PTC-W0039 + response_body_data: Optional[Union[dict, bytes]] = response_body + if response_body is not None and not isinstance(response_body, bytes): + try: + response_body_data = json.loads(response["body"]) + except json.decoder.JSONDecodeError: + message = _build_unexpected_body_error_message(response.get("body", "")) + raise err.SlackApiError(message, response) + + all_params: Dict[str, Any] = copy.copy(body_params) if body_params is not None else {} + if query_params: + all_params.update(query_params) + request_args["params"] = all_params # for backward-compatibility + + return SlackResponse( + client=self, + http_verb="POST", # you can use POST method for all the Web APIs + api_url=url, + req_args=request_args, + data=response_body_data, + headers=dict(response["headers"]), + status_code=response["status"], + ).validate() + finally: + for f in files_to_close: + if not f.closed: + f.close() + + def _perform_urllib_http_request(self, *, url: str, args: Dict[str, Dict[str, Any]]) -> Dict[str, Any]: + """Performs an HTTP request and parses the response. + + Args: + url: Complete URL (e.g., https://www.slack.com/api/chat.postMessage) + args: args has "headers", "data", "params", and "json" + "headers": Dict[str, str] + "data": Dict[str, Any] + "params": Dict[str, str], + "json": Dict[str, Any], + + Returns: + dict {status: int, headers: Headers, body: str} + """ + headers = args["headers"] + if args["json"]: + body = json.dumps(args["json"]) + headers["Content-Type"] = "application/json;charset=utf-8" + elif args["data"]: + boundary = f"--------------{uuid.uuid4()}" + sep_boundary = b"\r\n--" + boundary.encode("ascii") + end_boundary = sep_boundary + b"--\r\n" + body = io.BytesIO() + data = args["data"] + for key, value in data.items(): + readable = getattr(value, "readable", None) + if readable and value.readable(): + filename = "Uploaded file" + name_attr = getattr(value, "name", None) + if name_attr: + filename = name_attr.decode("utf-8") if isinstance(name_attr, bytes) else name_attr + if "filename" in data: + filename = data["filename"] + mimetype = mimetypes.guess_type(filename)[0] or "application/octet-stream" + title = ( + f'\r\nContent-Disposition: form-data; name="{key}"; filename="{filename}"\r\n' + + f"Content-Type: {mimetype}\r\n" + ) + value = value.read() + else: + title = f'\r\nContent-Disposition: form-data; name="{key}"\r\n' + value = str(value).encode("utf-8") + body.write(sep_boundary) + body.write(title.encode("utf-8")) + body.write(b"\r\n") + body.write(value) + + body.write(end_boundary) + body = body.getvalue() + headers["Content-Type"] = f"multipart/form-data; boundary={boundary}" + headers["Content-Length"] = len(body) + elif args["params"]: + body = urlencode(args["params"]) + headers["Content-Type"] = "application/x-www-form-urlencoded" + else: + body = None + + if isinstance(body, str): + body = body.encode("utf-8") + + # NOTE: Intentionally ignore the `http_verb` here + # Slack APIs accepts any API method requests with POST methods + req = Request(method="POST", url=url, data=body, headers=headers) + resp = None + last_error = None + + retry_state = RetryState() + counter_for_safety = 0 + while counter_for_safety < 100: + counter_for_safety += 1 + # If this is a retry, the next try started here. We can reset the flag. + retry_state.next_attempt_requested = False + + try: + resp = self._perform_urllib_http_request_internal(url, req) + # The resp is a 200 OK response + return resp + + except HTTPError as e: + # As adding new values to HTTPError#headers can be ignored, building a new dict object here + response_headers = dict(e.headers.items()) + resp = {"status": e.code, "headers": response_headers} + if e.code == 429: + # for compatibility with aiohttp + if "retry-after" not in response_headers and "Retry-After" in response_headers: + response_headers["retry-after"] = response_headers["Retry-After"] + if "Retry-After" not in response_headers and "retry-after" in response_headers: + response_headers["Retry-After"] = response_headers["retry-after"] + + # read the response body here + charset = e.headers.get_content_charset() or "utf-8" + response_body: str = e.read().decode(charset) + resp["body"] = response_body + + # Try to find a retry handler for this error + retry_request = RetryHttpRequest.from_urllib_http_request(req) + retry_response = RetryHttpResponse( + status_code=e.code, + headers={k: [v] for k, v in response_headers.items()}, + data=response_body.encode("utf-8") if response_body is not None else None, + ) + for handler in self.retry_handlers: + if handler.can_retry( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ): + if self._logger.level <= logging.DEBUG: + self._logger.info( + f"A retry handler found: {type(handler).__name__} for {req.method} {req.full_url} - {e}" + ) + handler.prepare_for_next_attempt( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ) + break + + if retry_state.next_attempt_requested is False: + return resp + + except Exception as err: + last_error = err + self._logger.error(f"Failed to send a request to Slack API server: {err}") + + # Try to find a retry handler for this error + retry_request = RetryHttpRequest.from_urllib_http_request(req) + for handler in self.retry_handlers: + if handler.can_retry( + state=retry_state, + request=retry_request, + response=None, + error=err, + ): + if self._logger.level <= logging.DEBUG: + self._logger.info( + f"A retry handler found: {type(handler).__name__} for {req.method} {req.full_url} - {err}" + ) + handler.prepare_for_next_attempt( + state=retry_state, + request=retry_request, + response=None, + error=err, + ) + self._logger.info(f"Going to retry the same request: {req.method} {req.full_url}") + break + + if retry_state.next_attempt_requested is False: + raise err + + if resp is not None: + return resp + raise last_error + + def _perform_urllib_http_request_internal( + self, + url: str, + req: Request, + ) -> Dict[str, Any]: + # urllib not only opens http:// or https:// URLs, but also ftp:// and file://. + # With this it might be possible to open local files on the executing machine + # which might be a security risk if the URL to open can be manipulated by an external user. + # (BAN-B310) + if url.lower().startswith("http"): + opener: Optional[OpenerDirector] = None + if self.proxy is not None: + if isinstance(self.proxy, str): + opener = urllib.request.build_opener( + ProxyHandler({"http": self.proxy, "https": self.proxy}), + HTTPSHandler(context=self.ssl), + ) + else: + raise SlackRequestError(f"Invalid proxy detected: {self.proxy} must be a str value") + + # NOTE: BAN-B310 is already checked above + resp: Optional[HTTPResponse] = None + if opener: + resp = opener.open(req, timeout=self.timeout) # skipcq: BAN-B310 + else: + resp = urlopen(req, context=self.ssl, timeout=self.timeout) # skipcq: BAN-B310 + if resp.headers.get_content_type() == "application/gzip": + # admin.analytics.getFile + body: bytes = resp.read() + if self._logger.level <= logging.DEBUG: + self._logger.debug( + "Received the following response - " + f"status: {resp.code}, " + f"headers: {dict(resp.headers)}, " + f"body: (binary)" + ) + return {"status": resp.code, "headers": resp.headers, "body": body} + + charset = resp.headers.get_content_charset() or "utf-8" + body: str = resp.read().decode(charset) # read the response body here + if self._logger.level <= logging.DEBUG: + self._logger.debug( + "Received the following response - " + f"status: {resp.code}, " + f"headers: {dict(resp.headers)}, " + f"body: {body}" + ) + return {"status": resp.code, "headers": resp.headers, "body": body} + raise SlackRequestError(f"Invalid URL detected: {url}") + + def _build_urllib_request_headers( + self, token: str, has_json: bool, has_files: bool, additional_headers: dict + ) -> Dict[str, str]: + headers = {"Content-Type": "application/x-www-form-urlencoded"} + headers.update(self.headers) + if token: + headers.update({"Authorization": "Bearer {}".format(token)}) + if additional_headers: + headers.update(additional_headers) + if has_json: + headers.update({"Content-Type": "application/json;charset=utf-8"}) + if has_files: + # will be set afterwards + headers.pop("Content-Type", None) + return headers + + # ================================================================= + + @staticmethod + def validate_slack_signature(*, signing_secret: str, data: str, timestamp: str, signature: str) -> bool: + """ + Slack creates a unique string for your app and shares it with you. Verify + requests from Slack with confidence by verifying signatures using your + signing secret. + + On each HTTP request that Slack sends, we add an X-Slack-Signature HTTP + header. The signature is created by combining the signing secret with the + body of the request we're sending using a standard HMAC-SHA256 keyed hash. + + https://api.slack.com/docs/verifying-requests-from-slack#how_to_make_a_request_signature_in_4_easy_steps__an_overview + + Args: + signing_secret: Your application's signing secret, available in the + Slack API dashboard + data: The raw body of the incoming request - no headers, just the body. + timestamp: from the 'X-Slack-Request-Timestamp' header + signature: from the 'X-Slack-Signature' header - the calculated signature + should match this. + + Returns: + True if signatures matches + """ + warnings.warn( + "As this method is deprecated since slackclient 2.6.0, " + "use `from slack.signature import SignatureVerifier` instead", + DeprecationWarning, + ) + format_req = str.encode(f"v0:{timestamp}:{data}") + encoded_secret = str.encode(signing_secret) + request_hash = hmac.new(encoded_secret, format_req, hashlib.sha256).hexdigest() + calculated_signature = f"v0={request_hash}" + return hmac.compare_digest(calculated_signature, signature) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/web/client.py b/core_service/aws_lambda/project/packages/slack_sdk/web/client.py new file mode 100644 index 0000000..5da8fed --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/web/client.py @@ -0,0 +1,4407 @@ +"""A Python module for interacting with Slack's Web API.""" +import json +import os +from io import IOBase +from typing import Union, Sequence, Optional, Dict, Tuple, Any, List + +import slack_sdk.errors as e +from slack_sdk.models.views import View +from .base_client import BaseClient, SlackResponse +from .internal_utils import ( + _parse_web_class_objects, + _update_call_participants, + _warn_if_text_or_attachment_fallback_is_missing, + _remove_none_values, +) +from ..models.attachments import Attachment +from ..models.blocks import Block +from ..models.metadata import Metadata + + +class WebClient(BaseClient): + """A WebClient allows apps to communicate with the Slack Platform's Web API. + + https://api.slack.com/methods + + The Slack Web API is an interface for querying information from + and enacting change in a Slack workspace. + + This client handles constructing and sending HTTP requests to Slack + as well as parsing any responses received into a `SlackResponse`. + + Attributes: + token (str): A string specifying an `xoxp-*` or `xoxb-*` token. + base_url (str): A string representing the Slack API base URL. + Default is `'https://www.slack.com/api/'` + timeout (int): The maximum number of seconds the client will wait + to connect and receive a response from Slack. + Default is 30 seconds. + ssl (SSLContext): An [`ssl.SSLContext`][1] instance, helpful for specifying + your own custom certificate chain. + proxy (str): String representing a fully-qualified URL to a proxy through + which to route all requests to the Slack API. Even if this parameter + is not specified, if any of the following environment variables are + present, they will be loaded into this parameter: `HTTPS_PROXY`, + `https_proxy`, `HTTP_PROXY` or `http_proxy`. + headers (dict): Additional request headers to attach to all requests. + + Methods: + `api_call`: Constructs a request and executes the API call to Slack. + + Example of recommended usage: + ```python + import os + from slack_sdk import WebClient + + client = WebClient(token=os.environ['SLACK_API_TOKEN']) + response = client.chat_postMessage( + channel='#random', + text="Hello world!") + assert response["ok"] + assert response["message"]["text"] == "Hello world!" + ``` + + Example manually creating an API request: + ```python + import os + from slack_sdk import WebClient + + client = WebClient(token=os.environ['SLACK_API_TOKEN']) + response = client.api_call( + api_method='chat.postMessage', + json={'channel': '#random','text': "Hello world!"} + ) + assert response["ok"] + assert response["message"]["text"] == "Hello world!" + ``` + + Note: + Any attributes or methods prefixed with _underscores are + intended to be "private" internal use only. They may be changed or + removed at anytime. + + [1]: https://docs.python.org/3/library/ssl.html#ssl.SSLContext + """ + + def admin_analytics_getFile( + self, + *, + type: str, + date: Optional[str] = None, + metadata_only: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Retrieve analytics data for a given date, presented as a compressed JSON file + https://api.slack.com/methods/admin.analytics.getFile + """ + kwargs.update({"type": type}) + if date is not None: + kwargs.update({"date": date}) + if metadata_only is not None: + kwargs.update({"metadata_only": metadata_only}) + return self.api_call("admin.analytics.getFile", params=kwargs) + + def admin_apps_approve( + self, + *, + app_id: Optional[str] = None, + request_id: Optional[str] = None, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Approve an app for installation on a workspace. + Either app_id or request_id is required. + These IDs can be obtained either directly via the app_requested event, + or by the admin.apps.requests.list method. + https://api.slack.com/methods/admin.apps.approve + """ + if app_id: + kwargs.update({"app_id": app_id}) + elif request_id: + kwargs.update({"request_id": request_id}) + else: + raise e.SlackRequestError("The app_id or request_id argument must be specified.") + + kwargs.update( + { + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return self.api_call("admin.apps.approve", params=kwargs) + + def admin_apps_approved_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """List approved apps for an org or workspace. + https://api.slack.com/methods/admin.apps.approved.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return self.api_call("admin.apps.approved.list", http_verb="GET", params=kwargs) + + def admin_apps_clearResolution( + self, + *, + app_id: str, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Clear an app resolution + https://api.slack.com/methods/admin.apps.clearResolution + """ + kwargs.update( + { + "app_id": app_id, + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return self.api_call("admin.apps.clearResolution", http_verb="POST", params=kwargs) + + def admin_apps_requests_cancel( + self, + *, + request_id: str, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """List app requests for a team/workspace. + https://api.slack.com/methods/admin.apps.requests.cancel + """ + kwargs.update( + { + "request_id": request_id, + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return self.api_call("admin.apps.requests.cancel", http_verb="POST", params=kwargs) + + def admin_apps_requests_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """List app requests for a team/workspace. + https://api.slack.com/methods/admin.apps.requests.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + } + ) + return self.api_call("admin.apps.requests.list", http_verb="GET", params=kwargs) + + def admin_apps_restrict( + self, + *, + app_id: Optional[str] = None, + request_id: Optional[str] = None, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Restrict an app for installation on a workspace. + Exactly one of the team_id or enterprise_id arguments is required, not both. + Either app_id or request_id is required. These IDs can be obtained either directly + via the app_requested event, or by the admin.apps.requests.list method. + https://api.slack.com/methods/admin.apps.restrict + """ + if app_id: + kwargs.update({"app_id": app_id}) + elif request_id: + kwargs.update({"request_id": request_id}) + else: + raise e.SlackRequestError("The app_id or request_id argument must be specified.") + + kwargs.update( + { + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return self.api_call("admin.apps.restrict", params=kwargs) + + def admin_apps_restricted_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """List restricted apps for an org or workspace. + https://api.slack.com/methods/admin.apps.restricted.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return self.api_call("admin.apps.restricted.list", http_verb="GET", params=kwargs) + + def admin_apps_uninstall( + self, + *, + app_id: str, + enterprise_id: Optional[str] = None, + team_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> SlackResponse: + """Uninstall an app from one or many workspaces, or an entire enterprise organization. + With an org-level token, enterprise_id or team_ids is required. + https://api.slack.com/methods/admin.apps.uninstall + """ + kwargs.update({"app_id": app_id}) + if enterprise_id is not None: + kwargs.update({"enterprise_id": enterprise_id}) + if team_ids is not None: + if isinstance(team_ids, (list, Tuple)): + kwargs.update({"team_ids": ",".join(team_ids)}) + else: + kwargs.update({"team_ids": team_ids}) + return self.api_call("admin.apps.uninstall", http_verb="POST", params=kwargs) + + def admin_auth_policy_getEntities( + self, + *, + policy_name: str, + cursor: Optional[str] = None, + entity_type: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> SlackResponse: + """Fetch all the entities assigned to a particular authentication policy by name. + https://api.slack.com/methods/admin.auth.policy.getEntities + """ + kwargs.update({"policy_name": policy_name}) + if cursor is not None: + kwargs.update({"cursor": cursor}) + if entity_type is not None: + kwargs.update({"entity_type": entity_type}) + if limit is not None: + kwargs.update({"limit": limit}) + return self.api_call("admin.auth.policy.getEntities", http_verb="POST", params=kwargs) + + def admin_auth_policy_assignEntities( + self, + *, + entity_ids: Union[str, Sequence[str]], + policy_name: str, + entity_type: str, + **kwargs, + ) -> SlackResponse: + """Assign entities to a particular authentication policy. + https://api.slack.com/methods/admin.auth.policy.assignEntities + """ + if isinstance(entity_ids, (list, Tuple)): + kwargs.update({"entity_ids": ",".join(entity_ids)}) + else: + kwargs.update({"entity_ids": entity_ids}) + kwargs.update({"policy_name": policy_name}) + kwargs.update({"entity_type": entity_type}) + return self.api_call("admin.auth.policy.assignEntities", http_verb="POST", params=kwargs) + + def admin_auth_policy_removeEntities( + self, + *, + entity_ids: Union[str, Sequence[str]], + policy_name: str, + entity_type: str, + **kwargs, + ) -> SlackResponse: + """Remove specified entities from a specified authentication policy. + https://api.slack.com/methods/admin.auth.policy.removeEntities + """ + if isinstance(entity_ids, (list, Tuple)): + kwargs.update({"entity_ids": ",".join(entity_ids)}) + else: + kwargs.update({"entity_ids": entity_ids}) + kwargs.update({"policy_name": policy_name}) + kwargs.update({"entity_type": entity_type}) + return self.api_call("admin.auth.policy.removeEntities", http_verb="POST", params=kwargs) + + def admin_barriers_create( + self, + *, + barriered_from_usergroup_ids: Union[str, Sequence[str]], + primary_usergroup_id: str, + restricted_subjects: Union[str, Sequence[str]], + **kwargs, + ) -> SlackResponse: + """Create an Information Barrier + https://api.slack.com/methods/admin.barriers.create + """ + kwargs.update({"primary_usergroup_id": primary_usergroup_id}) + if isinstance(barriered_from_usergroup_ids, (list, Tuple)): + kwargs.update({"barriered_from_usergroup_ids": ",".join(barriered_from_usergroup_ids)}) + else: + kwargs.update({"barriered_from_usergroup_ids": barriered_from_usergroup_ids}) + if isinstance(restricted_subjects, (list, Tuple)): + kwargs.update({"restricted_subjects": ",".join(restricted_subjects)}) + else: + kwargs.update({"restricted_subjects": restricted_subjects}) + return self.api_call("admin.barriers.create", http_verb="POST", params=kwargs) + + def admin_barriers_delete( + self, + *, + barrier_id: str, + **kwargs, + ) -> SlackResponse: + """Delete an existing Information Barrier + https://api.slack.com/methods/admin.barriers.delete + """ + kwargs.update({"barrier_id": barrier_id}) + return self.api_call("admin.barriers.delete", http_verb="POST", params=kwargs) + + def admin_barriers_update( + self, + *, + barrier_id: str, + barriered_from_usergroup_ids: Union[str, Sequence[str]], + primary_usergroup_id: str, + restricted_subjects: Union[str, Sequence[str]], + **kwargs, + ) -> SlackResponse: + """Update an existing Information Barrier + https://api.slack.com/methods/admin.barriers.update + """ + kwargs.update({"barrier_id": barrier_id, "primary_usergroup_id": primary_usergroup_id}) + if isinstance(barriered_from_usergroup_ids, (list, Tuple)): + kwargs.update({"barriered_from_usergroup_ids": ",".join(barriered_from_usergroup_ids)}) + else: + kwargs.update({"barriered_from_usergroup_ids": barriered_from_usergroup_ids}) + if isinstance(restricted_subjects, (list, Tuple)): + kwargs.update({"restricted_subjects": ",".join(restricted_subjects)}) + else: + kwargs.update({"restricted_subjects": restricted_subjects}) + return self.api_call("admin.barriers.update", http_verb="POST", params=kwargs) + + def admin_barriers_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> SlackResponse: + """Get all Information Barriers for your organization + https://api.slack.com/methods/admin.barriers.list""" + kwargs.update( + { + "cursor": cursor, + "limit": limit, + } + ) + return self.api_call("admin.barriers.list", http_verb="GET", params=kwargs) + + def admin_conversations_create( + self, + *, + is_private: bool, + name: str, + description: Optional[str] = None, + org_wide: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Create a public or private channel-based conversation. + https://api.slack.com/methods/admin.conversations.create + """ + kwargs.update( + { + "is_private": is_private, + "name": name, + "description": description, + "org_wide": org_wide, + "team_id": team_id, + } + ) + return self.api_call("admin.conversations.create", params=kwargs) + + def admin_conversations_delete( + self, + *, + channel_id: str, + **kwargs, + ) -> SlackResponse: + """Delete a public or private channel. + https://api.slack.com/methods/admin.conversations.delete + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("admin.conversations.delete", params=kwargs) + + def admin_conversations_invite( + self, + *, + channel_id: str, + user_ids: Union[str, Sequence[str]], + **kwargs, + ) -> SlackResponse: + """Invite a user to a public or private channel. + https://api.slack.com/methods/admin.conversations.invite + """ + kwargs.update({"channel_id": channel_id}) + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + # NOTE: the endpoint is unable to handle Content-Type: application/json as of Sep 3, 2020. + return self.api_call("admin.conversations.invite", params=kwargs) + + def admin_conversations_archive( + self, + *, + channel_id: str, + **kwargs, + ) -> SlackResponse: + """Archive a public or private channel. + https://api.slack.com/methods/admin.conversations.archive + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("admin.conversations.archive", params=kwargs) + + def admin_conversations_unarchive( + self, + *, + channel_id: str, + **kwargs, + ) -> SlackResponse: + """Unarchive a public or private channel. + https://api.slack.com/methods/admin.conversations.archive + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("admin.conversations.unarchive", params=kwargs) + + def admin_conversations_rename( + self, + *, + channel_id: str, + name: str, + **kwargs, + ) -> SlackResponse: + """Rename a public or private channel. + https://api.slack.com/methods/admin.conversations.rename + """ + kwargs.update({"channel_id": channel_id, "name": name}) + return self.api_call("admin.conversations.rename", params=kwargs) + + def admin_conversations_search( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + query: Optional[str] = None, + search_channel_types: Optional[Union[str, Sequence[str]]] = None, + sort: Optional[str] = None, + sort_dir: Optional[str] = None, + team_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> SlackResponse: + """Search for public or private channels in an Enterprise organization. + https://api.slack.com/methods/admin.conversations.search + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "query": query, + "sort": sort, + "sort_dir": sort_dir, + } + ) + + if isinstance(search_channel_types, (list, Tuple)): + kwargs.update({"search_channel_types": ",".join(search_channel_types)}) + else: + kwargs.update({"search_channel_types": search_channel_types}) + + if isinstance(team_ids, (list, Tuple)): + kwargs.update({"team_ids": ",".join(team_ids)}) + else: + kwargs.update({"team_ids": team_ids}) + + return self.api_call("admin.conversations.search", params=kwargs) + + def admin_conversations_convertToPrivate( + self, + *, + channel_id: str, + **kwargs, + ) -> SlackResponse: + """Convert a public channel to a private channel. + https://api.slack.com/methods/admin.conversations.convertToPrivate + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("admin.conversations.convertToPrivate", params=kwargs) + + def admin_conversations_setConversationPrefs( + self, + *, + channel_id: str, + prefs: Union[str, Dict[str, str]], + **kwargs, + ) -> SlackResponse: + """Set the posting permissions for a public or private channel. + https://api.slack.com/methods/admin.conversations.setConversationPrefs + """ + kwargs.update({"channel_id": channel_id}) + if isinstance(prefs, dict): + kwargs.update({"prefs": json.dumps(prefs)}) + else: + kwargs.update({"prefs": prefs}) + return self.api_call("admin.conversations.setConversationPrefs", params=kwargs) + + def admin_conversations_getConversationPrefs( + self, + *, + channel_id: str, + **kwargs, + ) -> SlackResponse: + """Get conversation preferences for a public or private channel. + https://api.slack.com/methods/admin.conversations.getConversationPrefs + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("admin.conversations.getConversationPrefs", params=kwargs) + + def admin_conversations_disconnectShared( + self, + *, + channel_id: str, + leaving_team_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> SlackResponse: + """Disconnect a connected channel from one or more workspaces. + https://api.slack.com/methods/admin.conversations.disconnectShared + """ + kwargs.update({"channel_id": channel_id}) + if isinstance(leaving_team_ids, (list, Tuple)): + kwargs.update({"leaving_team_ids": ",".join(leaving_team_ids)}) + else: + kwargs.update({"leaving_team_ids": leaving_team_ids}) + return self.api_call("admin.conversations.disconnectShared", params=kwargs) + + def admin_conversations_ekm_listOriginalConnectedChannelInfo( + self, + *, + channel_ids: Optional[Union[str, Sequence[str]]] = None, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> SlackResponse: + """List all disconnected channels—i.e., + channels that were once connected to other workspaces and then disconnected—and + the corresponding original channel IDs for key revocation with EKM. + https://api.slack.com/methods/admin.conversations.ekm.listOriginalConnectedChannelInfo + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + } + ) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + if isinstance(team_ids, (list, Tuple)): + kwargs.update({"team_ids": ",".join(team_ids)}) + else: + kwargs.update({"team_ids": team_ids}) + return self.api_call("admin.conversations.ekm.listOriginalConnectedChannelInfo", params=kwargs) + + def admin_conversations_restrictAccess_addGroup( + self, + *, + channel_id: str, + group_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Add an allowlist of IDP groups for accessing a channel. + https://api.slack.com/methods/admin.conversations.restrictAccess.addGroup + """ + kwargs.update( + { + "channel_id": channel_id, + "group_id": group_id, + "team_id": team_id, + } + ) + return self.api_call( + "admin.conversations.restrictAccess.addGroup", + http_verb="GET", + params=kwargs, + ) + + def admin_conversations_restrictAccess_listGroups( + self, + *, + channel_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """List all IDP Groups linked to a channel. + https://api.slack.com/methods/admin.conversations.restrictAccess.listGroups + """ + kwargs.update( + { + "channel_id": channel_id, + "team_id": team_id, + } + ) + return self.api_call( + "admin.conversations.restrictAccess.listGroups", + http_verb="GET", + params=kwargs, + ) + + def admin_conversations_restrictAccess_removeGroup( + self, + *, + channel_id: str, + group_id: str, + team_id: str, + **kwargs, + ) -> SlackResponse: + """Remove a linked IDP group linked from a private channel. + https://api.slack.com/methods/admin.conversations.restrictAccess.removeGroup + """ + kwargs.update( + { + "channel_id": channel_id, + "group_id": group_id, + "team_id": team_id, + } + ) + return self.api_call( + "admin.conversations.restrictAccess.removeGroup", + http_verb="GET", + params=kwargs, + ) + + def admin_conversations_setTeams( + self, + *, + channel_id: str, + org_channel: Optional[bool] = None, + target_team_ids: Optional[Union[str, Sequence[str]]] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Set the workspaces in an Enterprise grid org that connect to a public or private channel. + https://api.slack.com/methods/admin.conversations.setTeams + """ + kwargs.update( + { + "channel_id": channel_id, + "org_channel": org_channel, + "team_id": team_id, + } + ) + if isinstance(target_team_ids, (list, Tuple)): + kwargs.update({"target_team_ids": ",".join(target_team_ids)}) + else: + kwargs.update({"target_team_ids": target_team_ids}) + return self.api_call("admin.conversations.setTeams", params=kwargs) + + def admin_conversations_getTeams( + self, + *, + channel_id: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> SlackResponse: + """Set the workspaces in an Enterprise grid org that connect to a channel. + https://api.slack.com/methods/admin.conversations.getTeams + """ + kwargs.update( + { + "channel_id": channel_id, + "cursor": cursor, + "limit": limit, + } + ) + return self.api_call("admin.conversations.getTeams", params=kwargs) + + def admin_conversations_getCustomRetention( + self, + *, + channel_id: str, + **kwargs, + ) -> SlackResponse: + """Get a channel's retention policy + https://api.slack.com/methods/admin.conversations.getCustomRetention + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("admin.conversations.getCustomRetention", params=kwargs) + + def admin_conversations_removeCustomRetention( + self, + *, + channel_id: str, + **kwargs, + ) -> SlackResponse: + """Remove a channel's retention policy + https://api.slack.com/methods/admin.conversations.removeCustomRetention + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("admin.conversations.removeCustomRetention", params=kwargs) + + def admin_conversations_setCustomRetention( + self, + *, + channel_id: str, + duration_days: int, + **kwargs, + ) -> SlackResponse: + """Set a channel's retention policy + https://api.slack.com/methods/admin.conversations.setCustomRetention + """ + kwargs.update({"channel_id": channel_id, "duration_days": duration_days}) + return self.api_call("admin.conversations.setCustomRetention", params=kwargs) + + def admin_emoji_add( + self, + *, + name: str, + url: str, + **kwargs, + ) -> SlackResponse: + """Add an emoji. + https://api.slack.com/methods/admin.emoji.add + """ + kwargs.update({"name": name, "url": url}) + return self.api_call("admin.emoji.add", http_verb="GET", params=kwargs) + + def admin_emoji_addAlias( + self, + *, + alias_for: str, + name: str, + **kwargs, + ) -> SlackResponse: + """Add an emoji alias. + https://api.slack.com/methods/admin.emoji.addAlias + """ + kwargs.update({"alias_for": alias_for, "name": name}) + return self.api_call("admin.emoji.addAlias", http_verb="GET", params=kwargs) + + def admin_emoji_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> SlackResponse: + """List emoji for an Enterprise Grid organization. + https://api.slack.com/methods/admin.emoji.list + """ + kwargs.update({"cursor": cursor, "limit": limit}) + return self.api_call("admin.emoji.list", http_verb="GET", params=kwargs) + + def admin_emoji_remove( + self, + *, + name: str, + **kwargs, + ) -> SlackResponse: + """Remove an emoji across an Enterprise Grid organization. + https://api.slack.com/methods/admin.emoji.remove + """ + kwargs.update({"name": name}) + return self.api_call("admin.emoji.remove", http_verb="GET", params=kwargs) + + def admin_emoji_rename( + self, + *, + name: str, + new_name: str, + **kwargs, + ) -> SlackResponse: + """Rename an emoji. + https://api.slack.com/methods/admin.emoji.rename + """ + kwargs.update({"name": name, "new_name": new_name}) + return self.api_call("admin.emoji.rename", http_verb="GET", params=kwargs) + + def admin_users_session_reset( + self, + *, + user_id: str, + mobile_only: Optional[bool] = None, + web_only: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Wipes all valid sessions on all devices for a given user. + https://api.slack.com/methods/admin.users.session.reset + """ + kwargs.update( + { + "user_id": user_id, + "mobile_only": mobile_only, + "web_only": web_only, + } + ) + return self.api_call("admin.users.session.reset", params=kwargs) + + def admin_users_session_resetBulk( + self, + *, + user_ids: Union[str, Sequence[str]], + mobile_only: Optional[bool] = None, + web_only: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Enqueues an asynchronous job to wipe all valid sessions on all devices for a given list of users + https://api.slack.com/methods/admin.users.session.resetBulk + """ + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + kwargs.update( + { + "mobile_only": mobile_only, + "web_only": web_only, + } + ) + return self.api_call("admin.users.session.resetBulk", params=kwargs) + + def admin_users_session_invalidate( + self, + *, + session_id: str, + team_id: str, + **kwargs, + ) -> SlackResponse: + """Invalidate a single session for a user by session_id. + https://api.slack.com/methods/admin.users.session.invalidate + """ + kwargs.update({"session_id": session_id, "team_id": team_id}) + return self.api_call("admin.users.session.invalidate", params=kwargs) + + def admin_users_session_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + user_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Lists all active user sessions for an organization + https://api.slack.com/methods/admin.users.session.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + "user_id": user_id, + } + ) + return self.api_call("admin.users.session.list", params=kwargs) + + def admin_teams_settings_setDefaultChannels( + self, + *, + team_id: str, + channel_ids: Union[str, Sequence[str]], + **kwargs, + ) -> SlackResponse: + """Set the default channels of a workspace. + https://api.slack.com/methods/admin.teams.settings.setDefaultChannels + """ + kwargs.update({"team_id": team_id}) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return self.api_call("admin.teams.settings.setDefaultChannels", http_verb="GET", params=kwargs) + + def admin_users_session_getSettings( + self, + *, + user_ids: Union[str, Sequence[str]], + **kwargs, + ) -> SlackResponse: + """Get user-specific session settings—the session duration + and what happens when the client closes—given a list of users. + https://api.slack.com/methods/admin.users.session.getSettings + """ + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + return self.api_call("admin.users.session.getSettings", params=kwargs) + + def admin_users_session_setSettings( + self, + *, + user_ids: Union[str, Sequence[str]], + desktop_app_browser_quit: Optional[bool] = None, + duration: Optional[int] = None, + **kwargs, + ) -> SlackResponse: + """Configure the user-level session settings—the session duration + and what happens when the client closes—for one or more users. + https://api.slack.com/methods/admin.users.session.setSettings + """ + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + kwargs.update( + { + "desktop_app_browser_quit": desktop_app_browser_quit, + "duration": duration, + } + ) + return self.api_call("admin.users.session.setSettings", params=kwargs) + + def admin_users_session_clearSettings( + self, + *, + user_ids: Union[str, Sequence[str]], + **kwargs, + ) -> SlackResponse: + """Clear user-specific session settings—the session duration + and what happens when the client closes—for a list of users. + https://api.slack.com/methods/admin.users.session.clearSettings + """ + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + return self.api_call("admin.users.session.clearSettings", params=kwargs) + + def admin_users_unsupportedVersions_export( + self, + *, + date_end_of_support: Optional[Union[str, int]] = None, + date_sessions_started: Optional[Union[str, int]] = None, + **kwargs, + ) -> SlackResponse: + """Ask Slackbot to send you an export listing all workspace members using unsupported software, + presented as a zipped CSV file. + https://api.slack.com/methods/admin.users.unsupportedVersions.export + """ + kwargs.update( + { + "date_end_of_support": date_end_of_support, + "date_sessions_started": date_sessions_started, + } + ) + return self.api_call("admin.users.unsupportedVersions.export", params=kwargs) + + def admin_inviteRequests_approve( + self, + *, + invite_request_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Approve a workspace invite request. + https://api.slack.com/methods/admin.inviteRequests.approve + """ + kwargs.update({"invite_request_id": invite_request_id, "team_id": team_id}) + return self.api_call("admin.inviteRequests.approve", params=kwargs) + + def admin_inviteRequests_approved_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """List all approved workspace invite requests. + https://api.slack.com/methods/admin.inviteRequests.approved.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + } + ) + return self.api_call("admin.inviteRequests.approved.list", params=kwargs) + + def admin_inviteRequests_denied_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """List all denied workspace invite requests. + https://api.slack.com/methods/admin.inviteRequests.denied.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + } + ) + return self.api_call("admin.inviteRequests.denied.list", params=kwargs) + + def admin_inviteRequests_deny( + self, + *, + invite_request_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Deny a workspace invite request. + https://api.slack.com/methods/admin.inviteRequests.deny + """ + kwargs.update({"invite_request_id": invite_request_id, "team_id": team_id}) + return self.api_call("admin.inviteRequests.deny", params=kwargs) + + def admin_inviteRequests_list( + self, + **kwargs, + ) -> SlackResponse: + """List all pending workspace invite requests.""" + return self.api_call("admin.inviteRequests.list", params=kwargs) + + def admin_teams_admins_list( + self, + *, + team_id: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> SlackResponse: + """List all of the admins on a given workspace. + https://api.slack.com/methods/admin.inviteRequests.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + } + ) + return self.api_call("admin.teams.admins.list", http_verb="GET", params=kwargs) + + def admin_teams_create( + self, + *, + team_domain: str, + team_name: str, + team_description: Optional[str] = None, + team_discoverability: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Create an Enterprise team. + https://api.slack.com/methods/admin.teams.create + """ + kwargs.update( + { + "team_domain": team_domain, + "team_name": team_name, + "team_description": team_description, + "team_discoverability": team_discoverability, + } + ) + return self.api_call("admin.teams.create", params=kwargs) + + def admin_teams_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> SlackResponse: + """List all teams on an Enterprise organization. + https://api.slack.com/methods/admin.teams.list + """ + kwargs.update({"cursor": cursor, "limit": limit}) + return self.api_call("admin.teams.list", params=kwargs) + + def admin_teams_owners_list( + self, + *, + team_id: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> SlackResponse: + """List all of the admins on a given workspace. + https://api.slack.com/methods/admin.teams.owners.list + """ + kwargs.update({"team_id": team_id, "cursor": cursor, "limit": limit}) + return self.api_call("admin.teams.owners.list", http_verb="GET", params=kwargs) + + def admin_teams_settings_info( + self, + *, + team_id: str, + **kwargs, + ) -> SlackResponse: + """Fetch information about settings in a workspace + https://api.slack.com/methods/admin.teams.settings.info + """ + kwargs.update({"team_id": team_id}) + return self.api_call("admin.teams.settings.info", params=kwargs) + + def admin_teams_settings_setDescription( + self, + *, + team_id: str, + description: str, + **kwargs, + ) -> SlackResponse: + """Set the description of a given workspace. + https://api.slack.com/methods/admin.teams.settings.setDescription + """ + kwargs.update({"team_id": team_id, "description": description}) + return self.api_call("admin.teams.settings.setDescription", params=kwargs) + + def admin_teams_settings_setDiscoverability( + self, + *, + team_id: str, + discoverability: str, + **kwargs, + ) -> SlackResponse: + """Sets the icon of a workspace. + https://api.slack.com/methods/admin.teams.settings.setDiscoverability + """ + kwargs.update({"team_id": team_id, "discoverability": discoverability}) + return self.api_call("admin.teams.settings.setDiscoverability", params=kwargs) + + def admin_teams_settings_setIcon( + self, + *, + team_id: str, + image_url: str, + **kwargs, + ) -> SlackResponse: + """Sets the icon of a workspace. + https://api.slack.com/methods/admin.teams.settings.setIcon + """ + kwargs.update({"team_id": team_id, "image_url": image_url}) + return self.api_call("admin.teams.settings.setIcon", http_verb="GET", params=kwargs) + + def admin_teams_settings_setName( + self, + *, + team_id: str, + name: str, + **kwargs, + ) -> SlackResponse: + """Sets the icon of a workspace. + https://api.slack.com/methods/admin.teams.settings.setName + """ + kwargs.update({"team_id": team_id, "name": name}) + return self.api_call("admin.teams.settings.setName", params=kwargs) + + def admin_usergroups_addChannels( + self, + *, + channel_ids: Union[str, Sequence[str]], + usergroup_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Add one or more default channels to an IDP group. + https://api.slack.com/methods/admin.usergroups.addChannels + """ + kwargs.update({"team_id": team_id, "usergroup_id": usergroup_id}) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return self.api_call("admin.usergroups.addChannels", params=kwargs) + + def admin_usergroups_addTeams( + self, + *, + usergroup_id: str, + team_ids: Union[str, Sequence[str]], + auto_provision: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Associate one or more default workspaces with an organization-wide IDP group. + https://api.slack.com/methods/admin.usergroups.addTeams + """ + kwargs.update({"usergroup_id": usergroup_id, "auto_provision": auto_provision}) + if isinstance(team_ids, (list, Tuple)): + kwargs.update({"team_ids": ",".join(team_ids)}) + else: + kwargs.update({"team_ids": team_ids}) + return self.api_call("admin.usergroups.addTeams", params=kwargs) + + def admin_usergroups_listChannels( + self, + *, + usergroup_id: str, + include_num_members: Optional[bool] = None, + team_id: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Add one or more default channels to an IDP group. + https://api.slack.com/methods/admin.usergroups.listChannels + """ + kwargs.update( + { + "usergroup_id": usergroup_id, + "include_num_members": include_num_members, + "team_id": team_id, + } + ) + return self.api_call("admin.usergroups.listChannels", params=kwargs) + + def admin_usergroups_removeChannels( + self, + *, + usergroup_id: str, + channel_ids: Union[str, Sequence[str]], + **kwargs, + ) -> SlackResponse: + """Add one or more default channels to an IDP group. + https://api.slack.com/methods/admin.usergroups.removeChannels + """ + kwargs.update({"usergroup_id": usergroup_id}) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return self.api_call("admin.usergroups.removeChannels", params=kwargs) + + def admin_users_assign( + self, + *, + team_id: str, + user_id: str, + channel_ids: Optional[Union[str, Sequence[str]]] = None, + is_restricted: Optional[bool] = None, + is_ultra_restricted: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Add an Enterprise user to a workspace. + https://api.slack.com/methods/admin.users.assign + """ + kwargs.update( + { + "team_id": team_id, + "user_id": user_id, + "is_restricted": is_restricted, + "is_ultra_restricted": is_ultra_restricted, + } + ) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return self.api_call("admin.users.assign", params=kwargs) + + def admin_users_invite( + self, + *, + team_id: str, + email: str, + channel_ids: Union[str, Sequence[str]], + custom_message: Optional[str] = None, + email_password_policy_enabled: Optional[bool] = None, + guest_expiration_ts: Optional[Union[str, float]] = None, + is_restricted: Optional[bool] = None, + is_ultra_restricted: Optional[bool] = None, + real_name: Optional[str] = None, + resend: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Invite a user to a workspace. + https://api.slack.com/methods/admin.users.invite + """ + kwargs.update( + { + "team_id": team_id, + "email": email, + "custom_message": custom_message, + "email_password_policy_enabled": email_password_policy_enabled, + "guest_expiration_ts": str(guest_expiration_ts) if guest_expiration_ts is not None else None, + "is_restricted": is_restricted, + "is_ultra_restricted": is_ultra_restricted, + "real_name": real_name, + "resend": resend, + } + ) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return self.api_call("admin.users.invite", params=kwargs) + + def admin_users_list( + self, + *, + team_id: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> SlackResponse: + """List users on a workspace + https://api.slack.com/methods/admin.users.list + """ + kwargs.update( + { + "team_id": team_id, + "cursor": cursor, + "limit": limit, + } + ) + return self.api_call("admin.users.list", params=kwargs) + + def admin_users_remove( + self, + *, + team_id: str, + user_id: str, + **kwargs, + ) -> SlackResponse: + """Remove a user from a workspace. + https://api.slack.com/methods/admin.users.remove + """ + kwargs.update({"team_id": team_id, "user_id": user_id}) + return self.api_call("admin.users.remove", params=kwargs) + + def admin_users_setAdmin( + self, + *, + team_id: str, + user_id: str, + **kwargs, + ) -> SlackResponse: + """Set an existing guest, regular user, or owner to be an admin user. + https://api.slack.com/methods/admin.users.setAdmin + """ + kwargs.update({"team_id": team_id, "user_id": user_id}) + return self.api_call("admin.users.setAdmin", params=kwargs) + + def admin_users_setExpiration( + self, + *, + expiration_ts: int, + user_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Set an expiration for a guest user. + https://api.slack.com/methods/admin.users.setExpiration + """ + kwargs.update({"expiration_ts": expiration_ts, "team_id": team_id, "user_id": user_id}) + return self.api_call("admin.users.setExpiration", params=kwargs) + + def admin_users_setOwner( + self, + *, + team_id: str, + user_id: str, + **kwargs, + ) -> SlackResponse: + """Set an existing guest, regular user, or admin user to be a workspace owner. + https://api.slack.com/methods/admin.users.setOwner + """ + kwargs.update({"team_id": team_id, "user_id": user_id}) + return self.api_call("admin.users.setOwner", params=kwargs) + + def admin_users_setRegular( + self, + *, + team_id: str, + user_id: str, + **kwargs, + ) -> SlackResponse: + """Set an existing guest user, admin user, or owner to be a regular user. + https://api.slack.com/methods/admin.users.setRegular + """ + kwargs.update({"team_id": team_id, "user_id": user_id}) + return self.api_call("admin.users.setRegular", params=kwargs) + + def api_test( + self, + *, + error: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Checks API calling code. + https://api.slack.com/methods/api.test + """ + kwargs.update({"error": error}) + return self.api_call("api.test", params=kwargs) + + def apps_connections_open( + self, + *, + app_token: str, + **kwargs, + ) -> SlackResponse: + """Generate a temporary Socket Mode WebSocket URL that your app can connect to + in order to receive events and interactive payloads + https://api.slack.com/methods/apps.connections.open + """ + kwargs.update({"token": app_token}) + return self.api_call("apps.connections.open", http_verb="POST", params=kwargs) + + def apps_event_authorizations_list( + self, + *, + event_context: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> SlackResponse: + """Get a list of authorizations for the given event context. + Each authorization represents an app installation that the event is visible to. + https://api.slack.com/methods/apps.event.authorizations.list + """ + kwargs.update({"event_context": event_context, "cursor": cursor, "limit": limit}) + return self.api_call("apps.event.authorizations.list", params=kwargs) + + def apps_uninstall( + self, + *, + client_id: str, + client_secret: str, + **kwargs, + ) -> SlackResponse: + """Uninstalls your app from a workspace. + https://api.slack.com/methods/apps.uninstall + """ + kwargs.update({"client_id": client_id, "client_secret": client_secret}) + return self.api_call("apps.uninstall", params=kwargs) + + def auth_revoke( + self, + *, + test: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Revokes a token. + https://api.slack.com/methods/auth.revoke + """ + kwargs.update({"test": test}) + return self.api_call("auth.revoke", http_verb="GET", params=kwargs) + + def auth_test( + self, + **kwargs, + ) -> SlackResponse: + """Checks authentication & identity. + https://api.slack.com/methods/auth.test + """ + return self.api_call("auth.test", params=kwargs) + + def auth_teams_list( + self, + cursor: Optional[str] = None, + limit: Optional[int] = None, + include_icon: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """List the workspaces a token can access. + https://api.slack.com/methods/auth.teams.list + """ + kwargs.update({"cursor": cursor, "limit": limit, "include_icon": include_icon}) + return self.api_call("auth.teams.list", params=kwargs) + + def bookmarks_add( + self, + *, + channel_id: str, + title: str, + type: str, + emoji: Optional[str] = None, + entity_id: Optional[str] = None, + link: Optional[str] = None, # include when type is 'link' + parent_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Add bookmark to a channel. + https://api.slack.com/methods/bookmarks.add + """ + kwargs.update( + { + "channel_id": channel_id, + "title": title, + "type": type, + "emoji": emoji, + "entity_id": entity_id, + "link": link, + "parent_id": parent_id, + } + ) + return self.api_call("bookmarks.add", http_verb="POST", params=kwargs) + + def bookmarks_edit( + self, + *, + bookmark_id: str, + channel_id: str, + emoji: Optional[str] = None, + link: Optional[str] = None, + title: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Edit bookmark. + https://api.slack.com/methods/bookmarks.edit + """ + kwargs.update( + { + "bookmark_id": bookmark_id, + "channel_id": channel_id, + "emoji": emoji, + "link": link, + "title": title, + } + ) + return self.api_call("bookmarks.edit", http_verb="POST", params=kwargs) + + def bookmarks_list( + self, + *, + channel_id: str, + **kwargs, + ) -> SlackResponse: + """List bookmark for the channel. + https://api.slack.com/methods/bookmarks.list + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("bookmarks.list", http_verb="POST", params=kwargs) + + def bookmarks_remove( + self, + *, + bookmark_id: str, + channel_id: str, + **kwargs, + ) -> SlackResponse: + """Remove bookmark from the channel. + https://api.slack.com/methods/bookmarks.remove + """ + kwargs.update({"bookmark_id": bookmark_id, "channel_id": channel_id}) + return self.api_call("bookmarks.remove", http_verb="POST", params=kwargs) + + def bots_info( + self, + *, + bot: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Gets information about a bot user. + https://api.slack.com/methods/bots.info + """ + kwargs.update({"bot": bot, "team_id": team_id}) + return self.api_call("bots.info", http_verb="GET", params=kwargs) + + def calls_add( + self, + *, + external_unique_id: str, + join_url: str, + created_by: Optional[str] = None, + date_start: Optional[int] = None, + desktop_app_join_url: Optional[str] = None, + external_display_id: Optional[str] = None, + title: Optional[str] = None, + users: Optional[Union[str, Sequence[Dict[str, str]]]] = None, + **kwargs, + ) -> SlackResponse: + """Registers a new Call. + https://api.slack.com/methods/calls.add + """ + kwargs.update( + { + "external_unique_id": external_unique_id, + "join_url": join_url, + "created_by": created_by, + "date_start": date_start, + "desktop_app_join_url": desktop_app_join_url, + "external_display_id": external_display_id, + "title": title, + } + ) + _update_call_participants( # skipcq: PTC-W0039 + kwargs, + users if users is not None else kwargs.get("users"), # skipcq: PTC-W0039 + ) # skipcq: PTC-W0039 + return self.api_call("calls.add", http_verb="POST", params=kwargs) + + def calls_end( + self, + *, + id: str, + duration: Optional[int] = None, + **kwargs, + ) -> SlackResponse: # skipcq: PYL-W0622 + """Ends a Call. + https://api.slack.com/methods/calls.end + """ + kwargs.update({"id": id, "duration": duration}) + return self.api_call("calls.end", http_verb="POST", params=kwargs) + + def calls_info( + self, + *, + id: str, + **kwargs, + ) -> SlackResponse: # skipcq: PYL-W0622 + """Returns information about a Call. + https://api.slack.com/methods/calls.info + """ + kwargs.update({"id": id}) + return self.api_call("calls.info", http_verb="POST", params=kwargs) + + def calls_participants_add( + self, + *, + id: str, # skipcq: PYL-W0622 + users: Union[str, Sequence[Dict[str, str]]], + **kwargs, + ) -> SlackResponse: + """Registers new participants added to a Call. + https://api.slack.com/methods/calls.participants.add + """ + kwargs.update({"id": id}) + _update_call_participants(kwargs, users) + return self.api_call("calls.participants.add", http_verb="POST", params=kwargs) + + def calls_participants_remove( + self, + *, + id: str, # skipcq: PYL-W0622 + users: Union[str, Sequence[Dict[str, str]]], + **kwargs, + ) -> SlackResponse: + """Registers participants removed from a Call. + https://api.slack.com/methods/calls.participants.remove + """ + kwargs.update({"id": id}) + _update_call_participants(kwargs, users) + return self.api_call("calls.participants.remove", http_verb="POST", params=kwargs) + + def calls_update( + self, + *, + id: str, + desktop_app_join_url: Optional[str] = None, + join_url: Optional[str] = None, + title: Optional[str] = None, + **kwargs, + ) -> SlackResponse: # skipcq: PYL-W0622 + """Updates information about a Call. + https://api.slack.com/methods/calls.update + """ + kwargs.update( + { + "id": id, + "desktop_app_join_url": desktop_app_join_url, + "join_url": join_url, + "title": title, + } + ) + return self.api_call("calls.update", http_verb="POST", params=kwargs) + + # -------------------------- + # Deprecated: channels.* + # You can use conversations.* APIs instead. + # https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api + # -------------------------- + + def channels_archive( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Archives a channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.archive", json=kwargs) + + def channels_create( + self, + *, + name: str, + **kwargs, + ) -> SlackResponse: + """Creates a channel.""" + kwargs.update({"name": name}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.create", json=kwargs) + + def channels_history( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Fetches history of messages and events from a channel.""" + kwargs.update({"channel": channel}) + return self.api_call("channels.history", http_verb="GET", params=kwargs) + + def channels_info( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Gets information about a channel.""" + kwargs.update({"channel": channel}) + return self.api_call("channels.info", http_verb="GET", params=kwargs) + + def channels_invite( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> SlackResponse: + """Invites a user to a channel.""" + kwargs.update({"channel": channel, "user": user}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.invite", json=kwargs) + + def channels_join( + self, + *, + name: str, + **kwargs, + ) -> SlackResponse: + """Joins a channel, creating it if needed.""" + kwargs.update({"name": name}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.join", json=kwargs) + + def channels_kick( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> SlackResponse: + """Removes a user from a channel.""" + kwargs.update({"channel": channel, "user": user}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.kick", json=kwargs) + + def channels_leave( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Leaves a channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.leave", json=kwargs) + + def channels_list( + self, + **kwargs, + ) -> SlackResponse: + """Lists all channels in a Slack team.""" + return self.api_call("channels.list", http_verb="GET", params=kwargs) + + def channels_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> SlackResponse: + """Sets the read cursor in a channel.""" + kwargs.update({"channel": channel, "ts": ts}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.mark", json=kwargs) + + def channels_rename( + self, + *, + channel: str, + name: str, + **kwargs, + ) -> SlackResponse: + """Renames a channel.""" + kwargs.update({"channel": channel, "name": name}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.rename", json=kwargs) + + def channels_replies( + self, + *, + channel: str, + thread_ts: str, + **kwargs, + ) -> SlackResponse: + """Retrieve a thread of messages posted to a channel""" + kwargs.update({"channel": channel, "thread_ts": thread_ts}) + return self.api_call("channels.replies", http_verb="GET", params=kwargs) + + def channels_setPurpose( + self, + *, + channel: str, + purpose: str, + **kwargs, + ) -> SlackResponse: + """Sets the purpose for a channel.""" + kwargs.update({"channel": channel, "purpose": purpose}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.setPurpose", json=kwargs) + + def channels_setTopic( + self, + *, + channel: str, + topic: str, + **kwargs, + ) -> SlackResponse: + """Sets the topic for a channel.""" + kwargs.update({"channel": channel, "topic": topic}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.setTopic", json=kwargs) + + def channels_unarchive( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Unarchives a channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.unarchive", json=kwargs) + + # -------------------------- + + def chat_delete( + self, + *, + channel: str, + ts: str, + as_user: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Deletes a message. + https://api.slack.com/methods/chat.delete + """ + kwargs.update({"channel": channel, "ts": ts, "as_user": as_user}) + return self.api_call("chat.delete", params=kwargs) + + def chat_deleteScheduledMessage( + self, + *, + channel: str, + scheduled_message_id: str, + as_user: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Deletes a scheduled message. + https://api.slack.com/methods/chat.deleteScheduledMessage + """ + kwargs.update( + { + "channel": channel, + "scheduled_message_id": scheduled_message_id, + "as_user": as_user, + } + ) + return self.api_call("chat.deleteScheduledMessage", params=kwargs) + + def chat_getPermalink( + self, + *, + channel: str, + message_ts: str, + **kwargs, + ) -> SlackResponse: + """Retrieve a permalink URL for a specific extant message + https://api.slack.com/methods/chat.getPermalink + """ + kwargs.update({"channel": channel, "message_ts": message_ts}) + return self.api_call("chat.getPermalink", http_verb="GET", params=kwargs) + + def chat_meMessage( + self, + *, + channel: str, + text: str, + **kwargs, + ) -> SlackResponse: + """Share a me message into a channel. + https://api.slack.com/methods/chat.meMessage + """ + kwargs.update({"channel": channel, "text": text}) + return self.api_call("chat.meMessage", params=kwargs) + + def chat_postEphemeral( + self, + *, + channel: str, + user: str, + text: Optional[str] = None, + as_user: Optional[bool] = None, + attachments: Optional[Sequence[Union[Dict, Attachment]]] = None, + blocks: Optional[Sequence[Union[Dict, Block]]] = None, + thread_ts: Optional[str] = None, + icon_emoji: Optional[str] = None, + icon_url: Optional[str] = None, + link_names: Optional[bool] = None, + username: Optional[str] = None, + parse: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Sends an ephemeral message to a user in a channel. + https://api.slack.com/methods/chat.postEphemeral + """ + kwargs.update( + { + "channel": channel, + "user": user, + "text": text, + "as_user": as_user, + "attachments": attachments, + "blocks": blocks, + "thread_ts": thread_ts, + "icon_emoji": icon_emoji, + "icon_url": icon_url, + "link_names": link_names, + "username": username, + "parse": parse, + } + ) + _parse_web_class_objects(kwargs) + kwargs = _remove_none_values(kwargs) + _warn_if_text_or_attachment_fallback_is_missing("chat.postEphemeral", kwargs) + # NOTE: intentionally using json over params for the API methods using blocks/attachments + return self.api_call("chat.postEphemeral", json=kwargs) + + def chat_postMessage( + self, + *, + channel: str, + text: Optional[str] = None, + as_user: Optional[bool] = None, + attachments: Optional[Sequence[Union[Dict, Attachment]]] = None, + blocks: Optional[Sequence[Union[Dict, Block]]] = None, + thread_ts: Optional[str] = None, + reply_broadcast: Optional[bool] = None, + unfurl_links: Optional[bool] = None, + unfurl_media: Optional[bool] = None, + container_id: Optional[str] = None, + file_annotation: Optional[str] = None, + icon_emoji: Optional[str] = None, + icon_url: Optional[str] = None, + mrkdwn: Optional[bool] = None, + link_names: Optional[bool] = None, + username: Optional[str] = None, + parse: Optional[str] = None, # none, full + metadata: Optional[Union[Dict, Metadata]] = None, + **kwargs, + ) -> SlackResponse: + """Sends a message to a channel. + https://api.slack.com/methods/chat.postMessage + """ + kwargs.update( + { + "channel": channel, + "text": text, + "as_user": as_user, + "attachments": attachments, + "blocks": blocks, + "thread_ts": thread_ts, + "reply_broadcast": reply_broadcast, + "unfurl_links": unfurl_links, + "unfurl_media": unfurl_media, + "container_id": container_id, + "file_annotation": file_annotation, + "icon_emoji": icon_emoji, + "icon_url": icon_url, + "mrkdwn": mrkdwn, + "link_names": link_names, + "username": username, + "parse": parse, + "metadata": metadata, + } + ) + _parse_web_class_objects(kwargs) + kwargs = _remove_none_values(kwargs) + _warn_if_text_or_attachment_fallback_is_missing("chat.postMessage", kwargs) + # NOTE: intentionally using json over params for the API methods using blocks/attachments + return self.api_call("chat.postMessage", json=kwargs) + + def chat_scheduleMessage( + self, + *, + channel: str, + post_at: Union[str, int], + text: str, + as_user: Optional[bool] = None, + attachments: Optional[Sequence[Union[Dict, Attachment]]] = None, + blocks: Optional[Sequence[Union[Dict, Block]]] = None, + thread_ts: Optional[str] = None, + parse: Optional[str] = None, + reply_broadcast: Optional[bool] = None, + unfurl_links: Optional[bool] = None, + unfurl_media: Optional[bool] = None, + link_names: Optional[bool] = None, + metadata: Optional[Union[Dict, Metadata]] = None, + **kwargs, + ) -> SlackResponse: + """Schedules a message. + https://api.slack.com/methods/chat.scheduleMessage + """ + kwargs.update( + { + "channel": channel, + "post_at": post_at, + "text": text, + "as_user": as_user, + "attachments": attachments, + "blocks": blocks, + "thread_ts": thread_ts, + "reply_broadcast": reply_broadcast, + "parse": parse, + "unfurl_links": unfurl_links, + "unfurl_media": unfurl_media, + "link_names": link_names, + "metadata": metadata, + } + ) + _parse_web_class_objects(kwargs) + kwargs = _remove_none_values(kwargs) + _warn_if_text_or_attachment_fallback_is_missing("chat.scheduleMessage", kwargs) + # NOTE: intentionally using json over params for the API methods using blocks/attachments + return self.api_call("chat.scheduleMessage", json=kwargs) + + def chat_unfurl( + self, + *, + channel: str, + ts: str, + unfurls: Dict[str, Dict], + user_auth_blocks: Optional[Sequence[Union[Dict, Block]]] = None, + user_auth_message: Optional[str] = None, + user_auth_required: Optional[bool] = None, + user_auth_url: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Provide custom unfurl behavior for user-posted URLs. + https://api.slack.com/methods/chat.unfurl + """ + kwargs.update( + { + "channel": channel, + "ts": ts, + "unfurls": unfurls, + "user_auth_blocks": user_auth_blocks, + "user_auth_message": user_auth_message, + "user_auth_required": user_auth_required, + "user_auth_url": user_auth_url, + } + ) + kwargs = _remove_none_values(kwargs) + # NOTE: intentionally using json over params for API methods using blocks/attachments + return self.api_call("chat.unfurl", json=kwargs) + + def chat_update( + self, + *, + channel: str, + ts: str, + text: Optional[str] = None, + attachments: Optional[Sequence[Union[Dict, Attachment]]] = None, + blocks: Optional[Sequence[Union[Dict, Block]]] = None, + as_user: Optional[bool] = None, + file_ids: Optional[Union[str, Sequence[str]]] = None, + link_names: Optional[bool] = None, + parse: Optional[str] = None, # none, full + reply_broadcast: Optional[bool] = None, + metadata: Optional[Union[Dict, Metadata]] = None, + **kwargs, + ) -> SlackResponse: + """Updates a message in a channel. + https://api.slack.com/methods/chat.update + """ + kwargs.update( + { + "channel": channel, + "ts": ts, + "text": text, + "attachments": attachments, + "blocks": blocks, + "as_user": as_user, + "link_names": link_names, + "parse": parse, + "reply_broadcast": reply_broadcast, + "metadata": metadata, + } + ) + if isinstance(file_ids, (list, Tuple)): + kwargs.update({"file_ids": ",".join(file_ids)}) + else: + kwargs.update({"file_ids": file_ids}) + _parse_web_class_objects(kwargs) + kwargs = _remove_none_values(kwargs) + _warn_if_text_or_attachment_fallback_is_missing("chat.update", kwargs) + # NOTE: intentionally using json over params for API methods using blocks/attachments + return self.api_call("chat.update", json=kwargs) + + def chat_scheduledMessages_list( + self, + *, + channel: Optional[str] = None, + cursor: Optional[str] = None, + latest: Optional[str] = None, + limit: Optional[int] = None, + oldest: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Lists all scheduled messages. + https://api.slack.com/methods/chat.scheduledMessages.list + """ + kwargs.update( + { + "channel": channel, + "cursor": cursor, + "latest": latest, + "limit": limit, + "oldest": oldest, + "team_id": team_id, + } + ) + return self.api_call("chat.scheduledMessages.list", params=kwargs) + + def conversations_acceptSharedInvite( + self, + *, + channel_name: str, + channel_id: Optional[str] = None, + invite_id: Optional[str] = None, + free_trial_accepted: Optional[bool] = None, + is_private: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Accepts an invitation to a Slack Connect channel. + https://api.slack.com/methods/conversations.acceptSharedInvite + """ + if channel_id is None and invite_id is None: + raise e.SlackRequestError("Either channel_id or invite_id must be provided.") + kwargs.update( + { + "channel_name": channel_name, + "channel_id": channel_id, + "invite_id": invite_id, + "free_trial_accepted": free_trial_accepted, + "is_private": is_private, + "team_id": team_id, + } + ) + return self.api_call("conversations.acceptSharedInvite", http_verb="POST", params=kwargs) + + def conversations_approveSharedInvite( + self, + *, + invite_id: str, + target_team: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Approves an invitation to a Slack Connect channel. + https://api.slack.com/methods/conversations.approveSharedInvite + """ + kwargs.update({"invite_id": invite_id, "target_team": target_team}) + return self.api_call("conversations.approveSharedInvite", http_verb="POST", params=kwargs) + + def conversations_archive( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Archives a conversation. + https://api.slack.com/methods/conversations.archive + """ + kwargs.update({"channel": channel}) + return self.api_call("conversations.archive", params=kwargs) + + def conversations_close( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Closes a direct message or multi-person direct message. + https://api.slack.com/methods/conversations.close + """ + kwargs.update({"channel": channel}) + return self.api_call("conversations.close", params=kwargs) + + def conversations_create( + self, + *, + name: str, + is_private: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Initiates a public or private channel-based conversation + https://api.slack.com/methods/conversations.create + """ + kwargs.update({"name": name, "is_private": is_private, "team_id": team_id}) + return self.api_call("conversations.create", params=kwargs) + + def conversations_declineSharedInvite( + self, + *, + invite_id: str, + target_team: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Declines a Slack Connect channel invite. + https://api.slack.com/methods/conversations.declineSharedInvite + """ + kwargs.update({"invite_id": invite_id, "target_team": target_team}) + return self.api_call("conversations.declineSharedInvite", http_verb="GET", params=kwargs) + + def conversations_history( + self, + *, + channel: str, + cursor: Optional[str] = None, + inclusive: Optional[bool] = None, + include_all_metadata: Optional[bool] = None, + latest: Optional[str] = None, + limit: Optional[int] = None, + oldest: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Fetches a conversation's history of messages and events. + https://api.slack.com/methods/conversations.history + """ + kwargs.update( + { + "channel": channel, + "cursor": cursor, + "inclusive": inclusive, + "include_all_metadata": include_all_metadata, + "limit": limit, + "latest": latest, + "oldest": oldest, + } + ) + return self.api_call("conversations.history", http_verb="GET", params=kwargs) + + def conversations_info( + self, + *, + channel: str, + include_locale: Optional[bool] = None, + include_num_members: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Retrieve information about a conversation. + https://api.slack.com/methods/conversations.info + """ + kwargs.update( + { + "channel": channel, + "include_locale": include_locale, + "include_num_members": include_num_members, + } + ) + return self.api_call("conversations.info", http_verb="GET", params=kwargs) + + def conversations_invite( + self, + *, + channel: str, + users: Union[str, Sequence[str]], + **kwargs, + ) -> SlackResponse: + """Invites users to a channel. + https://api.slack.com/methods/conversations.invite + """ + kwargs.update({"channel": channel}) + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + return self.api_call("conversations.invite", params=kwargs) + + def conversations_inviteShared( + self, + *, + channel: str, + emails: Optional[Union[str, Sequence[str]]] = None, + user_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> SlackResponse: + """Sends an invitation to a Slack Connect channel. + https://api.slack.com/methods/conversations.inviteShared + """ + if emails is None and user_ids is None: + raise e.SlackRequestError("Either emails or user ids must be provided.") + kwargs.update({"channel": channel}) + if isinstance(emails, (list, Tuple)): + kwargs.update({"emails": ",".join(emails)}) + else: + kwargs.update({"emails": emails}) + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + return self.api_call("conversations.inviteShared", http_verb="GET", params=kwargs) + + def conversations_join( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Joins an existing conversation. + https://api.slack.com/methods/conversations.join + """ + kwargs.update({"channel": channel}) + return self.api_call("conversations.join", params=kwargs) + + def conversations_kick( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> SlackResponse: + """Removes a user from a conversation. + https://api.slack.com/methods/conversations.kick + """ + kwargs.update({"channel": channel, "user": user}) + return self.api_call("conversations.kick", params=kwargs) + + def conversations_leave( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Leaves a conversation. + https://api.slack.com/methods/conversations.leave + """ + kwargs.update({"channel": channel}) + return self.api_call("conversations.leave", params=kwargs) + + def conversations_list( + self, + *, + cursor: Optional[str] = None, + exclude_archived: Optional[bool] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + types: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> SlackResponse: + """Lists all channels in a Slack team. + https://api.slack.com/methods/conversations.list + """ + kwargs.update( + { + "cursor": cursor, + "exclude_archived": exclude_archived, + "limit": limit, + "team_id": team_id, + } + ) + if isinstance(types, (list, Tuple)): + kwargs.update({"types": ",".join(types)}) + else: + kwargs.update({"types": types}) + return self.api_call("conversations.list", http_verb="GET", params=kwargs) + + def conversations_listConnectInvites( + self, + *, + count: Optional[int] = None, + cursor: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """List shared channel invites that have been generated + or received but have not yet been approved by all parties. + https://api.slack.com/methods/conversations.listConnectInvites + """ + kwargs.update({"count": count, "cursor": cursor, "team_id": team_id}) + return self.api_call("conversations.listConnectInvites", params=kwargs) + + def conversations_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> SlackResponse: + """Sets the read cursor in a channel. + https://api.slack.com/methods/conversations.mark + """ + kwargs.update({"channel": channel, "ts": ts}) + return self.api_call("conversations.mark", params=kwargs) + + def conversations_members( + self, + *, + channel: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> SlackResponse: + """Retrieve members of a conversation. + https://api.slack.com/methods/conversations.members + """ + kwargs.update({"channel": channel, "cursor": cursor, "limit": limit}) + return self.api_call("conversations.members", http_verb="GET", params=kwargs) + + def conversations_open( + self, + *, + channel: Optional[str] = None, + return_im: Optional[bool] = None, + users: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> SlackResponse: + """Opens or resumes a direct message or multi-person direct message. + https://api.slack.com/methods/conversations.open + """ + if channel is None and users is None: + raise e.SlackRequestError("Either channel or users must be provided.") + kwargs.update({"channel": channel, "return_im": return_im}) + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + return self.api_call("conversations.open", params=kwargs) + + def conversations_rename( + self, + *, + channel: str, + name: str, + **kwargs, + ) -> SlackResponse: + """Renames a conversation. + https://api.slack.com/methods/conversations.rename + """ + kwargs.update({"channel": channel, "name": name}) + return self.api_call("conversations.rename", params=kwargs) + + def conversations_replies( + self, + *, + channel: str, + ts: str, + cursor: Optional[str] = None, + inclusive: Optional[bool] = None, + latest: Optional[str] = None, + limit: Optional[int] = None, + oldest: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Retrieve a thread of messages posted to a conversation + https://api.slack.com/methods/conversations.replies + """ + kwargs.update( + { + "channel": channel, + "ts": ts, + "cursor": cursor, + "inclusive": inclusive, + "limit": limit, + "latest": latest, + "oldest": oldest, + } + ) + return self.api_call("conversations.replies", http_verb="GET", params=kwargs) + + def conversations_setPurpose( + self, + *, + channel: str, + purpose: str, + **kwargs, + ) -> SlackResponse: + """Sets the purpose for a conversation. + https://api.slack.com/methods/conversations.setPurpose + """ + kwargs.update({"channel": channel, "purpose": purpose}) + return self.api_call("conversations.setPurpose", params=kwargs) + + def conversations_setTopic( + self, + *, + channel: str, + topic: str, + **kwargs, + ) -> SlackResponse: + """Sets the topic for a conversation. + https://api.slack.com/methods/conversations.setTopic + """ + kwargs.update({"channel": channel, "topic": topic}) + return self.api_call("conversations.setTopic", params=kwargs) + + def conversations_unarchive( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Reverses conversation archival. + https://api.slack.com/methods/conversations.unarchive + """ + kwargs.update({"channel": channel}) + return self.api_call("conversations.unarchive", params=kwargs) + + def dialog_open( + self, + *, + dialog: Dict[str, Any], + trigger_id: str, + **kwargs, + ) -> SlackResponse: + """Open a dialog with a user. + https://api.slack.com/methods/dialog.open + """ + kwargs.update({"dialog": dialog, "trigger_id": trigger_id}) + kwargs = _remove_none_values(kwargs) + # NOTE: As the dialog can be a dict, this API call works only with json format. + return self.api_call("dialog.open", json=kwargs) + + def dnd_endDnd( + self, + **kwargs, + ) -> SlackResponse: + """Ends the current user's Do Not Disturb session immediately. + https://api.slack.com/methods/dnd.endDnd + """ + return self.api_call("dnd.endDnd", params=kwargs) + + def dnd_endSnooze( + self, + **kwargs, + ) -> SlackResponse: + """Ends the current user's snooze mode immediately. + https://api.slack.com/methods/dnd.endSnooze + """ + return self.api_call("dnd.endSnooze", params=kwargs) + + def dnd_info( + self, + *, + team_id: Optional[str] = None, + user: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Retrieves a user's current Do Not Disturb status. + https://api.slack.com/methods/dnd.info + """ + kwargs.update({"team_id": team_id, "user": user}) + return self.api_call("dnd.info", http_verb="GET", params=kwargs) + + def dnd_setSnooze( + self, + *, + num_minutes: Union[int, str], + **kwargs, + ) -> SlackResponse: + """Turns on Do Not Disturb mode for the current user, or changes its duration. + https://api.slack.com/methods/dnd.setSnooze + """ + kwargs.update({"num_minutes": num_minutes}) + return self.api_call("dnd.setSnooze", http_verb="GET", params=kwargs) + + def dnd_teamInfo( + self, + users: Union[str, Sequence[str]], + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Retrieves the Do Not Disturb status for users on a team. + https://api.slack.com/methods/dnd.teamInfo + """ + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + kwargs.update({"team_id": team_id}) + return self.api_call("dnd.teamInfo", http_verb="GET", params=kwargs) + + def emoji_list( + self, + **kwargs, + ) -> SlackResponse: + """Lists custom emoji for a team. + https://api.slack.com/methods/emoji.list + """ + return self.api_call("emoji.list", http_verb="GET", params=kwargs) + + def files_comments_delete( + self, + *, + file: str, + id: str, + **kwargs, # skipcq: PYL-W0622 + ) -> SlackResponse: + """Deletes an existing comment on a file. + https://api.slack.com/methods/files.comments.delete + """ + kwargs.update({"file": file, "id": id}) + return self.api_call("files.comments.delete", params=kwargs) + + def files_delete( + self, + *, + file: str, + **kwargs, + ) -> SlackResponse: + """Deletes a file. + https://api.slack.com/methods/files.delete + """ + kwargs.update({"file": file}) + return self.api_call("files.delete", params=kwargs) + + def files_info( + self, + *, + file: str, + count: Optional[int] = None, + cursor: Optional[str] = None, + limit: Optional[int] = None, + page: Optional[int] = None, + **kwargs, + ) -> SlackResponse: + """Gets information about a team file. + https://api.slack.com/methods/files.info + """ + kwargs.update( + { + "file": file, + "count": count, + "cursor": cursor, + "limit": limit, + "page": page, + } + ) + return self.api_call("files.info", http_verb="GET", params=kwargs) + + def files_list( + self, + *, + channel: Optional[str] = None, + count: Optional[int] = None, + page: Optional[int] = None, + show_files_hidden_by_limit: Optional[bool] = None, + team_id: Optional[str] = None, + ts_from: Optional[str] = None, + ts_to: Optional[str] = None, + types: Optional[Union[str, Sequence[str]]] = None, + user: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Lists & filters team files. + https://api.slack.com/methods/files.list + """ + kwargs.update( + { + "channel": channel, + "count": count, + "page": page, + "show_files_hidden_by_limit": show_files_hidden_by_limit, + "team_id": team_id, + "ts_from": ts_from, + "ts_to": ts_to, + "user": user, + } + ) + if isinstance(types, (list, Tuple)): + kwargs.update({"types": ",".join(types)}) + else: + kwargs.update({"types": types}) + return self.api_call("files.list", http_verb="GET", params=kwargs) + + def files_remote_info( + self, + *, + external_id: Optional[str] = None, + file: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Retrieve information about a remote file added to Slack. + https://api.slack.com/methods/files.remote.info + """ + kwargs.update({"external_id": external_id, "file": file}) + return self.api_call("files.remote.info", http_verb="GET", params=kwargs) + + def files_remote_list( + self, + *, + channel: Optional[str] = None, + cursor: Optional[str] = None, + limit: Optional[int] = None, + ts_from: Optional[str] = None, + ts_to: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Retrieve information about a remote file added to Slack. + https://api.slack.com/methods/files.remote.list + """ + kwargs.update( + { + "channel": channel, + "cursor": cursor, + "limit": limit, + "ts_from": ts_from, + "ts_to": ts_to, + } + ) + return self.api_call("files.remote.list", http_verb="GET", params=kwargs) + + def files_remote_add( + self, + *, + external_id: str, + external_url: str, + title: str, + filetype: Optional[str] = None, + indexable_file_contents: Optional[Union[str, bytes, IOBase]] = None, + preview_image: Optional[Union[str, bytes, IOBase]] = None, + **kwargs, + ) -> SlackResponse: + """Adds a file from a remote service. + https://api.slack.com/methods/files.remote.add + """ + kwargs.update( + { + "external_id": external_id, + "external_url": external_url, + "title": title, + "filetype": filetype, + } + ) + files = None + # preview_image (file): Preview of the document via multipart/form-data. + if preview_image is not None or indexable_file_contents is not None: + files = { + "preview_image": preview_image, + "indexable_file_contents": indexable_file_contents, + } + + return self.api_call( + # Intentionally using "POST" method over "GET" here + "files.remote.add", + http_verb="POST", + data=kwargs, + files=files, + ) + + def files_remote_update( + self, + *, + external_id: Optional[str] = None, + external_url: Optional[str] = None, + file: Optional[str] = None, + title: Optional[str] = None, + filetype: Optional[str] = None, + indexable_file_contents: Optional[str] = None, + preview_image: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Updates an existing remote file. + https://api.slack.com/methods/files.remote.update + """ + kwargs.update( + { + "external_id": external_id, + "external_url": external_url, + "file": file, + "title": title, + "filetype": filetype, + } + ) + files = None + # preview_image (file): Preview of the document via multipart/form-data. + if preview_image is not None or indexable_file_contents is not None: + files = { + "preview_image": preview_image, + "indexable_file_contents": indexable_file_contents, + } + + return self.api_call( + # Intentionally using "POST" method over "GET" here + "files.remote.update", + http_verb="POST", + data=kwargs, + files=files, + ) + + def files_remote_remove( + self, + *, + external_id: Optional[str] = None, + file: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Remove a remote file. + https://api.slack.com/methods/files.remote.remove + """ + kwargs.update({"external_id": external_id, "file": file}) + return self.api_call("files.remote.remove", http_verb="POST", params=kwargs) + + def files_remote_share( + self, + *, + channels: Union[str, Sequence[str]], + external_id: Optional[str] = None, + file: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Share a remote file into a channel. + https://api.slack.com/methods/files.remote.share + """ + if external_id is None and file is None: + raise e.SlackRequestError("Either external_id or file must be provided.") + if isinstance(channels, (list, Tuple)): + kwargs.update({"channels": ",".join(channels)}) + else: + kwargs.update({"channels": channels}) + kwargs.update({"external_id": external_id, "file": file}) + return self.api_call("files.remote.share", http_verb="GET", params=kwargs) + + def files_revokePublicURL( + self, + *, + file: str, + **kwargs, + ) -> SlackResponse: + """Revokes public/external sharing access for a file + https://api.slack.com/methods/files.revokePublicURL + """ + kwargs.update({"file": file}) + return self.api_call("files.revokePublicURL", params=kwargs) + + def files_sharedPublicURL( + self, + *, + file: str, + **kwargs, + ) -> SlackResponse: + """Enables a file for public/external sharing. + https://api.slack.com/methods/files.sharedPublicURL + """ + kwargs.update({"file": file}) + return self.api_call("files.sharedPublicURL", params=kwargs) + + def files_upload( + self, + *, + file: Optional[Union[str, bytes, IOBase]] = None, + content: Optional[str] = None, + filename: Optional[str] = None, + filetype: Optional[str] = None, + initial_comment: Optional[str] = None, + thread_ts: Optional[str] = None, + title: Optional[str] = None, + channels: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> SlackResponse: + """Uploads or creates a file. + https://api.slack.com/methods/files.upload + """ + if file is None and content is None: + raise e.SlackRequestError("The file or content argument must be specified.") + if file is not None and content is not None: + raise e.SlackRequestError("You cannot specify both the file and the content argument.") + + if isinstance(channels, (list, Tuple)): + kwargs.update({"channels": ",".join(channels)}) + else: + kwargs.update({"channels": channels}) + kwargs.update( + { + "filename": filename, + "filetype": filetype, + "initial_comment": initial_comment, + "thread_ts": thread_ts, + "title": title, + } + ) + if file: + if kwargs.get("filename") is None and isinstance(file, str): + # use the local filename if filename is missing + if kwargs.get("filename") is None: + kwargs["filename"] = file.split(os.path.sep)[-1] + return self.api_call("files.upload", files={"file": file}, data=kwargs) + else: + kwargs["content"] = content + return self.api_call("files.upload", data=kwargs) + + # -------------------------- + # Deprecated: groups.* + # You can use conversations.* APIs instead. + # https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api + # -------------------------- + + def groups_archive( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Archives a private channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.archive", json=kwargs) + + def groups_create( + self, + *, + name: str, + **kwargs, + ) -> SlackResponse: + """Creates a private channel.""" + kwargs.update({"name": name}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.create", json=kwargs) + + def groups_createChild( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Clones and archives a private channel.""" + kwargs.update({"channel": channel}) + return self.api_call("groups.createChild", http_verb="GET", params=kwargs) + + def groups_history( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Fetches history of messages and events from a private channel.""" + kwargs.update({"channel": channel}) + return self.api_call("groups.history", http_verb="GET", params=kwargs) + + def groups_info( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Gets information about a private channel.""" + kwargs.update({"channel": channel}) + return self.api_call("groups.info", http_verb="GET", params=kwargs) + + def groups_invite( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> SlackResponse: + """Invites a user to a private channel.""" + kwargs.update({"channel": channel, "user": user}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.invite", json=kwargs) + + def groups_kick( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> SlackResponse: + """Removes a user from a private channel.""" + kwargs.update({"channel": channel, "user": user}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.kick", json=kwargs) + + def groups_leave( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Leaves a private channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.leave", json=kwargs) + + def groups_list( + self, + **kwargs, + ) -> SlackResponse: + """Lists private channels that the calling user has access to.""" + return self.api_call("groups.list", http_verb="GET", params=kwargs) + + def groups_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> SlackResponse: + """Sets the read cursor in a private channel.""" + kwargs.update({"channel": channel, "ts": ts}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.mark", json=kwargs) + + def groups_open( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Opens a private channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.open", json=kwargs) + + def groups_rename( + self, + *, + channel: str, + name: str, + **kwargs, + ) -> SlackResponse: + """Renames a private channel.""" + kwargs.update({"channel": channel, "name": name}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.rename", json=kwargs) + + def groups_replies( + self, + *, + channel: str, + thread_ts: str, + **kwargs, + ) -> SlackResponse: + """Retrieve a thread of messages posted to a private channel""" + kwargs.update({"channel": channel, "thread_ts": thread_ts}) + return self.api_call("groups.replies", http_verb="GET", params=kwargs) + + def groups_setPurpose( + self, + *, + channel: str, + purpose: str, + **kwargs, + ) -> SlackResponse: + """Sets the purpose for a private channel.""" + kwargs.update({"channel": channel, "purpose": purpose}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.setPurpose", json=kwargs) + + def groups_setTopic( + self, + *, + channel: str, + topic: str, + **kwargs, + ) -> SlackResponse: + """Sets the topic for a private channel.""" + kwargs.update({"channel": channel, "topic": topic}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.setTopic", json=kwargs) + + def groups_unarchive( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Unarchives a private channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.unarchive", json=kwargs) + + # -------------------------- + # Deprecated: im.* + # You can use conversations.* APIs instead. + # https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api + # -------------------------- + + def im_close( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Close a direct message channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("im.close", json=kwargs) + + def im_history( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Fetches history of messages and events from direct message channel.""" + kwargs.update({"channel": channel}) + return self.api_call("im.history", http_verb="GET", params=kwargs) + + def im_list( + self, + **kwargs, + ) -> SlackResponse: + """Lists direct message channels for the calling user.""" + return self.api_call("im.list", http_verb="GET", params=kwargs) + + def im_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> SlackResponse: + """Sets the read cursor in a direct message channel.""" + kwargs.update({"channel": channel, "ts": ts}) + kwargs = _remove_none_values(kwargs) + return self.api_call("im.mark", json=kwargs) + + def im_open( + self, + *, + user: str, + **kwargs, + ) -> SlackResponse: + """Opens a direct message channel.""" + kwargs.update({"user": user}) + kwargs = _remove_none_values(kwargs) + return self.api_call("im.open", json=kwargs) + + def im_replies( + self, + *, + channel: str, + thread_ts: str, + **kwargs, + ) -> SlackResponse: + """Retrieve a thread of messages posted to a direct message conversation""" + kwargs.update({"channel": channel, "thread_ts": thread_ts}) + return self.api_call("im.replies", http_verb="GET", params=kwargs) + + # -------------------------- + + def migration_exchange( + self, + *, + users: Union[str, Sequence[str]], + team_id: Optional[str] = None, + to_old: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """For Enterprise Grid workspaces, map local user IDs to global user IDs + https://api.slack.com/methods/migration.exchange + """ + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + kwargs.update({"team_id": team_id, "to_old": to_old}) + return self.api_call("migration.exchange", http_verb="GET", params=kwargs) + + # -------------------------- + # Deprecated: mpim.* + # You can use conversations.* APIs instead. + # https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api + # -------------------------- + + def mpim_close( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Closes a multiparty direct message channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("mpim.close", json=kwargs) + + def mpim_history( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Fetches history of messages and events from a multiparty direct message.""" + kwargs.update({"channel": channel}) + return self.api_call("mpim.history", http_verb="GET", params=kwargs) + + def mpim_list( + self, + **kwargs, + ) -> SlackResponse: + """Lists multiparty direct message channels for the calling user.""" + return self.api_call("mpim.list", http_verb="GET", params=kwargs) + + def mpim_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> SlackResponse: + """Sets the read cursor in a multiparty direct message channel.""" + kwargs.update({"channel": channel, "ts": ts}) + kwargs = _remove_none_values(kwargs) + return self.api_call("mpim.mark", json=kwargs) + + def mpim_open( + self, + *, + users: Union[str, Sequence[str]], + **kwargs, + ) -> SlackResponse: + """This method opens a multiparty direct message.""" + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + return self.api_call("mpim.open", params=kwargs) + + def mpim_replies( + self, + *, + channel: str, + thread_ts: str, + **kwargs, + ) -> SlackResponse: + """Retrieve a thread of messages posted to a direct message conversation from a + multiparty direct message. + """ + kwargs.update({"channel": channel, "thread_ts": thread_ts}) + return self.api_call("mpim.replies", http_verb="GET", params=kwargs) + + # -------------------------- + + def oauth_v2_access( + self, + *, + client_id: str, + client_secret: str, + # This field is required when processing the OAuth redirect URL requests + # while it's absent for token rotation + code: Optional[str] = None, + redirect_uri: Optional[str] = None, + # This field is required for token rotation + grant_type: Optional[str] = None, + # This field is required for token rotation + refresh_token: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Exchanges a temporary OAuth verifier code for an access token. + https://api.slack.com/methods/oauth.v2.access + """ + if redirect_uri is not None: + kwargs.update({"redirect_uri": redirect_uri}) + if code is not None: + kwargs.update({"code": code}) + if grant_type is not None: + kwargs.update({"grant_type": grant_type}) + if refresh_token is not None: + kwargs.update({"refresh_token": refresh_token}) + return self.api_call( + "oauth.v2.access", + data=kwargs, + auth={"client_id": client_id, "client_secret": client_secret}, + ) + + def oauth_access( + self, + *, + client_id: str, + client_secret: str, + code: str, + redirect_uri: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Exchanges a temporary OAuth verifier code for an access token. + https://api.slack.com/methods/oauth.access + """ + if redirect_uri is not None: + kwargs.update({"redirect_uri": redirect_uri}) + kwargs.update({"code": code}) + return self.api_call( + "oauth.access", + data=kwargs, + auth={"client_id": client_id, "client_secret": client_secret}, + ) + + def oauth_v2_exchange( + self, + *, + token: str, + client_id: str, + client_secret: str, + **kwargs, + ) -> SlackResponse: + """Exchanges a legacy access token for a new expiring access token and refresh token + https://api.slack.com/methods/oauth.v2.exchange + """ + kwargs.update({"client_id": client_id, "client_secret": client_secret, "token": token}) + return self.api_call("oauth.v2.exchange", params=kwargs) + + def openid_connect_token( + self, + client_id: str, + client_secret: str, + code: Optional[str] = None, + redirect_uri: Optional[str] = None, + grant_type: Optional[str] = None, + refresh_token: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Exchanges a temporary OAuth verifier code for an access token for Sign in with Slack. + https://api.slack.com/methods/openid.connect.token + """ + if redirect_uri is not None: + kwargs.update({"redirect_uri": redirect_uri}) + if code is not None: + kwargs.update({"code": code}) + if grant_type is not None: + kwargs.update({"grant_type": grant_type}) + if refresh_token is not None: + kwargs.update({"refresh_token": refresh_token}) + return self.api_call( + "openid.connect.token", + data=kwargs, + auth={"client_id": client_id, "client_secret": client_secret}, + ) + + def openid_connect_userInfo( + self, + **kwargs, + ) -> SlackResponse: + """Get the identity of a user who has authorized Sign in with Slack. + https://api.slack.com/methods/openid.connect.userInfo + """ + return self.api_call("openid.connect.userInfo", params=kwargs) + + def pins_add( + self, + *, + channel: str, + timestamp: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Pins an item to a channel. + https://api.slack.com/methods/pins.add + """ + kwargs.update({"channel": channel, "timestamp": timestamp}) + return self.api_call("pins.add", params=kwargs) + + def pins_list( + self, + *, + channel: str, + **kwargs, + ) -> SlackResponse: + """Lists items pinned to a channel. + https://api.slack.com/methods/pins.list + """ + kwargs.update({"channel": channel}) + return self.api_call("pins.list", http_verb="GET", params=kwargs) + + def pins_remove( + self, + *, + channel: str, + timestamp: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Un-pins an item from a channel. + https://api.slack.com/methods/pins.remove + """ + kwargs.update({"channel": channel, "timestamp": timestamp}) + return self.api_call("pins.remove", params=kwargs) + + def reactions_add( + self, + *, + channel: str, + name: str, + timestamp: str, + **kwargs, + ) -> SlackResponse: + """Adds a reaction to an item. + https://api.slack.com/methods/reactions.add + """ + kwargs.update({"channel": channel, "name": name, "timestamp": timestamp}) + return self.api_call("reactions.add", params=kwargs) + + def reactions_get( + self, + *, + channel: Optional[str] = None, + file: Optional[str] = None, + file_comment: Optional[str] = None, + full: Optional[bool] = None, + timestamp: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Gets reactions for an item. + https://api.slack.com/methods/reactions.get + """ + kwargs.update( + { + "channel": channel, + "file": file, + "file_comment": file_comment, + "full": full, + "timestamp": timestamp, + } + ) + return self.api_call("reactions.get", http_verb="GET", params=kwargs) + + def reactions_list( + self, + *, + count: Optional[int] = None, + cursor: Optional[str] = None, + full: Optional[bool] = None, + limit: Optional[int] = None, + page: Optional[int] = None, + team_id: Optional[str] = None, + user: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Lists reactions made by a user. + https://api.slack.com/methods/reactions.list + """ + kwargs.update( + { + "count": count, + "cursor": cursor, + "full": full, + "limit": limit, + "page": page, + "team_id": team_id, + "user": user, + } + ) + return self.api_call("reactions.list", http_verb="GET", params=kwargs) + + def reactions_remove( + self, + *, + name: str, + channel: Optional[str] = None, + file: Optional[str] = None, + file_comment: Optional[str] = None, + timestamp: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Removes a reaction from an item. + https://api.slack.com/methods/reactions.remove + """ + kwargs.update( + { + "name": name, + "channel": channel, + "file": file, + "file_comment": file_comment, + "timestamp": timestamp, + } + ) + return self.api_call("reactions.remove", params=kwargs) + + def reminders_add( + self, + *, + text: str, + time: str, + team_id: Optional[str] = None, + user: Optional[str] = None, + recurrence: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Creates a reminder. + https://api.slack.com/methods/reminders.add + """ + kwargs.update( + { + "text": text, + "time": time, + "team_id": team_id, + "user": user, + "recurrence": recurrence, + } + ) + return self.api_call("reminders.add", params=kwargs) + + def reminders_complete( + self, + *, + reminder: str, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Marks a reminder as complete. + https://api.slack.com/methods/reminders.complete + """ + kwargs.update({"reminder": reminder, "team_id": team_id}) + return self.api_call("reminders.complete", params=kwargs) + + def reminders_delete( + self, + *, + reminder: str, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Deletes a reminder. + https://api.slack.com/methods/reminders.delete + """ + kwargs.update({"reminder": reminder, "team_id": team_id}) + return self.api_call("reminders.delete", params=kwargs) + + def reminders_info( + self, + *, + reminder: str, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Gets information about a reminder. + https://api.slack.com/methods/reminders.info + """ + kwargs.update({"reminder": reminder, "team_id": team_id}) + return self.api_call("reminders.info", http_verb="GET", params=kwargs) + + def reminders_list( + self, + *, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Lists all reminders created by or for a given user. + https://api.slack.com/methods/reminders.list + """ + kwargs.update({"team_id": team_id}) + return self.api_call("reminders.list", http_verb="GET", params=kwargs) + + def rtm_connect( + self, + *, + batch_presence_aware: Optional[bool] = None, + presence_sub: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Starts a Real Time Messaging session. + https://api.slack.com/methods/rtm.connect + """ + kwargs.update({"batch_presence_aware": batch_presence_aware, "presence_sub": presence_sub}) + return self.api_call("rtm.connect", http_verb="GET", params=kwargs) + + def rtm_start( + self, + *, + batch_presence_aware: Optional[bool] = None, + include_locale: Optional[bool] = None, + mpim_aware: Optional[bool] = None, + no_latest: Optional[bool] = None, + no_unreads: Optional[bool] = None, + presence_sub: Optional[bool] = None, + simple_latest: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Starts a Real Time Messaging session. + https://api.slack.com/methods/rtm.start + """ + kwargs.update( + { + "batch_presence_aware": batch_presence_aware, + "include_locale": include_locale, + "mpim_aware": mpim_aware, + "no_latest": no_latest, + "no_unreads": no_unreads, + "presence_sub": presence_sub, + "simple_latest": simple_latest, + } + ) + return self.api_call("rtm.start", http_verb="GET", params=kwargs) + + def search_all( + self, + *, + query: str, + count: Optional[int] = None, + highlight: Optional[bool] = None, + page: Optional[int] = None, + sort: Optional[str] = None, + sort_dir: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Searches for messages and files matching a query. + https://api.slack.com/methods/search.all + """ + kwargs.update( + { + "query": query, + "count": count, + "highlight": highlight, + "page": page, + "sort": sort, + "sort_dir": sort_dir, + "team_id": team_id, + } + ) + return self.api_call("search.all", http_verb="GET", params=kwargs) + + def search_files( + self, + *, + query: str, + count: Optional[int] = None, + highlight: Optional[bool] = None, + page: Optional[int] = None, + sort: Optional[str] = None, + sort_dir: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Searches for files matching a query. + https://api.slack.com/methods/search.files + """ + kwargs.update( + { + "query": query, + "count": count, + "highlight": highlight, + "page": page, + "sort": sort, + "sort_dir": sort_dir, + "team_id": team_id, + } + ) + return self.api_call("search.files", http_verb="GET", params=kwargs) + + def search_messages( + self, + *, + query: str, + count: Optional[int] = None, + cursor: Optional[str] = None, + highlight: Optional[bool] = None, + page: Optional[int] = None, + sort: Optional[str] = None, + sort_dir: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Searches for messages matching a query. + https://api.slack.com/methods/search.messages + """ + kwargs.update( + { + "query": query, + "count": count, + "cursor": cursor, + "highlight": highlight, + "page": page, + "sort": sort, + "sort_dir": sort_dir, + "team_id": team_id, + } + ) + return self.api_call("search.messages", http_verb="GET", params=kwargs) + + def stars_add( + self, + *, + channel: Optional[str] = None, + file: Optional[str] = None, + file_comment: Optional[str] = None, + timestamp: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Adds a star to an item. + https://api.slack.com/methods/stars.add + """ + kwargs.update( + { + "channel": channel, + "file": file, + "file_comment": file_comment, + "timestamp": timestamp, + } + ) + return self.api_call("stars.add", params=kwargs) + + def stars_list( + self, + *, + count: Optional[int] = None, + cursor: Optional[str] = None, + limit: Optional[int] = None, + page: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Lists stars for a user. + https://api.slack.com/methods/stars.list + """ + kwargs.update( + { + "count": count, + "cursor": cursor, + "limit": limit, + "page": page, + "team_id": team_id, + } + ) + return self.api_call("stars.list", http_verb="GET", params=kwargs) + + def stars_remove( + self, + *, + channel: Optional[str] = None, + file: Optional[str] = None, + file_comment: Optional[str] = None, + timestamp: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Removes a star from an item. + https://api.slack.com/methods/stars.remove + """ + kwargs.update( + { + "channel": channel, + "file": file, + "file_comment": file_comment, + "timestamp": timestamp, + } + ) + return self.api_call("stars.remove", params=kwargs) + + def team_accessLogs( + self, + *, + before: Optional[Union[int, str]] = None, + count: Optional[Union[int, str]] = None, + page: Optional[Union[int, str]] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Gets the access logs for the current team. + https://api.slack.com/methods/team.accessLogs + """ + kwargs.update( + { + "before": before, + "count": count, + "page": page, + "team_id": team_id, + } + ) + return self.api_call("team.accessLogs", http_verb="GET", params=kwargs) + + def team_billableInfo( + self, + *, + team_id: Optional[str] = None, + user: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Gets billable users information for the current team. + https://api.slack.com/methods/team.billableInfo + """ + kwargs.update({"team_id": team_id, "user": user}) + return self.api_call("team.billableInfo", http_verb="GET", params=kwargs) + + def team_billing_info( + self, + **kwargs, + ) -> SlackResponse: + """Reads a workspace's billing plan information. + https://api.slack.com/methods/team.billing.info + """ + return self.api_call("team.billing.info", params=kwargs) + + def team_info( + self, + *, + team: Optional[str] = None, + domain: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Gets information about the current team. + https://api.slack.com/methods/team.info + """ + kwargs.update({"team": team, "domain": domain}) + return self.api_call("team.info", http_verb="GET", params=kwargs) + + def team_integrationLogs( + self, + *, + app_id: Optional[str] = None, + change_type: Optional[str] = None, + count: Optional[Union[int, str]] = None, + page: Optional[Union[int, str]] = None, + service_id: Optional[str] = None, + team_id: Optional[str] = None, + user: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Gets the integration logs for the current team. + https://api.slack.com/methods/team.integrationLogs + """ + kwargs.update( + { + "app_id": app_id, + "change_type": change_type, + "count": count, + "page": page, + "service_id": service_id, + "team_id": team_id, + "user": user, + } + ) + return self.api_call("team.integrationLogs", http_verb="GET", params=kwargs) + + def team_profile_get( + self, + *, + visibility: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Retrieve a team's profile. + https://api.slack.com/methods/team.profile.get + """ + kwargs.update({"visibility": visibility}) + return self.api_call("team.profile.get", http_verb="GET", params=kwargs) + + def team_preferences_list( + self, + **kwargs, + ) -> SlackResponse: + """Retrieve a list of a workspace's team preferences. + https://api.slack.com/methods/team.preferences.list + """ + return self.api_call("team.preferences.list", params=kwargs) + + def usergroups_create( + self, + *, + name: str, + channels: Optional[Union[str, Sequence[str]]] = None, + description: Optional[str] = None, + handle: Optional[str] = None, + include_count: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Create a User Group + https://api.slack.com/methods/usergroups.create + """ + kwargs.update( + { + "name": name, + "description": description, + "handle": handle, + "include_count": include_count, + "team_id": team_id, + } + ) + if isinstance(channels, (list, Tuple)): + kwargs.update({"channels": ",".join(channels)}) + else: + kwargs.update({"channels": channels}) + return self.api_call("usergroups.create", params=kwargs) + + def usergroups_disable( + self, + *, + usergroup: str, + include_count: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Disable an existing User Group + https://api.slack.com/methods/usergroups.disable + """ + kwargs.update({"usergroup": usergroup, "include_count": include_count, "team_id": team_id}) + return self.api_call("usergroups.disable", params=kwargs) + + def usergroups_enable( + self, + *, + usergroup: str, + include_count: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Enable a User Group + https://api.slack.com/methods/usergroups.enable + """ + kwargs.update({"usergroup": usergroup, "include_count": include_count, "team_id": team_id}) + return self.api_call("usergroups.enable", params=kwargs) + + def usergroups_list( + self, + *, + include_count: Optional[bool] = None, + include_disabled: Optional[bool] = None, + include_users: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """List all User Groups for a team + https://api.slack.com/methods/usergroups.list + """ + kwargs.update( + { + "include_count": include_count, + "include_disabled": include_disabled, + "include_users": include_users, + "team_id": team_id, + } + ) + return self.api_call("usergroups.list", http_verb="GET", params=kwargs) + + def usergroups_update( + self, + *, + usergroup: str, + channels: Optional[Union[str, Sequence[str]]] = None, + description: Optional[str] = None, + handle: Optional[str] = None, + include_count: Optional[bool] = None, + name: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Update an existing User Group + https://api.slack.com/methods/usergroups.update + """ + kwargs.update( + { + "usergroup": usergroup, + "description": description, + "handle": handle, + "include_count": include_count, + "name": name, + "team_id": team_id, + } + ) + if isinstance(channels, (list, Tuple)): + kwargs.update({"channels": ",".join(channels)}) + else: + kwargs.update({"channels": channels}) + return self.api_call("usergroups.update", params=kwargs) + + def usergroups_users_list( + self, + *, + usergroup: str, + include_disabled: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """List all users in a User Group + https://api.slack.com/methods/usergroups.users.list + """ + kwargs.update( + { + "usergroup": usergroup, + "include_disabled": include_disabled, + "team_id": team_id, + } + ) + return self.api_call("usergroups.users.list", http_verb="GET", params=kwargs) + + def usergroups_users_update( + self, + *, + usergroup: str, + users: Union[str, Sequence[str]], + include_count: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Update the list of users for a User Group + https://api.slack.com/methods/usergroups.users.update + """ + kwargs.update( + { + "usergroup": usergroup, + "include_count": include_count, + "team_id": team_id, + } + ) + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + return self.api_call("usergroups.users.update", params=kwargs) + + def users_conversations( + self, + *, + cursor: Optional[str] = None, + exclude_archived: Optional[bool] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + types: Optional[Union[str, Sequence[str]]] = None, + user: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """List conversations the calling user may access. + https://api.slack.com/methods/users.conversations + """ + kwargs.update( + { + "cursor": cursor, + "exclude_archived": exclude_archived, + "limit": limit, + "team_id": team_id, + "user": user, + } + ) + if isinstance(types, (list, Tuple)): + kwargs.update({"types": ",".join(types)}) + else: + kwargs.update({"types": types}) + return self.api_call("users.conversations", http_verb="GET", params=kwargs) + + def users_deletePhoto( + self, + **kwargs, + ) -> SlackResponse: + """Delete the user profile photo + https://api.slack.com/methods/users.deletePhoto + """ + return self.api_call("users.deletePhoto", http_verb="GET", params=kwargs) + + def users_getPresence( + self, + *, + user: str, + **kwargs, + ) -> SlackResponse: + """Gets user presence information. + https://api.slack.com/methods/users.getPresence + """ + kwargs.update({"user": user}) + return self.api_call("users.getPresence", http_verb="GET", params=kwargs) + + def users_identity( + self, + **kwargs, + ) -> SlackResponse: + """Get a user's identity. + https://api.slack.com/methods/users.identity + """ + return self.api_call("users.identity", http_verb="GET", params=kwargs) + + def users_info( + self, + *, + user: str, + include_locale: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Gets information about a user. + https://api.slack.com/methods/users.info + """ + kwargs.update({"user": user, "include_locale": include_locale}) + return self.api_call("users.info", http_verb="GET", params=kwargs) + + def users_list( + self, + *, + cursor: Optional[str] = None, + include_locale: Optional[bool] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Lists all users in a Slack team. + https://api.slack.com/methods/users.list + """ + kwargs.update( + { + "cursor": cursor, + "include_locale": include_locale, + "limit": limit, + "team_id": team_id, + } + ) + return self.api_call("users.list", http_verb="GET", params=kwargs) + + def users_lookupByEmail( + self, + *, + email: str, + **kwargs, + ) -> SlackResponse: + """Find a user with an email address. + https://api.slack.com/methods/users.lookupByEmail + """ + kwargs.update({"email": email}) + return self.api_call("users.lookupByEmail", http_verb="GET", params=kwargs) + + def users_setPhoto( + self, + *, + image: Union[str, IOBase], + crop_w: Optional[Union[int, str]] = None, + crop_x: Optional[Union[int, str]] = None, + crop_y: Optional[Union[int, str]] = None, + **kwargs, + ) -> SlackResponse: + """Set the user profile photo + https://api.slack.com/methods/users.setPhoto + """ + kwargs.update({"crop_w": crop_w, "crop_x": crop_x, "crop_y": crop_y}) + return self.api_call("users.setPhoto", files={"image": image}, data=kwargs) + + def users_setPresence( + self, + *, + presence: str, + **kwargs, + ) -> SlackResponse: + """Manually sets user presence. + https://api.slack.com/methods/users.setPresence + """ + kwargs.update({"presence": presence}) + return self.api_call("users.setPresence", params=kwargs) + + def users_profile_get( + self, + *, + user: Optional[str] = None, + include_labels: Optional[bool] = None, + **kwargs, + ) -> SlackResponse: + """Retrieves a user's profile information. + https://api.slack.com/methods/users.profile.get + """ + kwargs.update({"user": user, "include_labels": include_labels}) + return self.api_call("users.profile.get", http_verb="GET", params=kwargs) + + def users_profile_set( + self, + *, + name: Optional[str] = None, + value: Optional[str] = None, + user: Optional[str] = None, + profile: Optional[Dict] = None, + **kwargs, + ) -> SlackResponse: + """Set the profile information for a user. + https://api.slack.com/methods/users.profile.set + """ + kwargs.update( + { + "name": name, + "profile": profile, + "user": user, + "value": value, + } + ) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "profile" parameter + return self.api_call("users.profile.set", json=kwargs) + + def views_open( + self, + *, + trigger_id: str, + view: Union[dict, View], + **kwargs, + ) -> SlackResponse: + """Open a view for a user. + https://api.slack.com/methods/views.open + See https://api.slack.com/block-kit/surfaces/modals for details. + """ + kwargs.update({"trigger_id": trigger_id}) + if isinstance(view, View): + kwargs.update({"view": view.to_dict()}) + else: + kwargs.update({"view": view}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "view" parameter + return self.api_call("views.open", json=kwargs) + + def views_push( + self, + *, + trigger_id: str, + view: Union[dict, View], + **kwargs, + ) -> SlackResponse: + """Push a view onto the stack of a root view. + Push a new view onto the existing view stack by passing a view + payload and a valid trigger_id generated from an interaction + within the existing modal. + Read the modals documentation (https://api.slack.com/block-kit/surfaces/modals) + to learn more about the lifecycle and intricacies of views. + https://api.slack.com/methods/views.push + """ + kwargs.update({"trigger_id": trigger_id}) + if isinstance(view, View): + kwargs.update({"view": view.to_dict()}) + else: + kwargs.update({"view": view}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "view" parameter + return self.api_call("views.push", json=kwargs) + + def views_update( + self, + *, + view: Union[dict, View], + external_id: Optional[str] = None, + view_id: Optional[str] = None, + hash: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Update an existing view. + Update a view by passing a new view definition along with the + view_id returned in views.open or the external_id. + See the modals documentation (https://api.slack.com/block-kit/surfaces/modals#updating_views) + to learn more about updating views and avoiding race conditions with the hash argument. + https://api.slack.com/methods/views.update + """ + if isinstance(view, View): + kwargs.update({"view": view.to_dict()}) + else: + kwargs.update({"view": view}) + if external_id: + kwargs.update({"external_id": external_id}) + elif view_id: + kwargs.update({"view_id": view_id}) + else: + raise e.SlackRequestError("Either view_id or external_id is required.") + kwargs.update({"hash": hash}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "view" parameter + return self.api_call("views.update", json=kwargs) + + def views_publish( + self, + *, + user_id: str, + view: Union[dict, View], + hash: Optional[str] = None, + **kwargs, + ) -> SlackResponse: + """Publish a static view for a User. + Create or update the view that comprises an + app's Home tab (https://api.slack.com/surfaces/tabs) + https://api.slack.com/methods/views.publish + """ + kwargs.update({"user_id": user_id, "hash": hash}) + if isinstance(view, View): + kwargs.update({"view": view.to_dict()}) + else: + kwargs.update({"view": view}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "view" parameter + return self.api_call("views.publish", json=kwargs) + + def workflows_stepCompleted( + self, + *, + workflow_step_execute_id: str, + outputs: Optional[dict] = None, + **kwargs, + ) -> SlackResponse: + """Indicate a successful outcome of a workflow step's execution. + https://api.slack.com/methods/workflows.stepCompleted + """ + kwargs.update({"workflow_step_execute_id": workflow_step_execute_id}) + if outputs is not None: + kwargs.update({"outputs": outputs}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "outputs" parameter + return self.api_call("workflows.stepCompleted", json=kwargs) + + def workflows_stepFailed( + self, + *, + workflow_step_execute_id: str, + error: Dict[str, str], + **kwargs, + ) -> SlackResponse: + """Indicate an unsuccessful outcome of a workflow step's execution. + https://api.slack.com/methods/workflows.stepFailed + """ + kwargs.update( + { + "workflow_step_execute_id": workflow_step_execute_id, + "error": error, + } + ) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "error" parameter + return self.api_call("workflows.stepFailed", json=kwargs) + + def workflows_updateStep( + self, + *, + workflow_step_edit_id: str, + inputs: Optional[Dict[str, Any]] = None, + outputs: Optional[List[Dict[str, str]]] = None, + **kwargs, + ) -> SlackResponse: + """Update the configuration for a workflow extension step. + https://api.slack.com/methods/workflows.updateStep + """ + kwargs.update({"workflow_step_edit_id": workflow_step_edit_id}) + if inputs is not None: + kwargs.update({"inputs": inputs}) + if outputs is not None: + kwargs.update({"outputs": outputs}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "inputs" / "outputs" parameters + return self.api_call("workflows.updateStep", json=kwargs) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/web/deprecation.py b/core_service/aws_lambda/project/packages/slack_sdk/web/deprecation.py new file mode 100644 index 0000000..5ce5f06 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/web/deprecation.py @@ -0,0 +1,30 @@ +import os +import warnings + +# https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api +deprecated_method_prefixes_2020_01 = [ + "channels.", + "groups.", + "im.", + "mpim.", + "admin.conversations.whitelist.", +] + + +def show_2020_01_deprecation(method_name: str): + """Prints a warning if the given method is deprecated""" + + skip_deprecation = os.environ.get("SLACKCLIENT_SKIP_DEPRECATION") # for unit tests etc. + if skip_deprecation: + return + if not method_name: + return + + matched_prefixes = [prefix for prefix in deprecated_method_prefixes_2020_01 if method_name.startswith(prefix)] + if len(matched_prefixes) > 0: + message = ( + f"{method_name} is deprecated. Please use the Conversations API instead. " + "For more info, go to " + "https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api" + ) + warnings.warn(message) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/web/internal_utils.py b/core_service/aws_lambda/project/packages/slack_sdk/web/internal_utils.py new file mode 100644 index 0000000..f1a7ad9 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/web/internal_utils.py @@ -0,0 +1,301 @@ +import json +import os +import platform +import sys +import warnings +from ssl import SSLContext +from typing import Dict, Union, Optional, Any, Sequence +from urllib.parse import urljoin + +from slack_sdk import version +from slack_sdk.errors import SlackRequestError +from slack_sdk.models.attachments import Attachment +from slack_sdk.models.blocks import Block +from slack_sdk.models.metadata import Metadata + + +def convert_bool_to_0_or_1(params: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any]]: + """Converts all bool values in dict to "0" or "1". + + Slack APIs safely accept "0"/"1" as boolean values. + Using True/False (bool in Python) doesn't work with aiohttp. + This method converts only the bool values in top-level of a given dict. + + Args: + params: params as a dict + + Returns: + Modified dict + """ + if params: + return {k: _to_0_or_1_if_bool(v) for k, v in params.items()} + return None + + +def get_user_agent(prefix: Optional[str] = None, suffix: Optional[str] = None): + """Construct the user-agent header with the package info, + Python version and OS version. + + Returns: + The user agent string. + e.g. 'Python/3.6.7 slackclient/2.0.0 Darwin/17.7.0' + """ + # __name__ returns all classes, we only want the client + client = "{0}/{1}".format("slackclient", version.__version__) + python_version = "Python/{v.major}.{v.minor}.{v.micro}".format(v=sys.version_info) + system_info = "{0}/{1}".format(platform.system(), platform.release()) + user_agent_string = " ".join([python_version, client, system_info]) + prefix = f"{prefix} " if prefix else "" + suffix = f" {suffix}" if suffix else "" + return prefix + user_agent_string + suffix + + +def _get_url(base_url: str, api_method: str) -> str: + """Joins the base Slack URL and an API method to form an absolute URL. + + Args: + base_url (str): The base URL + api_method (str): The Slack Web API method. e.g. 'chat.postMessage' + + Returns: + The absolute API URL. + e.g. 'https://www.slack.com/api/chat.postMessage' + """ + return urljoin(base_url, api_method) + + +def _get_headers( + *, + headers: dict, + token: Optional[str], + has_json: bool, + has_files: bool, + request_specific_headers: Optional[dict], +) -> Dict[str, str]: + """Constructs the headers need for a request. + Args: + has_json (bool): Whether or not the request has json. + has_files (bool): Whether or not the request has files. + request_specific_headers (dict): Additional headers specified by the user for a specific request. + + Returns: + The headers dictionary. + e.g. { + 'Content-Type': 'application/json;charset=utf-8', + 'Authorization': 'Bearer xoxb-1234-1243', + 'User-Agent': 'Python/3.6.8 slack/2.1.0 Darwin/17.7.0' + } + """ + final_headers = { + "Content-Type": "application/x-www-form-urlencoded", + } + if headers is None or "User-Agent" not in headers: + final_headers["User-Agent"] = get_user_agent() + + if token: + final_headers.update({"Authorization": "Bearer {}".format(token)}) + if headers is None: + headers = {} + + # Merge headers specified at client initialization. + final_headers.update(headers) + + # Merge headers specified for a specific request. e.g. oauth.access + if request_specific_headers: + final_headers.update(request_specific_headers) + + if has_json: + final_headers.update({"Content-Type": "application/json;charset=utf-8"}) + + if has_files: + # These are set automatically by the aiohttp library. + final_headers.pop("Content-Type", None) + + return final_headers + + +def _set_default_params(target: dict, default_params: dict) -> None: + for name, value in default_params.items(): + if name not in target: + target[name] = value + + +def _build_req_args( + *, + token: Optional[str], + http_verb: str, + files: dict, + data: dict, + default_params: dict, + params: dict, + json: dict, # skipcq: PYL-W0621 + headers: dict, + auth: dict, + ssl: Optional[SSLContext], + proxy: Optional[str], +) -> dict: + has_json = json is not None + has_files = files is not None + if has_json and http_verb != "POST": + msg = "Json data can only be submitted as POST requests. GET requests should use the 'params' argument." + raise SlackRequestError(msg) + + if data is not None and isinstance(data, dict): + data = {k: v for k, v in data.items() if v is not None} + _set_default_params(data, default_params) + if files is not None and isinstance(files, dict): + files = {k: v for k, v in files.items() if v is not None} + # NOTE: We do not need to all #_set_default_params here + # because other parameters in binary data requests can exist + # only in either data or params, not in files. + if params is not None and isinstance(params, dict): + params = {k: v for k, v in params.items() if v is not None} + _set_default_params(params, default_params) + if json is not None and isinstance(json, dict): + _set_default_params(json, default_params) + + token: Optional[str] = token + if params is not None and "token" in params: + token = params.pop("token") + if json is not None and "token" in json: + token = json.pop("token") + req_args = { + "headers": _get_headers( + headers=headers, + token=token, + has_json=has_json, + has_files=has_files, + request_specific_headers=headers, + ), + "data": data, + "files": files, + "params": params, + "json": json, + "ssl": ssl, + "proxy": proxy, + "auth": auth, + } + return req_args + + +def _parse_web_class_objects(kwargs) -> None: + def to_dict(obj: Union[Dict, Block, Attachment, Metadata]): + if isinstance(obj, Block): + return obj.to_dict() + if isinstance(obj, Attachment): + return obj.to_dict() + if isinstance(obj, Metadata): + return obj.to_dict() + return obj + + blocks = kwargs.get("blocks", None) + if blocks is not None and isinstance(blocks, list): + dict_blocks = [to_dict(b) for b in blocks] + kwargs.update({"blocks": dict_blocks}) + + attachments = kwargs.get("attachments", None) + if attachments is not None and isinstance(attachments, list): + dict_attachments = [to_dict(a) for a in attachments] + kwargs.update({"attachments": dict_attachments}) + + metadata = kwargs.get("metadata", None) + if metadata is not None and isinstance(metadata, Metadata): + kwargs.update({"metadata": to_dict(metadata)}) + + +def _update_call_participants(kwargs, users: Union[str, Sequence[Dict[str, str]]]) -> None: + if users is None: + return + + if isinstance(users, list): + kwargs.update({"users": json.dumps(users)}) + elif isinstance(users, str): + kwargs.update({"users": users}) + else: + raise SlackRequestError("users must be either str or Sequence[Dict[str, str]]") + + +def _next_cursor_is_present(data) -> bool: + """Determine if the response contains 'next_cursor' + and 'next_cursor' is not empty. + + Returns: + A boolean value. + """ + # Only admin.conversations.search returns next_cursor at the top level + present = ("next_cursor" in data and data["next_cursor"] != "") or ( + "response_metadata" in data + and "next_cursor" in data["response_metadata"] + and data["response_metadata"]["next_cursor"] != "" + ) + return present + + +def _to_0_or_1_if_bool(v: Any) -> Union[Any, str]: + if isinstance(v, bool): + return "1" if v else "0" + return v + + +def _warn_if_text_or_attachment_fallback_is_missing(endpoint: str, kwargs: Dict[str, Any]) -> None: + text = kwargs.get("text") + if text and len(text.strip()) > 0: + # If a top-level text arg is provided, we are good. This is the recommended accessibility field to always provide. + return + + # for unit tests etc. + skip_deprecation = os.environ.get("SKIP_SLACK_SDK_WARNING") + if skip_deprecation: + return + + # At this point, at a minimum, text argument is missing. Warn the user about this. + message = ( + f"The top-level `text` argument is missing in the request payload for a {endpoint} call - " + f"It's a best practice to always provide a `text` argument when posting a message. " + f"The `text` argument is used in places where content cannot be rendered such as: " + "system push notifications, assistive technology such as screen readers, etc." + ) + warnings.warn(message, UserWarning) + + # Additionally, specifically for attachments, there is a legacy field available at the attachment level called `fallback` + # Even with a missing text, one can provide a `fallback` per attachment. + # More details here: https://api.slack.com/reference/messaging/attachments#legacy_fields + attachments = kwargs.get("attachments") + # Note that this method does not verify attachments + # if the value is already serialized as a single str value. + if ( + attachments is not None + and isinstance(attachments, list) + and not all( + [isinstance(attachment, dict) and len(attachment.get("fallback", "").strip()) > 0 for attachment in attachments] + ) + ): + # https://api.slack.com/reference/messaging/attachments + # Check if the fallback field exists for all the attachments + # Not all attachments have a fallback property; warn about this too! + message = ( + f"Additionally, the attachment-level `fallback` argument is missing in the request payload for a {endpoint} call" + f" - To avoid this warning, it is recommended to always provide a top-level `text` argument when posting a" + f" message. Alternatively you can provide an attachment-level `fallback` argument, though this is now considered" + f" a legacy field (see https://api.slack.com/reference/messaging/attachments#legacy_fields for more details)." + ) + warnings.warn(message, UserWarning) + + +def _build_unexpected_body_error_message(body: str) -> str: + body_for_logging = "".join([line.strip() for line in body.replace("\r", "\n").split("\n")]) + if len(body_for_logging) > 100: + body_for_logging = body_for_logging[:100] + "..." + message = f"Received a response in a non-JSON format: {body_for_logging}" + return message + + +def _remove_none_values(d: dict) -> dict: + # To avoid having null values in JSON (Slack API does not work with null in many situations) + # + # >>> import json + # >>> d = {"a": None, "b":123} + # >>> json.dumps(d) + # '{"a": null, "b": 123}' + # + return {k: v for k, v in d.items() if v is not None} diff --git a/core_service/aws_lambda/project/packages/slack_sdk/web/legacy_base_client.py b/core_service/aws_lambda/project/packages/slack_sdk/web/legacy_base_client.py new file mode 100644 index 0000000..08b1ff5 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/web/legacy_base_client.py @@ -0,0 +1,559 @@ +"""A Python module for interacting with Slack's Web API.""" + +import asyncio +import copy +import hashlib +import hmac +import io +import json +import logging +import mimetypes +import urllib +import uuid +import warnings +from http.client import HTTPResponse +from ssl import SSLContext +from typing import BinaryIO, Dict, List, Any +from typing import Optional, Union +from urllib.error import HTTPError +from urllib.parse import urlencode +from urllib.request import Request, urlopen, OpenerDirector, ProxyHandler, HTTPSHandler + +import aiohttp +from aiohttp import FormData, BasicAuth + +import slack_sdk.errors as err +from slack_sdk.errors import SlackRequestError +from .async_internal_utils import _files_to_data, _get_event_loop, _request_with_session +from .deprecation import show_2020_01_deprecation +from .internal_utils import ( + convert_bool_to_0_or_1, + get_user_agent, + _get_url, + _build_req_args, + _build_unexpected_body_error_message, +) +from .legacy_slack_response import LegacySlackResponse as SlackResponse +from ..proxy_env_variable_loader import load_http_proxy_from_env + + +class LegacyBaseClient: + BASE_URL = "https://www.slack.com/api/" + + def __init__( + self, + token: Optional[str] = None, + base_url: str = BASE_URL, + timeout: int = 30, + loop: Optional[asyncio.AbstractEventLoop] = None, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + run_async: bool = False, + use_sync_aiohttp: bool = False, + session: Optional[aiohttp.ClientSession] = None, + headers: Optional[dict] = None, + user_agent_prefix: Optional[str] = None, + user_agent_suffix: Optional[str] = None, + # for Org-Wide App installation + team_id: Optional[str] = None, + logger: Optional[logging.Logger] = None, + ): + self.token = None if token is None else token.strip() + """A string specifying an `xoxp-*` or `xoxb-*` token.""" + self.base_url = base_url + """A string representing the Slack API base URL. + Default is `'https://www.slack.com/api/'`.""" + self.timeout = timeout + """The maximum number of seconds the client will wait + to connect and receive a response from Slack. + Default is 30 seconds.""" + self.ssl = ssl + """An [`ssl.SSLContext`](https://docs.python.org/3/library/ssl.html#ssl.SSLContext) + instance, helpful for specifying your own custom + certificate chain.""" + self.proxy = proxy + """String representing a fully-qualified URL to a proxy through which + to route all requests to the Slack API. Even if this parameter + is not specified, if any of the following environment variables are + present, they will be loaded into this parameter: `HTTPS_PROXY`, + `https_proxy`, `HTTP_PROXY` or `http_proxy`.""" + self.run_async = run_async + self.use_sync_aiohttp = use_sync_aiohttp + self.session = session + self.headers = headers or {} + """`dict` representing additional request headers to attach to all requests.""" + self.headers["User-Agent"] = get_user_agent(user_agent_prefix, user_agent_suffix) + self.default_params = {} + if team_id is not None: + self.default_params["team_id"] = team_id + self._logger = logger if logger is not None else logging.getLogger(__name__) + if self.proxy is None or len(self.proxy.strip()) == 0: + env_variable = load_http_proxy_from_env(self._logger) + if env_variable is not None: + self.proxy = env_variable + + self._event_loop = loop + + def api_call( # skipcq: PYL-R1710 + self, + api_method: str, + *, + http_verb: str = "POST", + files: Optional[dict] = None, + data: Union[dict, FormData] = None, + params: Optional[dict] = None, + json: Optional[dict] = None, # skipcq: PYL-W0621 + headers: Optional[dict] = None, + auth: Optional[dict] = None, + ) -> Union[asyncio.Future, SlackResponse]: + """Create a request and execute the API call to Slack. + Args: + api_method (str): The target Slack API method. + e.g. 'chat.postMessage' + http_verb (str): HTTP Verb. e.g. 'POST' + files (dict): Files to multipart upload. + e.g. {image OR file: file_object OR file_path} + data: The body to attach to the request. If a dictionary is + provided, form-encoding will take place. + e.g. {'key1': 'value1', 'key2': 'value2'} + params (dict): The URL parameters to append to the URL. + e.g. {'key1': 'value1', 'key2': 'value2'} + json (dict): JSON for the body to attach to the request + (if files or data is not specified). + e.g. {'key1': 'value1', 'key2': 'value2'} + headers (dict): Additional request headers + auth (dict): A dictionary that consists of client_id and client_secret + Returns: + (SlackResponse) + The server's response to an HTTP request. Data + from the response can be accessed like a dict. + If the response included 'next_cursor' it can + be iterated on to execute subsequent requests. + Raises: + SlackApiError: The following Slack API call failed: + 'chat.postMessage'. + SlackRequestError: Json data can only be submitted as + POST requests. + """ + + api_url = _get_url(self.base_url, api_method) + + headers = headers or {} + headers.update(self.headers) + + if auth is not None: + if isinstance(auth, dict): + auth = BasicAuth(auth["client_id"], auth["client_secret"]) + elif isinstance(auth, BasicAuth): + headers["Authorization"] = auth.encode() + + req_args = _build_req_args( + token=self.token, + http_verb=http_verb, + files=files, + data=data, + default_params=self.default_params, + params=params, + json=json, # skipcq: PYL-W0621 + headers=headers, + auth=auth, + ssl=self.ssl, + proxy=self.proxy, + ) + + show_2020_01_deprecation(api_method) + + if self.run_async or self.use_sync_aiohttp: + if self._event_loop is None: + self._event_loop = _get_event_loop() + + future = asyncio.ensure_future( + self._send(http_verb=http_verb, api_url=api_url, req_args=req_args), + loop=self._event_loop, + ) + if self.run_async: + return future + if self.use_sync_aiohttp: + # Using this is no longer recommended - just keep this for backward-compatibility + return self._event_loop.run_until_complete(future) + + return self._sync_send(api_url=api_url, req_args=req_args) + + # ================================================================= + # aiohttp based async WebClient + # ================================================================= + + async def _send(self, http_verb: str, api_url: str, req_args: dict) -> SlackResponse: + """Sends the request out for transmission. + Args: + http_verb (str): The HTTP verb. e.g. 'GET' or 'POST'. + api_url (str): The Slack API url. e.g. 'https://slack.com/api/chat.postMessage' + req_args (dict): The request arguments to be attached to the request. + e.g. + { + json: { + 'attachments': [{"pretext": "pre-hello", "text": "text-world"}], + 'channel': '#random' + } + } + Returns: + The response parsed into a SlackResponse object. + """ + open_files = _files_to_data(req_args) + try: + if "params" in req_args: + # True/False -> "1"/"0" + req_args["params"] = convert_bool_to_0_or_1(req_args["params"]) + + res = await self._request(http_verb=http_verb, api_url=api_url, req_args=req_args) + finally: + for f in open_files: + f.close() + + data = { + "client": self, + "http_verb": http_verb, + "api_url": api_url, + "req_args": req_args, + "use_sync_aiohttp": self.use_sync_aiohttp, + } + return SlackResponse(**{**data, **res}).validate() + + async def _request(self, *, http_verb, api_url, req_args) -> Dict[str, Any]: + """Submit the HTTP request with the running session or a new session. + Returns: + A dictionary of the response data. + """ + return await _request_with_session( + current_session=self.session, + timeout=self.timeout, + logger=self._logger, + http_verb=http_verb, + api_url=api_url, + req_args=req_args, + ) + + # ================================================================= + # urllib based WebClient + # ================================================================= + + def _sync_send(self, api_url, req_args) -> SlackResponse: + params = req_args["params"] if "params" in req_args else None + data = req_args["data"] if "data" in req_args else None + files = req_args["files"] if "files" in req_args else None + _json = req_args["json"] if "json" in req_args else None + headers = req_args["headers"] if "headers" in req_args else None + token = params.get("token") if params and "token" in params else None + auth = req_args["auth"] if "auth" in req_args else None # Basic Auth for oauth.v2.access / oauth.access + if auth is not None: + headers = {} + if isinstance(auth, BasicAuth): + headers["Authorization"] = auth.encode() + elif isinstance(auth, str): + headers["Authorization"] = auth + else: + self._logger.warning(f"As the auth: {auth}: {type(auth)} is unsupported, skipped") + + body_params = {} + if params: + body_params.update(params) + if data: + body_params.update(data) + + return self._urllib_api_call( + token=token, + url=api_url, + query_params={}, + body_params=body_params, + files=files, + json_body=_json, + additional_headers=headers, + ) + + def _request_for_pagination(self, api_url: str, req_args: Dict[str, Dict[str, Any]]) -> Dict[str, Any]: + """This method is supposed to be used only for SlackResponse pagination + You can paginate using Python's for iterator as below: + for response in client.conversations_list(limit=100): + # do something with each response here + """ + response = self._perform_urllib_http_request(url=api_url, args=req_args) + return { + "status_code": int(response["status"]), + "headers": dict(response["headers"]), + "data": json.loads(response["body"]), + } + + def _urllib_api_call( + self, + *, + token: Optional[str] = None, + url: str, + query_params: Dict[str, str], + json_body: Dict, + body_params: Dict[str, str], + files: Dict[str, io.BytesIO], + additional_headers: Dict[str, str], + ) -> SlackResponse: + """Performs a Slack API request and returns the result. + + Args: + token: Slack API Token (either bot token or user token) + url: Complete URL (e.g., https://www.slack.com/api/chat.postMessage) + query_params: Query string + json_body: JSON data structure (it's still a dict at this point), + if you give this argument, body_params and files will be skipped + body_params: Form body params + files: Files to upload + additional_headers: Request headers to append + Returns: + API response + """ + files_to_close: List[BinaryIO] = [] + try: + # True/False -> "1"/"0" + query_params = convert_bool_to_0_or_1(query_params) + body_params = convert_bool_to_0_or_1(body_params) + + if self._logger.level <= logging.DEBUG: + + def convert_params(values: dict) -> dict: + if not values or not isinstance(values, dict): + return {} + return {k: ("(bytes)" if isinstance(v, bytes) else v) for k, v in values.items()} + + headers = {k: "(redacted)" if k.lower() == "authorization" else v for k, v in additional_headers.items()} + self._logger.debug( + f"Sending a request - url: {url}, " + f"query_params: {convert_params(query_params)}, " + f"body_params: {convert_params(body_params)}, " + f"files: {convert_params(files)}, " + f"json_body: {json_body}, " + f"headers: {headers}" + ) + + request_data = {} + if files is not None and isinstance(files, dict) and len(files) > 0: + if body_params: + for k, v in body_params.items(): + request_data.update({k: v}) + + for k, v in files.items(): + if isinstance(v, str): + f: BinaryIO = open(v.encode("utf-8", "ignore"), "rb") + files_to_close.append(f) + request_data.update({k: f}) + elif isinstance(v, (bytearray, bytes)): + request_data.update({k: io.BytesIO(v)}) + else: + request_data.update({k: v}) + + request_headers = self._build_urllib_request_headers( + token=token or self.token, + has_json=json is not None, + has_files=files is not None, + additional_headers=additional_headers, + ) + request_args = { + "headers": request_headers, + "data": request_data, + "params": body_params, + "files": files, + "json": json_body, + } + if query_params: + q = urlencode(query_params) + url = f"{url}&{q}" if "?" in url else f"{url}?{q}" + + response = self._perform_urllib_http_request(url=url, args=request_args) + body = response.get("body", None) # skipcq: PTC-W0039 + response_body_data: Optional[Union[dict, bytes]] = body + if body is not None and not isinstance(body, bytes): + try: + response_body_data = json.loads(response["body"]) + except json.decoder.JSONDecodeError: + message = _build_unexpected_body_error_message(response.get("body", "")) + raise err.SlackApiError(message, response) + + all_params: Dict[str, Any] = copy.copy(body_params) if body_params is not None else {} + if query_params: + all_params.update(query_params) + request_args["params"] = all_params # for backward-compatibility + + return SlackResponse( + client=self, + http_verb="POST", # you can use POST method for all the Web APIs + api_url=url, + req_args=request_args, + data=response_body_data, + headers=dict(response["headers"]), + status_code=response["status"], + use_sync_aiohttp=False, + ).validate() + finally: + for f in files_to_close: + if not f.closed: + f.close() + + def _perform_urllib_http_request(self, *, url: str, args: Dict[str, Dict[str, Any]]) -> Dict[str, Any]: + """Performs an HTTP request and parses the response. + + Args: + url: Complete URL (e.g., https://www.slack.com/api/chat.postMessage) + args: args has "headers", "data", "params", and "json" + "headers": Dict[str, str] + "data": Dict[str, Any] + "params": Dict[str, str], + "json": Dict[str, Any], + + Returns: + dict {status: int, headers: Headers, body: str} + """ + headers = args["headers"] + if args["json"]: + body = json.dumps(args["json"]) + headers["Content-Type"] = "application/json;charset=utf-8" + elif args["data"]: + boundary = f"--------------{uuid.uuid4()}" + sep_boundary = b"\r\n--" + boundary.encode("ascii") + end_boundary = sep_boundary + b"--\r\n" + body = io.BytesIO() + data = args["data"] + for key, value in data.items(): + readable = getattr(value, "readable", None) + if readable and value.readable(): + filename = "Uploaded file" + name_attr = getattr(value, "name", None) + if name_attr: + filename = name_attr.decode("utf-8") if isinstance(name_attr, bytes) else name_attr + if "filename" in data: + filename = data["filename"] + mimetype = mimetypes.guess_type(filename)[0] or "application/octet-stream" + title = ( + f'\r\nContent-Disposition: form-data; name="{key}"; filename="{filename}"\r\n' + + f"Content-Type: {mimetype}\r\n" + ) + value = value.read() + else: + title = f'\r\nContent-Disposition: form-data; name="{key}"\r\n' + value = str(value).encode("utf-8") + body.write(sep_boundary) + body.write(title.encode("utf-8")) + body.write(b"\r\n") + body.write(value) + + body.write(end_boundary) + body = body.getvalue() + headers["Content-Type"] = f"multipart/form-data; boundary={boundary}" + headers["Content-Length"] = len(body) + elif args["params"]: + body = urlencode(args["params"]) + headers["Content-Type"] = "application/x-www-form-urlencoded" + else: + body = None + + if isinstance(body, str): + body = body.encode("utf-8") + + # NOTE: Intentionally ignore the `http_verb` here + # Slack APIs accepts any API method requests with POST methods + try: + # urllib not only opens http:// or https:// URLs, but also ftp:// and file://. + # With this it might be possible to open local files on the executing machine + # which might be a security risk if the URL to open can be manipulated by an external user. + # (BAN-B310) + if url.lower().startswith("http"): + req = Request(method="POST", url=url, data=body, headers=headers) + opener: Optional[OpenerDirector] = None + if self.proxy is not None: + if isinstance(self.proxy, str): + opener = urllib.request.build_opener( + ProxyHandler({"http": self.proxy, "https": self.proxy}), + HTTPSHandler(context=self.ssl), + ) + else: + raise SlackRequestError(f"Invalid proxy detected: {self.proxy} must be a str value") + + # NOTE: BAN-B310 is already checked above + resp: Optional[HTTPResponse] = None + if opener: + resp = opener.open(req, timeout=self.timeout) # skipcq: BAN-B310 + else: + resp = urlopen(req, context=self.ssl, timeout=self.timeout) # skipcq: BAN-B310 + if resp.headers.get_content_type() == "application/gzip": + # admin.analytics.getFile + body: bytes = resp.read() + return {"status": resp.code, "headers": resp.headers, "body": body} + + charset = resp.headers.get_content_charset() or "utf-8" + body: str = resp.read().decode(charset) # read the response body here + return {"status": resp.code, "headers": resp.headers, "body": body} + raise SlackRequestError(f"Invalid URL detected: {url}") + except HTTPError as e: + # As adding new values to HTTPError#headers can be ignored, building a new dict object here + response_headers = dict(e.headers.items()) + resp = {"status": e.code, "headers": response_headers} + if e.code == 429: + # for compatibility with aiohttp + if "retry-after" not in response_headers and "Retry-After" in response_headers: + response_headers["retry-after"] = response_headers["Retry-After"] + if "Retry-After" not in response_headers and "retry-after" in response_headers: + response_headers["Retry-After"] = response_headers["retry-after"] + + # read the response body here + charset = e.headers.get_content_charset() or "utf-8" + body: str = e.read().decode(charset) + resp["body"] = body + return resp + + except Exception as err: + self._logger.error(f"Failed to send a request to Slack API server: {err}") + raise err + + def _build_urllib_request_headers( + self, token: str, has_json: bool, has_files: bool, additional_headers: dict + ) -> Dict[str, str]: + headers = {"Content-Type": "application/x-www-form-urlencoded"} + headers.update(self.headers) + if token: + headers.update({"Authorization": "Bearer {}".format(token)}) + if additional_headers: + headers.update(additional_headers) + if has_json: + headers.update({"Content-Type": "application/json;charset=utf-8"}) + if has_files: + # will be set afterwards + headers.pop("Content-Type", None) + return headers + + # ================================================================= + + @staticmethod + def validate_slack_signature(*, signing_secret: str, data: str, timestamp: str, signature: str) -> bool: + """ + Slack creates a unique string for your app and shares it with you. Verify + requests from Slack with confidence by verifying signatures using your + signing secret. + On each HTTP request that Slack sends, we add an X-Slack-Signature HTTP + header. The signature is created by combining the signing secret with the + body of the request we're sending using a standard HMAC-SHA256 keyed hash. + https://api.slack.com/docs/verifying-requests-from-slack#how_to_make_a_request_signature_in_4_easy_steps__an_overview + Args: + signing_secret: Your application's signing secret, available in the + Slack API dashboard + data: The raw body of the incoming request - no headers, just the body. + timestamp: from the 'X-Slack-Request-Timestamp' header + signature: from the 'X-Slack-Signature' header - the calculated signature + should match this. + Returns: + True if signatures matches + """ + warnings.warn( + "As this method is deprecated since slackclient 2.6.0, " + "use `from slack.signature import SignatureVerifier` instead", + DeprecationWarning, + ) + format_req = str.encode(f"v0:{timestamp}:{data}") + encoded_secret = str.encode(signing_secret) + request_hash = hmac.new(encoded_secret, format_req, hashlib.sha256).hexdigest() + calculated_signature = f"v0={request_hash}" + return hmac.compare_digest(calculated_signature, signature) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/web/legacy_client.py b/core_service/aws_lambda/project/packages/slack_sdk/web/legacy_client.py new file mode 100644 index 0000000..7a2430b --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/web/legacy_client.py @@ -0,0 +1,4418 @@ +# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +# +# *** DO NOT EDIT THIS FILE *** +# +# 1) Modify slack_sdk/web/client.py +# 2) Run `python setup.py codegen` +# +# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +from asyncio import Future + +"""A Python module for interacting with Slack's Web API.""" +import json +import os +from io import IOBase +from typing import Union, Sequence, Optional, Dict, Tuple, Any, List + +import slack_sdk.errors as e +from slack_sdk.models.views import View +from .legacy_base_client import LegacyBaseClient, SlackResponse +from .internal_utils import ( + _parse_web_class_objects, + _update_call_participants, + _warn_if_text_or_attachment_fallback_is_missing, + _remove_none_values, +) +from ..models.attachments import Attachment +from ..models.blocks import Block +from ..models.metadata import Metadata + + +class LegacyWebClient(LegacyBaseClient): + """A WebClient allows apps to communicate with the Slack Platform's Web API. + + https://api.slack.com/methods + + The Slack Web API is an interface for querying information from + and enacting change in a Slack workspace. + + This client handles constructing and sending HTTP requests to Slack + as well as parsing any responses received into a `SlackResponse`. + + Attributes: + token (str): A string specifying an `xoxp-*` or `xoxb-*` token. + base_url (str): A string representing the Slack API base URL. + Default is `'https://www.slack.com/api/'` + timeout (int): The maximum number of seconds the client will wait + to connect and receive a response from Slack. + Default is 30 seconds. + ssl (SSLContext): An [`ssl.SSLContext`][1] instance, helpful for specifying + your own custom certificate chain. + proxy (str): String representing a fully-qualified URL to a proxy through + which to route all requests to the Slack API. Even if this parameter + is not specified, if any of the following environment variables are + present, they will be loaded into this parameter: `HTTPS_PROXY`, + `https_proxy`, `HTTP_PROXY` or `http_proxy`. + headers (dict): Additional request headers to attach to all requests. + + Methods: + `api_call`: Constructs a request and executes the API call to Slack. + + Example of recommended usage: + ```python + import os + from slack_sdk.web.legacy_client import LegacyWebClient + + client = LegacyWebClient(token=os.environ['SLACK_API_TOKEN']) + response = client.chat_postMessage( + channel='#random', + text="Hello world!") + assert response["ok"] + assert response["message"]["text"] == "Hello world!" + ``` + + Example manually creating an API request: + ```python + import os + from slack_sdk.web.legacy_client import LegacyWebClient + + client = LegacyWebClient(token=os.environ['SLACK_API_TOKEN']) + response = client.api_call( + api_method='chat.postMessage', + json={'channel': '#random','text': "Hello world!"} + ) + assert response["ok"] + assert response["message"]["text"] == "Hello world!" + ``` + + Note: + Any attributes or methods prefixed with _underscores are + intended to be "private" internal use only. They may be changed or + removed at anytime. + + [1]: https://docs.python.org/3/library/ssl.html#ssl.SSLContext + """ + + def admin_analytics_getFile( + self, + *, + type: str, + date: Optional[str] = None, + metadata_only: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieve analytics data for a given date, presented as a compressed JSON file + https://api.slack.com/methods/admin.analytics.getFile + """ + kwargs.update({"type": type}) + if date is not None: + kwargs.update({"date": date}) + if metadata_only is not None: + kwargs.update({"metadata_only": metadata_only}) + return self.api_call("admin.analytics.getFile", params=kwargs) + + def admin_apps_approve( + self, + *, + app_id: Optional[str] = None, + request_id: Optional[str] = None, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Approve an app for installation on a workspace. + Either app_id or request_id is required. + These IDs can be obtained either directly via the app_requested event, + or by the admin.apps.requests.list method. + https://api.slack.com/methods/admin.apps.approve + """ + if app_id: + kwargs.update({"app_id": app_id}) + elif request_id: + kwargs.update({"request_id": request_id}) + else: + raise e.SlackRequestError("The app_id or request_id argument must be specified.") + + kwargs.update( + { + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return self.api_call("admin.apps.approve", params=kwargs) + + def admin_apps_approved_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List approved apps for an org or workspace. + https://api.slack.com/methods/admin.apps.approved.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return self.api_call("admin.apps.approved.list", http_verb="GET", params=kwargs) + + def admin_apps_clearResolution( + self, + *, + app_id: str, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Clear an app resolution + https://api.slack.com/methods/admin.apps.clearResolution + """ + kwargs.update( + { + "app_id": app_id, + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return self.api_call("admin.apps.clearResolution", http_verb="POST", params=kwargs) + + def admin_apps_requests_cancel( + self, + *, + request_id: str, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List app requests for a team/workspace. + https://api.slack.com/methods/admin.apps.requests.cancel + """ + kwargs.update( + { + "request_id": request_id, + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return self.api_call("admin.apps.requests.cancel", http_verb="POST", params=kwargs) + + def admin_apps_requests_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List app requests for a team/workspace. + https://api.slack.com/methods/admin.apps.requests.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + } + ) + return self.api_call("admin.apps.requests.list", http_verb="GET", params=kwargs) + + def admin_apps_restrict( + self, + *, + app_id: Optional[str] = None, + request_id: Optional[str] = None, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Restrict an app for installation on a workspace. + Exactly one of the team_id or enterprise_id arguments is required, not both. + Either app_id or request_id is required. These IDs can be obtained either directly + via the app_requested event, or by the admin.apps.requests.list method. + https://api.slack.com/methods/admin.apps.restrict + """ + if app_id: + kwargs.update({"app_id": app_id}) + elif request_id: + kwargs.update({"request_id": request_id}) + else: + raise e.SlackRequestError("The app_id or request_id argument must be specified.") + + kwargs.update( + { + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return self.api_call("admin.apps.restrict", params=kwargs) + + def admin_apps_restricted_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + enterprise_id: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List restricted apps for an org or workspace. + https://api.slack.com/methods/admin.apps.restricted.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "enterprise_id": enterprise_id, + "team_id": team_id, + } + ) + return self.api_call("admin.apps.restricted.list", http_verb="GET", params=kwargs) + + def admin_apps_uninstall( + self, + *, + app_id: str, + enterprise_id: Optional[str] = None, + team_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Uninstall an app from one or many workspaces, or an entire enterprise organization. + With an org-level token, enterprise_id or team_ids is required. + https://api.slack.com/methods/admin.apps.uninstall + """ + kwargs.update({"app_id": app_id}) + if enterprise_id is not None: + kwargs.update({"enterprise_id": enterprise_id}) + if team_ids is not None: + if isinstance(team_ids, (list, Tuple)): + kwargs.update({"team_ids": ",".join(team_ids)}) + else: + kwargs.update({"team_ids": team_ids}) + return self.api_call("admin.apps.uninstall", http_verb="POST", params=kwargs) + + def admin_auth_policy_getEntities( + self, + *, + policy_name: str, + cursor: Optional[str] = None, + entity_type: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Fetch all the entities assigned to a particular authentication policy by name. + https://api.slack.com/methods/admin.auth.policy.getEntities + """ + kwargs.update({"policy_name": policy_name}) + if cursor is not None: + kwargs.update({"cursor": cursor}) + if entity_type is not None: + kwargs.update({"entity_type": entity_type}) + if limit is not None: + kwargs.update({"limit": limit}) + return self.api_call("admin.auth.policy.getEntities", http_verb="POST", params=kwargs) + + def admin_auth_policy_assignEntities( + self, + *, + entity_ids: Union[str, Sequence[str]], + policy_name: str, + entity_type: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Assign entities to a particular authentication policy. + https://api.slack.com/methods/admin.auth.policy.assignEntities + """ + if isinstance(entity_ids, (list, Tuple)): + kwargs.update({"entity_ids": ",".join(entity_ids)}) + else: + kwargs.update({"entity_ids": entity_ids}) + kwargs.update({"policy_name": policy_name}) + kwargs.update({"entity_type": entity_type}) + return self.api_call("admin.auth.policy.assignEntities", http_verb="POST", params=kwargs) + + def admin_auth_policy_removeEntities( + self, + *, + entity_ids: Union[str, Sequence[str]], + policy_name: str, + entity_type: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Remove specified entities from a specified authentication policy. + https://api.slack.com/methods/admin.auth.policy.removeEntities + """ + if isinstance(entity_ids, (list, Tuple)): + kwargs.update({"entity_ids": ",".join(entity_ids)}) + else: + kwargs.update({"entity_ids": entity_ids}) + kwargs.update({"policy_name": policy_name}) + kwargs.update({"entity_type": entity_type}) + return self.api_call("admin.auth.policy.removeEntities", http_verb="POST", params=kwargs) + + def admin_barriers_create( + self, + *, + barriered_from_usergroup_ids: Union[str, Sequence[str]], + primary_usergroup_id: str, + restricted_subjects: Union[str, Sequence[str]], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Create an Information Barrier + https://api.slack.com/methods/admin.barriers.create + """ + kwargs.update({"primary_usergroup_id": primary_usergroup_id}) + if isinstance(barriered_from_usergroup_ids, (list, Tuple)): + kwargs.update({"barriered_from_usergroup_ids": ",".join(barriered_from_usergroup_ids)}) + else: + kwargs.update({"barriered_from_usergroup_ids": barriered_from_usergroup_ids}) + if isinstance(restricted_subjects, (list, Tuple)): + kwargs.update({"restricted_subjects": ",".join(restricted_subjects)}) + else: + kwargs.update({"restricted_subjects": restricted_subjects}) + return self.api_call("admin.barriers.create", http_verb="POST", params=kwargs) + + def admin_barriers_delete( + self, + *, + barrier_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Delete an existing Information Barrier + https://api.slack.com/methods/admin.barriers.delete + """ + kwargs.update({"barrier_id": barrier_id}) + return self.api_call("admin.barriers.delete", http_verb="POST", params=kwargs) + + def admin_barriers_update( + self, + *, + barrier_id: str, + barriered_from_usergroup_ids: Union[str, Sequence[str]], + primary_usergroup_id: str, + restricted_subjects: Union[str, Sequence[str]], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Update an existing Information Barrier + https://api.slack.com/methods/admin.barriers.update + """ + kwargs.update({"barrier_id": barrier_id, "primary_usergroup_id": primary_usergroup_id}) + if isinstance(barriered_from_usergroup_ids, (list, Tuple)): + kwargs.update({"barriered_from_usergroup_ids": ",".join(barriered_from_usergroup_ids)}) + else: + kwargs.update({"barriered_from_usergroup_ids": barriered_from_usergroup_ids}) + if isinstance(restricted_subjects, (list, Tuple)): + kwargs.update({"restricted_subjects": ",".join(restricted_subjects)}) + else: + kwargs.update({"restricted_subjects": restricted_subjects}) + return self.api_call("admin.barriers.update", http_verb="POST", params=kwargs) + + def admin_barriers_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Get all Information Barriers for your organization + https://api.slack.com/methods/admin.barriers.list""" + kwargs.update( + { + "cursor": cursor, + "limit": limit, + } + ) + return self.api_call("admin.barriers.list", http_verb="GET", params=kwargs) + + def admin_conversations_create( + self, + *, + is_private: bool, + name: str, + description: Optional[str] = None, + org_wide: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Create a public or private channel-based conversation. + https://api.slack.com/methods/admin.conversations.create + """ + kwargs.update( + { + "is_private": is_private, + "name": name, + "description": description, + "org_wide": org_wide, + "team_id": team_id, + } + ) + return self.api_call("admin.conversations.create", params=kwargs) + + def admin_conversations_delete( + self, + *, + channel_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Delete a public or private channel. + https://api.slack.com/methods/admin.conversations.delete + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("admin.conversations.delete", params=kwargs) + + def admin_conversations_invite( + self, + *, + channel_id: str, + user_ids: Union[str, Sequence[str]], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Invite a user to a public or private channel. + https://api.slack.com/methods/admin.conversations.invite + """ + kwargs.update({"channel_id": channel_id}) + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + # NOTE: the endpoint is unable to handle Content-Type: application/json as of Sep 3, 2020. + return self.api_call("admin.conversations.invite", params=kwargs) + + def admin_conversations_archive( + self, + *, + channel_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Archive a public or private channel. + https://api.slack.com/methods/admin.conversations.archive + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("admin.conversations.archive", params=kwargs) + + def admin_conversations_unarchive( + self, + *, + channel_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Unarchive a public or private channel. + https://api.slack.com/methods/admin.conversations.archive + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("admin.conversations.unarchive", params=kwargs) + + def admin_conversations_rename( + self, + *, + channel_id: str, + name: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Rename a public or private channel. + https://api.slack.com/methods/admin.conversations.rename + """ + kwargs.update({"channel_id": channel_id, "name": name}) + return self.api_call("admin.conversations.rename", params=kwargs) + + def admin_conversations_search( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + query: Optional[str] = None, + search_channel_types: Optional[Union[str, Sequence[str]]] = None, + sort: Optional[str] = None, + sort_dir: Optional[str] = None, + team_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Search for public or private channels in an Enterprise organization. + https://api.slack.com/methods/admin.conversations.search + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "query": query, + "sort": sort, + "sort_dir": sort_dir, + } + ) + + if isinstance(search_channel_types, (list, Tuple)): + kwargs.update({"search_channel_types": ",".join(search_channel_types)}) + else: + kwargs.update({"search_channel_types": search_channel_types}) + + if isinstance(team_ids, (list, Tuple)): + kwargs.update({"team_ids": ",".join(team_ids)}) + else: + kwargs.update({"team_ids": team_ids}) + + return self.api_call("admin.conversations.search", params=kwargs) + + def admin_conversations_convertToPrivate( + self, + *, + channel_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Convert a public channel to a private channel. + https://api.slack.com/methods/admin.conversations.convertToPrivate + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("admin.conversations.convertToPrivate", params=kwargs) + + def admin_conversations_setConversationPrefs( + self, + *, + channel_id: str, + prefs: Union[str, Dict[str, str]], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Set the posting permissions for a public or private channel. + https://api.slack.com/methods/admin.conversations.setConversationPrefs + """ + kwargs.update({"channel_id": channel_id}) + if isinstance(prefs, dict): + kwargs.update({"prefs": json.dumps(prefs)}) + else: + kwargs.update({"prefs": prefs}) + return self.api_call("admin.conversations.setConversationPrefs", params=kwargs) + + def admin_conversations_getConversationPrefs( + self, + *, + channel_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Get conversation preferences for a public or private channel. + https://api.slack.com/methods/admin.conversations.getConversationPrefs + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("admin.conversations.getConversationPrefs", params=kwargs) + + def admin_conversations_disconnectShared( + self, + *, + channel_id: str, + leaving_team_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Disconnect a connected channel from one or more workspaces. + https://api.slack.com/methods/admin.conversations.disconnectShared + """ + kwargs.update({"channel_id": channel_id}) + if isinstance(leaving_team_ids, (list, Tuple)): + kwargs.update({"leaving_team_ids": ",".join(leaving_team_ids)}) + else: + kwargs.update({"leaving_team_ids": leaving_team_ids}) + return self.api_call("admin.conversations.disconnectShared", params=kwargs) + + def admin_conversations_ekm_listOriginalConnectedChannelInfo( + self, + *, + channel_ids: Optional[Union[str, Sequence[str]]] = None, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List all disconnected channels—i.e., + channels that were once connected to other workspaces and then disconnected—and + the corresponding original channel IDs for key revocation with EKM. + https://api.slack.com/methods/admin.conversations.ekm.listOriginalConnectedChannelInfo + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + } + ) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + if isinstance(team_ids, (list, Tuple)): + kwargs.update({"team_ids": ",".join(team_ids)}) + else: + kwargs.update({"team_ids": team_ids}) + return self.api_call("admin.conversations.ekm.listOriginalConnectedChannelInfo", params=kwargs) + + def admin_conversations_restrictAccess_addGroup( + self, + *, + channel_id: str, + group_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Add an allowlist of IDP groups for accessing a channel. + https://api.slack.com/methods/admin.conversations.restrictAccess.addGroup + """ + kwargs.update( + { + "channel_id": channel_id, + "group_id": group_id, + "team_id": team_id, + } + ) + return self.api_call( + "admin.conversations.restrictAccess.addGroup", + http_verb="GET", + params=kwargs, + ) + + def admin_conversations_restrictAccess_listGroups( + self, + *, + channel_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List all IDP Groups linked to a channel. + https://api.slack.com/methods/admin.conversations.restrictAccess.listGroups + """ + kwargs.update( + { + "channel_id": channel_id, + "team_id": team_id, + } + ) + return self.api_call( + "admin.conversations.restrictAccess.listGroups", + http_verb="GET", + params=kwargs, + ) + + def admin_conversations_restrictAccess_removeGroup( + self, + *, + channel_id: str, + group_id: str, + team_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Remove a linked IDP group linked from a private channel. + https://api.slack.com/methods/admin.conversations.restrictAccess.removeGroup + """ + kwargs.update( + { + "channel_id": channel_id, + "group_id": group_id, + "team_id": team_id, + } + ) + return self.api_call( + "admin.conversations.restrictAccess.removeGroup", + http_verb="GET", + params=kwargs, + ) + + def admin_conversations_setTeams( + self, + *, + channel_id: str, + org_channel: Optional[bool] = None, + target_team_ids: Optional[Union[str, Sequence[str]]] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Set the workspaces in an Enterprise grid org that connect to a public or private channel. + https://api.slack.com/methods/admin.conversations.setTeams + """ + kwargs.update( + { + "channel_id": channel_id, + "org_channel": org_channel, + "team_id": team_id, + } + ) + if isinstance(target_team_ids, (list, Tuple)): + kwargs.update({"target_team_ids": ",".join(target_team_ids)}) + else: + kwargs.update({"target_team_ids": target_team_ids}) + return self.api_call("admin.conversations.setTeams", params=kwargs) + + def admin_conversations_getTeams( + self, + *, + channel_id: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Set the workspaces in an Enterprise grid org that connect to a channel. + https://api.slack.com/methods/admin.conversations.getTeams + """ + kwargs.update( + { + "channel_id": channel_id, + "cursor": cursor, + "limit": limit, + } + ) + return self.api_call("admin.conversations.getTeams", params=kwargs) + + def admin_conversations_getCustomRetention( + self, + *, + channel_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Get a channel's retention policy + https://api.slack.com/methods/admin.conversations.getCustomRetention + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("admin.conversations.getCustomRetention", params=kwargs) + + def admin_conversations_removeCustomRetention( + self, + *, + channel_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Remove a channel's retention policy + https://api.slack.com/methods/admin.conversations.removeCustomRetention + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("admin.conversations.removeCustomRetention", params=kwargs) + + def admin_conversations_setCustomRetention( + self, + *, + channel_id: str, + duration_days: int, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Set a channel's retention policy + https://api.slack.com/methods/admin.conversations.setCustomRetention + """ + kwargs.update({"channel_id": channel_id, "duration_days": duration_days}) + return self.api_call("admin.conversations.setCustomRetention", params=kwargs) + + def admin_emoji_add( + self, + *, + name: str, + url: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Add an emoji. + https://api.slack.com/methods/admin.emoji.add + """ + kwargs.update({"name": name, "url": url}) + return self.api_call("admin.emoji.add", http_verb="GET", params=kwargs) + + def admin_emoji_addAlias( + self, + *, + alias_for: str, + name: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Add an emoji alias. + https://api.slack.com/methods/admin.emoji.addAlias + """ + kwargs.update({"alias_for": alias_for, "name": name}) + return self.api_call("admin.emoji.addAlias", http_verb="GET", params=kwargs) + + def admin_emoji_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List emoji for an Enterprise Grid organization. + https://api.slack.com/methods/admin.emoji.list + """ + kwargs.update({"cursor": cursor, "limit": limit}) + return self.api_call("admin.emoji.list", http_verb="GET", params=kwargs) + + def admin_emoji_remove( + self, + *, + name: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Remove an emoji across an Enterprise Grid organization. + https://api.slack.com/methods/admin.emoji.remove + """ + kwargs.update({"name": name}) + return self.api_call("admin.emoji.remove", http_verb="GET", params=kwargs) + + def admin_emoji_rename( + self, + *, + name: str, + new_name: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Rename an emoji. + https://api.slack.com/methods/admin.emoji.rename + """ + kwargs.update({"name": name, "new_name": new_name}) + return self.api_call("admin.emoji.rename", http_verb="GET", params=kwargs) + + def admin_users_session_reset( + self, + *, + user_id: str, + mobile_only: Optional[bool] = None, + web_only: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Wipes all valid sessions on all devices for a given user. + https://api.slack.com/methods/admin.users.session.reset + """ + kwargs.update( + { + "user_id": user_id, + "mobile_only": mobile_only, + "web_only": web_only, + } + ) + return self.api_call("admin.users.session.reset", params=kwargs) + + def admin_users_session_resetBulk( + self, + *, + user_ids: Union[str, Sequence[str]], + mobile_only: Optional[bool] = None, + web_only: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Enqueues an asynchronous job to wipe all valid sessions on all devices for a given list of users + https://api.slack.com/methods/admin.users.session.resetBulk + """ + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + kwargs.update( + { + "mobile_only": mobile_only, + "web_only": web_only, + } + ) + return self.api_call("admin.users.session.resetBulk", params=kwargs) + + def admin_users_session_invalidate( + self, + *, + session_id: str, + team_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Invalidate a single session for a user by session_id. + https://api.slack.com/methods/admin.users.session.invalidate + """ + kwargs.update({"session_id": session_id, "team_id": team_id}) + return self.api_call("admin.users.session.invalidate", params=kwargs) + + def admin_users_session_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + user_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Lists all active user sessions for an organization + https://api.slack.com/methods/admin.users.session.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + "user_id": user_id, + } + ) + return self.api_call("admin.users.session.list", params=kwargs) + + def admin_teams_settings_setDefaultChannels( + self, + *, + team_id: str, + channel_ids: Union[str, Sequence[str]], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Set the default channels of a workspace. + https://api.slack.com/methods/admin.teams.settings.setDefaultChannels + """ + kwargs.update({"team_id": team_id}) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return self.api_call("admin.teams.settings.setDefaultChannels", http_verb="GET", params=kwargs) + + def admin_users_session_getSettings( + self, + *, + user_ids: Union[str, Sequence[str]], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Get user-specific session settings—the session duration + and what happens when the client closes—given a list of users. + https://api.slack.com/methods/admin.users.session.getSettings + """ + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + return self.api_call("admin.users.session.getSettings", params=kwargs) + + def admin_users_session_setSettings( + self, + *, + user_ids: Union[str, Sequence[str]], + desktop_app_browser_quit: Optional[bool] = None, + duration: Optional[int] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Configure the user-level session settings—the session duration + and what happens when the client closes—for one or more users. + https://api.slack.com/methods/admin.users.session.setSettings + """ + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + kwargs.update( + { + "desktop_app_browser_quit": desktop_app_browser_quit, + "duration": duration, + } + ) + return self.api_call("admin.users.session.setSettings", params=kwargs) + + def admin_users_session_clearSettings( + self, + *, + user_ids: Union[str, Sequence[str]], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Clear user-specific session settings—the session duration + and what happens when the client closes—for a list of users. + https://api.slack.com/methods/admin.users.session.clearSettings + """ + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + return self.api_call("admin.users.session.clearSettings", params=kwargs) + + def admin_users_unsupportedVersions_export( + self, + *, + date_end_of_support: Optional[Union[str, int]] = None, + date_sessions_started: Optional[Union[str, int]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Ask Slackbot to send you an export listing all workspace members using unsupported software, + presented as a zipped CSV file. + https://api.slack.com/methods/admin.users.unsupportedVersions.export + """ + kwargs.update( + { + "date_end_of_support": date_end_of_support, + "date_sessions_started": date_sessions_started, + } + ) + return self.api_call("admin.users.unsupportedVersions.export", params=kwargs) + + def admin_inviteRequests_approve( + self, + *, + invite_request_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Approve a workspace invite request. + https://api.slack.com/methods/admin.inviteRequests.approve + """ + kwargs.update({"invite_request_id": invite_request_id, "team_id": team_id}) + return self.api_call("admin.inviteRequests.approve", params=kwargs) + + def admin_inviteRequests_approved_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List all approved workspace invite requests. + https://api.slack.com/methods/admin.inviteRequests.approved.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + } + ) + return self.api_call("admin.inviteRequests.approved.list", params=kwargs) + + def admin_inviteRequests_denied_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List all denied workspace invite requests. + https://api.slack.com/methods/admin.inviteRequests.denied.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + } + ) + return self.api_call("admin.inviteRequests.denied.list", params=kwargs) + + def admin_inviteRequests_deny( + self, + *, + invite_request_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Deny a workspace invite request. + https://api.slack.com/methods/admin.inviteRequests.deny + """ + kwargs.update({"invite_request_id": invite_request_id, "team_id": team_id}) + return self.api_call("admin.inviteRequests.deny", params=kwargs) + + def admin_inviteRequests_list( + self, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List all pending workspace invite requests.""" + return self.api_call("admin.inviteRequests.list", params=kwargs) + + def admin_teams_admins_list( + self, + *, + team_id: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List all of the admins on a given workspace. + https://api.slack.com/methods/admin.inviteRequests.list + """ + kwargs.update( + { + "cursor": cursor, + "limit": limit, + "team_id": team_id, + } + ) + return self.api_call("admin.teams.admins.list", http_verb="GET", params=kwargs) + + def admin_teams_create( + self, + *, + team_domain: str, + team_name: str, + team_description: Optional[str] = None, + team_discoverability: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Create an Enterprise team. + https://api.slack.com/methods/admin.teams.create + """ + kwargs.update( + { + "team_domain": team_domain, + "team_name": team_name, + "team_description": team_description, + "team_discoverability": team_discoverability, + } + ) + return self.api_call("admin.teams.create", params=kwargs) + + def admin_teams_list( + self, + *, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List all teams on an Enterprise organization. + https://api.slack.com/methods/admin.teams.list + """ + kwargs.update({"cursor": cursor, "limit": limit}) + return self.api_call("admin.teams.list", params=kwargs) + + def admin_teams_owners_list( + self, + *, + team_id: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List all of the admins on a given workspace. + https://api.slack.com/methods/admin.teams.owners.list + """ + kwargs.update({"team_id": team_id, "cursor": cursor, "limit": limit}) + return self.api_call("admin.teams.owners.list", http_verb="GET", params=kwargs) + + def admin_teams_settings_info( + self, + *, + team_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Fetch information about settings in a workspace + https://api.slack.com/methods/admin.teams.settings.info + """ + kwargs.update({"team_id": team_id}) + return self.api_call("admin.teams.settings.info", params=kwargs) + + def admin_teams_settings_setDescription( + self, + *, + team_id: str, + description: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Set the description of a given workspace. + https://api.slack.com/methods/admin.teams.settings.setDescription + """ + kwargs.update({"team_id": team_id, "description": description}) + return self.api_call("admin.teams.settings.setDescription", params=kwargs) + + def admin_teams_settings_setDiscoverability( + self, + *, + team_id: str, + discoverability: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sets the icon of a workspace. + https://api.slack.com/methods/admin.teams.settings.setDiscoverability + """ + kwargs.update({"team_id": team_id, "discoverability": discoverability}) + return self.api_call("admin.teams.settings.setDiscoverability", params=kwargs) + + def admin_teams_settings_setIcon( + self, + *, + team_id: str, + image_url: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sets the icon of a workspace. + https://api.slack.com/methods/admin.teams.settings.setIcon + """ + kwargs.update({"team_id": team_id, "image_url": image_url}) + return self.api_call("admin.teams.settings.setIcon", http_verb="GET", params=kwargs) + + def admin_teams_settings_setName( + self, + *, + team_id: str, + name: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sets the icon of a workspace. + https://api.slack.com/methods/admin.teams.settings.setName + """ + kwargs.update({"team_id": team_id, "name": name}) + return self.api_call("admin.teams.settings.setName", params=kwargs) + + def admin_usergroups_addChannels( + self, + *, + channel_ids: Union[str, Sequence[str]], + usergroup_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Add one or more default channels to an IDP group. + https://api.slack.com/methods/admin.usergroups.addChannels + """ + kwargs.update({"team_id": team_id, "usergroup_id": usergroup_id}) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return self.api_call("admin.usergroups.addChannels", params=kwargs) + + def admin_usergroups_addTeams( + self, + *, + usergroup_id: str, + team_ids: Union[str, Sequence[str]], + auto_provision: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Associate one or more default workspaces with an organization-wide IDP group. + https://api.slack.com/methods/admin.usergroups.addTeams + """ + kwargs.update({"usergroup_id": usergroup_id, "auto_provision": auto_provision}) + if isinstance(team_ids, (list, Tuple)): + kwargs.update({"team_ids": ",".join(team_ids)}) + else: + kwargs.update({"team_ids": team_ids}) + return self.api_call("admin.usergroups.addTeams", params=kwargs) + + def admin_usergroups_listChannels( + self, + *, + usergroup_id: str, + include_num_members: Optional[bool] = None, + team_id: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Add one or more default channels to an IDP group. + https://api.slack.com/methods/admin.usergroups.listChannels + """ + kwargs.update( + { + "usergroup_id": usergroup_id, + "include_num_members": include_num_members, + "team_id": team_id, + } + ) + return self.api_call("admin.usergroups.listChannels", params=kwargs) + + def admin_usergroups_removeChannels( + self, + *, + usergroup_id: str, + channel_ids: Union[str, Sequence[str]], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Add one or more default channels to an IDP group. + https://api.slack.com/methods/admin.usergroups.removeChannels + """ + kwargs.update({"usergroup_id": usergroup_id}) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return self.api_call("admin.usergroups.removeChannels", params=kwargs) + + def admin_users_assign( + self, + *, + team_id: str, + user_id: str, + channel_ids: Optional[Union[str, Sequence[str]]] = None, + is_restricted: Optional[bool] = None, + is_ultra_restricted: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Add an Enterprise user to a workspace. + https://api.slack.com/methods/admin.users.assign + """ + kwargs.update( + { + "team_id": team_id, + "user_id": user_id, + "is_restricted": is_restricted, + "is_ultra_restricted": is_ultra_restricted, + } + ) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return self.api_call("admin.users.assign", params=kwargs) + + def admin_users_invite( + self, + *, + team_id: str, + email: str, + channel_ids: Union[str, Sequence[str]], + custom_message: Optional[str] = None, + email_password_policy_enabled: Optional[bool] = None, + guest_expiration_ts: Optional[Union[str, float]] = None, + is_restricted: Optional[bool] = None, + is_ultra_restricted: Optional[bool] = None, + real_name: Optional[str] = None, + resend: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Invite a user to a workspace. + https://api.slack.com/methods/admin.users.invite + """ + kwargs.update( + { + "team_id": team_id, + "email": email, + "custom_message": custom_message, + "email_password_policy_enabled": email_password_policy_enabled, + "guest_expiration_ts": str(guest_expiration_ts) if guest_expiration_ts is not None else None, + "is_restricted": is_restricted, + "is_ultra_restricted": is_ultra_restricted, + "real_name": real_name, + "resend": resend, + } + ) + if isinstance(channel_ids, (list, Tuple)): + kwargs.update({"channel_ids": ",".join(channel_ids)}) + else: + kwargs.update({"channel_ids": channel_ids}) + return self.api_call("admin.users.invite", params=kwargs) + + def admin_users_list( + self, + *, + team_id: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List users on a workspace + https://api.slack.com/methods/admin.users.list + """ + kwargs.update( + { + "team_id": team_id, + "cursor": cursor, + "limit": limit, + } + ) + return self.api_call("admin.users.list", params=kwargs) + + def admin_users_remove( + self, + *, + team_id: str, + user_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Remove a user from a workspace. + https://api.slack.com/methods/admin.users.remove + """ + kwargs.update({"team_id": team_id, "user_id": user_id}) + return self.api_call("admin.users.remove", params=kwargs) + + def admin_users_setAdmin( + self, + *, + team_id: str, + user_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Set an existing guest, regular user, or owner to be an admin user. + https://api.slack.com/methods/admin.users.setAdmin + """ + kwargs.update({"team_id": team_id, "user_id": user_id}) + return self.api_call("admin.users.setAdmin", params=kwargs) + + def admin_users_setExpiration( + self, + *, + expiration_ts: int, + user_id: str, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Set an expiration for a guest user. + https://api.slack.com/methods/admin.users.setExpiration + """ + kwargs.update({"expiration_ts": expiration_ts, "team_id": team_id, "user_id": user_id}) + return self.api_call("admin.users.setExpiration", params=kwargs) + + def admin_users_setOwner( + self, + *, + team_id: str, + user_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Set an existing guest, regular user, or admin user to be a workspace owner. + https://api.slack.com/methods/admin.users.setOwner + """ + kwargs.update({"team_id": team_id, "user_id": user_id}) + return self.api_call("admin.users.setOwner", params=kwargs) + + def admin_users_setRegular( + self, + *, + team_id: str, + user_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Set an existing guest user, admin user, or owner to be a regular user. + https://api.slack.com/methods/admin.users.setRegular + """ + kwargs.update({"team_id": team_id, "user_id": user_id}) + return self.api_call("admin.users.setRegular", params=kwargs) + + def api_test( + self, + *, + error: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Checks API calling code. + https://api.slack.com/methods/api.test + """ + kwargs.update({"error": error}) + return self.api_call("api.test", params=kwargs) + + def apps_connections_open( + self, + *, + app_token: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Generate a temporary Socket Mode WebSocket URL that your app can connect to + in order to receive events and interactive payloads + https://api.slack.com/methods/apps.connections.open + """ + kwargs.update({"token": app_token}) + return self.api_call("apps.connections.open", http_verb="POST", params=kwargs) + + def apps_event_authorizations_list( + self, + *, + event_context: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Get a list of authorizations for the given event context. + Each authorization represents an app installation that the event is visible to. + https://api.slack.com/methods/apps.event.authorizations.list + """ + kwargs.update({"event_context": event_context, "cursor": cursor, "limit": limit}) + return self.api_call("apps.event.authorizations.list", params=kwargs) + + def apps_uninstall( + self, + *, + client_id: str, + client_secret: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Uninstalls your app from a workspace. + https://api.slack.com/methods/apps.uninstall + """ + kwargs.update({"client_id": client_id, "client_secret": client_secret}) + return self.api_call("apps.uninstall", params=kwargs) + + def auth_revoke( + self, + *, + test: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Revokes a token. + https://api.slack.com/methods/auth.revoke + """ + kwargs.update({"test": test}) + return self.api_call("auth.revoke", http_verb="GET", params=kwargs) + + def auth_test( + self, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Checks authentication & identity. + https://api.slack.com/methods/auth.test + """ + return self.api_call("auth.test", params=kwargs) + + def auth_teams_list( + self, + cursor: Optional[str] = None, + limit: Optional[int] = None, + include_icon: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List the workspaces a token can access. + https://api.slack.com/methods/auth.teams.list + """ + kwargs.update({"cursor": cursor, "limit": limit, "include_icon": include_icon}) + return self.api_call("auth.teams.list", params=kwargs) + + def bookmarks_add( + self, + *, + channel_id: str, + title: str, + type: str, + emoji: Optional[str] = None, + entity_id: Optional[str] = None, + link: Optional[str] = None, # include when type is 'link' + parent_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Add bookmark to a channel. + https://api.slack.com/methods/bookmarks.add + """ + kwargs.update( + { + "channel_id": channel_id, + "title": title, + "type": type, + "emoji": emoji, + "entity_id": entity_id, + "link": link, + "parent_id": parent_id, + } + ) + return self.api_call("bookmarks.add", http_verb="POST", params=kwargs) + + def bookmarks_edit( + self, + *, + bookmark_id: str, + channel_id: str, + emoji: Optional[str] = None, + link: Optional[str] = None, + title: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Edit bookmark. + https://api.slack.com/methods/bookmarks.edit + """ + kwargs.update( + { + "bookmark_id": bookmark_id, + "channel_id": channel_id, + "emoji": emoji, + "link": link, + "title": title, + } + ) + return self.api_call("bookmarks.edit", http_verb="POST", params=kwargs) + + def bookmarks_list( + self, + *, + channel_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List bookmark for the channel. + https://api.slack.com/methods/bookmarks.list + """ + kwargs.update({"channel_id": channel_id}) + return self.api_call("bookmarks.list", http_verb="POST", params=kwargs) + + def bookmarks_remove( + self, + *, + bookmark_id: str, + channel_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Remove bookmark from the channel. + https://api.slack.com/methods/bookmarks.remove + """ + kwargs.update({"bookmark_id": bookmark_id, "channel_id": channel_id}) + return self.api_call("bookmarks.remove", http_verb="POST", params=kwargs) + + def bots_info( + self, + *, + bot: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Gets information about a bot user. + https://api.slack.com/methods/bots.info + """ + kwargs.update({"bot": bot, "team_id": team_id}) + return self.api_call("bots.info", http_verb="GET", params=kwargs) + + def calls_add( + self, + *, + external_unique_id: str, + join_url: str, + created_by: Optional[str] = None, + date_start: Optional[int] = None, + desktop_app_join_url: Optional[str] = None, + external_display_id: Optional[str] = None, + title: Optional[str] = None, + users: Optional[Union[str, Sequence[Dict[str, str]]]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Registers a new Call. + https://api.slack.com/methods/calls.add + """ + kwargs.update( + { + "external_unique_id": external_unique_id, + "join_url": join_url, + "created_by": created_by, + "date_start": date_start, + "desktop_app_join_url": desktop_app_join_url, + "external_display_id": external_display_id, + "title": title, + } + ) + _update_call_participants( # skipcq: PTC-W0039 + kwargs, + users if users is not None else kwargs.get("users"), # skipcq: PTC-W0039 + ) # skipcq: PTC-W0039 + return self.api_call("calls.add", http_verb="POST", params=kwargs) + + def calls_end( + self, + *, + id: str, + duration: Optional[int] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: # skipcq: PYL-W0622 + """Ends a Call. + https://api.slack.com/methods/calls.end + """ + kwargs.update({"id": id, "duration": duration}) + return self.api_call("calls.end", http_verb="POST", params=kwargs) + + def calls_info( + self, + *, + id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: # skipcq: PYL-W0622 + """Returns information about a Call. + https://api.slack.com/methods/calls.info + """ + kwargs.update({"id": id}) + return self.api_call("calls.info", http_verb="POST", params=kwargs) + + def calls_participants_add( + self, + *, + id: str, # skipcq: PYL-W0622 + users: Union[str, Sequence[Dict[str, str]]], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Registers new participants added to a Call. + https://api.slack.com/methods/calls.participants.add + """ + kwargs.update({"id": id}) + _update_call_participants(kwargs, users) + return self.api_call("calls.participants.add", http_verb="POST", params=kwargs) + + def calls_participants_remove( + self, + *, + id: str, # skipcq: PYL-W0622 + users: Union[str, Sequence[Dict[str, str]]], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Registers participants removed from a Call. + https://api.slack.com/methods/calls.participants.remove + """ + kwargs.update({"id": id}) + _update_call_participants(kwargs, users) + return self.api_call("calls.participants.remove", http_verb="POST", params=kwargs) + + def calls_update( + self, + *, + id: str, + desktop_app_join_url: Optional[str] = None, + join_url: Optional[str] = None, + title: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: # skipcq: PYL-W0622 + """Updates information about a Call. + https://api.slack.com/methods/calls.update + """ + kwargs.update( + { + "id": id, + "desktop_app_join_url": desktop_app_join_url, + "join_url": join_url, + "title": title, + } + ) + return self.api_call("calls.update", http_verb="POST", params=kwargs) + + # -------------------------- + # Deprecated: channels.* + # You can use conversations.* APIs instead. + # https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api + # -------------------------- + + def channels_archive( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Archives a channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.archive", json=kwargs) + + def channels_create( + self, + *, + name: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Creates a channel.""" + kwargs.update({"name": name}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.create", json=kwargs) + + def channels_history( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Fetches history of messages and events from a channel.""" + kwargs.update({"channel": channel}) + return self.api_call("channels.history", http_verb="GET", params=kwargs) + + def channels_info( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Gets information about a channel.""" + kwargs.update({"channel": channel}) + return self.api_call("channels.info", http_verb="GET", params=kwargs) + + def channels_invite( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Invites a user to a channel.""" + kwargs.update({"channel": channel, "user": user}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.invite", json=kwargs) + + def channels_join( + self, + *, + name: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Joins a channel, creating it if needed.""" + kwargs.update({"name": name}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.join", json=kwargs) + + def channels_kick( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Removes a user from a channel.""" + kwargs.update({"channel": channel, "user": user}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.kick", json=kwargs) + + def channels_leave( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Leaves a channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.leave", json=kwargs) + + def channels_list( + self, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Lists all channels in a Slack team.""" + return self.api_call("channels.list", http_verb="GET", params=kwargs) + + def channels_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sets the read cursor in a channel.""" + kwargs.update({"channel": channel, "ts": ts}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.mark", json=kwargs) + + def channels_rename( + self, + *, + channel: str, + name: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Renames a channel.""" + kwargs.update({"channel": channel, "name": name}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.rename", json=kwargs) + + def channels_replies( + self, + *, + channel: str, + thread_ts: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieve a thread of messages posted to a channel""" + kwargs.update({"channel": channel, "thread_ts": thread_ts}) + return self.api_call("channels.replies", http_verb="GET", params=kwargs) + + def channels_setPurpose( + self, + *, + channel: str, + purpose: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sets the purpose for a channel.""" + kwargs.update({"channel": channel, "purpose": purpose}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.setPurpose", json=kwargs) + + def channels_setTopic( + self, + *, + channel: str, + topic: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sets the topic for a channel.""" + kwargs.update({"channel": channel, "topic": topic}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.setTopic", json=kwargs) + + def channels_unarchive( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Unarchives a channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("channels.unarchive", json=kwargs) + + # -------------------------- + + def chat_delete( + self, + *, + channel: str, + ts: str, + as_user: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Deletes a message. + https://api.slack.com/methods/chat.delete + """ + kwargs.update({"channel": channel, "ts": ts, "as_user": as_user}) + return self.api_call("chat.delete", params=kwargs) + + def chat_deleteScheduledMessage( + self, + *, + channel: str, + scheduled_message_id: str, + as_user: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Deletes a scheduled message. + https://api.slack.com/methods/chat.deleteScheduledMessage + """ + kwargs.update( + { + "channel": channel, + "scheduled_message_id": scheduled_message_id, + "as_user": as_user, + } + ) + return self.api_call("chat.deleteScheduledMessage", params=kwargs) + + def chat_getPermalink( + self, + *, + channel: str, + message_ts: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieve a permalink URL for a specific extant message + https://api.slack.com/methods/chat.getPermalink + """ + kwargs.update({"channel": channel, "message_ts": message_ts}) + return self.api_call("chat.getPermalink", http_verb="GET", params=kwargs) + + def chat_meMessage( + self, + *, + channel: str, + text: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Share a me message into a channel. + https://api.slack.com/methods/chat.meMessage + """ + kwargs.update({"channel": channel, "text": text}) + return self.api_call("chat.meMessage", params=kwargs) + + def chat_postEphemeral( + self, + *, + channel: str, + user: str, + text: Optional[str] = None, + as_user: Optional[bool] = None, + attachments: Optional[Sequence[Union[Dict, Attachment]]] = None, + blocks: Optional[Sequence[Union[Dict, Block]]] = None, + thread_ts: Optional[str] = None, + icon_emoji: Optional[str] = None, + icon_url: Optional[str] = None, + link_names: Optional[bool] = None, + username: Optional[str] = None, + parse: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sends an ephemeral message to a user in a channel. + https://api.slack.com/methods/chat.postEphemeral + """ + kwargs.update( + { + "channel": channel, + "user": user, + "text": text, + "as_user": as_user, + "attachments": attachments, + "blocks": blocks, + "thread_ts": thread_ts, + "icon_emoji": icon_emoji, + "icon_url": icon_url, + "link_names": link_names, + "username": username, + "parse": parse, + } + ) + _parse_web_class_objects(kwargs) + kwargs = _remove_none_values(kwargs) + _warn_if_text_or_attachment_fallback_is_missing("chat.postEphemeral", kwargs) + # NOTE: intentionally using json over params for the API methods using blocks/attachments + return self.api_call("chat.postEphemeral", json=kwargs) + + def chat_postMessage( + self, + *, + channel: str, + text: Optional[str] = None, + as_user: Optional[bool] = None, + attachments: Optional[Sequence[Union[Dict, Attachment]]] = None, + blocks: Optional[Sequence[Union[Dict, Block]]] = None, + thread_ts: Optional[str] = None, + reply_broadcast: Optional[bool] = None, + unfurl_links: Optional[bool] = None, + unfurl_media: Optional[bool] = None, + container_id: Optional[str] = None, + file_annotation: Optional[str] = None, + icon_emoji: Optional[str] = None, + icon_url: Optional[str] = None, + mrkdwn: Optional[bool] = None, + link_names: Optional[bool] = None, + username: Optional[str] = None, + parse: Optional[str] = None, # none, full + metadata: Optional[Union[Dict, Metadata]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sends a message to a channel. + https://api.slack.com/methods/chat.postMessage + """ + kwargs.update( + { + "channel": channel, + "text": text, + "as_user": as_user, + "attachments": attachments, + "blocks": blocks, + "thread_ts": thread_ts, + "reply_broadcast": reply_broadcast, + "unfurl_links": unfurl_links, + "unfurl_media": unfurl_media, + "container_id": container_id, + "file_annotation": file_annotation, + "icon_emoji": icon_emoji, + "icon_url": icon_url, + "mrkdwn": mrkdwn, + "link_names": link_names, + "username": username, + "parse": parse, + "metadata": metadata, + } + ) + _parse_web_class_objects(kwargs) + kwargs = _remove_none_values(kwargs) + _warn_if_text_or_attachment_fallback_is_missing("chat.postMessage", kwargs) + # NOTE: intentionally using json over params for the API methods using blocks/attachments + return self.api_call("chat.postMessage", json=kwargs) + + def chat_scheduleMessage( + self, + *, + channel: str, + post_at: Union[str, int], + text: str, + as_user: Optional[bool] = None, + attachments: Optional[Sequence[Union[Dict, Attachment]]] = None, + blocks: Optional[Sequence[Union[Dict, Block]]] = None, + thread_ts: Optional[str] = None, + parse: Optional[str] = None, + reply_broadcast: Optional[bool] = None, + unfurl_links: Optional[bool] = None, + unfurl_media: Optional[bool] = None, + link_names: Optional[bool] = None, + metadata: Optional[Union[Dict, Metadata]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Schedules a message. + https://api.slack.com/methods/chat.scheduleMessage + """ + kwargs.update( + { + "channel": channel, + "post_at": post_at, + "text": text, + "as_user": as_user, + "attachments": attachments, + "blocks": blocks, + "thread_ts": thread_ts, + "reply_broadcast": reply_broadcast, + "parse": parse, + "unfurl_links": unfurl_links, + "unfurl_media": unfurl_media, + "link_names": link_names, + "metadata": metadata, + } + ) + _parse_web_class_objects(kwargs) + kwargs = _remove_none_values(kwargs) + _warn_if_text_or_attachment_fallback_is_missing("chat.scheduleMessage", kwargs) + # NOTE: intentionally using json over params for the API methods using blocks/attachments + return self.api_call("chat.scheduleMessage", json=kwargs) + + def chat_unfurl( + self, + *, + channel: str, + ts: str, + unfurls: Dict[str, Dict], + user_auth_blocks: Optional[Sequence[Union[Dict, Block]]] = None, + user_auth_message: Optional[str] = None, + user_auth_required: Optional[bool] = None, + user_auth_url: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Provide custom unfurl behavior for user-posted URLs. + https://api.slack.com/methods/chat.unfurl + """ + kwargs.update( + { + "channel": channel, + "ts": ts, + "unfurls": unfurls, + "user_auth_blocks": user_auth_blocks, + "user_auth_message": user_auth_message, + "user_auth_required": user_auth_required, + "user_auth_url": user_auth_url, + } + ) + kwargs = _remove_none_values(kwargs) + # NOTE: intentionally using json over params for API methods using blocks/attachments + return self.api_call("chat.unfurl", json=kwargs) + + def chat_update( + self, + *, + channel: str, + ts: str, + text: Optional[str] = None, + attachments: Optional[Sequence[Union[Dict, Attachment]]] = None, + blocks: Optional[Sequence[Union[Dict, Block]]] = None, + as_user: Optional[bool] = None, + file_ids: Optional[Union[str, Sequence[str]]] = None, + link_names: Optional[bool] = None, + parse: Optional[str] = None, # none, full + reply_broadcast: Optional[bool] = None, + metadata: Optional[Union[Dict, Metadata]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Updates a message in a channel. + https://api.slack.com/methods/chat.update + """ + kwargs.update( + { + "channel": channel, + "ts": ts, + "text": text, + "attachments": attachments, + "blocks": blocks, + "as_user": as_user, + "link_names": link_names, + "parse": parse, + "reply_broadcast": reply_broadcast, + "metadata": metadata, + } + ) + if isinstance(file_ids, (list, Tuple)): + kwargs.update({"file_ids": ",".join(file_ids)}) + else: + kwargs.update({"file_ids": file_ids}) + _parse_web_class_objects(kwargs) + kwargs = _remove_none_values(kwargs) + _warn_if_text_or_attachment_fallback_is_missing("chat.update", kwargs) + # NOTE: intentionally using json over params for API methods using blocks/attachments + return self.api_call("chat.update", json=kwargs) + + def chat_scheduledMessages_list( + self, + *, + channel: Optional[str] = None, + cursor: Optional[str] = None, + latest: Optional[str] = None, + limit: Optional[int] = None, + oldest: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Lists all scheduled messages. + https://api.slack.com/methods/chat.scheduledMessages.list + """ + kwargs.update( + { + "channel": channel, + "cursor": cursor, + "latest": latest, + "limit": limit, + "oldest": oldest, + "team_id": team_id, + } + ) + return self.api_call("chat.scheduledMessages.list", params=kwargs) + + def conversations_acceptSharedInvite( + self, + *, + channel_name: str, + channel_id: Optional[str] = None, + invite_id: Optional[str] = None, + free_trial_accepted: Optional[bool] = None, + is_private: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Accepts an invitation to a Slack Connect channel. + https://api.slack.com/methods/conversations.acceptSharedInvite + """ + if channel_id is None and invite_id is None: + raise e.SlackRequestError("Either channel_id or invite_id must be provided.") + kwargs.update( + { + "channel_name": channel_name, + "channel_id": channel_id, + "invite_id": invite_id, + "free_trial_accepted": free_trial_accepted, + "is_private": is_private, + "team_id": team_id, + } + ) + return self.api_call("conversations.acceptSharedInvite", http_verb="POST", params=kwargs) + + def conversations_approveSharedInvite( + self, + *, + invite_id: str, + target_team: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Approves an invitation to a Slack Connect channel. + https://api.slack.com/methods/conversations.approveSharedInvite + """ + kwargs.update({"invite_id": invite_id, "target_team": target_team}) + return self.api_call("conversations.approveSharedInvite", http_verb="POST", params=kwargs) + + def conversations_archive( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Archives a conversation. + https://api.slack.com/methods/conversations.archive + """ + kwargs.update({"channel": channel}) + return self.api_call("conversations.archive", params=kwargs) + + def conversations_close( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Closes a direct message or multi-person direct message. + https://api.slack.com/methods/conversations.close + """ + kwargs.update({"channel": channel}) + return self.api_call("conversations.close", params=kwargs) + + def conversations_create( + self, + *, + name: str, + is_private: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Initiates a public or private channel-based conversation + https://api.slack.com/methods/conversations.create + """ + kwargs.update({"name": name, "is_private": is_private, "team_id": team_id}) + return self.api_call("conversations.create", params=kwargs) + + def conversations_declineSharedInvite( + self, + *, + invite_id: str, + target_team: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Declines a Slack Connect channel invite. + https://api.slack.com/methods/conversations.declineSharedInvite + """ + kwargs.update({"invite_id": invite_id, "target_team": target_team}) + return self.api_call("conversations.declineSharedInvite", http_verb="GET", params=kwargs) + + def conversations_history( + self, + *, + channel: str, + cursor: Optional[str] = None, + inclusive: Optional[bool] = None, + include_all_metadata: Optional[bool] = None, + latest: Optional[str] = None, + limit: Optional[int] = None, + oldest: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Fetches a conversation's history of messages and events. + https://api.slack.com/methods/conversations.history + """ + kwargs.update( + { + "channel": channel, + "cursor": cursor, + "inclusive": inclusive, + "include_all_metadata": include_all_metadata, + "limit": limit, + "latest": latest, + "oldest": oldest, + } + ) + return self.api_call("conversations.history", http_verb="GET", params=kwargs) + + def conversations_info( + self, + *, + channel: str, + include_locale: Optional[bool] = None, + include_num_members: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieve information about a conversation. + https://api.slack.com/methods/conversations.info + """ + kwargs.update( + { + "channel": channel, + "include_locale": include_locale, + "include_num_members": include_num_members, + } + ) + return self.api_call("conversations.info", http_verb="GET", params=kwargs) + + def conversations_invite( + self, + *, + channel: str, + users: Union[str, Sequence[str]], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Invites users to a channel. + https://api.slack.com/methods/conversations.invite + """ + kwargs.update({"channel": channel}) + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + return self.api_call("conversations.invite", params=kwargs) + + def conversations_inviteShared( + self, + *, + channel: str, + emails: Optional[Union[str, Sequence[str]]] = None, + user_ids: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sends an invitation to a Slack Connect channel. + https://api.slack.com/methods/conversations.inviteShared + """ + if emails is None and user_ids is None: + raise e.SlackRequestError("Either emails or user ids must be provided.") + kwargs.update({"channel": channel}) + if isinstance(emails, (list, Tuple)): + kwargs.update({"emails": ",".join(emails)}) + else: + kwargs.update({"emails": emails}) + if isinstance(user_ids, (list, Tuple)): + kwargs.update({"user_ids": ",".join(user_ids)}) + else: + kwargs.update({"user_ids": user_ids}) + return self.api_call("conversations.inviteShared", http_verb="GET", params=kwargs) + + def conversations_join( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Joins an existing conversation. + https://api.slack.com/methods/conversations.join + """ + kwargs.update({"channel": channel}) + return self.api_call("conversations.join", params=kwargs) + + def conversations_kick( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Removes a user from a conversation. + https://api.slack.com/methods/conversations.kick + """ + kwargs.update({"channel": channel, "user": user}) + return self.api_call("conversations.kick", params=kwargs) + + def conversations_leave( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Leaves a conversation. + https://api.slack.com/methods/conversations.leave + """ + kwargs.update({"channel": channel}) + return self.api_call("conversations.leave", params=kwargs) + + def conversations_list( + self, + *, + cursor: Optional[str] = None, + exclude_archived: Optional[bool] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + types: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Lists all channels in a Slack team. + https://api.slack.com/methods/conversations.list + """ + kwargs.update( + { + "cursor": cursor, + "exclude_archived": exclude_archived, + "limit": limit, + "team_id": team_id, + } + ) + if isinstance(types, (list, Tuple)): + kwargs.update({"types": ",".join(types)}) + else: + kwargs.update({"types": types}) + return self.api_call("conversations.list", http_verb="GET", params=kwargs) + + def conversations_listConnectInvites( + self, + *, + count: Optional[int] = None, + cursor: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List shared channel invites that have been generated + or received but have not yet been approved by all parties. + https://api.slack.com/methods/conversations.listConnectInvites + """ + kwargs.update({"count": count, "cursor": cursor, "team_id": team_id}) + return self.api_call("conversations.listConnectInvites", params=kwargs) + + def conversations_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sets the read cursor in a channel. + https://api.slack.com/methods/conversations.mark + """ + kwargs.update({"channel": channel, "ts": ts}) + return self.api_call("conversations.mark", params=kwargs) + + def conversations_members( + self, + *, + channel: str, + cursor: Optional[str] = None, + limit: Optional[int] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieve members of a conversation. + https://api.slack.com/methods/conversations.members + """ + kwargs.update({"channel": channel, "cursor": cursor, "limit": limit}) + return self.api_call("conversations.members", http_verb="GET", params=kwargs) + + def conversations_open( + self, + *, + channel: Optional[str] = None, + return_im: Optional[bool] = None, + users: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Opens or resumes a direct message or multi-person direct message. + https://api.slack.com/methods/conversations.open + """ + if channel is None and users is None: + raise e.SlackRequestError("Either channel or users must be provided.") + kwargs.update({"channel": channel, "return_im": return_im}) + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + return self.api_call("conversations.open", params=kwargs) + + def conversations_rename( + self, + *, + channel: str, + name: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Renames a conversation. + https://api.slack.com/methods/conversations.rename + """ + kwargs.update({"channel": channel, "name": name}) + return self.api_call("conversations.rename", params=kwargs) + + def conversations_replies( + self, + *, + channel: str, + ts: str, + cursor: Optional[str] = None, + inclusive: Optional[bool] = None, + latest: Optional[str] = None, + limit: Optional[int] = None, + oldest: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieve a thread of messages posted to a conversation + https://api.slack.com/methods/conversations.replies + """ + kwargs.update( + { + "channel": channel, + "ts": ts, + "cursor": cursor, + "inclusive": inclusive, + "limit": limit, + "latest": latest, + "oldest": oldest, + } + ) + return self.api_call("conversations.replies", http_verb="GET", params=kwargs) + + def conversations_setPurpose( + self, + *, + channel: str, + purpose: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sets the purpose for a conversation. + https://api.slack.com/methods/conversations.setPurpose + """ + kwargs.update({"channel": channel, "purpose": purpose}) + return self.api_call("conversations.setPurpose", params=kwargs) + + def conversations_setTopic( + self, + *, + channel: str, + topic: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sets the topic for a conversation. + https://api.slack.com/methods/conversations.setTopic + """ + kwargs.update({"channel": channel, "topic": topic}) + return self.api_call("conversations.setTopic", params=kwargs) + + def conversations_unarchive( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Reverses conversation archival. + https://api.slack.com/methods/conversations.unarchive + """ + kwargs.update({"channel": channel}) + return self.api_call("conversations.unarchive", params=kwargs) + + def dialog_open( + self, + *, + dialog: Dict[str, Any], + trigger_id: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Open a dialog with a user. + https://api.slack.com/methods/dialog.open + """ + kwargs.update({"dialog": dialog, "trigger_id": trigger_id}) + kwargs = _remove_none_values(kwargs) + # NOTE: As the dialog can be a dict, this API call works only with json format. + return self.api_call("dialog.open", json=kwargs) + + def dnd_endDnd( + self, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Ends the current user's Do Not Disturb session immediately. + https://api.slack.com/methods/dnd.endDnd + """ + return self.api_call("dnd.endDnd", params=kwargs) + + def dnd_endSnooze( + self, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Ends the current user's snooze mode immediately. + https://api.slack.com/methods/dnd.endSnooze + """ + return self.api_call("dnd.endSnooze", params=kwargs) + + def dnd_info( + self, + *, + team_id: Optional[str] = None, + user: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieves a user's current Do Not Disturb status. + https://api.slack.com/methods/dnd.info + """ + kwargs.update({"team_id": team_id, "user": user}) + return self.api_call("dnd.info", http_verb="GET", params=kwargs) + + def dnd_setSnooze( + self, + *, + num_minutes: Union[int, str], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Turns on Do Not Disturb mode for the current user, or changes its duration. + https://api.slack.com/methods/dnd.setSnooze + """ + kwargs.update({"num_minutes": num_minutes}) + return self.api_call("dnd.setSnooze", http_verb="GET", params=kwargs) + + def dnd_teamInfo( + self, + users: Union[str, Sequence[str]], + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieves the Do Not Disturb status for users on a team. + https://api.slack.com/methods/dnd.teamInfo + """ + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + kwargs.update({"team_id": team_id}) + return self.api_call("dnd.teamInfo", http_verb="GET", params=kwargs) + + def emoji_list( + self, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Lists custom emoji for a team. + https://api.slack.com/methods/emoji.list + """ + return self.api_call("emoji.list", http_verb="GET", params=kwargs) + + def files_comments_delete( + self, + *, + file: str, + id: str, + **kwargs, # skipcq: PYL-W0622 + ) -> Union[Future, SlackResponse]: + """Deletes an existing comment on a file. + https://api.slack.com/methods/files.comments.delete + """ + kwargs.update({"file": file, "id": id}) + return self.api_call("files.comments.delete", params=kwargs) + + def files_delete( + self, + *, + file: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Deletes a file. + https://api.slack.com/methods/files.delete + """ + kwargs.update({"file": file}) + return self.api_call("files.delete", params=kwargs) + + def files_info( + self, + *, + file: str, + count: Optional[int] = None, + cursor: Optional[str] = None, + limit: Optional[int] = None, + page: Optional[int] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Gets information about a team file. + https://api.slack.com/methods/files.info + """ + kwargs.update( + { + "file": file, + "count": count, + "cursor": cursor, + "limit": limit, + "page": page, + } + ) + return self.api_call("files.info", http_verb="GET", params=kwargs) + + def files_list( + self, + *, + channel: Optional[str] = None, + count: Optional[int] = None, + page: Optional[int] = None, + show_files_hidden_by_limit: Optional[bool] = None, + team_id: Optional[str] = None, + ts_from: Optional[str] = None, + ts_to: Optional[str] = None, + types: Optional[Union[str, Sequence[str]]] = None, + user: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Lists & filters team files. + https://api.slack.com/methods/files.list + """ + kwargs.update( + { + "channel": channel, + "count": count, + "page": page, + "show_files_hidden_by_limit": show_files_hidden_by_limit, + "team_id": team_id, + "ts_from": ts_from, + "ts_to": ts_to, + "user": user, + } + ) + if isinstance(types, (list, Tuple)): + kwargs.update({"types": ",".join(types)}) + else: + kwargs.update({"types": types}) + return self.api_call("files.list", http_verb="GET", params=kwargs) + + def files_remote_info( + self, + *, + external_id: Optional[str] = None, + file: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieve information about a remote file added to Slack. + https://api.slack.com/methods/files.remote.info + """ + kwargs.update({"external_id": external_id, "file": file}) + return self.api_call("files.remote.info", http_verb="GET", params=kwargs) + + def files_remote_list( + self, + *, + channel: Optional[str] = None, + cursor: Optional[str] = None, + limit: Optional[int] = None, + ts_from: Optional[str] = None, + ts_to: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieve information about a remote file added to Slack. + https://api.slack.com/methods/files.remote.list + """ + kwargs.update( + { + "channel": channel, + "cursor": cursor, + "limit": limit, + "ts_from": ts_from, + "ts_to": ts_to, + } + ) + return self.api_call("files.remote.list", http_verb="GET", params=kwargs) + + def files_remote_add( + self, + *, + external_id: str, + external_url: str, + title: str, + filetype: Optional[str] = None, + indexable_file_contents: Optional[Union[str, bytes, IOBase]] = None, + preview_image: Optional[Union[str, bytes, IOBase]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Adds a file from a remote service. + https://api.slack.com/methods/files.remote.add + """ + kwargs.update( + { + "external_id": external_id, + "external_url": external_url, + "title": title, + "filetype": filetype, + } + ) + files = None + # preview_image (file): Preview of the document via multipart/form-data. + if preview_image is not None or indexable_file_contents is not None: + files = { + "preview_image": preview_image, + "indexable_file_contents": indexable_file_contents, + } + + return self.api_call( + # Intentionally using "POST" method over "GET" here + "files.remote.add", + http_verb="POST", + data=kwargs, + files=files, + ) + + def files_remote_update( + self, + *, + external_id: Optional[str] = None, + external_url: Optional[str] = None, + file: Optional[str] = None, + title: Optional[str] = None, + filetype: Optional[str] = None, + indexable_file_contents: Optional[str] = None, + preview_image: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Updates an existing remote file. + https://api.slack.com/methods/files.remote.update + """ + kwargs.update( + { + "external_id": external_id, + "external_url": external_url, + "file": file, + "title": title, + "filetype": filetype, + } + ) + files = None + # preview_image (file): Preview of the document via multipart/form-data. + if preview_image is not None or indexable_file_contents is not None: + files = { + "preview_image": preview_image, + "indexable_file_contents": indexable_file_contents, + } + + return self.api_call( + # Intentionally using "POST" method over "GET" here + "files.remote.update", + http_verb="POST", + data=kwargs, + files=files, + ) + + def files_remote_remove( + self, + *, + external_id: Optional[str] = None, + file: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Remove a remote file. + https://api.slack.com/methods/files.remote.remove + """ + kwargs.update({"external_id": external_id, "file": file}) + return self.api_call("files.remote.remove", http_verb="POST", params=kwargs) + + def files_remote_share( + self, + *, + channels: Union[str, Sequence[str]], + external_id: Optional[str] = None, + file: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Share a remote file into a channel. + https://api.slack.com/methods/files.remote.share + """ + if external_id is None and file is None: + raise e.SlackRequestError("Either external_id or file must be provided.") + if isinstance(channels, (list, Tuple)): + kwargs.update({"channels": ",".join(channels)}) + else: + kwargs.update({"channels": channels}) + kwargs.update({"external_id": external_id, "file": file}) + return self.api_call("files.remote.share", http_verb="GET", params=kwargs) + + def files_revokePublicURL( + self, + *, + file: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Revokes public/external sharing access for a file + https://api.slack.com/methods/files.revokePublicURL + """ + kwargs.update({"file": file}) + return self.api_call("files.revokePublicURL", params=kwargs) + + def files_sharedPublicURL( + self, + *, + file: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Enables a file for public/external sharing. + https://api.slack.com/methods/files.sharedPublicURL + """ + kwargs.update({"file": file}) + return self.api_call("files.sharedPublicURL", params=kwargs) + + def files_upload( + self, + *, + file: Optional[Union[str, bytes, IOBase]] = None, + content: Optional[str] = None, + filename: Optional[str] = None, + filetype: Optional[str] = None, + initial_comment: Optional[str] = None, + thread_ts: Optional[str] = None, + title: Optional[str] = None, + channels: Optional[Union[str, Sequence[str]]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Uploads or creates a file. + https://api.slack.com/methods/files.upload + """ + if file is None and content is None: + raise e.SlackRequestError("The file or content argument must be specified.") + if file is not None and content is not None: + raise e.SlackRequestError("You cannot specify both the file and the content argument.") + + if isinstance(channels, (list, Tuple)): + kwargs.update({"channels": ",".join(channels)}) + else: + kwargs.update({"channels": channels}) + kwargs.update( + { + "filename": filename, + "filetype": filetype, + "initial_comment": initial_comment, + "thread_ts": thread_ts, + "title": title, + } + ) + if file: + if kwargs.get("filename") is None and isinstance(file, str): + # use the local filename if filename is missing + if kwargs.get("filename") is None: + kwargs["filename"] = file.split(os.path.sep)[-1] + return self.api_call("files.upload", files={"file": file}, data=kwargs) + else: + kwargs["content"] = content + return self.api_call("files.upload", data=kwargs) + + # -------------------------- + # Deprecated: groups.* + # You can use conversations.* APIs instead. + # https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api + # -------------------------- + + def groups_archive( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Archives a private channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.archive", json=kwargs) + + def groups_create( + self, + *, + name: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Creates a private channel.""" + kwargs.update({"name": name}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.create", json=kwargs) + + def groups_createChild( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Clones and archives a private channel.""" + kwargs.update({"channel": channel}) + return self.api_call("groups.createChild", http_verb="GET", params=kwargs) + + def groups_history( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Fetches history of messages and events from a private channel.""" + kwargs.update({"channel": channel}) + return self.api_call("groups.history", http_verb="GET", params=kwargs) + + def groups_info( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Gets information about a private channel.""" + kwargs.update({"channel": channel}) + return self.api_call("groups.info", http_verb="GET", params=kwargs) + + def groups_invite( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Invites a user to a private channel.""" + kwargs.update({"channel": channel, "user": user}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.invite", json=kwargs) + + def groups_kick( + self, + *, + channel: str, + user: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Removes a user from a private channel.""" + kwargs.update({"channel": channel, "user": user}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.kick", json=kwargs) + + def groups_leave( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Leaves a private channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.leave", json=kwargs) + + def groups_list( + self, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Lists private channels that the calling user has access to.""" + return self.api_call("groups.list", http_verb="GET", params=kwargs) + + def groups_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sets the read cursor in a private channel.""" + kwargs.update({"channel": channel, "ts": ts}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.mark", json=kwargs) + + def groups_open( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Opens a private channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.open", json=kwargs) + + def groups_rename( + self, + *, + channel: str, + name: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Renames a private channel.""" + kwargs.update({"channel": channel, "name": name}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.rename", json=kwargs) + + def groups_replies( + self, + *, + channel: str, + thread_ts: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieve a thread of messages posted to a private channel""" + kwargs.update({"channel": channel, "thread_ts": thread_ts}) + return self.api_call("groups.replies", http_verb="GET", params=kwargs) + + def groups_setPurpose( + self, + *, + channel: str, + purpose: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sets the purpose for a private channel.""" + kwargs.update({"channel": channel, "purpose": purpose}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.setPurpose", json=kwargs) + + def groups_setTopic( + self, + *, + channel: str, + topic: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sets the topic for a private channel.""" + kwargs.update({"channel": channel, "topic": topic}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.setTopic", json=kwargs) + + def groups_unarchive( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Unarchives a private channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("groups.unarchive", json=kwargs) + + # -------------------------- + # Deprecated: im.* + # You can use conversations.* APIs instead. + # https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api + # -------------------------- + + def im_close( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Close a direct message channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("im.close", json=kwargs) + + def im_history( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Fetches history of messages and events from direct message channel.""" + kwargs.update({"channel": channel}) + return self.api_call("im.history", http_verb="GET", params=kwargs) + + def im_list( + self, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Lists direct message channels for the calling user.""" + return self.api_call("im.list", http_verb="GET", params=kwargs) + + def im_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sets the read cursor in a direct message channel.""" + kwargs.update({"channel": channel, "ts": ts}) + kwargs = _remove_none_values(kwargs) + return self.api_call("im.mark", json=kwargs) + + def im_open( + self, + *, + user: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Opens a direct message channel.""" + kwargs.update({"user": user}) + kwargs = _remove_none_values(kwargs) + return self.api_call("im.open", json=kwargs) + + def im_replies( + self, + *, + channel: str, + thread_ts: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieve a thread of messages posted to a direct message conversation""" + kwargs.update({"channel": channel, "thread_ts": thread_ts}) + return self.api_call("im.replies", http_verb="GET", params=kwargs) + + # -------------------------- + + def migration_exchange( + self, + *, + users: Union[str, Sequence[str]], + team_id: Optional[str] = None, + to_old: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """For Enterprise Grid workspaces, map local user IDs to global user IDs + https://api.slack.com/methods/migration.exchange + """ + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + kwargs.update({"team_id": team_id, "to_old": to_old}) + return self.api_call("migration.exchange", http_verb="GET", params=kwargs) + + # -------------------------- + # Deprecated: mpim.* + # You can use conversations.* APIs instead. + # https://api.slack.com/changelog/2020-01-deprecating-antecedents-to-the-conversations-api + # -------------------------- + + def mpim_close( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Closes a multiparty direct message channel.""" + kwargs.update({"channel": channel}) + kwargs = _remove_none_values(kwargs) + return self.api_call("mpim.close", json=kwargs) + + def mpim_history( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Fetches history of messages and events from a multiparty direct message.""" + kwargs.update({"channel": channel}) + return self.api_call("mpim.history", http_verb="GET", params=kwargs) + + def mpim_list( + self, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Lists multiparty direct message channels for the calling user.""" + return self.api_call("mpim.list", http_verb="GET", params=kwargs) + + def mpim_mark( + self, + *, + channel: str, + ts: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Sets the read cursor in a multiparty direct message channel.""" + kwargs.update({"channel": channel, "ts": ts}) + kwargs = _remove_none_values(kwargs) + return self.api_call("mpim.mark", json=kwargs) + + def mpim_open( + self, + *, + users: Union[str, Sequence[str]], + **kwargs, + ) -> Union[Future, SlackResponse]: + """This method opens a multiparty direct message.""" + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + return self.api_call("mpim.open", params=kwargs) + + def mpim_replies( + self, + *, + channel: str, + thread_ts: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieve a thread of messages posted to a direct message conversation from a + multiparty direct message. + """ + kwargs.update({"channel": channel, "thread_ts": thread_ts}) + return self.api_call("mpim.replies", http_verb="GET", params=kwargs) + + # -------------------------- + + def oauth_v2_access( + self, + *, + client_id: str, + client_secret: str, + # This field is required when processing the OAuth redirect URL requests + # while it's absent for token rotation + code: Optional[str] = None, + redirect_uri: Optional[str] = None, + # This field is required for token rotation + grant_type: Optional[str] = None, + # This field is required for token rotation + refresh_token: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Exchanges a temporary OAuth verifier code for an access token. + https://api.slack.com/methods/oauth.v2.access + """ + if redirect_uri is not None: + kwargs.update({"redirect_uri": redirect_uri}) + if code is not None: + kwargs.update({"code": code}) + if grant_type is not None: + kwargs.update({"grant_type": grant_type}) + if refresh_token is not None: + kwargs.update({"refresh_token": refresh_token}) + return self.api_call( + "oauth.v2.access", + data=kwargs, + auth={"client_id": client_id, "client_secret": client_secret}, + ) + + def oauth_access( + self, + *, + client_id: str, + client_secret: str, + code: str, + redirect_uri: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Exchanges a temporary OAuth verifier code for an access token. + https://api.slack.com/methods/oauth.access + """ + if redirect_uri is not None: + kwargs.update({"redirect_uri": redirect_uri}) + kwargs.update({"code": code}) + return self.api_call( + "oauth.access", + data=kwargs, + auth={"client_id": client_id, "client_secret": client_secret}, + ) + + def oauth_v2_exchange( + self, + *, + token: str, + client_id: str, + client_secret: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Exchanges a legacy access token for a new expiring access token and refresh token + https://api.slack.com/methods/oauth.v2.exchange + """ + kwargs.update({"client_id": client_id, "client_secret": client_secret, "token": token}) + return self.api_call("oauth.v2.exchange", params=kwargs) + + def openid_connect_token( + self, + client_id: str, + client_secret: str, + code: Optional[str] = None, + redirect_uri: Optional[str] = None, + grant_type: Optional[str] = None, + refresh_token: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Exchanges a temporary OAuth verifier code for an access token for Sign in with Slack. + https://api.slack.com/methods/openid.connect.token + """ + if redirect_uri is not None: + kwargs.update({"redirect_uri": redirect_uri}) + if code is not None: + kwargs.update({"code": code}) + if grant_type is not None: + kwargs.update({"grant_type": grant_type}) + if refresh_token is not None: + kwargs.update({"refresh_token": refresh_token}) + return self.api_call( + "openid.connect.token", + data=kwargs, + auth={"client_id": client_id, "client_secret": client_secret}, + ) + + def openid_connect_userInfo( + self, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Get the identity of a user who has authorized Sign in with Slack. + https://api.slack.com/methods/openid.connect.userInfo + """ + return self.api_call("openid.connect.userInfo", params=kwargs) + + def pins_add( + self, + *, + channel: str, + timestamp: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Pins an item to a channel. + https://api.slack.com/methods/pins.add + """ + kwargs.update({"channel": channel, "timestamp": timestamp}) + return self.api_call("pins.add", params=kwargs) + + def pins_list( + self, + *, + channel: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Lists items pinned to a channel. + https://api.slack.com/methods/pins.list + """ + kwargs.update({"channel": channel}) + return self.api_call("pins.list", http_verb="GET", params=kwargs) + + def pins_remove( + self, + *, + channel: str, + timestamp: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Un-pins an item from a channel. + https://api.slack.com/methods/pins.remove + """ + kwargs.update({"channel": channel, "timestamp": timestamp}) + return self.api_call("pins.remove", params=kwargs) + + def reactions_add( + self, + *, + channel: str, + name: str, + timestamp: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Adds a reaction to an item. + https://api.slack.com/methods/reactions.add + """ + kwargs.update({"channel": channel, "name": name, "timestamp": timestamp}) + return self.api_call("reactions.add", params=kwargs) + + def reactions_get( + self, + *, + channel: Optional[str] = None, + file: Optional[str] = None, + file_comment: Optional[str] = None, + full: Optional[bool] = None, + timestamp: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Gets reactions for an item. + https://api.slack.com/methods/reactions.get + """ + kwargs.update( + { + "channel": channel, + "file": file, + "file_comment": file_comment, + "full": full, + "timestamp": timestamp, + } + ) + return self.api_call("reactions.get", http_verb="GET", params=kwargs) + + def reactions_list( + self, + *, + count: Optional[int] = None, + cursor: Optional[str] = None, + full: Optional[bool] = None, + limit: Optional[int] = None, + page: Optional[int] = None, + team_id: Optional[str] = None, + user: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Lists reactions made by a user. + https://api.slack.com/methods/reactions.list + """ + kwargs.update( + { + "count": count, + "cursor": cursor, + "full": full, + "limit": limit, + "page": page, + "team_id": team_id, + "user": user, + } + ) + return self.api_call("reactions.list", http_verb="GET", params=kwargs) + + def reactions_remove( + self, + *, + name: str, + channel: Optional[str] = None, + file: Optional[str] = None, + file_comment: Optional[str] = None, + timestamp: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Removes a reaction from an item. + https://api.slack.com/methods/reactions.remove + """ + kwargs.update( + { + "name": name, + "channel": channel, + "file": file, + "file_comment": file_comment, + "timestamp": timestamp, + } + ) + return self.api_call("reactions.remove", params=kwargs) + + def reminders_add( + self, + *, + text: str, + time: str, + team_id: Optional[str] = None, + user: Optional[str] = None, + recurrence: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Creates a reminder. + https://api.slack.com/methods/reminders.add + """ + kwargs.update( + { + "text": text, + "time": time, + "team_id": team_id, + "user": user, + "recurrence": recurrence, + } + ) + return self.api_call("reminders.add", params=kwargs) + + def reminders_complete( + self, + *, + reminder: str, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Marks a reminder as complete. + https://api.slack.com/methods/reminders.complete + """ + kwargs.update({"reminder": reminder, "team_id": team_id}) + return self.api_call("reminders.complete", params=kwargs) + + def reminders_delete( + self, + *, + reminder: str, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Deletes a reminder. + https://api.slack.com/methods/reminders.delete + """ + kwargs.update({"reminder": reminder, "team_id": team_id}) + return self.api_call("reminders.delete", params=kwargs) + + def reminders_info( + self, + *, + reminder: str, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Gets information about a reminder. + https://api.slack.com/methods/reminders.info + """ + kwargs.update({"reminder": reminder, "team_id": team_id}) + return self.api_call("reminders.info", http_verb="GET", params=kwargs) + + def reminders_list( + self, + *, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Lists all reminders created by or for a given user. + https://api.slack.com/methods/reminders.list + """ + kwargs.update({"team_id": team_id}) + return self.api_call("reminders.list", http_verb="GET", params=kwargs) + + def rtm_connect( + self, + *, + batch_presence_aware: Optional[bool] = None, + presence_sub: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Starts a Real Time Messaging session. + https://api.slack.com/methods/rtm.connect + """ + kwargs.update({"batch_presence_aware": batch_presence_aware, "presence_sub": presence_sub}) + return self.api_call("rtm.connect", http_verb="GET", params=kwargs) + + def rtm_start( + self, + *, + batch_presence_aware: Optional[bool] = None, + include_locale: Optional[bool] = None, + mpim_aware: Optional[bool] = None, + no_latest: Optional[bool] = None, + no_unreads: Optional[bool] = None, + presence_sub: Optional[bool] = None, + simple_latest: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Starts a Real Time Messaging session. + https://api.slack.com/methods/rtm.start + """ + kwargs.update( + { + "batch_presence_aware": batch_presence_aware, + "include_locale": include_locale, + "mpim_aware": mpim_aware, + "no_latest": no_latest, + "no_unreads": no_unreads, + "presence_sub": presence_sub, + "simple_latest": simple_latest, + } + ) + return self.api_call("rtm.start", http_verb="GET", params=kwargs) + + def search_all( + self, + *, + query: str, + count: Optional[int] = None, + highlight: Optional[bool] = None, + page: Optional[int] = None, + sort: Optional[str] = None, + sort_dir: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Searches for messages and files matching a query. + https://api.slack.com/methods/search.all + """ + kwargs.update( + { + "query": query, + "count": count, + "highlight": highlight, + "page": page, + "sort": sort, + "sort_dir": sort_dir, + "team_id": team_id, + } + ) + return self.api_call("search.all", http_verb="GET", params=kwargs) + + def search_files( + self, + *, + query: str, + count: Optional[int] = None, + highlight: Optional[bool] = None, + page: Optional[int] = None, + sort: Optional[str] = None, + sort_dir: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Searches for files matching a query. + https://api.slack.com/methods/search.files + """ + kwargs.update( + { + "query": query, + "count": count, + "highlight": highlight, + "page": page, + "sort": sort, + "sort_dir": sort_dir, + "team_id": team_id, + } + ) + return self.api_call("search.files", http_verb="GET", params=kwargs) + + def search_messages( + self, + *, + query: str, + count: Optional[int] = None, + cursor: Optional[str] = None, + highlight: Optional[bool] = None, + page: Optional[int] = None, + sort: Optional[str] = None, + sort_dir: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Searches for messages matching a query. + https://api.slack.com/methods/search.messages + """ + kwargs.update( + { + "query": query, + "count": count, + "cursor": cursor, + "highlight": highlight, + "page": page, + "sort": sort, + "sort_dir": sort_dir, + "team_id": team_id, + } + ) + return self.api_call("search.messages", http_verb="GET", params=kwargs) + + def stars_add( + self, + *, + channel: Optional[str] = None, + file: Optional[str] = None, + file_comment: Optional[str] = None, + timestamp: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Adds a star to an item. + https://api.slack.com/methods/stars.add + """ + kwargs.update( + { + "channel": channel, + "file": file, + "file_comment": file_comment, + "timestamp": timestamp, + } + ) + return self.api_call("stars.add", params=kwargs) + + def stars_list( + self, + *, + count: Optional[int] = None, + cursor: Optional[str] = None, + limit: Optional[int] = None, + page: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Lists stars for a user. + https://api.slack.com/methods/stars.list + """ + kwargs.update( + { + "count": count, + "cursor": cursor, + "limit": limit, + "page": page, + "team_id": team_id, + } + ) + return self.api_call("stars.list", http_verb="GET", params=kwargs) + + def stars_remove( + self, + *, + channel: Optional[str] = None, + file: Optional[str] = None, + file_comment: Optional[str] = None, + timestamp: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Removes a star from an item. + https://api.slack.com/methods/stars.remove + """ + kwargs.update( + { + "channel": channel, + "file": file, + "file_comment": file_comment, + "timestamp": timestamp, + } + ) + return self.api_call("stars.remove", params=kwargs) + + def team_accessLogs( + self, + *, + before: Optional[Union[int, str]] = None, + count: Optional[Union[int, str]] = None, + page: Optional[Union[int, str]] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Gets the access logs for the current team. + https://api.slack.com/methods/team.accessLogs + """ + kwargs.update( + { + "before": before, + "count": count, + "page": page, + "team_id": team_id, + } + ) + return self.api_call("team.accessLogs", http_verb="GET", params=kwargs) + + def team_billableInfo( + self, + *, + team_id: Optional[str] = None, + user: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Gets billable users information for the current team. + https://api.slack.com/methods/team.billableInfo + """ + kwargs.update({"team_id": team_id, "user": user}) + return self.api_call("team.billableInfo", http_verb="GET", params=kwargs) + + def team_billing_info( + self, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Reads a workspace's billing plan information. + https://api.slack.com/methods/team.billing.info + """ + return self.api_call("team.billing.info", params=kwargs) + + def team_info( + self, + *, + team: Optional[str] = None, + domain: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Gets information about the current team. + https://api.slack.com/methods/team.info + """ + kwargs.update({"team": team, "domain": domain}) + return self.api_call("team.info", http_verb="GET", params=kwargs) + + def team_integrationLogs( + self, + *, + app_id: Optional[str] = None, + change_type: Optional[str] = None, + count: Optional[Union[int, str]] = None, + page: Optional[Union[int, str]] = None, + service_id: Optional[str] = None, + team_id: Optional[str] = None, + user: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Gets the integration logs for the current team. + https://api.slack.com/methods/team.integrationLogs + """ + kwargs.update( + { + "app_id": app_id, + "change_type": change_type, + "count": count, + "page": page, + "service_id": service_id, + "team_id": team_id, + "user": user, + } + ) + return self.api_call("team.integrationLogs", http_verb="GET", params=kwargs) + + def team_profile_get( + self, + *, + visibility: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieve a team's profile. + https://api.slack.com/methods/team.profile.get + """ + kwargs.update({"visibility": visibility}) + return self.api_call("team.profile.get", http_verb="GET", params=kwargs) + + def team_preferences_list( + self, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieve a list of a workspace's team preferences. + https://api.slack.com/methods/team.preferences.list + """ + return self.api_call("team.preferences.list", params=kwargs) + + def usergroups_create( + self, + *, + name: str, + channels: Optional[Union[str, Sequence[str]]] = None, + description: Optional[str] = None, + handle: Optional[str] = None, + include_count: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Create a User Group + https://api.slack.com/methods/usergroups.create + """ + kwargs.update( + { + "name": name, + "description": description, + "handle": handle, + "include_count": include_count, + "team_id": team_id, + } + ) + if isinstance(channels, (list, Tuple)): + kwargs.update({"channels": ",".join(channels)}) + else: + kwargs.update({"channels": channels}) + return self.api_call("usergroups.create", params=kwargs) + + def usergroups_disable( + self, + *, + usergroup: str, + include_count: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Disable an existing User Group + https://api.slack.com/methods/usergroups.disable + """ + kwargs.update({"usergroup": usergroup, "include_count": include_count, "team_id": team_id}) + return self.api_call("usergroups.disable", params=kwargs) + + def usergroups_enable( + self, + *, + usergroup: str, + include_count: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Enable a User Group + https://api.slack.com/methods/usergroups.enable + """ + kwargs.update({"usergroup": usergroup, "include_count": include_count, "team_id": team_id}) + return self.api_call("usergroups.enable", params=kwargs) + + def usergroups_list( + self, + *, + include_count: Optional[bool] = None, + include_disabled: Optional[bool] = None, + include_users: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List all User Groups for a team + https://api.slack.com/methods/usergroups.list + """ + kwargs.update( + { + "include_count": include_count, + "include_disabled": include_disabled, + "include_users": include_users, + "team_id": team_id, + } + ) + return self.api_call("usergroups.list", http_verb="GET", params=kwargs) + + def usergroups_update( + self, + *, + usergroup: str, + channels: Optional[Union[str, Sequence[str]]] = None, + description: Optional[str] = None, + handle: Optional[str] = None, + include_count: Optional[bool] = None, + name: Optional[str] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Update an existing User Group + https://api.slack.com/methods/usergroups.update + """ + kwargs.update( + { + "usergroup": usergroup, + "description": description, + "handle": handle, + "include_count": include_count, + "name": name, + "team_id": team_id, + } + ) + if isinstance(channels, (list, Tuple)): + kwargs.update({"channels": ",".join(channels)}) + else: + kwargs.update({"channels": channels}) + return self.api_call("usergroups.update", params=kwargs) + + def usergroups_users_list( + self, + *, + usergroup: str, + include_disabled: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List all users in a User Group + https://api.slack.com/methods/usergroups.users.list + """ + kwargs.update( + { + "usergroup": usergroup, + "include_disabled": include_disabled, + "team_id": team_id, + } + ) + return self.api_call("usergroups.users.list", http_verb="GET", params=kwargs) + + def usergroups_users_update( + self, + *, + usergroup: str, + users: Union[str, Sequence[str]], + include_count: Optional[bool] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Update the list of users for a User Group + https://api.slack.com/methods/usergroups.users.update + """ + kwargs.update( + { + "usergroup": usergroup, + "include_count": include_count, + "team_id": team_id, + } + ) + if isinstance(users, (list, Tuple)): + kwargs.update({"users": ",".join(users)}) + else: + kwargs.update({"users": users}) + return self.api_call("usergroups.users.update", params=kwargs) + + def users_conversations( + self, + *, + cursor: Optional[str] = None, + exclude_archived: Optional[bool] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + types: Optional[Union[str, Sequence[str]]] = None, + user: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """List conversations the calling user may access. + https://api.slack.com/methods/users.conversations + """ + kwargs.update( + { + "cursor": cursor, + "exclude_archived": exclude_archived, + "limit": limit, + "team_id": team_id, + "user": user, + } + ) + if isinstance(types, (list, Tuple)): + kwargs.update({"types": ",".join(types)}) + else: + kwargs.update({"types": types}) + return self.api_call("users.conversations", http_verb="GET", params=kwargs) + + def users_deletePhoto( + self, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Delete the user profile photo + https://api.slack.com/methods/users.deletePhoto + """ + return self.api_call("users.deletePhoto", http_verb="GET", params=kwargs) + + def users_getPresence( + self, + *, + user: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Gets user presence information. + https://api.slack.com/methods/users.getPresence + """ + kwargs.update({"user": user}) + return self.api_call("users.getPresence", http_verb="GET", params=kwargs) + + def users_identity( + self, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Get a user's identity. + https://api.slack.com/methods/users.identity + """ + return self.api_call("users.identity", http_verb="GET", params=kwargs) + + def users_info( + self, + *, + user: str, + include_locale: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Gets information about a user. + https://api.slack.com/methods/users.info + """ + kwargs.update({"user": user, "include_locale": include_locale}) + return self.api_call("users.info", http_verb="GET", params=kwargs) + + def users_list( + self, + *, + cursor: Optional[str] = None, + include_locale: Optional[bool] = None, + limit: Optional[int] = None, + team_id: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Lists all users in a Slack team. + https://api.slack.com/methods/users.list + """ + kwargs.update( + { + "cursor": cursor, + "include_locale": include_locale, + "limit": limit, + "team_id": team_id, + } + ) + return self.api_call("users.list", http_verb="GET", params=kwargs) + + def users_lookupByEmail( + self, + *, + email: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Find a user with an email address. + https://api.slack.com/methods/users.lookupByEmail + """ + kwargs.update({"email": email}) + return self.api_call("users.lookupByEmail", http_verb="GET", params=kwargs) + + def users_setPhoto( + self, + *, + image: Union[str, IOBase], + crop_w: Optional[Union[int, str]] = None, + crop_x: Optional[Union[int, str]] = None, + crop_y: Optional[Union[int, str]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Set the user profile photo + https://api.slack.com/methods/users.setPhoto + """ + kwargs.update({"crop_w": crop_w, "crop_x": crop_x, "crop_y": crop_y}) + return self.api_call("users.setPhoto", files={"image": image}, data=kwargs) + + def users_setPresence( + self, + *, + presence: str, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Manually sets user presence. + https://api.slack.com/methods/users.setPresence + """ + kwargs.update({"presence": presence}) + return self.api_call("users.setPresence", params=kwargs) + + def users_profile_get( + self, + *, + user: Optional[str] = None, + include_labels: Optional[bool] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Retrieves a user's profile information. + https://api.slack.com/methods/users.profile.get + """ + kwargs.update({"user": user, "include_labels": include_labels}) + return self.api_call("users.profile.get", http_verb="GET", params=kwargs) + + def users_profile_set( + self, + *, + name: Optional[str] = None, + value: Optional[str] = None, + user: Optional[str] = None, + profile: Optional[Dict] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Set the profile information for a user. + https://api.slack.com/methods/users.profile.set + """ + kwargs.update( + { + "name": name, + "profile": profile, + "user": user, + "value": value, + } + ) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "profile" parameter + return self.api_call("users.profile.set", json=kwargs) + + def views_open( + self, + *, + trigger_id: str, + view: Union[dict, View], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Open a view for a user. + https://api.slack.com/methods/views.open + See https://api.slack.com/block-kit/surfaces/modals for details. + """ + kwargs.update({"trigger_id": trigger_id}) + if isinstance(view, View): + kwargs.update({"view": view.to_dict()}) + else: + kwargs.update({"view": view}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "view" parameter + return self.api_call("views.open", json=kwargs) + + def views_push( + self, + *, + trigger_id: str, + view: Union[dict, View], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Push a view onto the stack of a root view. + Push a new view onto the existing view stack by passing a view + payload and a valid trigger_id generated from an interaction + within the existing modal. + Read the modals documentation (https://api.slack.com/block-kit/surfaces/modals) + to learn more about the lifecycle and intricacies of views. + https://api.slack.com/methods/views.push + """ + kwargs.update({"trigger_id": trigger_id}) + if isinstance(view, View): + kwargs.update({"view": view.to_dict()}) + else: + kwargs.update({"view": view}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "view" parameter + return self.api_call("views.push", json=kwargs) + + def views_update( + self, + *, + view: Union[dict, View], + external_id: Optional[str] = None, + view_id: Optional[str] = None, + hash: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Update an existing view. + Update a view by passing a new view definition along with the + view_id returned in views.open or the external_id. + See the modals documentation (https://api.slack.com/block-kit/surfaces/modals#updating_views) + to learn more about updating views and avoiding race conditions with the hash argument. + https://api.slack.com/methods/views.update + """ + if isinstance(view, View): + kwargs.update({"view": view.to_dict()}) + else: + kwargs.update({"view": view}) + if external_id: + kwargs.update({"external_id": external_id}) + elif view_id: + kwargs.update({"view_id": view_id}) + else: + raise e.SlackRequestError("Either view_id or external_id is required.") + kwargs.update({"hash": hash}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "view" parameter + return self.api_call("views.update", json=kwargs) + + def views_publish( + self, + *, + user_id: str, + view: Union[dict, View], + hash: Optional[str] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Publish a static view for a User. + Create or update the view that comprises an + app's Home tab (https://api.slack.com/surfaces/tabs) + https://api.slack.com/methods/views.publish + """ + kwargs.update({"user_id": user_id, "hash": hash}) + if isinstance(view, View): + kwargs.update({"view": view.to_dict()}) + else: + kwargs.update({"view": view}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "view" parameter + return self.api_call("views.publish", json=kwargs) + + def workflows_stepCompleted( + self, + *, + workflow_step_execute_id: str, + outputs: Optional[dict] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Indicate a successful outcome of a workflow step's execution. + https://api.slack.com/methods/workflows.stepCompleted + """ + kwargs.update({"workflow_step_execute_id": workflow_step_execute_id}) + if outputs is not None: + kwargs.update({"outputs": outputs}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "outputs" parameter + return self.api_call("workflows.stepCompleted", json=kwargs) + + def workflows_stepFailed( + self, + *, + workflow_step_execute_id: str, + error: Dict[str, str], + **kwargs, + ) -> Union[Future, SlackResponse]: + """Indicate an unsuccessful outcome of a workflow step's execution. + https://api.slack.com/methods/workflows.stepFailed + """ + kwargs.update( + { + "workflow_step_execute_id": workflow_step_execute_id, + "error": error, + } + ) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "error" parameter + return self.api_call("workflows.stepFailed", json=kwargs) + + def workflows_updateStep( + self, + *, + workflow_step_edit_id: str, + inputs: Optional[Dict[str, Any]] = None, + outputs: Optional[List[Dict[str, str]]] = None, + **kwargs, + ) -> Union[Future, SlackResponse]: + """Update the configuration for a workflow extension step. + https://api.slack.com/methods/workflows.updateStep + """ + kwargs.update({"workflow_step_edit_id": workflow_step_edit_id}) + if inputs is not None: + kwargs.update({"inputs": inputs}) + if outputs is not None: + kwargs.update({"outputs": outputs}) + kwargs = _remove_none_values(kwargs) + # NOTE: Intentionally using json for the "inputs" / "outputs" parameters + return self.api_call("workflows.updateStep", json=kwargs) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/web/legacy_slack_response.py b/core_service/aws_lambda/project/packages/slack_sdk/web/legacy_slack_response.py new file mode 100644 index 0000000..a9ca462 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/web/legacy_slack_response.py @@ -0,0 +1,223 @@ +"""A Python module for interacting and consuming responses from Slack.""" + +import asyncio + +# Standard Imports +import logging + +# Internal Imports +from typing import Union + +import slack_sdk.errors as e + + +class LegacySlackResponse(object): # skipcq: PYL-R0205 + """An iterable container of response data. + + Attributes: + data (dict): The json-encoded content of the response. Along + with the headers and status code information. + + Methods: + validate: Check if the response from Slack was successful. + get: Retrieves any key from the response data. + next: Retrieves the next portion of results, + if 'next_cursor' is present. + + Example: + ```python + import os + import slack + + client = slack.WebClient(token=os.environ['SLACK_API_TOKEN']) + + response1 = client.auth_revoke(test='true') + assert not response1['revoked'] + + response2 = client.auth_test() + assert response2.get('ok', False) + + users = [] + for page in client.users_list(limit=2): + TODO: This example should specify when to break. + users = users + page['members'] + ``` + + Note: + Some responses return collections of information + like channel and user lists. If they do it's likely + that you'll only receive a portion of results. This + object allows you to iterate over the response which + makes subsequent API requests until your code hits + 'break' or there are no more results to be found. + + Any attributes or methods prefixed with _underscores are + intended to be "private" internal use only. They may be changed or + removed at anytime. + """ + + def __init__( + self, + *, + client, + http_verb: str, + api_url: str, + req_args: dict, + data: Union[dict, bytes], # data can be binary data + headers: dict, + status_code: int, + use_sync_aiohttp: bool = True, # True for backward-compatibility + ): + self.http_verb = http_verb + self.api_url = api_url + self.req_args = req_args + self.data = data + self.headers = headers + self.status_code = status_code + self._initial_data = data + self._client = client # LegacyWebClient + self._use_sync_aiohttp = use_sync_aiohttp + self._logger = logging.getLogger(__name__) + + def __str__(self): + """Return the Response data if object is converted to a string.""" + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + return f"{self.data}" + + def __getitem__(self, key): + """Retrieves any key from the data store. + + Note: + This is implemented so users can reference the + SlackResponse object like a dictionary. + e.g. response["ok"] + + Returns: + The value from data or None. + """ + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + return self.data.get(key, None) + + def __iter__(self): + """Enables the ability to iterate over the response. + It's required for the iterator protocol. + + Note: + This enables Slack cursor-based pagination. + + Returns: + (SlackResponse) self + """ + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + self._iteration = 0 # skipcq: PYL-W0201 + self.data = self._initial_data + return self + + def __next__(self): + """Retrieves the next portion of results, if 'next_cursor' is present. + + Note: + Some responses return collections of information + like channel and user lists. If they do it's likely + that you'll only receive a portion of results. This + method allows you to iterate over the response until + your code hits 'break' or there are no more results + to be found. + + Returns: + (SlackResponse) self + With the new response data now attached to this object. + + Raises: + SlackApiError: If the request to the Slack API failed. + StopIteration: If 'next_cursor' is not present or empty. + """ + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + self._iteration += 1 + if self._iteration == 1: + return self + if self._next_cursor_is_present(self.data): # skipcq: PYL-R1705 + params = self.req_args.get("params", {}) + if params is None: + params = {} + params.update({"cursor": self.data["response_metadata"]["next_cursor"]}) + self.req_args.update({"params": params}) + + if self._use_sync_aiohttp: + # We no longer recommend going with this way + response = asyncio.get_event_loop().run_until_complete( + self._client._request( # skipcq: PYL-W0212 + http_verb=self.http_verb, + api_url=self.api_url, + req_args=self.req_args, + ) + ) + else: + # This method sends a request in a synchronous way + response = self._client._request_for_pagination( # skipcq: PYL-W0212 + api_url=self.api_url, req_args=self.req_args + ) + + self.data = response["data"] + self.headers = response["headers"] + self.status_code = response["status_code"] + return self.validate() + else: + raise StopIteration + + def get(self, key, default=None): + """Retrieves any key from the response data. + + Note: + This is implemented so users can reference the + SlackResponse object like a dictionary. + e.g. response.get("ok", False) + + Returns: + The value from data or the specified default. + """ + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + return self.data.get(key, default) + + def validate(self): + """Check if the response from Slack was successful. + + Returns: + (SlackResponse) + This method returns it's own object. e.g. 'self' + + Raises: + SlackApiError: The request to the Slack API failed. + """ + if self._logger.level <= logging.DEBUG: + body = self.data if isinstance(self.data, dict) else "(binary)" + self._logger.debug( + "Received the following response - " + f"status: {self.status_code}, " + f"headers: {dict(self.headers)}, " + f"body: {body}" + ) + if self.status_code == 200 and self.data and (isinstance(self.data, bytes) or self.data.get("ok", False)): + return self + msg = "The request to the Slack API failed." + raise e.SlackApiError(message=msg, response=self) + + @staticmethod + def _next_cursor_is_present(data): + """Determine if the response contains 'next_cursor' + and 'next_cursor' is not empty. + + Returns: + A boolean value. + """ + present = ( + "response_metadata" in data + and "next_cursor" in data["response_metadata"] + and data["response_metadata"]["next_cursor"] != "" + ) + return present diff --git a/core_service/aws_lambda/project/packages/slack_sdk/web/slack_response.py b/core_service/aws_lambda/project/packages/slack_sdk/web/slack_response.py new file mode 100644 index 0000000..936d496 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/web/slack_response.py @@ -0,0 +1,189 @@ +"""A Python module for interacting and consuming responses from Slack.""" + +import logging +from typing import Union + +import slack_sdk.errors as e +from .internal_utils import _next_cursor_is_present + + +class SlackResponse: + """An iterable container of response data. + + Attributes: + data (dict): The json-encoded content of the response. Along + with the headers and status code information. + + Methods: + validate: Check if the response from Slack was successful. + get: Retrieves any key from the response data. + next: Retrieves the next portion of results, + if 'next_cursor' is present. + + Example: + ```python + import os + import slack + + client = slack.WebClient(token=os.environ['SLACK_API_TOKEN']) + + response1 = client.auth_revoke(test='true') + assert not response1['revoked'] + + response2 = client.auth_test() + assert response2.get('ok', False) + + users = [] + for page in client.users_list(limit=2): + users = users + page['members'] + ``` + + Note: + Some responses return collections of information + like channel and user lists. If they do it's likely + that you'll only receive a portion of results. This + object allows you to iterate over the response which + makes subsequent API requests until your code hits + 'break' or there are no more results to be found. + + Any attributes or methods prefixed with _underscores are + intended to be "private" internal use only. They may be changed or + removed at anytime. + """ + + def __init__( + self, + *, + client, + http_verb: str, + api_url: str, + req_args: dict, + data: Union[dict, bytes], # data can be binary data + headers: dict, + status_code: int, + ): + self.http_verb = http_verb + self.api_url = api_url + self.req_args = req_args + self.data = data + self.headers = headers + self.status_code = status_code + self._initial_data = data + self._iteration = None # for __iter__ & __next__ + self._client = client + self._logger = logging.getLogger(__name__) + + def __str__(self): + """Return the Response data if object is converted to a string.""" + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + return f"{self.data}" + + def __contains__(self, key: str) -> bool: + return self.get(key) is not None + + def __getitem__(self, key): + """Retrieves any key from the data store. + + Note: + This is implemented so users can reference the + SlackResponse object like a dictionary. + e.g. response["ok"] + + Returns: + The value from data or None. + """ + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + if self.data is None: + raise ValueError("As the response.data is empty, this operation is unsupported") + return self.data.get(key, None) + + def __iter__(self): + """Enables the ability to iterate over the response. + It's required for the iterator protocol. + + Note: + This enables Slack cursor-based pagination. + + Returns: + (SlackResponse) self + """ + self._iteration = 0 + self.data = self._initial_data + return self + + def __next__(self): + """Retrieves the next portion of results, if 'next_cursor' is present. + + Note: + Some responses return collections of information + like channel and user lists. If they do it's likely + that you'll only receive a portion of results. This + method allows you to iterate over the response until + your code hits 'break' or there are no more results + to be found. + + Returns: + (SlackResponse) self + With the new response data now attached to this object. + + Raises: + SlackApiError: If the request to the Slack API failed. + StopIteration: If 'next_cursor' is not present or empty. + """ + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + self._iteration += 1 + if self._iteration == 1: + return self + if _next_cursor_is_present(self.data): # skipcq: PYL-R1705 + params = self.req_args.get("params", {}) + if params is None: + params = {} + next_cursor = self.data.get("response_metadata", {}).get("next_cursor") or self.data.get("next_cursor") + params.update({"cursor": next_cursor}) + self.req_args.update({"params": params}) + + # This method sends a request in a synchronous way + response = self._client._request_for_pagination( # skipcq: PYL-W0212 + api_url=self.api_url, req_args=self.req_args + ) + self.data = response["data"] + self.headers = response["headers"] + self.status_code = response["status_code"] + return self.validate() + else: + raise StopIteration + + def get(self, key, default=None): + """Retrieves any key from the response data. + + Note: + This is implemented so users can reference the + SlackResponse object like a dictionary. + e.g. response.get("ok", False) + + Returns: + The value from data or the specified default. + """ + if isinstance(self.data, bytes): + raise ValueError("As the response.data is binary data, this operation is unsupported") + if self.data is None: + return None + return self.data.get(key, default) + + def validate(self): + """Check if the response from Slack was successful. + + Returns: + (SlackResponse) + This method returns it's own object. e.g. 'self' + + Raises: + SlackApiError: The request to the Slack API failed. + """ + if self.status_code == 200 and self.data and (isinstance(self.data, bytes) or self.data.get("ok", False)): + return self + msg = f"The request to the Slack API failed. (url: {self.api_url})" + raise e.SlackApiError(message=msg, response=self) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/webhook/__init__.py b/core_service/aws_lambda/project/packages/slack_sdk/webhook/__init__.py new file mode 100644 index 0000000..ea52f6b --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/webhook/__init__.py @@ -0,0 +1,11 @@ +"""You can use slack_sdk.webhook.WebhookClient for Incoming Webhooks +and message responses using response_url in payloads. +""" +# from .async_client import AsyncWebhookClient +from .client import WebhookClient +from .webhook_response import WebhookResponse + +__all__ = [ + "WebhookClient", + "WebhookResponse", +] diff --git a/core_service/aws_lambda/project/packages/slack_sdk/webhook/async_client.py b/core_service/aws_lambda/project/packages/slack_sdk/webhook/async_client.py new file mode 100644 index 0000000..0198547 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/webhook/async_client.py @@ -0,0 +1,264 @@ +import json +import logging +from ssl import SSLContext +from typing import Dict, Union, Optional, Any, Sequence, List + +import aiohttp +from aiohttp import BasicAuth, ClientSession + +from slack_sdk.models.attachments import Attachment +from slack_sdk.models.blocks import Block +from .internal_utils import ( + _debug_log_response, + _build_request_headers, + _build_body, + get_user_agent, +) +from .webhook_response import WebhookResponse +from ..proxy_env_variable_loader import load_http_proxy_from_env + +from slack_sdk.http_retry.async_handler import AsyncRetryHandler +from slack_sdk.http_retry.builtin_async_handlers import async_default_handlers +from slack_sdk.http_retry.request import HttpRequest as RetryHttpRequest +from slack_sdk.http_retry.response import HttpResponse as RetryHttpResponse +from slack_sdk.http_retry.state import RetryState + + +class AsyncWebhookClient: + url: str + timeout: int + ssl: Optional[SSLContext] + proxy: Optional[str] + session: Optional[ClientSession] + trust_env_in_session: bool + auth: Optional[BasicAuth] + default_headers: Dict[str, str] + logger: logging.Logger + retry_handlers: List[AsyncRetryHandler] + + def __init__( + self, + url: str, + timeout: int = 30, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + session: Optional[ClientSession] = None, + trust_env_in_session: bool = False, + auth: Optional[BasicAuth] = None, + default_headers: Optional[Dict[str, str]] = None, + user_agent_prefix: Optional[str] = None, + user_agent_suffix: Optional[str] = None, + logger: Optional[logging.Logger] = None, + retry_handlers: Optional[List[AsyncRetryHandler]] = None, + ): + """API client for Incoming Webhooks and `response_url` + + https://api.slack.com/messaging/webhooks + + Args: + url: Complete URL to send data (e.g., `https://hooks.slack.com/XXX`) + timeout: Request timeout (in seconds) + ssl: `ssl.SSLContext` to use for requests + proxy: Proxy URL (e.g., `localhost:9000`, `http://localhost:9000`) + session: `aiohttp.ClientSession` instance + trust_env_in_session: True/False for `aiohttp.ClientSession` + auth: Basic auth info for `aiohttp.ClientSession` + default_headers: Request headers to add to all requests + user_agent_prefix: Prefix for User-Agent header value + user_agent_suffix: Suffix for User-Agent header value + logger: Custom logger + """ + self.url = url + self.timeout = timeout + self.ssl = ssl + self.proxy = proxy + self.trust_env_in_session = trust_env_in_session + self.session = session + self.auth = auth + self.default_headers = default_headers if default_headers else {} + self.default_headers["User-Agent"] = get_user_agent(user_agent_prefix, user_agent_suffix) + self.logger = logger if logger is not None else logging.getLogger(__name__) + self.retry_handlers = retry_handlers if retry_handlers is not None else async_default_handlers() + + if self.proxy is None or len(self.proxy.strip()) == 0: + env_variable = load_http_proxy_from_env(self.logger) + if env_variable is not None: + self.proxy = env_variable + + async def send( + self, + *, + text: Optional[str] = None, + attachments: Optional[Sequence[Union[Dict[str, Any], Attachment]]] = None, + blocks: Optional[Sequence[Union[Dict[str, Any], Block]]] = None, + response_type: Optional[str] = None, + replace_original: Optional[bool] = None, + delete_original: Optional[bool] = None, + unfurl_links: Optional[bool] = None, + unfurl_media: Optional[bool] = None, + headers: Optional[Dict[str, str]] = None, + ) -> WebhookResponse: + """Performs a Slack API request and returns the result. + + Args: + text: The text message (even when having blocks, setting this as well is recommended as it works as fallback) + attachments: A collection of attachments + blocks: A collection of Block Kit UI components + response_type: The type of message (either 'in_channel' or 'ephemeral') + replace_original: True if you use this option for response_url requests + delete_original: True if you use this option for response_url requests + unfurl_links: Option to indicate whether text url should unfurl + unfurl_media: Option to indicate whether media url should unfurl + headers: Request headers to append only for this request + + Returns: + Webhook response + """ + return await self.send_dict( + # It's fine to have None value elements here + # because _build_body() filters them out when constructing the actual body data + body={ + "text": text, + "attachments": attachments, + "blocks": blocks, + "response_type": response_type, + "replace_original": replace_original, + "delete_original": delete_original, + "unfurl_links": unfurl_links, + "unfurl_media": unfurl_media, + }, + headers=headers, + ) + + async def send_dict(self, body: Dict[str, Any], headers: Optional[Dict[str, str]] = None) -> WebhookResponse: + """Performs a Slack API request and returns the result. + + Args: + body: JSON data structure (it's still a dict at this point), + if you give this argument, body_params and files will be skipped + headers: Request headers to append only for this request + Returns: + Webhook response + """ + return await self._perform_http_request( + body=_build_body(body), + headers=_build_request_headers(self.default_headers, headers), + ) + + async def _perform_http_request(self, *, body: Dict[str, Any], headers: Dict[str, str]) -> WebhookResponse: + str_body: str = json.dumps(body) + headers["Content-Type"] = "application/json;charset=utf-8" + + session: Optional[ClientSession] = None + use_running_session = self.session and not self.session.closed + if use_running_session: + session = self.session + else: + session = aiohttp.ClientSession( + timeout=aiohttp.ClientTimeout(total=self.timeout), + auth=self.auth, + trust_env=self.trust_env_in_session, + ) + + last_error: Optional[Exception] = None + resp: Optional[WebhookResponse] = None + try: + request_kwargs = { + "headers": headers, + "data": str_body, + "ssl": self.ssl, + "proxy": self.proxy, + } + retry_request = RetryHttpRequest( + method="POST", + url=self.url, + headers=headers, + body_params=body, + ) + + retry_state = RetryState() + counter_for_safety = 0 + while counter_for_safety < 100: + counter_for_safety += 1 + # If this is a retry, the next try started here. We can reset the flag. + retry_state.next_attempt_requested = False + retry_response: Optional[RetryHttpResponse] = None + response_body = "" + + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"Sending a request - url: {self.url}, body: {str_body}, headers: {headers}") + + try: + async with session.request("POST", self.url, **request_kwargs) as res: + try: + response_body = await res.text() + retry_response = RetryHttpResponse( + status_code=res.status, + headers=res.headers, + data=response_body.encode("utf-8") if response_body is not None else None, + ) + except aiohttp.ContentTypeError: + self.logger.debug(f"No response data returned from the following API call: {self.url}") + + if res.status == 429: + for handler in self.retry_handlers: + if await handler.can_retry_async( + state=retry_state, + request=retry_request, + response=retry_response, + ): + if self.logger.level <= logging.DEBUG: + self.logger.info( + f"A retry handler found: {type(handler).__name__} " + f"for POST {self.url} - rate_limited" + ) + await handler.prepare_for_next_attempt_async( + state=retry_state, + request=retry_request, + response=retry_response, + ) + break + + if retry_state.next_attempt_requested is False: + resp = WebhookResponse( + url=self.url, + status_code=res.status, + body=response_body, + headers=res.headers, + ) + _debug_log_response(self.logger, resp) + return resp + + except Exception as e: + last_error = e + for handler in self.retry_handlers: + if await handler.can_retry_async( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ): + if self.logger.level <= logging.DEBUG: + self.logger.info( + f"A retry handler found: {type(handler).__name__} " f"for POST {self.url} - {e}" + ) + await handler.prepare_for_next_attempt_async( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ) + break + + if retry_state.next_attempt_requested is False: + raise last_error + + if resp is not None: + return resp + raise last_error + + finally: + if not use_running_session: + await session.close() + + return resp diff --git a/core_service/aws_lambda/project/packages/slack_sdk/webhook/client.py b/core_service/aws_lambda/project/packages/slack_sdk/webhook/client.py new file mode 100644 index 0000000..17df0b7 --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/webhook/client.py @@ -0,0 +1,277 @@ +import json +import logging +import urllib +from http.client import HTTPResponse +from ssl import SSLContext +from typing import Dict, Union, Sequence, Optional, List, Any +from urllib.error import HTTPError +from urllib.request import Request, urlopen, OpenerDirector, ProxyHandler, HTTPSHandler + +from slack_sdk.errors import SlackRequestError +from slack_sdk.models.attachments import Attachment +from slack_sdk.models.blocks import Block +from .internal_utils import ( + _build_body, + _build_request_headers, + _debug_log_response, + get_user_agent, +) +from .webhook_response import WebhookResponse +from slack_sdk.http_retry import default_retry_handlers +from slack_sdk.http_retry.handler import RetryHandler +from slack_sdk.http_retry.request import HttpRequest as RetryHttpRequest +from slack_sdk.http_retry.response import HttpResponse as RetryHttpResponse +from slack_sdk.http_retry.state import RetryState +from ..proxy_env_variable_loader import load_http_proxy_from_env + + +class WebhookClient: + url: str + timeout: int + ssl: Optional[SSLContext] + proxy: Optional[str] + default_headers: Dict[str, str] + logger: logging.Logger + retry_handlers: List[RetryHandler] + + def __init__( + self, + url: str, + timeout: int = 30, + ssl: Optional[SSLContext] = None, + proxy: Optional[str] = None, + default_headers: Optional[Dict[str, str]] = None, + user_agent_prefix: Optional[str] = None, + user_agent_suffix: Optional[str] = None, + logger: Optional[logging.Logger] = None, + retry_handlers: Optional[List[RetryHandler]] = None, + ): + """API client for Incoming Webhooks and `response_url` + + https://api.slack.com/messaging/webhooks + + Args: + url: Complete URL to send data (e.g., `https://hooks.slack.com/XXX`) + timeout: Request timeout (in seconds) + ssl: `ssl.SSLContext` to use for requests + proxy: Proxy URL (e.g., `localhost:9000`, `http://localhost:9000`) + default_headers: Request headers to add to all requests + user_agent_prefix: Prefix for User-Agent header value + user_agent_suffix: Suffix for User-Agent header value + logger: Custom logger + retry_handlers: Retry handlers + """ + self.url = url + self.timeout = timeout + self.ssl = ssl + self.proxy = proxy + self.default_headers = default_headers if default_headers else {} + self.default_headers["User-Agent"] = get_user_agent(user_agent_prefix, user_agent_suffix) + self.logger = logger if logger is not None else logging.getLogger(__name__) + self.retry_handlers = retry_handlers if retry_handlers is not None else default_retry_handlers() + + if self.proxy is None or len(self.proxy.strip()) == 0: + env_variable = load_http_proxy_from_env(self.logger) + if env_variable is not None: + self.proxy = env_variable + + def send( + self, + *, + text: Optional[str] = None, + attachments: Optional[Sequence[Union[Dict[str, Any], Attachment]]] = None, + blocks: Optional[Sequence[Union[Dict[str, Any], Block]]] = None, + response_type: Optional[str] = None, + replace_original: Optional[bool] = None, + delete_original: Optional[bool] = None, + unfurl_links: Optional[bool] = None, + unfurl_media: Optional[bool] = None, + headers: Optional[Dict[str, str]] = None, + ) -> WebhookResponse: + """Performs a Slack API request and returns the result. + + Args: + text: The text message + (even when having blocks, setting this as well is recommended as it works as fallback) + attachments: A collection of attachments + blocks: A collection of Block Kit UI components + response_type: The type of message (either 'in_channel' or 'ephemeral') + replace_original: True if you use this option for response_url requests + delete_original: True if you use this option for response_url requests + unfurl_links: Option to indicate whether text url should unfurl + unfurl_media: Option to indicate whether media url should unfurl + headers: Request headers to append only for this request + + Returns: + Webhook response + """ + return self.send_dict( + # It's fine to have None value elements here + # because _build_body() filters them out when constructing the actual body data + body={ + "text": text, + "attachments": attachments, + "blocks": blocks, + "response_type": response_type, + "replace_original": replace_original, + "delete_original": delete_original, + "unfurl_links": unfurl_links, + "unfurl_media": unfurl_media, + }, + headers=headers, + ) + + def send_dict(self, body: Dict[str, Any], headers: Optional[Dict[str, str]] = None) -> WebhookResponse: + """Performs a Slack API request and returns the result. + + Args: + body: JSON data structure (it's still a dict at this point), + if you give this argument, body_params and files will be skipped + headers: Request headers to append only for this request + Returns: + Webhook response + """ + return self._perform_http_request( + body=_build_body(body), + headers=_build_request_headers(self.default_headers, headers), + ) + + def _perform_http_request(self, *, body: Dict[str, Any], headers: Dict[str, str]) -> WebhookResponse: + body = json.dumps(body) + headers["Content-Type"] = "application/json;charset=utf-8" + + if self.logger.level <= logging.DEBUG: + self.logger.debug(f"Sending a request - url: {self.url}, body: {body}, headers: {headers}") + + url = self.url + # NOTE: Intentionally ignore the `http_verb` here + # Slack APIs accepts any API method requests with POST methods + req = Request(method="POST", url=url, data=body.encode("utf-8"), headers=headers) + resp = None + last_error = None + + retry_state = RetryState() + counter_for_safety = 0 + while counter_for_safety < 100: + counter_for_safety += 1 + # If this is a retry, the next try started here. We can reset the flag. + retry_state.next_attempt_requested = False + + try: + resp = self._perform_http_request_internal(url, req) + # The resp is a 200 OK response + return resp + + except HTTPError as e: + # read the response body here + charset = e.headers.get_content_charset() or "utf-8" + response_body: str = e.read().decode(charset) + # As adding new values to HTTPError#headers can be ignored, building a new dict object here + response_headers = dict(e.headers.items()) + resp = WebhookResponse( + url=url, + status_code=e.code, + body=response_body, + headers=response_headers, + ) + if e.code == 429: + # for backward-compatibility with WebClient (v.2.5.0 or older) + if "retry-after" not in resp.headers and "Retry-After" in resp.headers: + resp.headers["retry-after"] = resp.headers["Retry-After"] + if "Retry-After" not in resp.headers and "retry-after" in resp.headers: + resp.headers["Retry-After"] = resp.headers["retry-after"] + _debug_log_response(self.logger, resp) + + # Try to find a retry handler for this error + retry_request = RetryHttpRequest.from_urllib_http_request(req) + retry_response = RetryHttpResponse( + status_code=e.code, + headers={k: [v] for k, v in e.headers.items()}, + data=response_body.encode("utf-8") if response_body is not None else None, + ) + for handler in self.retry_handlers: + if handler.can_retry( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ): + if self.logger.level <= logging.DEBUG: + self.logger.info( + f"A retry handler found: {type(handler).__name__} for {req.method} {req.full_url} - {e}" + ) + handler.prepare_for_next_attempt( + state=retry_state, + request=retry_request, + response=retry_response, + error=e, + ) + break + + if retry_state.next_attempt_requested is False: + return resp + + except Exception as err: + last_error = err + self.logger.error(f"Failed to send a request to Slack API server: {err}") + + # Try to find a retry handler for this error + retry_request = RetryHttpRequest.from_urllib_http_request(req) + for handler in self.retry_handlers: + if handler.can_retry( + state=retry_state, + request=retry_request, + response=None, + error=err, + ): + if self.logger.level <= logging.DEBUG: + self.logger.info( + f"A retry handler found: {type(handler).__name__} for {req.method} {req.full_url} - {err}" + ) + handler.prepare_for_next_attempt( + state=retry_state, + request=retry_request, + response=None, + error=err, + ) + self.logger.info(f"Going to retry the same request: {req.method} {req.full_url}") + break + + if retry_state.next_attempt_requested is False: + raise err + + if resp is not None: + return resp + raise last_error + + def _perform_http_request_internal(self, url: str, req: Request): + opener: Optional[OpenerDirector] = None + # for security (BAN-B310) + if url.lower().startswith("http"): + if self.proxy is not None: + if isinstance(self.proxy, str): + opener = urllib.request.build_opener( + ProxyHandler({"http": self.proxy, "https": self.proxy}), + HTTPSHandler(context=self.ssl), + ) + else: + raise SlackRequestError(f"Invalid proxy detected: {self.proxy} must be a str value") + else: + raise SlackRequestError(f"Invalid URL detected: {url}") + + # NOTE: BAN-B310 is already checked above + http_resp: Optional[HTTPResponse] = None + if opener: + http_resp = opener.open(req, timeout=self.timeout) # skipcq: BAN-B310 + else: + http_resp = urlopen(req, context=self.ssl, timeout=self.timeout) # skipcq: BAN-B310 + charset: str = http_resp.headers.get_content_charset() or "utf-8" + response_body: str = http_resp.read().decode(charset) + resp = WebhookResponse( + url=url, + status_code=http_resp.status, + body=response_body, + headers=http_resp.headers, + ) + _debug_log_response(self.logger, resp) + return resp diff --git a/core_service/aws_lambda/project/packages/slack_sdk/webhook/internal_utils.py b/core_service/aws_lambda/project/packages/slack_sdk/webhook/internal_utils.py new file mode 100644 index 0000000..1dd94de --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/webhook/internal_utils.py @@ -0,0 +1,45 @@ +import logging +from typing import Optional, Dict, Any + +from slack_sdk.web.internal_utils import ( + _parse_web_class_objects, + get_user_agent, +) +from .webhook_response import WebhookResponse + + +def _build_body(original_body: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any]]: + if original_body: + body = {k: v for k, v in original_body.items() if v is not None} + _parse_web_class_objects(body) + return body + return None + + +def _build_request_headers( + default_headers: Dict[str, str], + additional_headers: Optional[Dict[str, str]], +) -> Dict[str, str]: + if default_headers is None and additional_headers is None: + return {} + + request_headers = { + "Content-Type": "application/json;charset=utf-8", + } + if default_headers is None or "User-Agent" not in default_headers: + request_headers["User-Agent"] = get_user_agent() + + request_headers.update(default_headers) + if additional_headers: + request_headers.update(additional_headers) + return request_headers + + +def _debug_log_response(logger, resp: WebhookResponse) -> None: + if logger.level <= logging.DEBUG: + logger.debug( + "Received the following response - " + f"status: {resp.status_code}, " + f"headers: {(dict(resp.headers))}, " + f"body: {resp.body}" + ) diff --git a/core_service/aws_lambda/project/packages/slack_sdk/webhook/webhook_response.py b/core_service/aws_lambda/project/packages/slack_sdk/webhook/webhook_response.py new file mode 100644 index 0000000..c45eeba --- /dev/null +++ b/core_service/aws_lambda/project/packages/slack_sdk/webhook/webhook_response.py @@ -0,0 +1,16 @@ +from typing import Dict, Any + + +class WebhookResponse: + def __init__( + self, + *, + url: str, + status_code: int, + body: str, + headers: Dict[str, Any], + ): + self.api_url = url + self.status_code = status_code + self.body = body + self.headers = headers diff --git a/core_service/deploy_core_service.py b/core_service/deploy_core_service.py index 094839e..292fa17 100644 --- a/core_service/deploy_core_service.py +++ b/core_service/deploy_core_service.py @@ -13,13 +13,11 @@ MODE_DB = '' #'stage_' GENERAL_SETUP = { - 'MODE': 'staging', - 'BUCKET_NAME': 'daita-client-data', - 'USER_POOL_ID': 'us-east-2_ZbwpnYN4g', - 'IDENTITY_POOL_ID': 'us-east-2:fa0b76bc-01fa-4bb8-b7cf-a5000954aafb', - 'CLIENT_ID':'4cpbb5etp3q7grnnrhrc7irjoa', - 'aws_access_key_id':'AKIAVKWNZXMINQ6JTYXY', - 'aws_secret_access_key':'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', + 'MODE': os.environ.get('MODE','staging'), + 'BUCKET_NAME': os.environ.get('BUCKET_NAME','daita-client-data'), + 'USER_POOL_ID': os.environ.get('USER_POOL_ID','us-east-2_ZbwpnYN4g'), + 'IDENTITY_POOL_ID': os.environ.get('IDENTITY_POOL_ID','us-east-2:fa0b76bc-01fa-4bb8-b7cf-a5000954aafb'), + 'CLIENT_ID':os.environ.get('CLIENT_ID','4cpbb5etp3q7grnnrhrc7irjoa'), 'DOWNLOAD_SERVICE_URL': '3.140.206.255', 'T_PROJECT' : MODE_DB + 'projects', @@ -33,7 +31,7 @@ 'T_INSTANCES': MODE_DB + 'ec2', 'T_EC2_TASK': MODE_DB + 'ec2_task', - 'T_TASKS' : MODE_DB + 'dev-generate-tasks', + 'T_TASKS' :os.environ.get('T_TASKS',MODE_DB + 'dev-generate-tasks'), 'T_METHODS' : MODE_DB + 'methods', "T_QUOTAS": MODE_DB + "quotas", "T_TASK_DOWNLOAD": MODE_DB + "down_tasks", diff --git a/core_service/dynamoDB/de_dynamodb.py b/core_service/dynamoDB/de_dynamodb.py index ddaaac0..2bf5472 100644 --- a/core_service/dynamoDB/de_dynamodb.py +++ b/core_service/dynamoDB/de_dynamodb.py @@ -1,21 +1,37 @@ from .dynamodb_service import DynamoDBService + def deploy_dynamoDB(general_info): dbservice = DynamoDBService() - dbservice.create_bd(general_info['T_PROJECT'], [('identity_id', 'HASH', 'S'), ('project_name', 'RANGE', 'S')], ) #range means short key, hash means partition key - dbservice.create_bd(general_info['T_PROJECT_DEL'], [('identity_id', 'HASH', 'S'), ('project_name', 'RANGE', 'S')], ) #range means short key, hash means partition key - dbservice.create_bd(general_info['T_DATA_ORI'], [('project_id', 'HASH', 'S'), ('filename', 'RANGE', 'S')], ) - dbservice.create_bd(general_info['T_DATA_PREPROCESS'], [('project_id', 'HASH', 'S'), ('filename', 'RANGE', 'S')], ) - dbservice.create_bd(general_info['T_DATA_AUGMENT'], [('project_id', 'HASH', 'S'), ('filename', 'RANGE', 'S')], ) - dbservice.create_bd(general_info['T_PROJECT_SUMMARY'], [('project_id', 'HASH', 'S'), ('type', 'RANGE', 'S')],) # type is one of 'ORI', 'PRE', 'AUG' - dbservice.create_bd(general_info['T_TASKS'], [('identity_id', 'HASH', 'S'), ('task_id', 'RANGE', 'S')],) - dbservice.create_bd(general_info['T_METHODS'], [('method_id', 'HASH', 'S'), ('method_name', 'RANGE', 'S')],) - dbservice.create_bd(general_info['T_QUOTAS'], [('identity_id', 'HASH', 'S'), ('type', 'RANGE', 'S')],) - dbservice.create_bd(general_info['T_INSTANCES'], [('ec2_id', 'HASH', 'S'), ('assi_id', 'RANGE', 'S')],) - dbservice.create_bd(general_info['T_EC2_TASK'], [('ec2_id', 'HASH', 'S'), ('task_id', 'RANGE', 'S')],) - dbservice.create_bd(general_info['T_TASK_DOWNLOAD'], [('identity_id', 'HASH', 'S'), ('task_id', 'RANGE', 'S')],) - dbservice.create_bd(general_info['T_TRIGGER_SEND_CODE'], [('user', 'HASH', 'S'),('code', 'RANGE', 'S')],) - dbservice.create_bd(general_info['T_EVENT_USER'], [('event_ID', 'HASH', 'S'), ('type', 'RANGE', 'S')],) - dbservice.create_bd(general_info['T_USER'], [('ID', 'HASH', 'S'), ('username', 'RANGE', 'S')],) - dbservice.create_bd(general_info['T_FEEDBACK'],[('ID','HASH','S'),]) \ No newline at end of file + dbservice.create_bd(general_info['T_PROJECT'], [('identity_id', 'HASH', 'S'), ( + 'project_name', 'RANGE', 'S')], ) # range means short key, hash means partition key + dbservice.create_bd(general_info['T_PROJECT_DEL'], [('identity_id', 'HASH', 'S'), ( + 'project_name', 'RANGE', 'S')], ) # range means short key, hash means partition key + dbservice.create_bd(general_info['T_DATA_ORI'], [ + ('project_id', 'HASH', 'S'), ('filename', 'RANGE', 'S')], ) + dbservice.create_bd(general_info['T_DATA_PREPROCESS'], [ + ('project_id', 'HASH', 'S'), ('filename', 'RANGE', 'S')], ) + dbservice.create_bd(general_info['T_DATA_AUGMENT'], [ + ('project_id', 'HASH', 'S'), ('filename', 'RANGE', 'S')], ) + dbservice.create_bd(general_info['T_PROJECT_SUMMARY'], [( + 'project_id', 'HASH', 'S'), ('type', 'RANGE', 'S')],) # type is one of 'ORI', 'PRE', 'AUG' + dbservice.create_bd(general_info['T_TASKS'], [ + ('identity_id', 'HASH', 'S'), ('task_id', 'RANGE', 'S')],) + dbservice.create_bd(general_info['T_METHODS'], [ + ('method_id', 'HASH', 'S'), ('method_name', 'RANGE', 'S')],) + dbservice.create_bd(general_info['T_QUOTAS'], [ + ('identity_id', 'HASH', 'S'), ('type', 'RANGE', 'S')],) + dbservice.create_bd(general_info['T_INSTANCES'], [ + ('ec2_id', 'HASH', 'S'), ('assi_id', 'RANGE', 'S')],) + dbservice.create_bd(general_info['T_EC2_TASK'], [ + ('ec2_id', 'HASH', 'S'), ('task_id', 'RANGE', 'S')],) + dbservice.create_bd(general_info['T_TASK_DOWNLOAD'], [ + ('identity_id', 'HASH', 'S'), ('task_id', 'RANGE', 'S')],) + dbservice.create_bd(general_info['T_TRIGGER_SEND_CODE'], [ + ('user', 'HASH', 'S'), ('code', 'RANGE', 'S')],) + dbservice.create_bd(general_info['T_EVENT_USER'], [ + ('event_ID', 'HASH', 'S'), ('type', 'RANGE', 'S')],) + dbservice.create_bd(general_info['T_USER'], [ + ('ID', 'HASH', 'S'), ('username', 'RANGE', 'S')],) + dbservice.create_bd(general_info['T_FEEDBACK'], [('ID', 'HASH', 'S'), ]) diff --git a/daita-app/ai-caller-service/ai_caller_service_template.yaml b/daita-app/ai-caller-service/ai_caller_service_template.yaml index bbec955..6a01df6 100644 --- a/daita-app/ai-caller-service/ai_caller_service_template.yaml +++ b/daita-app/ai-caller-service/ai_caller_service_template.yaml @@ -21,6 +21,11 @@ Globals: LOGGING: !Ref minimumLogLevel TABLE_GENERATE_TASK: !Ref TableGenerateTaskName ROOTEFS: !Ref ROOTEFS + EFSPATH: !Ref EFSStorageAI + MODE: !Ref Mode + TableProjectsName: !Ref TableProjectsName + TableDataOriginalName: !Ref TableDataOriginalName + TableDataPreprocessName: !Ref TableDataPreprocessName Layers: - !Ref CommonCodeLayerName @@ -37,6 +42,7 @@ Parameters: Type: String ApplicationPara: Type: String + EFSFileSystemId: Type: String Default: fs-0199771f2dfe97ace @@ -45,7 +51,7 @@ Parameters: Default: /mnt/generation-task ROOTEFS: Type: String - Default: /efs + SubnetIDsPara: Type: CommaDelimitedList SecurityGroupIdsPara: @@ -54,12 +60,7 @@ Parameters: Type: String TableGenerateTaskName: Type: String - TableProjectSumAll: - Type: String - Default: prj_sum_all - LambdaRoleARN: - Type: String - LambdaRoleARN: + TableProjectSumName: Type: String LambdaRoleARN: Type: String @@ -69,7 +70,33 @@ Parameters: Type: String TableDataPreprocessName: Type: String + TableProjectsName: + Type: String + Mode: + Type: String + + TaskAIPreprocessingDefinition: + Type: String + TaskAIAugmentationDefinition: + Type: String + + FuncProjectUploadUpdate: + Type: String + ReferenceImageStateMachineArn: + Type: String + RICalculateFunction: + Type: String + RIStatusFunction: + Type: String + RIInfoFunction: + Type: String + + ### for const table group + TableLsEc2Name: + Type: String + Resources: + LambdaExecutionRole: Type: AWS::IAM::Role Properties: @@ -207,7 +234,85 @@ Resources: Resource: - !Ref CallerServiceStateMachine + # Sub service + # ECSTasks: + # Type: AWS::Serverless::Application + # Properties: + # Location: ./ecs-tasks/template.yaml + # Parameters: + # SecurityGroupIds: !Join [",", !Ref SecurityGroupIdsPara] + # SubnetIds: !Join [",", !Ref SubnetIDsPara] + # StagePara: !Ref StagePara + # EFSFileSystemId: !Ref EFSFileSystemId + # EFSAccessPoint: !Ref EFSAccessPoint + # StagePara: !Ref StagePara + # ApplicationPara: !Ref ApplicationPara + #================ LAMBDA FUNCTIONS ========================================== + + ReferenceImageCalculateFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/download_task/get_reference_images + Environment: + Variables: + REFERENCE_IMAGE_SM_ARN: !Ref ReferenceImageStateMachineArn + FUNC_RI_CALCULATION: !Ref RICalculateFunction + FUNC_RI_STATUS: !Ref RIStatusFunction + FUNC_RI_INFO: !Ref RIInfoFunction + Policies: + - Statement: + - Effect: Allow + Action: + - "sqs:*" + Resource: "*" + - Sid: CloudWatchLogsPolicy + Effect: Allow + Action: + - "logs:CreateLogDelivery" + - "logs:GetLogDelivery" + - "logs:UpdateLogDelivery" + - "logs:DeleteLogDelivery" + - "logs:ListLogDeliveries" + - "logs:PutResourcePolicy" + - "logs:DescribeResourcePolicies" + - "logs:DescribeLogGroup" + - "logs:DescribeLogGroups" + Resource: "*" + - Effect: Allow + Action: + - "dynamodb:GetItem" + - "dynamodb:DeleteItem" + - "dynamodb:PutItem" + - "dynamodb:Scan" + - "dynamodb:Query" + - "dynamodb:UpdateItem" + - "dynamodb:BatchWriteItem" + - "dynamodb:BatchGetItem" + - "dynamodb:DescribeTable" + - "dynamodb:ConditionCheckItem" + - "lambda:InvokeFunction" + Resource: "*" + - Sid: CloudWatchEventsFullAccess + Effect: Allow + Action: + - "events:*" + Resource: "*" + - Sid: AmazonElasticFileSystemClientFullAccess + Effect: Allow + Action: + - elasticfilesystem:ClientMount + - elasticfilesystem:ClientRootAccess + - elasticfilesystem:ClientWrite + - elasticfilesystem:DescribeMountTargets + Resource: "*" + - Effect: Allow + Action: + - s3:GetObject + - s3:PutObject + - s3:PutObjectAcl + - states:* + Resource: "*" StopTaskStop: Type: AWS::Serverless::Function DependsOn: CallerServiceStateMachineLogGroup @@ -253,17 +358,11 @@ Resources: CodeUri: functions/handlers/generate_step/CompleteRequestAI Environment: Variables: - TABLE_GENERATE_TASK: !Ref TableGenerateTaskName TABLE_DATA_AUGMENT: !Ref TableDataAugmentName TABLE_DATA_ORIGINAL: !Ref TableDataOriginalName TABLE_DATA_PREPROCESS: !Ref TableDataPreprocessName - T_PROJECT_SUMMARY: !Ref TableProjectSumAll - # Events: - # SQSQueueEvent: - # Type: SQS - # Properties: - # BatchSize: 10 - # Queue: !GetAtt UpdateDatabaseRequestAI.Arn + TABLE_PROJECT_SUMMARY: !Ref TableProjectSumName + FUNC_DAITA_UPLOAD_UPDATE: !Ref FuncProjectUploadUpdate Policies: - Statement: - @@ -329,6 +428,7 @@ Resources: QUEUE_EC2_NAME_1: !GetAtt SqsEc2Work1.QueueName QUEUE_EC2_NAME_2: !GetAtt SqsEc2Work2.QueueName SF_CALL_SERVICE: !GetAtt CallerServiceStateMachine.Arn + TABLE_LS_EC2: !Ref TableLsEc2Name Events: ScheduledEvent: Type: Schedule @@ -465,6 +565,45 @@ Resources: - s3:PutObjectAcl Resource: "*" + # GlobOutputFiles: + # Type: AWS::Serverless::Function + # Properties: + # CodeUri: functions/handlers/generate_step/glob_output_files + # VpcConfig: + # SecurityGroupIds: !Split [',', !Join [',', !Ref SecurityGroupIdsPara]] + # SubnetIds: !Split [',', !Join [',', !Ref SubnetIDsPara]] + # FileSystemConfigs: + # - Arn: !GetAtt EFSAccessPoint.Arn + # LocalMountPath: !Ref EFSStorageAI + # Policies: + # - Statement: + # - Sid: CloudWatchLogsPolicy + # Effect: Allow + # Action: + # - "logs:CreateLogDelivery" + # - "logs:GetLogDelivery" + # - "logs:UpdateLogDelivery" + # - "logs:DeleteLogDelivery" + # - "logs:ListLogDeliveries" + # - "logs:PutResourcePolicy" + # - "logs:DescribeResourcePolicies" + # - "logs:DescribeLogGroup" + # - "logs:DescribeLogGroups" + # Resource: "*" + # - Sid: CloudWatchEventsFullAccess + # Effect: Allow + # Action: + # - "events:*" + # Resource: "*" + # - Sid: AmazonElasticFileSystemClientFullAccess + # Effect: Allow + # Action: + # - elasticfilesystem:ClientMount + # - elasticfilesystem:ClientRootAccess + # - elasticfilesystem:ClientWrite + # - elasticfilesystem:DescribeMountTargets + # Resource: "*" + HandleMergeResultRequestAI: Type: AWS::Serverless::Function Properties: @@ -529,6 +668,19 @@ Resources: Type: AWS::Serverless::Function Properties: CodeUri: functions/handlers/generate_step/RequestAI + # VpcConfig: + # SecurityGroupIds: !Split [',', !Join [',', !Ref SecurityGroupIdsPara]] + # SubnetIds: !Split [',', !Join [',', !Ref SubnetIDsPara]] + # FileSystemConfigs: + # - Arn: !GetAtt EFSAccessPoint.Arn + # LocalMountPath: !Ref EFSStorageAI + # Environment: + # Variables: + # PREPROCESSING_TASK_DEFINITION: abc #!GetAtt ECSTasks.Outputs.AIPreprocessingTask + # CLUSTER_NAME: abc #!GetAtt ECSTasks.Outputs.ECSCluster + # SECURITY_GROUP_IDs: !Join [",", !Ref SecurityGroupIdsPara] + # SUBNET_IDs : !Join [",", !Ref SubnetIDsPara] + # EFSLocalMountPath: !Ref EFSStorageAI Policies: - Statement: - Effect: Allow @@ -597,13 +749,65 @@ Resources: Action: - s3:* Resource: "*" + - Effect: Allow + Action: + - ecs:RunTask + # - ec2:DescribeNetworkInterfaces + # - ec2:CreateNetworkInterface + # - ec2:DeleteNetworkInterface + # - ec2:DescribeInstances + # - ec2:AttachNetworkInterface + - ec2:* + - elasticfilesystem:ClientMount + - elasticfilesystem:ClientRootAccess + - elasticfilesystem:ClientWrite + - elasticfilesystem:DescribeMountTargets + Resource: "*" + # - Sid: AmazonElasticFileSystemClientFullAccess + # Effect: Allow + # Action: + # - elasticfilesystem:ClientMount + # - elasticfilesystem:ClientRootAccess + # - elasticfilesystem:ClientWrite + # - elasticfilesystem:DescribeMountTargets + # Resource: "*" + # - PolicyName: "ECSRunTaskPermission" + # PolicyDocument: + # Statement: + # - + # Effect: Allow + # Action: + # - ecs:RunTask + # Resource: "*" + # - PolicyName: "IAMPassRolePermission" + # PolicyDocument: + # Statement: + # - + # Effect: Allow + # Action: + # - iam:PassRole + # Resource: "*" + # - PolicyName: 'AllowEC2NetworkInterface' + # PolicyDocument: + # Version: '2012-10-17' + # Statement: + # - Effect: Allow + # Action: + # - ec2:DescribeNetworkInterfaces + # - ec2:CreateNetworkInterface + # - ec2:DeleteNetworkInterface + # - ec2:DescribeInstances + # - ec2:AttachNetworkInterface + # Resource: + # - "*" + UpdateStatusTask: Type: AWS::Serverless::Function Properties: CodeUri: functions/handlers/generate_step/updateStatusTask VpcConfig: - SecurityGroupIds: !Split [',', !Join [',', !Ref SecurityGroupIdsPara]] - SubnetIds: !Split [',', !Join [',', !Ref SubnetIDsPara]] + SecurityGroupIds: !Split [',', !Join [',', !Ref SecurityGroupIdsPara]] + SubnetIds: !Split [',', !Join [',', !Ref SubnetIDsPara]] FileSystemConfigs: - Arn: !GetAtt EFSAccessPoint.Arn LocalMountPath: !Ref EFSStorageAI @@ -633,6 +837,11 @@ Resources: Type: AWS::Serverless::Function Properties: CodeUri: functions/handlers/preprocess_generate_batch/generate_batch + Environment: + Variables: + TableProjectsName: !Ref TableProjectsName + TableDataOriginalName: !Ref TableDataOriginalName + TableDataPreprocessName: !Ref TableDataPreprocessName Policies: - Statement: - Sid: ALLOWCRUDDynamoDB @@ -654,8 +863,8 @@ Resources: Type: AWS::Serverless::Function Properties: VpcConfig: - SecurityGroupIds: !Split [',', !Join [',', !Ref SecurityGroupIdsPara]] - SubnetIds: !Split [',', !Join [',', !Ref SubnetIDsPara]] + SecurityGroupIds: !Split [',', !Join [',', !Ref SecurityGroupIdsPara]] + SubnetIds: !Split [',', !Join [',', !Ref SubnetIDsPara]] FileSystemConfigs: - Arn: !GetAtt EFSAccessPoint.Arn LocalMountPath: !Ref EFSStorageAI @@ -832,6 +1041,7 @@ Resources: Variables: QUEUE_EC2_NAME_1: !GetAtt SqsEc2Work1.QueueName QUEUE_EC2_NAME_2: !GetAtt SqsEc2Work2.QueueName + TABLE_LS_EC2: !Ref TableLsEc2Name Policies: - SQSSendMessagePolicy: QueueName: !GetAtt SqsEc2Work1.QueueName @@ -904,7 +1114,7 @@ Resources: MAX_CONCURRENCY_TASKS : !Ref MaxConcurrencyTasks TABLE_DATA_AUGMENT: !Ref TableDataAugmentName TABLE_DATA_PREPROCESS: !Ref TableDataPreprocessName - T_PROJECT_SUMMARY: !Ref TableProjectSumAll + TABLE_PROJECT_SUMMARY: !Ref TableProjectSumName Events: ScheduledEvent: Type: Schedule @@ -977,7 +1187,7 @@ Resources: Type: AWS::Events::EventBus Properties: Name: !Sub "${StagePara}-${ApplicationPara}-StopProcessEventBus" - + ProcessAITaskEventBus: Type: AWS::Events::EventBus Properties: @@ -1147,6 +1357,7 @@ Resources: FunctionName: !Ref HandleDownloadImages FunctionName: !Ref HandleMergeResultDownloadImages FunctionName: !Ref HandleGetResultDownloadTask + FunctionName: !Ref ReferenceImageCalculateFunction - Statement: - Sid: ALLOWCRUDDynamoDB Effect: Allow @@ -1191,6 +1402,7 @@ Resources: HandleDownloadImages: !GetAtt HandleDownloadImages.Arn HandleMergeResultDownloadImages: !GetAtt HandleMergeResultDownloadImages.Arn HandleGetResultDownloadTask: !GetAtt HandleGetResultDownloadTask.Arn + ReferenceImageCalculateFunction: !GetAtt ReferenceImageCalculateFunction.Arn ############################################################################################################ GenerateStateMachine: Type: AWS::Serverless::StateMachine @@ -1199,6 +1411,15 @@ Resources: Properties: Type: STANDARD Name: !Sub "${StagePara}-${ApplicationPara}-GenerateSM" + DefinitionSubstitutions: + HandleGenerateStep: !GetAtt HandleGenerateStep.Arn + WorkerRequestAI: !GetAtt WorkerRequestAI.Arn + HandleMergeResultRequestAI: !GetAtt HandleMergeResultRequestAI.Arn + HandleCompleteRequestAI: !GetAtt HandleCompleteRequestAI.Arn + HandlerUploadBatchToS3: !GetAtt HandlerUploadBatchToS3.Arn + UpdateStatusTask: !GetAtt UpdateStatusTask.Arn + TaskAIPreprocessingDefinition: !Ref TaskAIPreprocessingDefinition + TaskAIAugmentationDefinition: !Ref TaskAIAugmentationDefinition Policies: - LambdaInvokePolicy: FunctionName: !Ref HandleGenerateStep @@ -1258,14 +1479,6 @@ Resources: Destinations: - CloudWatchLogsLogGroup: LogGroupArn: !GetAtt CallerServiceStateMachineLogGroup.Arn - DefinitionSubstitutions: - HandleGenerateStep: !GetAtt HandleGenerateStep.Arn - WorkerRequestAI: !GetAtt WorkerRequestAI.Arn - HandleMergeResultRequestAI: !GetAtt HandleMergeResultRequestAI.Arn - HandleCompleteRequestAI: !GetAtt HandleCompleteRequestAI.Arn - HandlerUploadBatchToS3: !GetAtt HandlerUploadBatchToS3.Arn - UpdateStatusTask: !GetAtt UpdateStatusTask.Arn - UploadStateMachine: Type: AWS::Serverless::StateMachine @@ -1274,7 +1487,7 @@ Resources: Name: !Sub "${StagePara}-${ApplicationPara}-UploadSM" Policies: - LambdaInvokePolicy: - FunctionName: !Ref PreprocessGenerateTask + FunctionName: !Ref PreprocessGenerateTask - Statement: - Sid: ALLOWCRUDDynamoDB Effect: Allow @@ -1315,6 +1528,69 @@ Resources: LogGroupArn: !GetAtt CallerServiceStateMachineLogGroup.Arn DefinitionSubstitutions: PreprocessGenerateTask: !GetAtt PreprocessGenerateTask.Arn + + ###### AI_caller_ecs_state_machine + AICallerECSSM: + Type: AWS::Serverless::StateMachine + Properties: + Type: STANDARD + Name: !Sub "${StagePara}-${ApplicationPara}-AICallerECSSM" + Policies: + - LambdaInvokePolicy: + FunctionName: !Ref PreprocessGenerateTask + - Statement: + - Sid: CloudWatchLogsPolicy + Effect: Allow + Action: + - "logs:CreateLogDelivery" + - "logs:GetLogDelivery" + - "logs:UpdateLogDelivery" + - "logs:DeleteLogDelivery" + - "logs:ListLogDeliveries" + - "logs:PutResourcePolicy" + - "logs:DescribeResourcePolicies" + - "logs:DescribeLogGroup" + - "logs:DescribeLogGroups" + Resource: "*" + - Sid: ALLOWCRUDDynamoDB + Effect: Allow + Action: + - "dynamodb:GetItem" + - "dynamodb:DeleteItem" + - "dynamodb:PutItem" + - "dynamodb:Scan" + - "dynamodb:Query" + - "dynamodb:UpdateItem" + - "dynamodb:BatchWriteItem" + - "dynamodb:BatchGetItem" + - "dynamodb:DescribeTable" + - "dynamodb:ConditionCheckItem" + - "lambda:InvokeFunction" + Resource: "*" + Tracing: + Enabled: true + DefinitionUri: statemachine/ai_caller_ecs/sm_ai_caller_ecs.asl.yaml + Logging: + Level: ALL + IncludeExecutionData: true + Destinations: + - CloudWatchLogsLogGroup: + LogGroupArn: !GetAtt CallerServiceStateMachineLogGroup.Arn + DefinitionSubstitutions: + FuncPreprocessInputRequestArn: !GetAtt FuncPreprocessInputRequest.Arn + ### Functions of AI_caller_ecs_state_machine + FuncPreprocessInputRequest: + Type: AWS::Serverless::Function + Properties: + CodeUri: statemachine/ai_caller_ecs/functions/preprocess_input_request + Handler: hdler_preprocess_input_request.lambda_handler + Policies: + - Statement: + - Sid: ALLOWCRUDDynamoDB + Effect: Allow + Action: + - "dynamodb:*" + Resource: "*" Outputs: # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function # Find out more about other implicit resources you can reference within SAM @@ -1339,6 +1615,8 @@ Outputs: Value: !Ref ProcessAITaskEventBus TaskQueueName: Value: !GetAtt TaskQueue.QueueName + AICallerECSSMArn: + Value: !GetAtt AICallerECSSM.Arn # LayerARN: # Description: "ARN of common code layer" # Value: !GetAtt CommonCodeLayer.Arn diff --git a/daita-app/ai-caller-service/functions/handlers/crontab/consumer/app.py b/daita-app/ai-caller-service/functions/handlers/crontab/consumer/app.py index ee348f8..e2a85cc 100644 --- a/daita-app/ai-caller-service/functions/handlers/crontab/consumer/app.py +++ b/daita-app/ai-caller-service/functions/handlers/crontab/consumer/app.py @@ -102,10 +102,10 @@ def lambda_handler(event, context): 'filename': each['filename'] }) total_size += each['size'] - # Update Table T_PROJECT_SUMMARY + # Update Table TABLE_PROJECT_SUMMARY prj_sum_all = db_resource.Table( - os.environ['T_PROJECT_SUMMARY']) + os.environ['TABLE_PROJECT_SUMMARY']) responsePrjSumAll = prj_sum_all.get_item(Key={ "project_id": detail['project_id'], "type": item.process_type diff --git a/daita-app/ai-caller-service/functions/handlers/crontab/stop_ec2/app.py b/daita-app/ai-caller-service/functions/handlers/crontab/stop_ec2/app.py index 41cd324..c49755a 100644 --- a/daita-app/ai-caller-service/functions/handlers/crontab/stop_ec2/app.py +++ b/daita-app/ai-caller-service/functions/handlers/crontab/stop_ec2/app.py @@ -31,9 +31,9 @@ def countTaskInQueue(queue_id): return int(num_task_in_queue) class EC2Model(object): - def __init__(self): + def __init__(self, table_ls_ec2_name = "ec2"): self.db_client = boto3.client('dynamodb') - self.TBL = 'ec2' + self.TBL = table_ls_ec2_name def scanTable(self,TableName,**kwargs): paginator = self.db_client.get_paginator("scan") @@ -81,7 +81,7 @@ def lambda_handler(event, context): ) ls_running_exe = response['executions'] - ec2Model = EC2Model() + ec2Model = EC2Model(os.environ["TABLE_LS_EC2"]) ec2free = ec2Model.getFreeEc2() if len(ls_running_exe) > 0: diff --git a/daita-app/ai-caller-service/functions/handlers/download_task/download_task/app.py b/daita-app/ai-caller-service/functions/handlers/download_task/download_task/app.py index 3bcac12..b379bb3 100644 --- a/daita-app/ai-caller-service/functions/handlers/download_task/download_task/app.py +++ b/daita-app/ai-caller-service/functions/handlers/download_task/download_task/app.py @@ -15,59 +15,65 @@ from identity_check import * from consts import ConstTbl + def batcher(iterable, size): iterator = iter(iterable) for first in iterator: yield list(chain([first], islice(iterator, size - 1))) + + constModel = ConstTbl() + class S3(object): - def __init__(self,project_prefix): + def __init__(self, project_prefix): self.s3 = boto3.client('s3') self.bucket = None - self.s3_key = None - self.project_prefix = project_prefix - self.bucket , self.folder = self.split(self.project_prefix) + self.s3_key = None + self.project_prefix = project_prefix + self.bucket, self.folder = self.split(self.project_prefix) self.prefix_pwd = os.environ['EFSPATH'] self.root_efs = os.environ['ROOTEFS'] - - def split(self,uri): + + def split(self, uri): if not 's3' in uri[:2]: temp = uri.split('/') bucket = temp[0] - filename = '/'.join([temp[i] for i in range(1,len(temp))]) + filename = '/'.join([temp[i] for i in range(1, len(temp))]) else: - match = re.match(r's3:\/\/(.+?)\/(.+)', uri) + match = re.match(r's3:\/\/(.+?)\/(.+)', uri) bucket = match.group(1) filename = match.group(2) - return bucket, filename + return bucket, filename - def download(self,uri,folder): - bucket , filename = self.split(uri) + def download(self, uri, folder): + bucket, filename = self.split(uri) basename = os.path.basename(filename) - new_image = os.path.join(folder,basename) + new_image = os.path.join(folder, basename) return new_image - def download_images(self,images,task_id): + def download_images(self, images, task_id): + + taskDir = os.path.join(self.prefix_pwd, task_id) + input_dir = os.path.join(taskDir, "raw_images") + output_dir = os.path.join(taskDir, "gen_images") - taskDir = os.path.join(self.prefix_pwd,task_id) - input_dir = os.path.join(taskDir,"raw_images") - output_dir = os.path.join(taskDir,"gen_images") + os.makedirs(input_dir, exist_ok=True) + os.makedirs(output_dir, exist_ok=True) - os.makedirs(input_dir,exist_ok=True) - os.makedirs(output_dir,exist_ok=True) + list_image = [self.download(it, input_dir) for it in images] - list_image = [self.download(it,input_dir) for it in images] def generator(): yield from list_image - batch_size = int(constModel.get_num_value(code= 'limit_request_batchsize_ai', threshold='THRESHOLD')) + batch_size = int(constModel.get_num_value( + code='limit_request_batchsize_ai', threshold='THRESHOLD')) batch_input = [] batch_output = [] total_len = 0 - jsonInputLoads = os.path.join(self.folder,'input_json') + jsonInputLoads = os.path.join(self.folder, 'input_json') # os.makedirs(jsonInputLoads,exist_ok=True) - for index, batch in enumerate(batcher(generator(),batch_size)): - output_dir_temp = output_dir + for index, batch in enumerate(batcher(generator(), batch_size)): + output_dir_temp = output_dir total_len += len(batch) jsonBatchImages = [self.root_efs+it for it in batch] # nameJsonBatches = os.path.join(jsonInputLoads,'input_batch_'+str(index)+'.json') @@ -75,35 +81,40 @@ def generator(): # json.dump(jsonBatchImages,f) self.s3.put_object( Body=json.dumps(jsonBatchImages), - Bucket= self.bucket, - Key= os.path.join(jsonInputLoads,str(index)+'_download_image.json') + Bucket=self.bucket, + Key=os.path.join(jsonInputLoads, str( + index)+'_download_image.json') ) - batch_input.append(self.bucket+'/'+jsonInputLoads+'/'+str(index)+'_download_image.json') - nameoutput = os.path.join(output_dir_temp,str(index)) - os.makedirs(nameoutput,exist_ok=True) + batch_input.append(self.bucket+'/'+jsonInputLoads + + '/'+str(index)+'_download_image.json') + nameoutput = os.path.join(output_dir_temp, str(index)) + os.makedirs(nameoutput, exist_ok=True) batch_output.append(self.root_efs+nameoutput) download_images = [] - for index,it in enumerate(images): - path ={'uri':it,'folder':input_dir} + for index, it in enumerate(images): + path = {'uri': it, 'folder': input_dir} self.s3.put_object( Body=json.dumps(path), - Bucket= self.bucket, - Key= os.path.join(self.folder,str(index)+'_download_image.json') + Bucket=self.bucket, + Key=os.path.join(self.folder, str( + index)+'_download_image.json') ) - download_images.append({'path':self.bucket+'/'+self.folder+'/'+str(index)+'_download_image.json'}) + download_images.append( + {'path': self.bucket+'/'+self.folder+'/'+str(index)+'_download_image.json'}) info = { 'images_download': len(list_image), - 'batches_input': batch_input , - 'batches_output':batch_output, + 'batches_input': batch_input, + 'batches_output': batch_output, 'batch_size': batch_size, } - return info , download_images + return info, download_images def downloadS3ToEFS(data): s3 = S3(project_prefix=data['project_prefix']) images = data['images'] - s3Resp, download_images = s3.download_images(images=images,task_id=data['task_id']) + s3Resp, download_images = s3.download_images( + images=images, task_id=data['task_id']) result = data result['download_task'] = s3Resp result['download_images'] = download_images @@ -118,4 +129,4 @@ def lambda_handler(event, context): print("body in event: ", body) data = body['data'] result = downloadS3ToEFS(data) - return result \ No newline at end of file + return result diff --git a/daita-app/ai-caller-service/functions/handlers/download_task/get_reference_images/__init__.py b/daita-app/ai-caller-service/functions/handlers/download_task/get_reference_images/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/ai-caller-service/functions/handlers/download_task/get_reference_images/app.py b/daita-app/ai-caller-service/functions/handlers/download_task/get_reference_images/app.py new file mode 100644 index 0000000..66b4ba1 --- /dev/null +++ b/daita-app/ai-caller-service/functions/handlers/download_task/get_reference_images/app.py @@ -0,0 +1,86 @@ +import requests +import time +from config import * +from response import * +from utils import * +from identity_check import * +from lambda_base_class import LambdaBaseClass + + +class GetReferenceImageClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + @LambdaBaseClass.parse_body + def parser(self, body): + pass + # self.process_type = body['process_type'] + # self.is_retry = body['is_retry'] + + + def _check_input_value(self): + pass + + def handle(self, event, context): + + if 'process_type' in event and event['process_type'] != 'PREPROCESS': + event['is_retry'] = False + return event + + if bool(event['reference_images']): + event['is_retry'] = False + + print("event start: \n", event) + + if not 'reference_image_task_id' in event: + input_data = { + 'id_token': event['id_token'], + 'project_id': event['project_id'], + 'ls_method_client_choose': [], + 'project_name': event['project_name'] + } + + response = self.invoke_lambda_func(self.env.FUNC_RI_CALCULATION, input_data) + + print("response calculation: ", response) + + if response["statusCode"] == 200: + data = json.loads(response["body"])["data"] + event['reference_image_task_id'] = data['task_id'] + print(f"Logging Debug :{data}") + + start = time.time() + while time.time() - start <= 120: + json_data = { + 'id_token': event['id_token'], + 'task_id': event['reference_image_task_id'], + } + reponseStatusRefImage = self.invoke_lambda_func(self.env.FUNC_RI_STATUS, json_data) + print("reponseStatusRefImage: \n", reponseStatusRefImage) + + data = json.loads(reponseStatusRefImage["body"])["data"] + + if data['status'] == 'FINISH': + json_data_info={ + 'project_id': event['project_id'], + 'id_token': event['id_token'], + } + reponseInfoRefImage = self.invoke_lambda_func(self.env.FUNC_RI_INFO, json_data_info) + print("reponseInfoRefImage: \n", reponseInfoRefImage) + + info = json.loads(reponseInfoRefImage["body"])["data"] + for it in info: + event['reference_images'][it['method_id'] + ] = "s3://{}".format(it['image_s3_path']) + event['is_retry'] = False + break + else: + event['is_retry'] = True + time.sleep(5) + + return event + + +@error_response +def lambda_handler(event, context): + return GetReferenceImageClass().handle(event=event, context=context) diff --git a/daita-app/ai-caller-service/functions/handlers/download_task/get_reference_images/requirements.txt b/daita-app/ai-caller-service/functions/handlers/download_task/get_reference_images/requirements.txt new file mode 100644 index 0000000..663bd1f --- /dev/null +++ b/daita-app/ai-caller-service/functions/handlers/download_task/get_reference_images/requirements.txt @@ -0,0 +1 @@ +requests \ No newline at end of file diff --git a/daita-app/ai-caller-service/functions/handlers/download_task/merge_download/app.py b/daita-app/ai-caller-service/functions/handlers/download_task/merge_download/app.py index 430bbda..193bcc3 100644 --- a/daita-app/ai-caller-service/functions/handlers/download_task/merge_download/app.py +++ b/daita-app/ai-caller-service/functions/handlers/download_task/merge_download/app.py @@ -16,18 +16,30 @@ from identity_check import * from s3_utils import split s3 = boto3.client('s3') +Mode = os.environ.get( + 'MODE', 'staging' +) + @error_response def lambda_handler(event, context): if 'reference_images' in event: root_efs = os.environ['ROOTEFS'] - prefix_pwd = os.path.join(os.environ['EFSPATH'],event['task_id']) - reference_images_folder = os.path.join(prefix_pwd,'reference_images') - os.makedirs(reference_images_folder,exist_ok=True) - for k , v in event['reference_images'].items(): - bucket , filename = split(v) + prefix_pwd = os.path.join(os.environ['EFSPATH'], event['task_id']) + reference_images_folder = os.path.join(prefix_pwd, 'reference_images') + os.makedirs(reference_images_folder, exist_ok=True) + for k, v in event['reference_images'].items(): + bucket, filename = split(v) basename = os.path.basename(filename) - new_image = os.path.join(reference_images_folder,basename) - s3.download_file(bucket,filename,new_image) - event['reference_images'][k] = root_efs +new_image - return event \ No newline at end of file + new_image = os.path.join(reference_images_folder, basename) + s3.download_file(bucket, filename, new_image) + ######################## Hot Fix for dev enviroment######### + if Mode == 'dev': + tmp = os.path.join('reference_images', basename) + task_id_dir = os.path.join(event['task_id'], tmp) + event['reference_images'][k] = os.path.join( + 'generation-task', task_id_dir) + else: + event['reference_images'][k] = root_efs + new_image + ############################################################ + return event diff --git a/daita-app/ai-caller-service/functions/handlers/generate_step/CompleteRequestAI/app.py b/daita-app/ai-caller-service/functions/handlers/generate_step/CompleteRequestAI/app.py index ebe54f3..e36f308 100644 --- a/daita-app/ai-caller-service/functions/handlers/generate_step/CompleteRequestAI/app.py +++ b/daita-app/ai-caller-service/functions/handlers/generate_step/CompleteRequestAI/app.py @@ -10,81 +10,108 @@ from itertools import chain, islice import os from config import * +from lambda_base_class import LambdaBaseClass from response import * from utils import * from identity_check import * from boto3.dynamodb.conditions import Key, Attr from models.generate_task_model import GenerateTaskModel from models.task_model import TaskModel -task_model = TaskModel(os.environ["TABLE_GENERATE_TASK"],None) -generate_task_model = GenerateTaskModel(os.environ["TABLE_GENERATE_TASK"]) -def batcher(iterable, size): - iterator = iter(iterable) - for first in iterator: - yield list(chain([first], islice(iterator, size - 1))) -def request_update_proj(update_pro_info,list_file_s3,gen_id,task_id): - batch_list_s3 = list(batcher(list_file_s3,20)) - for batch_file in batch_list_s3: - print("[DEBUG] batch file {}".format(batch_file)) - info = {'identity_id':update_pro_info['identity_id'], - 'id_token':update_pro_info['id_token'] , - 'project_id':update_pro_info['project_id'], - 'project_name':update_pro_info['project_name'], - 'type_method': update_pro_info['process_type'], - 'ls_object_info':[]} - for info_file in batch_file: - filename = os.path.basename(info_file['filename']) - info['ls_object_info'].append({ - 's3_key':os.path.join(update_pro_info['s3_key'],os.path.join(task_id,filename)) , - 'filename':filename, - 'hash':'', - 'size':info_file['size'], - 'is_ori':False, - 'gen_id' :gen_id - }) - print("[DEBUG] Log Request Update Upload : {}\n".format(info)) - update_project_output = requests.post('https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/projects/upload_update',json=info) - - print("[DEBUG] Request Update Upload: {}\n".format(update_project_output.text)) +class ComnpleteRequestAIClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + self.task_model = TaskModel(self.env.TABLE_GENERATE_TASK, None) + self.generate_task_model = GenerateTaskModel(self.env.TABLE_GENERATE_TASK) + + @LambdaBaseClass.parse_body + def parser(self, body): + self.info_upload_s3 = body["info_upload_s3"] + self.identity_id = body["identity_id"] + self.task_id = body["task_id"] + self.id_token = body["id_token"] + self.project_prefix = body["project_prefix"] + self.project_id = body['project_id'] + self.project_name = body['project_name'] + self.gen_id = body["gen_id"] + self.response = body["response"] + self.num_finish = body.get("num_finish", -1) + + def batcher(self, iterable, size): + iterator = iter(iterable) + for first in iterator: + yield list(chain([first], islice(iterator, size - 1))) + + def invokeUploadUpdateFunc(self, info): + lambdaInvokeClient = boto3.client('lambda') + payloadStr = json.dumps({'body': info}) + payloadBytesArr = bytes(payloadStr, encoding='utf8') + + lambdaInvokeReq = lambdaInvokeClient.invoke( + FunctionName=self.env.FUNC_DAITA_UPLOAD_UPDATE, + Payload=payloadBytesArr, + InvocationType="RequestResponse", + ) + payload = json.loads(lambdaInvokeReq['Payload'].read()) + print("Payload response: ", payload) + body = json.loads(payload['body']) + return body -@error_response -def lambda_handler(event, context): - print(event) - body = event - info_upload_s3 = body['info_upload_s3'] - item = generate_task_model.get_task_info(body['identity_id'] ,body['task_id']) - db_resource = boto3.resource('dynamodb',REGION) - resq = request_update_proj(update_pro_info={ - 'identity_id': body['identity_id'], - 'id_token':body['id_token'], - 's3_key': body['project_prefix'], - 'project_id': body['project_id'], - 'project_name': body['project_name'], - 'process_type': item.process_type - },list_file_s3= info_upload_s3, gen_id=body['gen_id'],task_id=body['task_id']) - - if item.process_type == 'AUGMENT': - table = db_resource.Table(os.environ['TABLE_DATA_AUGMENT']) - elif item.process_type == 'PREPROCESS': - table = db_resource.Table(os.environ['TABLE_DATA_PREPROCESS']) - - # queryResponse = table.query( - # KeyConditionExpression=Key('project_id').eq(body['project_id']), - # FilterExpression=Attr('s3_key').contains(body['task_id']), - # Limit=1000 - # ) - # print("output of querry: \n", queryResponse) - # print(f"number items of query: {len(queryResponse['Items'])}") - num_finish = event.get("num_finish", -1) - if num_finish<0: - pass - else: - task_model.update_number_files(task_id = body['task_id'], identity_id = body['identity_id'], - num_finish = num_finish) + def request_update_proj(self, update_pro_info, list_file_s3, gen_id,task_id): + batch_list_s3 = list(self.batcher(list_file_s3,20)) + for batch_file in batch_list_s3: + print("[DEBUG] batch file {}".format(batch_file)) + info = {'identity_id':update_pro_info['identity_id'], + 'id_token':update_pro_info['id_token'] , + 'project_id':update_pro_info['project_id'], + 'project_name':update_pro_info['project_name'], + 'type_method': update_pro_info['process_type'], + 'ls_object_info':[]} + for info_file in batch_file: + filename = os.path.basename(info_file['filename']) + info['ls_object_info'].append({ + 's3_key':os.path.join(update_pro_info['s3_key'],os.path.join(task_id,filename)) , + 'filename':filename, + 'hash':'', + 'size':info_file['size'], + 'is_ori':False, + 'gen_id' :gen_id + }) + print("[DEBUG] Log Request Update Upload : {}\n".format(info)) + + update_project_output = self.invokeUploadUpdateFunc(info) + print("[DEBUG] Request Update Upload: {}\n".format(update_project_output)) - return { - 'response':body['response'] - } \ No newline at end of file + def handle(self, event, context): + ### parse body + self.parser(event, is_event_as_body=True) + + item = self.generate_task_model.get_task_info(self.identity_id, self.task_id) + resq = self.request_update_proj(update_pro_info={ + 'identity_id': self.identity_id, + 'id_token': self.id_token, + 's3_key': self.project_prefix, + 'project_id': self.project_id, + 'project_name': self.project_name, + 'process_type': item.process_type + }, + list_file_s3 = self.info_upload_s3, + gen_id = self.gen_id, + task_id = self.task_id) + + + if self.num_finish<0: + pass + else: + self.task_model.update_number_files(task_id = self.task_id, identity_id = self.identity_id, + num_finish = self.num_finish) + + return { + 'response': self.response + } + +@error_response +def lambda_handler(event, context): + return ComnpleteRequestAIClass().handle(event=event, context=context) \ No newline at end of file diff --git a/daita-app/ai-caller-service/functions/handlers/generate_step/HandleBatchToS3/app.py b/daita-app/ai-caller-service/functions/handlers/generate_step/HandleBatchToS3/app.py index c08378f..7e78b92 100644 --- a/daita-app/ai-caller-service/functions/handlers/generate_step/HandleBatchToS3/app.py +++ b/daita-app/ai-caller-service/functions/handlers/generate_step/HandleBatchToS3/app.py @@ -19,7 +19,10 @@ task_model = TaskModel(os.environ["TABLE_GENERATE_TASK"], None) generate_task_model = GenerateTaskModel(os.environ["TABLE_GENERATE_TASK"]) ROOT_EFS = os.environ["ROOTEFS"] - +Mode = os.environ.get( + 'MODE', 'staging' +) +prefix_pwd = os.environ['EFSPATH'] s3 = boto3.client('s3') @@ -82,13 +85,24 @@ def lambda_handler(event, context): if event['response'] == 'OK': if 'output_images' in event: - output = list(map(lambda x: x.replace( - ROOT_EFS, ''), event['output_images'])) + ######################## Hot Fix for dev enviroment######### + if Mode == 'dev': + output = list(map(lambda x: x.replace( + '/mnt/efs', ''), event['output_images'])) + else: + output = list(map(lambda x: x.replace( + ROOT_EFS, ''), event['output_images'])) + infoUploadS3 = UploadImage( output=output, project_prefix=event['project_prefix'], task_id=event['task_id']) - output_folder = event["batch"]['request_json']['output_folder'].replace( - ROOT_EFS, '') + if Mode == 'dev': + output_folder = '/mnt/' + \ + event["batch"]['request_json']['output_folder'] + else: + output_folder = event["batch"]['request_json']['output_folder'].replace( + ROOT_EFS, '') + ls_split = output_folder.split(os.sep) path_check_finish = "/".join(ls_split[0:-1]) result["num_finish"] = get_number_files(path_check_finish) diff --git a/daita-app/ai-caller-service/functions/handlers/generate_step/RequestAI/app.py b/daita-app/ai-caller-service/functions/handlers/generate_step/RequestAI/app.py index 9fb0eda..6587fc2 100644 --- a/daita-app/ai-caller-service/functions/handlers/generate_step/RequestAI/app.py +++ b/daita-app/ai-caller-service/functions/handlers/generate_step/RequestAI/app.py @@ -16,19 +16,22 @@ from boto3.dynamodb.conditions import Key, Attr from models.generate_task_model import GenerateTaskModel s3 = boto3.client('s3') -sqs = boto3.resource("sqs",REGION) -sqsClient = boto3.client('sqs',REGION) +sqs = boto3.resource("sqs", REGION) +sqsClient = boto3.client('sqs', REGION) ec2_resource = boto3.resource('ec2', region_name=REGION) - +Mode = os.environ.get( + 'MODE', 'staging' +) generate_task_model = GenerateTaskModel(os.environ["TABLE_GENERATE_TASK"]) + def deleteMessageInQueue(task): queueSQS = sqs.get_queue_by_name(QueueName=task['queue']) QueueResp = queueSQS.receive_messages(VisibilityTimeout=60, - WaitTimeSeconds=0,MaxNumberOfMessages=1) + WaitTimeSeconds=0, MaxNumberOfMessages=1) print(f"Len of queueResp when deleteMessageInQueue: {len(QueueResp)}") - for message in QueueResp : + for message in QueueResp: # messageBody = message.body mss_id = message.message_id print("Delete QUEUE with id: \n", mss_id) @@ -38,59 +41,87 @@ def deleteMessageInQueue(task): # # # if messageBody == strTask: message.delete() - print(f"--Count AFTER DELETE message in queue: {task['queue']} is {countTaskInQueue(task['queue'])}") + print( + f"--Count AFTER DELETE message in queue: {task['queue']} is {countTaskInQueue(task['queue'])}") + def getQueue(queue_name_env): response = sqsClient.get_queue_url(QueueName=queue_name_env) return response['QueueUrl'] + def countTaskInQueue(queue_id): # get_queue_attributes # sqsName = sqsResourse.get_queue_by_name(QueueName=queue_id) response = sqsClient.get_queue_attributes( - QueueUrl=getQueue(queue_id), - AttributeNames=[ - 'ApproximateNumberOfMessages' - ] - ) + QueueUrl=getQueue(queue_id), + AttributeNames=[ + 'ApproximateNumberOfMessages' + ] + ) num_task_in_queue = response['Attributes']['ApproximateNumberOfMessages'] return int(num_task_in_queue) + @error_response def lambda_handler(event, context): result = event - if result['is_retry'] == True : + if result['is_retry'] == True: time.sleep(int(result['current_num_retries'])*15) print("Input event: ", event) batch = result['batch'] - if not isinstance(batch['request_json']['images_paths'],list): - bucket , filename = split(batch['request_json']['images_paths']) - resultS3 = s3.get_object(Bucket=bucket, Key=filename) - batch['request_json']['images_paths'] = json.loads(resultS3["Body"].read().decode()) + if not isinstance(batch['request_json']['images_paths'], list): + bucket, filename = split(batch['request_json']['images_paths']) + resultS3 = s3.get_object(Bucket=bucket, Key=filename) + batch['request_json']['images_paths'] = json.loads( + resultS3["Body"].read().decode()) + ######################## Hot Fix for dev enviroment######### + if Mode == 'dev' and len(batch['request_json']['images_paths']) != 0: + tmpStr = batch['request_json']['images_paths'][0] + tmp = tmpStr.split('/')[0] + if tmp != 'generation-task': + tmpArr = tmpStr.split('/') + index = None + for it, el in enumerate(tmpArr): + if el == 'generation-task': + index = it + tmpbatch = batch['request_json']['images_paths'] + tmpStr = None + batchArr = [] + for it in tmpbatch: + tmpStr = it.split('/') + batchArr.append('/'.join(tmpStr[index:])) + batch['request_json']['images_paths'] = batchArr + outputTmp = (batch['request_json']['output_folder']).split('/') + batch['request_json']['output_folder'] = '/'.join( + outputTmp[index:]) ###################################################### - item = generate_task_model.get_task_info(result['identity_id'] ,result['task_id']) + item = generate_task_model.get_task_info( + result['identity_id'], result['task_id']) if item.status == 'CANCEL': result['response'] = 'NOT_OK' result['is_retry'] = False deleteMessageInQueue(batch) return result - - print(f"--Count current message in queue: {batch['queue']} is {countTaskInQueue(batch['queue'])}") - + + print( + f"--Count current message in queue: {batch['queue']} is {countTaskInQueue(batch['queue'])}") + print("request AI body: \n", batch['request_json']) result['output_images'] = [] - try : + try: instance = ec2_resource.Instance(batch['ec2_id']) instance.load() - print(f"Current state of instance before send request: {batch['ec2_id']} is {instance.state['Name']}") + print( + f"Current state of instance before send request: {batch['ec2_id']} is {instance.state['Name']}") ip_public_current = instance.public_ip_address print(f"Current IP public {ip_public_current} and {batch['host']}") if not str(ip_public_current) in batch['host']: - batch['host'] = f"http://{ip_public_current}:8000/{batch['type']}" + batch['host'] = f"http://{ip_public_current}:8000/{batch['type']}" print(batch['host']) - output = requests.post(batch['host'],json=batch['request_json']) + output = requests.post(batch['host'], json=batch['request_json']) - ### use augment_codes for gen_id method for all images in batch + # use augment_codes for gen_id method for all images in batch json_data = output.json() result["augment_codes"] = json_data.get("augment_codes", None) print("Output from AI request: \n", output.text) @@ -113,8 +144,8 @@ def lambda_handler(event, context): # print(output.text) result['response'] = 'OK' result['is_retry'] = False - + print("-----Normal Delete message ------------------------ ") deleteMessageInQueue(batch) - return result \ No newline at end of file + return result diff --git a/daita-app/ai-caller-service/functions/handlers/generate_step/generate_task/app.py b/daita-app/ai-caller-service/functions/handlers/generate_step/generate_task/app.py index 9a13a8d..81d7954 100644 --- a/daita-app/ai-caller-service/functions/handlers/generate_step/generate_task/app.py +++ b/daita-app/ai-caller-service/functions/handlers/generate_step/generate_task/app.py @@ -15,88 +15,71 @@ from models.generate_task_model import GenerateTaskModel from s3_utils import * -task_model = TaskModel(os.environ["TABLE_GENERATE_TASK"],None) +task_model = TaskModel(os.environ["TABLE_GENERATE_TASK"], None) generate_task_model = GenerateTaskModel(os.environ["TABLE_GENERATE_TASK"]) -ec2Model = EC2Model() +ec2Model = EC2Model(os.environ["TABLE_LS_EC2"]) s3 = boto3.client('s3') + + def split(uri): if not 's3' in uri[:2]: temp = uri.split('/') bucket = temp[0] - filename = '/'.join([temp[i] for i in range(1,len(temp))]) + filename = '/'.join([temp[i] for i in range(1, len(temp))]) else: - match = re.match(r's3:\/\/(.+?)\/(.+)', uri) + match = re.match(r's3:\/\/(.+?)\/(.+)', uri) bucket = match.group(1) filename = match.group(2) return bucket, filename -""" - download_task: - images_download: - batched_input: - batched_output: - batch_size - output: - state: - list_request_ai : - [ - { - task_id - batch_input: - batch_out: - api: - project_prefix: - identity_id: - project_id: - } - ] - """ @error_response def lambda_handler(event, context): inputJson = event['body'] # with open(inputJson['path'],'r') as f: # data = json.load(f) - bucket , filename = split(inputJson['path']) + bucket, filename = split(inputJson['path']) resultS3 = s3.get_object(Bucket=bucket, Key=filename) data = json.loads(resultS3["Body"].read().decode()) print("Event body: \n", data) - downloadTask = data['download_task'] + downloadTask = data['download_task'] #############Get ec2 free######################## ec2FreeInstnaces = ec2Model.getFreeEc2() ################Start all ec2 free##################### # taskModel.create_item(identity_id=data['identity_id'],task_id=data['task_id'],project_id=data['project_id'] # ,num_gens=data['num_aug_per_imgs'] ,process_type=data['type_method'],IP='',EC2_ID='') - ### update num_gens for task - task_model.update_attribute(data['task_id'], data['identity_id'], [[TaskModel.FIELD_NUM_GENS_IMAGE, data["images"]]]) - task_model.update_generate_progress(task_id = data['task_id'], identity_id = data['identity_id'], num_finish = 0, status = 'PREPARING_HARDWARE') - + # update num_gens for task + task_model.update_attribute(data['task_id'], data['identity_id'], [ + [TaskModel.FIELD_NUM_GENS_IMAGE, data["images"]]]) + task_model.update_generate_progress( + task_id=data['task_id'], identity_id=data['identity_id'], num_finish=0, status='PREPARING_HARDWARE') + list_request_ai = assignTaskToEc2(ec2Instances=ec2FreeInstnaces, data=downloadTask, num_augments_per_image=data['num_aug_per_imgs'], type_method=data['type_method'], code=data['ls_method_id'], reference_images=data[KEY_NAME_REFERENCE_IMAGES], is_normalize_resolution=data[KEY_NAME_IS_RESOLUTION], - aug_parameters = data[KEY_NAME_AUG_PARAMS] + aug_parameters=data[KEY_NAME_AUG_PARAMS] ) time.sleep(5) task_model.update_generate_progress(task_id=data['task_id'], identity_id=data['identity_id'], num_finish=0, status='RUNNING') - print(list_request_ai) s3.delete_object(Bucket=bucket, Key=filename) - generate_task_model.init_messages_in_flight(identity_id=data['identity_id'], task_id=data['task_id'], ) + generate_task_model.init_messages_in_flight( + identity_id=data['identity_id'], task_id=data['task_id'], ) return { 'state': 'Request_AI', - 'list_request_ai':list_request_ai, - 'identity_id':data['identity_id'], + 'list_request_ai': list_request_ai, + 'identity_id': data['identity_id'], 'id_token': data['id_token'], 'task_id': data['task_id'], 'project_prefix': data['project_prefix'], 'current_num_retries': 0, 'max_retries': 13, - 'is_retry': False , + 'is_retry': False, "project_id": data['project_id'], "project_name": data['project_name'] - } \ No newline at end of file + } diff --git a/daita-app/ai-caller-service/functions/handlers/generate_step/generate_task/ec2.py b/daita-app/ai-caller-service/functions/handlers/generate_step/generate_task/ec2.py index 94784e4..1269682 100644 --- a/daita-app/ai-caller-service/functions/handlers/generate_step/generate_task/ec2.py +++ b/daita-app/ai-caller-service/functions/handlers/generate_step/generate_task/ec2.py @@ -5,9 +5,9 @@ clientEc2 = boto3.client('ec2',region_name=REGION) class EC2Model(object): - def __init__(self): + def __init__(self, table_ls_ec2_name = "ec2"): self.db_client = boto3.client('dynamodb') - self.TBL = 'ec2' + self.TBL = table_ls_ec2_name def scanTable(self,TableName,**kwargs): paginator = self.db_client.get_paginator("scan") diff --git a/daita-app/ai-caller-service/functions/handlers/generate_step/glob_output_files/__init__.py b/daita-app/ai-caller-service/functions/handlers/generate_step/glob_output_files/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/ai-caller-service/functions/handlers/generate_step/glob_output_files/app.py b/daita-app/ai-caller-service/functions/handlers/generate_step/glob_output_files/app.py new file mode 100644 index 0000000..14d9767 --- /dev/null +++ b/daita-app/ai-caller-service/functions/handlers/generate_step/glob_output_files/app.py @@ -0,0 +1,14 @@ +import os +import glob + + +@error_response +def lambda_handler(event, context): + result = event + intput_file = os.path.basename(event.get("intput_file")) + output_folder = event["request_json"]["output_folder"] + output_files = glob.glob(os.path.join(output_folder, "*.*")) # glob every files + if intput_file: + output_files = [f for f in output_files if os.path.basename(f) != intput_file] # exclude input_file + result["output_images"] = output_files + return result diff --git a/daita-app/ai-caller-service/functions/handlers/generate_step/updateStatusTask/app.py b/daita-app/ai-caller-service/functions/handlers/generate_step/updateStatusTask/app.py index 81ac3e6..86d8fd9 100644 --- a/daita-app/ai-caller-service/functions/handlers/generate_step/updateStatusTask/app.py +++ b/daita-app/ai-caller-service/functions/handlers/generate_step/updateStatusTask/app.py @@ -14,13 +14,16 @@ from models.generate_task_model import GenerateTaskModel generate_task_model = GenerateTaskModel(os.environ["TABLE_GENERATE_TASK"]) -task_model = TaskModel(os.environ["TABLE_GENERATE_TASK"],None) +task_model = TaskModel(os.environ["TABLE_GENERATE_TASK"], None) + @error_response def lambda_handler(event, context): - task_model.update_status(event['task_id'], event['identity_id'], event['status']) + task_model.update_status( + event['task_id'], event['identity_id'], event['status']) if event['status'] == 'FINISH': - folder = os.environ['ROOTEFS'] + os.environ['EFSPATH'] +'/'+ event['task_id'] +'/' + # folder = os.environ['ROOTEFS'] + os.environ['EFSPATH'] +'/'+ event['task_id'] +'/' + folder = os.environ['EFSPATH'] + '/' + event['task_id'] + '/' if os.path.exists(folder): shutil.rmtree(folder) - return event \ No newline at end of file + return event diff --git a/daita-app/ai-caller-service/functions/handlers/preprocess_generate_batch/generate_batch/app.py b/daita-app/ai-caller-service/functions/handlers/preprocess_generate_batch/generate_batch/app.py index 5d36ed9..0037fc0 100644 --- a/daita-app/ai-caller-service/functions/handlers/preprocess_generate_batch/generate_batch/app.py +++ b/daita-app/ai-caller-service/functions/handlers/preprocess_generate_batch/generate_batch/app.py @@ -13,9 +13,9 @@ from models.generate_task_model import GenerateTaskModel from models.task_model import TaskModel -TBL_data_original = 'data_original' -TBL_data_proprocess= 'data_preprocess' -TBL_PROJECT = 'projects' +TBL_data_original = os.getenv("TableDataOriginalName") +TBL_data_proprocess = os.getenv("TableDataPreprocessName") +TBL_PROJECT = os.getenv("TableProjectsName") MAX_NUMBER_GEN_PER_IMAGES = 5 generate_task_model = GenerateTaskModel(os.environ["TABLE_GENERATE_TASK"]) task_model = TaskModel(os.environ["TABLE_GENERATE_TASK"],None) @@ -36,13 +36,13 @@ def dydb_update_project_data_type_number(db_resource, identity_id, project_name, ':da': convert_current_date_to_iso8601(), ':tg': times_generated }, - UpdateExpression = 'SET #DA_TY = :va , updated_date = :da, times_generated = :tg' + UpdateExpression = 'SET #DA_TY = :va , updated_date = :da, times_generated = :tg' ) except Exception as e: print('Error: ', repr(e)) - raise + raise if response.get('Item', None): - return response['Item'] + return response['Item'] return None def dydb_get_project(db_resource, identity_id, project_name): @@ -57,9 +57,9 @@ def dydb_get_project(db_resource, identity_id, project_name): ) except Exception as e: print('Error: ', repr(e)) - raise + raise if response.get('Item', None): - return response['Item'] + return response['Item'] return None def dydb_update_class_data(table, project_id, filename, classtype): response = table.update_item( @@ -72,18 +72,18 @@ def dydb_update_class_data(table, project_id, filename, classtype): }, ExpressionAttributeValues = { ':ct': classtype, - + }, - UpdateExpression = 'SET #CT = :ct' + UpdateExpression = 'SET #CT = :ct' ) class ImageLoader(object): def __init__(self): - self.db_resource = boto3.resource('dynamodb') + self.db_resource = boto3.resource('dynamodb') ''' info_image: - identity_id - project_id + identity_id + project_id augment_code data_number data_type @@ -101,17 +101,17 @@ def __call__(self,info_image): infor = dydb_get_project(self.db_resource, identity_id, project_name) s3_prefix = infor['s3_prefix'] - + if data_type == 'ORIGINAL': table_name = TBL_data_original elif data_type == 'PREPROCESS': table_name = TBL_data_proprocess else: raise(Exception('data_type is not valid!')) - + if type_method == 'PREPROCESS': table_name = TBL_data_original - + # get list data table = self.db_resource.Table(table_name) response = table.query( @@ -119,7 +119,7 @@ def __call__(self,info_image): ProjectionExpression='filename, s3_key', ) ls_data = response['Items'] - + if type_method == 'PREPROCESS': ls_process = [item['s3_key'] for item in ls_data] # use all data in original for preprocessing elif type_method == 'AUGMENT': @@ -139,9 +139,9 @@ def __call__(self,info_image): else: ls_test.append(data) classtype = 'TEST' - + dydb_update_class_data(table, project_id, data["filename"], classtype) - return {"images": ls_process, "project_prefix":s3_prefix, 'type_method':type_method} + return {"images": ls_process, "project_prefix":s3_prefix, 'type_method':type_method} @error_response def lambda_handler(event, context): @@ -162,7 +162,7 @@ def lambda_handler(event, context): data['project_id'] = body['project_id'] data['project_name'] = body['project_name'] data['ls_method_id'] = body['ls_method_id'] - data['num_aug_per_imgs'] = body['num_aug_per_imgs'] if 'num_aug_per_imgs' in body else 1 + data['num_aug_per_imgs'] = body['num_aug_per_imgs'] if 'num_aug_per_imgs' in body else 1 data[KEY_NAME_PROCESS_TYPE] = body[KEY_NAME_PROCESS_TYPE] data[KEY_NAME_REFERENCE_IMAGES] = body[KEY_NAME_REFERENCE_IMAGES] data[KEY_NAME_IS_RESOLUTION] = body[KEY_NAME_IS_RESOLUTION] @@ -173,4 +173,3 @@ def lambda_handler(event, context): status_code=HTTPStatus.OK, data=data ) - diff --git a/daita-app/ai-caller-service/statemachine/ai_caller_ecs/functions/preprocess_input_request/hdler_preprocess_input_request.py b/daita-app/ai-caller-service/statemachine/ai_caller_ecs/functions/preprocess_input_request/hdler_preprocess_input_request.py new file mode 100644 index 0000000..8cca33d --- /dev/null +++ b/daita-app/ai-caller-service/statemachine/ai_caller_ecs/functions/preprocess_input_request/hdler_preprocess_input_request.py @@ -0,0 +1,43 @@ +import shutil +import time +import json +import boto3 +import random +from datetime import datetime +import requests +import queue +import threading +from itertools import chain, islice +import os +from config import * +from lambda_base_class import LambdaBaseClass +from response import * +from utils import * +from identity_check import * +from models.generate_task_model import GenerateTaskModel +from models.task_model import TaskModel + + +class PreprocessInputRequestClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + self.task_model = TaskModel(self.env.TABLE_GENERATE_TASK, None) + self.generate_task_model = GenerateTaskModel(self.env.TABLE_GENERATE_TASK) + + @LambdaBaseClass.parse_body + def parser(self, body): + pass + + def handle(self, event, context): + ### parse body + self.parser(event, is_event_as_body=True) + + + + return { + 'response': self.response + } + +@error_response +def lambda_handler(event, context): + return PreprocessInputRequestClass().handle(event=event, context=context) \ No newline at end of file diff --git a/daita-app/ai-caller-service/statemachine/ai_caller_ecs/sm_ai_caller_ecs.asl.yaml b/daita-app/ai-caller-service/statemachine/ai_caller_ecs/sm_ai_caller_ecs.asl.yaml new file mode 100644 index 0000000..1192ac9 --- /dev/null +++ b/daita-app/ai-caller-service/statemachine/ai_caller_ecs/sm_ai_caller_ecs.asl.yaml @@ -0,0 +1,25 @@ +StartAt: TaskFuncPreprocessInputRequest +States: + TaskFuncPreprocessInputRequest: + Type: Task + Resource: 'arn:aws:states:::lambda:invoke' + InputPath: $ + OutputPath: $ + Parameters: + FunctionName: '${FuncPreprocessInputRequestArn}' + Payload.$: $ + Next: PrepareComplete + Comment: >- + Check the level of parallelism, split requests into chunks and invoke + lamndas + Retry: + - ErrorEquals: + - RetriableCallerServiceError + IntervalSeconds: 1 + MaxAttempts: 2 + BackoffRate: 1 + PrepareComplete: + Type: Pass + Comment: Used for result aggregation + End: true +TimeoutSeconds: 18000 \ No newline at end of file diff --git a/daita-app/ai-caller-service/statemachine/download_state_machine.asl.yaml b/daita-app/ai-caller-service/statemachine/download_state_machine.asl.yaml index 510d0f0..f134085 100644 --- a/daita-app/ai-caller-service/statemachine/download_state_machine.asl.yaml +++ b/daita-app/ai-caller-service/statemachine/download_state_machine.asl.yaml @@ -61,8 +61,33 @@ States: MaxAttempts: 7 BackoffRate: 2 End: true - - StartAt: MergeDownloadImages + - StartAt: ReferenceImageCalculate States: + ReferenceImageCalculate: + Type: Task + Resource: 'arn:aws:states:::lambda:invoke' + InputPath: $ + OutputPath: $.Payload + Parameters: + FunctionName: '${ReferenceImageCalculateFunction}' + Payload.$ : $ + Next: IsCheckRetry + Retry: + - ErrorEquals: + - Lambda.TooManyRequestsException + - Lambda.ServiceException + IntervalSeconds: 3 + MaxAttempts: 7 + BackoffRate: 2 + IsCheckRetry: + Type: Choice + Choices: + - Variable: $.is_retry + BooleanEquals: true + Next: ReferenceImageCalculate + - Variable: $.is_retry + BooleanEquals: false + Next: MergeDownloadImages MergeDownloadImages: Type: Task Resource: 'arn:aws:states:::lambda:invoke' @@ -84,7 +109,6 @@ States: GetResult: Type: Task Resource: 'arn:aws:states:::lambda:invoke' - Next: DownloadComplete OutputPath: $ Parameters: FunctionName: '${HandleGetResultDownloadTask}' @@ -96,8 +120,5 @@ States: IntervalSeconds: 3 MaxAttempts: 7 BackoffRate: 2 - DownloadComplete: - Type: Pass - Comment: Used for result aggregation End: true -TimeoutSeconds: 1800 +TimeoutSeconds: 10800 diff --git a/daita-app/ai-caller-service/statemachine/generate_state_machine.asl.copy1.yaml b/daita-app/ai-caller-service/statemachine/generate_state_machine.asl.copy1.yaml deleted file mode 100644 index 338c73f..0000000 --- a/daita-app/ai-caller-service/statemachine/generate_state_machine.asl.copy1.yaml +++ /dev/null @@ -1,147 +0,0 @@ -StartAt: GenerateTask -States: - GenerateTask: - Type: Task - Resource: 'arn:aws:states:::lambda:invoke' - InputPath: $ - OutputPath: $.Payload - Parameters: - FunctionName: '${HandleGenerateStep}' - Payload.$: $ - Next: ChoiceStageRequestAI - Comment: >- - Check the level of parallelism split requests into chunks and invoke - lamndas - Retry: - - ErrorEquals: - - RetriableCallerServiceError - IntervalSeconds: 1 - MaxAttempts: 2 - BackoffRate: 1 - ChoiceStageRequestAI: - Type: Choice - Choices: - - Variable: $.state - StringEquals: Request_AI - Next: HandleBatchRequestAI - - Variable: $.state - StringEquals: FINISH - Next: UpdateStatusTask - - Variable: $.state - StringEquals: ERROR - Next: UpdateStatusTask - UpdateStatusTask: - Type: Task - Resource: 'arn:aws:states:::lambda:invoke' - Next: GenerateComplete - InputPath: $ - OutputPath: $ - Parameters: - FunctionName: '${UpdateStatusTask}' - Payload.$: $ - Retry: - - ErrorEquals: - - RetriableCallerServiceError - IntervalSeconds: 1 - MaxAttempts: 2 - BackoffRate: 1 - HandleBatchRequestAI: - Type: Map - MaxConcurrency: 5 - InputPath: $ - ItemsPath: $.list_request_ai - Parameters: - batch.$: $$.Map.Item.Value - current_num_retries.$: $.current_num_retries - max_retries.$: $.max_retries - is_retry.$: $.is_retry - identity_id.$: $$.Execution.Input.body.identity_id - task_id.$: $$.Execution.Input.body.task_id - Iterator: - StartAt: RequestAI - States: - RequestAI: - Type: Task - Resource: 'arn:aws:states:::lambda:invoke' - ResultSelector: - identity_id.$: $$.Execution.Input.body.identity_id - task_id.$: $$.Execution.Input.body.task_id - project_id.$: $$.Execution.Input.body.project_id - project_name.$: $$.Execution.Input.body.project_name - project_prefix.$: $$.Execution.Input.body.project_prefix - current_num_retries.$: $.Payload.current_num_retries - max_retries.$: $.Payload.max_retries - batch.$: $.Payload.batch - response.$: $.Payload.response - is_retry.$: $.Payload.is_retry - Parameters: - FunctionName: '${WorkerRequestAI}' - Payload.$: $ - ResultPath: $ - OutputPath: $ - Next: IsCheckRetry - IsCheckRetry: - Type: Choice - Choices: - - Variable: $.is_retry - BooleanEquals: true - Next: RequestAI - - Variable: $.is_retry - BooleanEquals: false - Next: UploadToS3 - UploadToS3: - Type: Task - Resource: 'arn:aws:states:::lambda:invoke' - InputPath: $ - OutputPath: $.Payload - Parameters: - FunctionName: '${HandlerUploadBatchToS3}' - Payload.$: $ - End: true - ResultPath: $ - Next: MergeResultRequestAI - MergeResultRequestAI: - Type: Task - Resource: 'arn:aws:states:::lambda:invoke' - InputPath: $ - ResultSelector: - id_token.$: $$.Execution.Input.body.id_token - identity_id.$: $$.Execution.Input.body.identity_id - task_id.$: $$.Execution.Input.body.task_id - project_id.$: $$.Execution.Input.body.project_id - project_name.$: $$.Execution.Input.body.project_name - project_prefix.$: $$.Execution.Input.body.project_prefix - response.$: $.Payload.response - state.$: $.Payload.state - status.$: $.Payload.status - info_update_s3.$: $.Payload.info_update_s3 - gen_id.$: $.Payload.gen_id - Parameters: - FunctionName: '${HandleMergeResultRequestAI}' - Payload.$: $ - ResultPath: $ - OutputPath: $ - Next: CheckResponse - CheckResponse: - Type: Choice - Choices: - - Variable: $.response - StringEquals: OK - Next: HandleCompleteRequestAI - - Variable: $.response - StringEquals: NOT_OK - Next: ChoiceStageRequestAI - HandleCompleteRequestAI: - Type: Task - Resource: 'arn:aws:states:::lambda:invoke' - InputPath: $ - OutputPath: $.Payload - Parameters: - FunctionName: '${HandleCompleteRequestAI}' - Payload.$: $ - Next: ChoiceStageRequestAI - GenerateComplete: - Type: Pass - Comment: Used for result aggregation - End: true -TimeoutSeconds: 1800 diff --git a/daita-app/ai-caller-service/statemachine/generate_state_machine.asl.yaml.copy b/daita-app/ai-caller-service/statemachine/generate_state_machine.asl.yaml.copy deleted file mode 100644 index f159ce8..0000000 --- a/daita-app/ai-caller-service/statemachine/generate_state_machine.asl.yaml.copy +++ /dev/null @@ -1,118 +0,0 @@ -StartAt: GenerateTask -States: - GenerateTask: - Type: Task - Resource: 'arn:aws:states:::lambda:invoke' - InputPath: $ - OutputPath: $.Payload - Parameters: - FunctionName: '${HandleGenerateStep}' - Payload.$: $ - Next: ChoiceStageRequestAI - Comment: >- - Check the level of parallelism, split requests into chunks and invoke - lamndas - Retry: - - ErrorEquals: - - RetriableCallerServiceError - IntervalSeconds: 1 - MaxAttempts: 2 - BackoffRate: 1 - ChoiceStageRequestAI: - Type: Choice - Choices: - - Variable: $.state - StringEquals: Request_AI - Next: HandleBatchRequestAI - - Variable: $.state - StringEquals: FINISH - Next: UpdateStatusTask - - Variable: $.state - StringEquals: ERROR - Next: UpdateStatusTask - UpdateStatusTask: - Type: Task - Resource: 'arn:aws:states:::lambda:invoke' - Next: GenerateComplete - InputPath: $ - OutputPath: $ - Parameters: - FunctionName: '${UpdateStatusTask}' - Payload.$: $ - Retry: - - ErrorEquals: - - RetriableCallerServiceError - IntervalSeconds: 1 - MaxAttempts: 2 - BackoffRate: 1 - HandleBatchRequestAI: - Type: Map - MaxConcurrency: 5 - InputPath: $ - ItemsPath: $.list_request_ai - Parameters: - batch.$: $$.Map.Item.Value - identity_id.$: $.identity_id - task_id.$: $.task_id - Iterator: - StartAt: RequestAI - States: - RequestAI: - Type: Task - Resource: 'arn:aws:states:::lambda:invoke' - OutputPath: $.Payload - Parameters: - FunctionName: '${WorkerRequestAI}' - Payload.$: $ - Next: IsCheckRetry - IsCheckRetry: - Type: Choice - Choices: - - Variable: $.is_retry - BooleanEquals: true - Next: RequestAI - - Variable: $.is_retry - BooleanEquals: false - Next: UploadToS3 - UploadToS3: - Type: Task - Resource: 'arn:aws:states:::lambda:invoke' - OutputPath: $.Payload - Parameters: - FunctionName: '${HandlerUploadBatchToS3}' - Payload.$: $ - End: true - ResultPath: $ - Next: MergeResultRequestAI - MergeResultRequestAI: - Type: Task - Resource: 'arn:aws:states:::lambda:invoke' - InputPath: $ - OutputPath: $.Payload - Parameters: - FunctionName: '${HandleMergeResultRequestAI}' - Payload.$: $ - Next: CheckResponse - CheckResponse: - Type: Choice - Choices: - - Variable: $.reponse - StringEquals: OK - Next: HandleCompleteRequestAI - - Variable: $.reponse - StringEquals: NOT_OK - Next: ChoiceStageRequestAI - HandleCompleteRequestAI: - Type: Task - Resource: 'arn:aws:states:::lambda:invoke' - InputPath: $ - OutputPath: $.Payload - Parameters: - FunctionName: '${HandleCompleteRequestAI}' - Payload.$: $ - Next: ChoiceStageRequestAI - GenerateComplete: - Type: Pass - Comment: Used for result aggregation - End: true -TimeoutSeconds: 1800 diff --git a/daita-app/auth-service/CognitoClient/template.yaml b/daita-app/auth-service/CognitoClient/template.yaml new file mode 100644 index 0000000..21e8376 --- /dev/null +++ b/daita-app/auth-service/CognitoClient/template.yaml @@ -0,0 +1,104 @@ +Parameters: + DomainDaita: + Type: String + CognitoUserPool: + Type: String + + StagePara: + Type: String + AuthHttpAPI: + Type: String + + GoogleClientID: + Type: String + GoogleClientSecret: + Type: String + GithubClientID: + Type: String + GithubClientSecret: + Type: String + + +Resources: + GoogleCognitoUserPoolIdentityProvider: + Type: AWS::Cognito::UserPoolIdentityProvider + Properties: + ProviderName: "Google" + AttributeMapping: + email: "email" + username: sub + middle_name: middle_name + family_name: family_name + given_name: given_name + picture: picture + ProviderDetails: + client_id: !Ref GoogleClientID + client_secret: !Ref GoogleClientSecret + authorize_scopes: profile email openid + ProviderType: Google + UserPoolId: + Ref : CognitoUserPool + GithubCognitoUserPoolIdentityProvider: + Type: AWS::Cognito::UserPoolIdentityProvider + Properties: + UserPoolId: !Ref CognitoUserPool + ProviderName: github + ProviderDetails: + client_id: !Ref GithubClientID + client_secret: !Ref GithubClientSecret + attributes_request_method: GET + oidc_issuer: !Sub "https://${AuthHttpAPI}.execute-api.${AWS::Region}.amazonaws.com/${StagePara}" + authorize_scopes: "openid read:user user:email" + jwks_uri: !Sub "https://${AuthHttpAPI}.execute-api.${AWS::Region}.amazonaws.com/${StagePara}/auth/github-openid-token-wrapper" + token_url: !Sub "https://${AuthHttpAPI}.execute-api.${AWS::Region}.amazonaws.com/${StagePara}/auth/github-openid-token-wrapper" + attributes_url: !Sub "https://${AuthHttpAPI}.execute-api.${AWS::Region}.amazonaws.com/${StagePara}/auth/github-openid-userinfo-wrapper" + authorize_url: https://github.com/login/oauth/authorize + ProviderType: "OIDC" + AttributeMapping: + email: "email" + username: sub + name: "email" + CognitoUserPoolClient: + Type: AWS::Cognito::UserPoolClient + DependsOn: + - GoogleCognitoUserPoolIdentityProvider + - GithubCognitoUserPoolIdentityProvider + Properties: + UserPoolId: !Ref CognitoUserPool + ClientName: user-pool-client + GenerateSecret: false + AllowedOAuthFlowsUserPoolClient: true + CallbackURLs: + - !Sub https://${AuthHttpAPI}.execute-api.${AWS::Region}.amazonaws.com/${StagePara}/auth/login_social + LogoutURLs: + - http://localhost:3000 + - !Ref DomainDaita + AllowedOAuthFlows: + - code + - implicit + AllowedOAuthScopes: + - phone + - email + - openid + - profile + - aws.cognito.signin.user.admin + SupportedIdentityProviders: + - COGNITO + - github + - Google + AccessTokenValidity: 2 + RefreshTokenValidity: 24 + IdTokenValidity: 2 + TokenValidityUnits: + AccessToken: hours + IdToken: hours + RefreshToken: hours + AllowedOAuthFlowsUserPoolClient: true + ExplicitAuthFlows: + - ALLOW_REFRESH_TOKEN_AUTH + - ALLOW_USER_PASSWORD_AUTH + - ALLOW_USER_SRP_AUTH + PreventUserExistenceErrors: ENABLED +Outputs: + UserPoolClientId: + Value: !Ref CognitoUserPoolClient \ No newline at end of file diff --git a/daita-app/auth-service/CognitoUserPool/api-defs/daita_http_api.yaml b/daita-app/auth-service/CognitoUserPool/api-defs/daita_http_api.yaml new file mode 100644 index 0000000..b2a7004 --- /dev/null +++ b/daita-app/auth-service/CognitoUserPool/api-defs/daita_http_api.yaml @@ -0,0 +1,72 @@ +openapi: "3.0.1" +info: + title: + Fn::Sub: "${StagePara}-Daita-Auth-HTTP-API" + version: "2021-04-07" +tags: +- name: "httpapi:createdBy" + x-amazon-apigateway-tag-value: "SAM" +paths: + /auth/login_social: + get: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${LoginSocialFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /auth/github-openid-userinfo-wrapper: + get: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${GithubUserinfoWrapperFunc.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + post: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${GithubUserinfoWrapperFunc.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /auth/github-openid-token-wrapper: + get: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${GithubTokenWrapperFunc.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + post: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${GithubTokenWrapperFunc.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" \ No newline at end of file diff --git a/daita-app/auth-service/CognitoUserPool/functions/github_openid_token_wrapper/__init__.py b/daita-app/auth-service/CognitoUserPool/functions/github_openid_token_wrapper/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/auth-service/CognitoUserPool/functions/github_openid_token_wrapper/app.py b/daita-app/auth-service/CognitoUserPool/functions/github_openid_token_wrapper/app.py new file mode 100644 index 0000000..a48e04d --- /dev/null +++ b/daita-app/auth-service/CognitoUserPool/functions/github_openid_token_wrapper/app.py @@ -0,0 +1,29 @@ +import json +import requests + +from config import * +from response import error_response +import base64 + +OAUTH_TOKEN_URL = "login/oauth/access_token" + + +@error_response +def lambda_handler(event, context): + print(event) + if event["isBase64Encoded"]: + event["body"] = base64.b64decode(event["body"]).decode("ascii") + response = requests.post( + url=f"{GITHUB_LOGIN_URL}/{OAUTH_TOKEN_URL}", + headers={ + 'Content-Type': 'application/x-www-form-urlencoded', + 'Accept': 'application/json' + }, + data=event["body"], + ) + print(response.json()) + # return { + # "body": json.dumps(response.json()), + # "isBase64Encoded": False + # } + return json.dumps(response.json()) \ No newline at end of file diff --git a/daita-app/auth-service/CognitoUserPool/functions/github_openid_token_wrapper/requirements.txt b/daita-app/auth-service/CognitoUserPool/functions/github_openid_token_wrapper/requirements.txt new file mode 100644 index 0000000..663bd1f --- /dev/null +++ b/daita-app/auth-service/CognitoUserPool/functions/github_openid_token_wrapper/requirements.txt @@ -0,0 +1 @@ +requests \ No newline at end of file diff --git a/daita-app/auth-service/CognitoUserPool/functions/github_openid_userinfo_wrapper/__init__.py b/daita-app/auth-service/CognitoUserPool/functions/github_openid_userinfo_wrapper/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/auth-service/CognitoUserPool/functions/github_openid_userinfo_wrapper/app.py b/daita-app/auth-service/CognitoUserPool/functions/github_openid_userinfo_wrapper/app.py new file mode 100644 index 0000000..0147eed --- /dev/null +++ b/daita-app/auth-service/CognitoUserPool/functions/github_openid_userinfo_wrapper/app.py @@ -0,0 +1,49 @@ +import json +import requests + +from config import * +from response import error_response + + +OAUTH_USERINFO_URL = "user" +OAUTH_USER_EMAIL_URL = "user/emails" + + +@error_response +def lambda_handler(event, context): + print(event) + oauth_token = event["headers"].get("authorization").split("Bearer ")[1] + userinfo_response = requests.get( + url=f"{GITHUB_API_URL}/{OAUTH_USERINFO_URL}", + headers={ + "Authorization": f"token {oauth_token}", + "Accept": "application/json" + }, + allow_redirects=False + ) + + useremail_response = requests.get( + url=f"{GITHUB_API_URL}/{OAUTH_USER_EMAIL_URL}", + headers={ + "Authorization": f"token {oauth_token}", + "Accept": "application/json" + }, + allow_redirects=False + ) + useremails = useremail_response.json() + for email in useremails: + if email["primary"]: + primary_email = email["email"] + break + else: + primary_email = None + + body = userinfo_response.json() + print(body) + body["email"] = primary_email + return json.dumps( + { + **body, + "sub": body["id"] + } + ) \ No newline at end of file diff --git a/daita-app/auth-service/CognitoUserPool/functions/github_openid_userinfo_wrapper/requirements.txt b/daita-app/auth-service/CognitoUserPool/functions/github_openid_userinfo_wrapper/requirements.txt new file mode 100644 index 0000000..663bd1f --- /dev/null +++ b/daita-app/auth-service/CognitoUserPool/functions/github_openid_userinfo_wrapper/requirements.txt @@ -0,0 +1 @@ +requests \ No newline at end of file diff --git a/daita-app/auth-service/CognitoUserPool/functions/login_social/app.py b/daita-app/auth-service/CognitoUserPool/functions/login_social/app.py new file mode 100644 index 0000000..97abd05 --- /dev/null +++ b/daita-app/auth-service/CognitoUserPool/functions/login_social/app.py @@ -0,0 +1,125 @@ +from email import header +import os +import json +import logging +import time +from datetime import datetime +from http import HTTPStatus +import os +import boto3 + +from error_messages import * +from response import * +from config import * +from lambda_base_class import LambdaBaseClass + +import base64 +from urllib.parse import urlencode +ACCESS_TOKEN_EXPIRATION = 24 * 60 * 60 +USERPOOLID = os.environ['COGNITO_USER_POOL'] +cog_provider_client = boto3.client('cognito-idp') +cog_identity_client = boto3.client('cognito-identity') + + +@error_response +def lambda_handler(event, context): + param = event['queryStringParameters'] + try: + code = param['code'] + except Exception as e: + print(e) + if 'error_description' in param: + location = LOCATION + headers = {"Location": location, + "Access-Control-Allow-Methods": "GET,HEAD,OPTIONS,POST,PUT"} + return { + "statusCode": 302, + "headers": headers, + "body": '', + "isBase64Encoded": False + } + raise Exception(e) + + if 'state' in param: + path = base64.b64decode(param['state']).decode('utf-8') + else: + path = 'http://localhost:3000/login' + mapping = { + 'token': '', + 'resfresh_token': '', + 'access_key': '', + 'session_key': '', + 'id_token': '', + 'credential_token_expires_in': '', + 'token_expires_in': '', + 'secret_key': '', + 'identity_id': '', + 'username': '', + 'code': code + } + location = path + '?' + urlencode(mapping, doseq=True) + headers = {"Location": location, + "Access-Control-Allow-Methods": "GET,HEAD,OPTIONS,POST,PUT"} + return { + "statusCode": 302, + "headers": headers, + "body": '', + "isBase64Encoded": False + } + + +class LoginSocialClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def parser(self, body): + self.code = body['code'] + + def handle(self, event, context): + param = event['queryStringParameters'] + try: + self.parser(param) + except Exception as e: + if 'error_description' in param: + location = LOCATION + headers = {"Location": location, + "Access-Control-Allow-Methods": "GET,HEAD,OPTIONS,POST,PUT"} + return { + "statusCode": 302, + "headers": headers, + "body": '', + "isBase64Encoded": False + } + raise Exception(e) + + if 'state' in param: + path = base64.b64decode(param['state']).decode('utf-8') + else: + path = 'http://localhost:3000/login' + mapping = { + 'token': '', + 'resfresh_token': '', + 'access_key': '', + 'session_key': '', + 'id_token': '', + 'credential_token_expires_in': '', + 'token_expires_in': '', + 'secret_key': '', + 'identity_id': '', + 'username': '', + 'code': self.code + } + location = path + '?' + urlencode(mapping, doseq=True) + headers = {"Location": location, + "Access-Control-Allow-Methods": "GET,HEAD,OPTIONS,POST,PUT"} + return { + "statusCode": 302, + "headers": headers, + "body": '', + "isBase64Encoded": False + } + + +@error_response +def lambda_handler(event, context): + return LoginSocialClass().handle(event=event, context=context) diff --git a/daita-app/auth-service/CognitoUserPool/template.yaml b/daita-app/auth-service/CognitoUserPool/template.yaml new file mode 100644 index 0000000..5853991 --- /dev/null +++ b/daita-app/auth-service/CognitoUserPool/template.yaml @@ -0,0 +1,199 @@ +AWSTemplateFormatVersion: '2010-09-09' +Transform: AWS::Serverless-2016-10-31 +Description: > + daita-caller-service-app + + Sample SAM Template for daita-caller-service-app +Parameters: + StagePara: + Type: String + Mode: + Type: String + CommonCodeLayerName: + Type: String + minimumLogLevel: + Type: String + Default: DEBUG + + CertificateUserpoolDomain: + Type: String + DomainUserPool: + Type: String + +Globals: + Function: + Timeout: 800 + Handler: app.lambda_handler + Runtime: python3.8 + Architectures: + - x86_64 + Environment: + Variables: + STAGE: !Ref StagePara + MODE: !Ref Mode + COGNITO_USER_POOL: !Ref CognitoUserPool + LOGGING: !Ref minimumLogLevel + Layers: + - !Ref CommonCodeLayerName +Resources: + ApiGatewayCallLambdaRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Principal: + Service: "apigateway.amazonaws.com" + Action: + - "sts:AssumeRole" + Policies: + - PolicyName: RestApiDirectInvokeLambda + PolicyDocument: + Version: "2012-10-17" + Statement: + Action: + - "lambda:InvokeFunction" + Effect: Allow + Resource: "*" + LambdaExecutionRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Principal: + Service: "lambda.amazonaws.com" + Action: + - "sts:AssumeRole" + Policies: + - PolicyName: 'SQS' + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - 'sqs:*' + Resource: '*' + - PolicyName: 'SecretsManagerParameterAccess' + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - ssm:GetParam* + - ssm:DescribeParam* + Resource: + - arn:aws:ssm:*:*:parameter/* + - PolicyName: 'CloudwatchPermission' + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - 'logs:*' + Resource: '*' + - PolicyName: 'CognitoPermission' + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - cognito-identity:* + - cognito-idp:* + Resource: '*' + - PolicyName: 'DynamoDBPermission' + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - dynamodb:* + Resource: "*" + - PolicyName: "OtherServicePermission" + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - events:PutEvents + Resource: "*" + - Effect: Allow + Action: + - states:StartExecution + - s3:* + Resource: "*" + - PolicyName: "InvokeFunction" + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - lambda:InvokeFunction + Resource: "*" + - PolicyName: "SES" + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - 'ses:*' + Resource: "*" + + AuthSocialHttpAPI: + Type: AWS::Serverless::HttpApi + Properties: + StageName: !Ref StagePara + DefinitionBody: + Fn::Transform: + Name: AWS::Include + Parameters: + Location: './api-defs/daita_http_api.yaml' + + CognitoUserPool: + Type: AWS::Cognito::UserPool + Properties: + UserPoolName: !Ref AWS::StackName + + # CognitoUserPoolDomain: + # Type: AWS::Cognito::UserPoolDomain + # Properties: + # Domain: !Ref DomainUserPool + # UserPoolId: !Ref CognitoUserPool + # CustomDomainConfig: + # CertificateArn: !Ref CertificateUserpoolDomain + + GithubTokenWrapperFunc: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/github_openid_token_wrapper + Role: !GetAtt LambdaExecutionRole.Arn + + GithubUserinfoWrapperFunc: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/github_openid_userinfo_wrapper + Role: !GetAtt LambdaExecutionRole.Arn + + LoginSocialFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/login_social + Role: !GetAtt LambdaExecutionRole.Arn + +Outputs: + UserPool: + Value: !Ref CognitoUserPool + AuthHttpAPI: + Value: !Ref AuthSocialHttpAPI + AuthApiURL: + Value: !Sub "https://${AuthSocialHttpAPI}.execute-api.${AWS::Region}.amazonaws.com/${StagePara}" + ProviderNameUserPool: + Value: !GetAtt CognitoUserPool.ProviderName \ No newline at end of file diff --git a/daita-app/auth-service/IdentityPool/template.yaml b/daita-app/auth-service/IdentityPool/template.yaml new file mode 100644 index 0000000..17db26e --- /dev/null +++ b/daita-app/auth-service/IdentityPool/template.yaml @@ -0,0 +1,17 @@ +Parameters: + UserPoolClientId: + Type: String + ProviderNameUserPool: + Type: String +Resources: + IdentityPool: + Type: "AWS::Cognito::IdentityPool" + Properties: + IdentityPoolName: !Ref AWS::StackName + AllowUnauthenticatedIdentities: true + CognitoIdentityProviders: + - ClientId: !Ref UserPoolClientId + ProviderName: !Ref ProviderNameUserPool +Outputs: + IdentityPool: + Value: !Ref IdentityPool \ No newline at end of file diff --git a/daita-app/auth-service/RoleIdentity/template.yaml b/daita-app/auth-service/RoleIdentity/template.yaml new file mode 100644 index 0000000..6971380 --- /dev/null +++ b/daita-app/auth-service/RoleIdentity/template.yaml @@ -0,0 +1,116 @@ +Parameters: + Bucket: + Type: String + IdentityPool: + Type: String + StagePara: + Type: String + ApplicationPara: + Type: String + AnnoBucket: + Type: String + +Resources: + CognitoUnAuthorizedRole: + Type: AWS::IAM::Role + Properties: + RoleName: !Sub "${StagePara}-${ApplicationPara}-unauthRole" + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Principal: + Federated: "cognito-identity.amazonaws.com" + Action: sts:AssumeRoleWithWebIdentity + Condition: + StringEquals: + cognito-identity.amazonaws.com:aud: + Ref: IdentityPool + ForAnyValue:StringLike: + cognito-identity.amazonaws.com:amr: unauthenticated + Policies: + - PolicyName: "CognitoUnAuthorizedPolicy" + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Action: + - "cognito-sync:*" + Resource: "*" + + CognitoAuthorizedRole: + Type: AWS::IAM::Role + Properties: + RoleName: !Sub "${StagePara}-${ApplicationPara}-authRole" + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Principal: + Federated: cognito-identity.amazonaws.com + Action: sts:AssumeRoleWithWebIdentity + Condition: + StringEquals: + cognito-identity.amazonaws.com:aud: + Ref: IdentityPool + ForAnyValue:StringLike: + cognito-identity.amazonaws.com:amr: authenticated + Policies: + - PolicyName: CognitoAuthorizedPolicy + PolicyDocument: + Version: "2012-10-17" + Statement: + - Sid: VisualEditor0 + Effect: Allow + Action: + - 's3:*' + Resource: + - !Join [ "", [ "arn:aws:s3:::", !Ref Bucket, "/${cognito-identity.amazonaws.com:sub}" ] ] + - !Join [ "", [ "arn:aws:s3:::", !Ref Bucket, "/${cognito-identity.amazonaws.com:sub}/*" ] ] + - !Join [ "", [ "arn:aws:s3:::", !Ref AnnoBucket, "/${cognito-identity.amazonaws.com:sub}" ] ] + - !Join [ "", [ "arn:aws:s3:::", !Ref AnnoBucket, "/${cognito-identity.amazonaws.com:sub}/*" ] ] + - Sid: VisualEditor1 + Effect: Allow + Action: 's3:ListBucket' + Resource: !Join [ "", [ "arn:aws:s3:::", !Ref Bucket ] ] ###"arn:aws:s3:::${Bucket}" + Condition: + StringLike: + 's3:prefix': + - '' + - / + - '${cognito-identity.amazonaws.com:sub}/*' + - Sid: VisualEditor4 + Effect: Allow + Action: 's3:ListBucket' + Resource: !Join [ "", [ "arn:aws:s3:::", !Ref AnnoBucket ] ] ###"arn:aws:s3:::${Bucket}" + Condition: + StringLike: + 's3:prefix': + - '' + - / + - '${cognito-identity.amazonaws.com:sub}/*' + - Sid: VisualEditor2 + Effect: Allow + Action: + - 's3:ListAllMyBuckets' + - 'kms:*' + - 's3:GetBucketLocation' + - "cognito-sync:*" + - "cognito-identity:*" + Resource: '*' + # - Sid: VisualEditor3 + # Effect: Allow + # Action: 's3:*' + # Resource: + # - !Join [ "", [ "arn:aws:s3:::", !Ref Bucket, "/${cognito-identity.amazonaws.com:sub}" ] ] # TODO check enviorment app + # - !Join [ "", [ "arn:aws:s3:::", !Ref Bucket, "/${cognito-identity.amazonaws.com:sub}/*" ] ] ##"arn:aws:s3:::client-data-test/${cognito-identity.amazonaws.com:sub}/*" # TODO check enviorment app + CognitoIdentityPoolRoles: + Type: AWS::Cognito::IdentityPoolRoleAttachment + Properties: + IdentityPoolId: + Ref: IdentityPool + Roles: + authenticated: + Fn::GetAtt: [CognitoAuthorizedRole,Arn] + unauthenticated: + Fn::GetAtt: [CognitoUnAuthorizedRole,Arn] \ No newline at end of file diff --git a/daita-app/build.sh b/daita-app/build.sh deleted file mode 100755 index e10fa16..0000000 --- a/daita-app/build.sh +++ /dev/null @@ -1,107 +0,0 @@ -#!/bin/bash - -read -p "Please input the stage name: " STAGE - -if [ -z $STAGE ] -then - echo Input the Stage name must not empty! - exit -fi - -STAGE="${STAGE,,}" - -if [[ $STAGE == "dev" ]] || [[ $STAGE == "prod" ]] -then - echo 1234 - read -p "Are you sure that you want to deploy with dev/prod env [y/N]: " confirm - confirm=${confirm:-n} - - if [[ $confirm == "n" ]] || [[ $confirm == "N" ]] - then - echo "Exit here" - exit - elif [[ $confirm == "y" ]] || [[ $confirm == "Y" ]] - then - echo =============================== - echo Start deploy with env: $STAGE - else - exit - fi -fi - -###=== AWS config ====== -AWS_REGION="us-east-2" -AWS_ACCOUNT_ID="366577564432" - - -sam build -sam deploy --no-confirm-changeset --disable-rollback --config-env $STAGE | tee output.txt -output=$(cat output.txt) -echo ========================================= -output=$(echo $output) - -echo @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ -echo Get value of ecr - -shopt -s extglob - - -keyname="DecompressEcrRepositoryName" -a=$output -[[ $a =~ Key[[:space:]]$keyname[[:space:]].+Key ]] -a=${BASH_REMATCH[0]} -[[ $a =~ Value.+Key ]] -a=${BASH_REMATCH[0]} -[[ $a =~ [[:space:]].+[[:space:]] ]] -a=${BASH_REMATCH[0]} -### strip space -a=${a##*( )} -a=${a%%*( )} - -keyname2="CompressDownloadEcrRepositoryName" -b=$output -echo ----------------------- -echo $b - -[[ $b =~ Key[[:space:]]$keyname2[[:space:]].+$keyname ]] -b=${BASH_REMATCH[0]} -[[ $b =~ Value.+Key ]] -b=${BASH_REMATCH[0]} -echo b0 $b -[[ $b =~ [[:space:]].+[[:space:]] ]] -b=${BASH_REMATCH[0]} -### strip space -b=${b##*( )} -b=${b%%*( )} - - -###=== ECR config========= -IMAGE_REPO_NAME=$a -IMAGE_TAG="latest" - -### Login to ecr -aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com - -### Build image for decompress -cd ./dataflow-service/decompress-upload-app/tasks/decompress -docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG . -docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG - -### Push image to ECR -docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG - -###=== ECR config========= -IMAGE_REPO_NAME1=$b -IMAGE_TAG1="latest" -echo build $IMAGE_REPO_NAME1 -cd .. -cd .. -cd .. -pwd -cd ./compress-download-app/tasks/download -pwd -docker build -t $IMAGE_REPO_NAME1:$IMAGE_TAG1 . -docker tag $IMAGE_REPO_NAME1:$IMAGE_TAG1 $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$IMAGE_REPO_NAME1:$IMAGE_TAG1 - -### Push image to ECR -docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$IMAGE_REPO_NAME1:$IMAGE_TAG1 \ No newline at end of file diff --git a/daita-app/build_daita.sh b/daita-app/build_daita.sh new file mode 100755 index 0000000..f156e9f --- /dev/null +++ b/daita-app/build_daita.sh @@ -0,0 +1,171 @@ +#!/bin/bash + + +### load config file +. "$1" + +OUTPUT_BUILD_DAITA=$2 +OUTPUT_FE_CONFIG=$3 + +### load output of daita-app +. "$OUTPUT_BUILD_DAITA" + + +cd daita-app + + +parameters_override="Mode=${MODE} Stage=${DAITA_STAGE} Application=${DAITA_APPLICATION} + SecurityGroupIds=${SECURITY_GROUP_IDS} SubnetIDs=${SUB_NET_IDS} + S3BucketName=${DAITA_S3_BUCKET} + EFSFileSystemId=${EFS_ID} + MaxConcurrencyTasks=${MAX_CONCURRENCY_TASK} + ROOTEFS=${ROOT_EFS} + VPCid=${VPC_ID} + CertificateUserpoolDomain=${CERTIFICATE_USERPOLL_DOMAIN} + S3AnnoBucket=${ANNO_S3_BUCKET} + PublicSubnetOne=${PublicSubnetOne} + PublicSubnetTwo=${PublicSubnetTwo} + ContainerSecurityGroup=${ContainerSecurityGroup} + VPC=${VPC} + VPCEndpointSQSDnsEntries=${VPCEndpointSQSDnsEntries} + TokenOauth2BotSlackFeedBack=${OAUTH2BOT_SLACK_FEED_BACK} + DomainUserPool=${DOMAIN_USER_POOL} + DomainDaita=${DOMAIN_DAITA} + GoogleClientID=${GOOGLE_CLIENT_ID} + GoogleClientSecret=${GOOGLE_CLIENT_SECRET} + GithubClientID=${GITHUB_CLIENT_ID} + GithubClientSecret=${GITHUB_CLIENT_SECRET} + OauthEndpoint=${OAUTH_ENPOINT} + CaptchaSiteKeyGoogle=${CAPTCHA_SITE_KEY_GOOGLE} + CaptchaSecretKeyGoogle=${CAPTCHA_SECRET_KEY_GOOGLE}" + +sam build +sam deploy --no-confirm-changeset --disable-rollback --resolve-s3 \ + --resolve-image-repos --config-env $DAITA_STAGE \ + --stack-name "$DAITA_STAGE-${DAITA_APPLICATION}-app" \ + --s3-prefix "$DAITA_STAGE-${DAITA_APPLICATION}-app" \ + --region $AWS_REGION \ + --parameter-overrides $parameters_override | tee output.txt + + +echo @@@@@@@@@@@@@@@@@@@@@@@@ Upload docker to ECR ========== + +shopt -s extglob + +declare -A dict_output +filename="output.txt" + +while read line; do + # reading each line + if [[ "$line" =~ "Key".+ ]]; then + + [[ "$line" =~ [[:space:]].+ ]] + a=${BASH_REMATCH[0]} + a=${a##*( )} + a=${a%%*( )} + key=$a + fi + if [[ "$line" =~ "Value".+ ]]; then + [[ "$line" =~ [[:space:]].+ ]] + value=${BASH_REMATCH[0]} + value=${value##*( )} + value=${value%%*( )} + + first_line=$value + is_first_line_value=true + else + if [[ "$line" =~ .*"-------".* ]]; then + echo "skip line" + else + if [ "$is_first_line_value" = true ]; then + final_line=$first_line$line + dict_output[$key]=$final_line + is_first_line_value=false + fi + fi + fi +done < $filename + +DecompressEcrRepositoryName=${dict_output["DecompressEcrRepositoryName"]} +CompressDownloadEcrRepositoryName=${dict_output["CompressDownloadEcrRepositoryName"]} +CognitoUserPoolRef=${dict_output["CognitoUserPoolRef"]} +CognitoIdentityPoolIdRef=${dict_output["CognitoIdentityPoolIdRef"]} +CommonCodeLayerRef=${dict_output["CommonCodeLayerRef"]} +TableDaitaProjectsName=${dict_output["TableDaitaProjectsName"]} +TableDaitaDataOriginalName=${dict_output["TableDaitaDataOriginalName"]} +TableUserName=${dict_output["TableUserName"]} +### for lambda functions +SendEmailIdentityIDFunction=${dict_output["SendEmailIdentityIDFunction"]} +### export for FE config +ApiDaitaAppUrl=${dict_output["ApiDaitaAppUrl"]} +ApiAuthDaitaUrl=${dict_output["ApiAuthDaitaUrl"]} +CognitoAppIntegrateID=${dict_output["CognitoAppIntegrateID"]} + + +###=== ECR config========= +IMAGE_REPO_NAME=$DecompressEcrRepositoryName +IMAGE_TAG="latest" + +### Login to ecr +aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com + +### Build image for decompress +cd ./dataflow-service/decompress-upload-app/tasks/decompress +docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG . +docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG + +### Push image to ECR +docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG + +###=== ECR config========= +IMAGE_REPO_NAME1=$CompressDownloadEcrRepositoryName +IMAGE_TAG1="latest" +echo build $IMAGE_REPO_NAME1 +cd .. +cd .. +cd .. +pwd +cd ./compress-download-app/tasks/download +pwd +docker build -t $IMAGE_REPO_NAME1:$IMAGE_TAG1 . +docker tag $IMAGE_REPO_NAME1:$IMAGE_TAG1 $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$IMAGE_REPO_NAME1:$IMAGE_TAG1 + +### Push image to ECR +docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$IMAGE_REPO_NAME1:$IMAGE_TAG1 + + +###======= Store output to file ========== +echo "CognitoUserPoolRef=$CognitoUserPoolRef" >> $OUTPUT_BUILD_DAITA +echo "CognitoIdentityPoolIdRef=$CognitoIdentityPoolIdRef" >> $OUTPUT_BUILD_DAITA +echo "CommonCodeLayerRef=$CommonCodeLayerRef" >> $OUTPUT_BUILD_DAITA +echo "TableDaitaProjectsName=$TableDaitaProjectsName" >> $OUTPUT_BUILD_DAITA +echo "TableDaitaDataOriginalName=$TableDaitaDataOriginalName" >> $OUTPUT_BUILD_DAITA +echo "TableUserName=$TableUserName" >> $OUTPUT_BUILD_DAITA +echo "SendEmailIdentityIDFunction=$SendEmailIdentityIDFunction" >> $OUTPUT_BUILD_DAITA + + + +###========= SAVE FE CONFIG =============== +echo "REACT_APP_AUTH_API_URL=$ApiDaitaAppUrl" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_INVITE_API_URL=$ApiDaitaAppUrl" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_PROJECT_API_URL=$ApiDaitaAppUrl" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_GENERATE_API_URL=$ApiDaitaAppUrl" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_HEALTH_CHECK_API_URL=$ApiDaitaAppUrl" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_DOWNLOAD_ZIP_API=$ApiDaitaAppUrl" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_UPLOAD_ZIP_API=$ApiDaitaAppUrl" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_TASK_API_URL=$ApiDaitaAppUrl" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_CREATE_PROJECT_SAMPLE=$ApiDaitaAppUrl" >> $OUTPUT_FE_CONFIG + +echo "REACT_APP_S3_BUCKET_NAME=$DAITA_S3_BUCKET" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_S3_BUCKET_ANNOTATION_NAME=$ANNO_S3_BUCKET" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_S3_REGION=$AWS_REGION" >> $OUTPUT_FE_CONFIG + +echo "REACT_APP_RECAPTCHA_SITE_KEY=6LcqEGMeAAAAAAEDnBue7fwR4pmvNO7JKWkHtAjl" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_API_AMAZON_COGNITO=https://${DOMAIN_USER_POOL}/oauth2/authorize" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_COGNITO_REDIRECT_URI=${ApiAuthDaitaUrl}/auth/login_social" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_COGNITO_CLIENTID=${CognitoAppIntegrateID}" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_API_LOGOUT_SOCIAL=https://${DOMAIN_USER_POOL}/logout?client_id=${CognitoAppIntegrateID}" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_GITHUB_IDENTITY_PROVIDER=github" >> $OUTPUT_FE_CONFIG + +echo "REACT_APP_FEEDBACK_SLACK=${ApiDaitaAppUrl}/webhook/client-feedback" >> $OUTPUT_FE_CONFIG +echo "REACT_APP_PRESIGN_URL_UPLOAD_FEEDBACK_IMAGE=${ApiDaitaAppUrl}/feedback/presign_url_image" >> $OUTPUT_FE_CONFIG \ No newline at end of file diff --git a/daita-app/core-service/api-defs/daita_http_api.yaml b/daita-app/core-service/api-defs/daita_http_api.yaml index f279601..49589b9 100644 --- a/daita-app/core-service/api-defs/daita_http_api.yaml +++ b/daita-app/core-service/api-defs/daita_http_api.yaml @@ -7,6 +7,255 @@ tags: - name: "httpapi:createdBy" x-amazon-apigateway-tag-value: "SAM" paths: + /webhook/client-feedback: + post: + responses: + default: + description: "FeedBack" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${SlackWebhookFeedbackFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + ##################################################################################################################################################################### + /auth/credential: + post: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${CredentialLoginFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /auth/user_signup: + post: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${RegisterFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /auth/user_logout: + post: + responses: + default: + description: "log out" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${LogoutFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /auth/user_login: + post: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${LoginFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /auth/auth_confirm: + post: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${AuthConfirmFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /auth/resend_confirmcode: + post: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ResendCodeAuthConfirmCodeFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /auth/confirm_code_forgot_password: + post: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ConfirmCodeForgotPasswordFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /auth/forgot_password: + post: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ForgotpasswordFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /auth/refresh_token: + post: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${LoginRefreshTokenFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /auth/template-invite-mail: + get: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${TemplateMailInviteFriendFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /send-mail/reference-email: + post: + responses: + default: + description: "Auth service transport" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ReferenceEmailFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + ##################################################################################################################################################################### + /feedback/presign_url_image: + post: + responses: + default: + description: "check daita upload token" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${CreatePresignUrlForImageFeedbackFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /cli/create_decompress_task: + post: + responses: + default: + description: "check daita upload token" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${CreateDecompressFileCLIFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /cli/check_existence_file: + post: + responses: + default: + description: "check daita upload token" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${CliCheckExistenceFileFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /cli/create_presignurl: + post: + responses: + default: + description: "check daita upload token" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${CreatePresignUrlFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /cli/check_daita_token: + get: + responses: + default: + description: "check daita upload token" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${CheckDaitaTokenFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /cli/upload_project: + post: + responses: + default: + description: "generate daita upload token" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${CliUploadProjectFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /generate/daita_upload_token: + post: + responses: + default: + description: "generate daita upload token" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${GenertateTokenUploadProjectFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" /generate/generate_images: post: responses: @@ -76,8 +325,36 @@ paths: type: "aws_proxy" payloadFormatVersion: "2.0" - ###============================== - ### For project functions + # ###============================== + # ### For project functions + /projects/list_prebuild_dataset: + post: + responses: + default: + description: "" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ListPrebuildDatasetFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + + /projects/create_project_from_prebuild: + post: + responses: + default: + description: "" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${FunctionCreateProjectFromPrebuild.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /projects/apply_expert_mode_param: post: responses: @@ -118,6 +395,137 @@ paths: httpMethod: "POST" type: "aws_proxy" payloadFormatVersion: "2.0" + /projects/create: + post: + responses: + default: + description: "create project" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ProjectCreateSampleFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /projects/create_sample: + post: + responses: + default: + description: "create project" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ProjectSampleFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /projects/list_info: + post: + responses: + default: + description: "list_info" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ProjectListInfoFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /projects/info: + post: + responses: + default: + description: "create project" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ProjectInfoFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /projects/update_info: + post: + responses: + default: + description: "create project" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ProjectUpdateInfoFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /projects/list_data: + post: + responses: + default: + description: "create project" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ProjectListDataFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /projects/download_create: + post: + responses: + default: + description: "create project" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ProjectDownloadCreateFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /projects/download_update: + post: + responses: + default: + description: "create project" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ProjectDownloadUpdateFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + /projects/upload_check: + post: + responses: + default: + description: "create project" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ProjectUpdateCheckFunction.Arn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" + + /projects/upload_update: + post: + responses: + default: + description: "create project" + x-amazon-apigateway-integration: + credentials: + Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] + uri: + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${FuncProjectUploadUpdateArn}/invocations + httpMethod: "POST" + type: "aws_proxy" + payloadFormatVersion: "2.0" ####=================================================================================== #### For healthcheck API /health_check/calculate: @@ -173,7 +581,7 @@ paths: credentials: Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] uri: - Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${RICalculateFunction.Arn}/invocations + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${RICalculateFunctionArn}/invocations httpMethod: "POST" type: "aws_proxy" payloadFormatVersion: "2.0" @@ -187,7 +595,7 @@ paths: credentials: Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] uri: - Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${RIStatusFunction.Arn}/invocations + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${RIStatusFunctionArn}/invocations httpMethod: "POST" type: "aws_proxy" payloadFormatVersion: "2.0" @@ -201,7 +609,7 @@ paths: credentials: Fn::GetAtt: [ApiGatewayCallLambdaRole, Arn] uri: - Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${RIInfoFunction.Arn}/invocations + Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${RIInfoFunctionArn}/invocations httpMethod: "POST" type: "aws_proxy" payloadFormatVersion: "2.0" diff --git a/daita-app/core-service/core_service_template.yaml b/daita-app/core-service/core_service_template.yaml index 0c85b7f..fa47429 100644 --- a/daita-app/core-service/core_service_template.yaml +++ b/daita-app/core-service/core_service_template.yaml @@ -16,20 +16,38 @@ Globals: - x86_64 Environment: Variables: + CHANNELWEBHOOK: !Ref ChannelWebhook + AUTH_ENDPOINT: !Ref AuthEndpoint STAGE: !Ref StagePara TABLE_NAME: global-table LOGGING: !Ref minimumLogLevel + REGION: !Ref AWS::Region TABLE_GENERATE_TASK: !Ref TableGenerateTaskName - TABLE_PROJECTS_NAME: !Ref TableProjectsName TABLE_METHODS_NAME: !Ref TableMethodsName TABLE_HEALTHCHECK_TASK: !Ref TableHealthCheckTasksName + TABLE_DATA_FLOW_TASK: !Ref ParaTableDataFlowTaskName + TABLE_REFERENCE_IMAGE_TASK: !Ref TableReferenceImageTasksName TABLE_HEALTHCHECK_INFO: !Ref TableHealthCheckInfoName INDEX_TASK_PROJECTID_TASKID: !Ref ParaIndexTaskProjectIDTaskID + TABLE_CONFIRM_CODE: !Ref TableConfirmCodeAuth + TABLE_USER: !Ref TableUser + COGNITO_USER_POOL: !Ref CognitoUserPool + COGNITO_CLIENT_ID: !Ref CognitoUserPoolClient + IDENTITY_POOL: !Ref CognitoIdentityPoolId + BUCKET_NAME: !Ref S3BucketName + TABLE_PROJECT: !Ref TableProjectsName + TABLE_TASK: !Ref TableTask + TABLE_PROJECT_SUMMARY: !Ref TableProjectSumName + TABLE_EVENTUSER: !Ref TableEventUser + MODE: !Ref Mode + IS_ENABLE_KMS: "False" Layers: - !Ref CommonCodeLayerName ## The general rule seems to be to use !Sub for in line substitutions and !ref for stand alone text Parameters: + ApplicationPara: + Type: String minimumLogLevel: Type: String Default: DEBUG @@ -43,15 +61,15 @@ Parameters: Type: String ProcessAITaskEventBusArn: Type: String - ProjectSummary: + + AICallerECSSMArn: + Type: String + + TableProjectSumName: Type: String - Default: prj_sum_all DataOrigin: Type: String Default: data_original - TableProject: - Type: String - Default: projects TableProjectDel: Type: String Default: projects_save_delete @@ -61,9 +79,7 @@ Parameters: Type: String TableMethodsName: Type: String - ParaTableDownloadTaskName: - Type: String - ParaTableProjectSumName: + TableProjectSumName: Type: String TableHealthCheckTasksName: Type: String @@ -77,6 +93,9 @@ Parameters: Type: String TableReferenceImageInfoName: Type: String + TableConstPrebuildDatasetName: + Type: String + MaxConcurrencyTasks: Type: String CommonCodeLayerName: @@ -90,19 +109,28 @@ Parameters: Type: String HealthCheckEventBusName: Type: String + ReferenceImageEventBusName: Type: String + RICalculateFunctionArn: + Type: String + RIStatusFunctionArn: + Type: String + RIInfoFunctionArn: + Type: String ParaDecompressFileStateMachineArn: Type: String ParaCompressDownloadStateMachineArn: Type: String - CognitoUserPoolId: - Type: String - Default: us-east-2_ZbwpnYN4g + CognitoIdentityPoolId: Type: String - Default: us-east-2:fa0b76bc-01fa-4bb8-b7cf-a5000954aafb + CognitoUserPool: + Type: String + CognitoUserPoolClient: + Type: String + StopProcessEventBusArn: Type: String StopProcessEventBusName: @@ -115,8 +143,117 @@ Parameters: Type: String TableDataAugmentName: Type: String + TableGenerateDaitaUploadToken: + Type: String + S3BucketName: + Type: String + Mode: + Type: String + + + + TableUser: + Type: String + CreateProjectPrebuildSMArn: + Type: String + TableConfirmCodeAuth: + Type: String + TableTask: + Type: String + FuncProjectUploadUpdateArn: + Type: String + TableDataPreprocess: + Type: String + TableDataAugment: + Type: String + StreamTableDataOriginalName: + Type: String + StreamTableDataPreprocessName: + Type: String + StreamTableDataAugmentName: + Type: String + TableEventUser: + Type: String + TableFeedback: + Type: String + AuthEndpoint: + Type: String + + TokenOauth2BotSlackFeedBack: + Type: String + ChannelWebhook: + Type: String + OauthEndpoint: + Type: String + CaptchaSiteKeyGoogle: + Type: String + CaptchaSecretKeyGoogle: + Type: String + Resources: + ThumbnailServiceEventBusRole: + Type: "AWS::IAM::Role" + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Principal: + Service: "events.amazonaws.com" + Action: + - "sts:AssumeRole" + Policies: + - PolicyName: DirectlyInvokeStepFunctions + PolicyDocument: + Version: "2012-10-17" + Statement: + Action: + - "states:StartExecution" + Effect: Allow + Resource: + - !Ref ThumbnailStateMachine + + TriggerThumbnailFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/thumbnail/trigger_thumbnail + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + EVENT_BUS_NAME: !Ref ThumbnailEventBus + ResizeImageThumbnailFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/thumbnail/resize_image + MemorySize: 512 + Role: !GetAtt LambdaExecutionRole.Arn + + DivideBatchImagesThumbnailFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/thumbnail/divide_batch + Role: !GetAtt LambdaExecutionRole.Arn + ThumbnailEventBus: + Type: AWS::Events::EventBus + Properties: + Name: !Sub "${StagePara}-${ApplicationPara}-ThumbnailEventBus" + ThumbnailServiceEventBusDefaultRule: + Type: AWS::Events::Rule + Properties: + Description: "Default Rule for Any event" + State: ENABLED + EventBusName: !Ref ThumbnailEventBus + EventPattern: + source: + - "source.events" + detail-type: + - "lambda.event" + Targets: + - + Arn: !GetAtt ThumbnailStateMachine.Arn + Id: "ThumbnailStateMachine" + RoleArn: !GetAtt ThumbnailServiceEventBusRole.Arn #================ ROLES ===================================================== # use this role for apigateway access lambda @@ -154,6 +291,14 @@ Resources: Action: - "sts:AssumeRole" Policies: + - PolicyName: 'SES' + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - 'ses:*' + Resource: '*' - PolicyName: 'SQS' PolicyDocument: Version: '2012-10-17' @@ -216,7 +361,44 @@ Resources: - states:StartExecution - s3:* Resource: "*" - + - PolicyName: "InvokeFunction" + PolicyDocument: + Version: '2012-10-17' + Statement: + - + Effect: Allow + Action: + - lambda:InvokeFunction + - lambda:* + Resource: "*" +#============================Event Source Mapping======================================== + EventSourceMappingDatabaseOriginal: + Type: AWS::Lambda::EventSourceMapping + Properties: + BatchSize: 1000 + Enabled: True + EventSourceArn: !Ref StreamTableDataOriginalName + FunctionName: !Ref TriggerThumbnailFunction + StartingPosition: LATEST + MaximumBatchingWindowInSeconds: 120 + EventSourceMappingDatabasePreprocess: + Type: AWS::Lambda::EventSourceMapping + Properties: + BatchSize: 1000 + Enabled: True + EventSourceArn: !Ref StreamTableDataPreprocessName + FunctionName: !Ref TriggerThumbnailFunction + StartingPosition: LATEST + MaximumBatchingWindowInSeconds: 120 + EventSourceMappingDatabaseAugment: + Type: AWS::Lambda::EventSourceMapping + Properties: + BatchSize: 1000 + Enabled: True + EventSourceArn: !Ref StreamTableDataAugmentName + FunctionName: !Ref TriggerThumbnailFunction + StartingPosition: LATEST + MaximumBatchingWindowInSeconds: 120 #================ CALLER SERVICE HTTP API =================================== DaitaHttpApi: @@ -228,10 +410,108 @@ Resources: Name: AWS::Include Parameters: Location: './api-defs/daita_http_api.yaml' + #================ LOGS FOR STEP FUNCTIONS =================================== + ThumbnaiServiceStateMachineLogGroup: + Type: AWS::Logs::LogGroup + Properties: + LogGroupName: !Sub "/aws/vendedlogs/states/${StagePara}-${ApplicationPara}-Thumbnail" + RetentionInDays: 7 + ThumbnailStateMachine: + Type: AWS::Serverless::StateMachine + Properties: + Type: STANDARD + Name: !Sub "${StagePara}-${ApplicationPara}-ThumbnailStatemachine" + Policies: + - LambdaInvokePolicy: + FunctionName: !Ref ResizeImageThumbnailFunction + FunctionName: !Ref DivideBatchImagesThumbnailFunction + - Statement: + - Sid: ALLOWCRUDDynamoDB + Effect: Allow + Action: + - "dynamodb:*" + - "lambda:InvokeFunction" + Resource: "*" + - Sid: CloudWatchLogsPolicy + Effect: Allow + Action: + - "logs:*" + Resource: "*" + Tracing: + Enabled: true + DefinitionUri: statemachine/thumbnail_step_function.asl.yaml + Logging: + Level: ALL + IncludeExecutionData: true + Destinations: + - CloudWatchLogsLogGroup: + LogGroupArn: !GetAtt ThumbnaiServiceStateMachineLogGroup.Arn + DefinitionSubstitutions: + ResizeImageThumbnailFunction: !GetAtt ResizeImageThumbnailFunction.Arn + DivideBatchImagesThumbnailFunction: !GetAtt DivideBatchImagesThumbnailFunction.Arn #================ LAMBDA FUNCTIONS ========================================== + CreatePresignUrlForImageFeedbackFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/feedback/presignUrl + Role: !GetAtt LambdaExecutionRole.Arn + CreateDecompressFileCLIFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/cli/create_decompress_task + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + T_GEN_DAITA_UPLOAD_TOKEN: !Ref TableGenerateDaitaUploadToken + DECOMPRESS_LAMBDA_INVOKE: !Ref CreateDecompressTaskFunction + CliCheckExistenceFileFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/cli/check_existence_file + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + T_GEN_DAITA_UPLOAD_TOKEN: !Ref TableGenerateDaitaUploadToken + LAMBDA_UPDATE_CHECK: !Ref ProjectUpdateCheckFunction + CreatePresignUrlFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/cli/create_presignurl_zip + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + T_GEN_DAITA_UPLOAD_TOKEN: !Ref TableGenerateDaitaUploadToken + CheckDaitaTokenFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/cli/check_daita_token + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + T_GEN_DAITA_UPLOAD_TOKEN: !Ref TableGenerateDaitaUploadToken + + CliUploadProjectFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/cli/cli_upload_project + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + T_GEN_DAITA_UPLOAD_TOKEN: !Ref TableGenerateDaitaUploadToken + TABLE_PROJECT_SUMMARY: !Ref TableProjectSumName + LAMBDA_UPLOAD_UPDATE: !Ref FuncProjectUploadUpdateArn + LAMBDA_UPLOAD_CHECK: !Ref ProjectUpdateCheckFunction + GenertateTokenUploadProjectFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/generate/daita_upload_token + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + T_GEN_DAITA_UPLOAD_TOKEN: !Ref TableGenerateDaitaUploadToken GenerateCheckConditionFunction: Type: AWS::Serverless::Function Properties: @@ -240,12 +520,14 @@ Resources: Environment: Variables: EVENT_BUS_NAME: !Ref ProcessAITaskEventBusName + AI_CALLER_ECS_SM_ARN: !Ref AICallerECSSMArn LIMIT_PROCESSING_TIMES: !Ref LimitPreprocessTimesName LIMIT_AUGMENT_TIMES: !Ref LimitAugmentTimesName - TABLE_PROJECT_SUM: !Ref ParaTableProjectSumName + TABLE_PROJECT_SUMMARY: !Ref TableProjectSumName TABLE_PREPROCESS: !Ref TableDataPreprocessName MAX_CONCURRENCY_TASK: !Ref MaxConcurrencyTasks QUEUE : !Ref TaskQueueName + TaskProgressFunction: Type: AWS::Serverless::Function Properties: @@ -283,89 +565,77 @@ Resources: DeleteImages: Type: AWS::Serverless::Function Properties: - CodeUri: functions/handlers/delete_images + CodeUri: functions/handlers/project/delete_images Role: !GetAtt LambdaExecutionRole.Arn Environment: Variables: TABLE_HEALTHCHECK_INFO: !Ref TableHealthCheckInfoName - T_PROJECT_SUMMARY: !Ref ProjectSummary - T_DATA_ORI: !Ref DataOrigin - T_PROJECT: !Ref TableProject + TABLE_PROJECT_SUMMARY: !Ref TableProjectSumName + T_DATA_ORI: !Ref TableDataOriginalName DeleteProject: Type: AWS::Serverless::Function Properties: - CodeUri: functions/handlers/delete_project + CodeUri: functions/handlers/project/delete_project Role: !GetAtt LambdaExecutionRole.Arn Environment: Variables: TABLE_HEALTHCHECK_INFO: !Ref TableHealthCheckInfoName TABLE_HEALTHCHECK_TASK: !Ref TableHealthCheckTasksName - T_PROJECT_SUMMARY: !Ref ProjectSummary - T_DATA_ORI: !Ref DataOrigin + TABLE_PROJECT_SUMMARY: !Ref TableProjectSumName + T_DATA_ORI: !Ref TableDataOriginalName T_TASKS: !Ref TableGenerateTaskName - T_PROJECT: !Ref TableProject T_PROJECT_DEL: !Ref TableProjectDel T_DATA_FLOW: !Ref ParaTableDataFlowTaskName T_REFERENCE_IMAGE: !Ref TableReferenceImageTasksName - ### for health check functions - HealthCheckFunction: - Type: AWS::Serverless::Function - Properties: - CodeUri: functions/handlers/health_check/calculate - Role: !GetAtt LambdaExecutionRole.Arn - Environment: - Variables: - EVENT_BUS_NAME: !Ref HealthCheckEventBusName - TABLE_HEALTHCHECK_TASK: !Ref TableHealthCheckTasksName - - HCStatusFunction: + ListPrebuildDatasetFunction: Type: AWS::Serverless::Function Properties: - CodeUri: functions/handlers/health_check/get_status + CodeUri: functions/handlers/project/list_prebuild_dataset Role: !GetAtt LambdaExecutionRole.Arn Environment: Variables: - TABLE_HEALTHCHECK_TASK: !Ref TableHealthCheckTasksName + T_CONST_PREBUILD_DATASET: !Ref TableConstPrebuildDatasetName - HCInfoFunction: + FunctionCreateProjectFromPrebuild: Type: AWS::Serverless::Function Properties: - CodeUri: functions/handlers/health_check/get_info + CodeUri: functions/handlers/project/create_prj_fr_prebuild Role: !GetAtt LambdaExecutionRole.Arn Environment: Variables: - TABLE_HEALTHCHECK_INFO: !Ref TableHealthCheckInfoName + T_CONST_PREBUILD_DATASET: !Ref TableConstPrebuildDatasetName + SM_CREATE_PRJ_PREBUILD: !Ref CreateProjectPrebuildSMArn - ### for reference images functions - RICalculateFunction: + ### for health check functions + HealthCheckFunction: Type: AWS::Serverless::Function Properties: - CodeUri: functions/handlers/reference_image/calculate + CodeUri: functions/handlers/health_check/calculate Role: !GetAtt LambdaExecutionRole.Arn Environment: Variables: - EVENT_BUS_NAME: !Ref ReferenceImageEventBusName - TABLE_REFERENCE_IMAGE_TASK: !Ref TableReferenceImageTasksName + EVENT_BUS_NAME: !Ref HealthCheckEventBusName + TABLE_HEALTHCHECK_TASK: !Ref TableHealthCheckTasksName - RIStatusFunction: + HCStatusFunction: Type: AWS::Serverless::Function Properties: - CodeUri: functions/handlers/reference_image/get_status + CodeUri: functions/handlers/health_check/get_status Role: !GetAtt LambdaExecutionRole.Arn Environment: Variables: - TABLE_REFERENCE_IMAGE_TASK: !Ref TableReferenceImageTasksName + TABLE_HEALTHCHECK_TASK: !Ref TableHealthCheckTasksName - RIInfoFunction: + HCInfoFunction: Type: AWS::Serverless::Function Properties: - CodeUri: functions/handlers/reference_image/get_info + CodeUri: functions/handlers/health_check/get_info Role: !GetAtt LambdaExecutionRole.Arn Environment: Variables: - TABLE_REFERENCE_IMAGE_INFO: !Ref TableReferenceImageInfoName + TABLE_HEALTHCHECK_INFO: !Ref TableHealthCheckInfoName ### For augment_img_review GetAugmentationImageReviewFunction: @@ -430,10 +700,207 @@ Resources: TableGenerateTaskName: !Ref TableGenerateTaskName TableReferenceImageName: !Ref TableReferenceImageTasksName + LoginFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/auth_service/login + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + CAPTCHA_SITE_KEY_GOOGLE: !Ref CaptchaSiteKeyGoogle + CAPTCHA_SECRET_KEY_GOOGLE: !Ref CaptchaSecretKeyGoogle + + LogoutFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/auth_service/logout + Role: !GetAtt LambdaExecutionRole.Arn + + RegisterFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/auth_service/sign_up + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + INVOKE_MAIL_LAMBDA: !Ref MailServiceFunction + CAPTCHA_SITE_KEY_GOOGLE: !Ref CaptchaSiteKeyGoogle + CAPTCHA_SECRET_KEY_GOOGLE: !Ref CaptchaSecretKeyGoogle + + LoginRefreshTokenFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/auth_service/login_refresh_token + Role: !GetAtt LambdaExecutionRole.Arn + + AuthConfirmFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/auth_service/auth_confirm + Role: !GetAtt LambdaExecutionRole.Arn + + ConfirmCodeForgotPasswordFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/auth_service/confirm_code_forgot_password + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + CAPTCHA_SITE_KEY_GOOGLE: !Ref CaptchaSiteKeyGoogle + CAPTCHA_SECRET_KEY_GOOGLE: !Ref CaptchaSecretKeyGoogle + + CredentialLoginFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/auth_service/credential_login + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + TBL_EVENTUSER: !Ref TableEventUser + OAUTH_ENPOINT: !Ref OauthEndpoint + + ResendCodeAuthConfirmCodeFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/auth_service/resend_confirmcode + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + INVOKE_MAIL_LAMBDA: !Ref MailServiceFunction + + ForgotpasswordFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/auth_service/forgot_password + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + INVOKE_MAIL_LAMBDA: !Ref MailServiceFunction + + MailServiceFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/auth_service/mail_service + Role: !GetAtt LambdaExecutionRole.Arn + TemplateMailInviteFriendFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/auth_service/template_mail + Role: !GetAtt LambdaExecutionRole.Arn + + ReferenceEmailFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/send-mail/reference-email + Role: !GetAtt LambdaExecutionRole.Arn +######################################################################################### + SlackWebhookFeedbackFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/feedback/slack_webhook + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + TABLE_FEEBACK: !Ref TableFeedback + TOKEN_OAUTH2BOT_SLACK_FEEDBACK: !Ref TokenOauth2BotSlackFeedBack + +##########################Project Service################################################ + + ProjectAsyCreateSampleFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/project/project_asy_create_sample + Role: !GetAtt LambdaExecutionRole.Arn + + ProjectCreateSampleFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/project/project_create + Role: !GetAtt LambdaExecutionRole.Arn + + ProjectDownloadCreateFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/project/project_download_create + Role: !GetAtt LambdaExecutionRole.Arn + + ProjectDownloadUpdateFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/project/project_download_update + Role: !GetAtt LambdaExecutionRole.Arn + + ProjectInfoFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/project/project_info + Role: !GetAtt LambdaExecutionRole.Arn + + ProjectListFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/project/project_list + Role: !GetAtt LambdaExecutionRole.Arn + + ProjectListDataFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/project/project_list_data + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + T_DATA_ORI: !Ref TableDataOriginalName + T_DATA_PREPROCESS: !Ref TableDataPreprocessName + T_DATA_AUGMENT: !Ref TableDataAugmentName + + ProjectListInfoFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/project/project_list_info + Role: !GetAtt LambdaExecutionRole.Arn + + ProjectSampleFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/project/project_sample + Role: !GetAtt LambdaExecutionRole.Arn + + ProjectUpdateInfoFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/project/project_update_info + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + T_DATA_ORI: !Ref TableDataOriginalName + T_DATA_PREPROCESS: !Ref TableDataPreprocessName + T_DATA_AUGMENT: !Ref TableDataAugmentName + ProjectUpdateCheckFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/project/project_upload_check + Role: !GetAtt LambdaExecutionRole.Arn + Environment: + Variables: + T_DATA_ORI: !Ref TableDataOriginalName + T_DATA_PREPROCESS: !Ref TableDataPreprocessName + T_DATA_AUGMENT: !Ref TableDataAugmentName + + # #### for email related functions + SendEmailIdentityIDFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/handlers/send-mail/send-email-identity-id + Handler: hdler_send_email_to_identityid.lambda_handler + Role: !GetAtt LambdaExecutionRole.Arn Outputs: CallerServiceHttpApiUrl: Description: "Url of the Caller service API" - Value: !Sub "https://${DaitaHttpApi}.execute-api.${AWS::Region}.amazonaws.com" + Value: !Sub "https://${DaitaHttpApi}.execute-api.${AWS::Region}.amazonaws.com/${StagePara}" LambdaRoleArn: - Value: !GetAtt LambdaExecutionRole.Arn \ No newline at end of file + Value: !GetAtt LambdaExecutionRole.Arn + + SendEmailIdentityIDFunction: + Value: !Ref SendEmailIdentityIDFunction + \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/auth_service/auth_confirm/__init__.py b/daita-app/core-service/functions/handlers/auth_service/auth_confirm/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/auth_service/auth_confirm/app.py b/daita-app/core-service/functions/handlers/auth_service/auth_confirm/app.py new file mode 100644 index 0000000..2e025d1 --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/auth_confirm/app.py @@ -0,0 +1,62 @@ +from cmath import inf +import os +import json +import logging +from http import HTTPStatus +import os +import boto3 +from dataclasses import dataclass +import re +from utils import * +from error_messages import * +from response import * +from config import * +from custom_mail import * +from datetime import datetime + +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] +cog_provider_client = boto3.client('cognito-idp') +cog_identity_client = boto3.client('cognito-identity') + + +@error_response +def lambda_handler(event, context): + try: + body = json.loads(event['body']) + username = body['username'] + confirmCode = body['confirm_code'] + except Exception as e: + print("error ", e) + return generate_response( + message=MessageUnmarshalInputJson, + data={}, + headers=RESPONSE_HEADER + ) + try: + DeleteConfirmCode({ + 'region': REGION, + 'user': username, + 'code': confirmCode, + 'confirm_code_Table': os.environ['TABLE_CONFIRM_CODE'] + }) + except Exception as e: + raise Exception(e) + _ = cog_provider_client.admin_confirm_sign_up( + UserPoolId=USERPOOLID, + Username=username + ) + + _ = cog_provider_client.admin_update_user_attributes( + UserPoolId=USERPOOLID, + Username=username, + UserAttributes=[ + { + 'Name': 'email_verified', + 'Value': 'true' + }, + ]) + return generate_response(message="Email successfully confirmed", data={}, + headers=RESPONSE_HEADER + ) diff --git a/daita-app/core-service/functions/handlers/auth_service/confirm_code_forgot_password/__init__.py b/daita-app/core-service/functions/handlers/auth_service/confirm_code_forgot_password/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/auth_service/confirm_code_forgot_password/app.py b/daita-app/core-service/functions/handlers/auth_service/confirm_code_forgot_password/app.py new file mode 100644 index 0000000..9f3c4aa --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/confirm_code_forgot_password/app.py @@ -0,0 +1,69 @@ +import re +import os +import boto3 +import json +from config import * +from error_messages import * +from custom_mail import * +from utils import * +from response import generate_response, error_response +from lambda_base_class import LambdaBaseClass + +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + +cog_provider_client = boto3.client('cognito-idp') +RESPONSE_HEADER = { + "Access-Control-Allow-Credentials": "true", + "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", +} +PASSWORD_REGEX = r"^(?=.*[a-z])(?=.*[A-Z])(?=.*[0-9])(?=.*[\^$*.\[\]{}\(\)?\-\"!@#%&\/,><\':;|_~`])\S{8,99}$" + + +class ConfirmCodeForgotPasswordClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + @LambdaBaseClass.parse_body + def parser(self, body): + self.username = body["username"] + self.password = body["password"] + self.confirm_code = body["confirm_code"] + + def handle(self, event, context): + self.parser(event) + try: + DeleteConfirmCode({ + 'region': REGION, + 'user': self.username, + 'code': self.confirm_code, + 'confirm_code_Table': os.environ['TABLE_CONFIRM_CODE'] + }) + except Exception as e: + raise Exception(e) + + if not re.match(PASSWORD_REGEX, self.password): + AddInsertConfirmCode( + info={'user': self.username, 'confirm_code': self.confirm_code, 'confirm_code_Table': os.environ['TABLE_CONFIRM_CODE']}) + raise Exception(MessageInvalidPassword) + + try: + response = cog_provider_client.admin_set_user_password( + UserPoolId=USERPOOLID, Username=self.username, Password=self.password, Permanent=True) + print(response) + except Exception as exc: + print(exc) + AddInsertConfirmCode( + info={'user': self.username, 'confirm_code': self.confirm_code, 'confirm_code_Table': os.environ['TABLE_CONFIRM_CODE']}) + raise Exception(MessageForgotPasswordConfirmcodeFailed) from exc + + return generate_response( + message=MessageForgotPasswordConfirmcodeSuccessfully, + headers=RESPONSE_HEADER + ) + + +@error_response +def lambda_handler(event, context): + return ConfirmCodeForgotPasswordClass().handle(event=event, context=context) diff --git a/daita-app/core-service/functions/handlers/auth_service/credential_login/__init__.py b/daita-app/core-service/functions/handlers/auth_service/credential_login/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/auth_service/credential_login/app.py b/daita-app/core-service/functions/handlers/auth_service/credential_login/app.py new file mode 100644 index 0000000..af1ae59 --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/credential_login/app.py @@ -0,0 +1,268 @@ +from email import header +import os +import json +import logging +import time +from datetime import datetime +from http import HTTPStatus +import os +import boto3 +import cognitojwt +from utils import aws_get_identity_id +from urllib.parse import urlparse, quote +from error_messages import * +from response import * +from config import * +from utils import * +from models.event_model import * +import requests +from urllib.parse import urlencode +from lambda_base_class import LambdaBaseClass + +ACCESS_TOKEN_EXPIRATION = 24 * 60 * 60 +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] +AUTH_ENDPOINT = os.environ['AUTH_ENDPOINT'] +REGION = os.environ['REGION'] +STAGE = os.environ['STAGE'] +cog_provider_client = boto3.client('cognito-idp') +cog_identity_client = boto3.client('cognito-identity') +# endpoint = 'https://devdaitaloginsocial.auth.us-east-2.amazoncognito.com/oauth2/token' + +endpoint = os.environ['OAUTH_ENPOINT'] + + +TableUser = os.environ['TABLE_USER'] + + +def getRedirectURI(): + return f'https://{AUTH_ENDPOINT}.execute-api.{REGION}.amazonaws.com/{STAGE}/auth/login_social' + +############################################################################################################################################################# + + +def getMail(user): + response = cog_provider_client.list_users( + UserPoolId=USERPOOLID + ) + # info_user = list(filter(lambda x : x['Username'] == user,response['Users'])) + + for _, data in enumerate(response['Users']): + if data['Username'] == user: + for info in data['Attributes']: + if info['Name'] == 'email': + return info['Value'] + return None + + +def checkInvalidUserRegister(user, mail): + isCheckMail = True + response = cog_provider_client.list_users( + UserPoolId=USERPOOLID + ) + for _, data in enumerate(response['Users']): + for info in data['Attributes']: + if info['Name'] == 'email' and info['Value'] == mail and data['Username'] != user: + isCheckMail = False + # break + return isCheckMail +############################################################################################################################################################# + + +def createKMSKey(identity): + alias_name = identity.split(":")[1] + kms = boto3.client("kms", region_name=REGION) + + key = kms.create_key() + key_id = key["KeyMetadata"]["KeyId"] + kms.create_alias( + AliasName="alias/"+alias_name, + TargetKeyId=key_id + ) + return key_id + +############################################################################################################# + + +def getCredentialsForIdentity(token_id): + PROVIDER = f'cognito-idp.{REGION}.amazonaws.com/{USERPOOLID}' + responseIdentity = aws_get_identity_id( + token_id, USERPOOLID, IDENTITY_POOL) + credentialsResponse = cog_identity_client.get_credentials_for_identity( + IdentityId=responseIdentity, + Logins={ + PROVIDER: token_id + }) + return { + 'secret_key': credentialsResponse['Credentials']['SecretKey'], + 'session_key': credentialsResponse['Credentials']['SessionToken'], + 'credential_token_expires_in': credentialsResponse['Credentials']['Expiration'].timestamp() * 1000, + 'access_key': credentialsResponse['Credentials']['AccessKeyId'], + 'identity_id': responseIdentity + } +############################################################################################################# + + +def Oauth2(code): + print(f'check {code} {getRedirectURI()}') + params = {"code": code, "grant_type": "authorization_code", "redirect_uri": getRedirectURI( + ), 'client_id': CLIENTPOOLID, 'scope': 'email+openid+phone+profile'} + headers = {"Content-Type": "application/x-www-form-urlencoded"} + data = urlencode(params) + result = requests.post(endpoint, data=data, headers=headers) + print(result.text) + return result +############################################################################################################ + + +def getDisplayName(username): + response = cog_provider_client.admin_get_user( + UserPoolId=USERPOOLID, + Username=username + ) + user_attributes = {} + for att in response["UserAttributes"]: + user_attributes[att["Name"]] = att["Value"] + + name = "" + if "name" in user_attributes: + name = user_attributes["name"] + elif "given_name" in user_attributes or \ + "family_name" in user_attributes: + name = f"{user_attributes.pop('given_name', '')} {user_attributes.pop('family_name', '')}" + else: + name = user_attributes["email"] + + return name + # return None +####################################################################################################### + + +def claimsToken(jwt_token, field): + """ + Validate JWT claims & retrieve user identifier + """ + token = jwt_token.replace("Bearer ", "") + try: + verified_claims = cognitojwt.decode( + token, REGION, USERPOOLID + ) + except Exception as e: + print(e) + verified_claims = {} + + return verified_claims.get(field) +############################################################################################################################################################# + + +class User(object): + def __init__(self): + self.db_client = boto3.resource('dynamodb') + self.TBL = TableUser + + def IsNotcheckFirstLogin(self, ID, username): + response = self.db_client.Table(self.TBL).get_item( + Key={ + 'ID': ID, + 'username': username + } + ) + if 'Item' in response and (response['Item']['status'] == "activate" or response['Item']['status'] == "deactivate"): + return True + return False + + def updateActivateUser(self, info): + self.db_client.Table(self.TBL).update_item( + Key={'ID': info['ID'], 'username': info['username']}, + UpdateExpression="SET #s = :s , #i = :i , #k = :k , #u = :u", + ExpressionAttributeValues={ + ':s': 'activate', + ':i': info['indentityID'], + ':k': info['kms'], + ':u': datetime.now().isoformat(), + }, + ExpressionAttributeNames={ + '#s': 'status', + '#i': 'identity_id', + '#k': 'kms_key_id', + '#u': 'update_at' + } + ) +############################################################################################################################################################# + + +class CredentialSocialLoginClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + @LambdaBaseClass.parse_body + def parser(self, body): + self.code = body['code'] + + def handle(self, event, context): + self.parser(event) + model = User() + resq = Oauth2(self.code) + if resq.status_code != 200: + raise Exception("Login Social Failed") + resqData = resq.json() + sub, username = claimsToken(resqData['access_token'], 'sub'), claimsToken( + resqData['access_token'], 'username') + mail = getMail(username) + checkemail = checkInvalidUserRegister(user=username, mail=mail) + if not CheckEventUserLogin(sub): + CreateEventUserLoginOauth2(sub, self.code) + try: + credentialsForIdentity = getCredentialsForIdentity( + resqData['id_token']) + except Exception as e: + print(e) + return generate_response( + message=MessageAuthenFailed, + data={}, + headers=RESPONSE_HEADER) + if not model.IsNotcheckFirstLogin(ID=sub, username=username): + if not checkemail: + resp = cog_provider_client.admin_delete_user( + UserPoolId=USERPOOLID, + Username=username + ) + print(resp) + raise Exception(MessageSignUPEmailInvalid) + responseIdentity = aws_get_identity_id( + resqData['id_token'], USERPOOLID, IDENTITY_POOL) + if 'IS_ENABLE_KMS' in os.environ and eval(os.environ['IS_ENABLE_KMS']) == True: + kms = createKMSKey(responseIdentity) + else: + kms = '' + model.updateActivateUser(info={ + 'indentityID': responseIdentity, + 'ID': sub, + 'username': username, + 'kms': kms, + }) + name = getDisplayName(username) + result = { + 'token': resqData['access_token'], + 'resfresh_token': resqData['refresh_token'], + 'access_key': credentialsForIdentity['access_key'], + 'session_key': credentialsForIdentity['session_key'], + 'id_token': resqData['id_token'], + 'credential_token_expires_in': credentialsForIdentity['credential_token_expires_in'], + 'token_expires_in': float(int((datetime.now().timestamp() + ACCESS_TOKEN_EXPIRATION)*1000)), + 'secret_key': credentialsForIdentity['secret_key'], + 'identity_id': credentialsForIdentity['identity_id'], + 'username': username, + 'name': name + } + return generate_response( + message=MessageSuccessfullyCredential, + data=result, + headers=RESPONSE_HEADER + ) + + +@error_response +def lambda_handler(event, context): + return CredentialSocialLoginClass().handle(event=event, context=context) diff --git a/daita-app/core-service/functions/handlers/auth_service/credential_login/requirements.txt b/daita-app/core-service/functions/handlers/auth_service/credential_login/requirements.txt new file mode 100644 index 0000000..f8e9300 --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/credential_login/requirements.txt @@ -0,0 +1,2 @@ +requests +cognitojwt \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/auth_service/forgot_password/__init__.py b/daita-app/core-service/functions/handlers/auth_service/forgot_password/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/auth_service/forgot_password/app.py b/daita-app/core-service/functions/handlers/auth_service/forgot_password/app.py new file mode 100644 index 0000000..7aecebd --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/forgot_password/app.py @@ -0,0 +1,106 @@ +from email import header, message +import os +from datetime import datetime +from http import HTTPStatus + +import boto3 +import json +from config import * +from error_messages import * +from verify_captcha import * +from custom_mail import * +from response import generate_response, error_response +from lambda_base_class import LambdaBaseClass +from utils import * + +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + +cog_provider_client = boto3.client('cognito-idp') +cog_identity_client = boto3.client('cognito-identity') +RESPONSE_HEADER = { + "Access-Control-Allow-Credentials": "true", + "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", +} + + +def getMail(user): + response = cog_provider_client.list_users( + UserPoolId=USERPOOLID + ) + # info_user = list(filter(lambda x : x['Username'] == user,response['Users'])) + + for _, data in enumerate(response['Users']): + if data['Username'] == user: + for info in data['Attributes']: + if info['Name'] == 'email': + return info['Value'] + return None + + +class ForgotPasswordClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + + @LambdaBaseClass.parse_body + def parser(self, body): + self.username = body["username"] + self.captcha = body["captcha"] + + def handle(self, event, context): + + self.parser(event) + + try: + verify_captcha(self.captcha, self.env.CAPTCHA_SITE_KEY_GOOGLE, self.env.CAPTCHA_SECRET_KEY_GOOGLE) + except Exception as exc: + raise Exception(MessageCaptchaFailed) from exc + + try: + response = cog_provider_client.admin_get_user( + UserPoolId=USERPOOLID, + Username=self.username + ) + except Exception as exc: + print(exc) + raise Exception(MessageForgotPasswordUsernotExist) from exc + + is_email_verified = True + for it in response['UserAttributes']: + if it['Name'] == 'email_verified' and it['Value'] == 'false': + is_email_verified = False + break + if not is_email_verified: + return generate_response( + message=MessageUserVerifyConfirmCode, + headers=RESPONSE_HEADER + ) + mail = getMail(self.username) + try: + AddTriggerCustomMail({ + 'lambda_name': os.environ['INVOKE_MAIL_LAMBDA'], + 'region': REGION, + 'user': self.username, + 'mail': mail, + 'subject': 'Your email confirmation code', + 'confirm_code_Table': os.environ['TABLE_CONFIRM_CODE'] + }) + except Exception as e: + print(e) + return generate_response( + message=MessageForgotPasswordSuccessfully, + headers=RESPONSE_HEADER, + error=True + ) + + return generate_response( + message=MessageForgotPasswordSuccessfully, + headers=RESPONSE_HEADER + ) + + +@error_response +def lambda_handler(event, context): + return ForgotPasswordClass().handle(event=event, context=context) diff --git a/daita-app/core-service/functions/handlers/auth_service/forgot_password/requirements.txt b/daita-app/core-service/functions/handlers/auth_service/forgot_password/requirements.txt new file mode 100644 index 0000000..663bd1f --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/forgot_password/requirements.txt @@ -0,0 +1 @@ +requests \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/auth_service/login/__init__.py b/daita-app/core-service/functions/handlers/auth_service/login/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/auth_service/login/app.py b/daita-app/core-service/functions/handlers/auth_service/login/app.py new file mode 100644 index 0000000..e8fc042 --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/login/app.py @@ -0,0 +1,255 @@ +import os +import json +import re +from datetime import datetime +import boto3 +from response import * +from config import * +from error_messages import * +from verify_captcha import * +from lambda_base_class import LambdaBaseClass +from utils import * + +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + +ACCESS_TOKEN_EXPIRATION = 24 * 60 * 60 +mailRegexString = re.compile( + '([A-Za-z0-9]+[.-_])*[A-Za-z0-9]+@[A-Za-z0-9-]+(\.[A-Z|a-z]{2,})+') +cog_provider_client = boto3.client('cognito-idp') +cog_identity_client = boto3.client('cognito-identity') + +RESPONSE_HEADER = { + "Access-Control-Allow-Creentials": "true", + "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", +} +TableUser = os.environ['TABLE_USER'] + +###################################################################################### + + +def getUsernameByEmail(email): + client = cog_provider_client.list_users( + UserPoolId=USERPOOLID) + for _, data in enumerate(client['Users']): + if data['UserStatus'] == 'EXTERNAL_PROVIDER': + continue + for info in data['Attributes']: + if info['Name'] == 'email' and info['Value'] == email: + if data['UserStatus'] == "CONFIRMED": + return data['Username'], None + elif data['UserStatus'] == "UNCONFIRMED": + return data['Username'], MessageUserVerifyConfirmCode + return None, None + +####################################################################################### + + +def getSub(access_token): + resp = cog_provider_client.get_user( + AccessToken=access_token + ) + sub = [a['Value'] for a in resp['UserAttributes'] if a['Name'] == 'sub'][0] + return sub + +####################################################################################### + + +def checkEmailVerified(access_token): + resp = cog_provider_client.get_user( + AccessToken=access_token + ) + + for it in resp['UserAttributes']: + if it['Name'] == 'email_verified' and it['Value'] == 'false': + return False + return True +#################################################################################### + + +class User(object): + def __init__(self): + self.db_client = boto3.resource('dynamodb', region_name=REGION) + self.TBL = TableUser + + def checkFirstLogin(self, ID, username): + response = self.db_client.Table(self.TBL).get_item( + Key={ + 'ID': ID, + 'username': username + } + ) + if 'Item' in response and response['Item']['status'] == "activate": + return True + + return False + + def updateActivateUser(self, info): + self.db_client.Table(self.TBL).update_item( + Key={'ID': info['ID'], 'username': info['username']}, + UpdateExpression="SET #s = :s , #i = :i , #k = :k , #u = :u", + ExpressionAttributeValues={ + ':s': 'activate', + ':i': info['indentityID'], + ':k': info['kms'], + ':u': datetime.now().isoformat(), + }, + ExpressionAttributeNames={ + '#s': 'status', + '#i': 'identity_id', + '#k': 'kms_key_id', + '#u': 'update_at' + } + ) +############################################################################################################# + + +def createKMSKey(identity): + alias_name = identity.split(":")[1] + kms = boto3.client("kms", region_name=REGION) + + key = kms.create_key() + key_id = key["KeyMetadata"]["KeyId"] + kms.create_alias( + AliasName="alias/"+alias_name, + TargetKeyId=key_id + ) + return key_id +############################################################################################################# + + +def getCredentialsForIdentity(token_id): + PROVIDER = f'cognito-idp.{REGION}.amazonaws.com/{USERPOOLID}' + responseIdentity = aws_get_identity_id(token_id, USERPOOLID, IDENTITY_POOL) + credentialsResponse = cog_identity_client.get_credentials_for_identity( + IdentityId=responseIdentity, + Logins={ + PROVIDER: token_id + }) + return { + 'secret_key': credentialsResponse['Credentials']['SecretKey'], + 'session_key': credentialsResponse['Credentials']['SessionToken'], + 'credential_token_expires_in': credentialsResponse['Credentials']['Expiration'].timestamp() * 1000, + 'access_key': credentialsResponse['Credentials']['AccessKeyId'], + 'identity_id': responseIdentity + } +############################################################################################################# + + +class LoginClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + @ LambdaBaseClass.parse_body + def parser(self, body): + self.username = body['username'] + self.password = body['password'] + self.captcha = body['captcha'] + + def handle(self, event, context): + self.parser(event) + try: + verify_captcha(self.captcha, self.env.CAPTCHA_SITE_KEY_GOOGLE, self.env.CAPTCHA_SECRET_KEY_GOOGLE) + except Exception as exc: + print(exc) + raise Exception(MessageCaptchaFailed) from exc + + if re.fullmatch(mailRegexString, self.username): + self.username, err = getUsernameByEmail(email=self.username) + print("[DEBUG] username {}".format(self.username)) + if not err is None: + raise Exception(MessageUserVerifyConfirmCode) + if self.username is None: + raise Exception(MessageLoginMailNotExist) + try: + model = User() + except Exception as e: + print(e) + return generate_response( + message=MessageUnmarshalInputJson, + data={}, + headers=RESPONSE_HEADER, + error=True + ) + + try: + authResponse = cog_provider_client.initiate_auth( + ClientId=CLIENTPOOLID, + AuthFlow='USER_PASSWORD_AUTH', + AuthParameters={ + 'USERNAME': self.username, + 'PASSWORD': self.password + } + ) + except cog_provider_client.exceptions.UserNotConfirmedException: + return generate_response( + message=MessageUserVerifyConfirmCode, + data={}, + headers=RESPONSE_HEADER, + error=True + ) + except Exception as e: + print(e) + return generate_response( + message=MessageLoginFailed, + data={}, + headers=RESPONSE_HEADER, + error=True + ) + + response = { + 'token': authResponse['AuthenticationResult']['AccessToken'], + 'resfresh_token': authResponse['AuthenticationResult']['RefreshToken'], + 'id_token': authResponse['AuthenticationResult']['IdToken'], + 'token_expires_in': float(int((datetime.now().timestamp() + ACCESS_TOKEN_EXPIRATION)*1000)) + } + + if not checkEmailVerified(response['token']): + raise Exception(MessageUserVerifyConfirmCode) + + try: + credentialsForIdentity = getCredentialsForIdentity( + authResponse['AuthenticationResult']['IdToken']) + except Exception as e: + print(e) + return generate_response( + message=MessageAuthenFailed, + data={}, + headers=RESPONSE_HEADER, + error=True) + + sub = getSub(response['token']) + + # # check the user is login another device + # if CheckEventUserLogin(sub): + # raise Exception(MessageAnotherUserIsLoginBefore) + # else: + # CreateEventUserLogin(sub) + + if not model.checkFirstLogin(ID=sub, username=self.username): + if 'IS_ENABLE_KMS' in os.environ and eval(os.environ['IS_ENABLE_KMS']) == True: + kms = createKMSKey(credentialsForIdentity['identity_id']) + else: + kms = '' + model.updateActivateUser(info={ + 'indentityID': credentialsForIdentity['identity_id'], + 'ID': sub, + 'username': self.username, + 'kms': kms, + }) + + for k, v in credentialsForIdentity.items(): + response[k] = v + response['username'] = self.username + response['name'] = self.username + return generate_response( + message=MessageSignInSuccessfully, + data=response, + headers=RESPONSE_HEADER + ) + + +@error_response +def lambda_handler(event, context): + return LoginClass().handle(event=event, context=context) diff --git a/daita-app/core-service/functions/handlers/auth_service/login/requirements.txt b/daita-app/core-service/functions/handlers/auth_service/login/requirements.txt new file mode 100644 index 0000000..663bd1f --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/login/requirements.txt @@ -0,0 +1 @@ +requests \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/auth_service/login_refresh_token/__init__.py b/daita-app/core-service/functions/handlers/auth_service/login_refresh_token/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/auth_service/login_refresh_token/app.py b/daita-app/core-service/functions/handlers/auth_service/login_refresh_token/app.py new file mode 100644 index 0000000..e450f68 --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/login_refresh_token/app.py @@ -0,0 +1,118 @@ +import os +import json +import logging +from datetime import datetime +from http import HTTPStatus + +import boto3 + +from config import * +from error_messages import * +from response import generate_response, error_response +from lambda_base_class import LambdaBaseClass +from utils import * + + +logger = logging.getLogger(__name__) +cog_provider_client = boto3.client('cognito-idp') +cog_identity_client = boto3.client('cognito-identity') +ACCESS_TOKEN_EXPIRATION = 24 * 60 * 60 +RESPONSE_HEADER = { + "Access-Control-Allow-Credentials": "true", + "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", +} +################################################################################################################## +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + + +def getDisplayName(username): + response = cog_provider_client.admin_get_user( + UserPoolId=USERPOOLID, + Username=username + ) + user_attributes = {} + for att in response["UserAttributes"]: + user_attributes[att["Name"]] = att["Value"] + + name = "" + if "name" in user_attributes: + name = user_attributes["name"] + elif "given_name" in user_attributes or \ + "family_name" in user_attributes: + name = f"{user_attributes.pop('given_name', '')} {user_attributes.pop('family_name', '')}" + else: + name = user_attributes["email"] + + return name + + +@error_response +def lambda_handler(event, context): + return RefreshTokenClass().handle(event, context) + + +class RefreshTokenClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + @LambdaBaseClass.parse_body + def parser(self, body): + self.username = body["username"] + self.refresh_token = body["refresh_token"] + + def handle(self, event, context): + self.parser(event) + try: + authenticated = cog_provider_client.initiate_auth( + AuthFlow="REFRESH_TOKEN_AUTH", + AuthParameters={ + "REFRESH_TOKEN": self.refresh_token, + "USERNAME": self.username, + }, + ClientId=CLIENTPOOLID, + ) + + id_token = authenticated["AuthenticationResult"]["IdToken"] + IdentityId = aws_get_identity_id( + id_token, USERPOOLID, IDENTITY_POOL) + + # call Cognito getCredentialsForIdentity + identity = cog_identity_client.get_credentials_for_identity( + IdentityId=IdentityId, + Logins={ + f'cognito-idp.{cog_identity_client.meta.region_name}.amazonaws.com/{USERPOOLID}': id_token + } + ) + except Exception as exc: + print(exc) + raise Exception(MessageRefreshTokenError) from exc + tempusername = self.username + tempusername = tempusername.lower() + if 'github' in tempusername or 'google' in tempusername: + name = getDisplayName(self.username) + else: + name = self.username + # reformat + user_data = { + "access_key": identity["Credentials"]["AccessKeyId"], + "secret_key": identity["Credentials"]["SecretKey"], + "session_key": identity["Credentials"]["SessionToken"], + "token": authenticated["AuthenticationResult"]["AccessToken"], + "identity_id": identity["IdentityId"], + # expire time in seconds + "credential_token_expires_in": (identity["Credentials"]["Expiration"].timestamp())*1000, + "id_token": id_token, + "token_expires_in": float(int((datetime.now().timestamp() + ACCESS_TOKEN_EXPIRATION)*1000)), + "username": self.username, + "name": name + } + + # return response + return generate_response( + message=MessageRefreshTokenSuccessfully, + status_code=HTTPStatus.OK, + headers=RESPONSE_HEADER, + data=user_data, + ) diff --git a/daita-app/core-service/functions/handlers/auth_service/login_refresh_token/requirements.txt b/daita-app/core-service/functions/handlers/auth_service/login_refresh_token/requirements.txt new file mode 100644 index 0000000..663bd1f --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/login_refresh_token/requirements.txt @@ -0,0 +1 @@ +requests \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/auth_service/login_social/__init__.py b/daita-app/core-service/functions/handlers/auth_service/login_social/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/auth_service/login_social/app.py b/daita-app/core-service/functions/handlers/auth_service/login_social/app.py new file mode 100644 index 0000000..ff2c649 --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/login_social/app.py @@ -0,0 +1,127 @@ +from email import header +import os +import json +import logging +import time +from datetime import datetime +from http import HTTPStatus +import os +import boto3 + +from error_messages import * +from response import * +from config import * +from lambda_base_class import LambdaBaseClass + +import base64 +from urllib.parse import urlencode +ACCESS_TOKEN_EXPIRATION = 24 * 60 * 60 +USERPOOLID = os.environ['COGNITO_USER_POOL'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + +cog_provider_client = boto3.client('cognito-idp') +cog_identity_client = boto3.client('cognito-identity') + + +@error_response +def lambda_handler(event, context): + param = event['queryStringParameters'] + try: + code = param['code'] + except Exception as e: + print(e) + if 'error_description' in param: + location = LOCATION + headers = {"Location": location, + "Access-Control-Allow-Methods": "GET,HEAD,OPTIONS,POST,PUT"} + return { + "statusCode": 302, + "headers": headers, + "body": '', + "isBase64Encoded": False + } + raise Exception(e) + + if 'state' in param: + path = base64.b64decode(param['state']).decode('utf-8') + else: + path = 'http://localhost:3000/login' + mapping = { + 'token': '', + 'resfresh_token': '', + 'access_key': '', + 'session_key': '', + 'id_token': '', + 'credential_token_expires_in': '', + 'token_expires_in': '', + 'secret_key': '', + 'identity_id': '', + 'username': '', + 'code': code + } + location = path + '?' + urlencode(mapping, doseq=True) + headers = {"Location": location, + "Access-Control-Allow-Methods": "GET,HEAD,OPTIONS,POST,PUT"} + return { + "statusCode": 302, + "headers": headers, + "body": '', + "isBase64Encoded": False + } + + +class LoginSocialClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def parser(self, body): + self.code = body['code'] + + def handle(self, event, context): + param = event['queryStringParameters'] + try: + self.parser(param) + except Exception as e: + if 'error_description' in param: + location = LOCATION + headers = {"Location": location, + "Access-Control-Allow-Methods": "GET,HEAD,OPTIONS,POST,PUT"} + return { + "statusCode": 302, + "headers": headers, + "body": '', + "isBase64Encoded": False + } + raise Exception(e) + + if 'state' in param: + path = base64.b64decode(param['state']).decode('utf-8') + else: + path = 'http://localhost:3000/login' + mapping = { + 'token': '', + 'resfresh_token': '', + 'access_key': '', + 'session_key': '', + 'id_token': '', + 'credential_token_expires_in': '', + 'token_expires_in': '', + 'secret_key': '', + 'identity_id': '', + 'username': '', + 'code': self.code + } + location = path + '?' + urlencode(mapping, doseq=True) + headers = {"Location": location, + "Access-Control-Allow-Methods": "GET,HEAD,OPTIONS,POST,PUT"} + return { + "statusCode": 302, + "headers": headers, + "body": '', + "isBase64Encoded": False + } + + +@error_response +def lambda_handler(event, context): + return LoginSocialClass.handle(event=event, context=context) diff --git a/daita-app/core-service/functions/handlers/auth_service/logout/__init__.py b/daita-app/core-service/functions/handlers/auth_service/logout/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/auth_service/logout/app.py b/daita-app/core-service/functions/handlers/auth_service/logout/app.py new file mode 100644 index 0000000..d44f9b3 --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/logout/app.py @@ -0,0 +1,53 @@ +import os +import boto3 +import cognitojwt +from error_messages import * +from response import * +from config import * +from models.event_model import EventUser + +cog_provider_client = boto3.client('cognito-idp') +cog_identity_client = boto3.client('cognito-identity') +RESPONSE_HEADER = { + "Access-Control-Allow-Creentials": "true", + "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", +} +eventModel = EventUser() +def claimsToken(jwt_token,field): + """ + Validate JWT claims & retrieve user identifier + """ + token = jwt_token.replace("Bearer ", "") + print(token) + try: + verified_claims = cognitojwt.decode( + token, os.environ['REGION'], os.environ['COGNITO_USER_POOL'] + ) + except Exception as e: + print(e) + raise e + + return verified_claims.get(field) + +@error_response +def lambda_handler(event, context): + headers = event['headers']['authorization'] + authorization_header = headers + if not len(authorization_header): + raise Exception(MessageMissingAuthorizationHeader) + try: + sub = claimsToken(authorization_header,'sub') + username = claimsToken(authorization_header,'username') + except Exception as e: + raise e + code = None + data = {} + if 'github' in username or 'google' in username: + code = eventModel.get_code_oauth2_cognito(sub) + data['code'] = code + + return generate_response( + message= MessageLogoutSuccessfully, + headers=RESPONSE_HEADER, + data = data + ) \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/auth_service/logout/requirements.txt b/daita-app/core-service/functions/handlers/auth_service/logout/requirements.txt new file mode 100644 index 0000000..5171ad5 --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/logout/requirements.txt @@ -0,0 +1 @@ +cognitojwt \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/auth_service/mail_service/__init__.py b/daita-app/core-service/functions/handlers/auth_service/mail_service/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/auth_service/mail_service/app.py b/daita-app/core-service/functions/handlers/auth_service/mail_service/app.py new file mode 100644 index 0000000..30c364b --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/mail_service/app.py @@ -0,0 +1,65 @@ +import json +import boto3 +import hashlib +import hmac +import base64 +import os +import uuid +from botocore.exceptions import ClientError + + +def convert_response(data): + return { + "statusCode": 200, + "headers": { + "Access-Control-Allow-Headers": "Content-Type", + "Access-Control-Allow-Origin": "*", + "Access-Control-Allow-Methods": "OPTIONS,POST,GET", + }, + "body": json.dumps(data), + } + + +def lambda_handler(event, context): + try: + destination_email = event["destination_email"] + message_email = event["message_email"] + message_email_text = event["message_email_text"] + subject = event["subject"] + except Exception as e: + print(e) + return convert_response( + {"error": True, "success": False, "message": repr(e), "data": None} + ) + + client = boto3.client("ses") + try: + response = client.send_email( + Destination={ + "ToAddresses": [destination_email], + }, + Message={ + "Body": { + "Html": { + "Charset": "UTF-8", + "Data": message_email, + }, + "Text": {"Charset": "UTF-8", "Data": message_email_text}, + }, + "Subject": {"Charset": "UTF-8", "Data": subject}, + }, + Source="DAITA Team ", + ) + except ClientError as e: + print(e) + return convert_response( + {"error": True, "success": False, "message": e, "data": None} + ) + return convert_response( + { + "error": False, + "success": True, + "message": response["MessageId"], + "data": None, + } + ) diff --git a/daita-app/core-service/functions/handlers/auth_service/resend_confirmcode/__init__.py b/daita-app/core-service/functions/handlers/auth_service/resend_confirmcode/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/auth_service/resend_confirmcode/app.py b/daita-app/core-service/functions/handlers/auth_service/resend_confirmcode/app.py new file mode 100644 index 0000000..3fa2986 --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/resend_confirmcode/app.py @@ -0,0 +1,79 @@ +from cmath import inf +import os +import json +import logging +from http import HTTPStatus +import os +import boto3 +from dataclasses import dataclass +import re + +from error_messages import * +from response import * +from config import * +from custom_mail import * +from datetime import datetime +from lambda_base_class import LambdaBaseClass + + +def convert_current_date_to_iso8601(): + my_date = datetime.now() + return my_date.isoformat() + + +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + +cog_provider_client = boto3.client('cognito-idp') +cog_identity_client = boto3.client('cognito-identity') +RESPONSE_HEADER = { + "Access-Control-Allow-Creentials": "true", + "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", +} + + +def getMail(user): + response = cog_provider_client.list_users( + UserPoolId=USERPOOLID + ) + # info_user = list(filter(lambda x : x['Username'] == user,response['Users'])) + + for _, data in enumerate(response['Users']): + if data['Username'] == user: + for info in data['Attributes']: + if info['Name'] == 'email': + return info['Value'] + return None + + +class ResendConfirmCodeClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + @LambdaBaseClass.parse_body + def parser(self, body): + self.username = body['username'] + self.mail = getMail(self.username) + + def handle(self, event, context): + self.parser(event) + ResendCodeConfirm({ + 'lambda_name': os.environ['INVOKE_MAIL_LAMBDA'], + 'region': REGION, + 'user': self.username, + 'mail': self.mail, + 'subject': 'Your email confirmation code', + 'confirm_code_Table': os.environ['TABLE_CONFIRM_CODE'] + + }) + return generate_response( + message=MessageResendEmailConfirmCodeSuccessfully, + data={}, + headers=RESPONSE_HEADER + ) + + +@error_response +def lambda_handler(event, context): + return ResendConfirmCodeClass.handle(event=event, context=context) diff --git a/daita-app/core-service/functions/handlers/auth_service/sign_up/__init__.py b/daita-app/core-service/functions/handlers/auth_service/sign_up/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/auth_service/sign_up/app.py b/daita-app/core-service/functions/handlers/auth_service/sign_up/app.py new file mode 100644 index 0000000..17a4aac --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/sign_up/app.py @@ -0,0 +1,148 @@ +import os +import json +import logging +from http import HTTPStatus +import os +import boto3 +from dataclasses import dataclass +import re + +from error_messages import * +from response import * +from config import * +from custom_mail import * +from verify_captcha import * +from lambda_base_class import LambdaBaseClass + +from datetime import datetime + + +def convert_current_date_to_iso8601(): + my_date = datetime.now() + return my_date.isoformat() + + +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + +cog_provider_client = boto3.client("cognito-idp") +cog_identity_client = boto3.client("cognito-identity") +RESPONSE_HEADER = { + "Access-Control-Allow-Creentials": "true", + "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", +} + + +PASSWORD_REGEX = r"^(?=.*[a-z])(?=.*[A-Z])(?=.*[0-9])(?=.*[\^$*.\[\]{}\(\)?\-\"!@#%&\/,><\':;|_~`])\S{8,99}$" + +TableUser = os.environ['TABLE_USER'] + + +class User(object): + def __init__(self): + self.db_client = boto3.resource("dynamodb", region_name=REGION) + self.TBL = TableUser + + def create_item(self, info): + self.db_client.Table(self.TBL).put_item( + Item={ + "ID": info["ID"], + "username": info["username"], + "role": "normal", + "status": "deactivate", + "create_at": convert_current_date_to_iso8601(), + } + ) + + +def checkInvalidUserRegister(user, mail): + response = cog_provider_client.list_users(UserPoolId=USERPOOLID) + info_user = list( + filter(lambda x: x["Username"] == user, response["Users"])) + if len(info_user): + return False, True + for _, data in enumerate(response["Users"]): + for info in data["Attributes"]: + if info["Name"] == "email" and info["Value"] == mail: + return True, False + return True, True + + +class SignUpClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + pass + + @LambdaBaseClass.parse_body + def parser(self, body): + self.username = body["username"] + self.mail = body["email"] + self.password = body["password"] + self.captcha = body["captcha"] + + def handle(self, event, context): + self.parser(event) + + try: + verify_captcha(self.captcha, self.env.CAPTCHA_SITE_KEY_GOOGLE, self.env.CAPTCHA_SECRET_KEY_GOOGLE) + except Exception as exc: + raise Exception(MessageCaptchaFailed) from exc + + if not re.match(PASSWORD_REGEX, self.password): + raise Exception(MessageInvalidPassword) + checkUsername, checkemail = checkInvalidUserRegister( + self.username, self.mail) + + if not checkUsername: + raise Exception(MessageSignUpUsernameInvalid) + elif not checkemail: + raise Exception(MessageSignUPEmailInvalid) + + try: + respUserSignUp = cog_provider_client.sign_up( + ClientId=CLIENTPOOLID, + Username=self.username, + Password=self.password, + UserAttributes=[{"Name": "email", "Value": self.mail}], + ) + except Exception as e: + print(e) + raise Exception(MessageSignUpFailed) + + model = User() + try: + model.create_item( + {"ID": respUserSignUp["UserSub"], "username": self.username}) + except Exception as e: + print(e) + raise Exception(MessageSignUpFailed) + try: + AddTriggerCustomMail( + { + 'lambda_name': os.environ['INVOKE_MAIL_LAMBDA'], + 'region': REGION, + 'user': self.username, + 'mail': self.mail, + 'subject': "Your email confirmation code", + 'confirm_code_Table': os.environ['TABLE_CONFIRM_CODE'] + } + ) + except Exception as e: + return generate_response( + message=str(e), + data={}, + headers=RESPONSE_HEADER, + error=True + ) + return generate_response( + message="Sign up for user {} was successful.".format( + self.username), + data={}, + headers=RESPONSE_HEADER, + ) + + +def lambda_handler(event, context): + return SignUpClass().handle(event=event, context=context) diff --git a/daita-app/core-service/functions/handlers/auth_service/sign_up/requirements.txt b/daita-app/core-service/functions/handlers/auth_service/sign_up/requirements.txt new file mode 100644 index 0000000..663bd1f --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/sign_up/requirements.txt @@ -0,0 +1 @@ +requests \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/auth_service/template_mail/__init__.py b/daita-app/core-service/functions/handlers/auth_service/template_mail/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/auth_service/template_mail/app.py b/daita-app/core-service/functions/handlers/auth_service/template_mail/app.py new file mode 100644 index 0000000..cbebea2 --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/template_mail/app.py @@ -0,0 +1,73 @@ +import os +import json +import logging +from datetime import datetime +from http import HTTPStatus +import os +import boto3 +import cognitojwt +from error_messages import * +from response import * +from config import * +cog_provider_client = boto3.client('cognito-idp') +cog_identity_client = boto3.client('cognito-identity') +RESPONSE_HEADER = { + "Access-Control-Allow-Creentials": "true", + "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", +} +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + + +def getMail(user): + response = cog_provider_client.list_users( + UserPoolId=USERPOOLID + ) + for _, data in enumerate(response['Users']): + if data['Username'] == user: + for info in data['Attributes']: + if info['Name'] == 'email': + return info['Value'] + return None + + +def claimsToken(jwt_token, field): + """ + Validate JWT claims & retrieve user identifier + """ + token = jwt_token.replace("Bearer ", "") + print(token) + try: + verified_claims = cognitojwt.decode( + token, os.environ['REGION'], USERPOOLID + ) + except Exception as e: + print(e) + verified_claims = {} + + return verified_claims.get(field) + + +@error_response +def lambda_handler(event, context): + headers = event['headers']['authorization'] + authorization_header = headers + if not len(authorization_header): + raise Exception(MessageMissingAuthorizationHeader) + username = claimsToken(authorization_header, 'username') + mail = getMail(username) + template = "

Hi,

{} has invited you to explore DAITA's recently launched " \ + "data augmentation platform.

" \ + "

Building a platform that machine learning engineers and data scientists " \ + "really love is truly hard. But that's our ultimate ambition!

Thus, your feedback" \ + " is greatly appreciated, as this first version will still be buggy and missing many features. Please send " \ + "all your thoughts, concerns, feature requests, etc. to contact@daita.tech or simply reply to this e-mail. " \ + "Please be assured that all your feedback will find its way into our product backlog.

All our services" \ + " are currently free of charge - so you can go wild! Try it now here.

Cheers,

The DAITA Team

".format( + mail) + return generate_response( + message=MessageGetTemapleMailSuccessFully, + data={"content": template}, + headers=RESPONSE_HEADER + ) diff --git a/daita-app/core-service/functions/handlers/auth_service/template_mail/requirements.txt b/daita-app/core-service/functions/handlers/auth_service/template_mail/requirements.txt new file mode 100644 index 0000000..5171ad5 --- /dev/null +++ b/daita-app/core-service/functions/handlers/auth_service/template_mail/requirements.txt @@ -0,0 +1 @@ +cognitojwt \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/cli/check_daita_token/__init__.py b/daita-app/core-service/functions/handlers/cli/check_daita_token/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/cli/check_daita_token/app.py b/daita-app/core-service/functions/handlers/cli/check_daita_token/app.py new file mode 100644 index 0000000..936e5dd --- /dev/null +++ b/daita-app/core-service/functions/handlers/cli/check_daita_token/app.py @@ -0,0 +1,34 @@ +import os +import json +from response import * +from config import * + +from models.generate_daita_upload_token import GenerateDaitaUploadTokenModel + +generate_daita_upload_token_model = GenerateDaitaUploadTokenModel( + os.environ['T_GEN_DAITA_UPLOAD_TOKEN']) + + +def lambda_handler(event, context): + param = event['queryStringParameters'] + try: + daita_token = param['daita_token'] + except Exception as e: + print(e) + return generate_response( + message=str(e), + data={}, + headers=RESPONSE_HEADER, + error=True) + info = generate_daita_upload_token_model.query_by_token(token=daita_token) + if info is None: + return generate_response( + message="Token is expired, please get another token!", + data={}, + headers=RESPONSE_HEADER, + error=True) + return generate_response( + message="OK", + data={}, + headers=RESPONSE_HEADER, + error=False) diff --git a/daita-app/core-service/functions/handlers/cli/check_existence_file/__init__.py b/daita-app/core-service/functions/handlers/cli/check_existence_file/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/cli/check_existence_file/app.py b/daita-app/core-service/functions/handlers/cli/check_existence_file/app.py new file mode 100644 index 0000000..97eb45c --- /dev/null +++ b/daita-app/core-service/functions/handlers/cli/check_existence_file/app.py @@ -0,0 +1,53 @@ +import os +import json +import boto3 +from response import * +from config import * + +from models.generate_daita_upload_token import GenerateDaitaUploadTokenModel + +generate_daita_upload_token_model = GenerateDaitaUploadTokenModel( + os.environ['T_GEN_DAITA_UPLOAD_TOKEN']) + + +def invokeUploadUpdateFunc(info): + lambdaInvokeClient = boto3.client('lambda') + lambdaInvokeReq = lambdaInvokeClient.invoke( + FunctionName=os.environ['LAMBDA_UPDATE_CHECK'], + Payload=json.dumps({'body': info}), + InvocationType="RequestResponse", + ) + payload = json.loads(lambdaInvokeReq['Payload'].read()) + body = json.loads(payload['body']) + return body + + +def lambda_handler(event, context): + try: + body = json.loads(event['body']) + daita_token = body['daita_token'] + ls_filename = body['ls_filename'] + except Exception as e: + print(e) + return generate_response( + message=str(e), + data={}, + headers=RESPONSE_HEADER, + error=True) + + info = generate_daita_upload_token_model.query_by_token(token=daita_token) + if info is None: + return generate_response( + message="Token is expired, please get another token!", + data={}, + headers=RESPONSE_HEADER, + error=True) + + project_id = info['project_id'] + id_token = info['id_token'] + payloadInfo = { + 'project_id': project_id, + 'id_token': id_token, + 'ls_filename': ls_filename + } + return invokeUploadUpdateFunc(json.dumps(payloadInfo)) diff --git a/daita-app/core-service/functions/handlers/cli/cli_upload_project/__init__.py b/daita-app/core-service/functions/handlers/cli/cli_upload_project/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/cli/cli_upload_project/app.py b/daita-app/core-service/functions/handlers/cli/cli_upload_project/app.py new file mode 100644 index 0000000..0071452 --- /dev/null +++ b/daita-app/core-service/functions/handlers/cli/cli_upload_project/app.py @@ -0,0 +1,133 @@ +import os +import json +import boto3 +from response import * +from config import * + +from models.generate_daita_upload_token import GenerateDaitaUploadTokenModel +from models.project_sum_model import ProjectSumModel + +generate_daita_upload_token_model = GenerateDaitaUploadTokenModel( + os.environ['T_GEN_DAITA_UPLOAD_TOKEN']) +project_sum_model = ProjectSumModel(os.environ["TABLE_PROJECT_SUMMARY"]) +s3 = boto3.client('s3') +bucket = os.environ['BUCKET_NAME'] +RESPONSE_HEADER = { + "Access-Control-Allow-Creentials": "true", + "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", +} + + +def invokeUploadUpdateFunc(info): + lambdaInvokeClient = boto3.client('lambda') + lambdaInvokeReq = lambdaInvokeClient.invoke( + FunctionName=os.environ['LAMBDA_UPLOAD_UPDATE'], + Payload=json.dumps({'body': info}), + InvocationType="RequestResponse", + ) + + +def invokeUploadCheck(info): + lambdaInvokeClient = boto3.client('lambda') + lambdaInvokeReq = lambdaInvokeClient.invoke( + FunctionName=os.environ['LAMBDA_UPLOAD_CHECK'], + Payload=json.dumps({'body': info}), + InvocationType="RequestResponse", + ) + + +def generate_presigned_url(object_keyname, expired=3600): + reponse = s3.generate_presigned_post( + Bucket=bucket, + Key=object_keyname, + ExpiresIn=expired + ) + return reponse + + +def lambda_handler(event, context): + + try: + body = json.loads(event['body']) + filenames = body['filenames'] + daita_token = body['daita_token'] + ls_object_info = body['ls_object_info'] + except Exception as e: + print(e) + return generate_response( + message=str(e), + data={}, + headers=RESPONSE_HEADER, + error=True) + data = {} + info = generate_daita_upload_token_model.query_by_token(token=daita_token) + if info is None: + return generate_response( + message="Token is expired, please get another token!", + data={}, + headers=RESPONSE_HEADER, + error=True) + identity_id = info['identity_id'] + project_id = info['project_id'] + id_token = info['id_token'] + project_name = info['project_name'] + # check number of images in project + prjSumAllResp = project_sum_model.get_item( + project_id=project_id, type_data='ORIGINAL') + + if prjSumAllResp is None: + return generate_response( + message="Something wrong with your project, Please check again!", + data={}, + headers=RESPONSE_HEADER, + error=True) + + countSumAllPrj = int(prjSumAllResp['count']) + if countSumAllPrj + len(filenames) >= 5000: + return generate_response( + message="Limited", + data={}, + headers=RESPONSE_HEADER, + error=True) + + folder = os.path.join(identity_id, project_id) + ls_filename = [] + for objectS3 in ls_object_info: + objectS3['s3_key'] = os.path.join( + bucket, os.path.join(folder, objectS3['filename'])) + ls_filename.append(objectS3['filename']) + try : + invokeUploadUpdateFunc(json.dumps({ + "id_token": id_token, + "project_id": project_id, + "project_name": project_name, + "ls_object_info": ls_object_info + })) + except Exception as e : + return generate_response( + message="Failed Invoke Upload Update: {}".format(e), + data=data, + headers=RESPONSE_HEADER, + error=True + ) + try: + invokeUploadCheck(json.dumps( + { + "id_token": id_token, + "ls_filename": ls_filename, + "project_id": project_id + } + )) + except Exception as e : + return generate_response( + message="Failed Invoke Upload Check: {}".format(e), + data=data, + headers=RESPONSE_HEADER, + error=True + ) + return generate_response( + message="Generate presign Url S3 successfully", + data=data, + headers=RESPONSE_HEADER, + error=False + ) diff --git a/daita-app/core-service/functions/handlers/cli/create_decompress_task/__init__.py b/daita-app/core-service/functions/handlers/cli/create_decompress_task/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/cli/create_decompress_task/app.py b/daita-app/core-service/functions/handlers/cli/create_decompress_task/app.py new file mode 100644 index 0000000..fe081cb --- /dev/null +++ b/daita-app/core-service/functions/handlers/cli/create_decompress_task/app.py @@ -0,0 +1,56 @@ +import os +import json +import boto3 +from response import * +from config import * +from models.generate_daita_upload_token import GenerateDaitaUploadTokenModel + +generate_daita_upload_token_model = GenerateDaitaUploadTokenModel( + os.environ['T_GEN_DAITA_UPLOAD_TOKEN']) + + +def invokeLambda(info): + lambdaInvokeClient = boto3.client('lambda') + lambdaInvokeReq = lambdaInvokeClient.invoke( + FunctionName=str(os.environ['DECOMPRESS_LAMBDA_INVOKE']), + Payload=json.dumps({'body': info}), + InvocationType="RequestResponse", + ) + payload = json.loads(lambdaInvokeReq['Payload'].read()) + body = json.loads(payload['body']) + return body + + +def lambda_handler(event, context): + try: + body = json.loads(event['body']) + s3 = body['s3'] + daita_token = body['daita_token'] + except Exception as e: + print(e) + return generate_response( + message=str(e), + data={}, + headers=RESPONSE_HEADER, + error=True) + info = generate_daita_upload_token_model.query_by_token(token=daita_token) + if info is None: + return generate_response( + message="Token is expired, please get another token!", + data={}, + headers=RESPONSE_HEADER, + error=True) + innvokeBodyLambda = json.dumps({ + 'id_token': info['id_token'], + 'project_id': info['project_id'], + 'project_name': info['project_name'], + 'type_method': "ORIGINAL", + 'file_url': s3 + }) + responseInvokeLambda = invokeLambda(innvokeBodyLambda) + responseInvokeLambda['data']['id_token'] = info['id_token'] + return generate_response( + message="Create decompress task successfully!", + data=responseInvokeLambda, + headers=RESPONSE_HEADER, + error=False) diff --git a/daita-app/core-service/functions/handlers/cli/create_presignurl_zip/__init__.py b/daita-app/core-service/functions/handlers/cli/create_presignurl_zip/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/cli/create_presignurl_zip/app.py b/daita-app/core-service/functions/handlers/cli/create_presignurl_zip/app.py new file mode 100644 index 0000000..36ad6d4 --- /dev/null +++ b/daita-app/core-service/functions/handlers/cli/create_presignurl_zip/app.py @@ -0,0 +1,74 @@ +import os +import json +import boto3 +from response import * +from config import * +from models.generate_daita_upload_token import GenerateDaitaUploadTokenModel + +s3 = boto3.client('s3') +bucket = os.environ['BUCKET_NAME'] +generate_daita_upload_token_model = GenerateDaitaUploadTokenModel( + os.environ['T_GEN_DAITA_UPLOAD_TOKEN']) + + +def generate_presigned_url(object_keyname, expired=3600): + response = s3.generate_presigned_post( + Bucket=bucket, + Key=object_keyname, + ExpiresIn=expired + ) + + return response + + +def lambda_handler(event, context): + try: + body = json.loads(event['body']) + filename = body['filename'] + daita_token = body['daita_token'] + is_zip = body['is_zip'] + except Exception as e: + print(e) + return generate_response( + message=str(e), + data={}, + headers=RESPONSE_HEADER, + error=True) + + info = generate_daita_upload_token_model.query_by_token(token=daita_token) + + if info is None: + return generate_response( + message="Token is expired, please get another token!", + data={}, + headers=RESPONSE_HEADER, + error=True) + + identity_id = info['identity_id'] + project_id = info['project_id'] + folder = os.path.join(identity_id, project_id) + data = {} + basename = os.path.basename(filename) + + if is_zip == False: + data['presign_url'] = generate_presigned_url( + object_keyname=os.path.join(folder, basename)) + data['s3_uri'] = f's3://{bucket}/{identity_id}/{project_id}/{basename}' + return generate_response( + message="Generate presign Url S3 successfully", + data=data, + headers=RESPONSE_HEADER, + error=False + ) + + folder = os.path.join('tmp', folder) + data['presign_url'] = generate_presigned_url( + object_keyname=os.path.join(folder, basename)) + data['s3_uri'] = f's3://{bucket}/tmp/{identity_id}/{project_id}/{basename}' + + return generate_response( + message="Generate presign Url S3 successfully", + data=data, + headers=RESPONSE_HEADER, + error=False + ) diff --git a/daita-app/core-service/functions/handlers/feedback/presignUrl/__init__.py b/daita-app/core-service/functions/handlers/feedback/presignUrl/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/feedback/presignUrl/app.py b/daita-app/core-service/functions/handlers/feedback/presignUrl/app.py new file mode 100644 index 0000000..82be607 --- /dev/null +++ b/daita-app/core-service/functions/handlers/feedback/presignUrl/app.py @@ -0,0 +1,53 @@ +import os +import json +import boto3 +from response import * +from config import * +from identity_check import * +s3 = boto3.client('s3') +bucket = os.environ['BUCKET_NAME'] + + +def generate_presigned_url(object_keyname, expired=3600): + response = s3.generate_presigned_post( + Bucket=bucket, + Key=object_keyname, + ExpiresIn=expired + ) + return response + + +def lambda_handler(event, context): + try: + body = json.loads(event['body']) + filename = body['filename'] + id_token = body['id_token'] + except Exception as e: + print(e) + return generate_response( + message=str(e), + data={}, + headers=RESPONSE_HEADER, + error=True) + try: + identity_id = aws_get_identity_id(id_token) + except Exception as e: + print(e) + return generate_response( + message=str(e), + data={}, + headers=RESPONSE_HEADER, + error=True) + data = {} + folder = os.path.join('feedback', identity_id) + basename = os.path.basename(filename) + folder = os.path.join(folder, basename) + data['presign_url'] = generate_presigned_url( + object_keyname=folder) + data['s3_uri'] = f's3://{bucket}/feedback/{identity_id}/{basename}' + return generate_response( + message='create presign url successfully!', + data=data, + headers=RESPONSE_HEADER, + error=False + ) diff --git a/daita-app/core-service/functions/handlers/feedback/slack_webhook/__init__.py b/daita-app/core-service/functions/handlers/feedback/slack_webhook/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/feedback/slack_webhook/app.py b/daita-app/core-service/functions/handlers/feedback/slack_webhook/app.py new file mode 100644 index 0000000..13d3872 --- /dev/null +++ b/daita-app/core-service/functions/handlers/feedback/slack_webhook/app.py @@ -0,0 +1,175 @@ +import requests +import re +import json +import pytz +import os +import json +from datetime import datetime +from http import HTTPStatus +import os +import uuid +import slack_sdk +import boto3 +import cognitojwt +from error_messages import * +from response import * +from config import * +import tempfile +from models.feedback_model import * +cog_provider_client = boto3.client('cognito-idp') +cog_identity_client = boto3.client('cognito-identity') +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] +REGION = os.environ['REGION'] +OAUTH2BOT = os.environ["TOKEN_OAUTH2BOT_SLACK_FEEDBACK"] +RESPONSE_HEADER = { + "Access-Control-Allow-Creentials": "true", + "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS, POST, PUT", +} +CHANNELWEBHOOK = '#'+ str(os.environ['CHANNELWEBHOOK']) +############################################################################################################ +s3 = boto3.client('s3') + + +def validFileImage(filename): + return (os.path.splitext(filename)[1]).lower() in ['.jpeg', '.png', '.jpg'] + + +def split(uri): + if not 's3' in uri[:2]: + temp = uri.split('/') + bucket = temp[0] + filename = '/'.join([temp[i] for i in range(1, len(temp))]) + else: + match = re.match(r's3:\/\/(.+?)\/(.+)', uri) + bucket = match.group(1) + filename = match.group(2) + return bucket, filename + + +def postMessageWithFiles(message, fileList, channel): + SLACK_TOKEN = OAUTH2BOT + client = slack_sdk.WebClient(token=SLACK_TOKEN) + for file in fileList: + upload = client.files_upload(file=file, filename=file) + message = message+"<"+upload['file']['permalink']+"| >" + outP = client.chat_postMessage( + channel=channel, + text=message + ) + + +def getDisplayName(username): + response = cog_provider_client.admin_get_user( + UserPoolId=USERPOOLID, + Username=username + ) + user_attributes = {} + for att in response["UserAttributes"]: + user_attributes[att["Name"]] = att["Value"] + + name = "" + if "name" in user_attributes: + name = user_attributes["name"] + elif "given_name" in user_attributes or "family_name" in user_attributes: + name = f"{user_attributes.pop('given_name', '')} {user_attributes.pop('family_name', '')}" + else: + name = user_attributes["email"] + + return name +#################################################################################################### + + +def claimsToken(jwt_token, field): + """ + Validate JWT claims & retrieve user identifier + """ + token = jwt_token.replace("Bearer ", "") + try: + verified_claims = cognitojwt.decode( + token, REGION, USERPOOLID + ) + except Exception as e: + print(e) + raise e + + return verified_claims.get(field) + + + + + +@error_response +def lambda_handler(event, context): + print(event) + headers = event['headers']['authorization'] + authorization_header = headers + token = authorization_header.replace('Bearer ', '') + feedbackDB = Feedback() + + if not len(authorization_header): + raise Exception(MessageMissingAuthorizationHeader) + + try: + body = json.loads(event['body']) + text = body['text'] + images = body['images'] + except Exception as e: + print(e) + raise Exception(MessageUnmarshalInputJson) + + if len(text) > 750: + raise Exception(MessageErrorFeedbackLimitword) + if not isinstance(images, list): + raise Exception(MessageErrorFeedbackInvalidType) + + if len(images) > 3: + raise Exception(MessageErrorFeedbackLimitImages) + for it in images: + if not validFileImage(it): + raise Exception(MessageErrorInvalidExtension) + try: + username = claimsToken(token, 'username') + except Exception as e: + raise e + info = {} + while True: + key = str(uuid.uuid4()) + UTC = pytz.utc + datetimeUTC = datetime.now(UTC) + datetimeString = datetimeUTC.strftime('%Y:%m:%d %H:%M:%S %Z %z') + if not feedbackDB.CheckKeyIsExist(key): + info = { + "ID": key, + "name": username, + "content": text, + "images": images, + "created_time": datetimeString + } + feedbackDB.CreateItem(info) + break + message = "Username: {}\n Time: {}\n Content: {}".format( + getDisplayName(info['name']), info['created_time'], info['content']) + fileList = [] + dir = tempfile.TemporaryDirectory(dir='/tmp') + dirname = dir.name + try: + for it in images: + bucket, filename = split(it) + resultS3 = s3.get_object(Bucket=bucket, Key=filename) + tmpfile = os.path.join(dirname, os.path.basename(filename)) + fileList.append(tmpfile) + with open(tmpfile, 'wb') as file: + file.write(resultS3['Body'].read()) + except Exception as e: + print(e) + raise Exception(e) + print(f'Debug Feedback{message}') + postMessageWithFiles(message, fileList, CHANNELWEBHOOK) + dir.cleanup() + return generate_response( + message=MessageSendFeedbackSuccessfully, + data={}, + headers=RESPONSE_HEADER + ) diff --git a/daita-app/core-service/functions/handlers/feedback/slack_webhook/requirements.txt b/daita-app/core-service/functions/handlers/feedback/slack_webhook/requirements.txt new file mode 100644 index 0000000..9e2c619 --- /dev/null +++ b/daita-app/core-service/functions/handlers/feedback/slack_webhook/requirements.txt @@ -0,0 +1,4 @@ +cognitojwt +slack-sdk +requests +pytz \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/generate/daita_upload_token/__init__.py b/daita-app/core-service/functions/handlers/generate/daita_upload_token/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/generate/daita_upload_token/app.py b/daita-app/core-service/functions/handlers/generate/daita_upload_token/app.py new file mode 100644 index 0000000..899fbef --- /dev/null +++ b/daita-app/core-service/functions/handlers/generate/daita_upload_token/app.py @@ -0,0 +1,63 @@ +import json +import os +import boto3 + +from config import * +from response import * +from error_messages import * +from identity_check import * +from utils import * +from models.generate_daita_upload_token import GenerateDaitaUploadTokenModel +generate_daita_upload_token_model = GenerateDaitaUploadTokenModel( + os.environ['T_GEN_DAITA_UPLOAD_TOKEN']) +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + +def lambda_handler(event, context): + + try: + body = json.loads(event['body']) + project_id = body['project_id'] + project_name = body['project_name'] + id_token = body["id_token"] + except Exception as e: + print(e) + return generate_response( + message=str(e), + status_code=HTTPStatus.OK, + data={}, + error=True) + + try: + identity_id = aws_get_identity_id(id_token, USERPOOLID, IDENTITY_POOL) + except Exception as e: + print(e) + return generate_response( + message=str(e), + status_code=HTTPStatus.OK, + data={}, + error=True) + + token_existed = generate_daita_upload_token_model.token_exsited( + identity_id=identity_id, project_id=project_id) + + if not token_existed is None: + token = token_existed['token'] + return generate_response( + status_code=HTTPStatus.OK, + message="Generate daita upload token successful!", + data={ + "token": token + }, + error=False) + + token = generate_daita_upload_token_model.create_new_token(id_token=id_token, + identity_id=identity_id, project_id=project_id, project_name=project_name) + return generate_response( + status_code=HTTPStatus.OK, + message="Generate daita upload token successful!", + data={ + "token": token + }, + error=False) diff --git a/daita-app/core-service/functions/handlers/generate/generate_images/app.py b/daita-app/core-service/functions/handlers/generate/generate_images/app.py index 6811fbc..957d6b3 100644 --- a/daita-app/core-service/functions/handlers/generate/generate_images/app.py +++ b/daita-app/core-service/functions/handlers/generate/generate_images/app.py @@ -17,15 +17,17 @@ from botocore.exceptions import ClientError + + class GenerateImageClass(LambdaBaseClass): def __init__(self) -> None: super().__init__() self.client_events = boto3.client('events') self.const = SystemParameterStore() - self.project_model = ProjectModel(os.environ["TABLE_PROJECTS_NAME"]) + self.project_model = ProjectModel(os.environ["TABLE_PROJECT"]) self.generate_task_model = GenerateTaskModel(os.environ["TABLE_GENERATE_TASK"]) - self.project_sum_model = ProjectSumModel(os.environ["TABLE_PROJECT_SUM"]) + self.project_sum_model = ProjectSumModel(os.environ["TABLE_PROJECT_SUMMARY"]) @LambdaBaseClass.parse_body def parser(self, body): @@ -35,13 +37,13 @@ def parser(self, body): self.project_id = body[KEY_NAME_PROJECT_ID] self.project_name = body[KEY_NAME_PROJECT_NAME] self.ls_methods_id = body[KEY_NAME_LS_METHOD_ID] - self.data_type = body.get(KEY_NAME_DATA_TYPE, 'ORIGINAL') # type is one of ORIGINAL or PREPROCESS, default is original + self.data_type = body.get(KEY_NAME_DATA_TYPE, 'ORIGINAL') # type is one of ORIGINAL or PREPROCESS, default is original self.num_aug_per_imgs = min(MAX_NUMBER_GEN_PER_IMAGES, body.get(KEY_NAME_NUM_AUG_P_IMG, 1)) # default is 1 self.data_number = body[KEY_NAME_DATA_NUMBER] # array of number data in train/val/test [100, 19, 1] self.process_type = body.get(KEY_NAME_PROCESS_TYPE, VALUE_TYPE_METHOD_PREPROCESS) self.reference_images = body.get(KEY_NAME_REFERENCE_IMAGES, {}) self.aug_parameters = body.get(KEY_NAME_AUG_PARAMS, {}) - self.is_normalize_resolution = body.get(KEY_NAME_IS_RESOLUTION, False) + self.is_normalize_resolution = body.get(KEY_NAME_IS_RESOLUTION, False) ### update value for ls_reference for method, s3_link in self.reference_images.items(): @@ -51,13 +53,13 @@ def parser(self, body): def _check_input_value(self): if len(self.data_number)>0: if self.data_number[0] == 0: - raise Exception(MESS_NUMBER_TRAINING) + raise Exception(MESS_NUMBER_TRAINING) for number in self.data_number: if number<0: - raise Exception(MESS_NUMBER_DATA) + raise Exception(MESS_NUMBER_DATA) - ### if len(ls_reference)>0, it means that we are in the expert mode, - ### we will only work with id PRE-2,3,4,5,6,8 + ### if len(ls_reference)>0, it means that we are in the expert mode, + ### we will only work with id PRE-2,3,4,5,6,8 ### and the code in ls_methods_id much match with code in ls_re # TODO @@ -66,8 +68,8 @@ def _check_input_value(self): def _check_running_task(self, generate_task_model:GenerateTaskModel, identity_id, project_id): """ Check any running tasks of this project - """ - ls_running_task = generate_task_model.query_running_tasks(identity_id, project_id) + """ + ls_running_task = generate_task_model.query_running_tasks(identity_id, project_id) if len(ls_running_task) > 0: raise Exception(MESS_ERROR_OVER_LIMIT_RUNNING_TASK) @@ -84,21 +86,21 @@ def _get_type_method(self, ls_methods_id): type_method = VALUE_TYPE_METHOD_PREPROCESS else: raise Exception(MESS_ERR_INVALID_LIST_METHOD) - + return type_method - def _check_generate_times_limitation(self, identity_id, project_name, type_method): + def _check_generate_times_limitation(self, identity_id, project_name, type_method): project_rec = self.project_model.get_project_info(identity_id, project_name) if project_rec is None: raise Exception(MESS_PROJECT_NOT_FOUND.format(project_name)) - + times_generated = int(project_rec.get_value_w_default(ProjectItem.FIELD_TIMES_AUGMENT, 0)) times_preprocess = int(project_rec.get_value_w_default(ProjectItem.FIELD_TIMES_PREPRO, 0)) - s3_prefix = project_rec.__dict__[ProjectItem.FIELD_S3_PREFIX] + s3_prefix = project_rec.__dict__[ProjectItem.FIELD_S3_PREFIX] - if type_method == VALUE_TYPE_METHOD_AUGMENT: + if type_method == VALUE_TYPE_METHOD_AUGMENT: if times_generated >= int(self.const.get_param(os.environ["LIMIT_AUGMENT_TIMES"])): - raise Exception(MESS_REACH_LIMIT_AUGMENT.format(self.const.get_param(os.environ["LIMIT_AUGMENT_TIMES"]))) + raise Exception(MESS_REACH_LIMIT_AUGMENT.format(self.const.get_param(os.environ["LIMIT_AUGMENT_TIMES"]))) elif type_method == VALUE_TYPE_METHOD_PREPROCESS: if times_preprocess >= int(self.const.get_param(os.environ["LIMIT_PROCESSING_TIMES"])): raise Exception(MESS_REACH_LIMIT_PREPROCESS.format(self.const.get_param(os.environ["LIMIT_PROCESSING_TIMES"]))) @@ -124,9 +126,9 @@ def _update_generate_times(self, identity_id, project_name, type_method, times_a def _create_task(self, identity_id, project_id, type_method): # create task id task_id = self.generate_task_model.create_new_generate_task(identity_id, project_id, type_method) - return task_id - - def _put_event_bus(self, detail_pass_para): + return task_id + + def _put_event_bus(self, detail_pass_para): response = self.client_events.put_events( Entries=[ @@ -140,7 +142,7 @@ def _put_event_bus(self, detail_pass_para): ) entries = response["Entries"] - return entries[0]["EventId"] + return entries[0]["EventId"] def _update_ls_method_for_preprocessing(self, ls_method_id): if "PRE-001" in ls_method_id: @@ -150,16 +152,16 @@ def _update_ls_method_for_preprocessing(self, ls_method_id): return ls_method_id def handle(self, event, context): - + ### parse body self.parser(event) ### check identity - identity_id = self.get_identity(self.id_token) + identity_id = self.get_identity(self.id_token) + + ### check running task + self._check_running_task(self.generate_task_model, identity_id, self.project_id) - ### check running task - self._check_running_task(self.generate_task_model, identity_id, self.project_id) - ### get type of process type_method = self._get_type_method(self.ls_methods_id) @@ -174,17 +176,17 @@ def handle(self, event, context): ### update the times_augment and times_preprocess to DB ### update reference images for last running - times_preprocess, times_augment = self._update_generate_times(identity_id, - self.project_name, type_method, - times_augment, times_preprocess, self.reference_images, self.aug_parameters) + times_preprocess, times_augment = self._update_generate_times(identity_id, + self.project_name, type_method, + times_augment, times_preprocess, self.reference_images, self.aug_parameters) ### update data number in case auto split for augmentation if type_method == VALUE_TYPE_METHOD_AUGMENT: - self.project_model.update_project_info(identity_id, self.project_name, self.data_type, self.data_number) + self.project_model.update_project_info(identity_id, self.project_name, self.data_type, self.data_number) ### check if preprocess then reset in prj sumary if type_method == VALUE_TYPE_METHOD_PREPROCESS: - self.project_sum_model.reset_prj_sum_preprocess(project_id = self.project_id, + self.project_sum_model.reset_prj_sum_preprocess(project_id = self.project_id, type_data=VALUE_TYPE_DATA_PREPROCESSED) ##delete image in preprocess table @@ -204,29 +206,42 @@ def handle(self, event, context): ### create taskID and update to DB task_id = self._create_task(identity_id, self.project_id, type_method) - ### push event to eventbridge + ### update parameters for next step detail_pass_para = { - KEY_NAME_IDENTITY_ID: identity_id, - KEY_NAME_PROJECT_ID: self.project_id, - KEY_NAME_PROJECT_NAME: self.project_name, - KEY_NAME_LS_METHOD_ID: self.ls_methods_id, - KEY_NAME_DATA_TYPE: self.data_type, - KEY_NAME_DATA_NUMBER: self.data_number, - KEY_NAME_S3_PREFIX: s3_prefix, - KEY_NAME_TIMES_AUGMENT: times_augment, - KEY_NAME_TIMES_PREPROCESS: times_preprocess, - KEY_NAME_TASK_ID: task_id, - KEY_NAME_ID_TOKEN: self.id_token, - KEY_NAME_PROCESS_TYPE: type_method, - KEY_NAME_REFERENCE_IMAGES: self.reference_images, - KEY_NAME_IS_RESOLUTION: self.is_normalize_resolution, - KEY_NAME_AUG_PARAMS: self.aug_parameters - } - event_id = self._put_event_bus(detail_pass_para) + KEY_NAME_IDENTITY_ID: identity_id, + KEY_NAME_PROJECT_ID: self.project_id, + KEY_NAME_PROJECT_NAME: self.project_name, + KEY_NAME_LS_METHOD_ID: self.ls_methods_id, + KEY_NAME_DATA_TYPE: self.data_type, + KEY_NAME_DATA_NUMBER: self.data_number, + KEY_NAME_S3_PREFIX: s3_prefix, + KEY_NAME_TIMES_AUGMENT: times_augment, + KEY_NAME_TIMES_PREPROCESS: times_preprocess, + KEY_NAME_TASK_ID: task_id, + KEY_NAME_ID_TOKEN: self.id_token, + KEY_NAME_PROCESS_TYPE: type_method, + KEY_NAME_REFERENCE_IMAGES: self.reference_images, + KEY_NAME_IS_RESOLUTION: self.is_normalize_resolution, + KEY_NAME_AUG_PARAMS: self.aug_parameters + } + + ### MUST FIX HERE, debug to run AI caller on ECS flow + if self.project_name == "test123": + sf = boto3.client('stepfunctions') + ### invoke stepfunction + response = sf.start_execution( + stateMachineArn = self.env.AI_CALLER_ECS_SM_ARN, + input = json.dumps(detail_pass_para) + ) + print("response when invoke exceution AI caller ECS: \n", response) + else: + ### push event to eventbridge + event_id = self._put_event_bus(detail_pass_para) + message = "OK" status_code = HTTPStatus.OK + sqsClient = boto3.client('sqs',REGION) - def getQueue(queue_name_env): try: response = sqsClient.get_queue_url(QueueName=queue_name_env) @@ -243,7 +258,7 @@ def countTaskInQueue(queue_id): ) num_task_in_queue = response['Attributes']['ApproximateNumberOfMessages'] print(f"QueueID: {queue_id} has len: {num_task_in_queue}") - return int(num_task_in_queue) + return int(num_task_in_queue) QueueResq = countTaskInQueue(os.environ['QUEUE']) print(QueueResq) if QueueResq > int(os.environ['MAX_CONCURRENCY_TASK']): @@ -260,11 +275,9 @@ def countTaskInQueue(queue_id): }, ) - + @error_response def lambda_handler(event, context): return GenerateImageClass().handle(event, context) - - \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/generate/split_data/app.py b/daita-app/core-service/functions/handlers/generate/split_data/app.py index fa6e820..fe821c1 100644 --- a/daita-app/core-service/functions/handlers/generate/split_data/app.py +++ b/daita-app/core-service/functions/handlers/generate/split_data/app.py @@ -12,52 +12,50 @@ class SplitDataClass(LambdaBaseClass): def __init__(self) -> None: super().__init__() - self.project_model = ProjectModel(os.environ["TABLE_PROJECTS_NAME"]) + self.project_model = ProjectModel(os.environ["TABLE_PROJECT"]) @LambdaBaseClass.parse_body def parser(self, body): self.logger.debug(f"body in main_parser: {body}") self.id_token = body[KEY_NAME_ID_TOKEN] - self.project_name = body[KEY_NAME_PROJECT_NAME] - self.data_type = body[KEY_NAME_DATA_TYPE].upper() # type is one of ORIGINAL or PREPROCESS, default is original + self.project_name = body[KEY_NAME_PROJECT_NAME] + self.data_type = body[KEY_NAME_DATA_TYPE].upper() # type is one of ORIGINAL or PREPROCESS, default is original self.data_number = body[KEY_NAME_DATA_NUMBER] # array of number data in train/val/test [100, 19, 1] def _check_input_value(self): if len(self.data_number)>0: if self.data_number[0] == 0: - raise Exception(MESS_NUMBER_TRAINING) + raise Exception(MESS_NUMBER_TRAINING) for number in self.data_number: if number<0: raise Exception(MESS_NUMBER_DATA) - + if self.data_type not in LS_ACCEPTABLE_VALUE_GENERATE: raise Exception(MESS_DATA_TYPE_INPUT.format(self.data_type, LS_ACCEPTABLE_VALUE_GENERATE)) return - + def handle(self, event, context): - + ### parse body self.parser(event) ### check identity - identity_id = self.get_identity(self.id_token) + identity_id = self.get_identity(self.id_token) ### update information self.project_model.update_project_info(identity_id, self.project_name, self.data_type, self.data_number) - + return generate_response( message="OK", status_code=HTTPStatus.OK, data=None, ) - + @error_response def lambda_handler(event, context): return SplitDataClass().handle(event, context) - - \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/project/apply_param_expert_mode/app.py b/daita-app/core-service/functions/handlers/project/apply_param_expert_mode/app.py index 989a13f..8809943 100644 --- a/daita-app/core-service/functions/handlers/project/apply_param_expert_mode/app.py +++ b/daita-app/core-service/functions/handlers/project/apply_param_expert_mode/app.py @@ -18,7 +18,7 @@ def __init__(self) -> None: super().__init__() self.client_events = boto3.client('events') self.const = SystemParameterStore() - self.project_model = ProjectModel(os.environ["TABLE_PROJECTS_NAME"]) + self.project_model = ProjectModel(os.environ["TABLE_PROJECT"]) @LambdaBaseClass.parse_body def parser(self, body): diff --git a/daita-app/core-service/functions/handlers/project/create_prj_fr_prebuild/app.py b/daita-app/core-service/functions/handlers/project/create_prj_fr_prebuild/app.py new file mode 100644 index 0000000..b4838ec --- /dev/null +++ b/daita-app/core-service/functions/handlers/project/create_prj_fr_prebuild/app.py @@ -0,0 +1,145 @@ +import boto3 +import json +import os +import uuid + +from config import * +from response import * +from error_messages import * +from utils import convert_current_date_to_iso8601, aws_get_identity_id, get_num_prj +import const + +from system_parameter_store import SystemParameterStore +from lambda_base_class import LambdaBaseClass +from models.prebuild_dataset_model import PrebuildDatasetModel +from models.project_model import ProjectModel, ProjectItem +from boto3.dynamodb.conditions import Key, Attr + + + +class CreatePrebuildDatasetClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.client_events = boto3.client('events') + self.client_step_func = boto3.client('stepfunctions') + self.prebuild_dataset_model = PrebuildDatasetModel(os.environ["T_CONST_PREBUILD_DATASET"]) + self.sm_create_prj_prebuild = os.environ["SM_CREATE_PRJ_PREBUILD"] + self.project_model = ProjectModel(os.environ["TABLE_PROJECT"]) + self.bucket_name = os.environ["BUCKET_NAME"] + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + + self.id_token = body[KEY_NAME_ID_TOKEN] + self.name_id_prebuild_dataset = body["name_id_prebuild"] + self.project_name = body["project_name"] + self.number_random = body["number_random"] + self.project_info = body["project_info"] + + def _check_input_value(self): + prebuild_dataset = self.prebuild_dataset_model.get_prebuild_dataset(self.name_id_prebuild_dataset) + if prebuild_dataset is None: + raise Exception(MESS_ERR_INVALID_PREBUILD_DATASET_NAME.format(self.name_id_prebuild_dataset)) + + ### check number max + if self.number_random <= 0 or self.number_random >= prebuild_dataset[PrebuildDatasetModel.FIELD_TOTAL_IMAGES]: + self.number_random = prebuild_dataset[PrebuildDatasetModel.FIELD_TOTAL_IMAGES] + if self.number_random > const.MAX_NUM_IMGAGES_CLONE_FROM_PREBUILD_DATASET: + raise (Exception( + f'The number of images should not exceed {const.MAX_NUM_IMGAGES_CLONE_FROM_PREBUILD_DATASET}!')) + + ### udpate the link to s3 + self.s3_key = prebuild_dataset[PrebuildDatasetModel.FIELD_S3_KEY] + self.visual_name = prebuild_dataset[PrebuildDatasetModel.FIELD_VISUAL_NAME] + + ### + try: + # check length of projectname and project info + if len(self.project_name) > const.MAX_LENGTH_PROJECT_NAME_INFO: + raise Exception(const.MES_LENGTH_OF_PROJECT_NAME) + if len(self.project_info) > const.MAX_LENGTH_PROJECT_DESCRIPTION: + raise Exception(const.MES_LENGTH_OF_PROJECT_INFO) + except Exception as e: + print('Error: ', repr(e)) + raise Exception(repr(e)) + + return + + def handle(self, event, context): + + ### parse body + self.parser(event) + + ### check identity + identity_id = self.get_identity(self.id_token) + + ### check limit project + num_prj=get_num_prj(identity_id) + if num_prj >= const.MAX_NUM_PRJ_PER_USER: + raise Exception(const.MES_REACH_LIMIT_NUM_PRJ) + + ### create project on DB + _uuid = uuid.uuid4().hex + project_id = f'{self.project_name}_{_uuid}' + s3_prefix = f'{self.bucket_name}/{identity_id}/{project_id}' + db_resource = boto3.resource('dynamodb') + try: + is_sample = False + gen_status = VALUE_STATUS_CREATE_SAMPLE_PRJ_GENERATING + item = { + 'ID': _uuid, + 'project_id': project_id, + 'identity_id': identity_id, + 'project_name': self.project_name, + 's3_prefix': s3_prefix, + 'project_info': self.project_info, + 'created_date': convert_current_date_to_iso8601(), + 'is_sample': is_sample, + 'gen_status': gen_status + } + condition = Attr(ProjectItem.FIELD_PROJECT_NAME).not_exists() & Attr(ProjectItem.FIELD_IDENTITY_ID).not_exists() + self.project_model.put_item_w_condition(item, condition=condition) + + except db_resource.meta.client.exceptions.ConditionalCheckFailedException as e: + print('Error condition: ', e) + raise Exception(MES_DUPLICATE_PROJECT_NAME.format(self.project_name)) + + except Exception as e: + print('Error: ', repr(e)) + raise Exception(repr(e)) + + ### call async step function + stepfunction_input = { + "identity_id": identity_id, + "project_id": project_id, + "project_name": self.project_name, + "bucket_name": self.bucket_name, + "s3_prefix_create": s3_prefix, + "number_random": self.number_random, + "s3_prefix_prebuild": self.s3_key if (f"s3://{self.bucket_name}" not in self.s3_key) else self.s3_key.replace(f"s3://{self.bucket_name}/", "") + } + response = self.client_step_func.start_execution( + stateMachineArn=self.sm_create_prj_prebuild, + input=json.dumps(stepfunction_input) + ) + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={ + "project_id": project_id, + "s3_prefix": s3_prefix, + "is_sample": is_sample, + "gen_status": gen_status, + "project_name": self.project_name + }, + ) + +@error_response +def lambda_handler(event, context): + + return CreatePrebuildDatasetClass().handle(event, context) + + \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/project/delete_images/__init__.py b/daita-app/core-service/functions/handlers/project/delete_images/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/delete_images/app.py b/daita-app/core-service/functions/handlers/project/delete_images/app.py similarity index 81% rename from daita-app/core-service/functions/handlers/delete_images/app.py rename to daita-app/core-service/functions/handlers/project/delete_images/app.py index fa44c42..8668cc9 100644 --- a/daita-app/core-service/functions/handlers/delete_images/app.py +++ b/daita-app/core-service/functions/handlers/project/delete_images/app.py @@ -12,7 +12,9 @@ from identity_check import * from utils import * MAX_NUMBER_ITEM_DELETE = 100 - +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] def delete_image_healthycheck_info(db_resource,project_id,healthcheck_id): table_healthycheck_info = db_resource.Table(os.environ['TABLE_HEALTHCHECK_INFO']) @@ -24,33 +26,34 @@ def delete_image_healthycheck_info(db_resource,project_id,healthcheck_id): ) def delete_reference_images(db_resource,identity_id,project_id,deletedfilename): print(f'Log Debug: {deletedfilename}') - table_project = db_resource.Table(os.environ['T_PROJECT']) + table_project = db_resource.Table(os.environ['TABLE_PROJECT']) queryResponse = table_project.query( KeyConditionExpression=Key('identity_id').eq(identity_id), FilterExpression=Attr('project_id').eq(project_id) ) with table_project.batch_writer() as batch: for each in queryResponse['Items']: - reference_images = each['reference_images'] - new_reference_image = {} - for k , v in reference_images.items(): - if not deletedfilename in v : - new_reference_image[k] = v - - resp = table_project.update_item( - Key={ - 'project_name': each['project_name'], - 'identity_id': identity_id , - }, - ExpressionAttributeNames= { - '#r': 'reference_images', - }, - ExpressionAttributeValues = { - ':r': new_reference_image - }, - UpdateExpression = 'SET #r = :r' - ) - print(resp) + if 'reference_images' in each: + reference_images = each['reference_images'] + new_reference_image = {} + for k , v in reference_images.items(): + if not deletedfilename in v : + new_reference_image[k] = v + + resp = table_project.update_item( + Key={ + 'project_name': each['project_name'], + 'identity_id': identity_id , + }, + ExpressionAttributeNames= { + '#r': 'reference_images', + }, + ExpressionAttributeValues = { + ':r': new_reference_image + }, + UpdateExpression = 'SET #r = :r' + ) + print(resp) def lambda_handler(event, context): @@ -60,7 +63,7 @@ def lambda_handler(event, context): #check authentication id_token = body["id_token"] - identity_id = aws_get_identity_id(id_token) + identity_id = aws_get_identity_id(id_token, USERPOOLID, IDENTITY_POOL) #get request data project_id = body['project_id'] @@ -101,7 +104,7 @@ def lambda_handler(event, context): # we use batch_write, it means that if key are existed in tables => overwrite db_resource = boto3.resource('dynamodb') try: - table_sum_all = db_resource.Table(os.environ['T_PROJECT_SUMMARY']) + table_sum_all = db_resource.Table(os.environ['TABLE_PROJECT_SUMMARY']) for key, value in dict_request.items(): @@ -114,11 +117,12 @@ def lambda_handler(event, context): 'project_id': project_id, 'filename': request['filename'] }) + print(f'log debug delete {item}') try: delete_reference_images(db_resource=db_resource,identity_id=identity_id,project_id=project_id,deletedfilename=request['filename']) except Exception as e: print(e) - if ('healthcheck_id' in item['Item']) and (not item['Item']['healthcheck_id'] is None or isinstance(item['Item']['healthcheck_id'],str)): + if 'Item' in item and ('healthcheck_id' in item['Item']) and (not item['Item']['healthcheck_id'] is None or isinstance(item['Item']['healthcheck_id'],str)): delete_image_healthycheck_info(db_resource=db_resource,project_id=project_id,healthcheck_id=item['Item']['healthcheck_id']) table.delete_item(Key={ diff --git a/daita-app/core-service/functions/handlers/project/delete_project/__init__.py b/daita-app/core-service/functions/handlers/project/delete_project/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/delete_project/app.py b/daita-app/core-service/functions/handlers/project/delete_project/app.py similarity index 92% rename from daita-app/core-service/functions/handlers/delete_project/app.py rename to daita-app/core-service/functions/handlers/project/delete_project/app.py index c9e9894..73fa7bc 100644 --- a/daita-app/core-service/functions/handlers/delete_project/app.py +++ b/daita-app/core-service/functions/handlers/project/delete_project/app.py @@ -10,7 +10,9 @@ from error_messages import * from identity_check import * from utils import * - +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] def CheckRunningAndDeleteTask(tableTask,identity_id,project_id,name): queryResp = tableTask.query( KeyConditionExpression=Key('identity_id').eq(identity_id), @@ -40,7 +42,7 @@ def lambda_handler(event, context): #check authentication id_token = body["id_token"] - identity_id = aws_get_identity_id(id_token) + identity_id = aws_get_identity_id(id_token, USERPOOLID, IDENTITY_POOL) #get request data project_id = body['project_id'] @@ -49,7 +51,7 @@ def lambda_handler(event, context): db_resource = boto3.resource('dynamodb') #### check task that belongs to the project id - tableGenerateTask = db_resource.Table(os.environ['T_TASKS']) + tableGenerateTask = db_resource.Table(os.environ['TABLE_TASK']) tableDataFlowstask = db_resource.Table(os.environ['T_DATA_FLOW']) tableReferenceImages = db_resource.Table(os.environ['T_REFERENCE_IMAGE']) tableHealthycheckTask = db_resource.Table(os.environ['TABLE_HEALTHCHECK_TASK']) @@ -76,7 +78,7 @@ def lambda_handler(event, context): 'healthcheck_id': each['healthcheck_id'] }) #### delete in project summary - table = db_resource.Table(os.environ['T_PROJECT_SUMMARY']) + table = db_resource.Table(os.environ['TABLE_PROJECT_SUMMARY']) for type_method in ['ORIGINAL', 'PREPROCESS', 'AUGMENT']: table.delete_item(Key={ 'project_id': project_id, @@ -84,7 +86,7 @@ def lambda_handler(event, context): }) #### delete project info - table_project = db_resource.Table(os.environ['T_PROJECT']) + table_project = db_resource.Table(os.environ['TABLE_PROJECT']) table_project_delete = db_resource.Table(os.environ['T_PROJECT_DEL']) dydb_update_delete_project(table_project, table_project_delete, identity_id, project_name) diff --git a/daita-app/core-service/functions/handlers/project/list_prebuild_dataset/app.py b/daita-app/core-service/functions/handlers/project/list_prebuild_dataset/app.py new file mode 100644 index 0000000..c873efb --- /dev/null +++ b/daita-app/core-service/functions/handlers/project/list_prebuild_dataset/app.py @@ -0,0 +1,55 @@ +import boto3 +import json +import os + +from config import * +from response import * +from error_messages import * +from identity_check import * + +from system_parameter_store import SystemParameterStore +from lambda_base_class import LambdaBaseClass +from models.prebuild_dataset_model import PrebuildDatasetModel + + +class ListPrebuildDatasetClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.client_events = boto3.client('events') + self.const = SystemParameterStore() + self.prebuild_dataset_model = PrebuildDatasetModel(os.environ["T_CONST_PREBUILD_DATASET"]) + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + + self.id_token = body[KEY_NAME_ID_TOKEN] + + def _check_input_value(self): + return + + def handle(self, event, context): + + ### parse body + self.parser(event) + + ### check identity + identity_id = self.get_identity(self.id_token) + + ### get list info + items = self.prebuild_dataset_model.get_list_prebuild_dataset() + ls_item_convert = [self.prebuild_dataset_model.convert_item_to_json(item) for item in items] + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data=ls_item_convert, + ) + +@error_response +def lambda_handler(event, context): + + return ListPrebuildDatasetClass().handle(event, context) + + \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/project/project_asy_create_sample/__init__.py b/daita-app/core-service/functions/handlers/project/project_asy_create_sample/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/project/project_asy_create_sample/app.py b/daita-app/core-service/functions/handlers/project/project_asy_create_sample/app.py new file mode 100644 index 0000000..9e44857 --- /dev/null +++ b/daita-app/core-service/functions/handlers/project/project_asy_create_sample/app.py @@ -0,0 +1,143 @@ +import json +import boto3 +import hashlib +import hmac +import base64 +import os +import uuid +from lambda_base_class import LambdaBaseClass +from utils import convert_response, convert_current_date_to_iso8601, aws_get_identity_id, move_data_s3 + + +class AsyncCreateSample(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def parser(self, body): + self.s3_prefix = body["s3_prefix"] + self.project_id = body["project_id"] + self.project_name = body["project_name"] + self.identity_id = body["identity_id"] + + def handle(self, event, context): + self.parser(json.loads(event['body'])) + db_client = boto3.client('dynamodb') + db_resource = boto3.resource('dynamodb') + + # move data in s3 + PREFIX_SAMPLE = "sample/" + ls_info = move_data_s3( + PREFIX_SAMPLE, self.s3_prefix, os.environ["BUCKET_NAME"]) + + # update to DB + # create the batch request from input data and summary the information + ls_item_request = [] + total_size = 0 + count = 0 + for object in ls_info: + # update summary information + size_old = 0 + total_size += (object[2]-size_old) + if size_old <= 0: + count += 1 + + is_ori = True + type_method = 'ORIGINAL' + item_request = { + 'project_id': self.project_id, # partition key + 's3_key': object[1], # sort_key + 'filename': object[0], + 'hash': '', # we use function get it mean that this field is optional in body + 'size': object[2], # size must be in Byte unit + 'is_ori': True, + 'type_method': type_method, + 'gen_id': '', # id of generation method + 'created_date': convert_current_date_to_iso8601() + } + ls_item_request.append(item_request) + + try: + table = db_resource.Table(os.environ["T_DATA_ORI"]) + with table.batch_writer() as batch: + for item in ls_item_request: + batch.put_item(Item=item) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + # update summary information + try: + response = db_client.update_item( + TableName=os.environ["TABLE_PROJECT_SUMMARY"], + Key={ + 'project_id': { + 'S': self.project_id + }, + 'type': { + 'S': type_method + } + }, + ExpressionAttributeNames={ + '#SI': 'total_size', + '#COU': 'count', + '#TK': 'thu_key', + '#TN': 'thu_name' + }, + ExpressionAttributeValues={ + ':si': { + 'N': str(total_size) + }, + ':cou': { + 'N': str(count) + }, + ':tk': { + 'S': ls_item_request[0]['s3_key'] + }, + ':tn': { + 'S': ls_item_request[0]['filename'] + } + }, + UpdateExpression='SET #TK = :tk, #TN = :tn ADD #SI :si, #COU :cou', + ) + print('response_summary: ', response) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + # update generate status + try: + table = db_resource.Table(os.environ['TABLE_PROJECT']) + response = table.update_item( + Key={ + 'identity_id': self.identity_id, + 'project_name': self.project_name, + }, + ExpressionAttributeValues={ + ':st': "FINISH", + }, + UpdateExpression='SET gen_status = :st' + ) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + return convert_response( + { + 'data': None, + "error": False, + "success": True, + "message": None + }) + + +def lambda_handler(event, context): + return AsyncCreateSample().handle(event=event, context=context) diff --git a/daita-app/core-service/functions/handlers/project/project_create/__init__.py b/daita-app/core-service/functions/handlers/project/project_create/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/project/project_create/app.py b/daita-app/core-service/functions/handlers/project/project_create/app.py new file mode 100644 index 0000000..ced7260 --- /dev/null +++ b/daita-app/core-service/functions/handlers/project/project_create/app.py @@ -0,0 +1,115 @@ +from lambda_base_class import LambdaBaseClass +import json +import boto3 +import hashlib +import hmac +import base64 +import os +import uuid +from boto3.dynamodb.conditions import Key, Attr + +from utils import convert_response, convert_current_date_to_iso8601, aws_get_identity_id, get_num_prj +import const +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + + +class CreateProject(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def parser(self, body): + self.id_token = body["id_token"] + self.access_token = body["access_token"] + self.project_name = body["project_name"] + self.project_info = body.get("project_info", '') + + def handle(self, event, context): + self.parser(json.loads(event['body'])) + try: + identity_id = aws_get_identity_id(self.id_token, USERPOOLID, IDENTITY_POOL) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + db_resource=boto3.resource("dynamodb") + + try: + # check length of projectname and project info + if len(self.project_name) > const.MAX_LENGTH_PROJECT_NAME_INFO: + raise Exception(const.MES_LENGTH_OF_PROJECT_NAME) + if len(self.project_info) > const.MAX_LENGTH_PROJECT_DESCRIPTION: + raise Exception(const.MES_LENGTH_OF_PROJECT_INFO) + + num_prj=get_num_prj(identity_id) + if num_prj >= const.MAX_NUM_PRJ_PER_USER: + raise Exception(const.MES_REACH_LIMIT_NUM_PRJ) + + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + _uuid=uuid.uuid4().hex + project_id=f'{self.project_name}_{_uuid}' + s3_prefix=f'{os.environ["BUCKET_NAME"]}/{identity_id}/{project_id}' + db_client=boto3.client('dynamodb') + try: + is_sample=False + gen_status="FINISH" + table_prj=db_resource.Table(os.environ["TABLE_PROJECT"]) + table_prj.put_item( + Item = { + 'ID': _uuid, + 'project_id': project_id, + 'identity_id': identity_id, + 'project_name': self.project_name, + 's3_prefix': s3_prefix, + 'project_info': self.project_info, + # 'sub': sub, + 'created_date': convert_current_date_to_iso8601(), + 'is_sample': is_sample, + 'gen_status': gen_status + }, + ConditionExpression = Attr('project_name').not_exists() & Attr( + 'identity_id').not_exists() + ) + + except db_resource.meta.client.exceptions.ConditionalCheckFailedException as e: + print('Error condition: ', e) + err_mess=const.MES_DUPLICATE_PROJECT_NAME.format( + self.project_name) + return convert_response({"error": True, + "success": False, + "message": err_mess, + "data": None}) + except Exception as e: + print('Error: ', e) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + return convert_response( + { + 'data': { + "project_id": project_id, + "s3_prefix": s3_prefix, + "is_sample": is_sample, + "gen_status": gen_status, + "project_name": self.project_name + }, + "error": False, + "success": True, + "message": None + }) + + +def lambda_handler(event, context): + return CreateProject().handle(event = event, context = context) diff --git a/daita-app/core-service/functions/handlers/project/project_download_create/__init__.py b/daita-app/core-service/functions/handlers/project/project_download_create/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/project/project_download_create/app.py b/daita-app/core-service/functions/handlers/project/project_download_create/app.py new file mode 100644 index 0000000..907e44f --- /dev/null +++ b/daita-app/core-service/functions/handlers/project/project_download_create/app.py @@ -0,0 +1,111 @@ +from tkinter import E +from lambda_base_class import LambdaBaseClass +import json +import boto3 +import hashlib +import hmac +import base64 +import os +import uuid +import requests +import random +from boto3.dynamodb.conditions import Key, Attr +from utils import convert_response, convert_current_date_to_iso8601, aws_get_identity_id +import const + +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + + +class ProjectDownloadCreateCls(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def parser(self, body): + self.id_token = body["id_token"] + self.project_id = body['project_id'] + self.project_name = body['project_name'] + self.down_type = body['down_type'] + + def handle(self, event, context): + self.parser(json.loads(event['body'])) + try: + identity_id = aws_get_identity_id( + self.id_token, USERPOOLID, IDENTITY_POOL) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + db_resource = boto3.resource('dynamodb') + + # create task id and save to DB + try: + table = db_resource.Table(os.environ["T_TASK_DOWNLOAD"]) + task_id = uuid.uuid4().hex + task_id = f"{convert_current_date_to_iso8601()}-{task_id}" + create_time = convert_current_date_to_iso8601() + table.put_item( + Item={ + "identity_id": identity_id, + "task_id": task_id, + "status": "RUNNING", + "process_type": "DOWNLOAD", + "project_id": self.project_id, + "created_time": create_time, + } + ) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + # send request to url + try: + url = 'http://'+os.environ["DOWNLOAD_SERVICE_URL"]+':8000/download' + request_body = { + "project_id": self.project_id, + "project_name": self.project_name, + "down_type": self.down_type, + "task_id": task_id, + "identity_id": identity_id + } + headers = {"Content-Type": "application/json"} + response = requests.post( + url, + json=request_body, + headers=headers, + ) + + if response.status_code == 500: + # error when download + raise Exception( + "There are some error when downloading, please try again") + + value = response.json() + print('response request: ', value) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + return convert_response( + { + 'data': { + "task_id": task_id + }, + "error": False, + "success": True, + "message": None + }) + + +def lambda_handler(event, context): + return ProjectDownloadCreateCls().handle(event=event, context=context) diff --git a/daita-app/core-service/functions/handlers/project/project_download_update/__init__.py b/daita-app/core-service/functions/handlers/project/project_download_update/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/project/project_download_update/app.py b/daita-app/core-service/functions/handlers/project/project_download_update/app.py new file mode 100644 index 0000000..6548975 --- /dev/null +++ b/daita-app/core-service/functions/handlers/project/project_download_update/app.py @@ -0,0 +1,78 @@ +from lambda_base_class import LambdaBaseClass +import json +import boto3 +import hashlib +import hmac +import base64 +import os +import uuid +import random +from boto3.dynamodb.conditions import Key, Attr +from utils import convert_response, convert_current_date_to_iso8601, aws_get_identity_id +import const + +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + + +class ProjectDownloadUpdateCls(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def parser(self, body): + self.id_token = body["id_token"] + self.task_id = body['task_id'] + + def handle(self, event, context): + self.parser(json.loads(event['body'])) + try: + identity_id = aws_get_identity_id( + self.id_token, USERPOOLID, IDENTITY_POOL) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + db_resource = boto3.resource('dynamodb') + + # create task id and save to DB + try: + table = db_resource.Table(os.environ["T_TASK_DOWNLOAD"]) + response = table.get_item( + Key={ + "identity_id": identity_id, + "task_id": self.task_id, + } + ) + if response.get("Item", None): + status = response["Item"].get("status") + s3_key = response["Item"].get("s3_key", None) + presign_url = response["Item"].get("presign_url", None) + else: + raise Exception(f"Task id ({self.task_id}) is not exist!") + + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + return convert_response( + { + 'data': { + "status": status, + "s3_key": s3_key, + "presign_url": presign_url + }, + "error": False, + "success": True, + "message": None + }) + + +def lambda_handler(event, context): + return ProjectDownloadUpdateCls().handle(event=event, context=context) diff --git a/daita-app/core-service/functions/handlers/project/project_info/__init__.py b/daita-app/core-service/functions/handlers/project/project_info/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/project/project_info/app.py b/daita-app/core-service/functions/handlers/project/project_info/app.py new file mode 100644 index 0000000..cb58f50 --- /dev/null +++ b/daita-app/core-service/functions/handlers/project/project_info/app.py @@ -0,0 +1,170 @@ +from lambda_base_class import LambdaBaseClass +import json +import boto3 +import hashlib +import hmac +import base64 +import os +from decimal import Decimal + +from boto3.dynamodb.conditions import Key, Attr +from utils import convert_response, aws_get_identity_id, dydb_get_project_id, dydb_get_project_full + +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + + +def get_running_task(table_name, db_resource, ls_tasks, identity_id, res_projectid, task_type=""): + table = db_resource.Table(table_name) + item_tasks = table.query( + ProjectionExpression='project_id, task_id, process_type', + KeyConditionExpression=Key('identity_id').eq(identity_id), + FilterExpression=Attr('status').ne('FINISH') & Attr('status').ne( + 'ERROR') & Attr('status').ne('CANCEL') & Attr('project_id').eq(res_projectid) + ) + for item in item_tasks['Items']: + ls_tasks.append({ + "task_id": item.get('task_id', ''), + "process_type": item.get('process_type', task_type) + }) + return ls_tasks + + +def from_dynamodb_to_json(item): + # print(f"Item to serialize: \n {item}") + for method, param in item.items(): + for key, value in param.items(): + if type(value) is Decimal: + param[key] = float(value) + + # print(f"Result after serialize: \n {serialize}") + return item + + +class ProjectInfoCls(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def parser(self, body): + self.id_token = body["id_token"] + self.project_name = body["project_name"] + + def handle(self, event, context): + self.parser(json.loads(event['body'])) + try: + identity_id = aws_get_identity_id( + + self.id_token, USERPOOLID, IDENTITY_POOL) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + db_resource = boto3.resource('dynamodb') + # get project_id + try: + table = db_resource.Table(os.environ['TABLE_PROJECT']) + res_project = dydb_get_project_full( + table, identity_id, self.project_name) + + # print(res_projectid) + res_projectid = res_project['project_id'] + is_sample = res_project.get("is_sample", False) + # default is finish, else GENERATING + gen_status = res_project.get("gen_status", "FINISH") + res_times_generated = int(res_project.get('times_generated', 0)) + reference_images = res_project.get("reference_images", {}) + aug_params = res_project.get("aug_parameters", {}) + reference_info = {} + for method, s3_path in reference_images.items(): + filename = s3_path.split("/")[-1] + reference_info[method] = { + "s3_path": s3_path, + "filename": filename + } + + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + # get info detail of a project + try: + table = db_resource.Table(os.environ['TABLE_PROJECT_SUMMARY']) + response = table.query( + KeyConditionExpression=Key('project_id').eq(res_projectid), + ) + # print(response) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + if response.get('Items', None): + groups = {} + for item in response['Items']: + type = item['type'] + data_num = res_project.get(type, None) + if data_num is not None: + data_num = [int(a) for a in data_num] + groups[type] = { + 'count': int(item['count']), + 'size': int(item['total_size']), + 'data_number': data_num + } + + # get running tasks of project + ls_tasks = [] + # get task of generation + ls_tasks = get_running_task( + os.environ['TABLE_TASK'], db_resource, ls_tasks, identity_id, res_projectid) + ls_tasks = get_running_task( + os.environ["TABLE_HEALTHCHECK_TASK"], db_resource, ls_tasks, identity_id, res_projectid, "HEALTHCHECK") + ls_tasks = get_running_task( + os.environ["TABLE_DATA_FLOW_TASK"], db_resource, ls_tasks, identity_id, res_projectid) + ls_tasks = get_running_task( + os.environ["TABLE_REFERENCE_IMAGE_TASK"], db_resource, ls_tasks, identity_id, res_projectid) + + return convert_response({'data': { + "identity_id": identity_id, + "project_name": self.project_name, + "project_id": res_projectid, + "times_generated": res_times_generated, + "is_sample": is_sample, + "gen_status": gen_status, + "ls_task": ls_tasks, + "groups": groups, + "reference_images": reference_info, + "aug_parameters": from_dynamodb_to_json(aug_params) + }, + "error": False, + "success": True, + "message": None}) + else: + return convert_response({'data': { + "identity_id": identity_id, + "project_name": self.project_name, + "project_id": res_projectid, + "times_generated": res_times_generated, + "is_sample": is_sample, + "gen_status": gen_status, + "ls_task": [], + "groups": None, + "reference_images": reference_info, + "aug_parameters": from_dynamodb_to_json(aug_params) + }, + "error": False, + "success": True, + "message": None}) + + +def lambda_handler(event, context): + return ProjectInfoCls().handle(event=event, context=context) + diff --git a/daita-app/core-service/functions/handlers/project/project_list/__init__.py b/daita-app/core-service/functions/handlers/project/project_list/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/project/project_list/app.py b/daita-app/core-service/functions/handlers/project/project_list/app.py new file mode 100644 index 0000000..a0f04e7 --- /dev/null +++ b/daita-app/core-service/functions/handlers/project/project_list/app.py @@ -0,0 +1,66 @@ +from lambda_base_class import LambdaBaseClass +import json +import boto3 +import hashlib +import hmac +import base64 +import os +from boto3.dynamodb.conditions import Key, Attr +from utils import convert_response, aws_get_identity_id +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + + +class ProjectListCls(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def parser(self, body): + self.id_token = body["id_token"] + + def handle(self, event, context): + self.parser(event['body']) + # get identity_id from id token, also check the authentication from client + try: + identity_id = aws_get_identity_id( + self.id_token, USERPOOLID, IDENTITY_POOL) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + # query list of projects + db_resource = boto3.resource('dynamodb') + try: + table = db_resource.Table(os.environ['TABLE_PROJECT']) + items = table.query( + ProjectionExpression='project_name, project_id, s3_prefix, is_sample, gen_status', + KeyConditionExpression=Key('identity_id').eq(identity_id), + ) + ls_item = [] + if items.get("Items", None): + for item in items["Items"]: + item["is_sample"] = item.get("is_sample", False) + item["gen_status"] = item.get("gen_status", "FINISH") + ls_item.append(item) + + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + return convert_response({'data': { + 'items': ls_item + }, + "error": False, + "success": True, + "message": None}) + + +def lambda_handler(event, context): + return ProjectListCls().handle(event=event, context=context) diff --git a/daita-app/core-service/functions/handlers/project/project_list_data/__init__.py b/daita-app/core-service/functions/handlers/project/project_list_data/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/project/project_list_data/app.py b/daita-app/core-service/functions/handlers/project/project_list_data/app.py new file mode 100644 index 0000000..c7a2ca5 --- /dev/null +++ b/daita-app/core-service/functions/handlers/project/project_list_data/app.py @@ -0,0 +1,115 @@ +from http.client import responses +from lambda_base_class import LambdaBaseClass +import json +import boto3 +import hashlib +import hmac +import base64 +import os +from boto3.dynamodb.conditions import Key, Attr + +from utils import convert_response, aws_get_identity_id + +MAX_NUMBER_LIMIT = 500 +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + + +def lambda_handler(event, context): + return ProjectListCls().handle(event, context) + + +class ProjectListCls(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def parser(self, body): + self.id_token = body["id_token"] + self.project_id = body["project_id"] + self.type_method = body["type_method"] + self.next_token = body["next_token"] + self.num_limit = min(MAX_NUMBER_LIMIT, body.get( + "num_limit", MAX_NUMBER_LIMIT)) + + def handle(self, event, context): + self.parser(json.loads(event['body'])) + # get identity_id from id token, also check the authentication from client + try: + identity_id = aws_get_identity_id( + self.id_token, USERPOOLID, IDENTITY_POOL) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + # query list data of project + dynamodb = boto3.resource('dynamodb') + try: + if self.type_method == 'ORIGINAL': + table_name = os.environ['T_DATA_ORI'] + elif self.type_method == 'PREPROCESS': + table_name = os.environ['T_DATA_PREPROCESS'] + elif self.type_method == 'AUGMENT': + table_name = os.environ['T_DATA_AUGMENT'] + else: + raise ( + Exception(f'type_method: {self.type_method} is not valid!')) + + table = dynamodb.Table(table_name) + if len(self.next_token) == 0: + response = table.query( + IndexName='index-created-sorted', + KeyConditionExpression=Key( + 'project_id').eq(self.project_id), + # ProjectionExpression='filename, s3_key, type_method, gen_id, created_date', + Limit=self.num_limit, + ScanIndexForward=False + ) + print('___Response first: ___', response) + else: + response = table.query( + IndexName='index-created-sorted', + KeyConditionExpression=Key( + 'project_id').eq(self.project_id), + # ProjectionExpression='filename, s3_key, type_method, gen_id, created_date', + ExclusiveStartKey=self.next_token, + Limit=self.num_limit, + ScanIndexForward=False + ) + print('___Response next: ___', response) + + self.next_token = None + # LastEvaluatedKey indicates that there are more results + if 'LastEvaluatedKey' in response: + self.next_token = response['LastEvaluatedKey'] + + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + items =[] + for it in response['Items']: + tempItem = { + 'created_date':it['created_date'], + 'filename': it['filename'], + 'gen_id':it['gen_id'], + 'type_method':it['type_method'], + 's3_key':it['s3_key'] + } + if 'thumbnail' in it and bool(it['thumbnail']): + tempItem['thumbnail'] = it['thumbnail'].replace('s3://','') + else: + tempItem['thumbnail'] = it['s3_key'] + items.append(tempItem) + return convert_response({'data': { + 'items': items, + 'next_token': self.next_token + }, + "error": False, + "success": True, + "message": None}) diff --git a/daita-app/core-service/functions/handlers/project/project_list_info/__init__.py b/daita-app/core-service/functions/handlers/project/project_list_info/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/project/project_list_info/app.py b/daita-app/core-service/functions/handlers/project/project_list_info/app.py new file mode 100644 index 0000000..87d0d3a --- /dev/null +++ b/daita-app/core-service/functions/handlers/project/project_list_info/app.py @@ -0,0 +1,116 @@ +from lambda_base_class import LambdaBaseClass +import json +import boto3 +import hashlib +import hmac +import base64 +import os +from boto3.dynamodb.conditions import Key, Attr +from utils import convert_response, aws_get_identity_id +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + + +def lambda_handler(event, context): + return ProjectListInfoCls().handle(event, context) + + +class ProjectListInfoCls(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def parser(self, body): + self.id_token = body["id_token"] + + def handle(self, event, context): + self.parser(json.loads(event['body'])) + # get identity_id from id token, also check the authentication from client + try: + identity_id = aws_get_identity_id( + self.id_token, USERPOOLID, IDENTITY_POOL) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + # query list of projects + db_resource = boto3.resource('dynamodb') + try: + table = db_resource.Table(os.environ['TABLE_PROJECT']) + items_project = table.query( + ProjectionExpression='project_name, project_id, s3_prefix, is_sample, gen_status, project_info', + KeyConditionExpression=Key('identity_id').eq(identity_id), + ) + + table = db_resource.Table(os.environ['TABLE_TASK']) + items_task = table.query( + ProjectionExpression='project_id, task_id', + KeyConditionExpression=Key('identity_id').eq(identity_id), + FilterExpression=Attr('status').eq('RUNNING') + ) + + print(items_project) + print(items_task) + + group_project_id = {} + + # add general project info + for index, item_project in enumerate(items_project['Items']): + print(f'Log debug: Index {index}: {item_project}') + # update value for is_sample and gen_status + item_project["is_sample"] = item_project.get( + "is_sample", False) + item_project["gen_status"] = item_project.get( + "gen_status", "FINISH") + item_project["description"] = item_project.get( + "project_info", "") + if not 'project_id' in item_project: + continue + group_project_id[item_project['project_id']] = item_project + group_project_id[item_project['project_id']]['ls_task'] = [] + + for item_task in items_task['Items']: + if len(item_task.get('project_id', '')) > 0: + group_project_id[item_task['project_id'] + ]['ls_task'].append(item_task) + + table = db_resource.Table(os.environ['TABLE_PROJECT_SUMMARY']) + + for key, value in group_project_id.items(): + response = table.query( + KeyConditionExpression=Key('project_id').eq(key), + ) + groups = {} + thumnail_key = None + for item in response['Items']: + type = item['type'] + groups[type] = { + 'count': int(item['count']), + 'size': int(item['total_size']), + } + if item.get('thu_key', None) is not None: + thumnail_key = item['thu_key'] + + value['groups'] = groups + value['thum_key'] = thumnail_key + + ls_items = [] + for key, value in group_project_id.items(): + ls_items.append(value) + + print('group_project_id: ', group_project_id) + + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + return convert_response({'data': ls_items, + "error": False, + "success": True, + "message": None}) diff --git a/daita-app/core-service/functions/handlers/project/project_sample/__init__.py b/daita-app/core-service/functions/handlers/project/project_sample/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/project/project_sample/app.py b/daita-app/core-service/functions/handlers/project/project_sample/app.py new file mode 100644 index 0000000..c97bcae --- /dev/null +++ b/daita-app/core-service/functions/handlers/project/project_sample/app.py @@ -0,0 +1,136 @@ +from lambda_base_class import LambdaBaseClass +import json +import boto3 +import hashlib +import hmac +import base64 +import os +import uuid +from boto3.dynamodb.conditions import Key, Attr +from utils import convert_response, convert_current_date_to_iso8601, aws_get_identity_id, move_data_s3, create_single_put_request, get_num_prj +import const + +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + + +def lambda_handler(event, context): + return ProjectSampleCls().handle(event, context) + + +class ProjectSampleCls(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def parser(self, body): + self.id_token = body["id_token"] + self.access_token = body["access_token"] + + self.project_name = const.SAMPLE_PROJECT_NAME + self.project_info = body.get( + 'project_info', const.SAMPLE_PROJECT_DESCRIPTION) + + def handle(self, event, context): + self.parser(json.loads(event['body'])) + try: + identity_id = aws_get_identity_id( + self.id_token, USERPOOLID, IDENTITY_POOL) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + # check limit of project per user + try: + num_prj = get_num_prj(identity_id) + if num_prj >= const.MAX_NUM_PRJ_PER_USER: + raise Exception(const.MES_REACH_LIMIT_NUM_PRJ) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + _uuid = uuid.uuid4().hex + project_id = f'{self.project_name}_{_uuid}' + s3_prefix = f'{os.environ["BUCKET_NAME"]}/{identity_id}/{project_id}' + db_client = boto3.client('dynamodb') + db_resource = boto3.resource('dynamodb') + try: + is_sample = True + gen_status = "GENERATING" + table_prj = db_resource.Table(os.environ["TABLE_PROJECT"]) + table_prj.put_item( + Item={ + 'ID': _uuid, + 'project_id': project_id, + 'identity_id': identity_id, + 'project_name': self.project_name, + 's3_prefix': s3_prefix, + 'project_info': self.project_info, + # 'sub': sub, + 'created_date': convert_current_date_to_iso8601(), + 'is_sample': is_sample, + 'gen_status': gen_status + }, + ConditionExpression=Attr('project_name').not_exists() & Attr( + 'identity_id').not_exists() + ) + + except db_resource.meta.client.exceptions.ConditionalCheckFailedException as e: + print('Error condition: ', e) + err_mess = const.MES_DUPLICATE_PROJECT_NAME.format( + self.project_name) + return convert_response({"error": True, + "success": False, + "message": err_mess, + "data": None}) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + # call asyn lambda for running + client = boto3.client('lambda') + try: + payload = { + "s3_prefix": s3_prefix, + "project_id": project_id, + "project_name": self.project_name, + "identity_id": identity_id + } + payloadStr = json.dumps(payload) + payloadBytesArr = bytes(payloadStr, encoding='utf8') + print('start send request') + response = client.invoke( + FunctionName="staging-project-asy-create-sample", + InvocationType='Event', + Payload=payloadBytesArr + ) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + print('response request from api gateway') + return convert_response( + { + 'data': { + "project_id": project_id, + "s3_prefix": s3_prefix, + "is_sample": is_sample, + "gen_status": gen_status, + "project_name": self.project_name + }, + "error": False, + "success": True, + "message": None + }) diff --git a/daita-app/core-service/functions/handlers/project/project_update_info/__init__.py b/daita-app/core-service/functions/handlers/project/project_update_info/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/project/project_update_info/app.py b/daita-app/core-service/functions/handlers/project/project_update_info/app.py new file mode 100644 index 0000000..057f196 --- /dev/null +++ b/daita-app/core-service/functions/handlers/project/project_update_info/app.py @@ -0,0 +1,131 @@ +from lambda_base_class import LambdaBaseClass +import json +import boto3 +import hashlib +import hmac +import base64 +import os +from boto3.dynamodb.conditions import Key, Attr +from utils import convert_response, aws_get_identity_id, dydb_get_project_id, dydb_get_project_full +import const + +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + + +def lambda_handler(event, context): + return ProjectUpdateCls().handle(event, context) + + +class ProjectUpdateCls(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def parser(self, body): + self.id_token = body["id_token"] + self.project_name = body["cur_project_name"] + self.new_prj_name = body.get("new_project_name", '') + self.new_description = body.get("new_description", '') + if self.project_name == self.new_prj_name: + raise Exception(const.MES_PROJECT_SAME.format(self.new_prj_name)) + + # check length of projectname and project info + if len(self.new_prj_name) > const.MAX_LENGTH_PROJECT_NAME_INFO: + raise Exception(const.MES_LENGTH_OF_PROJECT_NAME) + if len(self.new_description) > const.MAX_LENGTH_PROJECT_DESCRIPTION: + raise Exception(const.MES_LENGTH_OF_PROJECT_INFO) + + def handle(self, event, context): + self.parser(json.loads(event['body'])) + try: + identity_id = aws_get_identity_id( + self.id_token, USERPOOLID, IDENTITY_POOL) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + db_resource = boto3.resource('dynamodb') + # get project_id + try: + table = db_resource.Table(os.environ['TABLE_PROJECT']) + res_project = table.get_item( + Key={ + 'identity_id': identity_id, + 'project_name': self.project_name + } + ) + if res_project.get('Item', None): + current_info = res_project['Item'] + if len(self.new_description) == 0: + self.new_description = current_info['project_info'] + + if len(self.new_prj_name) > 0: + # check new name exist or not + response = table.get_item( + Key={ + 'identity_id': identity_id, + 'project_name': self.new_prj_name + } + ) + print(const.MES_PROJECT_ALREADY_EXIST.format( + self.new_prj_name)) + if response.get('Item', None): + raise Exception( + const.MES_PROJECT_ALREADY_EXIST.format(self.new_prj_name)) + else: + # add new project + current_info['project_info'] = self.new_description + current_info['project_name'] = self.new_prj_name + table.put_item( + Item=current_info + ) + + # delete current prj name + table.delete_item( + Key={ + 'identity_id': identity_id, + 'project_name': self.project_name, + } + ) + + else: + # only update description + table.update_item( + Key={ + 'identity_id': identity_id, + 'project_name': self.project_name, + }, + ExpressionAttributeValues={ + ':de': self.new_description + }, + UpdateExpression='SET project_info = :de' + ) + else: + raise Exception( + const.MES_PROJECT_NOT_FOUND.format(self.project_name)) + + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + return convert_response( + { + 'data': { + "project_id": current_info['project_id'], + "s3_prefix": current_info['s3_prefix'], + "is_sample": current_info['is_sample'], + "gen_status": current_info['gen_status'], + "project_name": current_info['project_name'], + "description": self.new_description + }, + "error": False, + "success": True, + "message": None + }) diff --git a/daita-app/core-service/functions/handlers/project/project_upload_check/__init__.py b/daita-app/core-service/functions/handlers/project/project_upload_check/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/project/project_upload_check/app.py b/daita-app/core-service/functions/handlers/project/project_upload_check/app.py new file mode 100644 index 0000000..a4d5b96 --- /dev/null +++ b/daita-app/core-service/functions/handlers/project/project_upload_check/app.py @@ -0,0 +1,110 @@ +from lambda_base_class import LambdaBaseClass +import json +import boto3 +import hashlib +import hmac +import base64 +import os +from utils import convert_response, aws_get_identity_id +import const + + +MAX_NUMBER_ITEM_QUERY = 1000 +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + + +def lambda_handler(event, context): + return ProjectUploadCheckCls().handle(event, context) + + +class ProjectUploadCheckCls(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def parser(self, body): + self.id_token = body["id_token"] + self.project_id = body["project_id"] + self.ls_filename = body["ls_filename"] + + # check quantiy of items + if len(self.ls_filename) > MAX_NUMBER_ITEM_QUERY: + raise Exception( + f'The number of items is over {MAX_NUMBER_ITEM_QUERY}') + + # create the batch request from input data + self.ls_batch_request = [] + for object in self.ls_filename: + request = { + 'project_id': {'S': self.project_id}, # partition key + 'filename': {'S': object} + } + self.ls_batch_request.append(request) + + def handle(self, event, context): + self.parser(json.loads(event['body'])) + # get identity_id from id token, also check the authentication from client + try: + identity_id = aws_get_identity_id( + self.id_token, USERPOOLID, IDENTITY_POOL) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + # query data from DB + try: + db_client = boto3.client('dynamodb') + ls_data = [] + start_idx = 0 + while start_idx < len(self.ls_batch_request): + next_idx = start_idx + 50 + response = db_client.batch_get_item( + RequestItems={ + os.environ["T_DATA_ORI"]: { + 'Keys': self.ls_batch_request[start_idx:next_idx], + 'ProjectionExpression': 'filename, size' + } + } + ) + start_idx = next_idx + print(start_idx, next_idx) + for data in response['Responses'][os.environ["T_DATA_ORI"]]: + ls_data.append( + {'filename': data['filename']['S'], 'size': data['size']['N']}) + + print(ls_data) + + # check available image is over the limitation + db_resource = boto3.resource("dynamodb") + table = db_resource.Table(os.environ["TABLE_PROJECT_SUMMARY"]) + response = table.get_item( + Key={ + "project_id": self.project_id, + "type": "ORIGINAL" + } + ) + if response.get('Item'): + current_num_data = response['Item'].get('count', 0) + else: + current_num_data = 0 + if len(self.ls_batch_request)-len(ls_data)+current_num_data > const.MAX_NUM_IMAGES_IN_ORIGINAL: + raise (Exception( + f'The number of images should not exceed {const.MAX_NUM_IMAGES_IN_ORIGINAL}!')) + + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + return convert_response({ + 'data': ls_data, + "error": False, + "success": True, + "message": None + }) diff --git a/daita-app/core-service/functions/handlers/send-mail/reference-email/app.py b/daita-app/core-service/functions/handlers/send-mail/reference-email/app.py new file mode 100644 index 0000000..2c74bbd --- /dev/null +++ b/daita-app/core-service/functions/handlers/send-mail/reference-email/app.py @@ -0,0 +1,112 @@ +import json +import boto3 +import os +from botocore.exceptions import ClientError + +from utils import convert_response +from config import * + +USERPOOLID = os.environ['COGNITO_USER_POOL'] + + +def get_email(user): + client = boto3.client("cognito-idp") + try: + resp = client.list_users( + UserPoolId=USERPOOLID, + ) + except Exception as e: + print(e) + return None, e + info_user = list(filter(lambda x: x["Username"] == user, resp["Users"])) + if len(info_user): + email = list( + filter(lambda x: x["Name"] == "email", info_user[0]["Attributes"])) + return email[0]["Value"], None + return None, None + + +def lambda_handler(event, context): + body = json.loads(event["body"]) + try: + source_user = body["username"] + destination_email = body["destination_email"] + except Exception as e: + return convert_response( + {"error": True, "success": False, "message": repr(e), "data": None} + ) + email_name, error = get_email(source_user) + if error != None: + return convert_response( + {"error": True, "success": False, + "message": str(error), "data": None} + ) + if email_name == None: + return convert_response( + { + "error": True, + "success": False, + "message": "The user is not exist", + "data": None, + } + ) + client = boto3.client("ses") + print("send mail") + message_email = """ +

Hi,

+

{} has invited you to explore DAITA's recently launched data augmentation platform.

+

Building a platform that machine learning engineers and data scientists really love is truly hard. But that's our ultimate ambition!

+

Thus, your feedback is greatly appreciated, as this first version will still be buggy and missing many features. Please send all your thoughts, concerns, feature requests, etc. to contact@daita.tech or simply reply to this e-mail. Please be assured that all your feedback will find its way into our product backlog.

+

All our services are currently free of charge - so you can go wild! Try it now here.

+

Cheers,

+

The DAITA Team

+ """.format( + email_name + ) + message_email_text = """ + Hi, + {} has invited you to explore DAITA's recently launched data augmentation platform https://app.daita.tech. + Building a platform that machine learning engineers and data scientists really love is truly hard. But that's our ultimate ambition! + Thus, your feedback is greatly appreciated, as this first version will still be buggy and missing many features. Please send all your thoughts, concerns, feature requests, etc. to contact@daita.tech or simply reply to this e-mail. Please be assured that all your feedback will find its way into our product backlog. + All our services are currently free of charge - so you can go wild! Try it now here https://app.daita.tech. + Cheers, + The DAITA Team + """.format( + email_name + ) + try: + response = client.send_email( + Destination={ + "ToAddresses": [destination_email], + }, + Message={ + "Body": { + "Html": { + "Charset": "UTF-8", + "Data": message_email, + }, + "Text": { + "Charset": "UTF-8", + "Data": message_email_text, + }, + }, + "Subject": { + "Charset": "UTF-8", + "Data": "You've been invited to try DAITA's data augmentation platform for free!", + }, + }, + Source="DAITA Team ", + ) + except ClientError as e: + print(e) + return convert_response( + {"error": True, "success": False, "message": e, "data": None} + ) + return convert_response( + { + "error": False, + "success": True, + "message": "Email sent! Message ID: {}".format(response["MessageId"]), + "data": None, + } + ) diff --git a/daita-app/core-service/functions/handlers/send-mail/send-email-identity-id/hdler_send_email_to_identityid.py b/daita-app/core-service/functions/handlers/send-mail/send-email-identity-id/hdler_send_email_to_identityid.py new file mode 100644 index 0000000..f3e5a94 --- /dev/null +++ b/daita-app/core-service/functions/handlers/send-mail/send-email-identity-id/hdler_send_email_to_identityid.py @@ -0,0 +1,105 @@ +import boto3 +import json +import os +import uuid + +from config import * +from response import * +from error_messages import * +from utils import convert_current_date_to_iso8601, aws_get_identity_id, get_num_prj +import const + + +from lambda_base_class import LambdaBaseClass +from boto3.dynamodb.conditions import Key, Attr + +table = boto3.client('dynamodb') +cognito_client = boto3.client('cognito-idp') + +class SendEmailIdentityIDClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + + self.identity_id = body["identity_id"] + self.message_email = body["message_email"] + self.message_email_text = body["message_email_text"] + + def _check_input_value(self): + return + + def get_mail_User(self, identity_id): + resq = table.scan(TableName=self.env.TABLE_USER, + FilterExpression='#id = :id', + ExpressionAttributeNames= + { + '#id':'identity_id' + } , + ExpressionAttributeValues={ + ':id':{'S':identity_id} + }) + + userInfo = resq['Items'] + + print("User info: ", userInfo) + if len(userInfo) > 0: + ID_User = userInfo[0]['ID']['S'] + response = cognito_client.list_users(UserPoolId = self.env.USER_POOL_ID, + AttributesToGet = ['email'], + Filter=f'sub=\"{ID_User}\"' + ) + print("response list cognito user: \n", response) + if len(response['Users']) > 0: + user_cognito = response['Users'][0] + mail = user_cognito['Attributes'][0]['Value'] + return mail + + return None + + def send_mail(self, mail, message_email, message_email_text): + client = boto3.client("ses") + print("send email to: ", mail) + response = client.send_email( + Destination={ + "ToAddresses": [mail], + }, + Message={ + "Body": { + "Html": { + "Charset": "UTF-8", + "Data": message_email, + }, + "Text": { + "Charset": "UTF-8", + "Data": message_email_text, + }, + }, + "Subject": { + "Charset": "UTF-8", + "Data": "Your AI detection results are ready", + }, + }, + Source="DAITA Team ", + ) + + return + + def handle(self, event, context): + print(event) + ### parse body + self.parser(event) + + email = self.get_mail_User(self.identity_id) + self.send_mail(email, self.message_email, self.message_email_text) + + return + + +@error_response +def lambda_handler(event, context): + + return SendEmailIdentityIDClass().handle(event, context) \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/thumbnail/divide_batch/app.py b/daita-app/core-service/functions/handlers/thumbnail/divide_batch/app.py new file mode 100644 index 0000000..9f55b54 --- /dev/null +++ b/daita-app/core-service/functions/handlers/thumbnail/divide_batch/app.py @@ -0,0 +1,31 @@ +import boto3 +from error_messages import * +from response import * +from config import * +from boto3.dynamodb.conditions import Key, Attr +from lambda_base_class import LambdaBaseClass +from itertools import chain, islice + + +def batcher(iterable, size): + iterator = iter(iterable) + for first in iterator: + yield list(chain([first], islice(iterator, size - 1))) + + +class divdeThumbnailsImageCls(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + def convert_to_image_batches(self, data): + def generator(): + yield from data + return batcher(generator(), 50) + + def handle(self, event, context): + res = {"batches":[it for it in self.convert_to_image_batches(event['detail']['body'])]} + return res + + +@error_response +def lambda_handler(event, context): + return divdeThumbnailsImageCls().handle(event, context) diff --git a/daita-app/core-service/functions/handlers/thumbnail/resize_image/app.py b/daita-app/core-service/functions/handlers/thumbnail/resize_image/app.py new file mode 100644 index 0000000..3e307aa --- /dev/null +++ b/daita-app/core-service/functions/handlers/thumbnail/resize_image/app.py @@ -0,0 +1,110 @@ +import os +import re +import boto3 +import PIL +import threading +from queue import Queue +from error_messages import * +from response import * +from config import * +from PIL import Image +from io import BytesIO +from lambda_base_class import LambdaBaseClass +from thumbnail import Thumbnail + + +class DynamoDBNewImageUpdated(object): + def __init__(self,info) -> None: + self.project_ID = info['project_id'] + self.filename = info['filename'] + self.thumbnail = None + self.s3_url = info['s3_urls'] + self.table_name = info['table'] + +class ResizeImageCls(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + self.s3 = boto3.client('s3') + self.table = boto3.resource('dynamodb') + + def split(self, uri): + if not 's3' in uri[:2]: + temp = uri.split('/') + bucket = temp[0] + filename = '/'.join([temp[i] for i in range(1, len(temp))]) + else: + match = re.match(r's3:\/\/(.+?)\/(.+)', uri) + bucket = match.group(1) + filename = match.group(2) + return bucket, filename + + def get_image_from_s3(self, url): + bucket, filename = self.split(url) + obj_body = self.s3.get_object(Bucket=bucket, Key=filename) + img = Image.open(BytesIO(obj_body["Body"].read())) + img = img.resize((int(img.size[1]*0.4),int(img.size[0]*0.4)), PIL.Image.ANTIALIAS) + buffer = BytesIO() + img.convert('RGB').save(buffer, 'JPEG') + buffer.seek(0) + return buffer + + def upload_s3(self, image, ext, bucket, filename): + self.s3.put_object(Bucket=bucket, Key=filename, Body=image) + s3_key = f's3://{bucket}/{filename}' + return s3_key + + def update_thumbnail(self, item : DynamoDBNewImageUpdated): + response = self.table.Table(item.table_name).update_item( + Key={ + 'project_id': item.project_ID, + 'filename': item.filename, + }, + ExpressionAttributeNames={ + '#thumbnail': 'thumbnail', + }, + ExpressionAttributeValues={ + ':thumbnail': item.thumbnail, + }, + UpdateExpression='SET #thumbnail = :thumbnail' + ) + print(f'Response : {response}') + + def resize_image(self, queue): + while True: + item = queue.get() + try: + ext = os.path.splitext(os.path.basename(item.s3_url))[-1] + image = self.get_image_from_s3(item.s3_url) + bucket, filename = self.split(item.s3_url) + filenameSlice = filename.split('/') + filenameSlice.insert(len(filenameSlice)-1,'thumbnail') + newfilename = '/'.join(filenameSlice) + item.thumbnail= self.upload_s3(image=image, ext=ext,bucket=bucket,filename=newfilename) + except Exception as e: + print(f'ERROR :{e}') + queue.task_done() + return + # print(f'Logs Check {item.thumbnail}') + self.update_thumbnail(item) + queue.task_done() + + def handle(self, event, context): + print(f'Log Debug {event}') + listRecord = [] + for it in event: + listRecord.append(Thumbnail(it)) + enclosure_queue = Queue() + for _ in range(5): + worker = threading.Thread( + target=self.resize_image, args=(enclosure_queue,)) + worker.daemon = True + worker.start() + for item in listRecord: + enclosure_queue.put(item) + enclosure_queue.join() + return {"message": "Trigger Successfully"} + + +@error_response +def lambda_handler(event, context): + return ResizeImageCls().handle(event, context) diff --git a/daita-app/core-service/functions/handlers/thumbnail/resize_image/requirements.txt b/daita-app/core-service/functions/handlers/thumbnail/resize_image/requirements.txt new file mode 100644 index 0000000..5873a22 --- /dev/null +++ b/daita-app/core-service/functions/handlers/thumbnail/resize_image/requirements.txt @@ -0,0 +1 @@ +Pillow \ No newline at end of file diff --git a/daita-app/core-service/functions/handlers/thumbnail/trigger_thumbnail/__init__.py b/daita-app/core-service/functions/handlers/thumbnail/trigger_thumbnail/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/daita-app/core-service/functions/handlers/thumbnail/trigger_thumbnail/app.py b/daita-app/core-service/functions/handlers/thumbnail/trigger_thumbnail/app.py new file mode 100644 index 0000000..faf0ac8 --- /dev/null +++ b/daita-app/core-service/functions/handlers/thumbnail/trigger_thumbnail/app.py @@ -0,0 +1,47 @@ +import os +import json +import boto3 +from error_messages import * +from response import * +from config import * +from lambda_base_class import LambdaBaseClass + +class CreateThumbnailCls(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + self.client_events = boto3.client('events') + + def handle(self,event,context): + records = event['Records'] + listRecord = [] + print(f'logs :{records}') + for record in records: + if record['eventName'] == 'INSERT': + tempItem = { + 'project_id': record['dynamodb']['Keys']['project_id']['S'], + 'filename' : record['dynamodb']['Keys']['filename']['S'], + 'table': record['eventSourceARN'].split(':')[5].split('/')[1], + 's3_urls':record['dynamodb']['NewImage']['s3_key']['S'] + } + listRecord.append(tempItem) + print(listRecord) + if len(listRecord) == 0 : + print("Nothing image need to create thumbnail") + return {"message":"ok"} + response = self.client_events.put_events( + Entries=[ + { + 'Source': 'source.events', + 'DetailType': 'lambda.event', + 'Detail': json.dumps({'body':listRecord}), + 'EventBusName': os.environ["EVENT_BUS_NAME"] + }, + ] + ) + print(f'Log: {response}') + return {"message":"ok"} + + +@error_response +def lambda_handler(event, context): + return CreateThumbnailCls().handle(event=event,context=context) \ No newline at end of file diff --git a/daita-app/core-service/statemachine/thumbnail_step_function.asl.yaml b/daita-app/core-service/statemachine/thumbnail_step_function.asl.yaml new file mode 100644 index 0000000..918077f --- /dev/null +++ b/daita-app/core-service/statemachine/thumbnail_step_function.asl.yaml @@ -0,0 +1,35 @@ +StartAt: DivideBatch +States: + DivideBatch: + Type: Task + Resource: 'arn:aws:states:::lambda:invoke' + InputPath: $ + OutputPath: $.Payload + Parameters: + FunctionName: '${DivideBatchImagesThumbnailFunction}' + Payload.$: $ + Next: ResizeImageMap + ResizeImageMap: + Type: Map + MaxConcurrency: 5 + InputPath: $ + ItemsPath: $.batches + Iterator: + StartAt: ResizeImage + States: + ResizeImage: + Type: Task + Resource: 'arn:aws:states:::lambda:invoke' + Parameters: + FunctionName: '${ResizeImageThumbnailFunction}' + Payload.$: $ + Retry: + - ErrorEquals: + - Lambda.TooManyRequestsException + - Lambda.ServiceException + IntervalSeconds: 3 + MaxAttempts: 7 + BackoffRate: 2 + End: true + End: true +TimeoutSeconds: 1800 diff --git a/daita-app/dataflow-service/compress-download-app/statemachine/compress_download.asl.yml b/daita-app/dataflow-service/compress-download-app/statemachine/compress_download.asl.yml index 5ce6873..15061f3 100644 --- a/daita-app/dataflow-service/compress-download-app/statemachine/compress_download.asl.yml +++ b/daita-app/dataflow-service/compress-download-app/statemachine/compress_download.asl.yml @@ -82,8 +82,7 @@ States: NetworkConfiguration: AwsvpcConfiguration: AssignPublicIp: ENABLED # Needed when pulling public Docker image - Subnets: - - ${SubnetId} + Subnets: !Join [ ${SubnetIds} ] Overrides: ContainerOverrides: - diff --git a/daita-app/dataflow-service/compress-download-app/template.yaml b/daita-app/dataflow-service/compress-download-app/template.yaml index 6c5b322..7163fce 100644 --- a/daita-app/dataflow-service/compress-download-app/template.yaml +++ b/daita-app/dataflow-service/compress-download-app/template.yaml @@ -8,12 +8,12 @@ Parameters: Type: String ApplicationPara: Type: String + SecurityGroupIds: - Type: String - Default: 'sg-af50cbde' - SubnetId: - Type: String - Default: subnet-31ff5b5a + Type: CommaDelimitedList + SubnetIds: + Type: CommaDelimitedList + EFSFileSystemId: Type: String Default: fs-0199771f2dfe97ace @@ -26,15 +26,12 @@ Parameters: Type: String TableMethodsName: Type: String - TableDataOriginal: + TableDataOriginalName: Type: String - Default: data_original - TableDataAugment: + TableDataAugmentName: Type: String - Default: data_augment - TableDataPreprocess: + TableDataPreprocessName: Type: String - Default: data_preprocess VPCEndpointForS3: Type: String VPCEndpointForDynamoDB: @@ -49,6 +46,8 @@ Parameters: Default: 'noreply@daita.tech' CommonCodeLayerName: Type: String + Mode: + Type: String # CompressedByLambdaMaxinumNrOfFiles: # Type: Number # Default: 100 @@ -63,7 +62,9 @@ Globals: - x86_64 Layers: - !Ref CommonCodeLayerName - + Environment: + Variables: + MODE: !Ref Mode Resources: #================ ROLES ===================================================== @@ -107,7 +108,7 @@ Resources: DefinitionSubstitutions: CompressDownloadTaskCluster: !GetAtt CompressDownloadTaskCluster.Arn CompressDownloadTask: !Ref CompressDownloadTask - SubnetId: !Ref SubnetId + SubnetIds: !Join [',', !Ref SubnetIds] SendCompletedMailFunction: !GetAtt SendCompletedMailFunction.Arn DivideDownloadKeysFunction: !GetAtt DivideDownloadKeysFunction.Arn DownloadFilesFunction: !GetAtt DownloadFilesFunction.Arn @@ -196,9 +197,9 @@ Resources: Resource: "*" Environment: Variables: - TableDataOriginal: !Ref TableDataOriginal - TableDataAugment: !Ref TableDataAugment - TableDataPreprocess: !Ref TableDataPreprocess + TableDataOriginal: !Ref TableDataOriginalName + TableDataAugment: !Ref TableDataAugmentName + TableDataPreprocess: !Ref TableDataPreprocessName CHUNK_SIZE: 8 DownloadFilesFunction: @@ -207,10 +208,8 @@ Resources: CodeUri: functions/download_files Timeout: 120 VpcConfig: - SecurityGroupIds: - - !Ref SecurityGroupIds - SubnetIds: - - !Ref SubnetId + SecurityGroupIds: !Split [',', !Join [',', !Ref SecurityGroupIds]] + SubnetIds: !Split [',', !Join [',', !Ref SubnetIds]] FileSystemConfigs: - Arn: !GetAtt EFSAccessPoint.Arn LocalMountPath: !Ref EFSMountPath @@ -228,10 +227,8 @@ Resources: CodeUri: functions/compress Timeout: 600 VpcConfig: - SecurityGroupIds: - - !Ref SecurityGroupIds - SubnetIds: - - !Ref SubnetId + SecurityGroupIds: !Split [',', !Join [',', !Ref SecurityGroupIds]] + SubnetIds: !Split [',', !Join [',', !Ref SubnetIds]] FileSystemConfigs: - Arn: !GetAtt EFSAccessPoint.Arn LocalMountPath: !Ref EFSMountPath @@ -281,15 +278,15 @@ Resources: Properties: RequiresCompatibilities: - "FARGATE" - ExecutionRoleArn: arn:aws:iam::366577564432:role/ecsTaskExecutionRole #TODO:create this in template - TaskRoleArn: arn:aws:iam::366577564432:role/DecompressTaskRole #TODO:create this in template + ExecutionRoleArn: !Sub "arn:aws:iam::${AWS::AccountId}:role/ecsTaskExecutionRole" #TODO:create this in template + TaskRoleArn: !Sub "arn:aws:iam::${AWS::AccountId}:role/DecompressTaskRole" #TODO:create this in template Cpu: 256 Memory: 512 NetworkMode: awsvpc ContainerDefinitions: - Name: "compress-download" - Image: !Sub "366577564432.dkr.ecr.${AWS::Region}.amazonaws.com/${CompressDownloadEcrRepositoryName}:latest" + Image: !Sub "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${CompressDownloadEcrRepositoryName}:latest" Cpu: 256 Memory: 512 Environment: diff --git a/daita-app/dataflow-service/decompress-upload-app/statemachine/decompress_file.asl.yml b/daita-app/dataflow-service/decompress-upload-app/statemachine/decompress_file.asl.yml index 8f1829e..91f3a5e 100644 --- a/daita-app/dataflow-service/decompress-upload-app/statemachine/decompress_file.asl.yml +++ b/daita-app/dataflow-service/decompress-upload-app/statemachine/decompress_file.asl.yml @@ -9,8 +9,7 @@ States: NetworkConfiguration: AwsvpcConfiguration: AssignPublicIp: ENABLED # Needed when pulling public Docker image - Subnets: - - ${SubnetId} + Subnets: !Join [ ${SubnetIds} ] Overrides: ContainerOverrides: - @@ -26,6 +25,11 @@ States: Value.$: $.identity_id TaskDefinition: ${DecompressTask} ResultPath: null # Forward previous step input as this step output + Catch: + - ErrorEquals: + - States.ALL + Next: CatchErrorTaskDynamoDB + ResultPath: null Next: DivideDecompressChunksFunction DivideDecompressChunksFunction: @@ -78,7 +82,7 @@ States: Resource: arn:aws:states:::lambda:invoke InputPath: $.Payload Parameters: - FunctionName: "${ProjectUploadUpdateFunction}" + FunctionName: "${FuncProjectUploadUpdate}" Payload: body.$: $.body Next: PostUploadFunction diff --git a/daita-app/dataflow-service/decompress-upload-app/template.yaml b/daita-app/dataflow-service/decompress-upload-app/template.yaml index c7d9197..3447275 100644 --- a/daita-app/dataflow-service/decompress-upload-app/template.yaml +++ b/daita-app/dataflow-service/decompress-upload-app/template.yaml @@ -8,12 +8,12 @@ Parameters: Type: String ApplicationPara: Type: String + SecurityGroupIds: - Type: String - Default: 'sg-af50cbde' - SubnetId: - Type: String - Default: subnet-31ff5b5a + Type: CommaDelimitedList + SubnetIds: + Type: CommaDelimitedList + EFSFileSystemId: Type: String Default: fs-0199771f2dfe97ace @@ -41,17 +41,30 @@ Parameters: UploadS3FileChunk: Type: String Default: 8 + Mode: + Type: String + TableDataOriginalName: + Type: String + TableDataAugmentName: + Type: String + TableDataPreprocessName: + Type: String + FuncProjectUploadUpdate: + Type: String + Globals: Function: - Timeout: 10 + Timeout: 180 Handler: app.lambda_handler Runtime: python3.8 Architectures: - x86_64 Layers: - !Ref CommonCodeLayerName - + Environment: + Variables: + MODE: !Ref Mode Resources: #================ ROLES ===================================================== @@ -125,12 +138,12 @@ Resources: DefinitionSubstitutions: DecompressTaskCluster: !GetAtt DecompressTaskCluster.Arn DecompressTask: !Ref DecompressTask - SubnetId: !Ref SubnetId + SubnetIds: !Join [',', !Ref SubnetIds] DivideDecompressChunksFunction: !GetAtt DivideDecompressChunksFunction.Arn UploadDecompressedFunction: !GetAtt UploadDecompressedFunction.Arn FinishDecompressTaskFunction: !GetAtt FinishDecompressTaskFunction.Arn DecompressTaskTable: !Ref TableDataFlowTaskName - ProjectUploadUpdateFunction: staging-project-upload-update + FuncProjectUploadUpdate: !Ref FuncProjectUploadUpdate #staging-project-upload-update PostUploadFunction: !Ref PostUploadFunction TableDataFlowTaskName: !Ref TableDataFlowTaskName Logging: @@ -149,7 +162,7 @@ Resources: - LambdaInvokePolicy: FunctionName: !Ref FinishDecompressTaskFunction - LambdaInvokePolicy: - FunctionName: staging-project-upload-update + FunctionName: !Ref FuncProjectUploadUpdate - LambdaInvokePolicy: FunctionName: !Ref PostUploadFunction - DynamoDBCrudPolicy: @@ -197,10 +210,8 @@ Resources: Properties: CodeUri: functions/divide_decompressed_chunks VpcConfig: - SecurityGroupIds: - - !Ref SecurityGroupIds - SubnetIds: - - !Ref SubnetId + SecurityGroupIds: !Split [',', !Join [',', !Ref SecurityGroupIds]] + SubnetIds: !Split [',', !Join [',', !Ref SubnetIds]] FileSystemConfigs: - Arn: !GetAtt EFSAccessPoint.Arn LocalMountPath: !Ref EFSMountPath @@ -235,10 +246,8 @@ Resources: CodeUri: functions/upload_decompressed Timeout: 30 VpcConfig: - SecurityGroupIds: - - !Ref SecurityGroupIds - SubnetIds: - - !Ref SubnetId + SecurityGroupIds: !Split [',', !Join [',', !Ref SecurityGroupIds]] + SubnetIds: !Split [',', !Join [',', !Ref SubnetIds]] FileSystemConfigs: - Arn: !GetAtt EFSAccessPoint.Arn LocalMountPath: !Ref EFSMountPath @@ -277,10 +286,8 @@ Resources: - elasticfilesystem:ClientMount Resource: "*" VpcConfig: - SecurityGroupIds: - - !Ref SecurityGroupIds - SubnetIds: - - !Ref SubnetId + SecurityGroupIds: !Split [',', !Join [',', !Ref SecurityGroupIds]] + SubnetIds: !Split [',', !Join [',', !Ref SubnetIds]] FileSystemConfigs: - Arn: !GetAtt EFSAccessPoint.Arn LocalMountPath: !Ref EFSMountPath @@ -302,15 +309,15 @@ Resources: Properties: RequiresCompatibilities: - "FARGATE" - ExecutionRoleArn: arn:aws:iam::366577564432:role/ecsTaskExecutionRole #TODO:create this in template - TaskRoleArn: arn:aws:iam::366577564432:role/DecompressTaskRole #TODO:create this in template + ExecutionRoleArn: !Sub "arn:aws:iam::${AWS::AccountId}:role/ecsTaskExecutionRole" #TODO:create this in template + TaskRoleArn: !Sub "arn:aws:iam::${AWS::AccountId}:role/DecompressTaskRole" #TODO:create this in template Cpu: 256 Memory: 512 NetworkMode: awsvpc ContainerDefinitions: - Name: "decompress-file" - Image: !Sub "366577564432.dkr.ecr.${AWS::Region}.amazonaws.com/${DecompressEcrRepositoryName}:latest" + Image: !Sub "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${DecompressEcrRepositoryName}:latest" Cpu: 256 Memory: 512 Environment: diff --git a/daita-app/dataflow-service/template.yaml b/daita-app/dataflow-service/template.yaml index 84e7ae3..17ea81f 100644 --- a/daita-app/dataflow-service/template.yaml +++ b/daita-app/dataflow-service/template.yaml @@ -8,15 +8,16 @@ Parameters: Type: String ApplicationPara: Type: String - SecurityGroupIds: - Type: String - Default: 'sg-af50cbde' + VpcId: Type: String Default: vpc-53239e38 - SubnetId: + + SecurityGroupIds: ### could not pass string over the nested app, so use string here + Type: String + SubnetIds: Type: String - Default: subnet-31ff5b5a + EFSFileSystemId: Type: String Default: fs-0199771f2dfe97ace @@ -44,16 +45,28 @@ Parameters: Default: 'noreply@daita.tech' CommonCodeLayerName: Type: String - + Mode: + Type: String + TableDataOriginalName: + Type: String + TableDataAugmentName: + Type: String + TableDataPreprocessName: + Type: String + + FuncProjectUploadUpdate: + Type: String Globals: Function: - Timeout: 10 + Timeout: 180 Handler: app.lambda_handler Runtime: python3.8 Architectures: - x86_64 - + Environment: + Variables: + MODE: !Ref Mode Resources: #================ ROLES ===================================================== @@ -144,7 +157,7 @@ Resources: StagePara: !Ref StagePara ApplicationPara: !Ref ApplicationPara SecurityGroupIds: !Ref SecurityGroupIds - SubnetId: !Ref SubnetId + SubnetIds: !Ref SubnetIds EFSMountPath: !Ref EFSMountPath EFSAccessPointRootPath: !Ref EFSAccessPointRootPath S3BucketName: !Ref S3BucketName @@ -153,7 +166,13 @@ Resources: VPCEndpointForS3: !Ref VPCEndpointForS3 VPCEndpointForDynamoDB: !Ref VPCEndpointForDynamoDB CommonCodeLayerName: !Ref CommonCodeLayerName - + EFSFileSystemId: !Ref EFSFileSystemId + Mode: !Ref Mode + TableDataOriginalName: !Ref TableDataOriginalName + TableDataAugmentName: !Ref TableDataAugmentName + TableDataPreprocessName: !Ref TableDataPreprocessName + FuncProjectUploadUpdate: !Ref FuncProjectUploadUpdate + CompressDownloadApp: Type: AWS::Serverless::Application Properties: @@ -162,7 +181,7 @@ Resources: StagePara: !Ref StagePara ApplicationPara: !Ref ApplicationPara SecurityGroupIds: !Ref SecurityGroupIds - SubnetId: !Ref SubnetId + SubnetIds: !Ref SubnetIds EFSMountPath: !Ref EFSMountPath EFSAccessPointRootPath: !Ref EFSAccessPointRootPath S3BucketName: !Ref S3BucketName @@ -173,8 +192,11 @@ Resources: VPCEndpointForDynamoDB: !Ref VPCEndpointForDynamoDB SESIdentityName: noreply@daita.tech CommonCodeLayerName: !Ref CommonCodeLayerName - - + EFSFileSystemId: !Ref EFSFileSystemId + Mode: !Ref Mode + TableDataOriginalName: !Ref TableDataOriginalName + TableDataAugmentName: !Ref TableDataAugmentName + TableDataPreprocessName: !Ref TableDataPreprocessName Outputs: DecompressFileStateMachineArn: diff --git a/daita-app/db-service/db_template.yaml b/daita-app/db-service/db_template.yaml index f8baf3a..762e5a9 100644 --- a/daita-app/db-service/db_template.yaml +++ b/daita-app/db-service/db_template.yaml @@ -6,12 +6,164 @@ Description: > Parameters: StagePara: Type: String + ApplicationPara: + Type: String IndexTaskProjectIDTaskIDName: Type: String Default: index-projectid-taskid + IndexDataName: + Type: String + Default: index-created-sorted Resources: #================ DYNAMODB ================================================== + + ###==== For Project ======== + TableConstPrebuildDataset: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-const-prebuild_dataset" + AttributeDefinitions: + - + AttributeName: name + AttributeType: S + KeySchema: + - + AttributeName: name + KeyType: HASH + BillingMode: PAY_PER_REQUEST + + ProjectDB: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-${ApplicationPara}-project" + AttributeDefinitions: + - + AttributeName: identity_id + AttributeType: S + - + AttributeName: project_name + AttributeType: S + KeySchema: + - + AttributeName: identity_id + KeyType: HASH + - + AttributeName: project_name + KeyType: RANGE + BillingMode: PAY_PER_REQUEST + + ProjectSummaryDB: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-${ApplicationPara}-prj-sum-all" + AttributeDefinitions: + - + AttributeName: project_id + AttributeType: S + - + AttributeName: type + AttributeType: S + KeySchema: + - + AttributeName: project_id + KeyType: HASH + - + AttributeName: type + KeyType: RANGE + BillingMode: PAY_PER_REQUEST + + DataOriginalDB: + Type: AWS::DynamoDB::Table + Properties: + AttributeDefinitions: + - + AttributeName: project_id + AttributeType: S + - + AttributeName: filename + AttributeType: S + KeySchema: + - + AttributeName: project_id + KeyType: HASH + - + AttributeName: filename + KeyType: RANGE + TableName: !Sub "${StagePara}-${ApplicationPara}-data-original" + GlobalSecondaryIndexes: + - + IndexName: !Ref IndexDataName + KeySchema: + - + AttributeName: project_id + KeyType: HASH + Projection: + ProjectionType: ALL + BillingMode: PAY_PER_REQUEST + StreamSpecification: + StreamViewType: NEW_IMAGE + + DataAugmentDB: + Type: AWS::DynamoDB::Table + Properties: + AttributeDefinitions: + - + AttributeName: project_id + AttributeType: S + - + AttributeName: filename + AttributeType: S + KeySchema: + - + AttributeName: project_id + KeyType: HASH + - + AttributeName: filename + KeyType: RANGE + TableName: !Sub "${StagePara}-${ApplicationPara}-data-augment" + GlobalSecondaryIndexes: + - + IndexName: !Ref IndexDataName + KeySchema: + - + AttributeName: project_id + KeyType: HASH + Projection: + ProjectionType: ALL + BillingMode: PAY_PER_REQUEST + StreamSpecification: + StreamViewType: NEW_IMAGE + DataPreprocessDB: + Type: AWS::DynamoDB::Table + Properties: + AttributeDefinitions: + - + AttributeName: project_id + AttributeType: S + - + AttributeName: filename + AttributeType: S + KeySchema: + - + AttributeName: project_id + KeyType: HASH + - + AttributeName: filename + KeyType: RANGE + TableName: !Sub "${StagePara}-${ApplicationPara}-data-preprocess" + GlobalSecondaryIndexes: + - + IndexName: !Ref IndexDataName + KeySchema: + - + AttributeName: project_id + KeyType: HASH + Projection: + ProjectionType: ALL + BillingMode: PAY_PER_REQUEST + StreamSpecification: + StreamViewType: NEW_IMAGE GenerateTaskDB: Type: AWS::DynamoDB::Table Properties: @@ -194,19 +346,162 @@ Resources: Projection: ProjectionType: ALL BillingMode: PAY_PER_REQUEST + TableGenerateDaitaUploadToken: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-generate-daita-upload-token" + AttributeDefinitions: + - + AttributeName: identity_id + AttributeType: S + - + AttributeName: token + AttributeType: S + KeySchema: + - + AttributeName: identity_id + KeyType: HASH + - + AttributeName: token + KeyType: RANGE + BillingMode: PAY_PER_REQUEST + TimeToLiveSpecification: + AttributeName: time_to_live + Enabled: True + GlobalSecondaryIndexes: + - + IndexName: !Sub "${StagePara}-generate-daita-upload-token-1" + KeySchema: + - + AttributeName: token + KeyType: HASH + Projection: + ProjectionType: ALL + TableConfirmCodeAuth: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-confirm-code" + AttributeDefinitions: + - + AttributeName: user + AttributeType: S + - + AttributeName: code + AttributeType: S + KeySchema: + - + AttributeName: user + KeyType: HASH + - + AttributeName: code + KeyType: RANGE + BillingMode: PAY_PER_REQUEST + TimeToLiveSpecification: + AttributeName: time_to_live + Enabled: True + TableUser: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-User" + AttributeDefinitions: + - + AttributeName: ID + AttributeType: S + - + AttributeName: username + AttributeType: S + KeySchema: + - + AttributeName: ID + KeyType: HASH + - + AttributeName: username + KeyType: RANGE + BillingMode: PAY_PER_REQUEST + TableFeedback: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-feedback" + AttributeDefinitions: + - + AttributeName: ID + AttributeType: S + KeySchema: + - + AttributeName: ID + KeyType: HASH + BillingMode: PAY_PER_REQUEST + + + TableTask: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-Task" + AttributeDefinitions: + - + AttributeName: identity_id + AttributeType: S + - + AttributeName: task_id + AttributeType: S + KeySchema: + - + AttributeName: identity_id + KeyType: HASH + - + AttributeName: task_id + KeyType: RANGE + BillingMode: PAY_PER_REQUEST + TableProjectDelete: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-Project-Delete" + AttributeDefinitions: + - + AttributeName: identity_id + AttributeType: S + - + AttributeName: project_name + AttributeType: S + KeySchema: + - + AttributeName: identity_id + KeyType: HASH + - + AttributeName: project_name + KeyType: RANGE + BillingMode: PAY_PER_REQUEST + TableEventUser: + Type: AWS::DynamoDB::Table + Properties: + TableName: !Sub "${StagePara}-EventUser" + AttributeDefinitions: + - + AttributeName: event_ID + AttributeType: S + - + AttributeName: type + AttributeType: S + KeySchema: + - + AttributeName: event_ID + KeyType: HASH + - + AttributeName: type + KeyType: RANGE + BillingMode: PAY_PER_REQUEST Outputs: TableGenerateTaskName: Description: "Name of table generate task" Value: !Ref GenerateTaskDB TableProjectsName: Description: "Name of table projects" - Value: projects + Value: !Ref ProjectDB TableProjectSumName: - Value: prj_sum_all - TableMethodsName: - Description: "Name of table methods" - Value: methods + Description: "Name of table projects" + Value: !Ref ProjectSummaryDB + TableHealthCheckTasksName: Description: "Name of table health check tasks" Value: !Ref HealthCheckTaskDB @@ -221,17 +516,46 @@ Outputs: Value: !Ref TableReferenceImageInfo TableDataAugmentName: Description: "Name of table data augment" - Value: data_augment + Value: !Ref DataAugmentDB TableDataOriginalName: Description: "Name of table data original" - Value: data_original + Value: !Ref DataOriginalDB TableDataPreprocessName: Description: "Name of table data preprocess" - Value: data_preprocess + Value: !Ref DataPreprocessDB TableDataFlowTaskName: Description: "Content task of download and upload service" Value: !Ref DataFlowTaskTable - TableDownloadTaskName: - Value: down_tasks + IndexTaskProjectIDTaskIDName: Value: !Ref IndexTaskProjectIDTaskIDName + + + TableGenerateDaitaUploadToken: + Value: !Ref TableGenerateDaitaUploadToken + TableConfirmCodeAuth: + Value: !Ref TableConfirmCodeAuth + TableUser: + Value: !Ref TableUser + TableFeedback: + Value: !Ref TableFeedback + + ### For const DB, we will create by code + TableConstPrebuildDatasetName: + Value: !Ref TableConstPrebuildDataset + TableLsEc2Name: + Value: ec2 + TableMethodsName: + Description: "Name of table methods" + Value: methods + + TableTask: + Value: !Ref TableTask + TableEventUser: + Value: !Ref TableEventUser + StreamTableDataOriginalName: + Value: !GetAtt DataOriginalDB.StreamArn + StreamTableDataPreprocessName: + Value: !GetAtt DataPreprocessDB.StreamArn + StreamTableDataAugmentName: + Value: !GetAtt DataAugmentDB.StreamArn \ No newline at end of file diff --git a/daita-app/ecs-ai-caller-app/ecs_segementation.yaml b/daita-app/ecs-ai-caller-app/ecs_segementation.yaml new file mode 100644 index 0000000..179ac0e --- /dev/null +++ b/daita-app/ecs-ai-caller-app/ecs_segementation.yaml @@ -0,0 +1,208 @@ +AWSTemplateFormatVersion: '2010-09-09' + +Parameters: + ECSAMI: + Type: String + Default: ami-09cb4bc2dcc083845 + + InstanceType: + Type: String + Default: c4.xlarge + AllowedValues: [t2.micro, t2.small, t2.medium, t2.large, m3.medium, m3.large, + m3.xlarge, m3.2xlarge, m4.large, m4.xlarge, m4.2xlarge, m4.4xlarge, m4.10xlarge, + c4.large, c4.xlarge, c4.2xlarge, c4.4xlarge, c4.8xlarge, c3.large, c3.xlarge, + c3.2xlarge, c3.4xlarge, c3.8xlarge, r3.large, r3.xlarge, r3.2xlarge, r3.4xlarge, + r3.8xlarge, i2.xlarge, i2.2xlarge, i2.4xlarge, i2.8xlarge] + ConstraintDescription: Please choose a valid instance type. + + PublicSubnetOne: + Type: String + PublicSubnetTwo: + Type: String + ContainerSecurityGroup: + Type: String + + ### currently, set directly in template + # DesiredCapacity: + # Type: Number + # Default: '1' + # Description: Number of EC2 instances to launch in your ECS cluster. + + StagePara: + Type: String + ApplicationPara: + Type: String + + EC2Role: + Type: String + + MaxSize: + Type: Number + Default: '6' + + ExecuteArn: + Type: String + + PreprocessingImageUrl: + Type: String + AugmentationImageUrl: + Type: String + + TaskRole: + Type: String + + ContainerPath: + Type: String + +Resources: + + ECSCluster: + Type: AWS::ECS::Cluster + Properties: + ClusterName: !Sub "${StagePara}-${ApplicationPara}-ECS-AICaller-Cluster" + + ECSAICallerAutoScalingGroup: + Type: AWS::AutoScaling::AutoScalingGroup + DependsOn: ECSCluster + Properties: + AutoScalingGroupName: !Sub "${StagePara}-${ApplicationPara}-AICaller-ECS" + # HealthCheckGracePeriod: 60 + # HealthCheckType: EC2 + VPCZoneIdentifier: + - !Ref PublicSubnetOne + - !Ref PublicSubnetTwo + LaunchConfigurationName: !Ref 'ContainerInstances' + NewInstancesProtectedFromScaleIn: false #if true this block scale in termination completely + MinSize: '0' + MaxSize: !Ref 'MaxSize' + DesiredCapacity: 0 + + EC2InstanceProfile: + Type: AWS::IAM::InstanceProfile + Properties: + Path: / + Roles: [!Ref 'EC2Role'] + + ContainerInstances: + Type: AWS::AutoScaling::LaunchConfiguration + Properties: + ImageId: !Ref ECSAMI + SecurityGroups: [!Ref ContainerSecurityGroup] + InstanceType: !Ref InstanceType + IamInstanceProfile: !Ref 'EC2InstanceProfile' + UserData: + Fn::Base64: !Sub | + #!/bin/bash -xe + echo ECS_CLUSTER=${ECSCluster} >> /etc/ecs/ecs.config + yum install -y aws-cfn-bootstrap + /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource ECSAICallerAutoScalingGroup --region ${AWS::Region} + + ECSTaskLogGroup: + Type: AWS::Logs::LogGroup + Properties: + LogGroupName: !Sub "/aws/vendedlogs/states/${StagePara}-MyECSAICallerTaskLogGroup-${AWS::StackName}" + RetentionInDays: 7 + + TaskAIPreprocessingDefinition: + Type: AWS::ECS::TaskDefinition + Properties: + ExecutionRoleArn: !Ref ExecuteArn + TaskRoleArn: !Ref TaskRole + NetworkMode: awsvpc + ContainerDefinitions: + - + Name: !Sub "${StagePara}-${ApplicationPara}-ecs-preprocessing" + Image: !Ref PreprocessingImageUrl + Cpu: 4092 + Memory: 4092 + LogConfiguration: + LogDriver: 'awslogs' + Options: + awslogs-group: !Ref ECSTaskLogGroup + awslogs-region: !Ref 'AWS::Region' + awslogs-create-group: true + awslogs-stream-prefix: !Ref 'ApplicationPara' + # MountPoints: + # - + # SourceVolume: "my-vol" + # ContainerPath: !Ref ContainerPath + # Volumes: + # - + # EFSVolumeConfiguration: + # AuthorizationConfig: + # AccessPointId: !Ref EFSAccessPoint + # FilesystemId: !Ref EFSFileSystemId + # TransitEncryption: ENABLED # enable this so maybe we don't need to config a access point https://docs.aws.amazon.com/pt_br/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-authorizationconfig.html + # Name: "my-vol" + + TaskAIAugmentationDefinition: + Type: AWS::ECS::TaskDefinition + Properties: + ExecutionRoleArn: !Ref ExecuteArn + TaskRoleArn: !Ref TaskRole + NetworkMode: awsvpc + ContainerDefinitions: + - + Name: !Sub "${StagePara}-${ApplicationPara}-ecs-augmentation" + Image: !Ref AugmentationImageUrl + Cpu: 4092 + Memory: 4092 + LogConfiguration: + LogDriver: 'awslogs' + Options: + awslogs-group: !Ref ECSTaskLogGroup + awslogs-region: !Ref 'AWS::Region' + awslogs-create-group: true + awslogs-stream-prefix: !Ref 'ApplicationPara' + # MountPoints: + # - + # SourceVolume: "my-vol" + # ContainerPath: !Ref ContainerPath + # Volumes: + # - + # EFSVolumeConfiguration: + # AuthorizationConfig: + # AccessPointId: !Ref EFSAccessPoint + # FilesystemId: !Ref EFSFileSystemId + # TransitEncryption: ENABLED # enable this so maybe we don't need to config a access point https://docs.aws.amazon.com/pt_br/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-authorizationconfig.html + # Name: "my-vol" + + ###________ CAPACITY CONFIG FOR CLUSTER ______________ + ###### doc: https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-target-tracking.html + CapacityProvider: + Type: AWS::ECS::CapacityProvider + DependsOn: ECSCluster + Properties: + Name: !Sub "${StagePara}-${ApplicationPara}-capacity-provider-ai-caller" + AutoScalingGroupProvider: + AutoScalingGroupArn: !Ref ECSAICallerAutoScalingGroup + ManagedScaling: + # InstanceWarmupPeriod: 300 + MaximumScalingStepSize: 2 + MinimumScalingStepSize: 1 + Status: ENABLED + TargetCapacity: 100 + # ManagedTerminationProtection: ENABLED + + ClusterCapacityProviderAssociation: + Type: AWS::ECS::ClusterCapacityProviderAssociations + Properties: + Cluster: !Ref ECSCluster + CapacityProviders: + - !Ref CapacityProvider + DefaultCapacityProviderStrategy: + - CapacityProvider: !Ref CapacityProvider + Weight: 1 + +Outputs: + + TaskAIPreprocessingDefinition: + Value: !Ref TaskAIPreprocessingDefinition + TaskAIAugmentationDefinition: + Value: !Ref TaskAIAugmentationDefinition + + ECSCluster: + Value: !Ref ECSCluster + + ContainerName: + Value: !Sub "${StagePara}-${ApplicationPara}-ecs-segmentations" \ No newline at end of file diff --git a/daita-app/ecs-ai-caller-app/role.yaml b/daita-app/ecs-ai-caller-app/role.yaml new file mode 100644 index 0000000..cff4d8e --- /dev/null +++ b/daita-app/ecs-ai-caller-app/role.yaml @@ -0,0 +1,137 @@ +AWSTemplateFormatVersion: '2010-09-09' + +Resources: + + ECSTaskExecutionRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Statement: + - Effect: Allow + Principal: + Service: [ecs-tasks.amazonaws.com] + AWS: !Sub "arn:aws:iam::${AWS::AccountId}:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS" + Action: ['sts:AssumeRole'] + Path: / + Policies: + - PolicyName: AmazonECSTaskExecutionRolePolicy + PolicyDocument: + Statement: + - Effect: Allow + Action: + - 'ecr:GetAuthorizationToken' + - 'ecr:BatchCheckLayerAvailability' + - 'ecr:GetDownloadUrlForLayer' + - 'ecr:BatchGetImage' + - 'logs:*' + - 'iam:PassRole' + Resource: '*' + + ECSRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Statement: + - Effect: Allow + Principal: + Service: [ + "ecs.amazonaws.com", + "lambda.amazonaws.com", + "ecs-tasks.amazonaws.com" + ] + AWS: !Sub "arn:aws:iam::${AWS::AccountId}:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS" + Action: ['sts:AssumeRole'] + Path: / + Policies: + - PolicyName: ecs-service + PolicyDocument: + Statement: + - Effect: Allow + Action: + + - 'ec2:AttachNetworkInterface' + - 'ec2:CreateNetworkInterface' + - 'ec2:CreateNetworkInterfacePermission' + - 'ec2:DeleteNetworkInterface' + - 'ec2:DeleteNetworkInterfacePermission' + - 'ec2:Describe*' + - 'ec2:DetachNetworkInterface' + + - 'elasticloadbalancing:DeregisterInstancesFromLoadBalancer' + - 'elasticloadbalancing:DeregisterTargets' + - 'elasticloadbalancing:Describe*' + - 'elasticloadbalancing:RegisterInstancesWithLoadBalancer' + - 'elasticloadbalancing:RegisterTargets' + + - s3:* + - s3-object-lambda:* + - logs:* + + - 'iam:PassRole' + Resource: '*' + + EC2Role: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Statement: + - Effect: Allow + Principal: + Service: [ec2.amazonaws.com] + Action: ['sts:AssumeRole'] + Path: / + ManagedPolicyArns: + - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role + Policies: + - PolicyName: ecs-service + PolicyDocument: + Statement: + - Effect: Allow + Action: + - 'ecs:CreateCluster' + - 'ecs:DeregisterContainerInstance' + - 'ecs:DiscoverPollEndpoint' + - 'ecs:Poll' + - 'ecs:RegisterContainerInstance' + - 'ecs:StartTelemetrySession' + - 'ecs:Submit*' + - 'logs:CreateLogStream' + - 'logs:PutLogEvents' + - 'ecr:GetAuthorizationToken' + - 'ecr:BatchGetImage' + - 'ecr:GetDownloadUrlForLayer' + + - 'iam:PassRole' + Resource: '*' + + AutoscalingRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Statement: + - Effect: Allow + Principal: + Service: [application-autoscaling.amazonaws.com] + Action: ['sts:AssumeRole'] + Path: / + Policies: + - PolicyName: Service-Autoscaling + PolicyDocument: + Statement: + - Effect: Allow + Action: + - 'application-autoscaling:*' + - 'cloudwatch:*' + - 'ecs:DescribeServices' + - 'ecs:UpdateService' + Resource: '*' + +Outputs: + EC2Role: + Value: !Ref EC2Role + + ECSTask: + Value: !Ref ECSTaskExecutionRole + + ECSRole: + Value: !Ref ECSRole diff --git a/daita-app/ecs-ai-caller-app/statemachine/ecs_task.asl.yaml b/daita-app/ecs-ai-caller-app/statemachine/ecs_task.asl.yaml new file mode 100644 index 0000000..22e070b --- /dev/null +++ b/daita-app/ecs-ai-caller-app/statemachine/ecs_task.asl.yaml @@ -0,0 +1,48 @@ +StartAt: DownloadS3toEFSFunction +States: + + DownloadS3toEFSFunction: + Type: Task + Resource: 'arn:aws:states:::lambda:invoke' + InputPath: $ + Parameters: + FunctionName: '${DownloadS3toEFSFunction}' + Payload.$: $ + Next: RunECSTask + + RunECSTask: + Type: Task + Resource: arn:aws:states:::ecs:runTask.sync + Parameters: + Cluster: ${AITaskECSClusterArn} + TaskDefinition: ${AITaskDefinitionArn} + NetworkConfiguration: + AwsvpcConfiguration: + Subnets: !Join [ ${Subnets} ] + SecurityGroups: !Join [ ${SecurityGroupIds} ] + # AssignPublicIp: ENABLED + Overrides: + ContainerOverrides: + - Name.$ : "$.Payload.Name" + Command.$: "$.Payload.Command" + Retry: + - ErrorEquals: + - RetriableCallerServiceError + - ErrorEquals: + - States.ALL + IntervalSeconds: 3 + MaxAttempts: 5 + BackoffRate: 1 + Next: UploadImage + + UploadImage: + Type: Task + Resource: 'arn:aws:states:::lambda:invoke' + InputPath: $ + Parameters: + FunctionName: '${UploadImage}' + Payload: + input_folder.$ : $$.Execution.Input.input_folder + records.$: $$.Execution.Input.records + End: true +TimeoutSeconds: 6000 \ No newline at end of file diff --git a/daita-app/ecs-ai-caller-app/statemachine/functions/hdler_download_image_to_efs.py b/daita-app/ecs-ai-caller-app/statemachine/functions/hdler_download_image_to_efs.py new file mode 100644 index 0000000..477d555 --- /dev/null +++ b/daita-app/ecs-ai-caller-app/statemachine/functions/hdler_download_image_to_efs.py @@ -0,0 +1,65 @@ +import boto3 +import re +import os +import json + +from response import * + +from lambda_base_class import LambdaBaseClass +s3 = boto3.client('s3') +def save_image_to_efs(s3_key : str,folder : str): + print(s3_key) + bucket, filename = split(s3_key) + basename = os.path.join(folder,os.path.basename(filename)) + new_image = os.path.join(str(os.environ['EFSPATH']),basename) + s3.download_file(bucket,filename,new_image) + + return basename + +def parse_json_ecs_segmentation(records,input_folder): + newJson = {'images':[]} + + for id ,record in enumerate(records): + newJson['images'].append( + { + "image_path":os.path.join(str(os.environ['CONTAINER_MOUNT']),save_image_to_efs('s3://{}'.format(record['s3_key']),input_folder)), + "image_id": id + }, + ) + + fileinput = os.path.join(input_folder,'input.json') + with open(os.path.join(str(os.environ['EFSPATH']), fileinput),'w') as f: + json.dump(newJson,f) + return fileinput + +def split(uri): + if not 's3' in uri[:2]: + temp = uri.split('/') + bucket = temp[0] + filename = '/'.join([temp[i] for i in range(1,len(temp))]) + else: + match = re.match(r's3:\/\/(.+?)\/(.+)', uri) + bucket = match.group(1) + filename = match.group(2) + return bucket, filename + +class DownloadImageEFSClass(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def handle(self,event, context): + input_folder = event['input_folder'] + os.makedirs(os.path.join(str(os.environ['EFSPATH']),input_folder),exist_ok=True) + inputJson = parse_json_ecs_segmentation(event['records'],input_folder) + inputJsonContainerVolume = os.path.join(str(os.environ['CONTAINER_MOUNT']),inputJson) + output_folder = os.path.join(input_folder,'output') + os.makedirs(os.path.join(str(os.environ['EFSPATH']),output_folder),exist_ok=True) + return { + "output_directory": output_folder, + "Name": os.environ['CONTAINER_NAME'], + "Command": ["--input_json_path",inputJsonContainerVolume,"--output_folder",os.path.join(str(os.environ['CONTAINER_MOUNT']),output_folder)] + } + +@error_response +def lambda_handler(event, context): + return DownloadImageEFSClass().handle(event=event,context=context) \ No newline at end of file diff --git a/daita-app/ecs-ai-caller-app/statemachine/functions/hdler_updoad_image.py b/daita-app/ecs-ai-caller-app/statemachine/functions/hdler_updoad_image.py new file mode 100644 index 0000000..2d97f3c --- /dev/null +++ b/daita-app/ecs-ai-caller-app/statemachine/functions/hdler_updoad_image.py @@ -0,0 +1,60 @@ +import boto3 +import re +import os +import json + +from response import * + +from lambda_base_class import LambdaBaseClass + +table = boto3.client('dynamodb') +s3 = boto3.client('s3') + +def update_s3_gen(project_id, filename, s3_key_gen): + response = table.update_item( + TableName=os.environ["TABLE"], + Key={ + 'project_id': { + 'S': project_id + }, + 'filename': { + 'S': filename + } + }, + ExpressionAttributeNames={ + '#gen': 's3_key_segm', + }, + ExpressionAttributeValues={ + ':gen':{ + 'S': s3_key_gen + } + }, + UpdateExpression='SET #gen = :gen', + ) + print(f'Response ',response) + +def upload_segmentation_s3(data,s3_key): + dirfilename = os.path.dirname(s3_key) + dirfilename = dirfilename.replace('raw_data','clone_project') + basename = os.path.splitext(os.path.basename(s3_key))[0] + '_segment.json' + filename = os.path.join(dirfilename, basename) + bucket = filename.split('/')[0] + key = '/'.join(filename.split('/')[1:]) + s3.put_object( + Body=data, + Bucket=bucket , + Key= key + ) + return filename + +@error_response +def lambda_handler(event, context): + output_folder = os.path.join(event['input_folder'],'output') + output_folder = os.path.join(os.environ['EFSPATH'],output_folder) + + for index , it in enumerate(event['records']): + with open(os.path.join(output_folder,str(index)+'.json'),'r') as f: + # data = json.load(f) + s3_key = upload_segmentation_s3(f.read(),s3_key=it['s3_key']) + update_s3_gen(it['project_id'], it['filename'],s3_key) + return {} \ No newline at end of file diff --git a/daita-app/ecs-ai-caller-app/template_ecs_ai_caller.yaml b/daita-app/ecs-ai-caller-app/template_ecs_ai_caller.yaml new file mode 100644 index 0000000..9b37801 --- /dev/null +++ b/daita-app/ecs-ai-caller-app/template_ecs_ai_caller.yaml @@ -0,0 +1,68 @@ +AWSTemplateFormatVersion: '2010-09-09' +Transform: AWS::Serverless-2016-10-31 +Description: > + SAM Template for Nested application resources + +## The general rule seems to be to use !Sub for in line substitutions and !ref for stand alone text +Parameters: + + StagePara: + Type: String + ApplicationPara: + Type: String + + ImageAIPreprocessingUrl: + Type: String + Default: 737589818430.dkr.ecr.us-east-2.amazonaws.com/ai-services-repo:preprocessing + ImageAIAugmentationUrl: + Type: String + Default: 737589818430.dkr.ecr.us-east-2.amazonaws.com/ai-services-repo:augmentation + + + + ContainerPath: + Type: String + Default: /app/data + + PublicSubnetOne: + Type: String + PublicSubnetTwo: + Type: String + ContainerSecurityGroup: + Type: String + VPC: + Type: String + +Resources: + + + #================ APPLICATIONS ============================================= + RoleApplication: + Type: AWS::Serverless::Application + Properties: + Location: ./role.yaml + + ECSServiceApplication: + Type: AWS::Serverless::Application + Properties: + Location: ./ecs_segementation.yaml + Parameters: + StagePara: !Ref StagePara + ApplicationPara: !Ref ApplicationPara + EC2Role: !GetAtt RoleApplication.Outputs.EC2Role + ### Network parameters + PublicSubnetOne: !Ref PublicSubnetOne + PublicSubnetTwo: !Ref PublicSubnetTwo + ContainerSecurityGroup: !Ref ContainerSecurityGroup + + ExecuteArn: !GetAtt RoleApplication.Outputs.ECSTask + TaskRole: !GetAtt RoleApplication.Outputs.ECSRole + PreprocessingImageUrl: !Ref ImageAIPreprocessingUrl + AugmentationImageUrl: !Ref ImageAIAugmentationUrl + ContainerPath: !Ref ContainerPath + +Outputs: + TaskAIPreprocessingDefinition: + Value: !GetAtt ECSServiceApplication.Outputs.TaskAIPreprocessingDefinition + TaskAIAugmentationDefinition: + Value: !GetAtt ECSServiceApplication.Outputs.TaskAIAugmentationDefinition \ No newline at end of file diff --git a/daita-app/health-check-service/health_check_service.yaml b/daita-app/health-check-service/health_check_service.yaml index ae6594e..6c34d01 100644 --- a/daita-app/health-check-service/health_check_service.yaml +++ b/daita-app/health-check-service/health_check_service.yaml @@ -28,7 +28,8 @@ Parameters: Type: String ApplicationPara: Type: String - + Mode: + Type: String Globals: Function: Timeout: 800 @@ -41,7 +42,7 @@ Globals: TABLE_DATA_AUGMENT: !Ref TableDataAugmentName TABLE_DATA_ORIGINAL: !Ref TableDataOriginalName TABLE_DATA_PREPROCESS: !Ref TableDataPreprocessName - + MODE: !Ref Mode Resources: #================ ROLES ===================================================== diff --git a/daita-app/project-service/functions/api_handler/hdler_project_upload_update.py b/daita-app/project-service/functions/api_handler/hdler_project_upload_update.py new file mode 100644 index 0000000..c60ba86 --- /dev/null +++ b/daita-app/project-service/functions/api_handler/hdler_project_upload_update.py @@ -0,0 +1,238 @@ +from lambda_base_class import LambdaBaseClass +import json +import boto3 +import hashlib +import hmac +import base64 +import os +from utils import convert_response, aws_get_identity_id, convert_current_date_to_iso8601 + +MAX_NUMBER_ITEM_PUT = 500 +MAX_NUM_IMAGES_IN_ORIGINAL = 500 + +USERPOOLID = os.environ['COGNITO_USER_POOL'] +CLIENTPOOLID = os.environ['COGNITO_CLIENT_ID'] +IDENTITY_POOL = os.environ['IDENTITY_POOL'] + + +def create_single_put_request(dict_value): + dict_re = { + 'PutRequest': { + 'Item': { + } + } + } + for key, value in dict_value.items(): + dict_re['PutRequest']['Item'][key] = { + value[0]: value[1] + } + return dict_re + + +def lambda_handler(event, context): + return ProjectUploadUpdateCls().handle(event, context) + + +class ProjectUploadUpdateCls(LambdaBaseClass): + def __init__(self) -> None: + super().__init__() + + def parser(self, body): + id_token = body["id_token"] + # self.client_events = boto3.client('events') + + self.identity_id = aws_get_identity_id( + id_token, USERPOOLID, IDENTITY_POOL) + + # get request data + self.project_id = body['project_id'] + self.project_name = body['project_name'] + self.ls_object_info = body['ls_object_info'] + self.type_method = body.get('type_method', 'ORIGINAL') + + # check quantiy of items + if len(self.ls_object_info) > MAX_NUMBER_ITEM_PUT: + raise Exception( + f'The number of items is over {MAX_NUMBER_ITEM_PUT}') + if len(self.ls_object_info) == 0: + raise Exception('The number of items must not empty') + + # create the batch request from input data and summary the information + self.ls_batch_request = [] + self.total_size = 0 + self.count = 0 + self.total_process = len(self.ls_object_info) + for object in self.ls_object_info: + # update summary information + size_old = object.get('size_old', 0) + self.total_size += (object['size']-size_old) + if size_old <= 0: + self.count += 1 + + self.is_ori = object['is_ori'] + request = { + 'project_id': self.project_id, # partition key + 's3_key': object['s3_key'], # sort_key + 'filename': object['filename'], + # we use function get it mean that this field is optional in body + 'hash': object.get('hash', ''), + # size must be in Byte unit + 'size': object['size'], + 'is_ori': object['is_ori'], + 'type_method': self.type_method, + 'gen_id': object.get('gen_id', ''), # id of generation method + 'created_date': convert_current_date_to_iso8601() + } + self.ls_batch_request.append(request) + + def handle(self, event, context): + if type(event['body']) is str: + body = json.loads(event['body']) + else: + body = event['body'] + self.parser(body) + + # check number images in original must smaller than a limitation + try: + db_resource = boto3.resource("dynamodb") + if self.is_ori: + table = db_resource.Table(os.environ["TABLE_PROJECT_SUMMARY"]) + # get current data in original + response = table.get_item( + Key={ + "project_id": self.project_id, + "type": "ORIGINAL" + } + ) + print('response get summary: ', response) + if response.get('Item'): + current_num_data = response['Item'].get( + 'num_exist_data', 0) + thumbnail_key = response['Item'].get('thu_key', None) + else: + current_num_data = 0 + thumbnail_key = None + + num_final = current_num_data + self.total_process + # move check in upload_check + # if num_final > MAX_NUM_IMAGES_IN_ORIGINAL: + # raise Exception(f'The total number in original data must smaller than {MAX_NUM_IMAGES_IN_ORIGINAL}! Current you already had {current_num_data}, but you tried to add {total_process} data.') + else: + num_final = 0 + + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + # update data to DB + # we use batch_write, it means that if key are existed in tables => overwrite + db_client = boto3.client('dynamodb') + db_resource = boto3.resource('dynamodb') + type_ = None + try: + if self.is_ori: + table = os.environ["T_DATA_ORI"] + type_ = os.environ["T_DATA_ORI"] + else: + if self.type_method == 'PREPROCESS': + table = os.environ["T_DATA_PREPROCESS"] + type_ = os.environ["T_DATA_PREPROCESS"] + elif self.type_method == 'AUGMENT': + table = os.environ["T_DATA_AUGMENT"] + type_ = os.environ["T_DATA_AUGMENT"] + else: + raise (Exception('Missing type_method parameters!')) + + table_pr = db_resource.Table(table) + with table_pr.batch_writer() as batch: + for item in self.ls_batch_request: + batch.put_item(Item=item) + + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + # self.client_events.put_events( + # Entries=[ + # { + # 'Source': 'source.events', + # 'DetailType': 'lambda.event', + # 'Detail': json.dumps({ + # 'project_id':self.project_id, + # 'type':type + # }), + # 'EventBusName': os.environ["THUMBNAIL_EVENT_BUS"] + # }, + # ] + # ) + # update summary information + try: + if self.is_ori and thumbnail_key is None: + # update thumbnail key to project + table = db_resource.Table(os.environ["TABLE_PROJECT_SUMMARY"]) + response = table.update_item( + Key={ + 'project_id': self.project_id, + 'type': self.type_method, + }, + ExpressionAttributeNames={ + '#SI': 'total_size', + '#COU': 'count', + '#NE': 'num_exist_data', + '#TK': 'thu_key', + '#TN': 'thu_name' + }, + ExpressionAttributeValues={ + ':si': self.total_size, + ':cou': self.count, + ':ne': num_final, + ':tk': self.ls_batch_request[0]['s3_key'], + ':tn': self.ls_batch_request[0]['filename'] + }, + UpdateExpression='SET #NE = :ne, #TK = :tk, #TN = :tn ADD #SI :si, #COU :cou', + ) + else: + response = db_client.update_item( + TableName=os.environ["TABLE_PROJECT_SUMMARY"], + Key={ + 'project_id': { + 'S': self.project_id + }, + 'type': { + 'S': self.type_method + } + }, + ExpressionAttributeNames={ + '#SI': 'total_size', + '#COU': 'count', + '#NE': 'num_exist_data' + }, + ExpressionAttributeValues={ + ':si': { + 'N': str(self.total_size) + }, + ':cou': { + 'N': str(self.count) + }, + ':ne': { + 'N': str(num_final) + } + }, + UpdateExpression='SET #NE = :ne ADD #SI :si, #COU :cou', + ) + except Exception as e: + print('Error: ', repr(e)) + return convert_response({"error": True, + "success": False, + "message": repr(e), + "data": None}) + + return convert_response({'data': {}, + "error": False, + "success": True, + "message": None}) diff --git a/daita-app/project-service/functions/hdler_move_s3_data.py b/daita-app/project-service/functions/hdler_move_s3_data.py new file mode 100644 index 0000000..535e701 --- /dev/null +++ b/daita-app/project-service/functions/hdler_move_s3_data.py @@ -0,0 +1,117 @@ +import boto3 +import json +import os +import random + +from config import * +from response import * +from error_messages import * +from identity_check import * + +from system_parameter_store import SystemParameterStore +from lambda_base_class import LambdaBaseClass +from models.data_model import DataModel, DataItem +from models.task_model import TaskModel + +from utils import create_unique_id, get_bucket_key_from_s3_uri, split_ls_into_batch + +class MoveS3DataClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.client_events = boto3.client('events') + self.const = SystemParameterStore() + self.s3 = boto3.client('s3') + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + self.s3_prefix_prebuild_store = body["s3_prefix_prebuild"] + self.s3_prefix_created_store = body["s3_prefix_create"] + self.project_id = body["project_id"] + self.project_name = body["project_name"] + self.identity_id = body["identity_id"] + self.bucket_name = body["bucket_name"] + self.number_random = body["number_random"] + + def _check_input_value(self): + pass + + def move_data_s3(self, source, target, bucket_name, number_random = -1): + ls_info = [] + #list all data in s3 + s3 = boto3.resource('s3') + bucket = s3.Bucket(bucket_name) + + ls_task_params = [] + for obj in bucket.objects.filter(Prefix=source): + if obj.key.endswith('/'): + continue + + old_source = { 'Bucket': bucket_name, + 'Key': obj.key} + # replace the prefix + new_prefix = target.replace(f"{bucket_name}/", "") + new_key = f'{new_prefix}/{obj.key.replace(source, "")}' + + task_params = (old_source, new_key, obj.size) + ls_task_params.append(task_params) + + if number_random>len(ls_task_params): + number_random = len(ls_task_params) + + ### shuffle the list parameters + random.shuffle(ls_task_params) + + for params in ls_task_params[:number_random]: + old_source = params[0] + new_key = params[1] + size = params[2] + + ### copy data to new s3 folder + s3.meta.client.copy(old_source, bucket_name, new_key) + + ## add to list info + ls_info.append((new_key.split('/')[-1], f"{bucket_name}/{new_key}", size)) + + return ls_info + + def handle(self, event, context): + + ### parse body + self.parser(event) + + # move data in s3 + ls_info = self.move_data_s3( + self.s3_prefix_prebuild_store, self.s3_prefix_created_store, self.bucket_name, number_random = self.number_random) + + if len(ls_info)>0: + bucket, folder = get_bucket_key_from_s3_uri(self.s3_prefix_created_store) + s3_key_path = os.path.join(folder, f"create_sample/RI_{create_unique_id()}.json") + self.s3.put_object( + Body=json.dumps(ls_info), + Bucket= bucket, + Key= s3_key_path + ) + else: + bucket = None + s3_key_path = None + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={ + "bucket_name": self.bucket_name, + "identity_id": self.identity_id, + "project_id": self.project_id, + "project_name": self.project_name, + "s3_key_path": s3_key_path + }, + is_in_stepfunction=True + ) + +def lambda_handler(event, context): + + return MoveS3DataClass().handle(event, context) + + \ No newline at end of file diff --git a/daita-app/project-service/functions/hdler_update_input_data.py b/daita-app/project-service/functions/hdler_update_input_data.py new file mode 100644 index 0000000..19c08f7 --- /dev/null +++ b/daita-app/project-service/functions/hdler_update_input_data.py @@ -0,0 +1,101 @@ +import boto3 +import json +import os + +from config import * +from response import * +from error_messages import * +from identity_check import * + +from system_parameter_store import SystemParameterStore +from lambda_base_class import LambdaBaseClass +from models.data_model import DataModel, DataItem +from models.task_model import TaskModel +from utils import get_bucket_key_from_s3_uri, split_ls_into_batch, convert_current_date_to_iso8601 + +db_resource = boto3.resource('dynamodb') + +class MoveUpdateDataClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.client_events = boto3.client('events') + self.const = SystemParameterStore() + self.s3 = boto3.client('s3') + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + self.identity_id = body[KEY_NAME_IDENTITY_ID] + self.project_id = body[KEY_NAME_PROJECT_ID] + self.bucket_name = body["bucket_name"] + self.s3_key_path = body["s3_key_path"] + self.project_name = body["project_name"] + + def _check_input_value(self): + pass + + def handle(self, event, context): + + ### parse body + self.parser(event) + + resultS3 = self.s3.get_object(Bucket=self.bucket_name, Key=self.s3_key_path) + ls_info = json.loads(resultS3["Body"].read().decode()) + + # update to DB + # create the batch request from input data and summary the information + ls_item_request = [] + total_size = 0 + count = 0 + for object in ls_info: + # update summary information + size_old = 0 + total_size += (object[2]-size_old) + if size_old <= 0: + count += 1 + + is_ori = True + type_method = VALUE_TYPE_DATA_ORIGINAL + item_request = { + 'project_id': self.project_id, # partition key + 's3_key': object[1], # sort_key + 'filename': object[0], + 'hash': '', # we use function get it mean that this field is optional in body + 'size': object[2], # size must be in Byte unit + 'is_ori': True, + 'type_method': type_method, + 'gen_id': '', # id of generation method + 'created_date': convert_current_date_to_iso8601() + } + ls_item_request.append(item_request) + + try: + table = db_resource.Table(os.environ["T_DATA_ORI"]) + with table.batch_writer() as batch: + for item in ls_item_request: + batch.put_item(Item=item) + except Exception as e: + print('Error: ', repr(e)) + raise Exception(repr(e)) + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={ + "total_size": total_size, + "count": count, + "thu_key": ls_item_request[0]['s3_key'], + "thu_name": ls_item_request[0]['filename'], + "project_id": self.project_id, + "project_name": self.project_name, + "identity_id": self.identity_id + }, + is_in_stepfunction=True + ) + +def lambda_handler(event, context): + + return MoveUpdateDataClass().handle(event, context) + + \ No newline at end of file diff --git a/daita-app/project-service/functions/hdler_update_sumary_db.py b/daita-app/project-service/functions/hdler_update_sumary_db.py new file mode 100644 index 0000000..176c681 --- /dev/null +++ b/daita-app/project-service/functions/hdler_update_sumary_db.py @@ -0,0 +1,114 @@ +import boto3 +import json +import os + +from config import * +from response import * +from error_messages import * +from identity_check import * + +from system_parameter_store import SystemParameterStore +from lambda_base_class import LambdaBaseClass +from models.project_model import ProjectModel, ProjectItem +from models.project_sum_model import ProjectSumModel +from utils import get_bucket_key_from_s3_uri, split_ls_into_batch + +db_client = boto3.client('dynamodb') +db_resource = boto3.resource('dynamodb') + +class UpdateSummaryDBClass(LambdaBaseClass): + + def __init__(self) -> None: + super().__init__() + self.client_events = boto3.client('events') + self.const = SystemParameterStore() + self.s3 = boto3.client('s3') + self.project_sum_model = ProjectSumModel(os.environ["TABLE_PROJECT_SUMMARY"]) + + @LambdaBaseClass.parse_body + def parser(self, body): + self.logger.debug(f"body in main_parser: {body}") + self.identity_id = body[KEY_NAME_IDENTITY_ID] + self.project_id = body[KEY_NAME_PROJECT_ID] + self.project_name = body["project_name"] + self.total_size = body["total_size"] + self.count = body["count"] + self.thu_key = body["thu_key"] + self.thu_name = body["thu_name"] + + def _check_input_value(self): + pass + + def handle(self, event, context): + + ### parse body + self.parser(event) + + # update summary information + try: + response = db_client.update_item( + TableName=os.environ["TABLE_PROJECT_SUMMARY"], + Key={ + 'project_id': { + 'S': self.project_id + }, + 'type': { + 'S': VALUE_TYPE_DATA_ORIGINAL + } + }, + ExpressionAttributeNames={ + '#SI': 'total_size', + '#COU': 'count', + '#TK': 'thu_key', + '#TN': 'thu_name' + }, + ExpressionAttributeValues={ + ':si': { + 'N': str(self.total_size) + }, + ':cou': { + 'N': str(self.count) + }, + ':tk': { + 'S': self.thu_key + }, + ':tn': { + 'S': self.thu_name + } + }, + UpdateExpression='SET #TK = :tk, #TN = :tn ADD #SI :si, #COU :cou', + ) + print('response_summary: ', response) + except Exception as e: + print('Error: ', repr(e)) + raise Exception(repr(e)) + + # update generate status + try: + table = db_resource.Table(os.environ['TABLE_PROJECT']) + response = table.update_item( + Key={ + 'identity_id': self.identity_id, + 'project_name': self.project_name, + }, + ExpressionAttributeValues={ + ':st': VALUE_STATUS_CREATE_SAMPLE_PRJ_FINISH, + }, + UpdateExpression='SET gen_status = :st' + ) + except Exception as e: + print('Error: ', repr(e)) + raise Exception(repr(e)) + + return generate_response( + message="OK", + status_code=HTTPStatus.OK, + data={}, + is_in_stepfunction=True + ) + +def lambda_handler(event, context): + + return UpdateSummaryDBClass().handle(event, context) + + \ No newline at end of file diff --git a/daita-app/project-service/project_service_template.yaml b/daita-app/project-service/project_service_template.yaml new file mode 100644 index 0000000..6b10dc5 --- /dev/null +++ b/daita-app/project-service/project_service_template.yaml @@ -0,0 +1,182 @@ +AWSTemplateFormatVersion: '2010-09-09' +Transform: AWS::Serverless-2016-10-31 +Description: > + daita-reference-image-service + + Sample SAM Template for daita-reference-image-service + + +Parameters: + minimumLogLevel: + Type: String + Default: DEBUG + StagePara: + Type: String + CommonCodeLayerName: + Type: String + Mode: + Type: String + ApplicationPara: + Type: String + LambdaRole: + Type: String + + TableDataOriginalName: + Type: String + TableDataPreprocess: + Type: String + TableDataAugment: + Type: String + + TableProjectSumName: + Type: String + TableProjectsName: + Type: String + + CognitoUserPool: + Type: String + CognitoUserPoolClient: + Type: String + CognitoIdentityPoolId: + Type: String + ThumbnailEventBus: + Type: String +Globals: + Function: + Timeout: 800 + Runtime: python3.8 + Architectures: + - x86_64 + Environment: + Variables: + STAGE: !Ref StagePara + LOGGING: !Ref minimumLogLevel + MODE: !Ref Mode + TABLE_PROJECT_SUMMARY: !Ref TableProjectSumName + TABLE_PROJECT: !Ref TableProjectsName + T_DATA_ORI: !Ref TableDataOriginalName + COGNITO_USER_POOL: !Ref CognitoUserPool + COGNITO_CLIENT_ID: !Ref CognitoUserPoolClient + IDENTITY_POOL: !Ref CognitoIdentityPoolId + THUMBNAIL_EVENT_BUS: !Ref ThumbnailEventBus + T_DATA_PREPROCESS: !Ref TableDataPreprocess + T_DATA_AUGMENT: !Ref TableDataAugment +Resources: + + #================ LAMBDA FUNCTIONS ========================================== + + ProjectUploadUpdateFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: functions/api_handler + Handler: hdler_project_upload_update.lambda_handler + Role: !Ref LambdaRole + MemorySize: 256 + Layers: + - !Ref CommonCodeLayerName + + FuncMoveS3Data: + Type: AWS::Serverless::Function + Properties: + Timeout: 900 + Handler: hdler_move_s3_data.lambda_handler + CodeUri: functions + Role: !Ref LambdaRole + MemorySize: 256 + Layers: + - !Ref CommonCodeLayerName + + FuncUpdateInputData: + Type: AWS::Serverless::Function + Properties: + Handler: hdler_update_input_data.lambda_handler + CodeUri: functions + Role: !Ref LambdaRole + MemorySize: 256 + Layers: + - !Ref CommonCodeLayerName + + FuncUpdateSumaryDatabase: + Type: AWS::Serverless::Function + Properties: + Handler: hdler_update_sumary_db.lambda_handler + CodeUri: functions + Role: !Ref LambdaRole + MemorySize: 256 + Layers: + - !Ref CommonCodeLayerName + + #================ LOGS FOR STEP FUNCTIONS =================================== + + CreateProjectPrebuildSMLogGroup: + Type: AWS::Logs::LogGroup + Properties: + LogGroupName: !Sub "/aws/vendedlogs/states/${StagePara}-${ApplicationPara}-CreateProjectPrebuild" + RetentionInDays: 7 + + + #================ PROCESSOR STATE MACHINE =================================== + + CreateProjectPrebuildSM: + Type: AWS::Serverless::StateMachine + Properties: + # Type: EXPRESS + Type: STANDARD + Name: !Sub "${StagePara}-${ApplicationPara}-CreateProjectPrebuildSM" + Policies: + - LambdaInvokePolicy: + FunctionName: !Ref FuncMoveS3Data + - LambdaInvokePolicy: + FunctionName: !Ref FuncUpdateInputData + - LambdaInvokePolicy: + FunctionName: !Ref FuncUpdateSumaryDatabase + - Statement: + - Sid: CloudWatchLogsPolicy + Effect: Allow + Action: + - "logs:CreateLogDelivery" + - "logs:GetLogDelivery" + - "logs:UpdateLogDelivery" + - "logs:DeleteLogDelivery" + - "logs:ListLogDeliveries" + - "logs:PutResourcePolicy" + - "logs:DescribeResourcePolicies" + - "logs:DescribeLogGroup" + - "logs:DescribeLogGroups" + Resource: "*" + - Sid: CloudWatchEventsFullAccess + Effect: Allow + Action: + - "events:*" + Resource: "*" + - Sid: IAMPassRoleForCloudWatchEvents + Effect: Allow + Action: + - "iam:PassRole" + Resource: "arn:aws:iam::*:role/AWS_Events_Invoke_Targets" + Tracing: + Enabled: true + DefinitionUri: ./statemachine/sm_create_project_fr_prebuild.asl.yaml + Logging: + Level: ALL + IncludeExecutionData: true + Destinations: + - CloudWatchLogsLogGroup: + LogGroupArn: !GetAtt CreateProjectPrebuildSMLogGroup.Arn + DefinitionSubstitutions: + Arn_FuncMoveS3Data: !GetAtt FuncMoveS3Data.Arn + Arn_FuncUpdateInputData: !GetAtt FuncUpdateInputData.Arn + Arn_FuncUpdateSumaryDatabase: !GetAtt FuncUpdateSumaryDatabase.Arn + +Outputs: + # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function + # Find out more about other implicit resources you can reference within SAM + # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api + CreateProjectPrebuildSMArn: + Description: "ARN of SM" + Value: !GetAtt CreateProjectPrebuildSM.Arn + FuncProjectUploadUpdate: + Value: !Ref ProjectUploadUpdateFunction + FuncProjectUploadUpdateArn: + Value: !GetAtt ProjectUploadUpdateFunction.Arn + diff --git a/daita-app/project-service/statemachine/sm_create_project_fr_prebuild.asl.yaml b/daita-app/project-service/statemachine/sm_create_project_fr_prebuild.asl.yaml new file mode 100644 index 0000000..74f85f2 --- /dev/null +++ b/daita-app/project-service/statemachine/sm_create_project_fr_prebuild.asl.yaml @@ -0,0 +1,49 @@ +StartAt: GetDataTask +States: + GetDataTask: + Type: Task + Resource: 'arn:aws:states:::lambda:invoke' + InputPath: $ + ResultSelector: + body.$: $.Payload.body.data + ResultPath: $ + OutputPath: $ + Parameters: + FunctionName: "${Arn_FuncMoveS3Data}" + Payload: + body.$: $ + Next: MoveUpdateDataMap + Comment: >- + Check the level of parallelism, split requests into chunks and invoke + lamndas + Retry: + - ErrorEquals: + - RetriableCallerServiceError + IntervalSeconds: 1 + MaxAttempts: 2 + BackoffRate: 1 + + MoveUpdateDataMap: + Type: Task + Resource: 'arn:aws:states:::lambda:invoke' + InputPath: $.body + ResultPath: $ + OutputPath: $.Payload + Parameters: + FunctionName: "${Arn_FuncUpdateInputData}" + Payload: + body.$: $ + Next: TaskUpdateSummaryDB + + TaskUpdateSummaryDB: + Type: Task + Resource: 'arn:aws:states:::lambda:invoke' + InputPath: $.body.data + OutputPath: $.Payload.body + Parameters: + FunctionName: "${Arn_FuncUpdateSumaryDatabase}" + Payload: + body.$: $ + End: true + +TimeoutSeconds: 6000 diff --git a/daita-app/core-service/functions/handlers/reference_image/calculate/app.py b/daita-app/reference-image-service/functions/api_handler/calculate/app.py similarity index 96% rename from daita-app/core-service/functions/handlers/reference_image/calculate/app.py rename to daita-app/reference-image-service/functions/api_handler/calculate/app.py index 9b5ac44..c68b529 100644 --- a/daita-app/core-service/functions/handlers/reference_image/calculate/app.py +++ b/daita-app/reference-image-service/functions/api_handler/calculate/app.py @@ -60,7 +60,7 @@ def handle(self, event, context): self.parser(event) ### check identity - identity_id = self.get_identity(self.id_token) + identity_id = self.get_identity(self.id_token, self.env.USER_POOL_ID, self.env.IDENTITY_POOL_ID) ### create taskID and update to DB task_id, process_type = self._create_task(identity_id, self.project_id, self.project_name, self.ls_method_id, self.ls_method_client_choose) diff --git a/daita-app/core-service/functions/handlers/reference_image/get_info/app.py b/daita-app/reference-image-service/functions/api_handler/get_info/app.py similarity index 93% rename from daita-app/core-service/functions/handlers/reference_image/get_info/app.py rename to daita-app/reference-image-service/functions/api_handler/get_info/app.py index c22e3dd..1f064a1 100644 --- a/daita-app/core-service/functions/handlers/reference_image/get_info/app.py +++ b/daita-app/reference-image-service/functions/api_handler/get_info/app.py @@ -36,7 +36,7 @@ def handle(self, event, context): self.parser(event) ### check identity - identity_id = self.get_identity(self.id_token) + identity_id = self.get_identity(self.id_token, self.env.USER_POOL_ID, self.env.IDENTITY_POOL_ID) ### get list info items = self.refer_info_model.get_info_of_project(self.project_id) diff --git a/daita-app/core-service/functions/handlers/reference_image/get_status/app.py b/daita-app/reference-image-service/functions/api_handler/get_status/app.py similarity index 94% rename from daita-app/core-service/functions/handlers/reference_image/get_status/app.py rename to daita-app/reference-image-service/functions/api_handler/get_status/app.py index 700e485..b6ce21c 100644 --- a/daita-app/core-service/functions/handlers/reference_image/get_status/app.py +++ b/daita-app/reference-image-service/functions/api_handler/get_status/app.py @@ -38,7 +38,7 @@ def handle(self, event, context): self.parser(event) ### check identity - identity_id = self.get_identity(self.id_token) + identity_id = self.get_identity(self.id_token, self.env.USER_POOL_ID, self.env.IDENTITY_POOL_ID) ### get status of task task_info = self._get_task_status(identity_id, self.task_id) diff --git a/daita-app/reference-image-service/reference_image_service.yaml b/daita-app/reference-image-service/reference_image_service.yaml index f64ed53..8010b1e 100644 --- a/daita-app/reference-image-service/reference_image_service.yaml +++ b/daita-app/reference-image-service/reference_image_service.yaml @@ -34,7 +34,17 @@ Parameters: Type: String MaxHeightResolutionImage: Type: String + Mode: + Type: String + CognitoIdentityPoolId: + Type: String + CognitoUserPool: + Type: String + CognitoUserPoolClient: + Type: String + + Globals: Function: Timeout: 800 @@ -49,7 +59,10 @@ Globals: BATCHSIZE_REF_IMG: !Ref BatchsizeCalculateReferenceImage MAX_WIDTH_RESOLUTION_IMG: !Ref MaxWidthResolutionImage MAX_HEIGHT_RESOLUTION_IMG: !Ref MaxHeightResolutionImage - + COGNITO_USER_POOL: !Ref CognitoUserPool + COGNITO_CLIENT_ID: !Ref CognitoUserPoolClient + IDENTITY_POOL: !Ref CognitoIdentityPoolId + MODE: !Ref Mode Resources: #================ ROLES ===================================================== @@ -99,6 +112,53 @@ Resources: Id: "ReferenceImageSMTarget" RoleArn: !GetAtt ReferenceImageServiceEventBusRole.Arn + #================= API HAMDLER LAMBDA FUNCTION ========================= + RICalculateFunction: + Type: AWS::Serverless::Function + Properties: + Runtime: python3.8 + Handler: app.lambda_handler + Architectures: + - x86_64 + CodeUri: functions/api_handler/calculate + Role: !Ref LambdaRole + Layers: + - !Ref CommonCodeLayerName + Environment: + Variables: + EVENT_BUS_NAME: !Ref ReferenceImageEventBus + TABLE_REFERENCE_IMAGE_TASK: !Ref TableReferenceImageTasksName + + RIStatusFunction: + Type: AWS::Serverless::Function + Properties: + Runtime: python3.8 + Handler: app.lambda_handler + Architectures: + - x86_64 + CodeUri: functions/api_handler/get_status + Role: !Ref LambdaRole + Layers: + - !Ref CommonCodeLayerName + Environment: + Variables: + TABLE_REFERENCE_IMAGE_TASK: !Ref TableReferenceImageTasksName + + RIInfoFunction: + Type: AWS::Serverless::Function + Properties: + Runtime: python3.8 + Handler: app.lambda_handler + Architectures: + - x86_64 + CodeUri: functions/api_handler/get_info + Role: !Ref LambdaRole + Layers: + - !Ref CommonCodeLayerName + Environment: + Variables: + TABLE_REFERENCE_IMAGE_INFO: !Ref TableReferenceImageInfoName + #================ LAMBDA FUNCTIONS ========================================== GetDataFunction: @@ -214,3 +274,18 @@ Outputs: ReferenceImageEventBusName: Description: "Name of EventBus" Value: !Ref ReferenceImageEventBus + ReferenceImageStateMachineArn: + Value: !GetAtt ReferenceImageStateMachine.Arn + + RICalculateFunction: + Value: !Ref RICalculateFunction + RICalculateFunctionArn: + Value: !GetAtt RICalculateFunction.Arn + RIStatusFunction: + Value: !Ref RIStatusFunction + RIStatusFunctionArn: + Value: !GetAtt RIStatusFunction.Arn + RIInfoFunction: + Value: !Ref RIInfoFunction + RIInfoFunctionArn: + Value: !GetAtt RIInfoFunction.Arn diff --git a/daita-app/samconfig.toml b/daita-app/samconfig.toml index 8cc3c13..8611488 100644 --- a/daita-app/samconfig.toml +++ b/daita-app/samconfig.toml @@ -2,48 +2,26 @@ version = 0.1 [default] [default.deploy] [default.deploy.parameters] -stack_name = "testapp" -s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-pemlzh8np33c" -s3_prefix = "testapp" +stack_name = "devdaitabeapp" +s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-eu58g5l8is1s" +s3_prefix = "devdaitabeapp" region = "us-east-2" confirm_changeset = true capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND" disable_rollback = true -image_repositories = [] -parameter_overrides = "Stage=\"testapp\" Application=\"testapp\" SecurityGroupIds=\"sg-0315a5ecee0dc69fe,sg-0b3b2fcc4dc7686ad,sg-af50cbde,sg-07c27f59bc172f180,sg-0796222bd5149736f\" SubnetIDs=\"subnet-31ff5b5a\" S3BucketName=\"daita-client-data\"" +image_repositories = ["HealthCheckService/CalculateHealthCheckFunction=737589818430.dkr.ecr.us-east-2.amazonaws.com/devdaitabeappb6b54d79/healthcheckservicecalculatehealthcheckfunction6d6be9a4repo", "ReferenceImageService/CalculateReferenceImageFunction=737589818430.dkr.ecr.us-east-2.amazonaws.com/devdaitabeappb6b54d79/referenceimageservicecalculatereferenceimagefunctione6710baarepo"] [dev.deploy.parameters] -stack_name = "dev-daita-app" -s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-pemlzh8np33c" -s3_prefix = "dev-daita-app" -region = "us-east-2" confirm_changeset = true -capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND" +capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" disable_rollback = true -image_repositories = ["HealthCheckService/CalculateHealthCheckFunction=366577564432.dkr.ecr.us-east-2.amazonaws.com/devdaitaapp620f3a3f/healthcheckservicecalculatehealthcheckfunction6d6be9a4repo", "ReferenceImageService/CalculateReferenceImageFunction=366577564432.dkr.ecr.us-east-2.amazonaws.com/devdaitaapp620f3a3f/referenceimageservicecalculatereferenceimagefunctione6710baarepo"] -parameter_overrides = "Stage=\"dev\" Application=\"daita\" SecurityGroupIds=\"sg-0315a5ecee0dc69fe,sg-0b3b2fcc4dc7686ad,sg-af50cbde,sg-07c27f59bc172f180,sg-0796222bd5149736f\" SubnetIDs=\"subnet-31ff5b5a\" S3BucketName=\"daita-client-data\" MaxConcurrencyTasks=\"3\"" +image_repositories = [] [prod.deploy.parameters] -stack_name = "prod-daita-app" -s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-pemlzh8np33c" -s3_prefix = "prod-daita-app" -region = "us-east-2" confirm_changeset = true -capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND" +capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" disable_rollback = true image_repositories = [] -parameter_overrides = "Stage=\"prod\"" - -[dte.deploy.parameters] -stack_name = "dte-daita-app" -s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-pemlzh8np33c" -s3_prefix = "dte-daita-app" -region = "us-east-2" -confirm_changeset = false -capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND" -disable_rollback = true -image_repositories = ["HealthCheckService/CalculateHealthCheckFunction=366577564432.dkr.ecr.us-east-2.amazonaws.com/dtedaitaappef3c8d7e/healthcheckservicecalculatehealthcheckfunction6d6be9a4repo"] -parameter_overrides = "Stage=\"dte\" Application=\"daita\" EFSpath=\"/mnt/efs\" AccessPointARN=\"arn:aws:elasticfilesystem:us-east-2:366577564432:access-point/fsap-0bdf8f0ae44bd5561\" SecurityGroupIds=\"sg-0315a5ecee0dc69fe,sg-0b3b2fcc4dc7686ad,sg-af50cbde,sg-07c27f59bc172f180,sg-0796222bd5149736f\" SubnetIDs=\"subnet-31ff5b5a\" S3BucketName=\"daita-client-data\"" [devbeapp] [devbeapp.deploy] @@ -108,18 +86,6 @@ capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND" parameter_overrides = "Stage=\"devbe\" Application=\"devbe\" SecurityGroupIds=\"sg-0315a5ecee0dc69fe,sg-0b3b2fcc4dc7686ad,sg-af50cbde,sg-07c27f59bc172f180,sg-0796222bd5149736f\" SubnetIDs=\"subnet-31ff5b5a\" S3BucketName=\"daita-client-data\"" image_repositories = [] -[devbele] -[devbele.deploy] -[devbele.deploy.parameters] -stack_name = "devbele" -s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-pemlzh8np33c" -s3_prefix = "devbele" -region = "us-east-2" -confirm_changeset = false -capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND" -parameter_overrides = "Stage=\"devbele\" Application=\"devbele\" SecurityGroupIds=\"sg-0315a5ecee0dc69fe,sg-0b3b2fcc4dc7686ad,sg-af50cbde,sg-07c27f59bc172f180,sg-0796222bd5149736f\" SubnetIDs=\"subnet-31ff5b5a\" S3BucketName=\"daita-client-data\"" -image_repositories = [] - [testdevapp] [testdevapp.deploy] [testdevapp.deploy.parameters] @@ -145,3 +111,37 @@ capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND" disable_rollback = true parameter_overrides = "Stage=\"testapp1\" Application=\"testapp1\" SecurityGroupIds=\"sg-0315a5ecee0dc69fe,sg-0b3b2fcc4dc7686ad,sg-af50cbde,sg-07c27f59bc172f180,sg-0796222bd5149736f\" SubnetIDs=\"subnet-31ff5b5a\" S3BucketName=\"daita-client-data\"" image_repositories = ["HealthCheckService/CalculateHealthCheckFunction=366577564432.dkr.ecr.us-east-2.amazonaws.com/testapp14acbe16e/healthcheckservicecalculatehealthcheckfunction6d6be9a4repo", "ReferenceImageService/CalculateReferenceImageFunction=366577564432.dkr.ecr.us-east-2.amazonaws.com/testapp14acbe16e/referenceimageservicecalculatereferenceimagefunctione6710baarepo"] + +[devdaitabeapp] +[devdaitabeapp.deploy] +[devdaitabeapp.deploy.parameters] +stack_name = "devdaitabeapp" +s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-eu58g5l8is1s" +s3_prefix = "devdaitabeapp" +region = "us-east-2" +confirm_changeset = true +capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" +disable_rollback = true +parameter_overrides = "Mode=\"dev\" Stage=\"devdaitabeapp\" Application=\"devdaitabeapp\" SecurityGroupIds=\"sg-0c9a0ca7844d7b128,sg-00d8b4ca79ee1e42f,sg-007caf776eee9bd32,sg-04b9c865721337372,sg-0b411b5391db8d7a3\" SecurityGroupsIdsDefault=\"sg-007caf776eee9bd32\" SubnetIDs=\"subnet-0642064673fd68d2e\" SubnetIDDefault=\"subnet-079365f0ecde37d92\" S3BucketName=\"client-data-test\" VPCid=\"vpc-057803c925fd8138a\" EFSFileSystemId=\"fs-01115862a24b75423\" MaxConcurrencyTasks=\"2\" ROOTEFS=\"/efs\" DomainUserPool=\"authdev.daita.tech\" LogoutUrl=\"http://localhost:3000,https://dev.daita.tech\"" +image_repositories = ["HealthCheckService/CalculateHealthCheckFunction=737589818430.dkr.ecr.us-east-2.amazonaws.com/devdaitabeappb6b54d79/healthcheckservicecalculatehealthcheckfunction6d6be9a4repo", "ReferenceImageService/CalculateReferenceImageFunction=737589818430.dkr.ecr.us-east-2.amazonaws.com/devdaitabeappb6b54d79/referenceimageservicecalculatereferenceimagefunctione6710baarepo"] + +[devbele] +[devbele.deploy] +[devbele.deploy.parameters] +stack_name = "devbele" +s3_prefix = "devbele" +region = "us-east-2" +confirm_changeset = false +capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" +disable_rollback = true +parameter_overrides = "Mode=\"dev\" Stage=\"devbele\" Application=\"devbele\" SecurityGroupIds=\"sg-0c9a0ca7844d7b128,sg-00d8b4ca79ee1e42f,sg-007caf776eee9bd32,sg-04b9c865721337372,sg-0b411b5391db8d7a3\" SecurityGroupsIdsDefault=\"sg-007caf776eee9bd32\" SubnetIDs=\"subnet-0642064673fd68d2e\" SubnetIDDefault=\"subnet-079365f0ecde37d92\" S3BucketName=\"client-data-test\" VPCid=\"vpc-057803c925fd8138a\" EFSFileSystemId=\"fs-01115862a24b75423\" MaxConcurrencyTasks=\"2\" ROOTEFS=\"/efs\" DomainUserPool=\"auth.daita.tech\" LogoutUrl=\"https://app.daita.tech\"" +s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-eu58g5l8is1s" +profile = "daita" +image_repositories = ["HealthCheckService/CalculateHealthCheckFunction=737589818430.dkr.ecr.us-east-2.amazonaws.com/devbele4f336314/healthcheckservicecalculatehealthcheckfunction6d6be9a4repo", "ReferenceImageService/CalculateReferenceImageFunction=737589818430.dkr.ecr.us-east-2.amazonaws.com/devbele4f336314/referenceimageservicecalculatereferenceimagefunctione6710baarepo"] + +[dev1.deploy.parameters] +s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-eu58g5l8is1s" +confirm_changeset = true +capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" +disable_rollback = true +image_repositories = [] \ No newline at end of file diff --git a/daita-app/shared-layer/commons/python/config.py b/daita-app/shared-layer/commons/python/config.py index 1ad4f24..35d5476 100644 --- a/daita-app/shared-layer/commons/python/config.py +++ b/daita-app/shared-layer/commons/python/config.py @@ -1,78 +1,115 @@ -CLIENT_POOL_ID = "4cpbb5etp3q7grnnrhrc7irjoa" -USER_POOL_ID = "us-east-2_ZbwpnYN4g" -REGION = "us-east-2" -IDENTITY_POOL_ID = "us-east-2:fa0b76bc-01fa-4bb8-b7cf-a5000954aafb" - -### config for status of generate task -VALUE_GENERATE_TASK_STATUS_FINISH = "FINISH" -VALUE_GENERATE_TASK_STATUS_ERROR = "ERROR" -VALUE_GENERATE_TASK_STATUS_PENDING = "PENDING" -VALUE_GENERATE_TASK_STATUS_PREPARING_DATA = "PREPARING_DATA" -VALUE_GENERATE_TASK_STATUS_CANCEL = "CANCEL" - -### config for process type (task type) -VALUE_PROCESS_TYPE_HEALTHCHECK = "HEALTHCHECK" -VALUE_PROCESS_TYPE_UPLOAD = "UPLOAD" -VALUE_PROCESS_TYPE_DOWNLOAD = "DOWNLOAD" -VALUE_PROCESS_TYPE_PREPROCESS = "PREPROCESS" -VALUE_PROCESS_TYPE_AUGMENT = "AUGMENT" +import os +config_env = { + 'CLIENTPOOLID': {'dev': '7v8h65t0d3elscfqll090acf9h', 'staging': '4cpbb5etp3q7grnnrhrc7irjoa'}, + 'USERPOOLID': {'dev': 'us-east-2_6Sc8AZij7', 'staging': 'us-east-2_ZbwpnYN4g'}, + 'IDENTITYPOOLID': {'dev': 'us-east-2:639788f0-a9b0-460d-9f50-23bbe5bc7140', 'staging': 'us-east-2:fa0b76bc-01fa-4bb8-b7cf-a5000954aafb'}, + 'LOCATION': {'dev': "https://dev.daita.tech/", 'prod': 'https://app.daita.tech/'}, + 'ENDPPOINTREDIRCTLOGINSOCIALOAUTH': {'dev': 'https://izugd01pv1.execute-api.us-east-2.amazonaws.com/dev/auth/login_social', 'staging': 'https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/login_social'}, + + 'OAUTHENPOINT': {'dev': 'https://authdev.daita.tech/oauth2/token', 'staging': 'https://auth.daita.tech/oauth2/token'}, + + 'WEBHOOK': {'dev': 'https://hooks.slack.com/services/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', 'staging': 'https://hooks.slack.com/services/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'}, + 'CHANNELWEBHOOK': {'dev': '#feedback-daita', 'staging': '#user-feedback'}, + 'OAUTH2OFBOT': {'dev': 'xoxb-2966360820768-3760970933602-MoApe9duMpoO5KAa6HaCUzzY'} +} + + +REGION = "us-east-2" + +# TODO: refactor URL to configurable by env var or something +# URL = "https://uflt5029de.execute-api.us-east-2.amazonaws.com/devdaitabeapp/" if os.environ[ +# 'MODE'] == 'dev' else "https://119u2071ul.execute-api.us-east-2.amazonaws.com/dev/" +LOCATION = config_env['LOCATION'][os.environ['MODE']] + +ENDPOINTCAPTCHAVERIFY = "https://www.google.com/recaptcha/api/siteverify" + +# Github OpenID wrapper +# Change these if used with GitHub Enterprise (see below) +GITHUB_API_URL = "https://api.github.com" +GITHUB_LOGIN_URL = "https://github.com" +# config for status of generate task +VALUE_GENERATE_TASK_STATUS_FINISH = "FINISH" +VALUE_GENERATE_TASK_STATUS_ERROR = "ERROR" +VALUE_GENERATE_TASK_STATUS_PENDING = "PENDING" +VALUE_GENERATE_TASK_STATUS_PREPARING_DATA = "PREPARING_DATA" +VALUE_GENERATE_TASK_STATUS_CANCEL = "CANCEL" + +# config for process type (task type) +VALUE_PROCESS_TYPE_HEALTHCHECK = "HEALTHCHECK" +VALUE_PROCESS_TYPE_UPLOAD = "UPLOAD" +VALUE_PROCESS_TYPE_DOWNLOAD = "DOWNLOAD" +VALUE_PROCESS_TYPE_PREPROCESS = "PREPROCESS" +VALUE_PROCESS_TYPE_AUGMENT = "AUGMENT" VALUE_PROCESS_TYPE_REFERENCE_IM = "REFERENCE_IMAGE" -VALUE_LS_PROCESS_TYPE = [VALUE_PROCESS_TYPE_HEALTHCHECK, VALUE_PROCESS_TYPE_DOWNLOAD, VALUE_PROCESS_TYPE_AUGMENT, - VALUE_PROCESS_TYPE_PREPROCESS, VALUE_PROCESS_TYPE_REFERENCE_IM, - VALUE_PROCESS_TYPE_UPLOAD] - -### config for status of healthcheck task -VALUE_HEALTHCHECK_TASK_STATUS_RUNNING = "RUNNING" -VALUE_HEALTHCHECK_TASK_STATUS_FINISH = "FINISH" -VALUE_HEALTHCHECK_TASK_STATUS_ERROR = "ERROR" - -VALUE_TASK_RUNNING = "RUNNING" -VALUE_TASK_FINISH = "FINISH" -VALUE_TASK_ERROR = "ERROR" - -### config value type method -VALUE_TYPE_METHOD_AUGMENT = "AUGMENT" -VALUE_TYPE_METHOD_PREPROCESS = "PREPROCESS" -VALUE_TYPE_METHOD_NAME_AUGMENT = "AUG" -VALUE_TYPE_METHOD_NAME_PREPROCESS = "PRE" - -### config value data type (type of data will be use) -VALUE_TYPE_DATA_ORIGINAL = "ORIGINAL" -VALUE_TYPE_DATA_PREPROCESSED = "PREPROCESS" -VALUE_TYPE_DATA_AUGMENT = "AUGMENT" -LS_ACCEPTABLE_VALUE_GENERATE = [VALUE_TYPE_DATA_ORIGINAL, VALUE_TYPE_DATA_PREPROCESSED] - -### request + response body + state machine key -KEY_NAME_ID_TOKEN = "id_token" -KEY_NAME_PROJECT_ID = "project_id" -KEY_NAME_PROJECT_NAME = "project_name" -KEY_NAME_LS_METHOD_ID = "ls_method_id" -KEY_NAME_DATA_TYPE = "data_type" -KEY_NAME_NUM_AUG_P_IMG = "num_aug_p_img" -KEY_NAME_DATA_NUMBER = "data_number" -KEY_NAME_S3_PREFIX = "s3_prefix" -KEY_NAME_TIMES_AUGMENT = "times_augment" -KEY_NAME_TIMES_PREPROCESS = "times_preprocess" -KEY_NAME_TASK_ID = "task_id" -KEY_NAME_IDENTITY_ID = "identity_id" -KEY_NAME_TASK_STATUS = "status" -KEY_NAME_FILTER = "filter" # for task bashboard API -KEY_NAME_PAGINATION = "pagination" # for task dashboard API -KEY_SIZE_LS_ITEM_QUERY = "size_list_items_query" # for task dashboard API -KEY_NAME_PROCESS_TYPE = "process_type" -KEY_NAME_CREATED_TIME = "created_time" -KEY_NAME_REFERENCE_IMAGES = "reference_images" -KEY_NAME_METHOD_ID = "method_id" -KEY_NAME_IS_RESOLUTION = "is_normalize_resolution" -KEY_NAME_AUG_PARAMS = "aug_parameters" - -KEY_NAME_LS_METHOD_CHOOSE = "ls_method_client_choose" # for save result of reference image to database project table - -KEY_NAME_RES_AUMENTATION = "augmentation" -KEY_NAME_RES_PREPROCESSING = "preprocessing" - -MAX_NUMBER_GEN_PER_IMAGES = 1 +VALUE_LS_PROCESS_TYPE = [VALUE_PROCESS_TYPE_HEALTHCHECK, VALUE_PROCESS_TYPE_DOWNLOAD, VALUE_PROCESS_TYPE_AUGMENT, + VALUE_PROCESS_TYPE_PREPROCESS, VALUE_PROCESS_TYPE_REFERENCE_IM, + VALUE_PROCESS_TYPE_UPLOAD] + +# config for status of healthcheck task +VALUE_HEALTHCHECK_TASK_STATUS_RUNNING = "RUNNING" +VALUE_HEALTHCHECK_TASK_STATUS_FINISH = "FINISH" +VALUE_HEALTHCHECK_TASK_STATUS_ERROR = "ERROR" + +### value status create sample project from prebuild dataset +VALUE_STATUS_CREATE_SAMPLE_PRJ_GENERATING = "GENERATING" +VALUE_STATUS_CREATE_SAMPLE_PRJ_FINISH = "FINISH" + + +VALUE_TASK_RUNNING = "RUNNING" +VALUE_TASK_FINISH = "FINISH" +VALUE_TASK_ERROR = "ERROR" + +# config value type method +VALUE_TYPE_METHOD_AUGMENT = "AUGMENT" +VALUE_TYPE_METHOD_PREPROCESS = "PREPROCESS" +VALUE_TYPE_METHOD_NAME_AUGMENT = "AUG" +VALUE_TYPE_METHOD_NAME_PREPROCESS = "PRE" + +# config value data type (type of data will be use) +VALUE_TYPE_DATA_ORIGINAL = "ORIGINAL" +VALUE_TYPE_DATA_PREPROCESSED = "PREPROCESS" +VALUE_TYPE_DATA_AUGMENT = "AUGMENT" +LS_ACCEPTABLE_VALUE_GENERATE = [ + VALUE_TYPE_DATA_ORIGINAL, VALUE_TYPE_DATA_PREPROCESSED] + +# request + response body + state machine key +KEY_NAME_ID_TOKEN = "id_token" +KEY_NAME_PROJECT_ID = "project_id" +KEY_NAME_PROJECT_NAME = "project_name" +KEY_NAME_LS_METHOD_ID = "ls_method_id" +KEY_NAME_DATA_TYPE = "data_type" +KEY_NAME_NUM_AUG_P_IMG = "num_aug_p_img" +KEY_NAME_DATA_NUMBER = "data_number" +KEY_NAME_S3_PREFIX = "s3_prefix" +KEY_NAME_TIMES_AUGMENT = "times_augment" +KEY_NAME_TIMES_PREPROCESS = "times_preprocess" +KEY_NAME_TASK_ID = "task_id" +KEY_NAME_IDENTITY_ID = "identity_id" +KEY_NAME_TASK_STATUS = "status" +KEY_NAME_FILTER = "filter" # for task bashboard API +KEY_NAME_PAGINATION = "pagination" # for task dashboard API +KEY_SIZE_LS_ITEM_QUERY = "size_list_items_query" # for task dashboard API +KEY_NAME_PROCESS_TYPE = "process_type" +KEY_NAME_CREATED_TIME = "created_time" +KEY_NAME_REFERENCE_IMAGES = "reference_images" +KEY_NAME_METHOD_ID = "method_id" +KEY_NAME_IS_RESOLUTION = "is_normalize_resolution" +KEY_NAME_AUG_PARAMS = "aug_parameters" + +# for save result of reference image to database project table +KEY_NAME_LS_METHOD_CHOOSE = "ls_method_client_choose" + +KEY_NAME_RES_AUMENTATION = "augmentation" +KEY_NAME_RES_PREPROCESSING = "preprocessing" + +MAX_NUMBER_GEN_PER_IMAGES = 1 MAX_LS_ITEM_QUERY_TASK_DASHBOARD = 100 -LS_METHOD_ID_SUPPORT_REFERENCE_IMG = ["PRE-002", "PRE-003", "PRE-004", "PRE-005", "PRE-006", "PRE-008"] -LS_METHOD_KEEP_IF_EXIST_PRE001 = ["PRE-001", "PRE-000", "PRE-009"] \ No newline at end of file +LS_METHOD_ID_SUPPORT_REFERENCE_IMG = [ + "PRE-002", "PRE-003", "PRE-004", "PRE-005", "PRE-006", "PRE-008"] +LS_METHOD_KEEP_IF_EXIST_PRE001 = ["PRE-001", "PRE-000", "PRE-009"] + + +### key defined of lambda env +KEY_TABLE_PROJECT = "TABLE_PROJECT" +KEY_TABLE_PROJECT_SUM = "TABLE_PROJECT_SUMMARY" diff --git a/daita-app/shared-layer/commons/python/const.py b/daita-app/shared-layer/commons/python/const.py new file mode 100644 index 0000000..80777dd --- /dev/null +++ b/daita-app/shared-layer/commons/python/const.py @@ -0,0 +1,44 @@ +import boto3 +import os + + +db_resource = boto3.resource('dynamodb') +table = db_resource.Table("consts") + +MAX_NUM_IMGAGES_CLONE_FROM_PREBUILD_DATASET = 800 +MAX_NUM_IMAGES_IN_ORIGINAL = 1000 + +MAX_NUM_PRJ_PER_USER = 5 +MAX_TIMES_AUGMENT_IMAGES = 'limit_times_augment' +MAX_TIMES_PREPROCESS_IMAGES = 'limit_times_prepro' + +SAMPLE_PROJECT_NAME = 'Driving Dataset Sample' # 'prj_sample' # +SAMPLE_PROJECT_DESCRIPTION = 'An open-source autonomous driving dataset sample to kick-start.' + +MAX_LENGTH_PROJECT_NAME_INFO = 75 +MAX_LENGTH_PROJECT_DESCRIPTION = 300 + +MES_LENGTH_OF_PROJECT_NAME = f'Length of project name must smaller than {MAX_LENGTH_PROJECT_NAME_INFO}.' +MES_LENGTH_OF_PROJECT_INFO = f'Length of project description must smaller than {MAX_LENGTH_PROJECT_DESCRIPTION}.' +MES_REACH_LIMIT_NUM_PRJ = f'You have reached the threshold of {MAX_NUM_PRJ_PER_USER} custom projects per user.' +MES_DUPLICATE_PROJECT_NAME = "{} already exists. Please choose another name." +MES_REACH_LIMIT_AUGMENT = "You have reached the threshold of {} augmentation runs per project." +MES_REACH_LIMIT_PREPROCESS = "You have reached the threshold of {} preprocessing runs per project." + +MES_PROJECT_NOT_FOUND = "Project {} is not found." +MES_PROJECT_ALREADY_EXIST = "Project {} exists, please choose another name." +MES_PROJECT_SAME = "New project name {} must different with current name." + +### For annotaion const +FOLDER_RAW_DATA_NAME = "raw_data" +FOLDER_LABEL_NAME = "labels" + + +def get_const_db(code): + response = table.get_item( + Key={ + 'code': code, + 'type': "THRESHOLD" + } + ) + return response.get("Item")["num_value"] diff --git a/daita-app/shared-layer/commons/python/custom_mail.py b/daita-app/shared-layer/commons/python/custom_mail.py new file mode 100644 index 0000000..6d00e26 --- /dev/null +++ b/daita-app/shared-layer/commons/python/custom_mail.py @@ -0,0 +1,141 @@ +import boto3 +import json +import random +import time +from boto3 import resource +from boto3.dynamodb.conditions import Key + + +def invoke_sendmail_cognito_service(lambda_name, subject, destination_email, message_email, message_email_text): + client = boto3.client("lambda") + payload_json = None + try: + response = client.invoke( + FunctionName=lambda_name, + InvocationType="RequestResponse", + Payload=json.dumps( + { + "subject": subject, + "destination_email": destination_email, + "message_email": message_email, + "message_email_text": message_email_text + } + ), + ) + payload_json = json.loads(response["Payload"].read()) + except Exception as e: + return None, e + + return payload_json, None + + +class TriggerCustomMailcode: + def __init__(self, REGION, Table): + self.db_client = boto3.resource("dynamodb", region_name=REGION) + self.TBL = Table + + def create_item(self, info): + self.db_client.Table(self.TBL).put_item( + Item={"user": info["user"], "code": info["code"], + "time_to_live": 90*60 + 5 + int(time.time())} + ) + + def query_all_partition_key(self, value): + filteringExp = Key("user").eq(value) + return self.db_client.Table(self.TBL).query(KeyConditionExpression=filteringExp) + + def delete_item(self, info): + items = (self.query_all_partition_key(value=info["user"])).get("Items") + for item in items: + self.db_client.Table(self.TBL).delete_item( + Key={"user": item["user"], "code": item["code"]} + ) + + def find_item(self, info): + response = self.db_client.Table(self.TBL).get_item( + Key={"user": info["user"], "code": info["code"]} + ) + return True if "Item" in response else False + + +def AddTriggerCustomMail(info): + confirmCode = str(random.randint(100000, 999999)) + modelTrigger = TriggerCustomMailcode( + REGION=info["region"], Table=info['confirm_code_Table']) + modelTrigger.create_item({"user": info["user"], "code": confirmCode}) + invoke_sendmail_cognito_service( + lambda_name=info['lambda_name'], + subject=info["subject"], + destination_email=info["mail"], + message_email=""" +

Your confirmation code is {}.

+

Best,

+

The DAITA Team

+

---

+

In case you encounter any issues or questions, please contact us at contact@daita.tech.

+ """.format( + confirmCode + ), + message_email_text=""" + Your confirmation code is {}. + Best, + The DAITA Team + --- + In case you encounter any issues or questions, please contact us at contact@daita.tech. + """.format( + confirmCode + ) + ) + + +def DeleteConfirmCode(info): + modelTrigger = TriggerCustomMailcode( + REGION=info["region"], Table=info['confirm_code_Table']) + if not modelTrigger.find_item({"user": info["user"], "code": info["code"]}): + raise Exception( + "A wrong confirmation code has been entered. If you have requested a new confirmation code, use only the latest code." + ) + + modelTrigger.delete_item({"user": info["user"]}) + + +def AddInsertConfirmCode(info): + modelTrigger = TriggerCustomMailcode( + REGION=info["region"], Table=info['confirm_code_Table']) + modelTrigger.create_item( + {"user": info["user"], "code": info["confirm_code"]}) + + +def ResendCodeConfirm(info): + modelTrigger = TriggerCustomMailcode( + REGION=info["region"], Table=info['confirm_code_Table']) + + try: + modelTrigger.delete_item({"user": info["user"]}) + except Exception as e: + raise Exception(e) + confirmCode = str(random.randint(100000, 999999)) + modelTrigger.create_item({"user": info["user"], "code": confirmCode}) + invoke_sendmail_cognito_service( + lambda_name=info['lambda_name'], + subject=info["subject"], + destination_email=info["mail"], + message_email=""" +

Your confirmation code is {}.

+

Best,

+

The DAITA Team

+

---

+

In case you encounter any issues or questions, please contact us at contact@daita.tech.

+ """.format( + confirmCode + ), + message_email_text=""" + Your confirmation code is {}. + Best, + The DAITA Team + --- + In case you encounter any issues or questions, please contact us at contact@daita.tech. + """.format( + confirmCode + ) + ) diff --git a/daita-app/shared-layer/commons/python/error_messages.py b/daita-app/shared-layer/commons/python/error_messages.py index 771f6d1..96a9ab5 100644 --- a/daita-app/shared-layer/commons/python/error_messages.py +++ b/daita-app/shared-layer/commons/python/error_messages.py @@ -1,17 +1,71 @@ -## Put error message here -MESS_AUTHEN_FAILED = "Authentication failed" -MESS_INVALID_JSON_INPUT = "Invalid JSON input" -MESS_NUMBER_TRAINING = "The number of training images must be greater than 0!" -MESS_NUMBER_DATA = "Number of train/val/test must greater than 0" -MESS_LIST_METHODS_EMPTY = "List method id must not empty!" -MESS_ERROR_OVER_LIMIT_RUNNING_TASK = "You have been running a task in this project, please wait!" -MESS_ERR_INVALID_LIST_METHOD = "List method is not valid!" -MESS_REACH_LIMIT_AUGMENT = "You have reached the threshold of {} augmentation runs per project." -MESS_REACH_LIMIT_PREPROCESS = "You have reached the threshold of {} preprocessing runs per project." -MESS_TASK_NOT_EXIST = "Task ID {} does not exist" -MESS_DATA_TYPE_INPUT = "Data type value {} does not belong to {}." -MESS_PROJECT_NOT_FOUND = "Project {} not found." -MESS_PROCESS_TYPE_IS_INVALID = "Process type {} is invalid!" +# Put error message here +MESS_AUTHEN_FAILED = "Authentication failed" +MESS_INVALID_JSON_INPUT = "Invalid JSON input" +MESS_NUMBER_TRAINING = "The number of training images must be greater than 0!" +MESS_NUMBER_DATA = "Number of train/val/test must greater than 0" +MESS_LIST_METHODS_EMPTY = "List method id must not empty!" +MESS_ERROR_OVER_LIMIT_RUNNING_TASK = "You have been running a task in this project, please wait!" +MESS_ERR_INVALID_LIST_METHOD = "List method is not valid!" +MESS_REACH_LIMIT_AUGMENT = "You have reached the threshold of {} augmentation runs per project." +MESS_REACH_LIMIT_PREPROCESS = "You have reached the threshold of {} preprocessing runs per project." +MESS_TASK_NOT_EXIST = "Task ID {} does not exist" +MESS_DATA_TYPE_INPUT = "Data type value {} does not belong to {}." +MESS_PROJECT_NOT_FOUND = "Project {} not found." +MESS_PROCESS_TYPE_IS_INVALID = "Process type {} is invalid!" MESS_METHOD_REFERENCE_IMAGE_INVALID = "Method id must belong to prerpocess method!" -MESS_METHOD_DOES_NOT_SUPPORT = "System does not support method_id {}." -MESS_NO_DATA_IN_VIEWED_TAB = "There are no images on the selected tab {}." +MESS_METHOD_DOES_NOT_SUPPORT = "System does not support method_id {}." +MESS_NO_DATA_IN_VIEWED_TAB = "There are no images on the selected tab {}." +MessageUnmarshalInputJson = "Unmarshal unexpected end of JSON input." +MessageAuthenFailed = "Authentication failed." +MessageLoginFailed = "Username or password is incorrect." +MessageSignUpFailed = "Sign up failed." +MessageSignInSuccessfully = "Login successful." +MessageMissingAuthorizationHeader = "Missing authorization header." +MessageConfirmWrongCodeSignUP = "A wrong confirmation code has been entered. If you have requested a new confirmation code, use only the latest code." +MessageUserVerifyConfirmCode = "User needs to verify confirmation code." +MessageSignUPEmailInvalid = "This email address is already being used." +MessageSignUpUsernameInvalid = "This username is already being used." +MessageSignUpInvalid = "Both username and email are already being used." +MessageResendEmailConfirmCodeSuccessfully = "Email successfully sent." +MessageResendEmailConfirmCodeFailed = "Email delivery failed." +MessageInvalidPassword = "Password rules are at least 8 characters, at least 1 lower case letter, at least 1 upper case letter, at least 1 number, and at least 1 special character." +MessageVerifyConfirmcodeWrong = "Invalid confirmation code provided." +MessageTheValueNotExistInDatabase = "The value does not exist in the database." +MessageRefreshTokenError = "Refreshing the token failed." +MessageRefreshTokenSuccessfully = "Refreshing the token was successful." +MessageForgotPasswordFailed = "Forgot password failed." +MessageForgotPasswordUsernotExist = "Username does not exist." +MessageForgotPasswordSuccessfully = "Email for password recovery successfully sent." +MessageForgotPasswordConfirmcodeSuccessfully = "Confirmation code for password recovery was successful." +MessageForgotPasswordConfirmcodeFailed = "Confirmation code for password recovery failed." +MessageCannotRegisterMoreUser = "We cannot register more users currently." +MessageTokenInvalid = "Token is invalid." +MessageGetTemapleMailSuccessFully = "Template for friend invitation received successfully." +MessageErrorDeleteUserSignUp = "Error deleting user." +MessageCaptchaFailed = "Captcha token verification failed." +MessageSendFeedbackFailed = "Sending feeback failed." +MessageSendFeedbackSuccessfully = "Successfully sent feedback." +MessageLoginMailNotExist = "Email does not exist." +MessageAnotherUserIsLoginBefore = "You are already logged in on another device." +MessageErrorUserdoesnotlogin = "Current user did not login to the application!" +MessageLogoutSuccessfully = "User log out successful." +MessageErrorCredential = "Failed to retrieve credentials." +MessageSuccessfullyCredential = "Successfully retrieved credential." +MessageErrorFeedbackInvalidType = "Please check format json!" +MessageErrorFeedbackLimitword = "Word limit for feedback!" +MessageErrorFeedbackLimitImages = "Attachment file count limit!" +MessageErrorInvalidExtension = "Invalid extension of file attachment!" + +###========== Error message for project ===================### +### Error create prebuild dataset +MESS_ERR_INVALID_PREBUILD_DATASET_NAME = "The prebuild dataset {} is not valid now." + +MES_LENGTH_OF_PROJECT_NAME = 'Length of project name must smaller than {}.' +# MES_LENGTH_OF_PROJECT_INFO = 'Length of project description must smaller than {}.' +MES_REACH_LIMIT_NUM_PRJ = 'You have reached the threshold of {} custom projects per user.' +MES_DUPLICATE_PROJECT_NAME = "{} already exists. Please choose another name." +MES_REACH_LIMIT_AUGMENT = "You have reached the threshold of {} augmentation runs per project." +MES_REACH_LIMIT_PREPROCESS = "You have reached the threshold of {} preprocessing runs per project." + +### for project annotation +MESS_PROJECT_NOT_EXIST = "Project {} does not exist in daita system." diff --git a/daita-app/shared-layer/commons/python/identity_check.py b/daita-app/shared-layer/commons/python/identity_check.py index 6327609..f3e9cb5 100644 --- a/daita-app/shared-layer/commons/python/identity_check.py +++ b/daita-app/shared-layer/commons/python/identity_check.py @@ -3,15 +3,16 @@ from config import * from error_messages import * -def aws_get_identity_id(id_token): +def aws_get_identity_id(id_token, USER_POOL_ID=os.environ.get("COGNITO_USER_POOL", ""), IDENTITY_POOL_ID=os.environ.get("IDENTITY_POOL", "")): identity_client = boto3.client('cognito-identity') PROVIDER = f'cognito-idp.{identity_client.meta.region_name}.amazonaws.com/{USER_POOL_ID}' try: identity_response = identity_client.get_id( - IdentityPoolId=IDENTITY_POOL_ID, - Logins = {PROVIDER: id_token}) + IdentityPoolId=IDENTITY_POOL_ID, + Logins={PROVIDER: id_token}) except Exception as e: + print('Error: ', repr(e)) raise Exception(MESS_AUTHEN_FAILED) from e identity_id = identity_response['IdentityId'] diff --git a/daita-app/shared-layer/commons/python/lambda_base_class.py b/daita-app/shared-layer/commons/python/lambda_base_class.py index 20b2980..875f610 100644 --- a/daita-app/shared-layer/commons/python/lambda_base_class.py +++ b/daita-app/shared-layer/commons/python/lambda_base_class.py @@ -2,23 +2,30 @@ import os from identity_check import * import json +from load_env_lambda_function import LambdaEnv + + class LambdaBaseClass(object): def __init__(self) -> None: self.logger = logging.getLogger(self.__class__.__name__) self.logger.setLevel(os.environ["LOGGING"]) + self.env = LambdaEnv() def handle(self, event, context): raise NotImplementedError @classmethod def parse_body(cls, func): - def parser(object, event): - if type(event['body']) is str: - body = json.loads(event['body']) + def parser(object, event, is_event_as_body = False): + if is_event_as_body: + body = event else: - body = event['body'] + if type(event['body']) is str: + body = json.loads(event['body']) + else: + body = event['body'] object.logger.info("Body: {}".format(body)) try: @@ -35,7 +42,21 @@ def _check_input_value(self): pass return - def get_identity(self, id_token): - identity = aws_get_identity_id(id_token) + def get_identity(self, id_token, user_pool_id=None, identity_pool_id=None): + if user_pool_id is None or identity_pool_id is None: + identity = aws_get_identity_id(id_token) + else: + identity = aws_get_identity_id(id_token=id_token, + USER_POOL_ID=user_pool_id, IDENTITY_POOL_ID=identity_pool_id) self.logger.info(f"identity: {identity}") - return identity \ No newline at end of file + return identity + + def invoke_lambda_func(self, function_name, body_info, type_request="RequestResponse"): + lambdaInvokeClient = boto3.client('lambda') + lambdaInvokeReq = lambdaInvokeClient.invoke( + FunctionName=function_name, + Payload=json.dumps({'body': body_info}), + InvocationType=type_request, + ) + + return json.loads(lambdaInvokeReq['Payload'].read().decode("utf-8")) diff --git a/daita-app/shared-layer/commons/python/load_const_lambda_func.py b/daita-app/shared-layer/commons/python/load_const_lambda_func.py new file mode 100644 index 0000000..a124bcf --- /dev/null +++ b/daita-app/shared-layer/commons/python/load_const_lambda_func.py @@ -0,0 +1,16 @@ +import boto3 + +class LambdaConst(): + + FIELD_GROUP_NAME = "group" + FIELD_NAME = "name" + FIELD_VALUE = "value" + + VALUE_GROUP = "anno" + + VALUE_NAME = "" + + def __init__(self, table_name) -> None: + self.table = boto3.resource('dynamodb').Table(table_name) + + self.const \ No newline at end of file diff --git a/daita-app/shared-layer/commons/python/load_env_lambda_function.py b/daita-app/shared-layer/commons/python/load_env_lambda_function.py new file mode 100644 index 0000000..eeb2bad --- /dev/null +++ b/daita-app/shared-layer/commons/python/load_env_lambda_function.py @@ -0,0 +1,46 @@ +import os + +class LambdaEnv(): + def __init__(self) -> None: + + self.USER_POOL_ID = os.environ.get('COGNITO_USER_POOL', "") + self.IDENTITY_POOL_ID = os.environ.get('IDENTITY_POOL', "") + self.COGNITO_CLIENT_ID = os.environ.get("COGNITO_CLIENT_ID", "") + + self.CAPTCHA_SITE_KEY_GOOGLE = os.environ.get("CAPTCHA_SITE_KEY_GOOGLE", "") + self.CAPTCHA_SECRET_KEY_GOOGLE = os.environ.get("CAPTCHA_SECRET_KEY_GOOGLE", "") + + + self.TABLE_ANNO_PROJECT = os.environ.get("TABLE_ANNO_PROJECT", "") + self.TABLE_ANNO_DATA_ORI = os.environ.get("TABLE_ANNO_DATA_ORI", "") + self.TABLE_ANNO_PROJECT_SUMMARY = os.environ.get("TABLE_ANNO_PROJECT_SUMMARY", "") + self.TABLE_ANNO_LABEL_INFO = os.environ.get("TABLE_ANNO_LABEL_INFO", "") + self.TABLE_ANNO_CATEGORY_INFO = os.environ.get("TABLE_ANNO_CATEGORY_INFO", "") + self.TABLE_ANNO_CLASS_INFO = os.environ.get("TABLE_ANNO_CLASS_INFO", "") + self.TABLE_ANNO_AI_DEFAULT_CLASS = os.environ.get("TABLE_ANNO_AI_DEFAULT_CLASS", "") + self.TABLE_ANNO_DELETED_PRJ = os.environ.get("TABLE_ANNO_DELETED_PRJ", "") + + self.S3_ANNO_BUCKET_NAME = os.environ.get("S3_ANNO_BUCKET_NAME", "") + self.S3_DAITA_BUCKET_NAME = os.environ.get("S3_DAITA_BUCKET_NAME", "") + + self.TABLE_DAITA_DATA_ORIGINAL = os.environ.get("TABLE_DAITA_DATA_ORIGINAL", "") + self.TABLE_DAITA_PROJECT = os.environ.get("TABLE_DAITA_PROJECT", "") + self.TABLE_GENERATE_TASK = os.environ.get("TABLE_GENERATE_TASK", "") + + self.TABLE_USER = os.environ.get("TABLE_USER", "") + + self.TABLE_CONFIG_PARA_LAMBDA = os.environ.get("TABLE_CONFIG_PARA_LAMBDA", "") + + + + self.SM_CLONE_PROJECT_ARN = os.environ.get("SM_CLONE_PROJECT_ARN", "") + + self.FUNC_DAITA_UPLOAD_UPDATE = os.environ.get("FUNC_DAITA_UPLOAD_UPDATE", "") + + self.IS_USE_ECS_AI_CALLER_FLOW = os.environ.get("IS_USE_ECS_AI_CALLER_FLOW", False) + self.AI_CALLER_ECS_SM_ARN = os.environ.get("AI_CALLER_ECS_SM_ARN", "") + + self.FUNC_RI_CALCULATION = os.environ.get("FUNC_RI_CALCULATION", "") + self.FUNC_RI_STATUS = os.environ.get("FUNC_RI_STATUS", "") + self.FUNC_RI_INFO = os.environ.get("FUNC_RI_INFO", "") + diff --git a/daita-app/shared-layer/commons/python/models/annotaition/anno_category_info.py b/daita-app/shared-layer/commons/python/models/annotaition/anno_category_info.py new file mode 100644 index 0000000..0b16ca0 --- /dev/null +++ b/daita-app/shared-layer/commons/python/models/annotaition/anno_category_info.py @@ -0,0 +1,46 @@ +import boto3 +from boto3.dynamodb.conditions import Key, Attr +from config import * +from typing import List +from utils import convert_current_date_to_iso8601, create_unique_id +from models.base_model import BaseModel + +class AnnoCategoryInfoModel(BaseModel): + + FIELD_PROJECT_ID = "project_id" ### hash + FIELD_CATEGORY_ID = "category_id" ### range + FIELD_CATEGORY_NAME = "category_name" + FIELD_CATEGORY_DES = "category_des" + FIELD_S3_KEY_JSON_LABEL = "s3key_jsonlabel" + FIELD_CREATED_TIME = "created_time" + FIELD_UPDATED_TIME = "updated_time" + + + def __init__(self, table_name) -> None: + self.table = boto3.resource('dynamodb').Table(table_name) + + def create_new_category(self, project_id, category_name, category_des): + category_id = f"{category_name}_{create_unique_id()}" + item = { + self.FIELD_CREATED_TIME: convert_current_date_to_iso8601(), + self.FIELD_UPDATED_TIME: convert_current_date_to_iso8601(), + self.FIELD_PROJECT_ID: project_id, + self.FIELD_CATEGORY_ID: category_id, + self.FIELD_CATEGORY_NAME: category_name, + self.FIELD_CATEGORY_DES: category_des + } + condition = Attr(self.FIELD_PROJECT_ID).not_exists() & Attr(self.FIELD_CATEGORY_ID).not_exists() + self.put_item_w_condition(item, condition) + + return category_id + + + def query_all_category_label(self, file_id): + response = self.table.query( + KeyConditionExpression=Key(self.FIELD_FILE_ID).eq(file_id), + ) + + return response.get("Items", []) + + + \ No newline at end of file diff --git a/daita-app/shared-layer/commons/python/models/annotaition/anno_class_info.py b/daita-app/shared-layer/commons/python/models/annotaition/anno_class_info.py new file mode 100644 index 0000000..58ccab2 --- /dev/null +++ b/daita-app/shared-layer/commons/python/models/annotaition/anno_class_info.py @@ -0,0 +1,91 @@ +import boto3 +from boto3.dynamodb.conditions import Key, Attr +from config import * +from typing import List +from utils import convert_current_date_to_iso8601, create_unique_id +from models.base_model import BaseModel + +class AnnoClassInfoModel(BaseModel): + + + FIELD_CATEGORY_ID = "category_id" ### hash + FIELD_CLASS_NAME = "class_name" ### range + FIELD_CLASS_ID = "class_id" + FIELD_CLASS_DES = "description" + + FIELD_CREATED_TIME = "created_time" + FIELD_UPDATED_TIME = "updated_time" + + + def __init__(self, table_name) -> None: + self.table = boto3.resource('dynamodb').Table(table_name) + + def create_new_class(self, category_id, class_name): + class_id = f"{class_name}_{create_unique_id()}" + item = { + self.FIELD_CREATED_TIME: convert_current_date_to_iso8601(), + self.FIELD_UPDATED_TIME: convert_current_date_to_iso8601(), + self.FIELD_CATEGORY_ID: category_id, + self.FIELD_CLASS_NAME: class_name, + self.FIELD_CLASS_ID: class_id + } + condition = Attr(self.FIELD_CATEGORY_ID).not_exists() & Attr(self.FIELD_CLASS_NAME).not_exists() + + try: + self.put_item_w_condition(item, condition) + except Exception as e: + print(e) + return False + + return True + + def get_all_AI_default_class(self): + response = self.table.scan() + items = response['Items'] + + return items + + def add_default_AI_class(self, category_id, ls_items): + ls_item_requests = [] + for item in ls_items: + item_request = { + self.FIELD_CATEGORY_ID: category_id, + self.FIELD_CLASS_NAME: item[self.FIELD_CLASS_NAME], + self.FIELD_CLASS_ID: item[self.FIELD_CLASS_ID], + self.FIELD_CREATED_TIME: convert_current_date_to_iso8601(), + self.FIELD_UPDATED_TIME: convert_current_date_to_iso8601() + } + ## add to list requests + ls_item_requests.append(item_request) + + ### batch write to DB + print("ls_item_requests: ", ls_item_requests) + self.batch_write(ls_item_requests) + + return + + def add_list_class(self, category_id, ls_class_name): + ls_ok = [] + ls_fail = [] + for class_name in ls_class_name: + is_ok = self.create_new_class(category_id, class_name) + if is_ok: + ls_ok.append(class_name) + else: + ls_fail.append(class_name) + + return ls_ok, ls_fail + + def query_all_class_of_category(self, category_id, ls_fields_projection=[]): + if len(ls_fields_projection) == 0: + ls_fields_projection = [self.FIELD_CLASS_NAME, self.FIELD_CLASS_ID] + + response = self.table.query( + KeyConditionExpression=Key(self.FIELD_CATEGORY_ID).eq(category_id), + ProjectionExpression= ",".join(ls_fields_projection), + ) + + return response.get("Items", []) + + + \ No newline at end of file diff --git a/daita-app/shared-layer/commons/python/models/annotaition/anno_data_model.py b/daita-app/shared-layer/commons/python/models/annotaition/anno_data_model.py new file mode 100644 index 0000000..a3a07c4 --- /dev/null +++ b/daita-app/shared-layer/commons/python/models/annotaition/anno_data_model.py @@ -0,0 +1,238 @@ +import boto3 +from boto3.dynamodb.conditions import Key, Attr +from config import * +from typing import List +from utils import convert_current_date_to_iso8601, create_unique_id +from models.base_model import BaseModel + +class AnnoDataModel(BaseModel): + + FIELD_PROJECT_ID = "project_id" + FIELD_FILENAME = "filename" + FIELD_FILE_ID = "file_id" + FIELD_CREATED_TIME = "created_time" + FIELD_UPDATED_TIME = "updated_time" + FIELD_GEN_ID = "gen_id" + FIELD_HASH = "hash" + FIELD_IS_ORIGINAL = "is_ori" + FIELD_S3_KEY = "s3_key" + FIELD_TYPE_METHOD = "type_method" + FIELD_SIZE = "size" + FIELD_S3_SEGME_LABEL = "s3_key_segm" + + REQUEST_TYPE_ALL = "all" + + def __init__(self, table_name) -> None: + self.table = boto3.resource('dynamodb').Table(table_name) + + def get_item(self, project_id, filename, ls_fields = []): + if len(ls_fields)==0: + ls_fields = [self.FIELD_FILENAME, self.FIELD_FILE_ID, self.FIELD_S3_KEY, self.FIELD_S3_SEGME_LABEL, + self.FIELD_SIZE, self.FIELD_CREATED_TIME] + response = self.table.get_item( + Key={ + self.FIELD_PROJECT_ID: project_id, + self.FIELD_FILENAME: filename, + }, + ProjectionExpression= ",".join(ls_fields) + ) + item = response.get('Item', None) + + if item: + item[self.FIELD_SIZE] = int(item[self.FIELD_SIZE]) + + return item + + def _query_project_wo_healthcheck_id(self, project_id): + response = self.table.query ( + KeyConditionExpression=Key(self.FIELD_PROJECT_ID).eq(project_id), + FilterExpression=Attr(self.FIELD_HEALTHCHECK_ID).not_exists() + ) + items = response.get("Items", []) + + return items + + def _update_healthcheck_id(self, project_id, filename, healthcheck_id): + response = self.table.update_item( + Key={ + self.FIELD_PROJECT_ID: project_id, + self.FIELD_FILENAME: filename, + }, + ExpressionAttributeNames= { + '#HC': self.FIELD_HEALTHCHECK_ID, + '#UP_DATE': self.FIELD_UPDATED_TIME + }, + ExpressionAttributeValues = { + ':hc': healthcheck_id, + ':da': convert_current_date_to_iso8601(), + }, + UpdateExpression = 'SET #HC = :hc , #UP_DATE = :da' + ) + return + + def get_all_wo_healthcheck_id(self, project_id: str): + """ + get all data that does not have healthcheck_id + Args: + project_id (str): project id + + Returns: + _type_: _description_ + """ + + items = self._query_project_wo_healthcheck_id(project_id) + + ls_s3_key = [(item[self.FIELD_FILENAME], item[self.FIELD_S3_KEY]) for item in items] + + return ls_s3_key + + def get_all_data_in_project(self, project_id, ls_fields_projection = []): + if len(ls_fields_projection) == 0: + ls_fields_projection = [self.FIELD_FILENAME, self.FIELD_S3_KEY, + self.FIELD_SIZE] + + response = self.table.query ( + KeyConditionExpression=Key(self.FIELD_PROJECT_ID).eq(project_id), + ProjectionExpression=",".join(ls_fields_projection), + ) + ls_items = response.get("Items", []) + + while 'LastEvaluatedKey' in response: + next_token = response['LastEvaluatedKey'] + response = self.table.query ( + KeyConditionExpression=Key(self.FIELD_PROJECT_ID).eq(project_id), + ProjectionExpression=",".join(ls_fields_projection), + ExclusiveStartKey=next_token + ) + + ls_items += response.get("Items", []) + + return [self.convert_decimal_indb_item(item) for item in ls_items] + + def update_healthcheck_id(self, project_id, filename, healthcheck_id): + self._update_healthcheck_id(project_id, filename, healthcheck_id) + + def query_data_follow_batch(self, project_id, next_token, num_limit, ls_fields_projection=[]): + if len(ls_fields_projection) == 0: + ls_fields_projection = [self.FIELD_CREATED_TIME, self.FIELD_FILENAME, + self.FIELD_S3_KEY, self.FIELD_S3_SEGME_LABEL, self.FIELD_SIZE] + + if len(next_token) == 0: + response = self.table.query( + # IndexName='index-created-sorted', + KeyConditionExpression=Key(self.FIELD_PROJECT_ID).eq(project_id), + ProjectionExpression= ",".join(ls_fields_projection), + Limit = num_limit, + ScanIndexForward = False + ) + print('___Response first: ___', response) + else: + response = self.table.query( + # IndexName='index-created-sorted', + KeyConditionExpression = Key(self.FIELD_PROJECT_ID).eq(project_id), + ProjectionExpression = ",".join(ls_fields_projection), + ExclusiveStartKey=next_token, + Limit=num_limit, + ScanIndexForward=False + ) + print('___Response next: ___', response) + + next_token = None + # LastEvaluatedKey indicates that there are more results + if 'LastEvaluatedKey' in response: + next_token = response['LastEvaluatedKey'] + + ls_items = response.get("Items", []) + + return next_token, [self.convert_decimal_indb_item(item) for item in ls_items] + + def delete_project(self,project_id): + + # delete in data + query = None + with self.table.batch_writer() as batch: + while query is None or 'LastEvaluatedKey' in query: + if query is not None and 'LastEvaluatedKey' in query: + query = self.table.query( + KeyConditionExpression=Key(self.FIELD_PROJECT_ID).eq(project_id), + ExclusiveStartKey=query['LastEvaluatedKey'] + ) + else: + query = self.table.query( + KeyConditionExpression=Key(self.FIELD_PROJECT_ID).eq(project_id) + ) + + for item in query['Items']: + batch.delete_item( + Key={ + self.FIELD_PROJECT_ID: project_id, + self.FIELD_FILENAME: item[self.FIELD_FILENAME] + } + ) + + return + + def get_item_from_list(self, project_id, ls_filename: List[str]): + # create the batch request from input data + ls_batch_request = [] + for filename in ls_filename: + request = { + self.FIELD_PROJECT_ID: project_id, # partition key + self.FIELD_FILENAME: filename + } + ls_batch_request.append(request) + + batch_keys = { + self.table.name: { + 'Keys': ls_batch_request, + 'ProjectionExpression': f'{self.FIELD_FILENAME}, {self.FIELD_SIZE}' + } + } + + response = boto3.resource('dynamodb').batch_get_item(RequestItems=batch_keys)["Responses"] + + ls_data_res = response.get(self.table.name, []) + + return self.convert_decimal_ls_item(ls_data_res) + + def put_item_from_ls_object(self, project_id, ls_object_info): + # create the batch request from input data and summary the information + ls_batch_request = [] + for object in ls_object_info: + request = { + self.FIELD_PROJECT_ID: project_id, # partition key + self.FIELD_S3_KEY: object['s3_key'], # sort_key + self.FIELD_FILENAME: object['filename'], + self.FIELD_FILE_ID: create_unique_id(), + self.FIELD_HASH: object.get('hash', ''), # we use function get it mean that this field is optional in body + self.FIELD_SIZE: object['size'], # size must be in Byte unit + self.FIELD_IS_ORIGINAL: True, + self.FIELD_CREATED_TIME: convert_current_date_to_iso8601() + } + ls_batch_request.append(request) + + # update data to DB + # we use batch_write, it means that if key are existed in tables => overwrite + with self.table.batch_writer() as batch: + for item in ls_batch_request: + batch.put_item(Item=item) + + return + + def query_progress_ai_segm(self, project_id): + """ + Check all data of a project, count total and the number of data that exist ai_segmentation + """ + ls_fields_projection = [self.FIELD_FILENAME, self.FIELD_S3_SEGME_LABEL] + + ls_items = self.get_all_data_in_project(project_id, ls_fields_projection) + ls_ai_finished = [] + + for item in ls_items: + if item.get(self.FIELD_S3_SEGME_LABEL) is not None: + if len(item[self.FIELD_S3_SEGME_LABEL])>1: + ls_ai_finished.append(item) + + return len(ls_items), len(ls_ai_finished) + + diff --git a/daita-app/shared-layer/commons/python/models/annotaition/anno_label_info_model.py b/daita-app/shared-layer/commons/python/models/annotaition/anno_label_info_model.py new file mode 100644 index 0000000..08e1389 --- /dev/null +++ b/daita-app/shared-layer/commons/python/models/annotaition/anno_label_info_model.py @@ -0,0 +1,80 @@ +import boto3 +from boto3.dynamodb.conditions import Key, Attr +from config import * +from typing import List +from utils import convert_current_date_to_iso8601 +from models.base_model import BaseModel + +class AnnoLabelInfoModel(BaseModel): + + FIELD_FILE_ID = "file_id" ### hash + FIELD_CATEGORY_ID = "category_id" ### range + FIELD_CATEGORY_NAME = "category_name" + FIELD_CATEGORY_DES = "category_des" + FIELD_S3_KEY_JSON_LABEL = "s3key_jsonlabel" + FIELD_CREATED_TIME = "created_time" + FIELD_UPDATED_TIME = "updated_time" + + + def __init__(self, table_name) -> None: + self.table = boto3.resource('dynamodb').Table(table_name) + + ### move to category model + def create_new_category(self, file_id, category_id, category_name, category_des): + item = { + self.FIELD_CREATED_TIME: convert_current_date_to_iso8601(), + self.FIELD_UPDATED_TIME: convert_current_date_to_iso8601(), + self.FIELD_FILE_ID: file_id, + self.FIELD_CATEGORY_ID: category_id, + self.FIELD_CATEGORY_NAME: category_name, + self.FIELD_CATEGORY_DES: category_des + } + condition = Attr(self.FIELD_FILE_ID).not_exists() & Attr(self.FIELD_CATEGORY_ID).not_exists() + self.put_item_w_condition(item, condition) + + return + + def update_label_for_category(self, file_id, category_id, s3_key_json_label): + response = self.table.update_item( + Key={ + self.FIELD_FILE_ID: file_id, + self.FIELD_CATEGORY_ID: category_id, + }, + ExpressionAttributeNames= { + '#S3_K': self.FIELD_S3_KEY_JSON_LABEL, + '#UP_DATE': self.FIELD_UPDATED_TIME + }, + ExpressionAttributeValues = { + ':s3_k': s3_key_json_label, + ':da': convert_current_date_to_iso8601(), + }, + UpdateExpression = 'SET #S3_K = :s3_k , #UP_DATE = :da' + ) + + return + + def query_all_category_label(self, file_id): + response = self.table.query( + KeyConditionExpression=Key(self.FIELD_FILE_ID).eq(file_id), + ) + + return response.get("Items", []) + + def get_label_info_of_category(self, file_id, category_id, ls_fields = []): + if len(ls_fields)==0: + ls_fields = [self.FIELD_S3_KEY_JSON_LABEL] + + response = self.table.get_item( + Key={ + self.FIELD_FILE_ID: file_id, + self.FIELD_CATEGORY_ID: category_id, + }, + ProjectionExpression= ",".join(ls_fields) + ) + item = response.get('Item', None) + + return item + + + + \ No newline at end of file diff --git a/daita-app/shared-layer/commons/python/models/annotaition/anno_project_model.py b/daita-app/shared-layer/commons/python/models/annotaition/anno_project_model.py new file mode 100644 index 0000000..326ce70 --- /dev/null +++ b/daita-app/shared-layer/commons/python/models/annotaition/anno_project_model.py @@ -0,0 +1,240 @@ +import boto3 +from boto3.dynamodb.conditions import Key, Attr +from config import * +from utils import * + + + + +class AnnoProjectModel(): + FIELD_IDENTITY_ID = "identity_id" + FIELD_PROJECT_ID = "project_id" + FIELD_PROJECT_NAME = "project_name" + FIELD_GEN_STATUS = "gen_status" ### status of generating progress + FIELD_S3_PREFIX = "s3_prefix" + FIELD_S3_LABEL = "s3_label" + FIELD_S3_PRJ_ROOT = "s3_prj_root" + FIELD_IS_SAMPLE = "is_sample" + FIELD_PROJECT_INFO = "project_info" + FIELD_TIMES_AUGMENT = "times_generated" + FIELD_TIMES_PREPRO = "times_propr" + FIELD_UPDATE_DATE = "updated_date" + FIELD_CREATED_DATE = "created_date" + FIELD_LINKED_PROJECT = "link_daita_prj_id" + FIELD_CATEGORY_DEFAULT = "defa_category_id" ### the default category of project, remove it after has category logic + FIELD_REFERENCE_IMAGES = KEY_NAME_REFERENCE_IMAGES + FIELD_DATANUM_ORIGINAL = VALUE_TYPE_DATA_ORIGINAL + FIELD_DATANUM_PREPROCESS = VALUE_TYPE_DATA_PREPROCESSED + FIELD_AUG_PARAMETERS = KEY_NAME_AUG_PARAMS + FIELD_IS_DELETED = "is_dele" + + VALUE_GEN_STATUS_GENERATING = "GENERATING" + VALUE_GEN_STATUS_FINISH = "FINISH" + + def __init__(self, table_name) -> None: + print("table name: ", table_name) + self.table = boto3.resource('dynamodb').Table(table_name) + + def get_all_project(self, identity_id): + + ls_fields_projection = [self.FIELD_PROJECT_ID, self.FIELD_PROJECT_NAME, + self.FIELD_GEN_STATUS, self.FIELD_CREATED_DATE] + + response = self.table.query ( + KeyConditionExpression=Key(self.FIELD_IDENTITY_ID).eq(identity_id), + ProjectionExpression= ",".join(ls_fields_projection), + ) + ls_items = response.get("Items", []) + + while 'LastEvaluatedKey' in response: + next_token = response['LastEvaluatedKey'] + response = self.table.query ( + KeyConditionExpression=Key(self.FIELD_IDENTITY_ID).eq(identity_id), + ProjectionExpression = ",".join(ls_fields_projection), + ExclusiveStartKey=next_token + ) + + ls_items += response.get("Items", []) + + return ls_items + + def get_project_info(self, identity_id, project_name, ls_projection_fields = []) -> dict: + ### set default fields for projection + if len(ls_projection_fields) == 0: + ls_projection_fields = [self.FIELD_PROJECT_ID, self.FIELD_PROJECT_NAME, self.FIELD_GEN_STATUS] + + ### query + response = self.table.get_item( + Key={ + self.FIELD_IDENTITY_ID: identity_id, + self.FIELD_PROJECT_NAME: project_name, + }, + ProjectionExpression= ",".join(ls_projection_fields) + ) + item = response.get('Item', None) + print(item) + + return item + + def update_project_info(self, identity_id, project_name, data_type, data_number): + response = self.table.update_item( + Key={ + self.FIELD_IDENTITY_ID: identity_id, + self.FIELD_PROJECT_NAME: project_name, + }, + ExpressionAttributeNames= { + '#DA_TY': data_type, + '#UP_DATE': self.FIELD_UPDATE_DATE + }, + ExpressionAttributeValues = { + ':va': data_number, + ':da': convert_current_date_to_iso8601(), + }, + UpdateExpression = 'SET #DA_TY = :va , #UP_DATE = :da' + ) + + return + + def update_project_gen_status(self, identity_id, project_name, status): + response = self.table.update_item( + Key={ + 'identity_id': identity_id, + 'project_name': project_name, + }, + ExpressionAttributeValues={ + ':st': status, + }, + UpdateExpression='SET gen_status = :st' + ) + + return + + def update_project_gen_status_category_default(self, identity_id, project_name, status, category_id): + response = self.table.update_item( + Key={ + 'identity_id': identity_id, + 'project_name': project_name, + }, + ExpressionAttributeValues={ + ':st': status, + ':cat': category_id, + ':up': convert_current_date_to_iso8601() + }, + ExpressionAttributeNames= { + '#GEN': self.FIELD_GEN_STATUS, + '#CAT': self.FIELD_CATEGORY_DEFAULT, + '#UP': self.FIELD_UPDATE_DATE, + }, + UpdateExpression='SET #GEN = :st, #CAT = :cat, #UP = :up' + ) + + return + + # def update_project_attributes(self, identity_id, project_name, ls_attributes_info): + # """ + # update the attributes of project in general input parameters, the update expression default will be set + + # params: + # - ls_attributes_info: should be in format [[, ], [, ], ...] + # """ + # ex_attri_name = {} + # ex_value = {} + # ls_update_ex = [] + # for name, value in ls_attributes_info: + # ex_attri_name + + + def update_project_generate_times(self, identity_id, project_name, times_augment, times_preprocess, + reference_images, aug_params): + response = self.table.update_item( + Key={ + self.FIELD_IDENTITY_ID: identity_id, + self.FIELD_PROJECT_NAME: project_name, + }, + ExpressionAttributeNames= { + '#P_T': self.FIELD_TIMES_PREPRO, + '#A_T': self.FIELD_TIMES_AUGMENT, + '#UP_DATE': self.FIELD_UPDATE_DATE, + "#RE_IM": self.FIELD_REFERENCE_IMAGES, + "#AU_PA": self.FIELD_AUG_PARAMETERS + }, + ExpressionAttributeValues = { + ':vp_t': times_preprocess, + ':va_t': times_augment, + ':da': convert_current_date_to_iso8601(), + ':re_im': reference_images, + ':au_pa': aug_params + }, + UpdateExpression = 'SET #P_T = :vp_t , #A_T = :va_t, #AU_PA = :au_pa, #UP_DATE = :da, #RE_IM = :re_im' + ) + + return + + def update_generate_expert_mode_param(self, identity_id, project_name, reference_images, aug_params): + response = self.table.update_item( + Key={ + self.FIELD_IDENTITY_ID: identity_id, + self.FIELD_PROJECT_NAME: project_name, + }, + ExpressionAttributeNames= { + '#UP_DATE': self.FIELD_UPDATE_DATE, + "#RE_IM": self.FIELD_REFERENCE_IMAGES, + "#AU_PA": self.FIELD_AUG_PARAMETERS + }, + ExpressionAttributeValues = { + ':da': convert_current_date_to_iso8601(), + ':re_im': reference_images, + ':au_pa': aug_params + }, + UpdateExpression = 'SET #AU_PA = :au_pa, #UP_DATE = :da, #RE_IM = :re_im' + ) + + return + + def update_project_reference_images(self, identity_id, project_name, reference_images): + """ + update project reference images after finish calculate reference images with ls_method choosen by client + """ + response = self.table.update_item( + Key={ + self.FIELD_IDENTITY_ID: identity_id, + self.FIELD_PROJECT_NAME: project_name, + }, + ExpressionAttributeNames= { + '#UP_DATE': self.FIELD_UPDATE_DATE, + "#RE_IM": self.FIELD_REFERENCE_IMAGES, + }, + ExpressionAttributeValues = { + ':da': convert_current_date_to_iso8601(), + ':re_im': reference_images, + }, + UpdateExpression = 'SET #RE_IM = :re_im, #UP_DATE = :da' + ) + + return + + def put_item_w_condition(self, item, condition = None): + if condition is None: + self.table.put_item( + Item = item + ) + else: + self.table.put_item( + Item = item, + ConditionExpression = condition + ) + return + + def delete_project(self, identity_id, project_name): + response = self.table.delete_item( + Key={ + self.FIELD_IDENTITY_ID: identity_id, + self.FIELD_PROJECT_NAME: project_name, + }, + ReturnValues='ALL_OLD' + ) + return response.get("Attributes", None) + + def find_project_by_project_ID(self,project_id): + response = self.table.scan(FilterExpression=Attr("project_id").eq(project_id)) + return response['Items'] \ No newline at end of file diff --git a/daita-app/shared-layer/commons/python/models/annotaition/anno_project_sum_model.py b/daita-app/shared-layer/commons/python/models/annotaition/anno_project_sum_model.py new file mode 100644 index 0000000..06fc14f --- /dev/null +++ b/daita-app/shared-layer/commons/python/models/annotaition/anno_project_sum_model.py @@ -0,0 +1,174 @@ +import boto3 +from boto3.dynamodb.conditions import Key, Attr +from config import * +from utils import * + + +class AnnoProjectSumModel(): + + FIELD_IDENTITY_ID = "identity_id" + FIELD_PROJECT_ID = "project_id" + FIELD_TYPE = "type" + FIELD_IS_DELETED = "is_dele" + FIELD_UPDATED_DATE = "updated_date" + FIELD_COUNT = "count" ### total data in a project + FIELD_TOTAL_SIZE = "total_size" + FIELD_THUM_KEY = "thu_key" + FIELD_THUM_FILENAME = "thu_name" + FIELD_NUM_EXIST_DATA = "num_exist_data" + + + VALUE_TYPE_ORIGINAL = "ORIGINAL" + + def __init__(self, table_name) -> None: + self.table = boto3.resource('dynamodb').Table(table_name) + + def reset_prj_sum_preprocess(self, project_id, type_data): + response = self.table.update_item( + Key={ + self.FIELD_PROJECT_ID: project_id, + self.FIELD_TYPE: type_data, + }, + ExpressionAttributeNames={ + '#CO': 'count', + '#TS': 'total_size' + }, + ExpressionAttributeValues={ + ':ts': 0, + ':co': 0 + }, + UpdateExpression='SET #TS = :ts, #CO = :co' + ) + return response + + def update_project_sum(self, project_id, type_data, total_size, count, thu_key, thu_name): + try: + response = self.table.update_item( + Key={ + self.FIELD_PROJECT_ID: project_id, + self.FIELD_TYPE: type_data, + }, + ExpressionAttributeNames={ + '#SI': 'total_size', + '#COU': 'count', + '#TK': 'thu_key', + '#TN': 'thu_name' + }, + ExpressionAttributeValues={ + ':si': total_size, + ':cou': count, + ':tk': thu_key, + ':tn': thu_name + }, + UpdateExpression='SET #TK = :tk, #TN = :tn ADD #SI :si, #COU :cou' + ) + except Exception as e: + print('Error: ', repr(e)) + raise Exception(repr(e)) + + def get_item(self, project_id, type_data): + response = self.table.get_item( + Key={ + 'project_id': project_id, + 'type': type_data, + } + ) + print(response) + if 'Item' in response: + return response['Item'] + elif 'ResponseMetadata' in response and response['ResponseMetadata']['HTTPStatusCode'] == 200: + return {'count': 0} + return None + + def get_current_number_data_in_prj(self, project_id, type_data = VALUE_TYPE_ORIGINAL): + response = self.table.get_item( + Key={ + self.FIELD_PROJECT_ID: project_id, + self.FIELD_TYPE: type_data, + } + ) + + if 'Item' in response: + count = int(response['Item'].get(self.FIELD_COUNT, 0)) + else: + count = 0 + + return count + + def get_item_prj_sum_info(self, project_id, type_data = VALUE_TYPE_ORIGINAL, ls_fields_projection = []): + if len(ls_fields_projection)==0: + response = self.table.get_item( + Key={ + self.FIELD_PROJECT_ID: project_id, + self.FIELD_TYPE: type_data, + } + # ProjectionExpression= ",".join(ls_fields_projection) + ) + # ls_fields_projection = [self.FIELD_COUNT, self.FIELD_TOTAL_SIZE, self.FIELD_THUM_KEY, self.FIELD_THUM_FILENAME] + else: + response = self.table.get_item( + Key={ + self.FIELD_PROJECT_ID: project_id, + self.FIELD_TYPE: type_data, + }, + ProjectionExpression= ",".join(ls_fields_projection) + ) + item = response.get('Item', None) + + return item + + + def query_data_project_id(self, project_id): + response = self.table.query( + KeyConditionExpression=Key(self.FIELD_PROJECT_ID).eq(project_id), + ) + + return response.get("Items", []) + + def update_deleted_status(self, project_id, type_data): + response = self.table.update_item( + Key={ + self.FIELD_PROJECT_ID: project_id, + self.FIELD_TYPE: type_data, + }, + ExpressionAttributeNames={ + '#IS_DE': self.FIELD_IS_DELETED, + '#UP_DATE': self.FIELD_UPDATED_DATE + }, + ExpressionAttributeValues={ + ':is_de': True, + ':da': convert_current_date_to_iso8601(), + }, + UpdateExpression='SET #UP_DATE = :da, #IS_DE = :is_de' + ) + + return response + + def update_upload_new_data(self, project_id, total_size, count, num_final, + thum_s3_key, thum_filename, type_data = VALUE_TYPE_ORIGINAL): + exp_att_name = { + '#SI': self.FIELD_TOTAL_SIZE, + '#COU': self.FIELD_COUNT, + '#NE': self.FIELD_NUM_EXIST_DATA, + '#TK': self.FIELD_THUM_KEY, + '#TN': self.FIELD_THUM_FILENAME + } + exp_att_value = { + ':si': total_size, + ':cou': count, + ':ne': num_final, + ':tk': thum_s3_key, + ':tn': thum_filename + } + + response = self.table.update_item( + Key={ + self.FIELD_PROJECT_ID: project_id, + self.FIELD_TYPE: type_data, + }, + ExpressionAttributeNames = exp_att_name, + ExpressionAttributeValues = exp_att_value, + UpdateExpression = 'SET #NE = :ne, #TK = :tk, #TN = :tn ADD #SI :si, #COU :cou', + ) + + return \ No newline at end of file diff --git a/daita-app/shared-layer/commons/python/models/base_model.py b/daita-app/shared-layer/commons/python/models/base_model.py new file mode 100644 index 0000000..a33679f --- /dev/null +++ b/daita-app/shared-layer/commons/python/models/base_model.py @@ -0,0 +1,50 @@ + + +from decimal import Decimal + + +class BaseModel(): + def __init__(self) -> None: + pass + + def convert_decimal_ls_item(self, ls_item): + return [self.convert_decimal_indb_item(item) for item in ls_item] + + def convert_decimal_indb_item(self, item): + if item: + for key, value in item.items(): + if type(value) is Decimal: + item[key] = int(value) + + return item + + def put_item_w_condition(self, item, condition): + self.table.put_item( + Item = item, + ConditionExpression = condition + ) + return + + def batch_write(self, ls_item_request): + try: + with self.table.batch_writer() as batch: + for item in ls_item_request: + batch.put_item(Item=item) + except Exception as e: + print('Error: ', repr(e)) + raise Exception(repr(e)) + + + """ + ### Helper class to convert a DynamoDB item to JSON. + class DecimalEncoder(json.JSONEncoder): + def default(self, o): + if isinstance(o, decimal.Decimal): + if o % 1 > 0: + return float(o) + else: + return int(o) + return super(DecimalEncoder, self).default(o) + + print(json.dumps(item, indent=4, cls=DecimalEncoder)) + """ \ No newline at end of file diff --git a/daita-app/shared-layer/commons/python/models/data_model.py b/daita-app/shared-layer/commons/python/models/data_model.py index d17f62d..df9a825 100644 --- a/daita-app/shared-layer/commons/python/models/data_model.py +++ b/daita-app/shared-layer/commons/python/models/data_model.py @@ -3,6 +3,7 @@ from config import * from typing import List from utils import convert_current_date_to_iso8601 +from models.base_model import BaseModel class DataItem(): @@ -54,7 +55,7 @@ def to_dict(self, request = REQUEST_TYPE_ALL): dict_info = { self.FIELD_PROJECT_ID: self.project_id, self.FIELD_FILENAME: self.filename, - self.GEN_ID: self.gen_id, + self.FIELD_GEN_ID: self.gen_id, self.FIELD_HASH: self.hash, self.FIELD_IS_ORIGINAL: self.is_original, self.FIELD_S3_KEY: self.s3_key, @@ -66,7 +67,7 @@ def to_dict(self, request = REQUEST_TYPE_ALL): return dict_info -class DataModel(): +class DataModel(BaseModel): def __init__(self, table_name) -> None: self.table = boto3.resource('dynamodb').Table(table_name) @@ -114,13 +115,24 @@ def get_all_wo_healthcheck_id(self, project_id: str): return ls_s3_key def get_all_data_in_project(self, project_id): + response = self.table.query ( KeyConditionExpression=Key(DataItem.FIELD_PROJECT_ID).eq(project_id), - ProjectionExpression=f"{DataItem.FIELD_FILENAME}, {DataItem.FIELD_S3_KEY}", + ProjectionExpression=f"{DataItem.FIELD_FILENAME}, {DataItem.FIELD_S3_KEY}, {DataItem.FIELD_SIZE}", + ) + ls_items = response.get("Items", []) + + while 'LastEvaluatedKey' in response: + next_token = response['LastEvaluatedKey'] + response = self.table.query ( + KeyConditionExpression=Key(DataItem.FIELD_PROJECT_ID).eq(project_id), + ProjectionExpression=f"{DataItem.FIELD_FILENAME}, {DataItem.FIELD_S3_KEY}, {DataItem.FIELD_SIZE}", + ExclusiveStartKey=next_token ) - items = response.get("Items", []) - return items + ls_items += response.get("Items", []) + + return [self.convert_decimal_indb_item(item) for item in ls_items] def update_healthcheck_id(self, project_id, filename, healthcheck_id): self._update_healthcheck_id(project_id, filename, healthcheck_id) diff --git a/daita-app/shared-layer/commons/python/models/event_model.py b/daita-app/shared-layer/commons/python/models/event_model.py new file mode 100644 index 0000000..5bb89b2 --- /dev/null +++ b/daita-app/shared-layer/commons/python/models/event_model.py @@ -0,0 +1,78 @@ +import boto3 +import os +class EventUser: + def __init__(self): + self.db_client = boto3.resource('dynamodb',region_name=os.environ['REGION']) + self.TBL = os.environ['TABLE_EVENTUSER'] + def create_item(self,info): + self.db_client.Table(self.TBL).put_item(Item={ + 'event_ID':info['event_ID'], + 'type':info['type'] + }) + def create_item_cognito(self,info): + self.db_client.Table(self.TBL).put_item(Item={ + 'event_ID':info['event_ID'], + 'type':info['type'], + 'code':info['code'] + }) + + def delete_item(self,info): + self.db_client.Table(self.TBL).delete_item( + Key={ + 'event_ID': info['event_ID'], + 'type': info['type'] + } + ) + + def get_code_oauth2(self,info): + response = self.db_client.Table(self.TBL).get_item( + Key={ + 'event_ID':info['event_ID'], + 'type':info['type'] + } + ) + return response['Item'] + + def find_item(self,info): + response = self.db_client.Table(self.TBL).get_item( + Key={ + 'event_ID':info['event_ID'], + 'type':info['type'] + } + ) + return True if 'Item' in response else False +model = EventUser() + + +def CreateEventUserLogin(sub): + model.create_item(info={ + 'event_ID':sub, + 'type':'AUTH' + }) + +def CreateEventUserLoginOauth2(sub,code): + model.create_item_cognito(info={ + 'event_ID':sub, + 'type':'AUTH', + 'code':code + }) + +def get_code_oauth2_cognito(sub): + return model.get_code_oauth2( + info={ + 'event_ID':sub, + 'type':'AUTH' + } + )['code'] + + +def CheckEventUserLogin(sub): + return model.find_item(info={ + 'event_ID':sub, + 'type':'AUTH' + }) +def EventUserLogout(sub): + model.delete_item(info={ + 'event_ID':sub, + 'type':'AUTH' + }) \ No newline at end of file diff --git a/daita-app/shared-layer/commons/python/models/feedback_model.py b/daita-app/shared-layer/commons/python/models/feedback_model.py new file mode 100644 index 0000000..ce2550c --- /dev/null +++ b/daita-app/shared-layer/commons/python/models/feedback_model.py @@ -0,0 +1,24 @@ +import os +import boto3 + +class Feedback(object): + def __init__(self): + self.db_client = boto3.resource('dynamodb') + self.TBL = os.environ['TABLE_FEEBACK'] + + def CreateItem(self, info): + self.db_client.Table(self.TBL).put_item(Item={ + "ID": info["ID"], + "name": info["name"], + "content": info["content"], + "images": info["images"], + "created_time": info["created_time"], + }) + + def CheckKeyIsExist(self, ID): + response = self.db_client.Table(self.TBL).get_item(Key={ + "ID": ID + }) + if 'Item' in response: + return True + return False \ No newline at end of file diff --git a/daita-app/shared-layer/commons/python/models/generate_daita_upload_token.py b/daita-app/shared-layer/commons/python/models/generate_daita_upload_token.py new file mode 100644 index 0000000..c41ac0c --- /dev/null +++ b/daita-app/shared-layer/commons/python/models/generate_daita_upload_token.py @@ -0,0 +1,110 @@ +import boto3 +import time +import random +import hashlib +from datetime import datetime +from boto3.dynamodb.conditions import Key, Attr +from config import * +from utils import * + +stringRandom = '0123456789qwertyuiopasdfghjklzxcvbnm' + + +def randomToken(): + m = hashlib.md5() + now = str(datetime.now()) + m.update(now.encode('utf-8')) + token = m.hexdigest() + for _ in range(0, 15): + token += stringRandom[random.randint(0, len(stringRandom)-1)] + return token + + +class GenerateDaitaUploadTokenkItem(): + FIELD_IDENTITY_ID = "identity_id" + FIELD_PROJECT_ID = "project_id" + FIELD_TOKEN = "token" + FIELD_ID_TOKEN = "id_token" + FIELD_TTL = "time_to_live" + FIELD_PROJECT_NAME = "project_name" + + def __init__(self) -> None: + self.identity_id = "" + self.token = "" + self.id_token = "" + self.project_id = "" + self.ttl = 0 + self.project_name = "" + + def to_dict(self): + dict_info = { + self.FIELD_IDENTITY_ID: self.identity_id, + self.FIELD_PROJECT_ID: self.project_id, + self.FIELD_TOKEN: self.token, + self.FIELD_TTL: self.ttl, + self.FIELD_ID_TOKEN: self.id_token, + self.FIELD_PROJECT_NAME: self.project_name + } + return dict_info + + def from_db_item(self, item_info): + self.identity_id = item_info.get(self.FIELD_IDENTITY_ID) + self.token = item_info.get(self.FIELD_TOKEN) + self.project_id = item_info.get(self.FIELD_PROJECT_ID) + self.ttl = item_info.get(self.FIELD_TTL) + self.id_token = item_info.get(self.FIELD_ID_TOKEN) + self.project_name = item_info.get(self.FIELD_PROJECT_NAME) + return self + + @classmethod + def create_new_item(cls, id_token, identity_id, project_id, project_name): + object = cls() + object.token = randomToken() + object.identity_id = identity_id + object.project_id = project_id + object.id_token = id_token + object.project_name = project_name + object.ttl = 60*60 + int(time.time()) + return object + + +class GenerateDaitaUploadTokenModel(): + + def __init__(self, table_name) -> None: + self.table = boto3.resource('dynamodb').Table(table_name) + self.IndexToken = table_name + '-1' + + def insert_new_token(self, item) -> None: + response = self.table.put_item( + Item=item.to_dict() + ) + return + + def token_exsited(self, identity_id, project_id): + response = self.table.query(KeyConditionExpression=Key('identity_id').eq( + identity_id), FilterExpression=Attr('project_id').eq(project_id)) + + if len(response['Items']) <= 0: + return None + + return response['Items'][0] + + def create_new_token(self, id_token, identity_id, project_id, project_name): + generate_token_item = GenerateDaitaUploadTokenkItem.create_new_item(id_token=id_token, + identity_id=identity_id, project_id=project_id, project_name=project_name) + + self.insert_new_token(generate_token_item) + return generate_token_item.token + + def query_by_token(self, token): + resp = self.table.query( + IndexName=self.IndexToken, + KeyConditionExpression=Key('token').eq(token), + Limit=1, + ScanIndexForward=False + ) + + if len(resp['Items']) <= 0: + return None + + return resp['Items'][0] diff --git a/daita-app/shared-layer/commons/python/models/generate_task_model.py b/daita-app/shared-layer/commons/python/models/generate_task_model.py index 51d1403..bc00769 100644 --- a/daita-app/shared-layer/commons/python/models/generate_task_model.py +++ b/daita-app/shared-layer/commons/python/models/generate_task_model.py @@ -21,25 +21,25 @@ class GenerateTaskItem(): FIELD_EXECUTEARN = 'ExecutionArn' FIELD_WAITINGINQUEUE = 'waiting_in_queue' FIELD_MESSAGEINFLIGHT = 'messages_in_flight' - REQUEST_TYPE_ALL = "all" - REQUEST_TYPE_TASK_PROGRESS = "task_progress" + REQUEST_TYPE_ALL = "all" + REQUEST_TYPE_TASK_PROGRESS = "task_progress" def __init__(self) -> None: - self.identity_id = "" - self.task_id = "" - self.status = "" - self.type_method = "" - self.number_finished = 0 - self.number_gen_images = 0 - self.project_id = "" - self.created_time = convert_current_date_to_iso8601() - self.updated_time = convert_current_date_to_iso8601() - self.process_type = "" - self.executeArn = "" - self.waitingInQueue = True + self.identity_id = "" + self.task_id = "" + self.status = "" + self.type_method = "" + self.number_finished = 0 + self.number_gen_images = 0 + self.project_id = "" + self.created_time = convert_current_date_to_iso8601() + self.updated_time = convert_current_date_to_iso8601() + self.process_type = "" + self.executeArn = "" + self.waitingInQueue = True self.messages_in_flight = 0 - def to_dict(self, request = REQUEST_TYPE_ALL): + def to_dict(self, request=REQUEST_TYPE_ALL): print(self.__dict__) if request == self.REQUEST_TYPE_TASK_PROGRESS: dict_info = { @@ -50,9 +50,9 @@ def to_dict(self, request = REQUEST_TYPE_ALL): self.FIELD_NUMBER_FINISHED: self.number_finished, self.FIELD_NUM_GEN_IMAGES: self.number_gen_images, self.FIELD_PROJECT_ID: self.project_id, - self.FIELD_EXECUTEARN : self.executeArn, + self.FIELD_EXECUTEARN: self.executeArn, self.FIELD_WAITINGINQUEUE: self.waitingInQueue, - self.FIELD_MESSAGEINFLIGHT : self.messages_in_flight + self.FIELD_MESSAGEINFLIGHT: self.messages_in_flight } else: dict_info = { @@ -66,9 +66,9 @@ def to_dict(self, request = REQUEST_TYPE_ALL): self.FIELD_CREATE_TIME: self.created_time, self.FIELD_UPDATE_TIME: self.updated_time, self.FIELD_PROCESS_TYPE: self.process_type, - self.FIELD_EXECUTEARN : self.executeArn, + self.FIELD_EXECUTEARN: self.executeArn, self.FIELD_WAITINGINQUEUE: self.waitingInQueue, - self.FIELD_MESSAGEINFLIGHT : self.messages_in_flight + self.FIELD_MESSAGEINFLIGHT: self.messages_in_flight } return dict_info @@ -76,18 +76,21 @@ def from_db_item(self, item_info): if item_info is None: return None else: - self.identity_id = item_info.get(self.FIELD_IDENTITY_ID) - self.task_id = item_info.get(self.FIELD_TASK_ID) - self.updated_time = item_info.get(self.FIELD_UPDATE_TIME) - self.status = item_info.get(self.FIELD_STATUS) - self.process_type = item_info.get(self.FIELD_PROCESS_TYPE) - self.number_finished = int(item_info.get(self.FIELD_NUMBER_FINISHED)) - self.number_gen_images = int(item_info.get(self.FIELD_NUM_GEN_IMAGES)) - self.project_id = item_info.get(self.FIELD_PROJECT_ID) - self.executeArn = item_info.get(self.FIELD_EXECUTEARN) - self.waitingInQueue = item_info.get(self.FIELD_WAITINGINQUEUE) - self.created_time = item_info.get(self.FIELD_CREATE_TIME) - self.messages_in_flight = int(item_info.get(self.FIELD_MESSAGEINFLIGHT)) + self.identity_id = item_info.get(self.FIELD_IDENTITY_ID) + self.task_id = item_info.get(self.FIELD_TASK_ID) + self.updated_time = item_info.get(self.FIELD_UPDATE_TIME) + self.status = item_info.get(self.FIELD_STATUS) + self.process_type = item_info.get(self.FIELD_PROCESS_TYPE) + self.number_finished = int( + item_info.get(self.FIELD_NUMBER_FINISHED)) + self.number_gen_images = int( + item_info.get(self.FIELD_NUM_GEN_IMAGES)) + self.project_id = item_info.get(self.FIELD_PROJECT_ID) + self.executeArn = item_info.get(self.FIELD_EXECUTEARN) + self.waitingInQueue = item_info.get(self.FIELD_WAITINGINQUEUE) + self.created_time = item_info.get(self.FIELD_CREATE_TIME) + self.messages_in_flight = int( + item_info.get(self.FIELD_MESSAGEINFLIGHT)) return self @classmethod @@ -103,23 +106,24 @@ def create_new_generate_task(cls, identity_id, project_id, type_method): return object - class GenerateTaskModel(): def __init__(self, table_name) -> None: self.table = boto3.resource('dynamodb').Table(table_name) def query_running_tasks(self, identity_id, project_id): - response = self.table.query ( - KeyConditionExpression=Key(GenerateTaskItem.FIELD_IDENTITY_ID).eq(identity_id), - FilterExpression=Attr(GenerateTaskItem.FIELD_PROJECT_ID).eq(project_id) & - Attr(GenerateTaskItem.FIELD_STATUS).ne(VALUE_GENERATE_TASK_STATUS_FINISH) & - Attr(GenerateTaskItem.FIELD_STATUS).ne(VALUE_GENERATE_TASK_STATUS_ERROR) & - Attr(GenerateTaskItem.FIELD_STATUS).ne(VALUE_GENERATE_TASK_STATUS_CANCEL) - ) + response = self.table.query( + KeyConditionExpression=Key( + GenerateTaskItem.FIELD_IDENTITY_ID).eq(identity_id), + FilterExpression=Attr(GenerateTaskItem.FIELD_PROJECT_ID).eq(project_id) & + Attr(GenerateTaskItem.FIELD_STATUS).ne(VALUE_GENERATE_TASK_STATUS_FINISH) & + Attr(GenerateTaskItem.FIELD_STATUS).ne(VALUE_GENERATE_TASK_STATUS_ERROR) & + Attr(GenerateTaskItem.FIELD_STATUS).ne( + VALUE_GENERATE_TASK_STATUS_CANCEL) + ) return response.get("Items", []) - def get_task_info(self, identity_id, task_id)->GenerateTaskItem: + def get_task_info(self, identity_id, task_id) -> GenerateTaskItem: response = self.table.get_item( Key={ GenerateTaskItem.FIELD_IDENTITY_ID: identity_id, @@ -131,18 +135,19 @@ def get_task_info(self, identity_id, task_id)->GenerateTaskItem: def insert_new_generate_task(self, item) -> None: response = self.table.put_item( - Item = item.to_dict() - ) + Item=item.to_dict() + ) return def create_new_generate_task(self, identity_id, project_id, type_method): - generate_task_item = GenerateTaskItem.create_new_generate_task(identity_id, project_id, type_method) + generate_task_item = GenerateTaskItem.create_new_generate_task( + identity_id, project_id, type_method) self.insert_new_generate_task(generate_task_item) return generate_task_item.task_id - def update_status(self,identity_id, task_id,status): + def update_status(self, identity_id, task_id, status): self.table.update_item( Key={ GenerateTaskItem.FIELD_IDENTITY_ID: identity_id, @@ -156,7 +161,8 @@ def update_status(self,identity_id, task_id,status): '#s': 'status', } ) - def update_task_dequeue(self,identity_id, task_id): + + def update_task_dequeue(self, identity_id, task_id): self.table.update_item( Key={ GenerateTaskItem.FIELD_IDENTITY_ID: identity_id, @@ -170,13 +176,14 @@ def update_task_dequeue(self,identity_id, task_id): '#e': 'waiting_in_queue', } ) - def update_messages_in_flight(self,identity_id, task_id,messages_in_flight): + + def update_messages_in_flight(self, identity_id, task_id, messages_in_flight): self.table.update_item( Key={ GenerateTaskItem.FIELD_IDENTITY_ID: identity_id, GenerateTaskItem.FIELD_TASK_ID: task_id, }, - ExpressionAttributeValues={ + ExpressionAttributeValues={ ':m': messages_in_flight }, ExpressionAttributeNames={ @@ -185,13 +192,14 @@ def update_messages_in_flight(self,identity_id, task_id,messages_in_flight): UpdateExpression="ADD #m :m", ReturnValues="UPDATED_NEW" ) - def init_messages_in_flight(self,identity_id, task_id): + + def init_messages_in_flight(self, identity_id, task_id): self.table.update_item( Key={ GenerateTaskItem.FIELD_IDENTITY_ID: identity_id, GenerateTaskItem.FIELD_TASK_ID: task_id, }, - ExpressionAttributeValues={ + ExpressionAttributeValues={ ':m': 0 }, ExpressionAttributeNames={ @@ -199,7 +207,8 @@ def init_messages_in_flight(self,identity_id, task_id): }, UpdateExpression="SET #m = :m", ) - def update_ExecutionArn(self,identity_id, task_id,executionArn): + + def update_ExecutionArn(self, identity_id, task_id, executionArn): self.table.update_item( Key={ GenerateTaskItem.FIELD_IDENTITY_ID: identity_id, diff --git a/daita-app/shared-layer/commons/python/models/prebuild_dataset_model.py b/daita-app/shared-layer/commons/python/models/prebuild_dataset_model.py new file mode 100644 index 0000000..697e243 --- /dev/null +++ b/daita-app/shared-layer/commons/python/models/prebuild_dataset_model.py @@ -0,0 +1,49 @@ +import boto3 +from decimal import Decimal +from boto3.dynamodb.conditions import Key, Attr + +class PrebuildDatasetModel(): + FILE_NAME_ID = "name" + FIELD_S3_KEY = "s3_key" + FIELD_VISUAL_NAME = "visual_name" + FIELD_IS_ACTIVE = "is_active" + FIELD_TOTAL_IMAGES = "total_images" + + def __init__(self, table_name) -> None: + self.tablename = table_name + self.table = boto3.resource('dynamodb').Table(table_name) + + def get_list_prebuild_dataset(self): + response = self.table.scan( + FilterExpression=Attr(self.FIELD_IS_ACTIVE).eq(True) + ) + items = response["Items"] + + return items + + def get_prebuild_dataset(self, name_id): + response = self.table.get_item( + Key = { + self.FILE_NAME_ID: name_id + } + ) + item = self.convert_item_to_json(response.get("Item", None)) + + return item + + def convert_item_to_json(self, item): + if item: + for key, value in item.items(): + if type(value) is Decimal: + item[key] = int(value) + + return item + + def get_const_with_code(self, code, type="THRESHOLD"): + response = self.table.get_item( + Key = { + 'code': code, + 'type': type + } + ) + return response.get("Item")[self.FIELD_NUM_VALUE] \ No newline at end of file diff --git a/daita-app/shared-layer/commons/python/models/project_model.py b/daita-app/shared-layer/commons/python/models/project_model.py index 62dd6a1..1d39306 100644 --- a/daita-app/shared-layer/commons/python/models/project_model.py +++ b/daita-app/shared-layer/commons/python/models/project_model.py @@ -27,7 +27,7 @@ def __init__(self, dict_info) -> None: self.__dict__[key]=value print("init success: ", self.__dict__) - def get_value_w_default(self, name, default_value): + def get_value_w_default(self, name, default_value=None): if self.__dict__.get(name, None): return self.__dict__[name] else: @@ -143,3 +143,10 @@ def update_project_reference_images(self, identity_id, project_name, reference_i ) return + + def put_item_w_condition(self, item, condition): + self.table.put_item( + Item = item, + ConditionExpression = condition + ) + return diff --git a/daita-app/shared-layer/commons/python/models/project_sum_model.py b/daita-app/shared-layer/commons/python/models/project_sum_model.py index 03928c1..fcb20de 100644 --- a/daita-app/shared-layer/commons/python/models/project_sum_model.py +++ b/daita-app/shared-layer/commons/python/models/project_sum_model.py @@ -6,29 +6,41 @@ class ProjectSumModel(): - FIELD_IDENTITY_ID = "identity_id" - FIELD_PROJECT_ID = "project_id" - FIELD_TYPE = "type" - + FIELD_IDENTITY_ID = "identity_id" + FIELD_PROJECT_ID = "project_id" + FIELD_TYPE = "type" def __init__(self, table_name) -> None: - self.table = boto3.resource('dynamodb').Table(table_name) + self.table = boto3.resource('dynamodb').Table(table_name) - def reset_prj_sum_preprocess(self, project_id, type_data): + def reset_prj_sum_preprocess(self, project_id, type_data): response = self.table.update_item( - Key={ - self.FIELD_PROJECT_ID: project_id, - self.FIELD_TYPE: type_data, - }, - ExpressionAttributeNames= { - '#CO': 'count', - '#TS': 'total_size' - }, - ExpressionAttributeValues = { - ':ts': 0, - ':co': 0 - }, - UpdateExpression = 'SET #TS = :ts, #CO = :co' - ) + Key={ + self.FIELD_PROJECT_ID: project_id, + self.FIELD_TYPE: type_data, + }, + ExpressionAttributeNames={ + '#CO': 'count', + '#TS': 'total_size' + }, + ExpressionAttributeValues={ + ':ts': 0, + ':co': 0 + }, + UpdateExpression='SET #TS = :ts, #CO = :co' + ) return response - + + def get_item(self, project_id, type_data): + response = self.table.get_item( + Key={ + 'project_id': project_id, + 'type': type_data, + } + ) + print(response) + if 'Item' in response: + return response['Item'] + elif 'ResponseMetadata' in response and response['ResponseMetadata']['HTTPStatusCode'] == 200: + return {'count': 0} + return None diff --git a/daita-app/shared-layer/commons/python/s3_utils.py b/daita-app/shared-layer/commons/python/s3_utils.py index d4bfb7f..06587e5 100644 --- a/daita-app/shared-layer/commons/python/s3_utils.py +++ b/daita-app/shared-layer/commons/python/s3_utils.py @@ -39,4 +39,46 @@ def generate_presigned_url(s3_link, expired=3600, bucket = None, object_key = No ExpiresIn=1*60*60 ) - return reponse \ No newline at end of file + return reponse + +def move_data_s3(source, target, bucket_name): + ls_info = [] + #list all data in s3 + s3 = boto3.resource('s3') + bucket = s3.Bucket(bucket_name) + + for obj in bucket.objects.filter(Prefix=source): + if obj.key.endswith('/'): + continue + + old_source = { 'Bucket': bucket_name, + 'Key': obj.key} + # replace the prefix + new_prefix = target.replace(f"{bucket_name}/", "") + new_key = f'{new_prefix}/{obj.key.replace(source, "")}' + s3.meta.client.copy(old_source, bucket_name, new_key) + + ls_info.append((new_key.split('/')[-1], f"{bucket_name}/{new_key}", obj.size)) + + return ls_info + +def separate_s3_uri(s3_uri, bucket_name): + """ + split the bucket_name and the key of file from s3 uri + EXP: + client-data-test/us-east-2:1d66c2cf-28c7-4222-b517-73c78c83f132/jhgjhgj_a95bac00047140429e027876861b7bcd, bucket_name = client-data-test + -> output: ["client-data-test", "us-east-2:1d66c2cf-28c7-4222-b517-73c78c83f132/jhgjhgj_a95bac00047140429e027876861b7bcd"] + s3://client-data-test/us-east-2:1d66c2cf-28c7-4222-b517-73c78c83f132/jhgjhgj_a95bac00047140429e027876861b7bcd, bucket_name = client-data-test + -> output: ["client-data-test", "us-east-2:1d66c2cf-28c7-4222-b517-73c78c83f132/jhgjhgj_a95bac00047140429e027876861b7bcd"] + """ + if not 's3://' in s3_uri[:2]: + temp = s3_uri.split('/') + + else: + s3_uri = s3_uri.replace("s3://", "") + temp = s3_uri.split('/') + + bucket = temp[0] + filename = '/'.join(temp[1:]) + + return bucket, filename \ No newline at end of file diff --git a/daita-app/shared-layer/commons/python/thumbnail.py b/daita-app/shared-layer/commons/python/thumbnail.py new file mode 100644 index 0000000..48fc910 --- /dev/null +++ b/daita-app/shared-layer/commons/python/thumbnail.py @@ -0,0 +1,8 @@ + +class Thumbnail(object): + def __init__(self,info) -> None: + self.project_ID = info['project_id'] + self.filename = info['filename'] + self.thumbnail = None + self.s3_url = info['s3_urls'] + self.table_name = info['table'] \ No newline at end of file diff --git a/daita-app/shared-layer/commons/python/utils.py b/daita-app/shared-layer/commons/python/utils.py index 43c07ad..222cadc 100644 --- a/daita-app/shared-layer/commons/python/utils.py +++ b/daita-app/shared-layer/commons/python/utils.py @@ -2,27 +2,35 @@ from decimal import Decimal import uuid from boto3.dynamodb.types import TypeDeserializer, TypeSerializer +from boto3.dynamodb.conditions import Key, Attr import re +import boto3 +import os +import json def convert_current_date_to_iso8601(): my_date = datetime.now() return my_date.isoformat() + def create_unique_id(): """ Create an unique id """ return str(uuid.uuid4()) + + def create_task_id_w_created_time(): """ Create unique id combine with current day for task_id of all task table """ return f"{convert_current_date_to_iso8601()}-{create_unique_id()}" + def from_dynamodb_to_json(item): # print(f"Item to serialize: \n {item}") - d = TypeDeserializer() + d = TypeDeserializer() serialize = {k: d.deserialize(value=v) for k, v in item.items()} for key, value in serialize.items(): @@ -32,16 +40,18 @@ def from_dynamodb_to_json(item): # print(f"Result after serialize: \n {serialize}") return serialize + def get_bucket_key_from_s3_uri(uri: str): if not 's3' in uri[:2]: temp = uri.split('/') bucket = temp[0] - filename = '/'.join([temp[i] for i in range(1,len(temp))]) + filename = '/'.join([temp[i] for i in range(1, len(temp))]) else: - match = re.match(r's3:\/\/(.+?)\/(.+)', uri) + match = re.match(r's3:\/\/(.+?)\/(.+)', uri) bucket = match.group(1) filename = match.group(2) - return bucket, filename + return bucket, filename + def split_ls_into_batch(ls_info, batch_size=8): """ @@ -50,14 +60,16 @@ def split_ls_into_batch(ls_info, batch_size=8): ls_batch = [] ls_batch_current = [] for info in ls_info: - if len(ls_batch_current)==batch_size: + if len(ls_batch_current) == batch_size: ls_batch.append(ls_batch_current) ls_batch_current = [] ls_batch_current.append(info) - if len(ls_batch_current)>0: + if len(ls_batch_current) > 0: ls_batch.append(ls_batch_current) - + return ls_batch + + def get_data_table_name(type_method): if type_method == 'ORIGINAL': return 'data_original' @@ -67,32 +79,58 @@ def get_data_table_name(type_method): return 'data_preprocess' else: raise Exception(f'Method {type_method} is not exist!') + +def get_num_prj(identity_id): + db_resource = boto3.resource("dynamodb") + table = db_resource.Table(os.environ['TABLE_PROJECT']) + response = table.query( + KeyConditionExpression=Key('identity_id').eq(identity_id), + Select = 'COUNT' + ) + return response.get("Count", 0) def get_table_dydb_object(db_resource, type_method): table_name = get_data_table_name(type_method) return db_resource.Table(table_name) -def dydb_update_prj_sum(table, project_id, type_method, count_add, size_add): - response = table.update_item( +def dydb_get_project_id(table, identity_id, project_name): + try: + response = table.get_item( Key={ - 'project_id': project_id, - 'type': type_method, - }, - ExpressionAttributeNames= { - '#CO': 'count', - '#TS': 'total_size' - }, - ExpressionAttributeValues = { - ':ts': -size_add, - ':co': -count_add + 'identity_id': identity_id, + 'project_name': project_name, }, - UpdateExpression = 'ADD #TS :ts, #CO :co' + ProjectionExpression= 'project_id' ) + except Exception as e: + print('Error: ', repr(e)) + raise + if response.get('Item', None): + return response['Item']['project_id'] + return None +def dydb_update_prj_sum(table, project_id, type_method, count_add, size_add): + response = table.update_item( + Key={ + 'project_id': project_id, + 'type': type_method, + }, + ExpressionAttributeNames={ + '#CO': 'count', + '#TS': 'total_size' + }, + ExpressionAttributeValues={ + ':ts': -size_add, + ':co': -count_add + }, + UpdateExpression='ADD #TS :ts, #CO :co' + ) + + def dydb_update_delete_project(table, table_project_delete, identity_id, project_name): # get current project response = table.get_item(Key={ - 'identity_id': identity_id, - 'project_name': project_name, - }) + 'identity_id': identity_id, + 'project_name': project_name, + }) if response.get('Item'): new_item = response['Item'] @@ -100,13 +138,94 @@ def dydb_update_delete_project(table, table_project_delete, identity_id, project # create new item in table project delete table_project_delete.put_item( - Item = new_item - ) + Item=new_item + ) # delete item in table project table.delete_item( - Key = { - 'identity_id': identity_id, - 'project_name': project_name, - } - ) \ No newline at end of file + Key={ + 'identity_id': identity_id, + 'project_name': project_name, + } + ) + + +def aws_get_identity_id(id_token, USER_POOL_ID, IDENTITY_POOL_ID): + identity_client = boto3.client('cognito-identity') + PROVIDER = f'cognito-idp.{identity_client.meta.region_name}.amazonaws.com/{USER_POOL_ID}' + + try: + identity_response = identity_client.get_id( + IdentityPoolId=IDENTITY_POOL_ID, + Logins={PROVIDER: id_token}) + except Exception as e: + print('Error: ', repr(e)) + raise + + identity_id = identity_response['IdentityId'] + + return identity_id + +def convert_response(data): + if data.get('message', None): + # print("convert: ",data['message']) + # print("type: ", type(data['message'])) + data['message'] = data['message'].replace("Exception('", "").replace("')", "") + return { + "statusCode": 200, + 'headers': { + 'Access-Control-Allow-Headers': 'Content-Type', + 'Access-Control-Allow-Origin': '*', + 'Access-Control-Allow-Methods': 'OPTIONS,POST,GET' + }, + "body": json.dumps(data), + } + +def move_data_s3(source, target, bucket_name): + ls_info = [] + #list all data in s3 + s3 = boto3.resource('s3') + bucket = s3.Bucket(bucket_name) + + for obj in bucket.objects.filter(Prefix=source): + if obj.key.endswith('/'): + continue + + old_source = { 'Bucket': bucket_name, + 'Key': obj.key} + # replace the prefix + new_prefix = target.replace(f"{bucket_name}/", "") + new_key = f'{new_prefix}/{obj.key.replace(source, "")}' + s3.meta.client.copy(old_source, bucket_name, new_key) + + ls_info.append((new_key.split('/')[-1], f"{bucket_name}/{new_key}", obj.size)) + + return ls_info + +def create_single_put_request(dict_value): + dict_re = { + 'PutRequest': { + 'Item': { + } + } + } + for key, value in dict_value.items(): + dict_re['PutRequest']['Item'][key] = { + value[0]: value[1] + } + return dict_re + +def dydb_get_project_full(table, identity_id, project_name): + try: + response = table.get_item( + Key={ + 'identity_id': identity_id, + 'project_name': project_name, + }, + ) + except Exception as e: + print('Error: ', repr(e)) + raise + if response.get('Item', None): + return response['Item'] + return None \ No newline at end of file diff --git a/daita-app/shared-layer/commons/python/verify_captcha.py b/daita-app/shared-layer/commons/python/verify_captcha.py new file mode 100644 index 0000000..3cbcd77 --- /dev/null +++ b/daita-app/shared-layer/commons/python/verify_captcha.py @@ -0,0 +1,21 @@ +from http import HTTPStatus + +import requests + +from config import * + + +def verify_captcha(token: str, site_key_google, secret_key_google): + payload = { + "secret": secret_key_google, + "sitekey": site_key_google, + "response": token + } + + response = requests.post( + ENDPOINTCAPTCHAVERIFY, + params=payload + ) + print(response.json()) + if not response.json()["success"]: + raise Exception("Verify captcha failed") diff --git a/daita-app/template.yaml b/daita-app/template.yaml index 0ad7de9..663d133 100644 --- a/daita-app/template.yaml +++ b/daita-app/template.yaml @@ -5,24 +5,73 @@ Description: > ## The general rule seems to be to use !Sub for in line substitutions and !ref for stand alone text Parameters: + Mode: + Type: String + Default: staging Stage: Type: String - Default: dev Application: Type: String - Default: daita + SecurityGroupIds: Type: String - Default: 'sg-0315a5ecee0dc69fe,sg-0b3b2fcc4dc7686ad,sg-af50cbde,sg-07c27f59bc172f180,sg-0796222bd5149736f' SubnetIDs: Type: String - Default: subnet-31ff5b5a + S3BucketName: Type: String - Default: daita-client-data + S3AnnoBucket: + Type: String + + VPCid: + Type: String + EFSFileSystemId: + Type: String MaxConcurrencyTasks: Type: String Default: 3 + ROOTEFS: + Type: String + + DomainUserPool: + Type: String + DomainDaita: + Type: String + + ### parameter from infra + PublicSubnetOne: + Type: String + PublicSubnetTwo: + Type: String + ContainerSecurityGroup: + Type: String + VPC: + Type: String + + CertificateUserpoolDomain: + Type: String + + ### for token key + TokenOauth2BotSlackFeedBack: + Type: String + GoogleClientID: + Type: String + GoogleClientSecret: + Type: String + GithubClientID: + Type: String + GithubClientSecret: + Type: String + ChannelWebhook: + Type: String + Default: user-feedback + OauthEndpoint: + Type: String + CaptchaSiteKeyGoogle: + Type: String + CaptchaSecretKeyGoogle: + Type: String + Resources: #=============== SYSTEM PARAMETER CONFIGURATON ============================== LimitPreprocessTimes: @@ -30,7 +79,7 @@ Resources: Properties: Name: !Sub "${Stage}-LimitPreprocessTimes" Type: String - Value: "1" + Value: "5" LimitAugmentTimes: Type: AWS::SSM::Parameter Properties: @@ -56,7 +105,6 @@ Resources: Type: String Value: "4000" - #=============== ECR ======================================================== DecompressEcrRepository: Type: AWS::ECR::Repository @@ -147,6 +195,7 @@ Resources: CompatibleRuntimes: - python3.8 + #================ APPLICATIONS ============================================= AICallerService: Type: AWS::Serverless::Application @@ -165,7 +214,20 @@ Resources: TableDataAugmentName: !GetAtt DatabaseService.Outputs.TableDataAugmentName TableDataOriginalName: !GetAtt DatabaseService.Outputs.TableDataOriginalName TableDataPreprocessName: !GetAtt DatabaseService.Outputs.TableDataPreprocessName + TableProjectSumName: !GetAtt DatabaseService.Outputs.TableProjectSumName + TableProjectsName: !GetAtt DatabaseService.Outputs.TableProjectsName + TableLsEc2Name: !GetAtt DatabaseService.Outputs.TableLsEc2Name MaxConcurrencyTasks: !Ref MaxConcurrencyTasks + EFSFileSystemId: !Ref EFSFileSystemId + ROOTEFS: !Ref ROOTEFS + Mode: !Ref Mode + TaskAIPreprocessingDefinition: !GetAtt ECSAICallerApp.Outputs.TaskAIPreprocessingDefinition + TaskAIAugmentationDefinition: !GetAtt ECSAICallerApp.Outputs.TaskAIAugmentationDefinition + FuncProjectUploadUpdate: !GetAtt ProjectService.Outputs.FuncProjectUploadUpdate + ReferenceImageStateMachineArn: !GetAtt ReferenceImageService.Outputs.ReferenceImageStateMachineArn + RICalculateFunction: !GetAtt ReferenceImageService.Outputs.RICalculateFunction + RIStatusFunction: !GetAtt ReferenceImageService.Outputs.RIStatusFunction + RIInfoFunction: !GetAtt ReferenceImageService.Outputs.RIInfoFunction DataFlowService: Type: AWS::Serverless::Application @@ -180,6 +242,15 @@ Resources: CompressDownloadEcrRepositoryName: !Ref CompressDownloadEcrRepository TableMethodsName: !GetAtt DatabaseService.Outputs.TableMethodsName CommonCodeLayerName: !Ref CommonCodeLayer + VpcId: !Ref VPCid + SubnetIds: !Ref SubnetIDs + EFSFileSystemId: !Ref EFSFileSystemId + SecurityGroupIds: !Ref SecurityGroupIds + Mode: !Ref Mode + TableDataOriginalName: !GetAtt DatabaseService.Outputs.TableDataOriginalName + TableDataAugmentName: !GetAtt DatabaseService.Outputs.TableDataAugmentName + TableDataPreprocessName: !GetAtt DatabaseService.Outputs.TableDataPreprocessName + FuncProjectUploadUpdate: !GetAtt ProjectService.Outputs.FuncProjectUploadUpdate DatabaseService: Type: AWS::Serverless::Application @@ -187,19 +258,84 @@ Resources: Location: db-service/db_template.yaml Parameters: StagePara: !Ref Stage + ApplicationPara: !Ref Application + ProjectService: + Type: AWS::Serverless::Application + Properties: + Location: project-service/project_service_template.yaml + Parameters: + StagePara: !Ref Stage + Mode: !Ref Mode + CommonCodeLayerName: !Ref CommonCodeLayer + ApplicationPara: !Ref Application + LambdaRole: !GetAtt GeneralLambdaExecutionRole.Arn + TableProjectSumName: !GetAtt DatabaseService.Outputs.TableProjectSumName + TableProjectsName: !GetAtt DatabaseService.Outputs.TableProjectsName + TableDataOriginalName: !GetAtt DatabaseService.Outputs.TableDataOriginalName + TableDataPreprocess: !GetAtt DatabaseService.Outputs.TableDataPreprocessName + TableDataAugment: !GetAtt DatabaseService.Outputs.TableDataAugmentName + CognitoUserPoolClient: !GetAtt CognitoClient.Outputs.UserPoolClientId + CognitoUserPool: !GetAtt CognitoUserPool.Outputs.UserPool + CognitoIdentityPoolId: !GetAtt IdentityPool.Outputs.IdentityPool + ThumbnailEventBus: temp #!GetAtt CoreService.Outputs.ThumbnailEventBus + + CognitoUserPool: + Type: AWS::Serverless::Application + Properties: + Location: auth-service/CognitoUserPool/template.yaml + Parameters: + StagePara: !Ref Stage + Mode: !Ref Mode + CommonCodeLayerName: !Ref CommonCodeLayer + CertificateUserpoolDomain: !Ref CertificateUserpoolDomain + DomainUserPool: !Ref DomainUserPool + + CognitoClient: + Type: AWS::Serverless::Application + Properties: + Location: auth-service/CognitoClient/template.yaml + Parameters: + DomainDaita: !Ref DomainDaita + CognitoUserPool: !GetAtt CognitoUserPool.Outputs.UserPool + StagePara: !Ref Stage + AuthHttpAPI: !GetAtt CognitoUserPool.Outputs.AuthHttpAPI + GoogleClientID: !Ref GoogleClientID + GoogleClientSecret: !Ref GoogleClientSecret + GithubClientID: !Ref GithubClientID + GithubClientSecret: !Ref GithubClientSecret + + IdentityPool: + Type: AWS::Serverless::Application + Properties: + Location: auth-service/IdentityPool/template.yaml + Parameters: + UserPoolClientId: !GetAtt CognitoClient.Outputs.UserPoolClientId + ProviderNameUserPool: !GetAtt CognitoUserPool.Outputs.ProviderNameUserPool + + RoleIdentityS3: + Type: AWS::Serverless::Application + Properties: + Location: auth-service/RoleIdentity/template.yaml + Parameters: + Bucket: !Ref S3BucketName + IdentityPool: !GetAtt IdentityPool.Outputs.IdentityPool + StagePara: !Ref Stage + ApplicationPara: !Ref Application + AnnoBucket: !Ref S3AnnoBucket + CoreService: Type: AWS::Serverless::Application Properties: Location: core-service/core_service_template.yaml Parameters: + ApplicationPara: !Ref Application StagePara: !Ref Stage CallerServiceEventBusArn: !GetAtt AICallerService.Outputs.CallerServiceEventBusArn CallerServiceEventBusName: !GetAtt AICallerService.Outputs.CallerServiceEventBusName StopProcessEventBusArn: !GetAtt AICallerService.Outputs.StopProcessEventBusArn StopProcessEventBusName: !GetAtt AICallerService.Outputs.StopProcessEventBusName TableGenerateTaskName: !GetAtt DatabaseService.Outputs.TableGenerateTaskName - TableProjectsName: !GetAtt DatabaseService.Outputs.TableProjectsName TableMethodsName: !GetAtt DatabaseService.Outputs.TableMethodsName TableHealthCheckTasksName: !GetAtt DatabaseService.Outputs.TableHealthCheckTasksName TableHealthCheckInfoName: !GetAtt DatabaseService.Outputs.TableHealthCheckInfoName @@ -213,21 +349,56 @@ Resources: ParaTableDataFlowTaskName: !GetAtt DatabaseService.Outputs.TableDataFlowTaskName ParaDecompressFileStateMachineArn: !GetAtt DataFlowService.Outputs.DecompressFileStateMachineArn ParaCompressDownloadStateMachineArn: !GetAtt DataFlowService.Outputs.CompressDownloadStateMachineArn - ParaTableDownloadTaskName: !GetAtt DatabaseService.Outputs.TableDownloadTaskName - ParaTableProjectSumName: !GetAtt DatabaseService.Outputs.TableProjectSumName ParaIndexTaskProjectIDTaskID: !GetAtt DatabaseService.Outputs.IndexTaskProjectIDTaskIDName ProcessAITaskEventBusArn: !GetAtt AICallerService.Outputs.ProcessAITaskEventBusArn ProcessAITaskEventBusName: !GetAtt AICallerService.Outputs.ProcessAITaskEventBusName + ### for ECS AI Caller + AICallerECSSMArn: !GetAtt AICallerService.Outputs.AICallerECSSMArn + ### for project table + TableConstPrebuildDatasetName: !GetAtt DatabaseService.Outputs.TableConstPrebuildDatasetName ### for reference image para TableReferenceImageTasksName: !GetAtt DatabaseService.Outputs.TableReferenceImageTasksName TableReferenceImageInfoName: !GetAtt DatabaseService.Outputs.TableReferenceImageInfoName ReferenceImageEventBusName: !GetAtt ReferenceImageService.Outputs.ReferenceImageEventBusName + RICalculateFunctionArn: !GetAtt ReferenceImageService.Outputs.RICalculateFunctionArn + RIStatusFunctionArn: !GetAtt ReferenceImageService.Outputs.RIStatusFunctionArn + RIInfoFunctionArn: !GetAtt ReferenceImageService.Outputs.RIInfoFunctionArn ####### Data Preprocess Table TableDataPreprocessName: !GetAtt DatabaseService.Outputs.TableDataPreprocessName MaxConcurrencyTasks: !Ref MaxConcurrencyTasks TaskQueueName: !GetAtt AICallerService.Outputs.TaskQueueName TableDataOriginalName: !GetAtt DatabaseService.Outputs.TableDataOriginalName TableDataAugmentName: !GetAtt DatabaseService.Outputs.TableDataAugmentName + TableGenerateDaitaUploadToken: !GetAtt DatabaseService.Outputs.TableGenerateDaitaUploadToken + TableUser: !GetAtt DatabaseService.Outputs.TableUser + S3BucketName: !Ref S3BucketName + Mode: !Ref Mode + ########################################Auth Service############################################# + CognitoUserPoolClient: !GetAtt CognitoClient.Outputs.UserPoolClientId + CognitoUserPool: !GetAtt CognitoUserPool.Outputs.UserPool + CognitoIdentityPoolId: !GetAtt IdentityPool.Outputs.IdentityPool + TableEventUser: !GetAtt DatabaseService.Outputs.TableEventUser + ### For project service + CreateProjectPrebuildSMArn: !GetAtt ProjectService.Outputs.CreateProjectPrebuildSMArn + TableConfirmCodeAuth: !GetAtt DatabaseService.Outputs.TableConfirmCodeAuth + TableProjectsName: !GetAtt DatabaseService.Outputs.TableProjectsName + TableTask: !GetAtt DatabaseService.Outputs.TableTask + TableProjectSumName: !GetAtt DatabaseService.Outputs.TableProjectSumName + TableDataPreprocess: !GetAtt DatabaseService.Outputs.TableDataPreprocessName + TableDataAugment: !GetAtt DatabaseService.Outputs.TableDataAugmentName + TableFeedback: !GetAtt DatabaseService.Outputs.TableFeedback + StreamTableDataOriginalName: !GetAtt DatabaseService.Outputs.StreamTableDataOriginalName + StreamTableDataPreprocessName: !GetAtt DatabaseService.Outputs.StreamTableDataPreprocessName + StreamTableDataAugmentName: !GetAtt DatabaseService.Outputs.StreamTableDataAugmentName + ##### for project function + FuncProjectUploadUpdateArn: !GetAtt ProjectService.Outputs.FuncProjectUploadUpdateArn + AuthEndpoint: !GetAtt CognitoUserPool.Outputs.AuthHttpAPI + ### For token + TokenOauth2BotSlackFeedBack: !Ref TokenOauth2BotSlackFeedBack + ChannelWebhook: !Ref ChannelWebhook + OauthEndpoint: !Ref OauthEndpoint + CaptchaSiteKeyGoogle: !Ref CaptchaSiteKeyGoogle + CaptchaSecretKeyGoogle: !Ref CaptchaSecretKeyGoogle HealthCheckService: Type: AWS::Serverless::Application @@ -243,6 +414,7 @@ Resources: TableDataPreprocessName: !GetAtt DatabaseService.Outputs.TableDataPreprocessName CommonCodeLayerName: !Ref CommonCodeLayer ApplicationPara: !Ref Application + Mode: !Ref Mode ReferenceImageService: Type: AWS::Serverless::Application @@ -260,14 +432,54 @@ Resources: BatchsizeCalculateReferenceImage: !Ref BatchsizeCalculateReferenceImage MaxWidthResolutionImage: !Ref MaxWidthResolutionImage MaxHeightResolutionImage: !Ref MaxHeightResolutionImage + CognitoUserPoolClient: !GetAtt CognitoClient.Outputs.UserPoolClientId + CognitoUserPool: !GetAtt CognitoUserPool.Outputs.UserPool + CognitoIdentityPoolId: !GetAtt IdentityPool.Outputs.IdentityPool + Mode: !Ref Mode + ECSAICallerApp: + Type: AWS::Serverless::Application + Properties: + Location: ecs-ai-caller-app/template_ecs_ai_caller.yaml + Parameters: + StagePara: !Ref Stage + ApplicationPara: !Ref Application + ### params infra network + PublicSubnetOne: !Ref PublicSubnetOne + PublicSubnetTwo: !Ref PublicSubnetTwo + ContainerSecurityGroup: !Ref ContainerSecurityGroup + VPC: !Ref VPC Outputs: - ZZBC: - Value: Stage DecompressEcrRepositoryName: Value: !Ref DecompressEcrRepository CompressDownloadEcrRepositoryName: Value: !Ref CompressDownloadEcrRepository - ABC: - Value: !Ref Stage + + CognitoUserPoolRef: + Value: !GetAtt CognitoUserPool.Outputs.UserPool + CognitoIdentityPoolIdRef: + Value: !GetAtt IdentityPool.Outputs.IdentityPool + + CommonCodeLayerRef: + Value: !Ref CommonCodeLayer + + TableDaitaProjectsName: + Value: !GetAtt DatabaseService.Outputs.TableProjectsName + TableDaitaDataOriginalName: + Value: !GetAtt DatabaseService.Outputs.TableDataOriginalName + TableUserName: + Value: !GetAtt DatabaseService.Outputs.TableUser + + ApiDaitaAppUrl: + Value: !GetAtt CoreService.Outputs.CallerServiceHttpApiUrl + ApiAuthDaitaUrl: + Value: !GetAtt CognitoUserPool.Outputs.AuthApiURL + CognitoAppIntegrateID: + Value: !GetAtt CognitoClient.Outputs.UserPoolClientId + + TableUser: + Value: !GetAtt DatabaseService.Outputs.TableUser + + SendEmailIdentityIDFunction: + Value: !GetAtt CoreService.Outputs.SendEmailIdentityIDFunction \ No newline at end of file diff --git a/docs/build_instruction.md b/docs/build_instruction.md new file mode 100644 index 0000000..93a577f --- /dev/null +++ b/docs/build_instruction.md @@ -0,0 +1,20 @@ +1. Copy and replace env in [daita-app/samconfig.toml] , do not change the config inside +[dev.deploy.parameters] +s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-eu58g5l8is1s" +confirm_changeset = true +capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" +disable_rollback = true +image_repositories = [] + +2. Copy and replace env in [annotation-app/samconfig.toml] , do not change the config inside +[dev.deploy.parameters] +confirm_changeset = true +capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" +disable_rollback = true +image_repositories = [] + +3. replace dev env trong [main_config.cnf] +DAITA_STAGE=dev +ANNOTATION_STAGE=dev + +4. Use `bash main_build.sh` to build full flow. diff --git a/docs/url-endpoint-annotation-app.md b/docs/url-endpoint-annotation-app.md new file mode 100644 index 0000000..84979d2 --- /dev/null +++ b/docs/url-endpoint-annotation-app.md @@ -0,0 +1,299 @@ + + +URL_ROOT = [dev] `https://tq2sk9wusj.execute-api.us-east-2.amazonaws.com/dev` + +## Clone project from daita to annotation +- Path: [POST] `/annotation/project/clone_from_daita` +- Request + - Body: + + ```json + { + "id_token": {{id_token}}, + "anno_project_name": "test3_anno", + "daita_project_name": "test3", + "project_info": "test annotation" + } + ``` +- Response + - Http code: 200 + + ```json + { + "message": "OK", + "data": { + "project_id": "test3_anno_46795ea6be594903b4cf5c621b65405d", + ### use this value for add new image to s3 + "s3_prefix": "client-annotation-bucket/us-east-2:1fce7cc6-e794-4c1b-801c-77f5cda2f0da/test3_anno_46795ea6be594903b4cf5c621b65405d/raw_data", + "s3_prj_root": "client-annotation-bucket/us-east-2:1fce7cc6-e794-4c1b-801c-77f5cda2f0da/test3_anno_46795ea6be594903b4cf5c621b65405d", + "gen_status": "GENERATING", + "project_name": "test3_anno", + "link_daita_prj_id": "test3_03e9cf50593e420ab1e035cd7db7f0dd" + }, + "error": false + } + ``` + +## Get annotation project info +- Path: [POST] `/annotation/project/get_info` +- Request + - Body: + + ```json + { + "id_token": {{id_token}}, + "project_name": "test123_anno" + } + ``` + +- Response + - Http code: 200 + + ```json + { + "data": { + "identity_id": "us-east-2:706ba872-01d3-4e77-b2a1-759d711f97c6", + "project_name": "test123_anno", + "project_id": "test123_anno_d92bfec0b0984685961d94a78658344b", + "groups": { + "ORIGINAL": { + "count": 8, + "size": 26746970 + } + } + }, + "error": false, + "success": true, + "message": null + } + ``` + +## List all projects of a user +- Path: [POST] `/annotation/project/list_project` +- Request + - Body: + + ```json + { + "id_token": {{id_token}}, + } + ``` + +- Response + - Http code: 200 + + ```json + { + "message": "OK", + "data": { + "items": [ + { + "gen_status": "FINISH", + "project_name": "test123_anno", + "project_id": "test123_anno_d92bfec0b0984685961d94a78658344b", + "created_date": "2022-09-27T16:44:56.791574" + } + ] + }, + "error": false + } + ``` + +## List all files of a project +- Path: [POST] `/annotation/project/list_project` +- Request + - Body: + + ```json + { + "id_token": {{id_token}}, + "project_id": "test123_anno_d92bfec0b0984685961d94a78658344b", + "next_token": "", + "num_limit": 3 + } + ``` + +- Response + - Http code: 200 + + ```json + { + "message": "OK", + "data": { + "items": [ + { + "filename": "20190401145936_camera_sideright_000017967.png", + "size": 3096452, + "created_time": "2022-09-27T16:45:04.307939", + "s3_key": "client-annotation-bucket/us-east-2:706ba872-01d3-4e77-b2a1-759d711f97c6/test123_anno_d92bfec0b0984685961d94a78658344b/raw_data/20190401145936_camera_sideright_000017967.png" + }, + { + "filename": "20190401145936_camera_sideright_000017963.png", + "size": 3090755, + "created_time": "2022-09-27T16:45:04.307930", + "s3_key": "client-annotation-bucket/us-east-2:706ba872-01d3-4e77-b2a1-759d711f97c6/test123_anno_d92bfec0b0984685961d94a78658344b/raw_data/20190401145936_camera_sideright_000017963.png" + }, + { + "filename": "20190401145936_camera_sideright_000017937.png", + "size": 3025504, + "created_time": "2022-09-27T16:45:04.307921", + "s3_key": "client-annotation-bucket/us-east-2:706ba872-01d3-4e77-b2a1-759d711f97c6/test123_anno_d92bfec0b0984685961d94a78658344b/raw_data/20190401145936_camera_sideright_000017937.png" + } + ], + "next_token": { + "filename": "20190401145936_camera_sideright_000017937.png", + "project_id": "test123_anno_d92bfec0b0984685961d94a78658344b" + } + }, + "error": false + } + ``` + +- Request + - Body: + + ```json + { + "id_token": {{id_token}}, + "project_id": "test123_anno_d92bfec0b0984685961d94a78658344b", + "next_token": { + "filename": "20190401145936_camera_sideright_000017937.png", + "project_id": "test123_anno_d92bfec0b0984685961d94a78658344b" + }, + "num_limit": 3 + } + ``` + +- Response + - Http code: 200 + + ```json + { + "message": "OK", + "data": { + "items": [ + { + "filename": "20190401145936_camera_rearcenter_000017952.png", + "size": 3901712, + "created_time": "2022-09-27T16:45:04.307912", + "s3_key": "client-annotation-bucket/us-east-2:706ba872-01d3-4e77-b2a1-759d711f97c6/test123_anno_d92bfec0b0984685961d94a78658344b/raw_data/20190401145936_camera_rearcenter_000017952.png" + }, + { + "filename": "20190401145936_camera_frontright_000018002.png", + "size": 3006391, + "created_time": "2022-09-27T16:45:04.307903", + "s3_key": "client-annotation-bucket/us-east-2:706ba872-01d3-4e77-b2a1-759d711f97c6/test123_anno_d92bfec0b0984685961d94a78658344b/raw_data/20190401145936_camera_frontright_000018002.png" + }, + { + "filename": "20190401121727_camera_frontleft_000013485.png", + "size": 3706888, + "created_time": "2022-09-27T16:45:04.307894", + "s3_key": "client-annotation-bucket/us-east-2:706ba872-01d3-4e77-b2a1-759d711f97c6/test123_anno_d92bfec0b0984685961d94a78658344b/raw_data/20190401121727_camera_frontleft_000013485.png" + } + ], + "next_token": { + "filename": "20190401121727_camera_frontleft_000013485.png", + "project_id": "test123_anno_d92bfec0b0984685961d94a78658344b" + } + }, + "error": false + } + ``` + +## Create label category +- Path: [POST] `/annotation/file/create_lable_category` +- Request + - Body: + + ```json + { + "id_token": {{id_token}}, + "file_id": "90830ffc-3255-467f-b9e6-993a2aeeb0d1", + "category_name": "object segmentation", + "category_des": "object segmentation description" + } + ``` + +- Response + - Http code: 200 + + ```json + { + "message": "OK", + "data": { + "category_id": "4d9509ec-42d0-44c7-83eb-472e1d13e1c5", + "category_name": "object segmentation", + "category_des": "object segmentation description" + }, + "error": false + } + ``` + +## Save label +- Path: [POST] `/annotation/file/create_lable_category` +- Request + - Body: + + ```json + { + "id_token": {{id_token}}, + "file_id": "90830ffc-3255-467f-b9e6-993a2aeeb0d1", + "dict_s3_key": { + "4d9509ec-42d0-44c7-83eb-472e1d13e1c5": "s3_path_json_label" + } + } + ``` + +- Response + - Http code: 200 + + ```json + { + "message": "OK", + "data": {}, + "error": false + } + ``` + +## Get file information and the label of file for all category +- Path: [POST] `/annotation/file/get_file_info_n_label` +- Request + - Body: + + ```json + { + "id_token": {{id_token}}, + "project_id": "test123_anno_d92bfec0b0984685961d94a78658344b", + "filename": "20180810150607_camera_frontleft_000000083.png" + } + ``` + +- Response + - Http code: 200 + + ```json + { + "message": "OK", + "data": { + "file_info": { + "filename": "20180810150607_camera_frontleft_000000083.png", + "size": 3043989, + "created_time": "2022-09-27T16:45:04.307857", + "s3_key": "client-annotation-bucket/us-east-2:706ba872-01d3-4e77-b2a1-759d711f97c6/test123_anno_d92bfec0b0984685961d94a78658344b/raw_data/20180810150607_camera_frontleft_000000083.png", + "file_id": "90830ffc-3255-467f-b9e6-993a2aeeb0d1" + }, + "label_info": [ + { + "s3key_jsonlabel": "s3_path_json_label", + "updated_time": "2022-09-27T17:21:31.200059", + "category_id": "4d9509ec-42d0-44c7-83eb-472e1d13e1c5", + "category_name": "object segmentation", + "file_id": "90830ffc-3255-467f-b9e6-993a2aeeb0d1", + "category_des": "object segmentation description", + "created_time": "2022-09-27T16:52:04.972645" + } + ] + }, + "error": false + } + ``` \ No newline at end of file diff --git a/docs/url-endpoint-aws-service.md b/docs/url-endpoint-aws-service.md index 42d02c5..49bcd34 100644 --- a/docs/url-endpoint-aws-service.md +++ b/docs/url-endpoint-aws-service.md @@ -39,6 +39,8 @@ URL: https://4cujdvfmd4.execute-api.us-east-2.amazonaws.com/staging/projects/create +DEV_URL: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/projects/create + Create new project for user with id_token from project name REQUEST BODY: @@ -67,6 +69,8 @@ EXCEPTION EXPLAINATION: URL: https://4cujdvfmd4.execute-api.us-east-2.amazonaws.com/staging/projects/create_sample +DEV_URL: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/projects/create_sample + FUNCTION: Create sample project with provided data @@ -115,6 +119,8 @@ EXAMPLE URL: https://119u2071ul.execute-api.us-east-2.amazonaws.com/dev/projects/delete_images +Dev_URL: https://uflt5029de.execute-api.us-east-2.amazonaws.com/devdaitabeapp/projects/delete_images + FUNCTION Delete selected images in project. @@ -162,6 +168,8 @@ EXAMPLE URL: https://119u2071ul.execute-api.us-east-2.amazonaws.com/dev/projects/delete +DEV_URL: https://uflt5029de.execute-api.us-east-2.amazonaws.com/devdaitabeapp/projects/delete_images + FUNCTION Delete selected project. @@ -204,6 +212,8 @@ EXAMPLE URL: https://4cujdvfmd4.execute-api.us-east-2.amazonaws.com/staging/projects/create +Dev_URL: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/projects/list + FUNCTION: List all projects of user @@ -237,6 +247,8 @@ EXCEPTION EXPLAINATION: URL: https://4cujdvfmd4.execute-api.us-east-2.amazonaws.com/staging/projects/list_info +Dev_URL : https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/projects/list_info + FUNCTION: List all projects with all info: @@ -321,6 +333,8 @@ EXAMPLE URL: https://4cujdvfmd4.execute-api.us-east-2.amazonaws.com/staging/projects/info +Dev_URL: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/projects/info + FUNCTION Get detail information of a project name, including total data, total size, total original data, total generation data @@ -419,6 +433,8 @@ EXAMPLE URL: https://4cujdvfmd4.execute-api.us-east-2.amazonaws.com/staging/projects/update_info +Dev_URL: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/projects/update_info + FUNCTION Update current information of project including project_name or description @@ -478,6 +494,8 @@ EXAMPLE URL: https://4cujdvfmd4.execute-api.us-east-2.amazonaws.com/staging/projects/list_data +Dev_URL: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/projects/list_data + FUNCTION: List all s3_key of data in project name. The result will be return as paginator @@ -602,6 +620,8 @@ EXAMPLE URL: https://4cujdvfmd4.execute-api.us-east-2.amazonaws.com/staging/projects/download_create +Dev_URL: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/projects/download_create + FUNCTION Create an download task @@ -644,6 +664,8 @@ EXAMPLE URL: https://4cujdvfmd4.execute-api.us-east-2.amazonaws.com/staging/projects/download_update +Dev_URL: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/projects/download_update + FUNCTION Get information of current progress of download task @@ -701,6 +723,8 @@ EXAMPLE URL: https://4cujdvfmd4.execute-api.us-east-2.amazonaws.com/staging/projects/upload_check +Dev_URL: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/projects/upload_check + FUNCTION Check list data prepare for uploading with current data in DB. @@ -754,6 +778,8 @@ EXAMPLE URL: https://4cujdvfmd4.execute-api.us-east-2.amazonaws.com/staging/projects/upload_update +Dev_URL: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/projects/upload_update + FUNCTION After uploaded successfully from client to S3, client will send information of these data to server and server will update to DB. In here, we will not auto trigger with put action in client from s3 to lambda. @@ -814,6 +840,8 @@ EXAMPLE URL: https://4cujdvfmd4.execute-api.us-east-2.amazonaws.com/staging/generate/list_method +Dev_URL: https://uflt5029de.execute-api.us-east-2.amazonaws.com/devdaitabeapp/generate/list_method + FUNCTION Get list method for proprocessing and augmenting. @@ -872,6 +900,8 @@ EXAMPLE URL: https://4cujdvfmd4.execute-api.us-east-2.amazonaws.com/staging/generate/images +Dev_URL: https://uflt5029de.execute-api.us-east-2.amazonaws.com/devdaitabeapp/generate/images + FUNCTION Generate images with preprocessing or augmentation. @@ -924,6 +954,8 @@ EXAMPLE URL: https://4cujdvfmd4.execute-api.us-east-2.amazonaws.com/staging/generate/task_progress +Dev_URL: https://uflt5029de.execute-api.us-east-2.amazonaws.com/devdaitabeapp/generate/task_progress + FUNCTION Get information of current progress of task @@ -1036,8 +1068,8 @@ EXAMPLE ### API Sign Up - POST ``` - staging: https://rtv81e9ysk.execute-api.us-east-2.amazonaws.com/staging/user_signup - dev: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/user_signup + Dev: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/auth/user_signup + Product: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/user_signup ``` - Request - Content-type: application/json @@ -1055,8 +1087,8 @@ EXAMPLE ### API Login - POST ``` - staging: https://rtv81e9ysk.execute-api.us-east-2.amazonaws.com/staging/user_login - dev: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/user_login + Dev: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/auth/user_login + Product: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/user_login ``` - Request - Content-type: application/json @@ -1081,8 +1113,8 @@ EXAMPLE ### API Confirmation Email - POST ``` - staging: https://rtv81e9ysk.execute-api.us-east-2.amazonaws.com/staging/auth_confirm - dev: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/auth_confirm + Dev: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/auth/auth_confirm + Product: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/auth_confirm ``` - Request - Content-type: application/json @@ -1106,8 +1138,8 @@ EXAMPLE ### API Resend Confirmation Code - POST ``` - staging: https://rtv81e9ysk.execute-api.us-east-2.amazonaws.com/staging/resend_confirmcode - dev: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/resend_confirmcode + Dev: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/auth/resend_confirmcode + Product: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/resend_confirmcode ``` - Request - Content-type: application/json @@ -1130,8 +1162,8 @@ EXAMPLE ### API Resend Confirmation Code - Forgot Password - POST ``` - staging: https://4145bk5g67.execute-api.us-east-2.amazonaws.com/staging/confirm_code_forgot_password - dev: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/confirm_code_forgot_password + Dev: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/auth/confirm_code_forgot_password + Product: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/confirm_code_forgot_password ``` - Request - Content-type: application/json @@ -1156,8 +1188,8 @@ EXAMPLE ### API Forgot Password - POST ``` - staging: https://4145bk5g67.execute-api.us-east-2.amazonaws.com/staging/forgot_password - dev: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/forgot_password + Dev: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/auth/forgot_password + Product: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/forgot_password ``` - Request - Content-type: application/json @@ -1180,8 +1212,8 @@ EXAMPLE ### API Refresh Token - POST ``` - staging: https://4145bk5g67.execute-api.us-east-2.amazonaws.com/staging/refresh_token - dev: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/refresh_token + Dev: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/auth/refresh_token + Product: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/refresh_token ``` - Request - Content-type: application/json @@ -1213,8 +1245,8 @@ EXAMPLE ### API Send Email - POST ``` - staging: https://54upf5w03c.execute-api.us-east-2.amazonaws.com/staging/send-mail/reference-email - dev: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/send-mail/reference-email + Dev: https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/auth/reference-email + Product: https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/send-mail/reference-email ``` - Request - Content-type: application/json @@ -1238,8 +1270,8 @@ EXAMPLE ### API Template Invitation Email - GET ``` - staging :https://4145bk5g67.execute-api.us-east-2.amazonaws.com/staging/template-invite-mail - dev :https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/template-invite-mail + Dev : https://yf6ayuvru1.execute-api.us-east-2.amazonaws.com/dev/auth/template-invite-mail + Product : https://nzvw2zvu3d.execute-api.us-east-2.amazonaws.com/staging/auth/template-invite-mail ``` - Request - Header: Authorization: "Bearer "The Token"" @@ -1274,7 +1306,10 @@ EXAMPLE "error": boolean } ``` - +### API logout social login +``` + DEV_URL: https://daitasociallogin.auth.us-east-2.amazoncognito.com/logout?client_id=7v8h65t0d3elscfqll090acf9h&logout_uri=https://dev.daita.tech +``` ## Download images APIs ### Create Download task @@ -1334,4 +1369,4 @@ EXAMPLE }, "error": boolean } - ``` \ No newline at end of file + ``` diff --git a/download_service/app_download.py b/download_service/app_download.py deleted file mode 100644 index eefed17..0000000 --- a/download_service/app_download.py +++ /dev/null @@ -1,52 +0,0 @@ -from flask import Flask, jsonify -from flask import request -from itsdangerous import exc -from flask_cors import CORS, cross_origin - -import download_task -from task_worker import make_celery -import time - -app = Flask(__name__) -app.config.from_object('settings') -celery = make_celery(app) - -cors = CORS(app, resources={r"/*": {"origins": "*"}}) -STATUS_FINISH = "FINISH" -STATUS_ERROR = "ERROR" - - -@celery.task(name="task_download") -def task_download(data): - try: - task_id = data["task_id"] - identity_id = data["identity_id"] - url, s3_key = download_task.download(data) - download_task.upload_progress_db(STATUS_FINISH, identity_id, task_id, url, s3_key) - print(url, s3_key) - return "success" - except Exception as e: - print(e) - url= None - s3_key = None - download_task.upload_progress_db(STATUS_ERROR, identity_id, task_id, url, s3_key) - return "Error" - -@app.route("/download", methods=['POST']) -def download(): - try: - data = request.get_json() - print(data) - task = task_download.delay(data) - print(task) - return jsonify({ - "status": "OK" - }) - - except Exception as e: - print(e) - return "Error" - - -if __name__ == "__main__": - app.run(host="0.0.0.0", port=8000) \ No newline at end of file diff --git a/download_service/download_task.py b/download_service/download_task.py deleted file mode 100644 index 4005177..0000000 --- a/download_service/download_task.py +++ /dev/null @@ -1,346 +0,0 @@ -import os -import boto3 -import time -import json -from zipfile import ZipFile -import shutil -from botocore.client import Config -from boto3.dynamodb.conditions import Key, Attr -from concurrent import futures -from datetime import datetime -import logging -from pathlib import Path -from boto3.s3.transfer import TransferConfig -from botocore.exceptions import ClientError -import time - - - -relative_path = '/mnt/efs/images' -os.makedirs(relative_path, exist_ok=True) -bucket_name = 'daita-client-data' -s3 = boto3.client('s3') - -IDENTITY_POOL_ID = 'us-east-2:fa0b76bc-01fa-4bb8-b7cf-a5000954aafb' #'us-east-2:639788f0-a9b0-460d-9f50-23bbe5bc7140' -USER_POOL_ID = 'us-east-2_ZbwpnYN4g' #'us-east-2_6Sc8AZij7' - -# max_workers = 5 -# self.abs_path = os.path.abspath(relative_path) -MAX_WORKERS = 8 - -# config for uploading zipfile to s3 -MULTIPART_THRESHOLD = 1 -MULTIPART_CONCURRENCY = 2 -MAX_RETRY_COUNT = 3 - - -log = logging.getLogger('s3_uploader') - -def convert_current_date_to_iso8601(): - my_date = datetime.now() - return my_date.isoformat() - -def fetch(info): - abs_path = os.path.abspath(relative_path) - key = info["s3_key"] - type_method = info["type_method"] - - folder = os.path.split(key)[0] - filename = os.path.split(key)[1] - file = f'{abs_path}/{folder}/{type_method}/{filename}' - os.makedirs(os.path.split(file)[0], exist_ok=True) - - key = key.replace(f'{bucket_name}/', "") - # print('key request: ', key) - with open(file, 'wb') as data: - s3.download_fileobj(bucket_name, key, data) - - return (file, f'{type_method}/{filename}', info) - -def fetch_all(keys): - print("fetch_all") - with futures.ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor: - future_to_key = {executor.submit(fetch, key): key for key in keys} - - print("All URLs submitted.") - - for future in futures.as_completed(future_to_key): - key = future_to_key[future] - exception = future.exception() - - if not exception: - yield key, future.result() - else: - yield key, exception - -def get_all_file_paths(directory): - # initializing empty file paths list - file_paths = [] - - # crawling through directory and subdirectories - for root, directories, files in os.walk(directory): - for filename in files: - # join the two strings in order to form the full filepath. - filepath = os.path.join(root, filename) - file_paths.append(filepath) - - # returning all file paths - return file_paths - -def aws_get_identity_id(id_token): - - identity_client = boto3.client('cognito-identity') - PROVIDER = f'cognito-idp.{identity_client.meta.region_name}.amazonaws.com/{USER_POOL_ID}' - - try: - identity_response = identity_client.get_id( - IdentityPoolId=IDENTITY_POOL_ID, - Logins = {PROVIDER: id_token}) - except Exception as e: - print('Error: ', repr(e)) - raise Exception("Id_token is invalid!") - - identity_id = identity_response['IdentityId'] - - return identity_id - -def convert_method_name(dict_method, ls_method_id_str): - """ - convert generation method id to method string name - """ - ls_method_id_str = ls_method_id_str.replace(']', '').replace('[', '').replace("'", "") - if len(ls_method_id_str) == 0: - return "" - - ls_method_id = ls_method_id_str.split(",") - # print("ls_methof_id: ", ls_method_id) - str_final = "" - for method_id in ls_method_id: - # print(f"method_id: {method_id.strip()}, method_name: {dict_method[method_id.strip()]}") - str_final += f'{dict_method.get(method_id.strip(), "")}, ' - # print("string final: ", str_final) - - return str_final - -def upload_progress_db(status, identity_id, task_id, presign_url, s3_key): - db_resource = boto3.resource("dynamodb") - table = db_resource.Table('down_tasks') - response = table.update_item( - Key={ - 'identity_id': identity_id, - 'task_id': task_id, - }, - ExpressionAttributeNames= { - '#ST': "status" - }, - ExpressionAttributeValues = { - ':da': convert_current_date_to_iso8601(), - ":ke": s3_key, - ":ur": presign_url, - ":st": status - }, - UpdateExpression = 'SET s3_key = :ke, presign_url = :ur, #ST = :st, updated_date = :da' - ) - return - -def put_zip_to_s3(filepath, bucket_name, key_name, metadata=None): - """ - Upload zipfile to s3 and process with multipart and multi thread setting - """ - log.info("Uploading [" + filepath + "] to [" + bucket_name + "] bucket ...") - log.info("S3 path: [ s3://" + bucket_name + "/" + key_name + " ]") - # Multipart transfers occur when the file size exceeds the value of the multipart_threshold attribute - if not Path(filepath).is_file: - log.error("File [" + filepath + "] does not exist!") - raise Exception("File not found!") - - if key_name is None: - log.error("object_path is null!") - raise Exception("S3 object must be set!") - - GB = 1024 ** 3 - mp_threshold = MULTIPART_THRESHOLD*GB - concurrency = MULTIPART_CONCURRENCY - transfer_config = TransferConfig(multipart_threshold=mp_threshold, use_threads=True, max_concurrency=concurrency) - - login_attempt = False - retry = MAX_RETRY_COUNT - - while retry > 0: - try: - s3.upload_file(filepath, bucket_name, key_name, Config=transfer_config, ExtraArgs=metadata) - - log.info("File [" + filepath + "] uploaded successfully") - retry = 0 - - except ClientError as e: - log.error("Failed to upload object!") - log.exception(e) - if e.response['Error']['Code'] == 'ExpiredToken': - log.warning('Login token expired') - retry -= 1 - log.debug("retry = " + str(retry)) - else: - log.error("Unhandled error code:") - log.debug(e.response['Error']['Code']) - raise Exception("Error") - - except boto3.exceptions.S3UploadFailedError as e: - log.error("Failed to upload object!") - log.exception(e) - if 'ExpiredToken' in str(e): - log.warning('Login token expired') - log.info("Handling...") - retry -= 1 - log.debug("retry = " + str(retry)) - else: - log.error("Unknown error!") - raise Exception("Error") - - except Exception as e: - log.error("Unknown exception occured!") - template = "An exception of type {0} occurred. Arguments:\n{1!r}" - message = template.format(type(e).__name__, e.args) - log.debug(message) - log.exception(e) - raise Exception("Error") - -def download(data): - try: - # id_token: str = data["id_token"] - down_type: str = data["down_type"] # ALL, augment, preprocess, original - project_name: str = data["project_name"] - project_id: str = data["project_id"] - - # project_name = 'Driving Dataset Sample' - # project_id = 'Driving Dataset Sample_c64bd36f28a84897ad77b598046bfbd1' - # print("call aws get_identity: ", id_token) - # identity_id = aws_get_identity_id(id_token) - # print(identity_id) - - # get list key of the project - db_resource = boto3.resource("dynamodb") - - ls_table = [] - if down_type == "ALL": - ls_table.append(db_resource.Table('data_original')) - ls_table.append(db_resource.Table('data_augment')) - ls_table.append(db_resource.Table('data_preprocess')) - elif down_type == "ORIGINAL": - ls_table.append(db_resource.Table('data_original')) - elif down_type == "PREPROCESS": - ls_table.append(db_resource.Table('data_preprocess')) - elif down_type == "AUGMENT": - ls_table.append(db_resource.Table('data_augment')) - else: - return "Error" - # return Response( - # status_code=500, - # content=f"Field 'down_type' must be ALL | ORIGINAL | PREPROCESS | AUGMENT. Got type={down_type}" - # ) - - ## get all dowloaded object information from DB - ls_object = [] - for table in ls_table: - response = table.query( - KeyConditionExpression = Key('project_id').eq(project_id), - ProjectionExpression='filename, s3_key, classtype, gen_id, type_method, size', - Limit = 500 - ) - ls_object = ls_object + response['Items'] - # print("total len response: ", len(response['Items'])) - next_token = response.get('LastEvaluatedKey', None) - while next_token is not None: - response = table.query( - KeyConditionExpression = Key('project_id').eq(project_id), - ProjectionExpression='filename, s3_key, classtype, gen_id, type_method, size', - Limit = 500, - ExclusiveStartKey=next_token, - ) - next_token = response.get('LastEvaluatedKey', None) - # print("total len response next: ", len(response['Items'])) - ls_object = ls_object + response['Items'] - - ## get all methods - table = db_resource.Table('methods') - response = table.scan() - dict_method = {} - for item in response["Items"]: - dict_method[item["method_id"]] = item["method_name"] - # print("dict_method: \n", dict_method) - - - if len(ls_object) == 0: - return {"s3_key": None} - print("Final len: ", len(ls_object)) - - # setup dir - folder_s3 = ls_object[0]['s3_key'].replace(f'{bucket_name}/', "").replace(f"/{ls_object[0]['filename']}", "") - delete_dir = ls_object[0]['s3_key'].replace(f"/{ls_object[0]['filename']}", "") - - - # dowload and zip - starttime = time.time() - zipfile_name = f"{project_name}_{down_type}.zip" - zip_dir = f"/mnt/efs/zip" - os.makedirs(zip_dir, exist_ok=True) - zip_path = os.path.join(zip_dir, zipfile_name) - json_object = {} - with ZipFile(zip_path,'w') as zip: - for key, result in fetch_all(ls_object): - # print("result: ", result) - zip.write(result[0], result[1]) - json_object[result[2]["filename"]] = { - "name": result[2]["filename"], - "typeOfImage": result[2]["type_method"], - "class": result[2].get("classtype", ""), - "methodToCreate": convert_method_name(dict_method, result[2].get("gen_id", "")), - "size": int(result[2].get("size", 0)), - "nameOfProject": project_name - } - - # write this object to json file - json_filename = f"{project_name}_{down_type}_{str(int(time.time()))}.json" - dir_path = f'{relative_path}/{delete_dir}' - filepath = os.path.join(dir_path, json_filename) - with open(filepath, 'w') as outfile: - json.dump(json_object, outfile) - - # write to zip file - zip.write(filepath, f"{json_filename}") - endtime_down_zip = time.time() - - - - # put to s3 - key_name = f"{folder_s3}/download/{zipfile_name}" - put_zip_to_s3(zip_path, bucket_name, key_name) - endtime_put_s3 = time.time() - - - # delete data in EFS - dir_path = f'{relative_path}/{delete_dir}' - print("delete dir path: ", dir_path) - try: - shutil.rmtree(dir_path) - except OSError as e: - print("Error: %s : %s" % (dir_path, e.strerror)) - - # get presign url for this zip file: - s3_conf = boto3.client('s3', config=Config(signature_version='s3v4')) - url = s3_conf.generate_presigned_url( - ClientMethod='get_object', - Params={ - 'Bucket': bucket_name, - 'Key': key_name - }, - ExpiresIn=1*60*60 - ) - - print(f"Processing time of down_zip: {endtime_down_zip-starttime} puts3: {endtime_put_s3-endtime_down_zip}") - - return url, key_name - except Exception as e: - print("error") - print(e) - raise Exception("Error!") \ No newline at end of file diff --git a/download_service/requirements.txt b/download_service/requirements.txt deleted file mode 100644 index d0b107a..0000000 --- a/download_service/requirements.txt +++ /dev/null @@ -1,49 +0,0 @@ -amqp==5.0.9 -awscli==1.22.39 -billiard==3.6.4.0 -boto3==1.20.39 -botocore==1.23.39 -cached-property==1.5.2 -celery==5.2.3 -cffi==1.15.0 -click==8.0.3 -click-didyoumean==0.3.0 -click-plugins==1.1.1 -click-repl==0.2.0 -colorama==0.4.3 -cryptography==36.0.1 -Deprecated==1.2.13 -docutils==0.15.2 -Flask==2.0.2 -Flask-Cors==3.0.10 -flower==1.0.0 -gunicorn==20.1.0 -humanize==3.13.1 -importlib-metadata==4.10.1 -itsdangerous==2.0.1 -Jinja2==3.0.3 -jmespath==0.10.0 -kombu==5.2.3 -MarkupSafe==2.0.1 -packaging==21.3 -prometheus-client==0.12.0 -prompt-toolkit==3.0.24 -pyasn1==0.4.8 -pycparser==2.21 -pyOpenSSL==21.0.0 -pyparsing==3.0.7 -python-dateutil==2.8.2 -pytz==2021.3 -PyYAML==5.4.1 -redis==4.1.1 -rsa==4.7.2 -s3transfer==0.5.0 -six==1.16.0 -tornado==6.1 -typing-extensions==4.0.1 -urllib3==1.26.8 -vine==5.0.0 -wcwidth==0.2.5 -Werkzeug==2.0.2 -wrapt==1.13.3 -zipp==3.7.0 \ No newline at end of file diff --git a/download_service/settings.py b/download_service/settings.py deleted file mode 100644 index 88c6e2e..0000000 --- a/download_service/settings.py +++ /dev/null @@ -1,6 +0,0 @@ -# -*- coding: utf-8 -*- -import os - -CELERY_BROKER_URL='redis://localhost:6379/0' -CELERY_RESULT_BACKEND='redis://localhost:6379/0' - diff --git a/download_service/task_worker.py b/download_service/task_worker.py deleted file mode 100644 index e9cfba6..0000000 --- a/download_service/task_worker.py +++ /dev/null @@ -1,17 +0,0 @@ -from celery import Celery - -def make_celery(app): - celery = Celery( - app.import_name, - backend=app.config['CELERY_RESULT_BACKEND'], - broker=app.config['CELERY_BROKER_URL'] - ) - celery.conf.update(app.config) - - class ContextTask(celery.Task): - def __call__(self, *args, **kwargs): - with app.app_context(): - return self.run(*args, **kwargs) - - celery.Task = ContextTask - return celery \ No newline at end of file diff --git a/env.dev b/env.dev new file mode 100644 index 0000000..8bfba89 --- /dev/null +++ b/env.dev @@ -0,0 +1,8 @@ +MODE=dev +BUCKET_NAME=client-data-test +USER_POOL_ID=us-east-2_6Sc8AZij7 +IDENTITY_POOL_ID=us-east-2:639788f0-a9b0-460d-9f50-23bbe5bc7140 +CLIENT_ID=7v8h65t0d3elscfqll090acf9h +ROLE=arn:aws:iam::737589818430:role/datai_be_roles_1 +T_TASKS=devdaitabeapp-generate-tasks +ROLEGATEWAY=arn:aws:iam::737589818430:role/daita_apigw_lambda_cloud_watch \ No newline at end of file diff --git a/infrastructure-def-app/build_infra_app.sh b/infrastructure-def-app/build_infra_app.sh new file mode 100644 index 0000000..417d265 --- /dev/null +++ b/infrastructure-def-app/build_infra_app.sh @@ -0,0 +1,86 @@ +#!/bin/bash + + +### load config file +. "$1" + +OUTPUT_BUILD_DAITA=$2 +OUTPUT_FE_CONFIG=$3 + + +cd infrastructure-def-app + + +parameters_override="Mode=${MODE} Stage=${DAITA_STAGE} Application=${INFRA_APPLICATION}" + +sam build --template-file template_infra.yaml +sam deploy --template-file template_infra.yaml --no-confirm-changeset --disable-rollback \ + --resolve-image-repos --resolve-s3 --config-env $DAITA_STAGE \ + --stack-name "$DAITA_STAGE-${INFRA_APPLICATION}-app" \ + --s3-prefix "$DAITA_STAGE-${INFRA_APPLICATION}-app" \ + --region $AWS_REGION \ + --parameter-overrides $parameters_override | tee -a output.txt + + +shopt -s extglob + +declare -A dict_output +filename="output.txt" + +is_first_line_value=false +first_line="" +while read line; do + # reading each line + if [[ "$line" =~ "Key".+ ]]; then + + [[ "$line" =~ [[:space:]].+ ]] + a=${BASH_REMATCH[0]} + a=${a##*( )} + a=${a%%*( )} + key=$a + fi + if [[ "$line" =~ "Value".+ ]]; then + [[ "$line" =~ [[:space:]].+ ]] + value=${BASH_REMATCH[0]} + value=${value##*( )} + value=${value%%*( )} + + first_line=$value + is_first_line_value=true + else + if [[ "$line" =~ .*"-------".* ]]; then + echo "skip line" + else + if [ "$is_first_line_value" = true ]; then + final_line=$first_line$line + dict_output[$key]=$final_line + is_first_line_value=false + fi + fi + fi +done < $filename + + +PublicSubnetOne=${dict_output["PublicSubnetOne"]} +PublicSubnetTwo=${dict_output["PublicSubnetTwo"]} +ContainerSecurityGroup=${dict_output["ContainerSecurityGroup"]} +VPC=${dict_output["VPC"]} +VPCEndpointSQSDnsEntries=${dict_output["VPCEndpointSQSDnsEntries"]} +VpcEndointSQS=${dict_output["VpcEndointSQS"]} +VPCEndpointS3=${dict_output["VPCEndpointS3"]} + +EFSFileSystemId=${dict_output["EFSFileSystemId"]} +EFSAccessPoint=${dict_output["EFSAccessPoint"]} +EFSAccessPointArn=${dict_output["EFSAccessPointArn"]} + +###======= Store output to file ========== +echo "PublicSubnetOne=$PublicSubnetOne" > $OUTPUT_BUILD_DAITA +echo "PublicSubnetTwo=$PublicSubnetTwo" >> $OUTPUT_BUILD_DAITA +echo "ContainerSecurityGroup=$ContainerSecurityGroup" >> $OUTPUT_BUILD_DAITA +echo "VPC=$VPC" >> $OUTPUT_BUILD_DAITA +echo "VPCEndpointSQSDnsEntries=$VPCEndpointSQSDnsEntries" >> $OUTPUT_BUILD_DAITA + +echo "EFSFileSystemId=$EFSFileSystemId" >> $OUTPUT_BUILD_DAITA +echo "EFSAccessPoint=$EFSAccessPoint" >> $OUTPUT_BUILD_DAITA +echo "EFSAccessPointArn=$EFSAccessPointArn" >> $OUTPUT_BUILD_DAITA + diff --git a/infrastructure-def-app/network.yaml b/infrastructure-def-app/network.yaml new file mode 100644 index 0000000..f0589d2 --- /dev/null +++ b/infrastructure-def-app/network.yaml @@ -0,0 +1,250 @@ +AWSTemplateFormatVersion: '2010-09-09' +Description: > + ECS-workers-app + + + +Parameters: + + StagePara: + Type: String + + ApplicationPara: + Type: String + +Mappings: + # Hard values for the subnet masks. These masks define + # the range of internal IP addresses that can be assigned. + # The VPC can have all IP's from 10.0.0.0 to 10.0.255.255 + # There are four subnets which cover the ranges: + # + # 10.0.0.0 - 10.0.0.255 + # 10.0.1.0 - 10.0.1.255 + # 10.0.2.0 - 10.0.2.255 + # 10.0.3.0 - 10.0.3.255 + # + # If you need more IP addresses (perhaps you have so many + # instances that you run out) then you can customize these + # ranges to add more + SubnetConfig: + VPC: + CIDR: '10.0.0.0/16' + PublicOne: + CIDR: '10.0.0.0/24' + PublicTwo: + CIDR: '10.0.1.0/24' + PublicThree: + CIDR: '10.0.2.0/24' + PublicFour: + CIDR: '10.0.3.0/24' + +Resources: + VPC: + Type: AWS::EC2::VPC + Properties: + EnableDnsSupport: true + EnableDnsHostnames: true + CidrBlock: !FindInMap ['SubnetConfig', 'VPC', 'CIDR'] + Tags: + - Key: "Name" + Value: !Sub "${StagePara}_${ApplicationPara}_vpc_config" + + PublicSubnetOne: + Type: AWS::EC2::Subnet + Properties: + AvailabilityZone: + Fn::Select: + - 0 + - Fn::GetAZs: {Ref: 'AWS::Region'} + VpcId: !Ref VPC + CidrBlock: !FindInMap ['SubnetConfig', 'PublicOne', 'CIDR'] + MapPublicIpOnLaunch: true + Tags: + - Key: "Name" + Value: !Sub "${StagePara}_${ApplicationPara}_pb_subnet_1" + + PublicSubnetTwo: + Type: AWS::EC2::Subnet + Properties: + AvailabilityZone: + Fn::Select: + - 1 + - Fn::GetAZs: {Ref: 'AWS::Region'} + VpcId: !Ref VPC + CidrBlock: !FindInMap ['SubnetConfig', 'PublicTwo', 'CIDR'] + MapPublicIpOnLaunch: true + Tags: + - Key: "Name" + Value: !Sub "${StagePara}_${ApplicationPara}_pb_subnet_2" + + InternetGateway: + Type: AWS::EC2::InternetGateway + + GatewayAttachment: + Type: AWS::EC2::VPCGatewayAttachment + Properties: + VpcId: !Ref VPC + InternetGatewayId: !Ref InternetGateway + + PublicRouteTable: + Type: AWS::EC2::RouteTable + Properties: + VpcId: !Ref VPC + Tags: + - Key: "Name" + Value: !Sub "${StagePara}_${ApplicationPara}_pb_route_table" + + PublicRoute: + Type: AWS::EC2::Route + Properties: + RouteTableId: !Ref PublicRouteTable + DestinationCidrBlock: '0.0.0.0/0' + GatewayId: !Ref InternetGateway + + PublicSubnetOneRouteTableAssociation: + Type: AWS::EC2::SubnetRouteTableAssociation + Properties: + SubnetId: !Ref PublicSubnetOne + RouteTableId: !Ref PublicRouteTable + + PublicSubnetTwoRouteTableAssociation: + Type: AWS::EC2::SubnetRouteTableAssociation + Properties: + SubnetId: !Ref PublicSubnetTwo + RouteTableId: !Ref PublicRouteTable + + ContainerSecurityGroup: + Type: AWS::EC2::SecurityGroup + Properties: + GroupDescription: Access container + VpcId: !Ref VPC + Tags: + - Key: "Name" + Value: !Sub "${StagePara}_${ApplicationPara}_security_group" + + EcsSecurityGroupIngressFromSelf: + Type: AWS::EC2::SecurityGroupIngress + Properties: + Description: Ingress from container in the same security group + GroupId: !Ref ContainerSecurityGroup + IpProtocol: -1 + SourceSecurityGroupId: !Ref ContainerSecurityGroup + + DynamoDBEndpoint: + Type: AWS::EC2::VPCEndpoint + Properties: + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: "*" + Principal: "*" + Resource: "*" + RouteTableIds: + - !Ref 'PublicRouteTable' + ServiceName: !Sub com.amazonaws.${AWS::Region}.dynamodb + VpcId: !Ref VPC + + S3VPCEndpoint: + Type: AWS::EC2::VPCEndpoint + Properties: + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: "*" + Principal: "*" + Resource: "*" + RouteTableIds: + - !Ref 'PublicRouteTable' + # SecurityGroupIds: + # - !Ref ContainerSecurityGroup + # SubnetIds: + # - !Ref PublicSubnetOne + # - !Ref PublicSubnetTwo + ServiceName: !Sub com.amazonaws.${AWS::Region}.s3 + VpcId: !Ref VPC + + ECRPullImageEndpoint: + Type: AWS::EC2::VPCEndpoint + Properties: + VpcEndpointType: Interface + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: "*" + Principal: "*" + Resource: "*" + ServiceName: !Sub com.amazonaws.${AWS::Region}.ecr.dkr + VpcId: !Ref VPC + + EC2Endpoint: + Type: AWS::EC2::VPCEndpoint + Properties: + VpcEndpointType: Interface + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: "*" + Principal: "*" + Resource: "*" + ServiceName: !Sub com.amazonaws.${AWS::Region}.ec2 + VpcId: !Ref VPC + + LambdaEndpoint: + Type: AWS::EC2::VPCEndpoint + Properties: + VpcEndpointType: Interface + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: "*" + Principal: "*" + Resource: "*" + ServiceName: !Sub com.amazonaws.${AWS::Region}.lambda + VpcId: !Ref VPC + + + SQSEndpoint: + Type: AWS::EC2::VPCEndpoint + Properties: + VpcEndpointType: Interface + VpcId: !Ref VPC + SecurityGroupIds: + - !Ref ContainerSecurityGroup + SubnetIds: + - !Ref PublicSubnetOne + - !Ref PublicSubnetTwo + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: "*" + Principal: "*" + Resource: "*" + ServiceName: !Sub com.amazonaws.${AWS::Region}.sqs + +Outputs: + PublicSubnetOne: + Value: !Ref PublicSubnetOne + + PublicSubnetTwo: + Value: !Ref PublicSubnetTwo + + ContainerSecurityGroup: + Value: !Ref ContainerSecurityGroup + + VPC: + Value: !Ref VPC + + VpcEndointSQS: + Value: !Ref SQSEndpoint + + S3VPCEndpoint: + Value: !Ref S3VPCEndpoint + + VPCSQSEndpointDnsEntries: + Value: !Select [0, !GetAtt SQSEndpoint.DnsEntries] \ No newline at end of file diff --git a/infrastructure-def-app/samconfig.toml b/infrastructure-def-app/samconfig.toml new file mode 100644 index 0000000..6b7ba84 --- /dev/null +++ b/infrastructure-def-app/samconfig.toml @@ -0,0 +1,19 @@ +version = 0.1 + +[dev.deploy.parameters] +confirm_changeset = true +capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" +disable_rollback = true +image_repositories = [] + +[prod.deploy.parameters] +confirm_changeset = true +capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" +disable_rollback = true +image_repositories = [] + +[dev1.deploy.parameters] +confirm_changeset = true +capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" +disable_rollback = true +image_repositories = [] \ No newline at end of file diff --git a/infrastructure-def-app/storage.yaml b/infrastructure-def-app/storage.yaml new file mode 100644 index 0000000..fdf4dce --- /dev/null +++ b/infrastructure-def-app/storage.yaml @@ -0,0 +1,79 @@ +AWSTemplateFormatVersion: '2010-09-09' + +Parameters: + VPC: + Type: String + + PublicSubnetOne: + Type: String + + PublicSubnetTwo: + Type: String + + SecurityGroup: + Type: String + + EFSAccessPointRootPath: + Type: String + Default: /app/data + + StagePara: + Type: String + + ApplicationPara: + Type: String + +Resources: + + EFSFileSystem: + Type: AWS::EFS::FileSystem + # FileSystemTags: + # - Key: "Name" + # Value: !Sub "${StagePara}_${ApplicationPara}_efs_ecs_segmentation" + + MountTargeSubnetOne: + Type: AWS::EFS::MountTarget + Properties: + FileSystemId: !Ref EFSFileSystem + SubnetId: !Ref PublicSubnetOne + SecurityGroups: + - !Ref SecurityGroup + + MountTargeSubnetTwo: + Type: AWS::EFS::MountTarget + Properties: + FileSystemId: !Ref EFSFileSystem + SubnetId: !Ref PublicSubnetTwo + SecurityGroups: + - !Ref SecurityGroup + + AccessPoint: + Type: AWS::EFS::AccessPoint + Properties: + FileSystemId: !Ref EFSFileSystem + PosixUser: + Gid: "1000" + Uid: "1000" + RootDirectory: + Path: !Ref EFSAccessPointRootPath + CreationInfo: + OwnerGid: "1000" + OwnerUid: "1000" + Permissions: "777" + +Outputs: + + MountTargeSubnetOne: + Value: !Ref MountTargeSubnetOne + + MountTargeSubnetTwo: + Value: !Ref MountTargeSubnetTwo + + EFSFileSystemId: + Value: !Ref EFSFileSystem + EFSAccessPoint: + Value: !Ref AccessPoint + EFSAccessPointArn: + Value: !GetAtt AccessPoint.Arn + + diff --git a/infrastructure-def-app/template_infra.yaml b/infrastructure-def-app/template_infra.yaml new file mode 100644 index 0000000..2498b60 --- /dev/null +++ b/infrastructure-def-app/template_infra.yaml @@ -0,0 +1,65 @@ +AWSTemplateFormatVersion: '2010-09-09' +Transform: AWS::Serverless-2016-10-31 +Description: > + SAM Template for Nested application resources + +## The general rule seems to be to use !Sub for in line substitutions and !ref for stand alone text +Parameters: + + Stage: + Type: String + + Application: + Type: String + +Resources: + + + #================ APPLICATIONS ============================================= + NetworkLayerApplication: + Type: AWS::Serverless::Application + Properties: + Location: ./network.yaml + Parameters: + StagePara: !Ref Stage + ApplicationPara: !Ref Application + + StorageLayerApplication: + Type: AWS::Serverless::Application + Properties: + Location: ./storage.yaml + Parameters: + VPC: !GetAtt NetworkLayerApplication.Outputs.VPC + PublicSubnetOne: !GetAtt NetworkLayerApplication.Outputs.PublicSubnetOne + PublicSubnetTwo: !GetAtt NetworkLayerApplication.Outputs.PublicSubnetTwo + SecurityGroup: !GetAtt NetworkLayerApplication.Outputs.ContainerSecurityGroup + StagePara: !Ref Stage + ApplicationPara: !Ref Application + +Outputs: + PublicSubnetOne: + Value: !GetAtt NetworkLayerApplication.Outputs.PublicSubnetOne + PublicSubnetTwo: + Value: !GetAtt NetworkLayerApplication.Outputs.PublicSubnetTwo + ContainerSecurityGroup: + Value: !GetAtt NetworkLayerApplication.Outputs.ContainerSecurityGroup + + VPC: + Value: !GetAtt NetworkLayerApplication.Outputs.VPC + + VpcEndointSQS: + Value: !GetAtt NetworkLayerApplication.Outputs.VpcEndointSQS + VPCEndpointS3: + Value: !GetAtt NetworkLayerApplication.Outputs.S3VPCEndpoint + VPCEndpointSQSDnsEntries: + Value: !GetAtt NetworkLayerApplication.Outputs.VPCSQSEndpointDnsEntries + + ### storage output + EFSFileSystemId: + Value: !GetAtt StorageLayerApplication.Outputs.EFSFileSystemId + EFSAccessPoint: + Value: !GetAtt StorageLayerApplication.Outputs.EFSAccessPoint + EFSAccessPointArn: + Value: !GetAtt StorageLayerApplication.Outputs.EFSAccessPointArn + + \ No newline at end of file diff --git a/main_build.sh b/main_build.sh new file mode 100644 index 0000000..8b270a2 --- /dev/null +++ b/main_build.sh @@ -0,0 +1,72 @@ +#! /bin/bash + +SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) +echo $SCRIPT_DIR + +###===== read from config file +echo ==1==. Read the configure file +if [[ $1 == "dev" ]] +then + configfile=$SCRIPT_DIR/main_config_dev.cnf +else + if [[ $1 == "prod" ]] + then + configfile=$SCRIPT_DIR/main_config_prod.cnf + else + echo "Please choose the env is dev/prod" + exit + fi +fi + +echo $configfile + + +keys=( $(grep -oP '\w+(?==)' "$configfile") ) +. "$configfile" + +### confirm build daita +read -p "Do you want to build DAITA [Y/n]: " IS_BUILD_DAITA + confirm=${IS_BUILD_DAITA:-y} +read -p "Do you want to build ANNOTATION [Y/n]: " IS_BUILD_ANNOTATION + confirm=${IS_BUILD_ANNOTATION:-y} + +for var in "${keys[@]}"; do + printf "%s\t=> %s\n" "$var" "${!var}" +done + +### confirm the config again +read -p "Are you sure that you want to continue with this configuration [y/N]: " confirm + confirm=${confirm:-n} + +if [[ $confirm == "n" ]] || [[ $confirm == "N" ]] +then + exit +fi + +### output data path +output_data=$SCRIPT_DIR/$OUTPUT_BUILD_DAITA_NAME +output_fe_config=$SCRIPT_DIR/$OUTPUT_FOR_FE + + +### save FE config +echo "### config for daita_env: $DAITA_STAGE annotation_env: $ANNOTATION_STAGE ###" > $output_fe_config +echo "REACT_APP_ENV=development" >> $output_fe_config + + +### build infrastructure that use for another application +echo ==============Building: INFRA ========================== +bash ./infrastructure-def-app/build_infra_app.sh "$configfile" "$output_data" "$output_fe_config" + +### build daita app and annotation ap +if [[ $IS_BUILD_DAITA == "y" ]] || [[ $IS_BUILD_DAITA == "Y" ]] +then + echo ==============Building: DAITA ========================== + bash ./daita-app/build_daita.sh "$configfile" "$output_data" "$output_fe_config" +fi + + +if [[ $IS_BUILD_ANNOTATION == "y" ]] || [[ $IS_BUILD_ANNOTATION == "Y" ]] +then + echo ==============Building: ANNOTATION ========================== + bash ./annotation-app/build_annotation.sh "$configfile" "$output_data" "$output_fe_config" +fi \ No newline at end of file diff --git a/main_config_dev.cnf b/main_config_dev.cnf new file mode 100644 index 0000000..54c8233 --- /dev/null +++ b/main_config_dev.cnf @@ -0,0 +1,61 @@ +AWS_REGION=us-east-2 +AWS_ACCOUNT_ID="737589818430" + +OUTPUT_BUILD_DAITA_NAME=output_daita.txt +OUTPUT_FOR_FE=output_fe_config.txt + +###--------------- parameter value --------------- +###____ do not change this config when you change the env_______ + +SUB_NET_IDS=subnet-019da2a6738756f88,subnet-0642064673fd68d2e +SECURITY_GROUP_IDS=sg-0c9a0ca7844d7b128,sg-00d8b4ca79ee1e42f,sg-007caf776eee9bd32,sg-04b9c865721337372,sg-0b411b5391db8d7a3 +VPC_ID=vpc-057803c925fd8138a + + +EFS_ID=fs-01115862a24b75423 +ROOT_EFS=/efs + +DAITA_S3_BUCKET=client-data-test +ANNO_S3_BUCKET=client-annotation-bucket +DAITA_APPLICATION=daita +ANNO_APPLICATION=anno +INFRA_APPLICATION=infra + +CERTIFICATE_USERPOLL_DOMAIN=arn:aws:acm:us-east-1:737589818430:certificate/997bd837-32c3-466d-927b-2e1d4dee7bf0 +OAUTH2BOT_SLACK_FEED_BACK=xoxb-1117947584070-4388501155699-kEtvbOiFTEVEqVEUEREYj0IY + + +MODE=dev + +GOOGLE_CLIENT_ID=639730110991-9t82efunb20f6m4stek56f6ut9t0kjfu.apps.googleusercontent.com +GOOGLE_CLIENT_SECRET=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 +GITHUB_CLIENT_ID=0cec5cf3d1f070b36b63 +GITHUB_CLIENT_SECRET=5929cb027e02330533a587a8e9f5d2e0fd40e48e + +OAUTH_ENPOINT=https://authdev.daita.tech/oauth2/token + +CAPTCHA_SITE_KEY_GOOGLE=6LfR4ioiAAAAAH7RrdcoRQLiNbKFpFMNH2nqG8fv +CAPTCHA_SECRET_KEY_GOOGLE=6LfR4ioiAAAAAPR6AwPkkg3xAENhqK83hr1caBk7 + +MAX_SIZE_EC2_AUTOSCALING_ECS=2 + +###__________________________________ +###__________________________________ + +###-------config for dev env-------------------------------------- +###--------------------------------------------------------------- +IS_BUILD_DAITA=yes +IS_BUILD_ANNOTATION=yes + +DAITA_STAGE=dev +ANNOTATION_STAGE=dev +CAPABILITIES="CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" + +MAX_CONCURRENCY_TASK=2 + +DOMAIN_USER_POOL=authdev.daita.tech +DOMAIN_DAITA=https://dev.daita.tech + +IMAGE_AI_SEGMENTATION_URL=737589818430.dkr.ecr.us-east-2.amazonaws.com/ai-services-repo:segformer + +####------------------------------------------------------------- diff --git a/main_config_prod.cnf b/main_config_prod.cnf new file mode 100644 index 0000000..ea8c303 --- /dev/null +++ b/main_config_prod.cnf @@ -0,0 +1,63 @@ +AWS_REGION=us-east-2 +AWS_ACCOUNT_ID="366577564432" + +OUTPUT_BUILD_DAITA_NAME=output_daita.txt +OUTPUT_FOR_FE=output_fe_config.txt + +DAITA_APPLICATION=daita +ANNO_APPLICATION=anno +INFRA_APPLICATION=infra + +###--------------- parameter value --------------- +###____ do not change this config when you change the env_______ + +SUB_NET_IDS=subnet-31ff5b5a +SECURITY_GROUP_IDS=sg-0315a5ecee0dc69fe,sg-0b3b2fcc4dc7686ad,sg-af50cbde,sg-07c27f59bc172f180,sg-0796222bd5149736f +VPC_ID=vpc-53239e38 + + +EFS_ID=fs-0199771f2dfe97ace +ROOT_EFS=/efs + +DAITA_S3_BUCKET=daita-client-data +ANNO_S3_BUCKET=annotation-client-data + + +CERTIFICATE_USERPOLL_DOMAIN=arn:aws:acm:us-east-1:366577564432:certificate/8c1361ca-e287-4171-91ba-472b28b9c69a + +MODE=prod + +OAUTH2BOT_SLACK_FEED_BACK=xoxb-1117947584070-4388501155699-kEtvbOiFTEVEqVEUEREYj0IY + +GOOGLE_CLIENT_ID=639730110991-p66c4enhi3sbd6o41uq4o2ct4l1it8k6.apps.googleusercontent.com +GOOGLE_CLIENT_SECRET=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 +GITHUB_CLIENT_ID=107117788aa64f363ee5 +GITHUB_CLIENT_SECRET=c054671d4091df5d5e327b630b50e40d0db1de6d + +CAPTCHA_SITE_KEY_GOOGLE=6LfR4ioiAAAAAH7RrdcoRQLiNbKFpFMNH2nqG8fv +CAPTCHA_SECRET_KEY_GOOGLE=6LfR4ioiAAAAAPR6AwPkkg3xAENhqK83hr1caBk7 + +OAUTH_ENPOINT=https://auth.daita.tech/oauth2/token + +MAX_SIZE_EC2_AUTOSCALING_ECS=4 + +###__________________________________ +###__________________________________ + +###-------config for prod env-------------------------------------- +###--------------------------------------------------------------- +IS_BUILD_DAITA=yes +IS_BUILD_ANNOTATION=yes + +DAITA_STAGE=prod +ANNOTATION_STAGE=prod +CAPABILITIES="CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM" + +MAX_CONCURRENCY_TASK=2 + +DOMAIN_USER_POOL=auth.daita.tech +DOMAIN_DAITA=https://app.daita.tech + +IMAGE_AI_SEGMENTATION_URL=366577564432.dkr.ecr.us-east-2.amazonaws.com/ai-services-repo:segformer + +####------------------------------------------------------------- diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..dff1d0d --- /dev/null +++ b/requirements.txt @@ -0,0 +1,2 @@ +aws-sam-cli==1.53.0 +awscli==1.25.51 \ No newline at end of file