Merge pull request #217 from creativecommons/new-projects

Update to manage open and untracked issues and pull requests using new projects

Author: TimidRobot

.github/workflows/manage_issues.yml.DISABLED renamed to .github/workflows/manage_issues.yml

@@ -29,12 +29,7 @@ jobs:
     - name: Install Python dependencies
       run: pipenv sync --system
 
-    - name: 'run script to move closed issues to Active Sprint: Done'
-      run: ./move_closed_issues.py
-      env:
-        ADMIN_GITHUB_TOKEN: ${{ secrets.ADMIN_GITHUB_TOKEN }}
-
-    - name: run script to track open untriaged issues and pull requests
-      run: ./track_issues_and_pull_requests.py
+    - name: run script to track open and untracked issues and pull requests
+      run: ./manage_new_issues_and_pull_requests.py
       env:
         ADMIN_GITHUB_TOKEN: ${{ secrets.ADMIN_GITHUB_TOKEN }}

Pipfile

@@ -13,10 +13,13 @@ asana = ">=3.2,<4"  # "Please use v3.2.X for stable / production environments"
 cryptography = ">=42.0.4"  # dependency, pinned for security
 emoji = "*"
 GitPython = ">=3.1.41"
+gql = "*"
 idna = ">=3.7"  # dependency, pinned for security
 PyGithub = "*"
+Pygments = "*"
 PyYAML = "*"
 requests = "*"
+requests_toolbelt = "*"
 urllib3 = ">=2.0.6"  # dependency, pinned for security
 
 [requires]

Pipfile.lock

@@ -62,44 +62,42 @@ The following workflows are ordered by schedule frequency and start time.
 
 This creates GitHub teams for the Community teams and updates their membership
 based on the [`community_team_members.json`][databag] Lektor databag.
- - The databag is used to:
-   - create the [Community Team Members — Creative Commons Open
-     Source][ctlistpage] page
-   - configure GitHub team memberships and repository permissions
- - The databag is kept up-to-date by [Push data to CC Open
-   Source](#push-data-to-cc-open-source), below
+- The databag is used to:
+  - create the [Community Team Members — Creative Commons Open
+    Source][ctlistpage] page
+  - configure GitHub team memberships and repository permissions
+- The databag is kept up-to-date by [Push data to CC Open
+  Source](#push-data-to-cc-open-source), below
 
 [sync_teams_yml]: .github/workflows/sync_community_teams.yml
 [teams_file]: sync_community_teams.py
 [databag]: https://github.com/creativecommons/creativecommons.github.io-source/blob/master/databags/community_team_members.json
 [ctlistpage]: https://opensource.creativecommons.org/community/community-team/members/
 
 
-### Manage issues and pull requests in projects
+### Manage new issues and pull requests in projects
 
 | **Workflow** | | |
 | -- | --: | --- |
 | | Schedule: | Hourly at 45 minutes past the hour (`**:45`) |
 | | YAML: | [`manage_issues.yml`][manage_issues] |
 | **Script** | | |
-| | File: | [`move_closed_issues.py`][move_file] |
-| | File: | [`track_issues_and_pull_requests.py`][track_issues] |
+| | File: | [`manage_new_issues_and_pull_requests.py`][manage_new_issues] |
 | | Common Modules: | [`ccos/`](ccos/) |
 | **Env** | | |
 | | Required: | `ADMIN_GITHUB_TOKEN` |
 
-This manages issues and pull requests to ensure they are properly tracked
-within the [Active Sprint][active_sprint] and [Backlog][backlog] projects:
-  - Move closed issues from [Backlog][backlog] to [Active
-    Sprint][active_sprint]: Done
-  - Track open issues in [Backlog][backlog]: Pending Review
-  - Track open pull requests in [Active Sprint][active_sprint]: Code Review
+This manages new issues and pull requests to ensure they are properly tracked
+in a GitHub project:
+- [Applications project][proj_applications]
+- [Systems project][proj_systems]
+- [WebDev project][proj_webdev]
 
 [manage_issues]: .github/workflows/manage_issues.yml
-[move_file]: move_closed_issues.py
-[track_issues]: track_issues_and_pull_requests.py
-[active_sprint]: https://github.com/orgs/creativecommons/projects/7
-[backlog]: https://github.com/orgs/creativecommons/projects/10
+[manage_new_issues]: manage_new_issues_and_pull_requests.py
+[proj_applications]: https://github.com/orgs/creativecommons/projects/15/views/1
+[proj_systems]: https://github.com/orgs/creativecommons/projects/22/views/1
+[proj_webdev]: https://github.com/orgs/creativecommons/projects/23/views/1
 
 
 ###  Normalize Repos
@@ -206,37 +204,59 @@ GitHub Action:
 ```
 
 
-## Python Dependencies
+## Development
+
+Local development and testing is facilitated by helper scripts:
+- `./dev/tools.sh`: Checks and updates Python formatting
+- `.dev/test.sh`: Uses act and Docker to test workflows
+  - [nektos/act](https://github.com/nektos/act): _Run your GitHub Actions
+    locally 🚀_
+
 
-- [Asana/python-asana][python-asana]: Official Python client library for the
-  Asana API v1
-- [carpedm20/emoji][emoji]: emoji terminal output for Python
-- [gitpython-developers/GitPython][gitpython]: GitPython is a python library
-  used to interact with Git repositories.
-- [PyGithub/PyGithub][pygithub]: Typed interactions with the GitHub API v3
-- [PyYAML][pyyaml] is a full-featured YAML framework for the Python programming
-  language
-- [Requests][requests]: HTTP for Humans™
+### Python Dependencies
+
+- [Asana/python-asana][python-asana]: _Official Python client library for the
+  Asana API v1_
+- [carpedm20/emoji][emoji]: _e_moji terminal output for Python_
+- [gitpython-developers/GitPython][gitpython]: _GitPython is a python library
+  used to interact with Git repositories._
+- [graphql-python/gql][pygql]: _A GraphQL client in Python_
+- [PyGithub/PyGithub][pygithub]: _Typed interactions with the GitHub API v3_
+- [PyYAML][pyyaml]: _a full-featured YAML framework for the Python
+  programming language_
+- [Requests][requests]: _HTTP for Humans™_
 
 [python-asana]: https://github.com/asana/python-asana
 [emoji]: https://github.com/carpedm20/emoji/
 [gitpython]: https://github.com/gitpython-developers/GitPython
+[pygql]: https://github.com/graphql-python/gql
 [pygithub]: https://github.com/pygithub/pygithub
 [pyyaml]: https://pyyaml.org/
 [requests]: https://requests.readthedocs.io/en/latest/
 
 
-## Development
+### GitHub GraphQL API
 
-Local development and testing is facilitated by helper scripts:
-- `./dev/tools.sh`: Checks and updates Python formatting
-- `.dev/test.sh`: Uses act and Docker to test workflows
-  - [nektos/act](https://github.com/nektos/act): _Run your GitHub Actions
-    locally 🚀_
+- [Using the API to manage Projects - GitHub Docs][projectsv2api]
+- [Forming calls with GraphQL - GitHub Docs][formingcalls]
+
+[projectsv2api]: https://docs.github.com/en/issues/planning-and-tracking-with-projects/automating-your-project/using-the-api-to-manage-projects
+[formingcalls]: https://docs.github.com/en/graphql/guides/forming-calls-with-graphql
 
 
 ## License
 
 - [`LICENSE`](LICENSE) (Expat/[MIT][mit] License)
 
 [mit]: http://www.opensource.org/licenses/MIT "The MIT License | Open Source Initiative"
+
+
+### GitHub GraphQL API schema
+
+The GitHub GraphQL API public schema
+([`ccos/schema.docs.graphql`][graphqlschema]) was downloaded from [Public
+schema - GitHub Docs][publicschema] and is not within scope of the Expat/MIT
+license of this project.
+
+[graphqlschema]: ccos/schema.docs.graphql
+[publicschema]: https://docs.github.com/en/graphql/overview/public-schema

README.md

@@ -3,15 +3,32 @@
 import os
 import re
 import sys
+import textwrap
 
 # Third-party
 from github import Github
 from github.GithubException import BadCredentialsException
+from gql import Client, gql
+from gql.transport.requests import RequestsHTTPTransport
+from gql.transport.requests import log as gql_requests_log
+from graphql.error.syntax_error import GraphQLSyntaxError
+from pygments import highlight
+from pygments.formatters import TerminalFormatter
+from pygments.lexers import GraphQLLexer
 from urllib3.util.retry import Retry
 
 GITHUB_ORGANIZATION = "creativecommons"
+GITHUB_RETRY_STATUS_FORCELIST = [
+    408,  # Request Timeout
+    429,  # Too Many Requests
+    500,  # Internal Server Error
+    502,  # Bad Gateway
+    503,  # Service Unavailable
+    504,  # Gateway Timeout
+]
 GITHUB_USERNAME_DEFAULT = "cc-creativecommons-github-io-bot"
 LOG = logging.root
+gql_requests_log.setLevel(logging.WARNING)
 
 
 def get_credentials():
@@ -27,9 +44,43 @@ def get_credentials():
     return github_username, github_token
 
 
-def set_up_github_client():
+def gql_query(query):
+    try:
+        validated_query = gql(query)
+    except GraphQLSyntaxError as e:
+        query_formatted = highlight(
+            textwrap.indent(textwrap.dedent(query), "    "),
+            GraphQLLexer(),
+            TerminalFormatter(),
+        )
+        error_formatted = textwrap.indent(f"{e}", "    ")
+        LOG.error(
+            f"Invalid GraphQL syntax:\n{query_formatted}\n{error_formatted}"
+        )
+        sys.exit(1)
+    return validated_query
+
+
+def setup_github_gql_client():
     _, github_token = get_credentials()
-    LOG.info("Setting up GitHub client...")
+    LOG.info("Setting up GitHub GraphQL API client")
+    transport = RequestsHTTPTransport(
+        url="https://api.github.com/graphql",
+        headers={"Authorization": f"bearer {github_token}"},
+        timeout=10,
+        retries=5,
+        retry_backoff_factor=10,
+        retry_status_forcelist=GITHUB_RETRY_STATUS_FORCELIST,
+    )
+    with open("ccos/schema.docs.graphql") as file_obj:
+        gh_schema = file_obj.read()
+    github_gql_client = Client(transport=transport, schema=gh_schema)
+    return github_gql_client
+
+
+def setup_github_rest_client():
+    _, github_token = get_credentials()
+    LOG.info("Setting up GitHub Rest API client")
     # TODO: Remove retry parameter (urllib3.util.retry.Retry object) once we
     # are using PyGithub v2.0
     # https://github.com/creativecommons/ccos-scripts/issues/179
@@ -38,7 +89,7 @@ def set_up_github_client():
         # for specified HTTP status codes
         total=5,
         backoff_factor=10,
-        status_forcelist=list(range(500, 600)),
+        status_forcelist=GITHUB_RETRY_STATUS_FORCELIST,
         allowed_methods={
             "DELETE",
             "GET",
@@ -49,14 +100,13 @@ def set_up_github_client():
             "TRACE",
         },
     )
-    github_client = Github(login_or_token=github_token, retry=retry)
-    LOG.success("done.")
-    return github_client
+    github_rest_client = Github(login_or_token=github_token, retry=retry)
+    return github_rest_client
 
 
 def get_cc_organization(github_client=None):
     if github_client is None:
-        github_client = set_up_github_client()
+        github_client = setup_github_rest_client()
     LOG.info("Getting CC's GitHub organization...")
     try:
         gh_org_cc = github_client.get_organization(GITHUB_ORGANIZATION)

ccos/gh_utils.py

@@ -12,6 +12,7 @@ class IndentFormatter(logging.Formatter):
     indentation using tab characters.
     """
 
+    # https://en.wikipedia.org/wiki/ANSI_escape_code
     color_map = {  # ............. Level     ##    Color    ##
         logging.CRITICAL: 31,  # . CRITICAL  50    red      31
         logging.ERROR: 31,  # .... ERROR     40    red      31

ccos/log.py

@@ -0,0 +1,77 @@
+---
+Applications:
+  repos:
+  - .github
+  - cc-legal-tools-app
+  - cc-legal-tools-data
+  - ccos-scripts
+  - ccrel
+  - ccrel-guide
+  - creativecommons.github.io
+  - creativecommons.github.io-source
+  - creativecommons.org
+  - data-science-playground
+  - eng-misc-scripts
+  - faq
+  - legaldb
+  - licensebuttons
+  - magical-pony
+  - mp
+  - quantifying
+  - scholars-addenda
+  - sre-report-to-wikijs
+  - sre-wp-pull
+  - stateof
+Systems:
+  repos:
+  - ansible-dev
+  - CCID-cas
+  - CCID-MediaWiki
+  - index-performance-testing
+  - rubycas-client-rails
+  - sre-salt
+  - sre-salt-prime
+  - sre-wiki-js
+  - tech-support
+WebDev:
+  repos:
+  - attribution-license-plugin
+  - australian-chapter
+  - business-toolkit-files
+  - candela-utility
+  - cc-accidenz-commons
+  - cc-assets
+  - cc-license-chooser
+  - cc-resource-archive
+  - cc-usability-prototypes
+  - certificates-landing-page-theme
+  - chapters
+  - chooser
+  - commoners
+  - Creative-Commons-Post-Republisher
+  - creativecommons-base
+  - creativecommons-certificate
+  - global-network-strategy
+  - index-dev-env
+  - index-prototype
+  - network-platforms
+  - og-image-generator
+  - open4us.org
+  - queulat
+  - reversionary-rights
+  - search
+  - taaccct
+  - team-open
+  - termination-of-transfer
+  - the-power-of-open
+  - vocabulary
+  - vocabulary-theme
+  - wp-plugin-cc-global-network
+  - wp-plugin-creativecommons
+  - wp-plugin-creativecommons-website
+  - wp-theme-cc-chapter
+  - wp-theme-cc-commoners
+  - wp-theme-cctoolkit
+  - wp-theme-creativecommons.org
+  - wp-theme-openglam
+  - wp-theme-summit