ci: add markdownlint and json schema (#5545)

Updates to precommit:
* verify some markdown files (ignore examples for now)
  * update several markdown files to comply
* verify dependabot and workflow definitions
* version bump on yamllint

.determined_markdown.style

@@ -0,0 +1,11 @@
+# determiend-ai project markdown style
+all
+
+# line-length
+exclude_rule 'MD013'
+# inline HTML
+exclude_rule 'MD033'
+## trailing punctuation in header
+#exclude_rule 'MD026'
+# remove colon from default trailing punctuation chars
+rule 'MD026', :punctuation => '.,;!?'

.github/pull_request_template.md

@@ -1,38 +1,44 @@
 ## Description
 
 <!---
-Lead with the intended commit body in this description field. For breaking changes,
-please include "BREAKING CHANGE:" at the beginning of your commit body.
-At a minimum, this section should include a bracketed reference to the Jira ticket,
-e.g. "[DET-1234]". When squash-and-merging, copy this directly into the description field.
+Lead with the intended commit body in this description field. For breaking
+changes, please include "BREAKING CHANGE:" at the beginning of your commit
+body.  At a minimum, this section should include a bracketed reference to the
+Jira ticket, e.g. "[DET-1234]". When squash-and-merging, copy this directly
+into the description field.
 -->
 
 
+
 ## Test Plan
 
 <!---
-Describe the situations in which you've tested your change, and/or a screenshot as appropriate.
-Reviewers may ask questions about this test plan to ensure adequate manual coverage of changes.
+Describe the situations in which you've tested your change, and/or a screenshot
+as appropriate.  Reviewers may ask questions about this test plan to ensure
+adequate manual coverage of changes.
 -->
 
 
+
 ## Commentary (optional)
 
 <!---
-Use this section of your description to add context to the PR. Could be for particularly
-tricky bits of code that could use extra scrutiny, historical context useful for reviewers, etc.
+Use this section of your description to add context to the PR. Could be for
+particularly tricky bits of code that could use extra scrutiny, historical
+context useful for reviewers, etc.
 You may intentionally leave this section blank and remove the title.
 --->
 
 
+
 ## Checklist
+
 - [ ] Changes have been manually QA'd
 - [ ] User-facing API changes need the "User-facing API Change" label.
 - [ ] Release notes should be added as a separate file under `docs/release-notes/`.
-See [Release Note](https://github.com/determined-ai/determined/blob/master/docs/release-notes/README.md) for details.
+  See [Release Note](https://github.com/determined-ai/determined/blob/master/docs/release-notes/README.md) for details.
 - [ ] Licenses should be included for new code which was copied and/or modified from any external code.
 
-
 <!---
 ## Title
 
@@ -42,12 +48,14 @@ Specifically, this title should contain a type and a description
 of the change being made:
 
 User-facing change types:
+
 - docs: docs-only change
 - feat: new user-facing feature
 - fix: bug fix
 - perf: performance improvement
 
 Internal change types:
+
 - build: build system change (anything in a `Makefile`, mostly)
 - chore: any internal change not covered by another type
 - ci: anything that touches `.circleci`
@@ -58,8 +66,10 @@ Internal change types:
 See https://www.conventionalcommits.org/en/v1.0.0/ for background.
 
 The first line should also:
+
 - be at most 89 characters long
 - contain a description that is at most 72 characters long
 - not end with sentence-ending punctuation
 - start (after the type) with a lowercase imperative ("add", "fix")
+
 -->

.mdlrc

@@ -0,0 +1,5 @@
+# include the "custom" style
+# (we have to use a style definition to change rule parameters)
+style File.join(File.dirname(__FILE__), '.determined_markdown.style')
+# # enable parser warnings from Ruby
+# show_kramdown_warnings true

.pre-commit-config.yaml

@@ -1,3 +1,4 @@
+---
 repos:
   - repo: local
     hooks:
@@ -33,14 +34,58 @@ repos:
         args: [-w, -s, -l, -i, '4', -ci, -bn]
 
   - repo: https://github.com/adrienverge/yamllint.git
-    rev: v1.17.0
+    rev: v1.28.0
     hooks:
       - id: yamllint
-        files: '^.github/'
+        files: >
+          (?x)^(
+          .github/
+          | .pre-commit-config.yaml
+          )$
+
+  - repo: https://github.com/python-jsonschema/check-jsonschema
+    rev: 0.19.2
+    hooks:
+      - id: check-dependabot
+      - id: check-github-actions
+      - id: check-github-workflows
+
   - repo: https://github.com/psf/black
-    rev: '20.8b1'
+    rev: 20.8b1
     hooks:
       - id: black
-        # known issue fixed in a newer version: https://github.com/psf/black/issues/2964#issuecomment-1080974737
+        # known issue fixed in a newer version:
+        # https://github.com/psf/black/issues/2964#issuecomment-1080974737
         additional_dependencies: ['click==8.0.4']
         exclude: ^(examples|harness/determined/common/api/bindings.py)
+
+  - repo: https://github.com/jumanjihouse/pre-commit-hooks
+    rev: 3.0.0
+    hooks:
+      - id: markdownlint  # Configure in .mdlrc
+        exclude: >
+          (?x)^(
+          examples/.*
+          | model_hub/examples/.*
+          | (webui/react/src/shared/)?\.github/.*template\.md
+          | docs/release-notes/README\.md
+          )$
+      - id: markdownlint
+        name: special markdown lint for templates (skip header & blank lines)
+        args:
+          - --rules
+          - ~MD002,~MD041,~MD012
+        files: >
+          (?x)^(
+          (webui/react/src/shared/)?.github/.*template\.md
+          )$
+      - id: markdownlint
+        # bug in markdownlint handling ul inside of ol :/
+        # https://github.com/markdownlint/markdownlint/issues/313
+        name: special markdown lint for release notes (bullet indentation)
+        args:
+          - --rules
+          - ~MD007
+        files: ^docs/release-notes/README\.md$
+      # - id: git-check
+      # - id: git-dirty

CODE_OF_CONDUCT.md

@@ -63,4 +63,4 @@ maintain confidentiality with regard to the reporter of an incident.
 ## Attribution
 
 This Code of Conduct is adapted from the Contributor Covenant, version 1.4,
-available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+available at <https://www.contributor-covenant.org/version/1/4/code-of-conduct.html>

CONTRIBUTING.md

@@ -9,7 +9,7 @@ to get support and talk with other users and developers in real-time.
 
 ## Project Roadmap
 
-https://github.com/determined-ai/determined/wiki/Project-Roadmap
+<https://github.com/determined-ai/determined/wiki/Project-Roadmap>
 
 ## Contributing Changes
 
@@ -52,9 +52,9 @@ git clone --recurse-submodules https://github.com/determined-ai/determined.git
 
 - Go (>= 1.18)
 - Python (>= 3.6, <= 3.9), including:
-   - python3-venv
-   - python3-wheel
-   - python3-dev
+  - python3-venv
+  - python3-wheel
+  - python3-dev
 - Node (>= 16.13, < 17)
 - NPM (>= 8)
 - Docker (>= 19.03)
@@ -65,14 +65,14 @@ git clone --recurse-submodules https://github.com/determined-ai/determined.git
 - jq (>= 1.6)
 - socat (>= 1.7)
 
-If you are installing prerequisites from your Linux distribution's package 
-repository, ensure that they meet the version requirements above, particularly 
+If you are installing prerequisites from your Linux distribution's package
+repository, ensure that they meet the version requirements above, particularly
 Python and Node.
 
 #### Install Prerequisites with Homebrew for Linux and macOS
 
-Because the versions of prerequisites from Linux distribution package 
-repositories can vary widely, we recommend installing the Determined build 
+Because the versions of prerequisites from Linux distribution package
+repositories can vary widely, we recommend installing the Determined build
 prerequisites with [Homebrew](https://brew.sh/).
 
 The following instructions are also applicable for building Determined on macOS.
@@ -135,12 +135,12 @@ or [direnv](https://direnv.net/) may help streamline the process.
 
 ### Building Errata
 
-If you are building Determined using Python 3.9, you must manually change the 
+If you are building Determined using Python 3.9, you must manually change the
 version of TensorFlow to 2.8.0 in the [end-to-end testing requirements file](https://github.com/determined-ai/determined/blob/master/e2e_tests/tests/requirements.txt#L8)
 because the version of TensorFlow currently set in the end-to-end requirements
 file is not compatible with Python 3.9:
 
-```
+```plain
 tensorflow==2.8.0; sys_platform != 'darwin' or platform_machine != 'arm64'
 ```
 
@@ -157,7 +157,7 @@ of master and one or more local agents.
 ### Accessing Determined
 
 After following either set of instructions above, the WebUI will be available at
-http://localhost:8080. You can also use our command-line tool, `det`, to
+`http://localhost:8080`. You can also use our command-line tool, `det`, to
 interact with Determined. For example, `det slot list` should print out a line
 for each GPU on your machine, if you have any, or a line for your CPU, if not.
 For more information, see [the reference
@@ -242,7 +242,6 @@ If this command displays one or more GPUs, the Determined agent should
 automatically detect the system's GPUs and make them available for
 running experiments.
 
-
 ### Secrets
 
 To prevent cloud credentials from accidentally being exposed on GitHub, install
@@ -253,6 +252,7 @@ For Mac, the tool can be installed via `brew install git-secrets`. For other
 OSes see installation instructions [here](https://github.com/awslabs/git-secrets#installing-git-secrets).
 
 Then navigate to the repository, set up the git hooks, and define the regexes:
+
 ```shell
 cd /path/to/my/repository
 

README.md

@@ -1,17 +1,17 @@
-<p align="center"><img src="determined-logo.png" alt="Determined AI Logo"></p>
-
 # Determined: Deep Learning Training Platform
 
+<p align="center"><img src="determined-logo.png" alt="Determined AI Logo"></p>
+
 Determined is an open-source deep learning training platform that makes building
 models fast and easy. Determined enables you to:
 
-- **Train models faster** using state-of-the-art distributed training, without
+* **Train models faster** using state-of-the-art distributed training, without
   changing your model code
-- **Automatically find high-quality models** with advanced hyperparameter tuning
+* **Automatically find high-quality models** with advanced hyperparameter tuning
   from the creators of Hyperband
-- **Get more from your GPUs** with smart scheduling and cut cloud GPU costs by
+* **Get more from your GPUs** with smart scheduling and cut cloud GPU costs by
   seamlessly using preemptible instances
-- **Track and reproduce your work** with experiment tracking that works
+* **Track and reproduce your work** with experiment tracking that works
   out-of-the-box, covering code versions, metrics, checkpoints, and
   hyperparameters
 

docs/release-notes/README.md

@@ -26,13 +26,13 @@ classifying the release note according to one of the following categories:
    The author of the PR has the most context about the change being made and should be
    aware of any caveats or additional context that users should be informed about.
 
-2. Write the release note entry as a separate file and add it to the
+1. Write the release note entry as a separate file and add it to the
    `determined/docs/release-notes` directory. See the [example](#release-note-example), below.
 
-   Writing guidelines can be relaxed for a release note to be more conversational than might
-   be acceptable in the core documentation. Spelling, grammar, coherence, and completeness
-   requirements still apply. Passing [Grammarly](https://app.grammarly.com/) checks is usually
-   sufficient.
+   Writing guidelines can be relaxed for a release note to be more
+   conversational than might be acceptable in the core documentation. Spelling,
+   grammar, coherence, and completeness requirements still apply. Passing
+   [Grammarly](https://app.grammarly.com/) checks is usually sufficient.
 
    * Give the file a descriptive name related to the change.
    * Write the release note using [reStructuredText](https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html), following Determined documentation [style conventions](https://determinedai.atlassian.net/l/c/53h3PrPo).
@@ -63,46 +63,50 @@ classifying the release note according to one of the following categories:
 
      **Note:** For a release note that might have particular significance for the user, use an `Important` admonition and highlight it. For example:
 
-         **Security Fixes**
-
-         -  CLI: **Important:** API requests executed through the Python bindings have been erroneously using the SSL
-            "noverify" option since version 0.17.6, making them potentially insecure. The option is now disabled.
+     > **Security Fixes**
+     >
+     > *  CLI: **Important:** API requests executed through the Python bindings have been erroneously using the SSL
+     >    "noverify" option since version 0.17.6, making them potentially insecure. The option is now disabled.
 
    * In one or more list items, provide additional information. Describe:
 
      * what changed
      * why the change was needed
      * how the change affects the user
 
-     Do not give details about how the change was implemented.
+       Do not give details about how the change was implemented.
 
-     If there is a Jira issue associated with the change, the Jira **Description** field, with some rewording can be used as the description.
+       If there is a Jira issue associated with the change, the Jira **Description** field, with some rewording can be used as the description.
 
-     Be sure to highlight API changes and backward incompatibility, including steps needed to upgrade safely.
+       Be sure to highlight API changes and backward incompatibility, including steps needed to upgrade safely.
 
-     Do not include:
+       Do not include:
 
-     * links to Determined documentation
-     * customer identifiers
-     * internal project status or plans
-     * Jira issue or PR identifiers
+       * links to Determined documentation
+       * customer identifiers
+       * internal project status or plans
+       * Jira issue or PR identifiers
 
 ### Release Note Example
 
-```
+```markdown
 :orphan:
 
 **New Features**
 
--  GCP: Add support for provisioning Nvidia A100 GPU instances.
+*  GCP: Add support for provisioning Nvidia A100 GPU instances.
 
-   -  Running workloads on A100 chips currently requires building a custom task environment with CUDA 11, because the default task environments provided by Determined contain either CUDA 10.0 or CUDA 10.1. The default task environments will be upgraded to CUDA 11 in a future release of Determined.
+   *  Running workloads on A100 chips currently requires building a custom task
+      environment with CUDA 11, because the default task environments provided
+      by Determined contain either CUDA 10.0 or CUDA 10.1. The default task
+      environments will be upgraded to CUDA 11 in a future release of
+      Determined.
 ```
 
 ## How to Collect and Publish the Release Notes for a Release
 
 As part of the release process, the release manager:
 
 1. Prepends the individual `docs/release-notes/` files to the `docs/release-notes.txt` file and creates a new version heading.
-2. Deletes the individual files from `docs/release-notes/`.
-3. Performs additional copy editing as needed.
+1. Deletes the individual files from `docs/release-notes/`.
+1. Performs additional copy editing as needed.