diff --git a/.determined_markdown.style b/.determined_markdown.style new file mode 100644 index 00000000000..376990f1d16 --- /dev/null +++ b/.determined_markdown.style @@ -0,0 +1,11 @@ +# determiend-ai project markdown style +all + +# line-length +exclude_rule 'MD013' +# inline HTML +exclude_rule 'MD033' +## trailing punctuation in header +#exclude_rule 'MD026' +# remove colon from default trailing punctuation chars +rule 'MD026', :punctuation => '.,;!?' diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 0d8c0fc99aa..503be9b7ef1 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,38 +1,44 @@ ## Description + ## Test Plan + ## Commentary (optional) + ## Checklist + - [ ] Changes have been manually QA'd - [ ] User-facing API changes need the "User-facing API Change" label. - [ ] Release notes should be added as a separate file under `docs/release-notes/`. -See [Release Note](https://github.com/determined-ai/determined/blob/master/docs/release-notes/README.md) for details. + See [Release Note](https://github.com/determined-ai/determined/blob/master/docs/release-notes/README.md) for details. - [ ] Licenses should be included for new code which was copied and/or modified from any external code. - diff --git a/.mdlrc b/.mdlrc new file mode 100644 index 00000000000..d23599d7ca4 --- /dev/null +++ b/.mdlrc @@ -0,0 +1,5 @@ +# include the "custom" style +# (we have to use a style definition to change rule parameters) +style File.join(File.dirname(__FILE__), '.determined_markdown.style') +# # enable parser warnings from Ruby +# show_kramdown_warnings true diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index b320fd138c5..d7922246632 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,3 +1,4 @@ +--- repos: - repo: local hooks: @@ -33,14 +34,58 @@ repos: args: [-w, -s, -l, -i, '4', -ci, -bn] - repo: https://github.com/adrienverge/yamllint.git - rev: v1.17.0 + rev: v1.28.0 hooks: - id: yamllint - files: '^.github/' + files: > + (?x)^( + .github/ + | .pre-commit-config.yaml + )$ + + - repo: https://github.com/python-jsonschema/check-jsonschema + rev: 0.19.2 + hooks: + - id: check-dependabot + - id: check-github-actions + - id: check-github-workflows + - repo: https://github.com/psf/black - rev: '20.8b1' + rev: 20.8b1 hooks: - id: black - # known issue fixed in a newer version: https://github.com/psf/black/issues/2964#issuecomment-1080974737 + # known issue fixed in a newer version: + # https://github.com/psf/black/issues/2964#issuecomment-1080974737 additional_dependencies: ['click==8.0.4'] exclude: ^(examples|harness/determined/common/api/bindings.py) + + - repo: https://github.com/jumanjihouse/pre-commit-hooks + rev: 3.0.0 + hooks: + - id: markdownlint # Configure in .mdlrc + exclude: > + (?x)^( + examples/.* + | model_hub/examples/.* + | (webui/react/src/shared/)?\.github/.*template\.md + | docs/release-notes/README\.md + )$ + - id: markdownlint + name: special markdown lint for templates (skip header & blank lines) + args: + - --rules + - ~MD002,~MD041,~MD012 + files: > + (?x)^( + (webui/react/src/shared/)?.github/.*template\.md + )$ + - id: markdownlint + # bug in markdownlint handling ul inside of ol :/ + # https://github.com/markdownlint/markdownlint/issues/313 + name: special markdown lint for release notes (bullet indentation) + args: + - --rules + - ~MD007 + files: ^docs/release-notes/README\.md$ + # - id: git-check + # - id: git-dirty diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index d83c8006797..956f1a1779e 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -63,4 +63,4 @@ maintain confidentiality with regard to the reporter of an incident. ## Attribution This Code of Conduct is adapted from the Contributor Covenant, version 1.4, -available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html +available at diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f8771a5c7d7..ffb0050a3f0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -9,7 +9,7 @@ to get support and talk with other users and developers in real-time. ## Project Roadmap -https://github.com/determined-ai/determined/wiki/Project-Roadmap + ## Contributing Changes @@ -52,9 +52,9 @@ git clone --recurse-submodules https://github.com/determined-ai/determined.git - Go (>= 1.18) - Python (>= 3.6, <= 3.9), including: - - python3-venv - - python3-wheel - - python3-dev + - python3-venv + - python3-wheel + - python3-dev - Node (>= 16.13, < 17) - NPM (>= 8) - Docker (>= 19.03) @@ -65,14 +65,14 @@ git clone --recurse-submodules https://github.com/determined-ai/determined.git - jq (>= 1.6) - socat (>= 1.7) -If you are installing prerequisites from your Linux distribution's package -repository, ensure that they meet the version requirements above, particularly +If you are installing prerequisites from your Linux distribution's package +repository, ensure that they meet the version requirements above, particularly Python and Node. #### Install Prerequisites with Homebrew for Linux and macOS -Because the versions of prerequisites from Linux distribution package -repositories can vary widely, we recommend installing the Determined build +Because the versions of prerequisites from Linux distribution package +repositories can vary widely, we recommend installing the Determined build prerequisites with [Homebrew](https://brew.sh/). The following instructions are also applicable for building Determined on macOS. @@ -135,12 +135,12 @@ or [direnv](https://direnv.net/) may help streamline the process. ### Building Errata -If you are building Determined using Python 3.9, you must manually change the +If you are building Determined using Python 3.9, you must manually change the version of TensorFlow to 2.8.0 in the [end-to-end testing requirements file](https://github.com/determined-ai/determined/blob/master/e2e_tests/tests/requirements.txt#L8) because the version of TensorFlow currently set in the end-to-end requirements file is not compatible with Python 3.9: -``` +```plain tensorflow==2.8.0; sys_platform != 'darwin' or platform_machine != 'arm64' ``` @@ -157,7 +157,7 @@ of master and one or more local agents. ### Accessing Determined After following either set of instructions above, the WebUI will be available at -http://localhost:8080. You can also use our command-line tool, `det`, to +`http://localhost:8080`. You can also use our command-line tool, `det`, to interact with Determined. For example, `det slot list` should print out a line for each GPU on your machine, if you have any, or a line for your CPU, if not. For more information, see [the reference @@ -242,7 +242,6 @@ If this command displays one or more GPUs, the Determined agent should automatically detect the system's GPUs and make them available for running experiments. - ### Secrets To prevent cloud credentials from accidentally being exposed on GitHub, install @@ -253,6 +252,7 @@ For Mac, the tool can be installed via `brew install git-secrets`. For other OSes see installation instructions [here](https://github.com/awslabs/git-secrets#installing-git-secrets). Then navigate to the repository, set up the git hooks, and define the regexes: + ```shell cd /path/to/my/repository diff --git a/README.md b/README.md index 773be591626..f0ac8f9a996 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,17 @@ -

Determined AI Logo

- # Determined: Deep Learning Training Platform +

Determined AI Logo

+ Determined is an open-source deep learning training platform that makes building models fast and easy. Determined enables you to: -- **Train models faster** using state-of-the-art distributed training, without +* **Train models faster** using state-of-the-art distributed training, without changing your model code -- **Automatically find high-quality models** with advanced hyperparameter tuning +* **Automatically find high-quality models** with advanced hyperparameter tuning from the creators of Hyperband -- **Get more from your GPUs** with smart scheduling and cut cloud GPU costs by +* **Get more from your GPUs** with smart scheduling and cut cloud GPU costs by seamlessly using preemptible instances -- **Track and reproduce your work** with experiment tracking that works +* **Track and reproduce your work** with experiment tracking that works out-of-the-box, covering code versions, metrics, checkpoints, and hyperparameters diff --git a/docs/release-notes/README.md b/docs/release-notes/README.md index 904e3d2b00f..0a6db1479cf 100644 --- a/docs/release-notes/README.md +++ b/docs/release-notes/README.md @@ -26,13 +26,13 @@ classifying the release note according to one of the following categories: The author of the PR has the most context about the change being made and should be aware of any caveats or additional context that users should be informed about. -2. Write the release note entry as a separate file and add it to the +1. Write the release note entry as a separate file and add it to the `determined/docs/release-notes` directory. See the [example](#release-note-example), below. - Writing guidelines can be relaxed for a release note to be more conversational than might - be acceptable in the core documentation. Spelling, grammar, coherence, and completeness - requirements still apply. Passing [Grammarly](https://app.grammarly.com/) checks is usually - sufficient. + Writing guidelines can be relaxed for a release note to be more + conversational than might be acceptable in the core documentation. Spelling, + grammar, coherence, and completeness requirements still apply. Passing + [Grammarly](https://app.grammarly.com/) checks is usually sufficient. * Give the file a descriptive name related to the change. * Write the release note using [reStructuredText](https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html), following Determined documentation [style conventions](https://determinedai.atlassian.net/l/c/53h3PrPo). @@ -63,10 +63,10 @@ classifying the release note according to one of the following categories: **Note:** For a release note that might have particular significance for the user, use an `Important` admonition and highlight it. For example: - **Security Fixes** - - - CLI: **Important:** API requests executed through the Python bindings have been erroneously using the SSL - "noverify" option since version 0.17.6, making them potentially insecure. The option is now disabled. + > **Security Fixes** + > + > * CLI: **Important:** API requests executed through the Python bindings have been erroneously using the SSL + > "noverify" option since version 0.17.6, making them potentially insecure. The option is now disabled. * In one or more list items, provide additional information. Describe: @@ -74,29 +74,33 @@ classifying the release note according to one of the following categories: * why the change was needed * how the change affects the user - Do not give details about how the change was implemented. + Do not give details about how the change was implemented. - If there is a Jira issue associated with the change, the Jira **Description** field, with some rewording can be used as the description. + If there is a Jira issue associated with the change, the Jira **Description** field, with some rewording can be used as the description. - Be sure to highlight API changes and backward incompatibility, including steps needed to upgrade safely. + Be sure to highlight API changes and backward incompatibility, including steps needed to upgrade safely. - Do not include: + Do not include: - * links to Determined documentation - * customer identifiers - * internal project status or plans - * Jira issue or PR identifiers + * links to Determined documentation + * customer identifiers + * internal project status or plans + * Jira issue or PR identifiers ### Release Note Example -``` +```markdown :orphan: **New Features** -- GCP: Add support for provisioning Nvidia A100 GPU instances. +* GCP: Add support for provisioning Nvidia A100 GPU instances. - - Running workloads on A100 chips currently requires building a custom task environment with CUDA 11, because the default task environments provided by Determined contain either CUDA 10.0 or CUDA 10.1. The default task environments will be upgraded to CUDA 11 in a future release of Determined. + * Running workloads on A100 chips currently requires building a custom task + environment with CUDA 11, because the default task environments provided + by Determined contain either CUDA 10.0 or CUDA 10.1. The default task + environments will be upgraded to CUDA 11 in a future release of + Determined. ``` ## How to Collect and Publish the Release Notes for a Release @@ -104,5 +108,5 @@ classifying the release note according to one of the following categories: As part of the release process, the release manager: 1. Prepends the individual `docs/release-notes/` files to the `docs/release-notes.txt` file and creates a new version heading. -2. Deletes the individual files from `docs/release-notes/`. -3. Performs additional copy editing as needed. +1. Deletes the individual files from `docs/release-notes/`. +1. Performs additional copy editing as needed. diff --git a/harness/determined/pytorch/README.md b/harness/determined/pytorch/README.md index 889b88cf53c..a387a618634 100644 --- a/harness/determined/pytorch/README.md +++ b/harness/determined/pytorch/README.md @@ -3,7 +3,6 @@ "Most of PyTorchTrial is simple. But the Horovod / AMP / Gradient Aggregation combo is not." - ## Basics of PyTorch ### The forward pass @@ -24,7 +23,6 @@ Gather up all of the gradents for the tensors this optimizer is responsible for tuning, and apply the `gradient * learning_rate` to each weight. Now you have new weights, congratulations! - ## Horovod optimizer The horovod optimizer calculates gradient updates on each worker, then @@ -75,7 +73,6 @@ This is not AMP, and it does not behave like the GradScaler. It just blindly casts gradient update tensors to fp16 for network communication and blindly casts them back to fp32 afterwards. - ## Gradient Aggregation: Gradient aggregation by a factor of N is just like multiplying the batch size @@ -87,16 +84,15 @@ such as with batch norm). The only operation required for gradient aggregation are: - * do N forward/backward passes before doing anything with the gradients that - result from those backward passes. Those in-place gradients will add - together naturally. - - * Divide the gradients by N to match the behavior of an N-times-larger batch - size. +* do N forward/backward passes before doing anything with the gradients that + result from those backward passes. Those in-place gradients will add + together naturally. - * Call optimizer.step() after every N batches to act on the aggregated - gradients. +* Divide the gradients by N to match the behavior of an N-times-larger batch + size. +* Call optimizer.step() after every N batches to act on the aggregated + gradients. ## PyTorch-Native AMP: diff --git a/master/static/migrations/README.md b/master/static/migrations/README.md index 414129ec81d..426c45d5b91 100644 --- a/master/static/migrations/README.md +++ b/master/static/migrations/README.md @@ -1,6 +1,6 @@ # Database Migrations -We use `go-pg/migrations`: https://github.com/go-pg/migrations +We use `go-pg/migrations`: ## Running migrations manually diff --git a/model_hub/README.md b/model_hub/README.md index e1eeddaae30..d70ff1d0448 100644 --- a/model_hub/README.md +++ b/model_hub/README.md @@ -1,19 +1,25 @@ # Determined's Model-Hub Library -Determined's Model-Hub library makes it super easy to use popular external libraries with -Determined's training platform. Support for each external library will provide a Trial interface, -helper functions, and official examples to allows users to easily use these libraries with all of -Determined features (e.g., distributed training, experiment tracking, hyperparameter search). + +Determined's Model-Hub library makes it super easy to use popular external +libraries with Determined's training platform. Support for each external +library will provide a Trial interface, helper functions, and official examples +to allows users to easily use these libraries with all of Determined features +(e.g., distributed training, experiment tracking, hyperparameter search). ## Supported Libraries -Model-Hub currently supports + +Model-Hub currently supports + * [Huggingface transformers](https://github.com/huggingface/transformers) * [MMDetection](https://github.com/open-mmlab/mmdetection) -Please refer to [our documentation](https://docs.determined.ai/latest/model-hub/index.html) for -more information on how to use these libraries with Determined. +Please refer to [our documentation]( +https://docs.determined.ai/latest/model-hub/index.html) for more information on +how to use these libraries with Determined. Libraries we are working to support in the future include: + * [detectron2](https://github.com/facebookresearch/detectron2) -If there is a library you want us to support for use with Determined, please open an issue -to submit a request. +If there is a library you want us to support for use with Determined, please +open an issue to submit a request. diff --git a/schemas/README.md b/schemas/README.md index 9adb1640a04..c86b26d18fe 100644 --- a/schemas/README.md +++ b/schemas/README.md @@ -5,72 +5,72 @@ for validation, null handling, default values, and some custom rules. - To make changes to the configuration schema, change the according files below: - - See `schemas/expconf` for logics that are shared across languages. - - See `schemas/test_cases` for test cases that are shared across languages. - - See `master/pkg/schemas/expconf` for go struct definitions. - - See `harness/determined/common/schemas/expconf` for python class definitions. + - See `schemas/expconf` for logics that are shared across languages. + - See `schemas/test_cases` for test cases that are shared across languages. + - See `master/pkg/schemas/expconf` for go struct definitions. + - See `harness/determined/common/schemas/expconf` for python class definitions. - We generate code that contains the definitions and utility functions of structs. See `schemas/gen.py`. - Validation: - - Validation happens in two steps: - - Sanity check: make sure everything is parsable - - Validation: make sure everything is present - - Sanity check is applied to raw user inputs - - Validation is applied after cluster defaults have been applied - - The implementation difference between sanity check and validation is the - inclusion of a single `eventuallyRequired` schema in the latter. + - Validation happens in two steps: + - Sanity check: make sure everything is parsable + - Validation: make sure everything is present + - Sanity check is applied to raw user inputs + - Validation is applied after cluster defaults have been applied + - The implementation difference between sanity check and validation is the + inclusion of a single `eventuallyRequired` schema in the latter. - Null handling: - - Anything with a null value is treated as not present. - - Reason: there is no pythonic or golangic way to represent values which were - provided in the configuration as literal nulls, rather than values which - were not provided at all. In theory, you could have singleton - "NotProvided" pointers, which you would check for every time that you - checked a value in the schema, but in practice that would be a pain in the - ass. Additionally, the golang json-schema library we use treats - not-present values a nil values anyway. + - Anything with a null value is treated as not present. + - Reason: there is no pythonic or golangic way to represent values which were + provided in the configuration as literal nulls, rather than values which + were not provided at all. In theory, you could have singleton + "NotProvided" pointers, which you would check for every time that you + checked a value in the schema, but in practice that would be a pain. + Additionally, the golang json-schema library we use treats not-present + values a nil values anyway. - Default values: - - Default values are defined in JSON Schema. - - However, other default values might be populated on the master side. + - Default values are defined in JSON Schema. + - However, other default values might be populated on the master side. - Customization rules: - - We define some JSON schema extensions to provide the behavior we need. - JSON Schema itself does not provide the behavior we want out of the box. - Most JSON Schema libraries have great support for custom validation logic. - - We have json-schema extensions for the following keywords: - - `checks`: Custom messages for arbitrary validation logic - - `compareProperties`: Support customizable value comparisons - - `disallowProperties`: Custom messages when disallowing properties - - `eventuallyRequired`: Support two-step validation - - `union`: Excellent error messages when validating union types - - `optionalRef`: like `$ref`, but only enforced for non-null values - - `eventually`: Defer validation of inner clause till completeness validation phase - - The canonical implementations (with thorough comments) may be found in - `harness/determined/common/schemas/extensions.py`. + - We define some JSON schema extensions to provide the behavior we need. + JSON Schema itself does not provide the behavior we want out of the box. + Most JSON Schema libraries have great support for custom validation logic. + - We have json-schema extensions for the following keywords: + - `checks`: Custom messages for arbitrary validation logic + - `compareProperties`: Support customizable value comparisons + - `disallowProperties`: Custom messages when disallowing properties + - `eventuallyRequired`: Support two-step validation + - `union`: Excellent error messages when validating union types + - `optionalRef`: like `$ref`, but only enforced for non-null values + - `eventually`: Defer validation of inner clause till completeness validation phase + - The canonical implementations (with thorough comments) may be found in + `harness/determined/common/schemas/extensions.py`. - Migration and Versioning: - - Migration logics are implemented in the master. See `pkg/schemas/expconf/parse.go`. - - Bumping the version is only necessary when previously-valid configs become - no longer valid. For instance, removing a once-available field requires a - version bump, but appending a new field or accepting a new type in an - existing field does not. - - To bump versions, take the following steps (assume you are adding `v2`): - - create a new `v2` directory, like `schemas/expconf/v2` - - copy the json schemas files you are updating from the `v1` directory into - the `v2` directory. Also copy any files which referenced any of those - files; for example, if you update `v1/hyperparameters.json`, you'll - have to copy `v1/experiment-config.json` as well since its `$ref` will need - to now point to the `v2/hyperparameters.json`. - - Populate `test_cases/v2` with test cases to the new json schemas you just - made. - - Make the golang changes: - - Create the corresponding objects in `pkg/schemas/expconf`, such as a - new `ExperimentConfigV2` and new `HyperparametersV2` - - update `expconf/latest.go` to point to the correct new object - - update any uses of that object throughout the codebase to reflect - the new structure. - - ensure that `make gen && go test ./pkg/schemas/expconf` passes - - Currently, no python or JS changes are needed. + - Migration logics are implemented in the master. See `pkg/schemas/expconf/parse.go`. + - Bumping the version is only necessary when previously-valid configs become + no longer valid. For instance, removing a once-available field requires a + version bump, but appending a new field or accepting a new type in an + existing field does not. + - To bump versions, take the following steps (assume you are adding `v2`): + - create a new `v2` directory, like `schemas/expconf/v2` + - copy the json schemas files you are updating from the `v1` directory + into the `v2` directory. Also copy any files which referenced any of + those files; for example, if you update `v1/hyperparameters.json`, + you'll have to copy `v1/experiment-config.json` as well since its + `$ref` will need to now point to the `v2/hyperparameters.json`. + - Populate `test_cases/v2` with test cases to the new json schemas you + just made. + - Make the golang changes: + - Create the corresponding objects in `pkg/schemas/expconf`, such as a + new `ExperimentConfigV2` and new `HyperparametersV2` + - update `expconf/latest.go` to point to the correct new object + - update any uses of that object throughout the codebase to reflect + the new structure. + - ensure that `make gen && go test ./pkg/schemas/expconf` passes + - Currently, no python or JS changes are needed. diff --git a/tools/k8s/README.md b/tools/k8s/README.md index 51bc21956e2..65dd5cacf12 100644 --- a/tools/k8s/README.md +++ b/tools/k8s/README.md @@ -23,7 +23,7 @@ create an API server that is not generally accessible. Read `tools/k8s/my-service-account.yaml` so you know what it's doing, then apply it: -``` +```sh kubectl apply -f tools/k8s/my-service-account.yaml ``` @@ -33,6 +33,6 @@ This step is only required once after creating your cluster. Read `tools/k8s/devcluster.yaml` so you know what it's doing, then run it: -``` +```sh devcluster -c tools/k8s/devcluster.yaml ``` diff --git a/tools/scripts/README.md b/tools/scripts/README.md index 479eef5b20a..62e65719883 100644 --- a/tools/scripts/README.md +++ b/tools/scripts/README.md @@ -25,13 +25,13 @@ be registered in the determined repo. Here is the process: ./refresh-ubuntu-amis.py --packer-json /path/to/environments-packer.json ``` -2. Wait for the post-merge-to-master CircleCI jobs on the environments repo to +1. Wait for the post-merge-to-master CircleCI jobs on the environments repo to finish. These will publish the relevant Docker/AWS/GCP images and create machine-readable artifacts containing the image tags. -3. Enter the `tools/scripts` directory of the determined repo. +1. Enter the `tools/scripts` directory of the determined repo. -4. Run `./update-bumpenvs-yaml.py bumpenvs.yaml THECOMMIT`. This will fetch the +1. Run `./update-bumpenvs-yaml.py bumpenvs.yaml THECOMMIT`. This will fetch the above-mentioned machine-readable artifacts from the CircleCI jobs of the environments repository, parse out the image tags, and update the relevant entries in `bumpenvs.yaml`. For every artifact found, this will set that @@ -39,13 +39,13 @@ be registered in the determined repo. Here is the process: value to the artifact produced by CI, including the task environments and the agent AMIs. -5. Run `./refresh-ubuntu-amis.py --bumpenvs-yaml bumpenvs.yaml`. This will +1. Run `./refresh-ubuntu-amis.py --bumpenvs-yaml bumpenvs.yaml`. This will fetch the up-to-date Ubuntu AMIs for each region for each of the `*_master_ami` and `*_bastion_ami` image tags in bumpenvs.yaml. This isn't strictly necessary; we just need to run it periodically, and now is a fine time. -6. Run `./bumpenvs.py bumpenvs.yaml`. This will do a simple string replacement +1. Run `./bumpenvs.py bumpenvs.yaml`. This will do a simple string replacement in the repository, replacing the `old` values with the `new` values for every image type identified in `bumpenvs.yaml`. @@ -67,20 +67,34 @@ headers contain metadata about the license and everything after the empty line should be the exact copy/pasted license text from the project in question. Here is an example: - Name: github.com/lib/pq - Type: mit - Agent: false - Master: true - Webui: false - - Copyright (c) 2011-2013, 'pq' Contributors - Portions Copyright (C) 2011 Blake Mizerany - - Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: - - The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. - - THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +```plain +Name: github.com/lib/pq +Type: mit +Agent: false +Master: true +Webui: false + +Copyright (c) 2011-2013, 'pq' Contributors +Portions Copyright (C) 2011 Blake Mizerany + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +``` The relevant metadata are: diff --git a/webui/README.md b/webui/README.md index 3ccd3411734..1cc91e0dd17 100644 --- a/webui/README.md +++ b/webui/README.md @@ -4,8 +4,8 @@ We use [React](https://reactjs.org/) as a WebUI framework. React has been the mo popular frontend framework and enjoys the benefits of being a mature and stable framework. As a result we see these benefits: -* True live development where new written code instantly transpiles into updated -browser experience via Hot Module Replacement (HMR). +* True live development where new written code instantly transpiles into + updated browser experience via Hot Module Replacement (HMR). * A very large number of supporting libraries and frameworks specific to React. * Many of the bugs and kinks have been identified and fixed. * Stable design patterns have mostly been figured out. @@ -18,22 +18,21 @@ Before starting this section, please get Determined set up properly by following the [Determined setup instructions](https://github.com/determined-ai/determined). Starting the master kick-starts a Go web server that serves WebUI static files. -There are two ways to start master. The more common way is to run master via Docker. -The other method is to run natively (without Docker) via `determined-master`. +There are two ways to start master. The more common way is to run master via +Docker. The other method is to run natively (without Docker) via +`determined-master`. 1. [Running Master via Docker](https://github.com/determined-ai/determined#local-deployment) -2. [Running Master without Docker (Natively)](https://github.com/determined-ai/determined/wiki/Useful-tools#master) - +1. [Running Master without Docker (Natively)](https://github.com/determined-ai/determined/wiki/Useful-tools#master) ## Local Development For local development, our goal is to set up an environment to... * Auto detect changes in the source code and update the WebUI on the browser to -speed up development. Also known as [Hot Module Replacement](https://webpack.js.org/concepts/hot-module-replacement/) + speed up development. Also known as [Hot Module Replacement](https://webpack.js.org/concepts/hot-module-replacement/) * Provide a debugging environment via source maps. Only applicable to the React SPA. - ### Running React Live To start React live, simply run the following and point your browser to `http://localhost:3000/`. @@ -59,7 +58,7 @@ Couple of things to note: To run unit tests for each of the SPAs issue `make test` in their respective directories. -``` +```sh make -C webui/react test ``` diff --git a/webui/react/README.md b/webui/react/README.md index 42d4d4ed067..8acf1c25a55 100644 --- a/webui/react/README.md +++ b/webui/react/README.md @@ -148,9 +148,9 @@ The bundle analyzer will look at the generated source maps for the `build` direc We are heavily leveraging a lot of goodness from **Create React App** discussed above. To continue benefitting from it, we need to avoid ejecting the project. Meaning we do not want to start managing the webpack configuration. The `npm run eject` command is a one-way operation and once you do it, **there is no going back**! The following describes what exactly happens when you do eject. > If you aren’t satisfied with the build tool and configuration choices, you can `eject` at any time. This command will remove the single build dependency from your project. - +> > Instead, it will copy all the configuration files and the transitive dependencies (Webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except `eject` will still work, but they will point to the copied scripts so you can tweak them. At this point you’re on your own. - +> > You don’t have to ever use `eject`. The curated feature set is suitable for small and middle deployments, and you shouldn’t feel obligated to use this feature. However we understand that this tool wouldn’t be useful if you couldn’t customize it when you are ready for it. All that being said, we do require some customization for library support, so we have a way around it described in the next section. diff --git a/webui/react/src/shared/README.md b/webui/react/src/shared/README.md index d13137e10e9..e51c60a4b92 100644 --- a/webui/react/src/shared/README.md +++ b/webui/react/src/shared/README.md @@ -4,8 +4,8 @@ We use this directory for shareable code snippets, UI components, etc. - separate reusable and shareable code across different Determined UIs - leave build and transpliation to target projects -- stepping stone for going to a solution such as packaging the components individually - or separately using Bit or Node registry. +- stepping stone for going to a solution such as packaging the components + individually or separately using Bit or Node registry. - assume mostly similar build and dev environments: - lower overhead, faster start up, simpler, faster, and smaller builds - familiarity: no new code management tool to learn @@ -19,7 +19,8 @@ Read more about the methodology and tradeoffs [here](https://hpe-my.sharepoint.c ## How -Code in this directory should not depend on internal dependencies that leave outside this directory +Code in this directory should not depend on internal dependencies that leave +outside this directory ## TODO