From 797ceeca45f7183ccd8f3c3706c19528ab9daf89 Mon Sep 17 00:00:00 2001 From: Danny Zhu Date: Mon, 9 May 2022 15:33:53 -0700 Subject: [PATCH] docs: add release notes for 0.18.0 (#4102) --- docs/release-notes.txt | 83 ++++++++++++++++++--- docs/release-notes/3938-job-queue-k8s.txt | 9 --- docs/release-notes/4055-native.txt | 7 -- docs/release-notes/4056-data-layer.txt | 7 -- docs/release-notes/4060-custom-reducers.txt | 8 -- docs/release-notes/4066-reset-params.txt | 5 -- docs/release-notes/4069-core-api.txt | 20 ----- docs/release-notes/fix-noverify.txt | 6 -- docs/release-notes/notebook-logs.txt | 5 -- docs/release-notes/remove-pbt.txt | 5 -- 10 files changed, 72 insertions(+), 83 deletions(-) delete mode 100644 docs/release-notes/3938-job-queue-k8s.txt delete mode 100644 docs/release-notes/4055-native.txt delete mode 100644 docs/release-notes/4056-data-layer.txt delete mode 100644 docs/release-notes/4060-custom-reducers.txt delete mode 100644 docs/release-notes/4066-reset-params.txt delete mode 100644 docs/release-notes/4069-core-api.txt delete mode 100644 docs/release-notes/fix-noverify.txt delete mode 100644 docs/release-notes/notebook-logs.txt delete mode 100644 docs/release-notes/remove-pbt.txt diff --git a/docs/release-notes.txt b/docs/release-notes.txt index 0437b6c9458..52f192717f1 100644 --- a/docs/release-notes.txt +++ b/docs/release-notes.txt @@ -4,6 +4,70 @@ Release Notes ############### +************** + Version 0.18 +************** + +Version 0.18.0 +============== + +**Release Date:** May 09, 2022 + +**New Features** + +- Add the Core API. The Core API is the first API offered by Determined that allows users to fully + integrate arbitrary models and training loops into the Determined platform. All of the features + offered by the higher-level Trial APIs, such as reporting metrics, pausing and reactivating, + hyperparameter search, and distributed training, are now available to arbitrary models, + frameworks, and training loops, with only light code changes. See the :ref:`user guide + ` and :ref:`reference documentation ` to get started. + +- **Breaking Change**: Checkpoints: The Python SDK's ``Checkpoint.download()`` method now writes a + differently formatted ``metadata.json`` file into the checkpoint directory. Previously, the JSON + content in the file contained many system-defined fields, plus a ``metadata`` field that + contained the user-defined metadata for the checkpoint, which was also available as a Python + object as ``Checkpoint.metadata``. Now, ``metadata.json`` contains only the user-defined + metadata, and those metadata appear as top-level keys. Some of the fields which were previously + system-defined are now considered user-defined, even though they are uploaded automatically in + Trial-based training. This decision is in line with the Trial APIs now being optional---that is, + part of userspace---after the release of the Core API. + +- Job queue: Add support for dynamic job modification on Kubernetes using the job queue. Users can + now use the WebUI or CLI to change the priority and queue position of jobs in k8s. To update jobs + through the WebUI, go to the Job Queue section, find the target job, and click on the Manage Job + option. To update jobs in the CLI, use the ``det job update`` command. Run ``det job update + --help`` for more information. + +**Bug Fixes** + +- CLI: API requests executed through the Python bindings have been erroneously using the SSL + "noverify" option since version 0.17.6, making them potentially insecure. The option is now + disabled. + +**Deprecated Features** + +- The Determined Data Layer has been deprecated and will be removed in a future version. New code + should not begin using it, but we will assist existing users to migrate to using `YogaDL + `__ directly before removing the feature. + +**Removed Features** + +- Python API: The old experimental namespace methods for custom reducers in both PyTorchTrial and + EstimatorTrial have been removed. The experimental names were deprecated in 0.15.2 (April 2021) + when custom reducers were promoted to general availability. Any users who have not already + migrated to the non-experimental namespace for custom reducer methods must do so. + +- Searcher: Remove the PBT searcher, which was deprecated in version 0.17.6 (January 2022). + +- API: Remove the notebook logs endpoint in favor of the new task logs endpoint. + +- Python API: Remove the remaining parts of the Native API, which was deprecated in version 0.13.5 + (September 2020). The only Native API functions that still remained were + ``det.experimental.create()`` and ``det.expeimental.create_trial_instance()``. + +- Python API: Remove the ``det.pytorch.reset_parameters()`` function, which was deprecated in + 0.12.13 (August 2020). + ************** Version 0.17 ************** @@ -31,8 +95,6 @@ Version 0.17.15 - Add a translation of DeepSpeed's DCGAN example using the new DeepSpeedTrial API. ----- - Version 0.17.14 =============== @@ -53,12 +115,11 @@ Version 0.17.13 - Support DeepSpeed with a new DeepSpeedTrial API. - - `DeepSpeed `_ is a powerful library for training large scale models. - With the new ``DeepSpeedTrial`` you can combine all the benefits of Determined with the - features available in DeepSpeed like the Zero Redundancy Optimizer and pipeline parallel - training. We also provide an example based on Eleuther AI's `GPT-NeoX - `_ repo to help you get started training - state-of-the-art language models. + `DeepSpeed `__ is a powerful library for training large scale models. + With the new ``DeepSpeedTrial`` you can combine all the benefits of Determined with the features + available in DeepSpeed like the Zero Redundancy Optimizer and pipeline parallel training. We also + provide an example based on Eleuther AI's `GPT-NeoX `__ + repo to help you get started training state-of-the-art language models. - CLI: Allow the CLI to accept any unique prefix of a task UUID to refer to the task, rather than requiring the entire UUID. In some places, Determined only displays the first few characters of a @@ -1039,7 +1100,7 @@ Version 0.14.6 - REST API: Add a new endpoint to delete experiments. This endpoint is only enabled for admin users and deletes all resources associated with an experiment. This includes checkpoint storage, - Tensorboards, trial logs from all backends and metadata such as history and metrics, stored in + TensorBoards, trial logs from all backends and metadata such as history and metrics, stored in PostgreSQL. - REST API: Add a new endpoint to fetch aggregated historical resource allocation information. @@ -1825,13 +1886,13 @@ Version 0.13.1 recommended to upgrade to version 0.13.1. Users already on version 0.13.0 should upgrade to version 0.13.1 as usual. -- Tensorboard: Fix a bug that prevents Tensorboards from experiments with old experiment +- TensorBoard: Fix a bug that prevents TensorBoards from experiments with old experiment configuration versions from being loaded. - WebUI: Fix an API response decoding issue on React where a null checkpoint resource was unhandled and could prevent trial detail page from rendering. -- WebUI: Fix an issue where terminated Tensorboard and notebook tasks were rendered as openable. +- WebUI: Fix an issue where terminated TensorBoard and notebook tasks were rendered as openable. Version 0.13.0 ============== diff --git a/docs/release-notes/3938-job-queue-k8s.txt b/docs/release-notes/3938-job-queue-k8s.txt deleted file mode 100644 index 9bcf9f55e24..00000000000 --- a/docs/release-notes/3938-job-queue-k8s.txt +++ /dev/null @@ -1,9 +0,0 @@ -:orphan: - -**New Features** - -- Job queue: Add support for dynamic job modification on Kubernetes using the job queue. Users can - now use the WebUI or CLI to change the priority and queue position of jobs in k8s. To update jobs - through the WebUI, go to the Job Queue section, find the target job, and click on the Manage Job - option. To update jobs in the CLI, use the det job update command. Run det job update --help for - more information. diff --git a/docs/release-notes/4055-native.txt b/docs/release-notes/4055-native.txt deleted file mode 100644 index d84b24ed9d8..00000000000 --- a/docs/release-notes/4055-native.txt +++ /dev/null @@ -1,7 +0,0 @@ -:orphan: - -**Removed Features** - -- Remove the remaining parts of the Native API (deprecated in 0.13.5, September 2020). The only - Native API functions which still remained were ``det.experimental.create()`` and - ``det.expeimental.create_trial_instance()``. diff --git a/docs/release-notes/4056-data-layer.txt b/docs/release-notes/4056-data-layer.txt deleted file mode 100644 index f78cac9cf74..00000000000 --- a/docs/release-notes/4056-data-layer.txt +++ /dev/null @@ -1,7 +0,0 @@ -:orphan: - -**Deprecated Features** - -- The Determined Data Layer has been deprecated and will be removed in a future version. New code - should not begin using it, but we will assist existing users to migrate to using `YogaDL - `_ directly before removing the feature. diff --git a/docs/release-notes/4060-custom-reducers.txt b/docs/release-notes/4060-custom-reducers.txt deleted file mode 100644 index 0b70dc4a5fe..00000000000 --- a/docs/release-notes/4060-custom-reducers.txt +++ /dev/null @@ -1,8 +0,0 @@ -:orphan: - -**Removed Features** - -- The old experimental namespace methods for custom reducers in both PyTorchTrial and - EstimatorTrial have been removed. The experimental names were deprecated in 0.15.2 (April 2021) - when custom reducers were promoted to general availability. Any users who have not already - migrated to the non-experimental namespace for custom reducer methods must do so. diff --git a/docs/release-notes/4066-reset-params.txt b/docs/release-notes/4066-reset-params.txt deleted file mode 100644 index d16cd296d15..00000000000 --- a/docs/release-notes/4066-reset-params.txt +++ /dev/null @@ -1,5 +0,0 @@ -:orphan: - -**Removed Features** - -- Remove the ``det.pytorch.reset_parameters()`` call, which was deprecated in 0.12.13 (Aug 2020). diff --git a/docs/release-notes/4069-core-api.txt b/docs/release-notes/4069-core-api.txt deleted file mode 100644 index c8c7374791c..00000000000 --- a/docs/release-notes/4069-core-api.txt +++ /dev/null @@ -1,20 +0,0 @@ -:orphan: - -**New Features** - -- Core API is the first API offered by Determined that allows users to fully integrate arbitrary - models and training loops into the Determined platform. All of the features offered by the - higher-level Trial APIs, such as reporting metrics, pausing and reactivating, hyperparameter - search, and distributed training, are now available to arbitrary models, frameworks, and training - loops, with only light code changes. See the :ref:`user guide ` and - :ref:`reference documentation ` to get started. - -- **Breaking Change** The Python SDK's ``Checkpoint.download()`` method now writes a differently - formatted ``metadata.json`` file into the checkpoint directory. Previously, the JSON content in - the file contained many system-defined fields, plus a ``metadata`` field that contained the - user-defined metadata for the checkpoint, which was also available as a python object as - ``Checkpoint.metadata``. Now, ``metadata.json`` contains only the user-defined metadata, and - those metadata appear as top-level keys. Some of fields which were previously system-defined are - now considered user-defined, even though they are uploaded automatically in Trial-based training. - This decision is in-line with the Trial APIs now being optional, that is part of userspace, after - the release of the Core API diff --git a/docs/release-notes/fix-noverify.txt b/docs/release-notes/fix-noverify.txt deleted file mode 100644 index 1305b0ea8c3..00000000000 --- a/docs/release-notes/fix-noverify.txt +++ /dev/null @@ -1,6 +0,0 @@ -:orphan: - -**Bug Fixes** - -- CLI: API requests executed through python bindings have been erroneously using SSL "noverify" - option since 0.17.6, making them potentially unsecure. This option is now disabled. diff --git a/docs/release-notes/notebook-logs.txt b/docs/release-notes/notebook-logs.txt deleted file mode 100644 index 8fd98c76f56..00000000000 --- a/docs/release-notes/notebook-logs.txt +++ /dev/null @@ -1,5 +0,0 @@ -:orphan: - -**Breaking Changes** - -- Remove the notebook logs endpoint in favor of the new task logs endpoint. diff --git a/docs/release-notes/remove-pbt.txt b/docs/release-notes/remove-pbt.txt deleted file mode 100644 index 7095492e000..00000000000 --- a/docs/release-notes/remove-pbt.txt +++ /dev/null @@ -1,5 +0,0 @@ -:orphan: - -**Breaking Changes** - -- HP Search: Remove the PBT searcher, which was deprecated in 0.17.6 (Jan 2022).