diff --git a/docs/Users_Guide/matthewjetstream.rst b/docs/Users_Guide/matthewjetstream.rst index 082c6ff..bcd9695 100644 --- a/docs/Users_Guide/matthewjetstream.rst +++ b/docs/Users_Guide/matthewjetstream.rst @@ -32,40 +32,37 @@ to avoid encountering issues during the process. Most sections refer to external documentation to provide details about the necessary steps and to offer additional background information. -Prepare to Use Jetstream2 -========================= +Get an ACCESS Account and Add an Allocation to It +================================================= To `get started with Jetstream2 `_, -you will need to: - -* Create an account with the `National Science Foundation (NSF) `_'s `ACCESS program `_. -* Request a computational "allocation" from ACCESS. -* Log in to Jetstream2's web portal. - -The sections below will guide you through this process. - -Create an ACCESS Account ------------------------- - +you will need to create an account with the `National Science Foundation (NSF) `_'s `ACCESS program `_. If you do not already have one, `register for an ACCESS account `_. -Note that you can either choose to associate your existing University/Organizational account or -create an entirely new ACCESS account when registering. - -Get an Allocation ------------------ - -With your ACCESS account set up, you may `request an allocation `_ +When registering your account, you can either choose to associate your existing University/Organizational account or +create an entirely new ACCESS account when registering. +The Jetstream2 team strongly recommends that you create a new ACCESS account, +as your organizational affiliation may change in the future. + +Once you have an account you will need to have a computational "allocation" added to that account. +Allocations provide the credits you will spend when running instances on Jetstream2. +If there are no allocation owners who can add you to their allocation and you are interested in obtaining your own, +you may `request an allocation `_ that will allow you to use an ACCESS-affiliated cyberinfrastructure resource. -Be sure to read all of the information on that page so that you make a suitable request. -An "Explore" project will be sufficient to work with this exercise, -and you will want to work with the resource "Indiana Jetstream2 CPU" (*not* GPU). +Note that allocations may only be requested by faculty, staff and graduate researchers, +so undergraduates must work with a faculty sponsor to requeste an allocation. + +Be sure to read all of the information on the request page so that you make a suitable request. +Note that you will need to describe the project for which the allocation is intended +and provide a CV or resume for the principal investigator. +An "Explore" project (400,000 credits) will be much more than enough to work with this exercise. +You will want to work with the resource "Indiana Jetstream2 CPU" (*not* **GPU**). The typical turnaround time for allocation requests is one business day. -Log in to the Exosphere Web Site --------------------------------- +Log in to Jetstream2's Exosphere Web Site +========================================= -Once you have an ACCESS account and allocation, -you can log in to their `Exosphere web dashboard `_. +Once you have an ACCESS account and an allocation has been added to it, +you can log in to Jetstream's `Exosphere web dashboard `_. The process of identifying your allocation and ACCESS ID to use Jetstream2 is described on `this page `__ of the `Introduction to Jetstream2 `_ Cornell Virtual Workshop, @@ -75,8 +72,8 @@ of the `Jetstream2 documentation `_. While adding an allocation to your account, it is recommended that you choose the "Indiana University" region of Jetstream2 for completing this exercise. -Create a Cloud Instance and Log In -================================== +Create a Cloud Instance on Jetstream2 +===================================== After you have logged in to Jetstream2 and added your allocation to your account, you are ready to create the cloud instance where you will run the simulation and verification. @@ -84,74 +81,64 @@ If you are not familiar with the cloud computing terms "image" and "instance", it is recommended that you `read about them `__ before proceeding. -Create an SSH Key ------------------ +For this tutorial, you will be able to log in to your instance using Exosphere's Web Desktop or Web Shell functionalities. +If you would rather log in using the "ssh" command from a shell on your own computer, +you will need to create an SSH key pair and upload it to Jetstream2 before creating your instance. +Optional information about doing those things is available here: -You must upload a public SSH key to Jetstream2 before creating your instance. -Jetstream2 injects that public key into the instance's default user account, -and you will need to provide the matching private SSH key to log in to the instance. -If you are not familiar with "SSH key pairs", you should -`read about them `__ before continuing. +.. dropdown:: Creating an SSH Key and uploading it to Jetstream2 -* First, `create an SSH Key on your computer `_ using the "ssh-keygen" command. That command allows you to specify the name and location of the private key file it creates, with the default being "id_rsa". The matching public key file is saved to the same location and name with ".pub" appended to the filename. Later instructions will assume that your private key file is named "id_rsa", but you may choose a different name now and use that name in those later instructions. -* Then, `upload the public key to Jetstream2 `_ through the Exosphere web interface. + You may choose to upload a public SSH key to Jetstream2 before creating your instance. + Jetstream2 will inject that public key into an instance's default user account, + and you will need to provide the matching private SSH key to log in to the instance. + If you are not familiar with "SSH key pairs", you should + `read about them `__ before continuing. -Create an Instance ------------------- + * First, `create an SSH Key on your computer `_ using the "ssh-keygen" command. That command allows you to specify the name and location of the private key file it creates, with the default being "id_rsa". The matching public key file is saved to the same location and name with ".pub" appended to the filename. Later instructions will assume that your private key file is named "id_rsa", but you may choose a different name now and use that name in those later instructions. + * Then, `upload the public key to Jetstream2 `_ through the Exosphere web interface. The Cornell Virtual Workshop topic `Creating an Instance `_ provides detailed information about creating a Jetstream2 instance. -While following those steps, be sure to make the following choices for this instance: +While following those steps for this tutorial, be sure to make the following choices for this instance: -* When choosing an image as the instance source, if viewing "By Type", select the "Ubuntu 22.04 (latest)" image. If viewing "By Image", choose the "Featured-Ubuntu22" image. +* When choosing an image as the instance source, if viewing "By Type", select the "Ubuntu 22.04" image. If viewing "By Image", choose the "Featured-Ubuntu22" image. * Choose the "Flavor" m3.quad (4 CPUs) to provide a faster simulation run-time. -* Select a custom disk size of 100 GB - large enough to hold this exercise's data and results. -* Select the SSH public key that you uploaded previously. +* Select a custom disk size of 100 GB, which is large enough to hold this exercise's data and results. +* For "Enable web desktop?", select Yes. +* For "Choose an SSH public key", select None unless you want to use your own SSH key that you uploaded previously. * You do not need to set any of the Advanced Options. After clicking the "Create" button, wait for the instance to enter the "Ready" state (it takes several minutes). Note that the instance will not only be created, but will be running so that you can log in right away. Log in to the Instance ----------------------- - -The Exosphere web dashboard provides the easy-to-use Web Shell for accessing your Jetstream2 instances, -but after encountering some issues with this exercise when using Web Shell, -we are recommending that you use the SSH command to access your instance from a shell on your computer. -The instructions for `connecting to Jetstream2 using SSH `_ -can executed in the Command Prompt on Windows (from the Start menu, type "cmd" and select Command Prompt) -or from the Terminal application on a Mac. - -In either case you will need to know the location and name of the private SSH key created on your computer (see above), -the IP address of your instance (found in the Exosphere web dashboard) -and the default username on your instance, which is "exouser". - -Once you are logged in to the instance you can proceed to the -"Install Software and Download Data" section below. -You will know that your login has been successful when the prompt has the form ``exouser@instance-name:~$``, -which indicates your username, the instance name, and your current working directory, followed by "$". +====================== -Managing a Jetstream2 Instance ------------------------------- +The Exosphere web dashboard provides two easy-to-use methods for logging in to your instance through a web browser. +The "Web Shell" button will open a terminal to your instance, +and the "Wed Desktop" button will open a view of the instance's graphical desktop (if enabled). +Both views open in a new browser tab, and Exosphere automatically logs you in to the instance. +For this tutorial you should open a Web Desktop so that you will be able to view +the plots that are generated from the simulation output. -In order to use cloud computing resources efficiently, you must know how to -`manage your instances `_. -Instances incur costs whenever they are running (on Jetstream2, this is when they are "Ready"). -"Shelving" an instance stops it from using the cloud's CPUs and memory, -and therefore stops it from incurring any charges against your allocation. +If you wish to log in to the instance from a shell on your computer, +you can do so following the information in this optional content: -When you are through working on this exercise, -be sure to use the instance's "Actions" menu in the web dashboard to -"Shelve" the instance so that it is no longer spending your credits. -If you later return to the dashboard and want to use the instance again, -Use the Action menu's "Unshelve" option to start the instance up again. -Note that any programs that were running when you shelve the instance will be lost, -but the contents of the disk are preserved when shelving. +.. dropdown:: Logging in to a Jetstream2 Instance using SSH From a Shell -You may also want to try the "Resize" action to change the number of CPUs of the instance. -Increasing the number of CPUs (say, to flavor "m3.8") can make your computations finish more quickly. -But of course, doubling the number of CPUs doubles the cost per hour to run the instance, -so Shelving as soon as you are done becomes even more important! + You can use the SSH command to access your instance from a shell on your computer. + The instructions for `connecting to Jetstream2 using SSH `_ + can be executed in the Command Prompt on Windows (from the Start menu, type "cmd" and select Command Prompt) + or from the Terminal application on a Mac. + + In either case you will need to know the location and name of the private SSH key created on your computer (see SSH section, above), + the IP address of your instance (found in the Exosphere web dashboard) + and the default username on your instance, which is "exouser". + + Once you are logged in to the instance you can proceed to the "Install Software and Download Data" section below. + +Once you are logged in to the instance, your shell prompt will have the form ``exouser@instance-name:~$``, +which indicates your username, the instance name, and your current working directory, followed by "$". Preparing the Environment ========================= @@ -169,7 +156,7 @@ The commands in each section can be copied using the button in the upper right c and then pasted into your shell by right-clicking. If your shell ever becomes unresponsive or disconnected from the instance, -you can recover from that situation by rebooting the instance. +you can recover from that situation by opening a new Web Desktop (if available) or rebooting the instance. In the Exosphere dashboard page for your instance, in the Actions menu, select "Reboot". The process takes several minutes, after which the instance status will return to "Ready". @@ -218,56 +205,27 @@ These commands perform the necessary operations:: cp ${WRF_CONFIG_DIR}/vars_io.txt ${WRF_DIR} cp ${WRF_CONFIG_DIR}/run.sh ${WRF_DIR} -Install Docker and Pull Docker Objects -====================================== - -Install Docker --------------- +Pull Docker Objects +=================== As mentioned above, the WRF and METplus software are provided as Docker images that will run as a `"container" `_ on your cloud instance. -To run a Docker container, you must first install the Docker Engine on your instance. +To run a Docker container, the Docker Engine must be installed on your instance. You can then "pull" (download) the WRF and METplus images that will be run as containers. - -The `instructions for installing Docker Engine on Ubuntu `_ -are very thorough and make a good reference, but we only need to perform a subset of those steps. -These commands run a script that sets up the Docker software repository on your instance, -then installs Docker:: - - curl --location https://bit.ly/3R3lqMU > install-docker.sh - source install-docker.sh - rm install-docker.sh - -If a text dialog is displayed asking which services should be restarted, type ``Enter``. -When the installation is complete, you can verify that the Docker command line tool works by asking for its version:: - - docker --version - -The Docker daemon should start automatically, but it sometimes runs into issues. -First, check to see if the daemon started successfully:: - - sudo systemctl --no-pager status docker - -If you see a message saying the daemon failed to start because a "Start request repeated too quickly", -wait a few minutes and issue this command to try again to start it:: - - sudo systemctl start docker - -If the command seems to succeed, confirm that the daemon is running using the status command above. -Repeat these efforts as necessary until it is started. +The Ubuntu instance you created already has the Docker Engine installed and running. Get the WRF and METplus Docker Images and the Observed Weather Data ------------------------------------------------------------------- -Once Docker is running, you must pull the correct versions of the WRF and METplus images onto your instance:: +You must pull the correct versions of the WRF and METplus images onto your instance:: docker pull ${WRF_IMAGE} docker pull ${METPLUS_IMAGE} -METplus is run to perform verification of the results of the WRF simulation using -observations gathered during Hurricane Matthew. -We download that data by pulling a Docker volume that holds it, +METplus is run to perform verification of the results of the WRF simulation +against observations gathered during Hurricane Matthew. +We download that observation data by pulling a Docker volume that holds it, and then referencing that volume when we run the METplus Docker container. The commands to pull and create the volume are:: @@ -350,8 +308,8 @@ Run METplus =========== After the WRF simulation has finished, you can run the METplus verification to compare the simulated results -to the actual weather observations during the hurricane. -The verification takes about five minutes to complete. +to the actual weather observations during the hurricane and generate plots of the simulation. +This process takes about nine minutes to complete. We use command line options to tell the METplus container several things, including where the observed data is located, where the METplus configuration can be found, @@ -379,4 +337,48 @@ The results of the METplus verification can be found in ``${WORKING_DIR}/metplus These files contain tabular output that can be viewed in a text editor. Turn off word wrapping for better viewing. Refer to the MET User's Guide for more information about the `Point-Stat output `_. -In the near future, this exercise will be extended to include instructions to visualize the results. + +View the Plotted Simulation Results +----------------------------------- + +The METplus container also plots the results of the simulation, outputting them as PNG images. +To view these images:: + +* Find the desktop shortcut "Files" on the left side of the desktop and click it to open a file browser. +* Double-click on the following folders in order: metplus, wrf, 20161006_00, then plots. +* Double-click on the first image in the folder, which opens an image viewing application. +* Click the Maximize button in the upper right to increase the viewer to full size. +* Click the button in the middle of the right side of the image to advance to the next image. +* Image legends are shown at the bottom and timeframes are shown in the upper right. +* Each of the six plot sequences contains 16 or 17 images. + +When you are finished running simulations and viewing their results, +you can close the web browser tab containing your Web Desktop. +Then, return to the Exosphere dashboard to manage your instance so it does not incur further charges. + +Managing Your Jetstream2 Instance +================================= + +In order to use cloud computing resources efficiently, you must know how to +`manage your instances `_. +Instances incur costs whenever they are running (on Jetstream2, this is when they are "Ready"). +"Shelving" an instance stops it from using the cloud's CPUs and memory, +and therefore stops it from incurring any charges against your allocation. + +When you are through working on this exercise, you should shelve your instance. +Note that any programs that are running when you shelve the instance will be terminated, +but the contents of the disk are preserved when shelving. + +To shelve, you need to be in the details page for your instance (with the "Actions" menu in the upper right). +If you are on the Instances page, click and instance's name to be taken to its details page. +From the Actions menu, select Shelve. +You will be prompted in that location to confirm the shelve action - click Yes to complete the action. +In the Instances page your instance will briefly be listed as "Shelving", +and then as "Shelved" when the operation is complete. + +When you later return to the dashboard and want to use the instance again, +use the Action menu's "Unshelve" option to start the instance up again. +You can also use the "Resize" action to change the flavor (number of CPUs and amount of RAM) of the instance. +Increasing the number of CPUs can make your computations finish more quickly, +but doubling the number of CPUs doubles the cost per hour to run the instance, +so Shelving as soon as you are done becomes even more important! diff --git a/docs/conf.py b/docs/conf.py index e0cbb83..07c7203 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -36,6 +36,7 @@ 'sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinx_copybutton', + 'sphinx_design', ] # settings for ReadTheDocs PDF creation