diff --git a/tutorial/.DS_Store b/tutorial/.DS_Store new file mode 100644 index 0000000..b24d5d1 Binary files /dev/null and b/tutorial/.DS_Store differ diff --git a/tutorial/CAP_Exercises.html b/tutorial/CAP_Exercises.html new file mode 100644 index 0000000..75af10d --- /dev/null +++ b/tutorial/CAP_Exercises.html @@ -0,0 +1,650 @@ + + + + + no title + + + + + + + +
+
+

2023 tutorial banner

+

Practice Exercises - Community Analysis Pipeline (CAP)

+

GCM workflow

+

CAP is a Python toolkit designed to simplify post-processing and plotting MGCM output. CAP consists of five Python executables:

+
    +
  1. MarsPull.py → accessing MGCM output
    2. +
  2. MarsFiles.py → reducing the files
    3. +
  3. MarsVars.py → performing variable operations
    4. +
  4. MarsInterp.py → interpolating the vertical grid
    5. +
  5. MarsPlot.py → plotting MGCM output
    6. +
+

The following exercises are organized into two parts by function. We will go through Part I on Monday Nov. 13 and Part II on Tuesday Nov. 14.

+

Part I: File Manipulations → MarsFiles.py, MarsVars.py, & MarsInterp.py

+

Part II: Plotting with CAP → MarsPlot.py

+
+

We will not be going over MarsPull.py because it is specifically for retrieving MGCM data from the online MGCM data repository. Instructions for MarsPull.py was covered in the 2021 Legacy Version Tutorial.

+
+
+

Table of Contents

+ +
+

Activating CAP

+

Activate the amesCAP virtual environment to use CAP:

+
(cloud)~$ source ~/amesCAP/bin/activate
+(amesCAP)~$
+

Confirm that CAP's executables are accessible by typing:

+
(amesCAP)~$ MarsVars.py -h
+

The --help [-h] argument prints documentation for the executable to the terminal. Now that we know CAP is configured, make a copy of the file amescap_profile in your home directory, and make it a hidden file:

+
(amesCAP)~$ cp ~/amesCAP/mars_templates/amescap_profile ~/.amescap_profile
+

CAP stores useful settings in amescap_profile. Copying it to our home directory ensures it is not overwritten if CAP is updated or reinstalled.

+

Following Along with the Tutorial

+

Part I covers file manipulations. Some exercises build off of previous exercises so it is important to complete them in order. If you make a mistake or get behind in the process, you can go back and catch up during a break or use the provided answer key before continuing on to Part II.

+

Part II demonstrates CAP's plotting routine. There is more flexibility in this part of the exercise.

+

We will perform every exercise together.

+
+

Feel free to put questions in the chat throughout the tutorial. Another MGCM member can help you as we go.

+
+

Return to Top

+

Part I: File Manipulations

+

CAP has dozens of post-processing capabilities. We will go over a few of the most commonly used functions in this tutorial. We will cover:

+ +

The required MGCM output files are already loaded in the cloud environment under tutorial_files/cap_exercises/. Change to that directory and look at the contents:

+
(amesCAP)~$ cd tutorial_files/cap_exercises
+(amesCAP)~$ ls
+03340.atmos_average.nc  03340.backup.zip  part_1_key.sh
+03340.atmos_diurn.nc    03340.fixed.nc    part_2_plots.in
+

The three MGCM output files have a 5-digit sol number appended to the front of the file name. The sol number indicates the day that a file's record begins. These contain output from the sixth year of a simulation. The zipped file is an archive of these three output files in case you need it.

+

The other two files, part_1_key.sh and part_2_plots.sh are discussed later. We can ignore them for now.

+
+

The output files we manipulate in Part I will be used to generating plots in Part II so do not delete any file you create!

+
+

Let's begin the tutorial.

+

1. MarsPlot's --inspect Function

+

The inspect function is part of MarsPlot.py and it prints netCDF file contents to the screen.

+

To use it on the average file, 03340.atmos_average.nc, type the following in the terminal:

+
(amesCAP)~$ MarsPlot.py -i 03340.atmos_average.nc
+
+

This is a good time to remind you that if you are unsure how to use a function, invoke the --help [-h] argument with any executable to see its documentation (e.g., MarsPlot.py -h).

+
+

Return to Part I

+
+

2. Editing Variable Names and Attributes

+

In the previous exercise, --inspect [-i] revealed a variable called opac in 03340.atmos_average.nc. opac is dust opacity per pascal and it is similar to another variable in the file, dustref, which is opacity per (model) level. Let's rename opac to dustref_per_pa to better indicate the relationship between these variables.

+

We can modify variable names, units, longnames, and even scale variables using the -edit function in MarsVars.py. The syntax for editing the variable name is:

+
(amesCAP)~$ MarsVars.py 03340.atmos_average.nc -edit opac -rename dustref_per_pa
+03340.atmos_average_tmp.nc was created
+03340.atmos_average.nc was updated
+

We can use --inspect [-i] again to confirm that opac was renamed dustref_per_pa:

+
(amesCAP)~$ MarsPlot.py -i 03340.atmos_average.nc
+

The --inspect [-i] function can also print a summary of the values of a variable to the screen. For example:

+
(amesCAP)~$ MarsPlot.py -i 03340.atmos_average.nc -stat dustref_per_pa
+_________________________________________________________
+  VAR           |   MIN     |    MEAN     |    MAX      |
+________________|___________|_____________|_____________|
+  dustref_per_pa|          0|  0.000384902|    0.0017573|
+________________|___________|_____________|_____________|
+

Finally, --inspect [-i] can print the values of a variable to the screen. For example:

+
(amesCAP)~$ MarsPlot.py -i 03340.atmos_average.nc -dump lat
+lat= 
+[-89. -87. -85. -83. -81. -79. -77. -75. -73. -71. -69. -67. -65. -63.
+ -61. -59. -57. -55. -53. -51. -49. -47. -45. -43. -41. -39. -37. -35.
+ -33. -31. -29. -27. -25. -23. -21. -19. -17. -15. -13. -11.  -9.  -7.
+  -5.  -3.  -1.   1.   3.   5.   7.   9.  11.  13.  15.  17.  19.  21.
+  1.   25.  27.  29.  31.  33.  35.  37.  39.  41.  43.  45.  47.  49.
+  2.   53.  55.  57.  59.  61.  63.  65.  67.  69.  71.  73.  75.  77.
+  3.   81.  83.  85.  87.  89.]
+

Return to Part I

+
+

3. Splitting Files in Time

+

Next we're going to trim the diurn and average files by Ls_s. We'll create files that only contain data around southern summer solstice, Ls_s=270. This greatly reduces the file size to make our next post-processing steps more efficient.

+

Syntax for trimming files by Ls_s is:

+
(amesCAP)~$ MarsFiles.py 03340.atmos_diurn.nc -split 265 275
+...
+/home/centos/tutorial_files/cap_exercises/03847.atmos_diurn_Ls265_275.nc was created
+
(amesCAP)~$ MarsFiles.py 03340.atmos_average.nc -split 265 275
+...
+/home/centos/tutorial_files/cap_exercises/03847.atmos_average_Ls265_275.nc was created
+

The trimmed files have the appendix _Ls265_275.nc and the simulation day has changed from 03340 to 03847 to reflect that the first day in the file has changed.

+

For future steps, we need a fixed file with the same simulation day number as the files we just created, so make a copy of the fixed file and rename it:

+
(amesCAP)~$ cp 03340.fixed.nc 03847.fixed.nc
+

Return to Part I

+
+

Break

+

Take 15 minutes to stretch, ask questions, or let us know your thoughts on CAP so far!

+
+

4. Deriving Secondary Variables

+

The --add function in MarsVars.py derives and adds secondary variables to MGCM output files provided that the variable(s) required for the derivation are already in the file. We will add the meridional mass streamfunction (msf) to the trimmed average file. To figure out what we need in order to do this, use the --help [-h] function on MarsVars.py:

+
(amesCAP)~$ MarsVars.py -h
+

The help function shows that streamfunction (msf) requires two things: that the meridional wind (vcomp) is in the average file, and that the average file is pressure-interpolated.

+

First, confirm that vcomp is in 03847.atmos_average_Ls265_275.nc using --inspect [-i]:

+
(amesCAP)~$ MarsPlot.py -i 03847.atmos_average_Ls265_275.nc
+...
+vcomp : ('time', 'pfull', 'lat', 'lon')= (3, 56, 90, 180), meridional wind  [m/sec]
+

Second, pressure-interpolate the average file using MarsInterp.py. The call to MarsInterp.py requires:

+ +
+

All interpolation types are listed in the --help [-h] documentation for MarsInterp.py. Additional grids are listed in ~/.amescap_profile, which accepts user-input grids as well.

+
+

We will also specify that only temperature (temp), winds (ucomp and vcomp), and surface pressure (ps) are to be included in this new file using -include. This will reduce the interpolated file size.

+

Finally, add the --grid [-g] flag at the end of prompt to print out the standard pressure grid levels that we are interpolating to:

+
(amesCAP)~$ MarsInterp.py 03847.atmos_average_Ls265_275.nc -t pstd -l pstd_default -include temp ucomp vcomp ps -g
+1100.0 1050.0 1000.0 950.0 900.0 850.0 800.0 750.0 700.0 650.0 600.0 550.0 500.0 450.0 400.0 350.0 300.0 250.0 200.0 150.0 100.0 70.0 50.0 30.0 20.0 10.0 7.0 5.0 3.0 2.0 1.0 0.5 0.3 0.2 0.1 0.05
+

To perform the interpolation, simply omit the --grid [-g] flag:

+
(amesCAP)~$ MarsInterp.py 03847.atmos_average_Ls265_275.nc -t pstd -l pstd_default -include temp ucomp vcomp ps
+...
+/home/centos/tutorial_files/cap_exercises/03847.atmos_average_Ls265_275_pstd.nc was created
+

Now we have a pressure-interpolated average file with vcomp in it. We can derive and add msf to it using MarsVars.py:

+
(amesCAP)~$ MarsVars.py 03847.atmos_average_Ls265_275_pstd.nc -add msf
+Processing: msf...
+msf: Done
+

Return to Part I

+
+

5. Time-Shifting Diurn Files

+

The diurn file is organized by time-of-day assuming universal time starting at the Martian prime meridian. The time-shift --tshift [-t] function interpolates the diurn file to uniform local time. This is especially useful when comparing MGCM output to satellite observations in fixed local time orbit.

+

Time-shifting can only be done on files with a local time dimension (time_of_day_24, i.e. diurn files). By default, MarsFiles.py time shifts all of the data in the file to 24 uniform local times and this generates very large files. To reduce file size and processing time, we will time-shift the data only to the local times we are interested in: 3 AM and 3 PM.

+

Time-shift the temperature (temp) and surface pressure (ps) in the trimmed diurn file to 3 AM / 3 PM local time like so:

+
(amesCAP)~$ MarsFiles.py 03847.atmos_diurn_Ls265_275.nc -t '3. 15.' -include temp ps
+...
+/home/centos/tutorial_files/cap_exercises/03847.atmos_diurn_Ls265_275_T.nc was created
+

A new diurn file called 03847.atmos_diurn_Ls265_275_T.nc is created. Use --inspect [-i] to confirm that only ps and temp (and their dimensions) are in the file and that the time_of_day dimension has a length of 2:

+
(amesCAP)~$ MarsPlot.py -i 03847.atmos_diurn_Ls265_275_T.nc
+...
+====================CONTENT==========================
+time           : ('time',)= (3,), sol number  [days since 0000-00-00 00:00:00]
+time_of_day_02 : ('time_of_day_02',)= (2,), time of day  [[hours since 0000-00-00 00:00:00]]
+pfull          : ('pfull',)= (56,), ref full pressure level  [mb]
+scalar_axis    : ('scalar_axis',)= (1,), none  [none]
+lon            : ('lon',)= (180,), longitude  [degrees_E]
+lat            : ('lat',)= (90,), latitude  [degrees_N]
+areo           : ('time', 'time_of_day_02', 'scalar_axis')= (3, 2, 1), areo  [degrees]
+ps             : ('time', 'time_of_day_02', 'lat', 'lon')= (3, 2, 90, 180), surface pressure  [Pa]
+temp           : ('time', 'time_of_day_02', 'pfull', 'lat', 'lon')= (3, 2, 56, 90, 180), temperature  [K]
+=====================================================
+

Return to Part I

+
+

6. Pressure-Interpolating the Vertical Axis

+

Now we can efficiently interpolate the diurn file to the standard pressure grid. Recall that interpolation is part of MarsInterp.py and requires:

+
    +
  1. Interpolation type (--type [-t]), and
    2. +
  2. Grid (--level [-l])
    3. +
+

As before, we will interpolate to standard pressure (pstd) using the default pressure grid in .amesgcm_profile (pstd_default):

+
(amesCAP)~$ MarsInterp.py 03847.atmos_diurn_Ls265_275_T.nc -t pstd -l pstd_default
+...
+/home/centos/tutorial_files/cap_exercises/03847.atmos_diurn_Ls265_275_T_pstd.nc was created
+
+

Note: Interpolation could be done before or after time-shifting, the order does not matter.

+
+

We now have four different diurn files in our directory:

+
03340.atmos_diurn.nc                  # Original MGCM file
+03847.atmos_diurn_Ls265_275.nc        # + Trimmed to Ls=265-275
+03847.atmos_diurn_Ls265_275_T.nc      # + Time-shifted; `ps` and `temp` only
+03847.atmos_diurn_Ls265_275_T_pstd.nc # + Pressure-interpolated
+

CAP always adds an appendix to the name of any new file it creates. This helps users keep track of what was done and in what order. The last file we created was trimmed, time-shifted, then pressure-interpolated. However, the same file could be generated by performing the three functions in any order.

+

Return to Part I

+

Optional: Use the Answer Key for Part I

+

This concludes Part I of the tutorial! If you messed up one of the exercises somewhere, you can run the part_1_key.sh script in this directory. It will delete the files you've made and performs all 6 Exercises in Part I for you. To do this, follow the steps below.

+
    +
  1. Source the amesCAP virtual environment
    2. +
  2. Change to the tutorial_files/cap_exercises/ directory
    3. +
  3. Run the executable:
    4. +
+
(amesCAP)~$ ./part_1_key.sh
+

The script will do all of Part I for you. This ensures you can follow along with the plotting routines in Part II.

+

Return to Part I

+

CAP Practical Day 2

+

This part of the CAP Practical covers how to generate plots with CAP. We will take a learn-by-doing approach, creating five sets of plots that demonstrate some of CAP's most often used plotting capabilities:

+
    +
  1. Custom Set 1 of 4: Zonal Mean Surface Plots Over Time
    2. +
  2. Custom Set 2 of 4: Global Mean Column-Integrated Dust Optical Depth Over Time
    3. +
  3. Custom Set 3 of 4: 50 Pa Temperatures at 3 AM and 3 PM
    4. +
  4. Custom Set 4 of 4: Zonal Mean Circulation Cross-Sections
    5. +
+

Plotting with CAP is done in 3 steps:

+

Step 1: Creating the Template (Custom.in)

+

Step 2: Editing Custom.in

+

Step 3: Generating the Plots

+

As in Part I, we will go through these steps together.

+

Part II: Plotting with CAP

+

CAP's plotting routine is MarsPlot.py. It works by generating a Custom.in file containing seven different plot templates that users can modify, then reading the Custom.in file to make the plots.

+

The plot templates in Custom.in include:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Plot TypeX, Y DimensionsName in Custom.in
MapLongitude, LatitudePlot 2D lon x lat
Time-varyingTime, LatitudePlot 2D time x lat
Time-varyingTime, levelPlot 2D time x lev
Time-varyingLongitude, TimePlot 2D lon x time
Cross-sectionLongitude, LevelPlot 2D lon x lev
Cross-sectionLatitude, LevelPlot 2D lat x lev
Line plot (1D)Dimension*, VariablePlot 1D
+
+

*Dimension is user-indicated and could be time (time), latitude (lat), longitude lon, or level (pfull, pstd, zstd, zagl).

+
+

Additionally, MarsPlot.py supports:

+ +

and so much more. You will learn to plot with MarsPlot.py by following along with the demonstration. We will generate the Custom.in template file, customize it, and pass it back into MarsPlot.py to create plots.

+

Return to Part II

+
+

Step 1: Creating the Template (Custom.in)

+

Generate the template file, Custom.in:

+
(amesCAP)~$ MarsPlot.py -template
+/home/centos/tutorial_files/cap_exercises/Custom.in was created 
+

A new file called Custom.in is created in your current working directory.

+
+

Step 2: Editing Custom.in

+

Open Custom.in using vim:

+
(amesCAP)~$ vim Custom.in
+

Scroll down until you see the first two templates shown in the image below:

+

custom input template

+

Since all of the templates have a similar structure, we can broadly describe how Custom.in works by going through the templates line-by-line.

+

Line 1

+
# Line 1                ┌ plot type  ┌ whether to create the plot
+<<<<<<<<<<<<<<| Plot 2D lon X lat = True |>>>>>>>>>>>>>
+

Line 1 indicates the plot type and whether to create the plot when passed into MarsPlot.py.

+

Line 2

+
# Line 2         ┌ file ┌ variable
+Title          = None
+

Line 2 is where we set the plot title.

+

Line 3

+
# Line 3         ┌ file ┌ variable
+Main Variable  = fixed.zsurf          # file.variable
+Main Variable  = [fixed.zsurf]/1000   # [] brackets for mathematical operations
+Main Variable  = diurn_T.temp{tod=3}  # {} brackets for dimension selection
+

Line 3 indicates the variable to plot and the file from which to pull the variable.

+

Additional customizations include:

+ +

Line 4

+
# Line 4
+Cmin, Cmax     = None           # automatic, or
+Cmin, Cmax     = -4,5           # contour limits, or
+Cmin, Cmax     = -4,-2,0,1,3,5  # explicit contour levels
+

Line 4 line defines the color-filled contours for Main Variable. Valid inputs are:

+ +

Lines 5 & 6

+
# Lines 5 & 6
+Ls 0-360       = None # for 'time' free dimension
+Level Pa/m     = None # for 'pstd' free dimension
+

Lines 5 & 6 handle the free dimension(s) for Main Variable (the dimensions that are not plot dimensions).

+

For example, temperature has four dimensions: (time, pstd, lat, lon). For a 2D lon X lat map of temperature, lon and lat provide the x and y dimensions of the plot. The free dimensions are then pstd (Level Pa/m) and time (Ls 0-360).

+

Lines 5 & 6 accept four input types:

+
    +
  1. integer selects the closest value
    2. +
  2. min,max averages over a range of the dimension
    3. +
  3. all averages over the entire dimension
    4. +
  4. None (default) depends on the free dimension:
    5. +
+
# ┌ free dimension      ┌ default setting
+Ls 0-360       = None   # most recent timestep
+Level Pa/m     = None   # surface level
+Lon +/-180     = None   # zonal mean over all longitudes
+Latitude       = None   # equatorial values only
+

Lines 7 & 8

+
# Line 7 & 8
+2nd Variable   = None           # no solid contours
+2nd Variable   = fixed.zsurf    # draw solid contours
+Contours Var 2  = -4,5          # contour range, or
+Contours Var 2  = -4,-2,0,1,3,5 # explicit contour levels
+

Lines 7 & 8 (optional) define the solid contours on the plot. Contours can be drawn for Main Variable or a different 2nd Variable.

+ +

Line 9

+
# Line 9        ┌ X axes limit      ┌ Y axes limit      ┌ colormap   ┌ cmap scale  ┌ projection
+ Axis Options : lon = [None,None] | lat = [None,None] | cmap = jet | scale = lin | proj = cart
+

Finally, Line 9 offers plot customization (e.g., axes limits, colormaps, map projections, linestyles, 1D axes labels).

+

Return to Part II

+
+

Step 3: Generating the Plots

+

Generate the plots set to True in Custom.in by saving and quitting the editor (:wq) and then passing the template file to MarsPlot.py. The first time we do this, we'll pass the --date [-d] flag to specify that we want to plot from the 03340 average and fixed files:

+
(amesCAP)~$ MarsPlot.py Custom.in -d 03340
+

Plots are created and saved in a file called Diagnostics.pdf.

+

default plots

+

Viewing Diagnostics.pdf

+

To open the file, we need to copy it to our local computer. We'll create an alias for the command that does this so we can easily pull Diagnostics.pdf from the cloud environment.

+

First, open a new terminal tab (CRTL-t).

+

Then, change to the directory hosting your token for this tutorial (the token is the file ending in .pem).

+

Finally, build the secure copy (scp) command OR use sftp.

+

Using scp(recommended)

+

To build your scp command:

+ +

Putting these together, the secure copy command is:

+
(local)~$ scp -i "mars-clusterXX.pem" centos@YOUR_IP_ADDRESS:tutorial_files/cap_exercises/Diagnostics.pdf .
+

To make the command into an alias named getpdf:

+
(local)~$ alias getpdf='scp -i "mars-clusterXX.pem" centos@YOUR_IP_ADDRESS:tutorial_files/cap_exercises/Diagnostics.pdf .'
+

Now we can pull the PDF to our local computer with one simple command:

+
(local)~$ getpdf # uses scp
+

Using sftp

+

Alternatively, if scp isn't working for you, you can use sftp. To do this, go to your new terminal tab and type:

+
(local)~$ sftp -i "mars-clusterXX.pem" centos@YOUR_IP_ADDRESS
+sftp> cd tutorial_files/cap_exercises
+

Then when you want to pull the PDF to your local computer, type:

+
sftp> get Diagnostics.pdf
+
+

Summary

+

Plotting with MarsPlot.py is done in 3 steps:

+
(amesCAP)~$ MarsPlot.py -template # generate Custom.in
+(amesCAP)~$ vim Custom.in         # edit Custom.in
+(amesCAP)~$ MarsPlot.py Custom.in # pass Custom.in back to MarsPlot
+

Now we will go through some examples.

+
+

Customizing the Plots

+

Open Custom.in in the editor:

+
(amesCAP)~$ vim Custom.in
+

Copy the first two templates that are set to True and paste them below the line Empty Templates (set to False). Then, set them to False. This way, we have all available templates saved at the bottom of the script.

+

We'll preserve the first two plots, but let's define the sol number of the average and fixed files in the template itself so we don't have to pass the --date [-d] argument every time:

+
# for the first plot (lon X lat topography):
+Main Variable  = 03340.fixed.zsurf
+# for the second plot (lat X lev zonal wind):
+Main Variable  = 03340.atmos_average.ucomp
+

Now we can omit the date (--date [-d]) when we pass Custom.in to MarsPlot.py.

+

Custom Set 1 of 4: Zonal Mean Surface Plots Over Time

+

The first set of plots we'll make are zonal mean surface fields over time: surface temperature, CO2_2 ice, and wind stress.

+

zonal mean surface plots

+

For each of the plots, source variables from the non-interpolated average file, 03340.atmos_average.nc.

+

For the surface temperature plot:

+ +

Let's pause here and pass the Custom.in file to MarsPlot.py.

+

Type ESC-:wq to save and close the file. Then, pass it to MarsPlot.py:

+
(amesCAP)~$ MarsPlot.py Custom.in
+

Now, go to your local terminal tab and retrieve the PDF:

+
(local)~$ getpdf # uses scp
+

or

+
sftp> get Diagnostics.pdf # uses sftp
+

Now we can open it and view our plot.

+

Go back to the cloud environment tab to finish generating the other plots on this page. Open Custom.in in vim:

+
(amesCAP)~$ vim Custom.in
+

Write a set of HOLD ON and HOLD OFF arguments around the surface temperature plot. We will paste the other templates within these arguments to tell MarsPlot.py to put these plots on the same page.

+

Copy/paste the Plot 2D time X lat template plot twice more. Make sure to set the boolean to True.

+

For the surface CO2_2 ice plot:

+ +

For the surface wind stress plot:

+ +

Save and quit the editor (ESC-:wq) and pass Custom.in to MarsPlot.py:

+
(amesCAP)~$ MarsPlot.py Custom.in
+

In your local terminal tab, retrieve the PDF to view the plots:

+
(local)~$ getpdf # uses scp
+

or

+
sftp> get Diagnostics.pdf # uses sftp
+

Return to Part II

+
+

Custom Set 2 of 4: Global Mean Column-Integrated Dust Optical Depth Over Time

+

Now we'll generate a 1D plot and practice plotting multiple lines on it.

+

global mean dust plot

+

Let's start by setting up our 1D plot template:

+ +

Create the visible dust optical depth plot first:

+ +

The input to Main Variable is not so straightforward this time. We want to plot the normalized dust optical depth, which is dervied as follows:

+
normalized_dust_OD = opacity / surface_pressure * reference_pressure
+

The MGCM outputs column-integrated visible dust opacity to the variable taudust_VIS, surface pressure is saved as ps, and we'll use a reference pressure of 610 Pa. Recall that element-wise operations are performed when square brackets [] are placed around the variable in Main Variable. Putting all that together, Main Variable is:

+
# ┌ norm. OD     ┌ opacity                         ┌ surface pressure       ┌ ref. P
+Main Variable  = [03340.atmos_average.taudust_VIS]/[03340.atmos_average.ps]*610
+

To finish up this plot, tell MarsPlot.py what to do to the dimensions of taudust_VIS (time, lon, lat):

+ +

The infrared dust optical depth plot is identical to the visible dust OD plot except for the variable being plotted, so duplicate the visible plot we just created. Make sure both templates are between HOLD ON and HOLD OFF Then, change two things:

+ +

Save and quit the editor (ESC-:wq). pass Custom.in to MarsPlot.py:

+
(amesCAP)~$ MarsPlot.py Custom.in
+

In your local terminal tab, retrieve the PDF to view the plots:

+
(local)~$ getpdf # uses scp
+

or

+
sftp> get Diagnostics.pdf # uses sftp
+

Notice we have two separate 1D plots on the same page. This is because of the HOLD ON and HOLD OFF arguments. Without those, these two plots would be on separate pages. But how do we overplot the lines on top of one another?

+

Go back to the cloud environment, open Custom.in, and type ADD LINE between the two 1D templates.

+

Save and quit again, pass it through MarsPlot.py, and retrieve the PDF locally. Now we have the overplotted lines we were looking for.

+

Return to Part II

+
+

Break

+

Take 15 minutes to stretch, ask questions, or let us know your thoughts on CAP so far!

+
+

Custom Set 3 of 4: 50 Pa Temperatures at 3 AM and 3 PM

+

The first two plots are 3 AM and 3 PM 50 Pa temperatures at Ls_s=270. Below is the 3 PM - 3 AM difference.

+

3 am 3 pm temperatures

+

We'll generate all three plots before passing Custom.in to MarsPlot.py, so copy/paste the Plot 2D lon X lat template three times between a set of HOLD ON and HOLD OFF arguments and set them to True.

+

For the first plot,

+ +
Main Variable  = 03847.atmos_diurn_Ls265_275_T_pstd.temp{tod=3}
+
+

Now, edit the second template for 3 PM temperatures the same way. The only differences are the:

+ +

For the difference plot, we will need to use square brackets in the input for Main Variable in order to subtract 3 AM temperatures from 3 PM temperatures. We'll also use a diverging colorbar to show temperature differences better.

+ +
Main Variable = [03847.atmos_diurn_Ls265_275_T_pstd.temp{tod=15}]-[03847.atmos_diurn_Ls265_275_T_pstd.temp{tod=3}]
+
+

Save and quit the editor (ESC-:wq). pass Custom.in to MarsPlot.py, and pull it to your local computer:

+
(amesCAP)~$ MarsPlot.py Custom.in
+# switch to the local terminal...
+(local)~$ getpdf # uses scp
+

or

+
sftp> get Diagnostics.pdf # uses sftp
+

Return to Part II

+
+

Custom Set 4 of 4: Zonal Mean Circulation Cross-Sections

+

For our final set of plots, we will generate four cross-section plots showing temperature, zonal (U) and meridional (V) winds, and mass streamfunction at Ls_s=270.

+

zonal mean circulation plots

+

Begin with the usual 3-step process:

+
    +
  1. Write a set of HOLD ON and HOLD OFF arguments
    2. +
  2. Copy-paste the Plot 2D lat X lev template between them
    3. +
  3. Set the template to True
    4. +
+

Since all four plots are going to have the same X and Y axis ranges and time selection, let's edit this template before copying it three more times:

+ +

Now copy/paste this template three more times. Let the first plot be temperature, the second be mass streamfunction, the third be zonal wind, and the fourth be meridional wind.

+

For temperature:

+
Title          = Temperature [K] (Ls=270)
+Main Variable  = 03847.atmos_average_Ls265_275_pstd.temp
+Cmin, Cmax     = 110,240
+...
+2nd Variable   = 03847.atmos_average_Ls265_275_pstd.temp
+

For streamfunction, define explicit solid contours under Contours Var 2 and set a diverging colormap.

+
Title          = Mass Stream Function [1.e8 kg s-1] (Ls=270)
+Main Variable  = 03847.atmos_average_Ls265_275_pstd.msf
+Cmin, Cmax     = -110,110
+...
+2nd Variable   = 03847.atmos_average_Ls265_275_pstd.msf
+Contours Var 2 = -5,-3,-1,-0.5,1,3,5,10,20,40,60,100,120
+# set cmap = bwr in Axis Options
+

For zonal and meridional wind, use the dual-toned colormap PiYG.

+
Title          = Zonal Wind [m/s] (Ls=270)
+Main Variable  = 03847.atmos_average_Ls265_275_pstd.ucomp
+Cmin, Cmax     = -230,230
+...
+2nd Variable   = 03847.atmos_average_Ls265_275_pstd.ucomp
+# set cmap = PiYG in Axis Options
+
Title          = Meridional Wind [m/s] (Ls=270)
+Main Variable  = 03847.atmos_average_Ls265_275_pstd.vcomp
+Cmin, Cmax     = -85,85
+...
+2nd Variable   = 03847.atmos_average_Ls265_275_pstd.vcomp
+# set cmap = PiYG in Axis Options
+

Save and quit the editor (ESC-:wq). pass Custom.in to MarsPlot.py, and pull it to your local computer:

+
(amesCAP)~$ MarsPlot.py Custom.in
+# switch to the local terminal...
+(local)~$ getpdf # uses scp
+

or

+
sftp> get Diagnostics.pdf # uses sftp
+

Return to Part II

+
+

End Credits

+

This concludes the practical exercise portion of the CAP tutorial. Please feel free to use these exercises as a reference when using CAP the future!

+

Written by Courtney Batterson, Alex Kling, and Victoria Hartwick. This document was created for the NASA Ames MGCM and CAP Tutorial held virtually November 13-15, 2023.

+

Questions, comments, or general feedback? Contact us.

+

Return to Top

+
+
+ + diff --git a/tutorial/CAP_Exercises.md b/tutorial/CAP_Exercises.md index 5af6ab5..4097e5e 100644 --- a/tutorial/CAP_Exercises.md +++ b/tutorial/CAP_Exercises.md @@ -12,7 +12,7 @@ CAP is a Python toolkit designed to simplify post-processing and plotting MGCM o 4. `MarsInterp.py` → interpolating the vertical grid 5. `MarsPlot.py` → plotting MGCM output -The following exercises are organized into two parts by function. We will go through Part I on Tuesday Nov. 14 and Part II on Wednesday Nov. 15. +The following exercises are organized into two parts by function. We will go through **Part I on Monday Nov. 13** and **Part II on Tuesday Nov. 14**. **[Part I: File Manipulations](#part-i-file-manipulations) → `MarsFiles.py`, `MarsVars.py`, & `MarsInterp.py`** @@ -319,7 +319,7 @@ We now have four different `diurn` files in our directory: ```bash 03340.atmos_diurn.nc # Original MGCM file -03847.atmos_diurn_Ls265_275.nc # + Trimmed to L$_s$=240-300 +03847.atmos_diurn_Ls265_275.nc # + Trimmed to Ls=265-275 03847.atmos_diurn_Ls265_275_T.nc # + Time-shifted; `ps` and `temp` only 03847.atmos_diurn_Ls265_275_T_pstd.nc # + Pressure-interpolated ``` @@ -545,32 +545,51 @@ To open the file, we need to copy it to our local computer. We'll create an alia First, open a new terminal tab (`CRTL-t`). -Then, change to the directory hosting your token for this tutorial (the token is the file ending in `.pem`) +Then, change to the directory hosting your token for this tutorial (the token is the file ending in `.pem`). -Finally, build the secure copy (`scp`) command using the following: +Finally, build the secure copy (`scp`) command OR use `sftp`. -- The name of your `.pem` file -- The address you used to login to the cloud (something like `user@IP.amazonaws.com`) -- The path to the PDF in the cloud (`path/to/Diagnostics.pdf`) +#### Using `scp`(recommended) + +To build your `scp` command: +- The name of your `.pem` file (e.g., `mars-clusterXX.pem`) +- The address you used to login to the cloud (something like `centos@YOUR_IP_ADDRESS`) +- The path to the PDF in the cloud: `tutorial_files/cap_exercises/Diagnostics.pdf` Putting these together, the secure copy command is: ```bash -(local)~$ scp -i "mars-test.pem" user@IP.amazonaws.com:path/to/Diagnostics.pdf . +(local)~$ scp -i "mars-clusterXX.pem" centos@YOUR_IP_ADDRESS:tutorial_files/cap_exercises/Diagnostics.pdf . ``` To make the command into an alias named `getpdf`: ```bash -(local)~$ alias getpdf='scp -i "mars-test.pem" user@IP.amazonaws.com:path/to/Diagnostics.pdf .' +(local)~$ alias getpdf='scp -i "mars-clusterXX.pem" centos@YOUR_IP_ADDRESS:tutorial_files/cap_exercises/Diagnostics.pdf .' ``` Now we can pull the PDF to our local computer with one simple command: ```bash -(local)~$ getpdf +(local)~$ getpdf # uses scp +``` + +#### Using `sftp` + +Alternatively, if `scp` isn't working for you, you can use `sftp`. To do this, go to your new terminal tab and type: + +```bash +(local)~$ sftp -i "mars-clusterXX.pem" centos@YOUR_IP_ADDRESS +sftp> cd tutorial_files/cap_exercises +``` + +Then when you want to pull the PDF to your local computer, type: + +```bash +sftp> get Diagnostics.pdf ``` + *** ### Summary @@ -637,7 +656,13 @@ Type `ESC-:wq` to save and close the file. Then, pass it to `MarsPlot.py`: Now, go to your **local terminal** tab and retrieve the PDF: ```bash -(local)~$ getpwd +(local)~$ getpdf # uses scp +``` + +or + +```bash +sftp> get Diagnostics.pdf # uses sftp ``` Now we can open it and view our plot. @@ -676,7 +701,13 @@ Save and quit the editor (`ESC-:wq`) and pass `Custom.in` to `MarsPlot.py`: In your **local terminal** tab, retrieve the PDF to view the plots: ```bash -(local)~$ getpwd +(local)~$ getpdf # uses scp +``` + +or + +```bash +sftp> get Diagnostics.pdf # uses sftp ``` *[Return to Part II](#cap-practical-day-2)* @@ -702,7 +733,9 @@ Create the **visible dust optical depth** plot first: The input to `Main Variable` is not so straightforward this time. We want to plot the *normalized* dust optical depth, which is dervied as follows: -`normalized_dust_OD = opacity / surface_pressure * reference_pressure` +``` +normalized_dust_OD = opacity / surface_pressure * reference_pressure +``` The MGCM outputs column-integrated visible dust opacity to the variable `taudust_VIS`, surface pressure is saved as `ps`, and we'll use a reference pressure of 610 Pa. Recall that element-wise operations are performed when square brackets `[]` are placed around the variable in `Main Variable`. Putting all that together, `Main Variable` is: @@ -732,7 +765,13 @@ Save and quit the editor (`ESC-:wq`). pass `Custom.in` to `MarsPlot.py`: In your **local terminal** tab, retrieve the PDF to view the plots: ```bash -(local)~$ getpwd +(local)~$ getpdf # uses scp +``` + +or + +```bash +sftp> get Diagnostics.pdf # uses sftp ``` Notice we have two separate 1D plots on the same page. This is because of the `HOLD ON` and `HOLD OFF` arguments. Without those, these two plots would be on separate pages. But how do we overplot the lines on top of one another? @@ -797,7 +836,13 @@ Save and quit the editor (`ESC-:wq`). pass `Custom.in` to `MarsPlot.py`, and pul ```bash (amesCAP)~$ MarsPlot.py Custom.in # switch to the local terminal... -(local)~$ getpwd +(local)~$ getpdf # uses scp +``` + +or + +```bash +sftp> get Diagnostics.pdf # uses sftp ``` *[Return to Part II](#cap-practical-day-2)* @@ -856,6 +901,13 @@ Cmin, Cmax = -230,230 2nd Variable = 03847.atmos_average_Ls265_275_pstd.ucomp # set cmap = PiYG in Axis Options ``` +Title = Zonal Wind [m/s] (Ls=270) +Main Variable = 03847.atmos_average_Ls265_275_pstd.ucomp +Cmin, Cmax = -230,230 +... +2nd Variable = 03847.atmos_average_Ls265_275_pstd.ucomp +# set cmap = PiYG in Axis Options +``` ```python Title = Meridional Wind [m/s] (Ls=270) @@ -871,7 +923,13 @@ Save and quit the editor (`ESC-:wq`). pass `Custom.in` to `MarsPlot.py`, and pul ```bash (amesCAP)~$ MarsPlot.py Custom.in # switch to the local terminal... -(local)~$ getpwd +(local)~$ getpdf # uses scp +``` + +or + +```bash +sftp> get Diagnostics.pdf # uses sftp ``` *[Return to Part II](#cap-practical-day-2)* diff --git a/tutorial/CAP_Exercises.pdf b/tutorial/CAP_Exercises.pdf new file mode 100644 index 0000000..8854ef4 Binary files /dev/null and b/tutorial/CAP_Exercises.pdf differ diff --git a/tutorial/atom.pdf b/tutorial/atom.pdf new file mode 100644 index 0000000..99da898 Binary files /dev/null and b/tutorial/atom.pdf differ diff --git a/tutorial/cap_tutorial_exercises_2023.code-workspace b/tutorial/cap_tutorial_exercises_2023.code-workspace new file mode 100644 index 0000000..876a149 --- /dev/null +++ b/tutorial/cap_tutorial_exercises_2023.code-workspace @@ -0,0 +1,8 @@ +{ + "folders": [ + { + "path": "." + } + ], + "settings": {} +} \ No newline at end of file diff --git a/tutorial/git.pdf b/tutorial/git.pdf new file mode 100644 index 0000000..6d58bf6 Binary files /dev/null and b/tutorial/git.pdf differ diff --git a/tutorial/out/CAP_Exercises.html b/tutorial/out/CAP_Exercises.html new file mode 100644 index 0000000..1e35856 --- /dev/null +++ b/tutorial/out/CAP_Exercises.html @@ -0,0 +1,3475 @@ + + + + + Practice Exercises - Community Analysis Pipeline (CAP) + + + + + + + + + + +
+
+

2023 tutorial banner

+

Practice Exercises - Community Analysis Pipeline (CAP)

+

GCM workflow

+

CAP is a Python toolkit designed to simplify post-processing and plotting MGCM output. CAP consists of five Python executables:

+
    +
  1. MarsPull.py for accessing MGCM output
    2. +
  2. MarsFiles.py for reducing the files
    3. +
  3. MarsVars.py for performing variable operations
    4. +
  4. MarsInterp.py for interpolating the vertical grid
    5. +
  5. MarsPlot.py for plotting MGCM output
    6. +
+

The following exercises are organized into two parts by function. We will go through Part I on Tuesday Nov. 14 and Part II on Wednesday Nov. 15.

+

Part I: File Manipulations → MarsFiles.py, MarsVars.py, & MarsInterp.py

+

Part II: Plotting with CAP → MarsPlot.py

+
+

We will not be going over MarsPull.py because it is specifically for retrieving Legacy MGCM data and this tutorial uses output from the new MGCM. MarsPull.py was covered in the 2021 Legacy Version Tutorial.

+
+
+

Table of Contents

+ +
+

Activating CAP

+

Activate the amesCAP virtual environment to use CAP:

+
(cloud)~$ source ~/amesCAP/bin/activate      # bash
+(amesCAP)~$
+
+

Your prompt will update to reflect that you're in the virtual environment. Before continuing, confirm that CAP's executables are accessible by typing:

+
(amesCAP)~$ MarsVars.py -h
+
+

This is the --help [-h] argument, which shows useful documentation for the executable indicated in the prompt. Now that we know CAP is set up correctly, copy the file amescap_profile provided by CAP to your home directory as a hidden file:

+
(amesCAP)~$ cp ~/amesCAP/mars_templates/amescap_profile ~/.amescap_profile
+
+

CAP stores useful settings in amescap_profile. We make a copy of it in our home directory so that it doesn't get overwritten if CAP is updated or reinstalled in the future.

+

Following Along with the Tutorial

+

We will perform every exercise together.

+

Part I covers file manipulations. Some of the exercises build off of previous exercises so it is important to complete them in order. If you make a mistake or get behind in the process, you can go back and catch up during a break or use the provided answer key before continuing on to Part II.

+

Part II demonstrates CAP's plotting routine and there is more flexibility in this part of the exercise.

+
+

Feel free to put questions in the chat throughout the exercises and another CAP developer can help you out as we go.

+
+

Return to Top

+

Part I: File Manipulations

+

CAP has dozens of post-processing capabilities and we will go over a few of the most commonly used functions including:

+ +

The necessary MGCM output files are already loaded in the cloud environment under tutorial_files/. Change to the tutorial_files/ directory and look at the contents:

+
(amesCAP)~$ cd tutorial_files
+(amesCAP)~$ ls
+03340.atmos_average.nc
+03340.atmos_diurn.nc
+03340.fixed.nc
+
+

These files are from the sixth year of a simulation. MGCM output file names are generated with a 5-digit sol number appended to the front of the file name. The sol number indicates the day that a file's record begins.

+
+

The files we manipulate here will be used to generating the plots in Part II so do not delete anything!

+
+

1. MarsPlot's --inspect Function

+

The average file is 03340.atmos_average.nc and the inspect function is in MarsPlot.py. To use it, type the following in the terminal:

+
(amesCAP)~$ MarsPlot.py -i 03340.atmos_average.nc
+
+
+

This is a good time to remind you that if you are unsure how to use a function, invoke the --help [-h] argument with any executable to see its documentation (e.g., MarsPlot.py -h).

+
+

Return to Part I

+
+

2. Editing Variable Names and Attributes

+

The --inspect [-i] function shows a variable called opac in 03340.atmos_average.nc. opac is dust opacity per Pascal, and we have another variable dustref that is opacity per (model) level. For the second exercise, we will rename opac to dustref_per_pa to better indicate the relationship between the variables.

+

The -edit function in MarsVars.py allows us to change variable names, units, and longnames in MGCM output files. The syntax is:

+
(amesCAP)~$ MarsVars.py 03340.atmos_average.nc -edit opac -rename dustref_per_pa
+03340.atmos_average_tmp.nc was created
+03340.atmos_average.nc was updated
+
+

We can use --inspect [-i] to see that 03340.atmos_average.nc reflects our changes:

+
(amesCAP)~$ MarsPlot.py -i 03340.atmos_average.nc
+
+

We can also see a summary of the values of dustref_per_pa using -stat with --inspect [-i]:

+
(amesCAP)~$ MarsPlot.py -i 03340.atmos_average.nc -stat dustref_per_pa
+__________________________________________________________________
+    VAR           |      MIN      |      MEAN     |      MAX      |
+__________________|_______________|_______________|_______________|
+    dustref_per_pa|              0|    0.000307308|     0.00175193|
+__________________|_______________|_______________|_______________|
+
+

and we can print the values of any variable to the terminal by adding an ncdump-like argument to --inspect [-i]. We demonstrate this with latitude (lat) because it is a relatively short 1D array:

+
(amesCAP)~$ MarsPlot.py -i 03340.atmos_average.nc -dump lat
+lat= 
+[-89. -87. -85. -83. -81. -79. -77. -75. -73. -71. -69. -67. -65. -63.
+ -61. -59. -57. -55. -53. -51. -49. -47. -45. -43. -41. -39. -37. -35.
+ -33. -31. -29. -27. -25. -23. -21. -19. -17. -15. -13. -11.  -9.  -7.
+  -5.  -3.  -1.   1.   3.   5.   7.   9.  11.  13.  15.  17.  19.  21.
+  1.   25.  27.  29.  31.  33.  35.  37.  39.  41.  43.  45.  47.  49.
+  2.   53.  55.  57.  59.  61.  63.  65.  67.  69.  71.  73.  75.  77.
+  3.   81.  83.  85.  87.  89.]
+
+

Return to Part I

+
+

3. Splitting Files in Time

+

Next we're going to trim the diurn and average files by L$_s$ to generate a new file that only contains data around southern summer solstice, L$_s$=270. This greatly reduces the size of the file so we can pressure-interpolate it faster later on.

+

Trim the files like so:

+
(amesCAP)~$ MarsFiles.py 03340.atmos_diurn.nc -split 265 275
+...
+/home/centos/tutorial_files/03847.atmos_diurn_Ls265_275.nc was created
+(amesCAP)~$ MarsFiles.py 03340.atmos_average.nc -split 265 275
+...
+/home/centos/tutorial_files/03847.atmos_average_Ls265_275.nc was created
+
+

The trimmed files have the appendix _Ls265_275.nc and the simulation day has changed from 03340 to 03847 to reflect that the first day in the file has changed.

+

For future steps, we need a fixed file with the same simulation day number as the files we just created, so make a copy of the fixed file and rename it:

+
(amesCAP)~$ cp 03340.fixed.nc 03847.fixed.nc
+
+

Return to Part I

+
+

4. Deriving Secondary Variables

+

The --add function in MarsVars.py derives and adds secondary variables to MGCM output files provided that the variable(s) required to derive the secondary variable exist(s) in the file. What variables do we need to derive the meridional mass streamfunction (msf)? We can check with the help function:

+
(amesCAP)~$ MarsVars.py -h
+
+

The help function shows that streamfunction (msf) requires meridional wind (vcomp) for derivation and that we can only derive streamfunction on a pressure-interpolated file. We can confirm that vcomp is in the 03847.atmos_average_Ls265_275.nc using the --inspect [-i] call to MarsPlot.py:

+
(amesCAP)~$ MarsPlot.py -i 03847.atmos_average_Ls265_275.nc
+...
+vcomp : ('time', 'pfull', 'lat', 'lon')= (3, 56, 90, 180), meridional wind  [m/sec]
+...
+
+

Now we can pressure-interpolate the average file using MarsInterp.py. This is done by specifying the interpolation type (--type [-t]), which is standard pressure (pstd), and the grid (--level [-l]) to interpolate to (pstd_default). Grid options are listed in ~/.amescap_profile.

+

We will also specify which variables to include in the interpolation using -include. This will reduce the interpolated file size. We will include temperature (temp), winds (ucomp and vcomp), and surface pressure (ps) in the interpolated file.

+

Before performing the interpolation, we can ask MarsInterp.py to print out the pressure levels of the grid that we are interpolating to by including the -g argument in the call to MarsInterp.py:

+

We add --grid [-g] in the call to MarsInterp.py to print out the pressure levels of the grid that we are interpolating to:

+
(amesCAP)~$ MarsInterp.py 03847.atmos_average_Ls265_275.nc -t pstd -l pstd_default -include temp ucomp vcomp ps -g
+1100.0 1050.0 1000.0 950.0 900.0 850.0 800.0 750.0 700.0 650.0 600.0 550.0 500.0 450.0 400.0 350.0 300.0 250.0 200.0 150.0 100.0 70.0 50.0 30.0 20.0 10.0 7.0 5.0 3.0 2.0 1.0 0.5 0.3 0.2 0.1 0.05
+
+

To perform the actual interpolation, omit the --grid [-g] flag:

+
(amesCAP)~$ MarsInterp.py 03847.atmos_average_Ls265_275.nc -t pstd -l pstd_default -include temp ucomp vcomp ps
+/home/centos/tutorial_files/03847.atmos_average_Ls265_275_pstd.nc was created
+
+

Now that we have a pressure-interpolated file, we can derive and add msf to it using MarsVars.py:

+
(amesCAP)~$ MarsVars.py 03847.atmos_average_Ls265_275_pstd.nc -add msf
+Processing: msf...
+msf: Done
+
+

Return to Part I

+
+

5. Time-Shifting Diurn Files

+

The diurn file is organized by time-of-day assuming universal time starts at the Martian prime meridian. The time-shift --tshift [-t] function converts the diurn file to uniform local time, which is useful for comparing MGCM output to observations from satellites in fixed local time orbit, for example. Time-shifting can only be done on files with a local time dimension (time_of_day_24, i.e. diurn files).

+

By default, MarsFiles.py time shifts all of the data in the file data to 24 uniform local times. This generates very large files. To reduce file size and processing time, there is an option to time-shift data only to user-specified local times. To do so, simply list the desired local times after the call to --tshift [-t].

+

In this example, we will time-shift temperature (temp) and surface pressure (ps) to 3 AM / 3 PM local time:

+
(amesCAP)~$ MarsFiles.py 03847.atmos_diurn_Ls265_275.nc -t '3. 15.' -include temp ps
+...
+/home/centos/tutorial_files/03847.atmos_diurn_Ls265_275_T.nc was created
+
+

This created a diurn file with "_T" appended to the file name: 03847.atmos_diurn_Ls265_275_T.nc. Using --inspect [-i], we can confirm that only ps and temp (and their dimensions) are in the file and that the time_of_day dimension has a length of 2:

+
(amesCAP)~$ MarsPlot.py -i 03847.atmos_diurn_Ls265_275_T.nc
+...
+====================CONTENT==========================
+scalar_axis    : ('scalar_axis',)= (1,), none  [none]
+pfull          : ('pfull',)= (56,), ref full pressure level  [mb]
+lat            : ('lat',)= (90,), latitude  [degrees_N]
+lon            : ('lon',)= (180,), longitude  [degrees_E]
+time           : ('time',)= (7,), sol number  [days since 0000-00-00 00:00:00]
+time_of_day_02 : ('time_of_day_02',)= (2,), time of day  [[hours since 0000-00-00 00:00:00]]
+areo           : ('time', 'time_of_day_02', 'scalar_axis')= (7, 2, 1), areo  [degrees]
+ps             : ('time', 'time_of_day_02', 'lat', 'lon')= (7, 2, 90, 180), surface pressure  [Pa]
+temp           : ('time', 'time_of_day_02', 'pfull', 'lat', 'lon')= (7, 2, 56, 90, 180), temperature  [K]
+=====================================================
+
+

Return to Part I

+
+

6. Pressure-Interpolating the Vertical Axis

+

After trimming the file and reducing the number of variables stored in the file, we can efficiently interpolate 03847.atmos_diurn_Ls265_275_T.nc to a standard pressure grid. Recall that interpolation is part of MarsInterp.py and requires two arguments:

+
    +
  1. Interpolation type (--type [-t]), and
    2. +
  2. Grid (--level [-l])
    3. +
+

As before, we will interpolate to the standard pressure grid (pstd_default), this time retaining all variables in the file:

+
(amesCAP)~$ MarsInterp.py 03847.atmos_diurn_Ls265_275_T.nc -t pstd -l pstd_default
+/home/centos/tutorial_files/03847.atmos_diurn_Ls265_275_T_pstd.nc was created
+
+
+

Note: Interpolation could be done before or after time-shifting, the order does not matter.

+
+

We now have four different diurn files in our directory:

+
03340.atmos_diurn.nc                  # Original MGCM file
+03847.atmos_diurn_Ls265_275.nc        # + Trimmed to L$_s$=240-300
+03847.atmos_diurn_Ls265_275_T.nc      # + Time-shifted; `ps` and `temp` only
+03847.atmos_diurn_Ls265_275_T_pstd.nc # + Pressure-interpolated
+
+

CAP always adds an appendix to the name of any new file it creates. This helps users keep track of what was done and in what order. The last file we created was trimmed, time-shifted, then pressure-interpolated. However, the same file could be generated by performing the three functions in any order.

+

Return to Part I

+

Optional: Use the Answer Key for Part I

+

This concludes Part I of the tutorial! If you messed up one of the exercises somewhere, you can run a script that performs all 6 Exercises in Part I for you. To do this, follow the steps below.

+
    +
  1. Source the amesCAP virtual environment
    2. +
  2. Change to the tutorial_files/ directory
    3. +
  3. Run the executable below
    4. +
+
(amesCAP)~$ ~/cap_backup/part_1_key.sh
+
+

The script will do all of Part I for you. This ensures you can follow along with the plotting routines in Part II.

+

Return to Part I

+

CAP Practical Day 2

+

This part of the CAP Practical covers how to generate plots with CAP. We will take a learn-by-doing approach, creating five sets of plots that demonstrate some of CAP's most often used plotting capabilities:

+
    +
  1. Page 5 of 5: Zonal Mean Circulation Cross-Sections
    2. +
  2. Page 1 of 5: Zonal Mean Surface Plots Over Time
    3. +
  3. Page 2 of 5: Zonal Mean Column-Integrated Dust Optical Depth Over Time
    4. +
  4. Page 3 of 5: Global Mean Column-Integrated Dust Optical Depth Over Time
    5. +
  5. Page 4 of 5: 50 Pa Temperatures at 3 AM and 3 PM
    6. +
+

Plotting with CAP is done in 3 steps:

+

Step 1: Creating the Template (Custom.in)

+

Step 2: Editing Custom.in

+

Step 3: Generating the Plots

+

As in Part I, we will go through these steps together.

+

Part II: Plotting with CAP

+

CAP's plotting routine is MarsPlot.py. It works by generating a Custom.in file containing seven different plot templates that users can modify, then reading the Custom.in file to make the plots.

+

The plot templates in Custom.in include:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Plot TypeX, Y DimensionsName in Custom.in
MapLongitude, LatitudePlot 2D lon x lat
Time-varyingTime, LatitudePlot 2D time x lat
Time-varyingTime, levelPlot 2D time x lev
Time-varyingLongitude, TimePlot 2D lon x time
Cross-sectionLongitude, LevelPlot 2D lon x lev
Cross-sectionLatitude, LevelPlot 2D lat x lev
Line plot (1D)Dimension*, VariablePlot 1D
+
+

*Dimension is user-indicated and could be time (time), latitude (lat), longitude lon, or level (pfull, pstd, zstd, zagl).

+
+

Additionally, MarsPlot.py supports:

+ +

and so much more. You will learn to plot with MarsPlot.py by following along with the demonstration. We will generate the Custom.in template file, customize it, and pass it back into MarsPlot.py to create plots.

+

Return to Part II

+
+

Step 1: Creating the Template (Custom.in)

+

Generate the template file, Custom.in:

+
(amesCAP)~$ MarsPlot.py -template
+/home/centos/tutorial_files/Custom.in was created 
+
+

A new file called Custom.in is created in your current working directory.

+
+

Step 2: Editing Custom.in

+

Open Custom.in using vim:

+
(amesCAP)~$ vim Custom.in
+
+

Scroll down until you see the first two templates shown in the image below:

+

custom input template

+

Since all of the templates have a similar structure, we can broadly describe how Custom.in works by going through the templates line-by-line.

+

Line 1

+
# Line 1                ┌ plot type  ┌ whether to create the plot
+<<<<<<<<<<<<<<| Plot 2D lon X lat = True |>>>>>>>>>>>>>
+
+

Line 1 indicates the plot type and whether to create the plot when passed into MarsPlot.py.

+

Line 2

+
# Line 2         ┌ file ┌ variable
+Main Variable  = fixed.zsurf          # file.variable
+Main Variable  = [fixed.zsurf]/1000   # [] brackets for mathematical operations
+Main Variable  = diurn_T.temp{tod=3}  # {} brackets for dimension selection
+
+

Line 2 indicates the variable to plot and the file from which to pull the variable.

+

Additional customizations include:

+ +

Line 3

+
# Line 3
+Cmin, Cmax     = None           # automatic, or
+Cmin, Cmax     = -4,5           # contour limits, or
+Cmin, Cmax     = -4,-2,0,1,3,5  # explicit contour levels
+
+

Line 3 line defines the color-filled contours for Main Variable. Valid inputs are:

+ +

Lines 4 & 5

+
# Lines 4 & 5
+Ls 0-360       = None # for 'time' free dimension
+Level Pa/m     = None # for 'pstd' free dimension
+
+

Lines 4 & 5 handle the free dimension(s) for Main Variable (the dimensions that are not plot dimensions).

+

For example, temperature has 4 dimensions: (time, pstd, lat, lon). For a 2D lon X lat map of temperature, lon and lat provide the x and y dimensions of the plot. The free dimensions are then pstd (Level Pa/m) and time (Ls 0-360).

+

Lines 4 & 5 accept four input types:

+
    +
  1. integer selects the closest value
    2. +
  2. min,max averages over a range of the dimension
    3. +
  3. all averages over the entire dimension
    4. +
  4. None (default) depends on the free dimension:
    5. +
+
# ┌ free dimension      ┌ default setting
+Ls 0-360       = None   # most recent timestep
+Level Pa/m     = None   # surface level
+Lon +/-180     = None   # zonal mean over all longitudes
+Latitude       = None   # equatorial values only
+
+

Lines 6 & 7

+
# Line 6 & 7
+2nd Variable   = None           # no solid contours
+2nd Variable   = fixed.zsurf    # draw solid contours
+Contours Var 2  = -4,5          # contour range, or
+Contours Var 2  = -4,-2,0,1,3,5 # explicit contour levels
+
+

Lines 6 & 7 (optional) define the solid contours on the plot. Contours can be drawn for Main Variable or a different 2nd Variable.

+ +

Line 8

+
# Line 8        ┌ X axes limit      ┌ Y axes limit      ┌ colormap   ┌ cmap scale  ┌ projection
+ Axis Options : lon = [None,None] | lat = [None,None] | cmap = jet | scale = lin | proj = cart
+
+

Finally, Line 8 offers plot customization (e.g., axes limits, colormaps, map projections, linestyles, 1D axes labels).

+

Return to Part II

+
+

Step 3: Generating the Plots

+

Generate the plots set to True in Custom.in by saving and quitting the editor (:wq) and then passing the template file to MarsPlot.py:

+
(amesCAP)~$ MarsPlot.py Custom.in
+
+

Plots are created and saved in a file called Diagnostics.pdf. Open the file to view the plots!

+
+

Summary

+

Plotting with MarsPlot.py is done in 3 steps:

+
(amesCAP)~$ MarsPlot.py -template # generate Custom.in
+(amesCAP)~$ vim Custom.in         # edit Custom.in
+(amesCAP)~$ MarsPlot.py Custom.in # pass Custom.in back to MarsPlot
+
+

Now we will go through some examples.

+
+

Customizing the Plots

+

Open Custom.in in the editor:

+
(amesCAP)~$ vim Custom.in
+
+

Copy the first two templates that are set to True and paste them below the line Empty Templates (set to False). Then, set them to False. This way, we have all available templates saved at the bottom of the script.

+

Page 1 of 5: Zonal Mean Surface Plots Over Time

+

The first set of plots we'll make are zonal mean surface fields over time: surface temperature, CO$_2$ ice, and wind stress.

+

zonal mean surface plots

+

We will always do three things when we create a new plot:

+
    +
  1. Write a set of HOLD ON and HOLD OFF arguments
    2. +
  2. Copy/paste the Plot 2D time X lat template three times between them
    3. +
  3. Set all three templates to True
    4. +
+

For each of the plots, source variables from the non-interpolated average file, 03340.atmos_average.nc.

+

For the surface temperature plot:

+ +

For the surface CO$_2$ ice plot:

+ +

For the surface wind stress plot:

+ +

Optionally, you can adjust any of the following Axis Options as you like. Leave some of the values to the default None to see what happens.

+ +

Make the plots by saving Custom.in (:wq) and passing it to MarsPlot.py:

+
(amesCAP)~$ MarsPlot.py Custom.in
+
+

Return to Part II

+
+

Page 2 of 5: Zonal Mean Column-Integrated Dust Optical Depth Over Time

+

The next set of plots are very similar to the first: zonal mean visible (taudust_VIS) and infrared (taudust_IR) dust optical depth over time. The difference is that the normalized dust optical depth has to be calculated from the dust opacity, so we'll use square brackets [] for element-wise operations.

+

zonal mean dust plots

+

Start with the 3-step process we did before:

+
    +
  1. Write a set of HOLD ON and HOLD OFF arguments
    2. +
  2. Copy/paste the Plot 2D time X lat template two times between them
    3. +
  3. Set all three templates to True
    4. +
+

For both plots, source dust opacities from the non-interpolated average file, 03340.atmos_average.nc.

+

Use square brackets [] around the input for Main Variable to calculate the normalized dust optical depth:

+

optical depth = [opacity] / [surface pressure] * (reference pressure)

+

where opacity is taudust_VIS, surface pressure is ps, and the reference pressure is 610 Pa. For the visible dust optical depth, then, Main Variable is:

+
Main Variable  = [03340.atmos_average.taudust_VIS]/[03340.atmos_average.ps]*610
+
+ +

Save and quit Custom.in (:wq) and pass it to MarsPlot.py:

+
(amesCAP)~$ MarsPlot.py Custom.in
+
+

Return to Part II

+
+

Page 3 of 5: Global Mean Column-Integrated Dust Optical Depth Over Time

+

Now we'll look at the global dust optical depth over time in a 1D plot.

+

global mean dust plot

+

Begin as usual: copy/paste the Plot 1D template twice between a set of HOLD ON and HOLD OFF arguments and set them to True.

+ +
Ls 0-360       = AXIS
+Latitude       = all
+Lon +/-180     = all
+Level [Pa/m]   = None # taudust_* vars are column-integrated
+Diurnal  [hr]  = None
+
+

If we were to run this through CAP now, we would end up with two separate 1D plots on the same page. To overplot both lines on the same plot instead, include ADD LINE between the templates.

+

Save and quit Custom.in (:wq) and pass it to MarsPlot.py:

+
(amesCAP)~$ MarsPlot.py Custom.in
+
+

Return to Part II

+
+

Page 4 of 5: 50 Pa Temperatures at 3 AM and 3 PM

+

This set of plots shows 50 Pa temperatures during southern summer solstice at 3 AM and 3 PM. As well as the difference between them.

+

3 am 3 pm temperatures

+

As usual, copy/paste the Plot 2D lon X lat template three times between a set of HOLD ON and HOLD OFF arguments and set them to True.

+

For each of the three plots:

+ +

For the AM and PM temperature plots:

+ +
Main Variable  = 03847.atmos_diurn_Ls265_275_T_pstd.temp{tod=3}  # for 3 AM
+Main Variable  = 03847.atmos_diurn_Ls265_275_T_pstd.temp{tod=15} # for 3 PM
+
+

For the difference plot, subtract Main Variable from the 3 PM and 3 AM lines. Since we're performing a mathematical operation on the values of temp, use square brackets [] around each variable like so:

+
Main Variable  = [03847.atmos_diurn_Ls265_275_T_pstd.temp{tod=15}]-[03847.atmos_diurn_Ls265_275_T_pstd.temp{tod=3}]
+
+ +

Save and quit Custom.in (:wq) and pass it to MarsPlot.py:

+
(amesCAP)~$ MarsPlot.py Custom.in
+
+

Return to Part II

+
+

Page 5 of 5: Zonal Mean Circulation Cross-Sections

+

Finally, we will make a set of cross-sectional plots showing temperature, U and V winds, and mass streamfunction during southern summer.

+

zonal mean circulation plots

+

Begin with the usual 3-step process:

+
    +
  1. Write a HOLD ON HOLD OFF set
    2. +
  2. Copy-paste the Plot 2D lat X lev template four times
    3. +
  3. Set every plot to True
    4. +
+

Set Main Variable to the variable to be plotted and Title the plot accordingly:

+
# temperature
+Title          = Temperature [K] (Ls=270)
+Main Variable  = 03847.atmos_average_Ls265_275_pstd.temp
+Cmin, Cmax     = 110,240
+Ls 0-360       = 270
+
+# mass streamfunction
+Title          = Mass Stream Function [1.e8 kg s-1] (Ls=270)
+Main Variable  = 03847.atmos_average_Ls265_275_pstd.msf
+Cmin, Cmax     = -110,110
+Ls 0-360       = 270
+
+# zonal wind
+Title          = Zonal Wind [m/s] (Ls=270)
+Main Variable  = 03847.atmos_average_Ls265_275_pstd.ucomp
+Cmin, Cmax     = -230,230
+Ls 0-360       = 270
+
+# meridional wind
+Title          = Meridional Wind [m/s] (Ls=270)
+Main Variable  = 03847.atmos_average_Ls265_275_pstd.vcomp
+Cmin, Cmax     = -85,85
+Ls 0-360       = 270
+
+

For mass streamfunction, add solid contours to the plot and specify the contour levels as follows:

+
2nd Variable   = 03847.atmos_average_Ls265_275_pstd.msf
+Contours Var 2 = -5,-3,-1,-0.5,1,3,5,10,20,40,60,100,120
+
+

Change the colormap for each plot. I like to use:

+ +

Finally, set the latitude and level limits explicitly under Axis Options:

+
Axis Options  : Lat = [-90,90] | level[Pa/m] = [1000,0.05] | cmap = rainbow |scale = lin
+
+
+

We are plotting from the pressure-interpolated (pstd) file so the vertical grid is pressure in Pascal and level [Pa/m] defines the axes limits in Pascal. If we were using the non-interpolated file, we would specify the axes limits in meters.

+
+

Save and quit Custom.in (:wq) and pass it to MarsPlot.py:

+
(amesCAP)~$ MarsPlot.py Custom.in
+
+

Return to Part II

+

End Credits

+

This concludes the practical exercise portion of the CAP tutorial. Please feel free to use these exercises as a reference when using CAP the future!

+

This document was completed in October 2023. Written by Courtney Batterson, Alex Kling, and Victoria Hartwick. Feedback can be directed to Alex Kling at alexandre.m.kling@nasa.gov

+

Return to Top

+
+
+
+ + + + + \ No newline at end of file diff --git a/tutorial/out/CAP_Install.html b/tutorial/out/CAP_Install.html new file mode 100644 index 0000000..e578859 --- /dev/null +++ b/tutorial/out/CAP_Install.html @@ -0,0 +1,3219 @@ + + + + + Installing the Community Analysis Pipeline (CAP) + + + + + + + + + + +
+
+

+
+

Installing the Community Analysis Pipeline (CAP)

+

Welcome!

+

This document contains the instructions for installing the NASA Ames MCMC's Community Analysis Pipeline (CAP). We ask that you come to the MGCM Tutorial on November 2-4 with CAP installed on your machine so that we can jump right into using it! On the second day of the tutorial (November 3rd), we will be using CAP to analyze MGCM output.

+

Installing CAP is fairly straightforward. We will create a Python virtual environment, download CAP, and then install CAP in the virtual environment. That's it!

+

A quick overview of what is covered in this installation document:

+
    +
  1. Creating the Virtual Environment
    2. +
  2. Installing CAP
    3. +
  3. Testing & Using CAP
    4. +
  4. Practical Tips
    5. +
  5. Do This Before Attending the Tutorial
    6. +
+
+

1. Creating the Virtual Environment

+

We begin by creating a virtual environment in which to install CAP. The virtual environment is an isolated Python environment cloned from an existing Python distribution. The virtual environment consists of the same directory trees as the original environment, but it includes activation and deactivation scripts that are used to move in and out of the virtual environment. Here's an illustration of how the two Python environments might differ:

+
     anaconda3                    virtual_env3/
+     ├── bin                      ├── bin
+     │   ├── pip       (copy)     │    ├── pip
+     │   └── python3    >>>>      │    ├── python3
+     └── lib                      │    ├── activate
+                                  │    ├── activate.csh
+                                  │    └── deactivate
+                                  └── lib             
+
+  ORIGINAL ENVIRONMENT           VIRTUAL ENVIRONMENT
+      (untouched)            (vanishes when deactivated)
+
+

We can install and upgrade packages in the virtual environment without breaking the main Python environment. In fact, it is safe to change or even completely delete the virtual environment without breaking the main distribution. This allows us to experiment freely in the virtual environment, making it the perfect location for installing and testing CAP.

+
+

Step 1: Identify Your Preferred Python Distribution

+

If you are already comfortable with Python's package management system, you are welcome to install the pipeline on top any python3 distribution already present on your computer. Jump to Step #2 and resolve any missing package dependency.

+

For all other users, we highly recommend using the latest version of the Anaconda Python distribution. It ships with pre-compiled math and plotting packages such as numpy and matplotlib as well as pre-compiled libraries like hdf5 headers for reading netCDF files (the preferred filetype for analysing MGCM output).

+

You can install the Anaconda Python distribution via the command-line or using a graphical interface (scroll to the very bottom of the page for all download options). You can install Anaconda at either the System/ level or the User/ level (the later does not require admin-priviledges). The instructions below are for the command-line installation and installs Anaconda in your home directory, which is the recommended location. Open a terminal and type the following:

+
(local)>$ chmod +x Anaconda3-2021.05-MacOSX-x86_64.sh   # make the .sh file executable (actual name may differ)
+(local)>$ ./Anaconda3-2021.05MacOSX-x86_64.sh           # runs the executable
+
+

Which will return:

+
> Welcome to Anaconda3 2021.05
+>
+> In order to continue the installation process, please review the license agreement.
+> Please, press ENTER to continue
+> >>>
+
+

Read (ENTER) and accept (yes) the terms, choose your installation location, and initialize Anaconda3:

+
(local)>$ [ENTER]
+> Do you accept the license terms? [yes|no]
+> >>>
+(local)>$ yes
+> Anaconda3 will now be installed into this location:
+> /Users/username/anaconda3
+>
+>  - Press ENTER to confirm the location
+>  - Press CTRL-C to abort the installation
+>  - Or specify a different location below
+>
+> [/Users/username/anaconda3] >>>
+(local)>$ [ENTER]
+> PREFIX=/Users/username/anaconda3
+> Unpacking payload ...
+> Collecting package metadata (current_repodata.json):
+>   done                                                       
+> Solving environment: done
+>
+> ## Package Plan ##
+> ...
+> Preparing transaction: done
+> Executing transaction: -
+> done
+> installation finished.
+> Do you wish the installer to initialize Anaconda3 by running conda init? [yes|no]
+> [yes] >>>
+(local)>$ yes
+
+
+

For Windows users, we recommend installing the pipeline in a Linux-type environment using Cygwin. This will enable the use of CAP command line tools. Simply download the Windows version of Anaconda on the Anaconda website and follow the instructions from the installation GUI. When asked about the installation location, make sure you install Python under your emulated-Linux home directory (/home/username) and not in the default location (/cygdrive/c/Users/username/anaconda3). From the installation GUI, the path you want to select is something like: C:/Program Files/cygwin64/home/username/anaconda3. Also be sure to check YES when prompted to "Add Anaconda to my PATH environment variable."

+
+

Confirm that your path to the Anaconda Python distribution is fully actualized by closing out of the current terminal, opening a new terminal, and typing:

+
(local)>$ python[TAB]
+
+

If this returns multiple options (e.g. python, python2, python 3.7, python.exe), then you have more than one version of Python sitting on your system (an old python2 executable located in /usr/local/bin/python, for example). You can see what these versions are by typing:

+
(local)>$ python3 --version     # Linux/MacOS
+(local)>$ python.exe --version  # Cygwin/Windows
+
+

Check your version of pip the same way, then find and set your $PATH environment variable to point to the Anaconda Python and Anaconda pip distributions. If you are planning to use Python for other projects, you can update these paths like so:

+
# with bash:
+(local)>$ echo 'export PATH=/Users/username/anaconda3/bin:$PATH' >> ~/.bash_profile
+# with csh/tsch:
+(local)>$ echo 'setenv PATH $PATH\:/Users/username/anaconda3/bin\:$HOME/bin\:.'  >> ~/.cshrc
+
+

Confirm these settings using the which command:

+
(local)>$ which python3         # Linux/MacOS
+(local)>$ which python.exe      # Cygwin/Windows
+
+

which hopefully returns a Python executable that looks like it was installed with Anaconda, such as:

+
> /username/anaconda3/bin/python3     # Linux/MacOS
+> /username/anaconda3/python.exe      # Cygwin/Windows
+
+

If which points to either of those locations, you are good to go and you can proceed from here using the shorthand path to your Anaconda Python distribution:

+
(local)>$ python3     # Linux/MacOS
+(local)>$ python.exe  # Cygwin/Windows
+
+

If, however, which points to some other location, such as /usr/local/bin/python, or more than one location, proceed from here using the full path to the Anaconda Python distribution:

+
(local)>$ /username/anaconda3/bin/python3 # Linux/MacOS
+(local)>$ /username/anaconda3/python.exe  # Cygwin/Windows
+
+
+

Step 2: Set Up the Virtual Environment:

+

Python virtual environments are created from the command line. Create an environment called amesCAP by typing:

+
(local)>$ python3 -m venv --system-site-packages amesCAP    # Linux/MacOS Use FULL PATH to python if needed
+(local)>$ python.exe -m venv –-system-site-packages amesCAP  # Cygwin/Windows Use FULL PATH to python if needed
+
+

First, find out if your terminal is using bash or a variation of C-shell (.csh, .tsch...) by typing:

+
(local)>$ echo $0
+> -bash
+
+

Depending on the answer, you can now activate the virtual environment with one of the options below:

+
(local)>$ source amesCAP/bin/activate          # bash
+(local)>$ source amesCAP/bin/activate.csh      # csh/tcsh
+(local)>$ source amesCAP/Scripts/activate.csh  # Cygwin/Windows
+(local)>$ conda amesCAP/bin/activate           # if you used conda
+
+
+

In Cygwin/Windows, the /bin directory may be named /Scripts.

+
+

You will notice that after sourcing amesCAP, your prompt changed indicate that you are now inside the virtual environment (i.e. (local)>$ changed to (amesCAP)>$).

+

We can verify that which python and which pip unambiguously point to amesCAP/bin/python3 and amesCAP/bin/pip, respectively, by calling which within the virtual environment:

+
(amesCAP)>$ which python3         # in bash, csh
+> amesCAP/bin/python3
+(amesCAP)>$ which pip
+> amesCAP/bin/pip
+
+(amesCAP)>$ which python.exe      # in Cygwin/Windows
+> amesCAP/Scripts/python.exe
+(amesCAP)>$ which pip
+> amesCAP/Scripts/pip            
+
+

There is therefore no need to reference the full paths while inside the virtual environment.

+
+

2. Installing CAP

+

Now we can download and install CAP in amesCAP. CAP was provided to you in the tarfile AmesCAP-master.zip that was sent along with these instructions. Download AmesCAP-master.zip. You can leave the file in Downloads/, or, if you encounter any permission issue, move it to a temporary location like your /home or /Desktop directories.

+

Using pip

+

Open a terminal window, activate the virtual environment, and untar the file or install from the github:

+
(local)>$ source ~/amesCAP/bin/activate          # bash
+(local)>$ source ~/amesCAP/bin/activate.csh      # cshr/tsch
+(local)>$ source ~/amesCAP/Scripts/activate.csh  #  Cygwin/Windows
+(local)>$ conda amesCAP/bin/activate             # if you used conda
+# FROM AN ARCHIVE:
+(amesCAP)>$ tar -xf AmesCAP-master.zip
+(amesCAP)>$ cd AmesCAP-master
+(amesCAP)>$ pip install .
+# OR FROM THE GITHUB:
+(amesCAP)>$ pip install git+https://github.com/NASA-Planetary-Science/AmesCAP.git
+
+
+

Please follow the instructions to upgrade pip if recommended during that steps. Instructions relevant the conda package manager are listed at the end of this section

+
+

That's it! CAP is installed in amesCAP and you can see the MarsXXXX.py executables stored in ~/amesCAP/bin/:

+
(local)>$ ls ~/amesCAP/bin/
+> Activate.ps1     MarsPull.py      activate.csh              nc4tonc3         pip3
+> MarsFiles.py     MarsVars.py      activate.fish             ncinfo           pip3.8
+> MarsInterp.py    MarsViewer.py    easy_install              normalizer       python
+> MarsPlot.py      activate         easy_install-3.8          pip              python3
+
+
+

Shall you need to modify any code, note that when you access the Mars tools above, those are not executed from the AmesCAP-master/ folder in your /Downloads directory, but instead from the amesCAP virtual environment where they were installed by pip. You can safely move AmesCAP-master.zip and the AmesCAP-master directory to a different location on your system.

+
+

Double check that the paths to the executables are correctly set in your terminal by exiting the virtual environment:

+
(amesCAP)>$ deactivate
+
+

then reactivating the virtual environment:

+
(local)>$ source ~/amesCAP/bin/activate          # bash
+(local)>$ source ~/amesCAP/bin/activate.csh      # csh/tsch
+(local)>$ source ~/amesCAP/Scripts/activate.csh  # cygwin
+(local)>$ conda amesCAP/bin/activate             # if you used conda
+
+

and checking the documentation for any CAP executable using the --help option:

+
(amesCAP)>$ MarsPlot.py --help
+(amesCAP)>$ MarsPlot.py -h
+
+

or using full paths:

+
(amesCAP)>$ ~/amesCAP/bin/MarsPlot.py -h     # Linux/MacOS
+(amesCAP)>$ ~/amesCAP/Scripts/MarsPlot.py -h # Cygwin/Windows
+
+

If the pipeline is installed correctly, --help will display documentation and command-line arguments for MarsPlot in the terminal.

+
+

If you have either purposely or accidentally installed the amescap package on top of your main python distribution (e.g. in ~/anaconda3/lib/python3.7/site-packages/ or ~/anaconda3/bin/) BEFORE setting-up the amesCAP virtual environment, the Mars*.py executables may not be present in the ~/amesCAP/bin/ directory of the virtual environment (~/amesCAP/Scripts/ on Cygwin). Because on Step 2 we created the virtual environment using the --system-site-packages flag, python will consider that amescap is already installed when creating the new virtual environment and pull the code from that location, which may change the structure of the ~/amesCAP/bin directory within the virtual environment. If that is the case, the recommended approach is to exit the virtual environment (deactivate), run pip uninstall amescap to remove CAP from the main python distribution, and start over at Step 2.

+
+

This completes the one-time installation of CAP in your virtual environment, amesCAP, which now looks like:

+
amesCAP/
+├── bin
+│   ├── MarsFiles.py
+│   ├── MarsInterp.py
+│   ├── MarsPlot.py
+│   ├── MarsPull.py
+│   ├── MarsVars.py
+│   ├── activate
+│   ├── activate.csh
+│   ├── deactivate
+│   ├── pip
+│   └── python3
+├── lib
+│   └── python3.7
+│       └── site-packages
+│           ├── netCDF4
+│           └── amescap
+│               ├── FV3_utils.py
+│               ├── Ncdf_wrapper.py
+│               └── Script_utils.py
+├── mars_data
+│   └── Legacy.fixed.nc
+└── mars_templates
+    ├──amescap_profile
+    └── legacy.in
+
+
+

Using conda

+

If you prefer using the conda package manager for setting up your virtual environment instead of pip, you may use the following commands to install CAP.

+

First, verify (using conda info or which conda) that you are using the intented conda executable (two or more versions of conda might be present if both Python2 and Python3 are installed on your system). Then, create the virtual environment with:

+
(local)>$ conda create -n amesCAP
+
+

Activate the virtual environment, then install CAP:

+
(local)>$ conda activate amesCAP
+(amesCAP)>$ conda install pip
+# FROM AN ARCHIVE:
+(amesCAP)>$ cd ~/Downloads
+(amesCAP)>$ tar -xf AmesCAP-master.zip
+(amesCAP)>$ cd AmesCAP-master
+(amesCAP)>$ pip install .
+# OR FROM THE GITHUB:
+(amesCAP)>$ pip install git+https://github.com/NASA-Planetary-Science/AmesCAP.git
+
+

The source code will be installed in:

+
/path/to/anaconda3/envs/amesCAP/
+
+

and the virtual environment may be activated and deactivated with conda:

+
(local)>$ conda activate amesCAP
+(amesCAP)>$ conda deactivate
+(local)>$
+
+
+

Note: CAP requires the following Python packages, which were automatically installed with CAP:

+
matplotlib        # the MatPlotLib plotting library
+numpy             # math library
+scipy             # math library and input/output for fortran binaries
+netCDF4 Python    # handling netCDF files
+requests          # downloading GCM output from the MCMC Data Portal
+
+
+
+

Removing CAP

+

To permanently remove CAP, activate the virtual environment and run the uninstall command:

+
(local)>$ source amesCAP/bin/activate          # bash
+(local)>$ source amesCAP/bin/activate.csh      # csh/tcsh
+(local)>$ source amesCAP/Scripts/activate.csh  # Cygwin/Windows
+(amesCAP)>$ pip uninstall amescap
+
+

You may also delete the amesCAP virtual environment directory at any time. This will uninstall CAP, remove the virtual environment from your machine, and will not affect your main Python distribution.

+
+

3. Testing & Using CAP

+

Whenever you want to use CAP, simply activate the virtual environment and all of CAP's executables will be accessible from the command line:

+
(local)>$ source amesCAP/bin/activate          #   bash
+(local)>$ source amesCAP/bin/activate.csh      #   csh/tcsh
+(local)>$ source amesCAP/Scripts/activate.csh  #   Cygwin/Windows
+
+

You can check that the tools are installed properly by typing Mars and then pressing the TAB key. No matter where you are on your system, you should see the following pop up:

+
(amesCAP)>$ Mars[TAB]
+> MarsFiles.py   MarsInterp.py  MarsPlot.py    MarsPull.py    MarsVars.py
+
+

If no executables show up then the paths have not been properly set in the virtual environment. You can either use the full paths to the executables:

+
(amesCAP)>$ ~/amesCAP/bin/MarsPlot.py
+
+

Or set up aliases in your ./bashrc or .cshrc:

+
# with bash:
+(local)>$ echo alias MarsPlot='/Users/username/amesCAP/bin/MarsPlot.py' >> ~/.bashrc
+(local)>$ source ~/.bashrc
+
+# with csh/tsch
+(local)>$ echo alias MarsPlot /username/amesCAP/bin/MarsPlot >> ~/.cshrc
+(local)>$ source ~/.cshrc
+
+
+

4. Practical Tips for Later Use During the Tutorial

+

Install ghostscript to Create Multiple-Page PDFs When Using MarsPlot

+

Installing ghostscript on your local machine allows CAP to generate a multiple-page PDF file instead of several individual PNGs when creating several plots. Without ghostcript, CAP defaults to generating multiple .png files instead of a single PDF file, and we therefore strongly recommend installing ghostscript to streamline the plotting process.

+

First, check whether you already have ghostscript on your machine. Open a terminal and type:

+
(local)>$ gs -version
+> GPL Ghostscript 9.54.0 (2021-03-30)
+> Copyright (C) 2021 Artifex Software, Inc.  All rights reserved.
+
+

If ghostscript is not installed, follow the directions on the ghostscript website to install it.

+
+

If gs -version returns a 'command not found error' but you are able to locate the gs executable on your system (e.g. /opt/local/bin/gs) you may need to add that specific directory (e.g. /opt/local/bin/) to your search $PATH as done for Python and pip in Step 1

+
+

Enable Syntax Highlighting for the Plot Template

+

The MarsPlot executable requires an input template with the .in file extension. We recommend using a text editor that provides language-specific (Python) syntax highlighting to make keywords more readable. A few options include: Atom and vim (compatible with MacOS, Windows, Linux), notepad++ (compatible with Windows), or gedit (compatible with Linux).

+

The most commonly used text editor is vim. Enabling proper syntax-highlighting for Python in vim can be done by adding the following lines to ~/.vimrc:

+
syntax on
+colorscheme default
+au BufReadPost *.in  set syntax=python
+
+
+

5. Do This Before Attending the Tutorial

+

In order to follow along with the practical part of the MGCM Tutorial, we ask that you download several MGCM output files beforehand. You should save these on the machine you'll be using during the tutorial.

+

We'll use CAP to retrieve these files from the MGCM Data Portal. To begin, activate the virtual environment:

+
(local)>$ source amesCAP/bin/activate      # bash
+(local)>$ source amesCAP/bin/activate.csh  # csh/tcsh
+
+

Choose a directory in which to store these MGCM output files on your machine. We will also create two sub- directories, one for an MGCM simulation with radiatively inert clouds (RIC) and one for an MGCM simulation with radiatively active clouds (RAC):

+
(amesCAP)>$ mkdir CAP_tutorial
+(amesCAP)>$ cd CAP_tutorial
+(amesCAP)>$ mkdir INERTCLDS ACTIVECLDS
+
+

Then, download the corresponding data in each directory:

+
(amesCAP)>$ cd INERTCLDS
+(amesCAP)>$ MarsPull.py -id INERTCLDS -ls 255 285
+(amesCAP)>$ cd ../ACTIVECLDS
+(amesCAP)>$ MarsPull.py -id ACTIVECLDS -ls 255 285
+
+

Finally, check for files integrity using the disk use command:

+
cd ..
+du -h INERTCLDS/fort.11*
+du -h ACTIVECLDS/fort.11*
+> 433M	fort.11_0719
+[...]
+
+

The files should be 433Mb each. That's it! CAP_tutorial now holds the necessary fort.11 files from the radiatively active and inert MGCM simulations:

+
CAP_tutorial/
+├── INERTCLDS/
+│   └── fort.11_0719  fort.11_0720  fort.11_0721  fort.11_0722  fort.11_0723
+└── ACTIVECLDS/
+    └── fort.11_0719  fort.11_0720  fort.11_0721  fort.11_0722  fort.11_0723
+
+

You can now deactivate the virtual environment:

+
(amesCAP)>$ deactivate
+
+
+

If you encounter an issue during the download process or if the files are not 433Mb, please verify the files availability on the MCMC Data Portal and try again later. You can re-attempt to download specific files as follows: MarsPull.py -id ACTIVECLDS -f fort.11_0720 fort.11_0723 (make sure to navigate to the appropriate simulation directory first), or simply download the 10 files listed above manually from the website.

+
+
+
+
+ + + + + \ No newline at end of file diff --git a/tutorial/out/CAP_lecture.html b/tutorial/out/CAP_lecture.html new file mode 100644 index 0000000..57b9be8 --- /dev/null +++ b/tutorial/out/CAP_lecture.html @@ -0,0 +1,3621 @@ + + + + + Table of Contents + + + + + + + + + + +
+
+

+ +

Table of Contents

+ + +
+

Introducing the Community Analysis Pipeline (CAP)

+

CAP is a toolkit designed to simplify the post-processing of MGCM output. CAP is written in Python and works with existing Python libraries, allowing any Python user to install and use CAP easily and free of charge. Without CAP, plotting MGCM output requires that a user provide their own scripts for post-processing, including code for interpolating the vertical grid, computing and adding derived variables to files, converting between file types, and creating diagnostic plots. In other words, a user would be responsible for the entire post-processing effort as illustrated in Figure 1.

+

Figure 1. The Typical Pipeline

+

Such a process requires that users be familiar with Fortran files and be able to write (or provide) script(s) to perform file manipulations and create plots. CAP standardizes the post-processing effort by providing executables that can perform file manipulations and create diagnostic plots from the command line. This enables users of almost any skill level to post-process and plot MGCM data (Figure 2).

+

Figure 2. The New Pipeline (CAP)

+

As a foreword, we will list a few design characteristics of CAP:

+ +

Specifically, CAP consists of five executables:

+
    +
  1. MarsPull.py Access MGCM output
    2. +
  2. MarsFiles.py Reduce the files
    3. +
  3. MarsVars.py Perform variable operations
    4. +
  4. MarsInterp.py Interpolate the vertical grid
    5. +
  5. MarsPlot.py Visualize the MGCM output
    6. +
+

These executables and their commonly-used functions are illustrated in the cheat sheet below in the order in which they are most often used. You should feel free to reference during and after the tutorial.

+

Cheat sheet

+

Figure 3. Quick Guide to Using CAP

+

CAP is designed to be modular. For example, a user could post-process and plot MGCM output exclusively with CAP or a user could employ their own post-processing routine and then use CAP to plot the data. Users are free to selectively integrate CAP into their own analysis routine to the extent they see fit.

+
+

The big question... How do I do this? > Ask for help!

+

Use the --help (-h for short) option on any executable to display documentation and examples.

+
(amesGCM3)>$ MarsPlot.py -h
+> usage: MarsPlot.py [-h] [-i INSPECT_FILE] [-d DATE [DATE ...]] [--template]
+>                   [-do DO] [-sy] [-o {pdf,eps,png}] [-vert] [-dir DIRECTORY]
+>                   [--debug]
+>                   [custom_file]
+
+
+

1. MarsPull.py - Downloading Raw MGCM Output

+

MarsPull is a utility for accessing MGCM output files hosted on the MCMC Data portal. MGCM data is archived in 1.5 hour intervals (16x/day) and packaged in files containing 10 sols. The files are named fort.11_XXXX in the order they were produced, but MarsPull maps those files to specific solar longitudes (Ls, in °). This allows users to request a file at a specific Ls or for a range of Ls using the -ls flag. Additionally the identifier (-id) flag is used to route MarsPull through a particular simulation. The filename (-f) flag can be used to parse specific files within a particular directory.

+
MarsPull.py -id INERTCLDS -ls 255 285
+MarsPull.py -id ACTIVECLDS -f fort.11_0720 fort.11_0723
+
+

Back to Top

+
+

2. MarsFiles.py - Reducing the Files

+

MarsFiles provides several tools for file manipulations, including code designed to create binned, averaged, and time-shifted files from MGCM output. The -fv3 flag is used to convert fort.11 binaries to the Netcdf data format (you can select one or more of the file format listed below):

+
(amesGCM3)>$ MarsFiles.py fort.11* -fv3 fixed average daily diurn
+
+

These are the file formats that MarsFiles can create from the fort.11 MGCM output files.

+

Primary files

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
File nameDescriptionTimesteps for 10 sols x 16 output/solRatio to daily file (430Mb)
atmos_daily.nccontinuous time series(16 x 10)=1601
atmos_diurn.ncdata binned by time of day and 5-day averaged(16 x 2)=32x5 smaller
atmos_average.nc5-day averages(1 x 2) = 2x80 smaller
fixed.ncstatics variable such as surface albedo and topographystaticfew kB
+

Secondary files

+ + + + + + + + + + + + + + + + + + + + + + + + + +
File namedescription
daily**_lpf**,_hpf,_bpflow, high and band pass filtered
diurn**_T**uniform local time (same time of day at all longitudes)
diurn**_tidal**tidally-decomposed files into harmonics
daily**_to_average** _to_diurncustom re-binning of daily files
+ +
> 07180.atmos_average.nc  07190.atmos_average.nc  07200.atmos_average.nc # 3 files with 10 days of output each
+(amesGCM3)>$ MarsFiles.py *atmos_average.nc -c
+> 07180.atmos_average.nc  # 1 file with 30 days of output
+
+

Figure X. MarsFiles options +3pm surface temperature before (left) and after (right) processing a diurn file with MarsFile to uniform local time (diurn_T.nc)

+

Back to Top

+
+

3. MarsVars.py - Performing Variable Operations

+

MarsVars provides several tools relating to variable operations such as adding and removing variables, and performing column integrations. With no other arguments, passing a file to MarsVars displays file content, much like ncdump:

+
(amesGCM3)>$ MarsVars.py 00000.atmos_average.nc
+>
+> ===================DIMENSIONS==========================
+> ['bnds', 'time', 'lat', 'lon', 'pfull', 'scalar_axis', 'phalf']
+> (etc)
+> ====================CONTENT==========================
+> pfull          : ('pfull',)= (30,), ref full pressure level  [Pa]
+> temp           : ('time', 'pfull', 'lat', 'lon')= (4, 30, 180, 360), temperature  [K]
+> (etc)
+
+

A typical option of MarsVars would be to add the atmospheric density rho to a file. Because the density is easily computed from the pressure and temperature fields, we do not archive in in the GCM output and instead provides a utility to add it as needed. This conservative approach to logging output allows to minimize disk space and speed-up post processing.

+
(amesGCM3)>$ MarsVars.py 00000.atmos_average.nc -add rho
+
+

We can see that rho was added by calling MarsVars with no argument as before:

+
(amesGCM3)>$ MarsVars.py 00000.atmos_average.nc
+>
+> ===================DIMENSIONS==========================
+> ['bnds', 'time', 'lat', 'lon', 'pfull', 'scalar_axis', 'phalf']
+> (etc)
+> ====================CONTENT==========================
+> pfull          : ('pfull',)= (30,), ref full pressure level  [Pa]
+> temp           : ('time', 'pfull', 'lat', 'lon')= (4, 30, 180, 360), temperature  [K]
+> rho            : ('time', 'pfull', 'lat', 'lon')= (4, 30, 180, 360), density (added postprocessing)  [kg/m3]
+
+

The help (-h) option provides information on available variables and needed fields for each operation.

+

Figure X. MarsVars

+

MarsVars also offers the following variable operations:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Commandflagaction
add-addadd a variable to the file
remove-rmremove a variable from a file
extract-extractextract a list of variables to a new file
col-colcolumn integration, applicable to mixing ratios in [kg/kg]
zdiff-zdiffvertical differentiation (e.g. compute gradients)
zonal_detrend-zdzonally detrend a variable
+

Back to Top

+
+

4. MarsInterp.py - Interpolating the Vertical Grid

+

Native MGCM output files use a terrain-following pressure coordinate as the vertical coordinate (pfull), which means the geometric heights and the actual mid-layer pressure of atmospheric layers vary based on the location (i.e. between adjacent grid points). In order to do any rigorous spatial averaging, it is therefore necessary to interpolate each vertical column to a same (standard) pressure grid (_pstd grid):

+

Figure X. MarsInterp

+

Pressure interpolation from the reference pressure grid to a standard pressure grid

+

MarsInterp is used to perform the vertical interpolation from reference (pfull) layers to standard (pstd) layers:

+
(amesGCM3)>$ MarsInterp.py  00000.atmos_average.nc
+
+

An inspection of the file shows that the pressure level axis which was pfull (30 layers) has been replaced by a standard pressure coordinate pstd (36 layers), and all 3- and 4-dimensional variables reflect the new shape:

+
(amesGCM3)>$ MarsInterp.py  00000.atmos_average.nc
+(amesGCM3)>$ MarsVars.py 00000.atmos_average_pstd.nc
+>
+> ===================DIMENSIONS==========================
+> ['bnds', 'time', 'lat', 'lon', 'scalar_axis', 'phalf', 'pstd']
+> ====================CONTENT==========================
+> pstd           : ('pstd',)= (36,), pressure  [Pa]
+> temp           : ('time', 'pstd', 'lat', 'lon')= (4, 36, 180, 360), temperature  [K]
+
+

MarsInterp support 3 types of vertical interpolation, which may be selected by using the --type (-t for short) flag:

+ + + + + + + + + + + + + + + + + + + + + + + + + +
file typedescriptionlow-level value in a deep crater
_pstdstandard pressure [Pa] (default)1000Pa
_zstdstandard altitude [m]-7000m
_zaglstandard altitude above ground level [m]0 m
+
+

Use of custom vertical grids

+

MarsInterp uses default grids for each of the interpolation listed above but it is possible for the user to specify the layers for the interpolation. This is done by editing a hidden file .amesgcm_profile(note the dot '.) in your home directory.

+

For the first use, you will need to copy a template of amesgcm_profile to your /home directory:

+
(amesGCM3)>$ cp ~/amesGCM3/mars_templates/amesgcm_profile ~/.amesgcm_profile # Note the dot '.' !!!
+
+

You can open ~/.amesgcm_profile with any text editor:

+
> <<<<<<<<<<<<<<| Pressure definitions for pstd |>>>>>>>>>>>>>
+
+>p44=[1.0e+03, 9.5e+02, 9.0e+02, 8.5e+02, 8.0e+02, 7.5e+02, 7.0e+02,
+>       6.5e+02, 6.0e+02, 5.5e+02, 5.0e+02, 4.5e+02, 4.0e+02, 3.5e+02,
+>       3.0e+02, 2.5e+02, 2.0e+02, 1.5e+02, 1.0e+02, 7.0e+01, 5.0e+01,
+>       3.0e+01, 2.0e+01, 1.0e+01, 7.0e+00, 5.0e+00, 3.0e+00, 2.0e+00,
+>       1.0e+00, 5.0e-01, 3.0e-01, 2.0e-01, 1.0e-01, 5.0e-02, 3.0e-02,
+>       1.0e-02, 5.0e-03, 3.0e-03, 5.0e-04, 3.0e-04, 1.0e-04, 5.0e-05,
+>       3.0e-05, 1.0e-05]
+>
+>phalf_mb=[50]
+
+

In the example above, the user custom-defined two vertical grids, one with 44 levels (named p44) and one with a single layer at 50 Pa =0.5mbar(named phalf_mb)

+

You can use these by calling MarsInterp with the -level (-l) argument followed by the name of the new grid defined in .amesgcm_profile.

+
(amesGCM3)>$ MarsInterp.py  00000.atmos_average.nc -t pstd -l  p44
+
+

Back to Top

+
+

5. MarsPlot.py - Plotting the Results

+

The last component of CAP is the plotting routine, MarsPlot, which accepts a modifiable template (Custom.in) containing a list of plots to create. MarsPlot is useful for creating plots from MGCM output quickly, and it is designed specifically for use with the netCDF output files (daily, diurn, average, fixed).

+

The following figure shows the three components of MarsPlot:

+ +

Figure 4. MarsPlot workflow

+

The default template, Custom.in, can be created by passing the -template argument to MarsPlot. Custom.in is pre-populated to draw two plots on one page: a topographical plot from the fixed file and a cross-section of the zonal wind from the average file. Creating the template and passing it into MarsPlot creates a PDF containing the plots:

+
(amesGCM3)>$ MarsPlot.py -template
+> /path/to/simulation/run_name/history/Custom.in was created
+(amesGCM3)>$
+(amesGCM3)>$ MarsPlot.py Custom.in
+> Reading Custom.in
+> [----------]  0 % (2D_lon_lat :fixed.zsurf)
+> [#####-----] 50 % (2D_lat_lev :atmos_average.ucomp, Ls= (MY 2) 252.30, zonal avg)
+> [##########]100 % (Done)
+> Merging figures...
+> /path/to/simulation/run_name/history/Diagnostics.pdf was generated
+
+

Specifically MarsPlot is designed to generate 2D cross - sections and 1D plots. +Let's remind ourselves that in order to create such plots from a multi-dimensional dataset, we first need to specify the free dimensions, meaning the ones that are not plotted.

+

Figure 4. MarsPlot cross section

+

A refresher on cross-section for multi-dimensional datasets

+

The data selection process to make any particular cross section is shown in the decision tree below. If an effort to make the process of generating multiple plots as streamlined as possible, MarsPlot selects a number of default settings for the user.

+
1.     Which simulation                                              ┌─
+    (e.g. ACTIVECLDS directory)                                      │  DEFAULT    1. ref> is current directory
+          │                                                          │  SETTINGS
+          └── 2.   Which XXXXX epoch                                 │             2. latest XXXXX.fixed in directory
+               (e.g. 00668, 07180)                                   └─
+                   │                                                 ┌─
+                   └── 3.   Which type of file                       │
+                        (e.g. diurn, average_pstd)                   │   USER      3. provided by user
+                            │                                        │ PROVIDES
+                            └── 4.   Which variable                  │             4. provided by user
+                                  (e.g. temp, ucomp)                 └─
+                                    │                                ┌─
+                                    └── 5. Which dimensions          │             5. see rule table below
+                                       (e.g lat =0°,Ls =270°)        │  DEFAULT
+                                           │                         │  SETTINGS
+                                           └── 6. plot customization │             6. default settings
+                                                  (e.g. colormap)    └─              
+
+
+

The free dimensions are set by default using day-to-day decisions from a climate modeler's perspective:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Free dimensionStatement for default settingImplementation
time"I am interested in the most recent events"time = Nt (last timestep)
level"I am more interested in the surface than any other vertical layer"level = sfc
latitude"If I have to pick a particular latitude, I would rather look at the equator"lat=0 (equator)
longitude"I am more interested in a zonal average than any particular longitude"lon=all (average over all values)
time of day"3pm =15hr Ok, this one is arbitrary. However if I use a diurn file, I have a specific time of day in mind"tod=15
+

Rule table for the default settings of the free dimensions

+

In practice, these cases cover 99% of the work typically done so whenever a setting is left to default (= None in MarsPlot's syntax) this is what is being used. This allows to considerably streamline the data selection process.

+

Custom.in can be modified using your preferred text editor (and renamed to your liking). This is an example of the code snippet in Custom.in used to generate a lon/lat cross-section. Note that the heading is set to = True, so that plot is activated for MarsPlot to process.

+
<<<<<<<<<<<<<<| Plot 2D lon X lat = True |>>>>>>>>>>>>>
+Title          = None
+Main Variable  = atmos_average.temp
+Cmin, Cmax     = None
+Ls 0-360       = None
+Level [Pa/m]   = None
+2nd Variable   = None
+Contours Var 2 = None
+Axis Options  : lon = [None,None] | lat = [None,None] | cmap = jet | scale = lin | proj = cart
+
+
+

In the example above, we are plotting the air temperature field temp from the atmos_average.nc file as a lon/lat map. temp is a 4D field (time, level, lat, lon) but since we left the time (Ls 0-360) and altitude (Level [Pa/m]) unspecified (i.e. set to None) MarsPlot will show us the last timestep in the file and the layer immediately adjacent to the surface. Similarly, MarsPlot will generate a default title for the figure with the variable's name (temperature), unit ([K]), selected dimensions (last timestep, at the surface), and makes educated choices for the range of the colormap, axis limits etc ... All those options are customizable, if desired. Finally, note the option of adding a secondary variable as solid contours. For example, one may set 2nd Variable = fixed.zsurf to plot the topography (zsurf) from the matching XXXXX.fixed.nc file.

+

To wrap-up (the use of {} to overwrite default settings is discussed later on), the following two working expressions are strictly equivalent for Main Variable = (shaded contours) or 2nd Variable = (solid contours) fields:

+
                     variable                                        variable
+                        │                     SIMPLIFY TO               │
+00668.atmos_average@1.temp{lev=1000;ls=270}     >>>      atmos_average.temp
+  │         │       │              │                           │
+epoch  file type simulation    free dimensions             file type
+                 directory
+
+

These are the four types of accepted entries for the free dimensions:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Accepted inputMeaningExample
NoneUse default settings from the rule table aboveLs 0-360 = None
valueReturn index closest to requested value in the figure'sunitLevel [Pa/m] = 50 (50 Pa)
Val Min, Val MaxReturn the average between two valuesLon +/-180 = -30,30
allall is a special keyword that return the average over all values along that dimensionLatitude = all
+

Accepted values for the Ls 0-360, Level [Pa/m] ,Lon +/-180, Latitude and time of day free dimensions

+
+

The time of day (tod) in diurn files is always specified using brackets{}, e.g. : Main Variable = atmos_diurn.temp{tod=15,18} for the average between 3pm and 6pm. This has allowed to streamlined all templates by not including the time of day free dimension, which is specific to diurn files.

+
+

MarsPlot.py: How to?

+

This section discusses MarsPlot capabilities. Note that a compact version of these instructions is present as comment at the very top of a new Custom.in and can be used as a quick reference:

+
===================== |MarsPlot V3.2|===================
+# QUICK REFERENCE:
+# > Find the matching  template for the desired plot type. Do not edit any labels left of any '=' sign
+# > Duplicate/remove any of the <<<< blocks>>>>, skip by setting <<<< block = False >>>>
+# > 'True', 'False' and 'None' are capitalized. Do not use quotes '' anywhere in this file
+etc...
+
+

Inspect the content of netCDF files

+

A handy function is MarsPlot's --inspect (-i for short) command which displays the content of a netCDF file:

+
(amesGCM3)> MarsPlot.py -i 07180.atmos_average.nc
+
+> ===================DIMENSIONS==========================
+> ['lat', 'lon', 'pfull', 'phalf', 'zgrid', 'scalar_axis', 'time']
+> [...]
+> ====================CONTENT==========================
+> pfull          : ('pfull',)= (24,), ref full pressure level  [Pa]
+> temp           : ('time', 'pfull', 'lat', 'lon')= (10, 24, 36, 60), temperature  [K]
+> ucomp          : ('time', 'pfull', 'lat', 'lon')= (10, 24, 36, 60), zonal wind  [m/sec]
+> [...]
+
+
+

Note that the -i method works with any netCDF file, not just the ones generated by CAP

+
+

The --inspect method can be combined with the --dump flag which is most useful to show the content of specific 1D arrays in the terminal.

+
(amesGCM3)>$ MarsPlot.py -i 07180.atmos_average.nc -dump pfull
+> pfull=
+> [8.7662227e-02 2.5499690e-01 5.4266089e-01 1.0518962e+00 1.9545468e+00
+> 3.5580616e+00 6.2466631e+00 1.0509957e+01 1.7400265e+01 2.8756382e+01
+> 4.7480076e+01 7.8348366e+01 1.2924281e+02 2.0770235e+02 3.0938846e+02
+> 4.1609518e+02 5.1308148e+02 5.9254102e+02 6.4705731e+02 6.7754218e+02
+> 6.9152936e+02 6.9731799e+02 6.9994830e+02 7.0082477e+02]
+> ______________________________________________________________________
+
+

The --stat flag is better suited to inspect large, multi-dimensional arrays. You can also request specific array indexes using quotes and square brackets '[]':

+
(amesGCM3)>$  MarsPlot.py -i 07180.atmos_average.nc --stat ucomp 'temp[:,-1,:,:]'
+__________________________________________________________________________
+           VAR            |      MIN      |      MEAN     |      MAX      |
+__________________________|_______________|_______________|_______________|
+                     ucomp|        -102.98|        6.99949|        192.088|
+            temp[:,-1,:,:]|        149.016|        202.508|         251.05|
+__________________________|_______________|_______________|_______________|
+
+
+

-1 refers to the last element in the that axis, following Python's indexing convention

+
+
+

Disable or add a new plot

+

Code blocks set to = True instruct MarsPlot to draw those plots. Other templates in Custom.in are set to = False by default, which instructs MarsPlot to skip those plots. In total, MarsPlot is equipped to create seven plot types:

+
<<<<<| Plot 2D lon X lat  = True |>>>>>
+<<<<<| Plot 2D lon X time = True |>>>>>
+<<<<<| Plot 2D lon X lev  = True |>>>>>
+<<<<<| Plot 2D lat X lev  = True |>>>>>
+<<<<<| Plot 2D time X lat = True |>>>>>
+<<<<<| Plot 2D time X lev = True |>>>>>
+<<<<<| Plot 1D            = True |>>>>> # Any 1D Plot Type (Dimension x Variable)
+
+

Adjust the color range and colormap

+

Cmin, Cmax (and Contours Var 2) are how the contours are set for the shaded (and solid) contours. If only two values are included, MarsPlot use 24 contours spaced between the max and min values. If more than two values are provided, MarsPlot will use those individual contours.

+
Main Variable  = atmos_average.temp     # filename.variable *REQUIRED
+Cmin, Cmax     = 240,290                # Colorbar limits (minimum, maximum)
+2nd Variable   = atmos_average.ucomp    # Overplot U winds
+Contours Var 2 = -200,-100,100,200      # List of contours for 2nd Variable or CMIN, CMAX
+Axis Options  : Ls = [None,None] | lat = [None,None] | cmap = jet |scale = lin
+
+

Note the option of setting the contour spacing linearly scale = lin or logarithmically (scale = log) if the range of values spans multiple order of magnitudes.

+

The default colormap cmap = jet may be changed using any Matplotlib colormaps. A selections of those are listed below:

+

Figure 4. MarsPlot workflow

+

Finally, note the use of the _r suffix (reverse) to reverse the order of the colormaps listed in the figure above. From example, using cmap = jet would have colors spanning from blue > red and cmap = jet_r red > blue instead

+

Supported colormaps in Marsplot. The figure was generated using code from the scipy webpage .

+
+

Make a 1D-plot

+

The 1D plot template is different from the others in a few key ways:

+ +
+

Some plots like vertical profiles and latitude plots use instead Y as the primary axis and plot the variable on the X axis

+
+
<<<<<<<<<<<<<<| Plot 1D = True |>>>>>>>>>>>>>
+Legend         = None                   # Legend instead of Title
+Main Variable  = atmos_average.temp
+Ls 0-360       = AXIS                   #       Any of these can be selected
+Latitude       = None                   #       as the X axis dimension, and
+Lon +/-180     = None                   #       the free dimensions can accept
+Level [Pa/m]   = None                   #       values as before. However,
+Diurnal  [hr]  = None                   #   ** Diurnal can ONLY be AXIS or None **
+
+

Customize 1D plots

+

Axis Options specify the axes limits, and linestyle 1D-plot:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
1D plot optionUsageExample
lat,lon+/-180,[Pa/m],sols = [None,None]range for X or Y axes limit depending on the plot typelat,lon+/-180,[Pa/m],sols = [1000,0.1]
var = [None,None]range for the plotted variable on the other axisvar = [120,250]
linestyle = - Line style following matplotlib's conventionlinestyle = -ob (solid line & blue circular markers)
axlabel = NoneChange the default name for the axisaxlabel = New Temperature [K]
+

Here is a sample of colors, linestyles and marker styles that can be used in 1D-plots

+

Figure 4. MarsPlot workflow

+

Supported styles for 1D plots. This figure was also generated using code from scipy-lectures.org

+
+

Put multiple plots on the same page

+

You can sandwich any number of plots between the HOLD ON and HOLD OFF keywords to group figures on the same page.

+
> HOLD ON
+>
+> <<<<<<| Plot 2D lon X lat = True |>>>>>>
+> Title    = Surface CO2 Ice (g/m2)
+> .. (etc) ..
+>
+> <<<<<<| Plot 2D lon X lat = True |>>>>>>
+> Title    = Surface Wind Speed (m/s)
+> .. (etc) ..
+>
+> HOLD OFF
+
+

By default, MarsPlot will use a default layout for the plots, this can be modified by adding the desired number of lines and number of columns, separated by a comma: HOLD ON 4,3 will organize the figure with a 4 -lines and 3-column layout.

+

Note that Custom.in comes with two plots pre-loaded on the same page.

+
+

Put multiple 1D-plots on the same page

+

Similarly adding the ADD LINE keywords between two (or more) templates can be used to place multiple 1D plots on the same figure.

+
> <<<<<<| Plot 1D = True |>>>>>>
+> Main Variable    = var1
+> .. (etc) ..
+>
+> ADD LINE
+>
+> <<<<<<| Plot 1D = True |>>>>>>
+> Main Variable    = var2
+> .. (etc) ..
+
+
+

Note that if you combine HOLD ON/HOLD OFF and ADD LINE to create a 1D figure with several sub-plots on a multi-figure page, the 1D plot has to be the LAST (and only 1D-figure with sub-plots) on that page.

+
+
+

Use a different epoch

+

If you have run a GCM simulation for a long time, you may have several files of the same type, e.g. :

+
00000.fixed.nc          00100.fixed.nc         00200.fixed.nc         00300.fixed.nc
+00000.atmos_average.nc  00100.atmos_average.nc 00200.atmos_average.nc 00300.atmos_average.nc
+
+

By default MarsPlot counts the fixed files in the directory and run the analysis on the last set of files, 00300.fixed.nc and 00300.atmos_average.nc in our example. Even though you may specify the epoch for each plot (e.g. Main Variable = 00200.atmos_average.temp for the file starting at 200 sols), it is more convenient to leave the epoch out of the Custom.in and instead pass the -date argument to MarsPlot.

+
MarsPlot.py Custom.in -d 200
+
+
+

-date also accepts a range of sols, e.g. MarsPlot.py Custom.in -d 100 300 which will run the plotting routine across multiple files.

+
+

When creating 1D plots of data spanning multiple years, you can overplot consecutive years on top of the other instead of sequentially by calling --stack_year (-sy) when submitting the template to MarsPlot.

+
+

Access simulation in a different directory

+

At the beginning of MarsPlot is the <<< Simulations >>> block which, is used to point MarsPlot to different directories containing MGCM outputs. When set to None, ref> (the simulation directory number @1, optional in the templates) refers to the current directory:

+
<<<<<<<<<<<<<<<<<<<<<< Simulations >>>>>>>>>>>>>>>>>>>>>
+ref> None
+2> /path/to/another/sim                            # another simulation
+3>
+=======================================================
+
+

Only 3 simulations have place holders but you can add additional ones if you would like (e.g. 4> ... ) +To access a variable from a file in another directory, just point to the correct simulation when calling Main Variable (or 2nd Variable) using the @ character:

+
Main Variable  = XXXXX.filename@N.variable`
+
+

Where N is the number in <<< Simulations >>> corresponding the the correct path.

+
+

Overwrite the free dimensions.

+

By default, MarsPlot uses the free dimensions provided in each template (Ls 0-360 and Level [Pa/m] in the example below) to reduce the data for both the Main Variable and the 2nd Variable. You can overwrite this behavior by using parenthesis {}, containing a list of specific free dimensions separated by semi-colons ; The free dimensions within the {} parenthesis will ultimately be the last one selected. In the example below, Main Variable (shaded contours) will use a solar longitude of 270° and a pressure of 10 Pa, but the 2nd Variable (solid contours) will use the average of solar longitudes between 90° and 180° and a pressure of 50 Pa.

+
<<<<<<<<<<<<<<| Plot 2D lon X lat = True |>>>>>>>>>>>>>
+...
+Main Variable  = atmos_average.var
+...
+Ls 0-360       = 270
+Level [Pa/m]   = 10
+2nd Variable   = atmos_average.var{ls=90,180;lev=50}
+
+
+

Keywords for the dimensions are ls, lev, lon, lat and tod. Accepted entries are Value (closest), Valmin,Valmax (average between two values) and all (average over all values)

+
+

Element-wise operations

+

You can encompass variables between square brackets [] to perform element-wise operations, which is useful to compare simulations, apply scaling etc... MarsPlot will first load each variables encompassed with the brackets, and then apply the algebraic expression outside the [] before plotting.

+

These are examples of potential applications:

+
 > Main Variable  = [fixed.zsurf]/(10.**3)                            (convert topography from [m] to [km])
+ > Main Variable  = [atmos_average.taudust_IR]/[atmos_average.ps]*610 (normalize the dust opacity)     
+ > Main Variable  = [atmos_average.temp]-[atmos_average@2.temp]       (temp. difference between ref simu and simu 2)
+ > Main Variable  = [atmos_average.temp]-[atmos_average.temp{lev=10}] (temp. difference between the default (near surface) and the 10 Pa level
+
+
+

Code comments and speed-up processing

+

Comments are preceded by #, following python's convention. Each <<<<| block |>>>> must stay integral so comments may be inserted between templates or comment all lines of the template (which is why it is generally easier to simply set the <<<<| block = False |>>>>) but not within a template.

+

You will notice the START key word at the very beginning of the template.

+
=======================================================
+START
+
+

This instructs MarsPlot to start parsing templates at this point. If you are already happy with multiple plots, you can move the START keyword further down in the Custom.in to skip those first plots instead of setting those to <<<<| Plot = False |>>>> individually. When you are done with your analysis, simply move START back to the top to generate a pdf with all the plots.

+

Similarly, you can use the keyword STOP (which is not initially present in Custom.in) to stop the parsing of templates. In this case, the only plots processed would be the ones between START and STOP.

+
+

Change projections

+

For Plot 2D lon X lat figures, MarsPlot supports 3 types of cylindrical projections : cart (cartesian), robin (robinson), moll (mollweide), and 3 types of azimuthal projections: Npole (north polar), Spole (south polar) and ortho (orthographic).

+

Figure 4. MarsPlot workflow +(Top) cylindrical projection cart, robin and moll. (Bottom) azimuthal projections Npole, Spole and ortho

+

The azimuthal projections accept optional arguments as follows:

+
proj = Npole lat_max                   # effectively zoom in/out on the North pole
+proj = Spole lat_min                   # effectively zoom in/out on the South pole
+proj = ortho lon_center, lat_center    # rotate the globe
+
+
+

Figure format, size

+ +
+

Access CAP libraries and make your own plots

+

CAP libraries are located (and documented) in FV3_utils.py. Spectral utilities are located in Spectral_utils.py, classes to parse fortran binaries and generate netCDf files are located in Ncdf_wrapper.py

+

The following code demonstrate how one can access CAP libraries and make plots for its own analysis:

+
#======================= Import python packages ================================
+import numpy as np                          # for array operations
+import matplotlib.pyplot as plt             # python plotting library
+from netCDF4 import Dataset                 # to read .nc files
+#===============================================================================
+
+# Open a fixed.nc file, read some variables and close it.
+f_fixed=Dataset('/path_to_file/00000.fixed.nc','r')
+lon=f_fixed.variables['lon'][:]
+lat=f_fixed.variables['lat'][:]
+zsurf=f_fixed.variables['zsurf'][:]  
+f_fixed.close()
+
+# Open a dataset and read the 'variables' attribute from the NETCDF FILE
+f_average_pstd=Dataset('/path_to_file/00000.atmos_average_pstd.nc','r')
+vars_list     =f_average_pstd.variables.keys()
+print('The variables in the atmos files are: ',vars_list)
+
+# Read the 'shape' and 'units' attribute from the temperature VARIABLE
+Nt,Nz,Ny,Nx = f_average_pstd.variables['temp'].shape
+units_txt   = f_average_pstd.variables['temp'].units
+print('The data dimensions are Nt,Nz,Ny,Nx=',Nt,Nz,Ny,Nx)
+# Read the pressure, time, and the temperature for an equatorial cross section
+pstd       = f_average_pstd.variables['pstd'][:]   
+areo       = f_average_pstd.variables['areo'][0] #solar longitude for the 1st timestep
+temp       = f_average_pstd.variables['temp'][0,:,18,:] #time, press, lat, lon
+f_average_pstd.close()
+
+# Get the latitude of the cross section.
+lat_cross=lat[18]
+
+# Example of accessing  functions from the Ames Pipeline if we wanted to plot
+# the data  in a different coordinate system  (0>360 instead of +/-180 )
+#----
+from amesgcm.FV3_utils import lon180_to_360,shiftgrid_180_to_360
+lon360=lon180_to_360(lon)
+temp360=shiftgrid_180_to_360(lon,temp)
+
+# Define some contours for plotting
+conts= np.linspace(150,250,32)
+
+#Create a figure with the data
+plt.close('all')
+ax=plt.subplot(111)
+plt.contourf(lon,pstd,temp,conts,cmap='jet',extend='both')
+plt.colorbar()
+# Axis labeling
+ax.invert_yaxis()
+ax.set_yscale("log")
+plt.xlabel('Longitudes')
+plt.ylabel('Pressure [Pa]')
+plt.title('Temperature [%s] at Ls %03i, lat= %.2f '%(units_txt,areo,lat_cross))
+plt.show()
+
+

will produce the following image:

+

+
+

Debugging

+

MarsPlot is designed to make plotting MGCM output easier and faster so it handles missing data and many errors by itself. It reports errors both in the terminal and in the generated figures. To by-pass this behavior (when debugging), use the --debug option with MarsPlot which will raise standard Python errors and stop the execution. One thing to always look for are typo/syntax errors in the template so you may want to cross-check your current plot against a pristine (empty) template.

+
+

Note that the errors raised with the --debug flag may reference to MarsPlot internal classes so they may not always be self-explanatory.

+
+

Back to Top

+
+
+
+ + + + + \ No newline at end of file diff --git a/tutorial/out/README.html b/tutorial/out/README.html new file mode 100644 index 0000000..6eaff69 --- /dev/null +++ b/tutorial/out/README.html @@ -0,0 +1,2882 @@ + + + + + 1. [Introduction to CAP and documentation of its functionalities](./CAP_lecture.md) + + + + + + + + + +
+
+

+

This directory contains tutorial documents describing the installation and functionality for CAP as well as a set of practise exercises. It is a great place to start for new users.

+

1. Introduction to CAP and documentation of its functionalities

+

2. Step-by-step installation

+

3. Practise exercises

+
+
+ + + + + \ No newline at end of file diff --git a/tutorial/tutorial_images/.DS_Store b/tutorial/tutorial_images/.DS_Store new file mode 100644 index 0000000..5008ddf Binary files /dev/null and b/tutorial/tutorial_images/.DS_Store differ diff --git a/tutorial/tutorial_images/50Pa_Temps.png b/tutorial/tutorial_images/50Pa_Temps.png new file mode 100644 index 0000000..efb8288 Binary files /dev/null and b/tutorial/tutorial_images/50Pa_Temps.png differ diff --git a/tutorial/tutorial_images/Default.png b/tutorial/tutorial_images/Default.png new file mode 100644 index 0000000..efa9981 Binary files /dev/null and b/tutorial/tutorial_images/Default.png differ diff --git a/tutorial/tutorial_images/GCM_Workflow_PRO.png b/tutorial/tutorial_images/GCM_Workflow_PRO.png index 25ddf59..9c80287 100644 Binary files a/tutorial/tutorial_images/GCM_Workflow_PRO.png and b/tutorial/tutorial_images/GCM_Workflow_PRO.png differ diff --git a/tutorial/tutorial_images/Global_Dust.png b/tutorial/tutorial_images/Global_Dust.png new file mode 100644 index 0000000..b840ed1 Binary files /dev/null and b/tutorial/tutorial_images/Global_Dust.png differ diff --git a/tutorial/tutorial_images/Tutorial_Banner_2023.png b/tutorial/tutorial_images/Tutorial_Banner_2023.png index f299cc8..802fb5d 100644 Binary files a/tutorial/tutorial_images/Tutorial_Banner_2023.png and b/tutorial/tutorial_images/Tutorial_Banner_2023.png differ diff --git a/tutorial/tutorial_images/Zonal_Circulation.png b/tutorial/tutorial_images/Zonal_Circulation.png new file mode 100644 index 0000000..14ba4ec Binary files /dev/null and b/tutorial/tutorial_images/Zonal_Circulation.png differ diff --git a/tutorial/tutorial_images/Zonal_Surface.png b/tutorial/tutorial_images/Zonal_Surface.png new file mode 100644 index 0000000..1a98ae7 Binary files /dev/null and b/tutorial/tutorial_images/Zonal_Surface.png differ