These instructions are intenteded to help those who have already configured and run UCERF3-ETAS to respond to a recent event. If you need to reconfigure UCERF3-ETAS-Launcher from scratch, first follow the instructions here.
If you're responding to an event, it's likely that you haven't updated UCERF3-ETAS-Launcher in some time. Do this on your local machine, and also on the cluster if you're going to run simulations in parallel.
cd $ETAS_LAUNCHER; git pull
Then, if you haven't already, define the ETAS_SIM_DIR environmental variable on all machines that you will use (the same way that you defined ETAS_LAUNCHER when you installed UCERF3-ETAS-Launcher). This should point to the directory where you want to store ETAS simulations, and allows you to use the same config.json file on multiple machines (e.g. to run simulations on a cluster, the copy back output and plot results on your local machine).
If you're working on USC CARC, ETAS_SIM_DIR should point to somewhere on /project/scec_608/$USER, e.g. /project/scec_608/$USER/ucerf3/etas_sim.
You'll need the ComCat event ID for the event, which you can determine from the URL on the USGS event page. For example, for the 2019 Ridgecrest M7.1, the full URL is https://earthquake.usgs.gov/earthquakes/eventpage/ci38457511/executive and the event ID is ci38457511.
Use the u3etas_comcat_event_config_builder.sh script to pull the event information directly from ComCat and configure a simulation. Full instructions are here, but here are the basics:
By default, only the given event will be included and the simulations will start immediately after the event. You can include foreshocks and aftershocks with --days-before <days>, --hours-before <hours>, --days-after <days>, --hours-after <hours>, and/or --end-now (the latter fetches all aftershocks up until the current moment and starts simulations at that time).
There are a few ways to describe finite fault surfaces. ShakeMap surfaces are currently recommended if available, use --finite-surf-shakemap if such a surface exists. You can also specify custom surfaces (e.g. drawn through seismicity) by following instructions here.
You'll probably want to run simulations on USC CARC, so include those options as well. The SCEC queue at USC CARC (accessible from endeavour) has 38 nodes. You can see how many of them are currently in use with the scec_queue_check.py command, then use at most 38 minus the reported number of nodes in use.
NOTE: If you're configuring the simulation on a login node and you see an error message like Error occurred during initialization of VM. Could not reserve enough space for <some value> object heap, then you'll need to set the ETAS_MEM_GB to a suitable level for a shared login node. 4 GB should suffice for this purpose on USC CARC or TACC Stamped2, so execute this command first: export ETAS_MEM_GB=4 and try again. TACC Frontera has stict memory limits on the login nodes, try export ETAS_MEM_GB=0.5 on that system.
Here's an example for 100,000 Ridgecrest simulations, starting immediately after the M7.1 (including seismicity 7 days before), using ShakeMap surfaces for all M>=5's, and configured to run on 36 nodes for up to 24 hours on the SCEC queue at USC CARC:
u3etas_comcat_event_config_builder.sh --event-id ci38457511 --num-simulations 100000 --days-before 7 --finite-surf-shakemap --finite-surf-shakemap-min-mag 5 --hpc-site USC_CARC --nodes 36 --hours 24 --queue scec
This command will generate the config.json and etas_sim_mpj.slurm files referenced below. It will also generate some diagnostic plots showing the input events in relation to UCERF fault sections in the config_input_plots subdirectory, which can be useful to see if you should reset the elastic rebound clock on any UCERF3 subsections (with the --reset-sects <id1,id2,...,idN> flag). Example output for the Ridgecrest M7.1 is available here.
Run simulations on a single machine with etas_launcher.sh config.json, or follow these instructions to submit them on a cluster.
If you're using a cluster, you'll then want to follow the instructions to monitor job progress.
When the job is complete, or at least some simulations have finished, you can plot the results with u3etas_plot_generator.sh config.json. This will generate many plots (in the 'plots' subdirectory) as well as an HTML index that can be opened in a web browser.
If you ran the simulations on a cluster, you have a few options:
- Copy results back to your local machine to process (either results_complete.bin or results_complete_partial.bin if the simulation is still in progress). This can be tedious if the results are very large
- This is no longer viable!
Use another SCEC machine. Don't try to run the plot generator code on the head node of a HPC center (they won't like it, it'll be crazy slow, and will probably get killed). If you want to plot results on a SCEC machine, try opensha.usc.edu (you should be able to login with the same credentials that you use on www.scec.org). If the simulations were run at USC CARC, then that filesystem (e.g. /home/scec-00) should be accessible directly - Submit the
plot_results.slurmfile that was automatcially generated to the batch scheduler. One complication of this route is that some HPC centers don't allow compute nodes to see the outside world. In that case, the "--no-maps" option should be enabled in the SLURM file which skips plots which would otherwise connect to opensha.usc.edu to produce maps. Another issue is that the compute nodes may not be able to access ComCat to fetch real events for comparison, in which case you should use the u3etas_comcat_catalog_fetcher.sh script to first download a catalog, then edit theplot_results.slurmto point to it. This is no longer a complication for USC CARC, which can now see the outside world.