Skip to content

Using WSL

Hayden Atchley edited this page May 1, 2023 · 4 revisions

Note steps 3 onward are all in a WSL terminal:

  1. Install WSL
  2. Open a WSL terminal
  3. Go to the WSL home directory (cd ~), if you then pwd it should be /home/<username>/
  4. From there clone the repository (within WSL, you can put it wherever relative to your ~ directory if you want) git clone --recurse-submodules https://github.com/byu-transpolab/wfrc_asim_scenario
  5. cd wfrc_asim_scenario
  6. Make sure R is installed (sudo apt install r-base r-base-dev)
  7. Open R from the terminal (R)
  8. You'll probably have to install the R packages since WSL can't use the windows-installed ones. The packages are: tidyverse, sf, tigris, tidycensus, targets, tarchetypes, and xml2. There's a good chance you'll run into errors with the installation due to missing packages etc. in WSL. Most of the errors will look something like the following:
------------ Missing dependencies ----------
The library libgdal is missing. The library can be found
in one of the following:

deb: libgdal-devel libgdal (debian, ubuntu, etc.)
rpm: lib-gdal (fedora, rhel, etc.)
etc.
---------------------------------------------

The exact wording is probably wrong since I'm typing this from memory, but there will be a text block that looks generally like this. It may be in the middle of the install text, so I'd watch the process and stop it with ctrl-c (you may need to push that a few times) once you see something like this. The only important line is the deb: line: install these library(ies) with: sudo apt install <copy everything after 'deb:' and before the parenthesis> (you'll need to do this outside an R session; it may be helpful to have two WSL windows side by side for this). Then try installing the R packages again. This is an iterative process and you will probably get different errors like this each time; just keep installing the dependencies and eventually all the R packages will install. You might also need to run sudo apt install build-essential; in fact I'd recommend doing this first. You can exit out of R for now (q()).


Now install conda (still from within WSL):

cd ~
sudo apt install wget
wget <https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh>
sh Miniconda3-latest-Linux-x86_64.sh

The defaults for the installation are probably fine (don't have it run conda init yet). After the install finishes, see if you can run conda. It will probably give you an error that the command can't be found. Note where the installation path was (probably /home/<username>/miniconda3). Now: cd ~, nano .bashrc, and add the following line somewhere: export PATH=$PATH:<conda install path>/bin. Save with ctrl-o , exit with ctrl-x. Now run nano .bash_profile and add source .bashrc to it (the file may be empty to start, that's fine). Exit, then source .bash_profile. See if conda works now, it should. Then run conda init. It should list a bunch of files and say some are modified. Exit out of any WSL window(s) and re-open them, you should see a (base) at the start of the terminal prompt. If so, then conda is good to go!


With WSL open, now set up the two conda environments:

cd ~/wfrc_asim_scenario #Change this to be wherever the repo is

#create activitysim conda environment
conda create -y -n ASIM_DEV python=3.9
conda activate ASIM_DEV
conda install -y pytables
pip install -r reference/asim_requirements.txt

#create populationsim conda environment
conda create -y -n popsim python=3.8
conda activate popsim

conda install -y pytables
pip install scipy
pip install populationsim

conda activate base

This may take some time. If the ASIM_DEV environment above doesn't work when you're running scripts, create it with conda env create --file=activitysim/conda-environments/activitysim-dev.yml --name ASIM_DEV instead (make sure you cd to wherever the repository is first). You'll have to conda remove --name ASIM_DEV --all before remaking it. This way of creating the environment will take probably 10-15 minutes to run.


Now you need a Census API key. https://api.census.gov/data/key_signup.html. You might get two emails, and the link to activate the key might be finnicky, but it worked after a few tries for me. Copy this key. In WSL cd to the repo directory, and nano .Renviron. Write CENSUS_API_KEY="<your key here>" (with the quotes) and save and exit.

Now start R (R) from the terminal in the root of the repository (probably cd ~/wfrc_asim_scenario). Run tar_make() and see what happens. It *should *make it to I think skims_file and then error (if it keeps going and doesn't error, then great! That should be fully working). Ideally the targets pipeline would run the .sh files itself, but I couldn't get it to outside of linux. So we need to do that part manually.

There are a few .sh scripts that need to be run at certain points in the targets pipeline, and the process will be similar for all of them. Find what the errored target does (for skims_file, it runs prepare_skims()). We want to find which .sh file is being run (sh/build_skims.sh) and then run those commands manually. The if statements about sourcing .bashrc or .bash_profile can be skipped, as can any printf lines. For build_skims.sh, that means we need to run conda activate ASIM_DEV and the python line (conda info just shows info about the current env, you can run it if you want). For the python line, though, we need to replace the $ arguments with actual folder paths. What those mean is the nth argument of the command for $n (counting from 0). So we go back to prepare_skims() and see that the args are the script and three directories. So python py/build_omx.py "$1" "$2" "$3" becomes python py/build_omx.py inputs/skims inputs/skims data_activitysim (we don't need to add the || exit 1). Run this in WSL (outside of R) and it *should *work.

If it works, great! You will need to edit the targets pipeline a bit to essentially bypass running the build_skims.sh file (and the other script files once you've gotten to them). Since the script file targets return a file path in the targets pipeline, you can just hard-code the path (in the skims case, it's data_activitysim/skims.omx, from the return statement in the prepare_skims() function). I believe this process needs to be done for build_skims.sh, run_popsim.sh, and run_activitysim.sh. Unfortunately, you'll have to alternate re-running tar_make() and these scripts manually each time you change something upstream.


I'm now looking at the code more closely, and I'm noticing a couple things that I think are incorrectly written. I'm not entirely sure since it's been a while, but you could try these changes if you run into errors.

  • In run_popsim.sh, the ASIM_DEV environment is activated instead of popsim. This might work, but you might need to use popsim.
  • You could try adding the line #!/bin/bash -v to the top of the three scripts, there's a chance this will make it work within the targets pipeline. Not sure though.
  • The sourcing of .bashrc and .bash_profile (lines 5 and 8) are backwards based on the if statements in the scripts. This shouldn't matter since you made both anyway, and if you're running the scripts manually you can skip these lines entirely, but fixing this may make the scripts work in the targets pipeline.

Not sure why these discrepancies, but it was probably just oversight on my part that happened to still work with the supercomputer setup.

Clone this wiki locally