usingr.Rmd

```{r setup, echo=FALSE, warning=FALSE, purl=FALSE, message=FALSE}
options(repos = "http://cran.rstudio.com/")
pkgs <- c("dplyr", "knitr")
x<-lapply(pkgs, library, character.only = TRUE)
opts_chunk$set(tidy = FALSE, message = F, warning = F)
```

# Using R for bioassessment data: What, why, and how

## Lesson Outline

- [Goals and Motivation]
- [RStudio]
- [R language fundamentals]
- [Data structures in R]
- [Getting your data into R]

## Lesson Exercises

- [Exercise 1]
- [Exercise 2]
- [Exercise 3]

## Goals and Motivation

[R](https://www.r-project.org/) is a language for statistical computing as well as a general purpose programming language. Increasingly, it has become one of the primary languages used in data science and for data analysis across many of the natural sciences. 

The goals of this training are to expose you to fundamentals and to develop an appreciation of what's possible with this software.  We also provide resources that you can use for follow-up learning on your own. You should be able to answer these questions at the end of this session:

* What is R and why should I use it?
* Why would I use RStudio and RStudio projects?
* How can I write, save, and run scripts in RStudio?
* Where can I go for help?
* What are the basic data structures in R?
* How do I import data? 

### Why should I invest time in R?

There are many programming languages available and each has it's specific benefits. R was originally created as a statistical programming language but now it is largely viewed as a 'data science' language. Why would you invest time in learning R compared to other languages?  

* The growth of R as explained in the [Stack Overflow blog](https://stackoverflow.blog/2017/10/10/impressive-growth-r/), [IEEE rating](http://blog.revolutionanalytics.com/2016/07/r-moves-up-to-5th-place-in-ieee-language-rankings.html)

R is also an open-source programming language - not only is it free, but this means anybody can contribute to it's development. As of `r format(Sys.time(), "%Y-%m-%d")`, there are `r nrow(available.packages())` supplemental packages for R on CRAN!   

## RStudio

In the old days, the only way to use R was directly from the Console - this is a bare bones way of running R only with direct input of commands. Now, [RStudio](https://www.rstudio.com/) is the go-to Interactive Development Environment (IDE) for R. Think of it like a car that is built around an engine. It is integrated with the console (engine) and includes many other features to improve the user's experience, such as version control, debugging, dynamic documents, package manager and creation, and code highlighting and completion. 

Let's get familiar with RStudio before we go on.  

### Open R and RStudio

If you haven't done so, download and install RStudio from the link above.  After it's installed, find the RStudio shortcut and fire it up (just watch for now).  You should see something like this: 

![](figure/rstudio.png)

There are four panes in RStudio: 

* Source
* console
* Environment, History, etc.
* Files, plots, etc.

### RStudio projects

I strongly encourage you to use RStudio projects when you are working with R.  The RStudio project provides a central location for working on a particular task.  It helps with file management and is portable because all the files live in the same project.  RStudio projects also remember history - what commands you used and what data objects are in your enviornment.  

To create a new project, click on the File menu at the top and select 'New project...'

![](figure/rstudio_proj.jpg)

Now we can use this project for our data and any scripts we create.  

### Scripting

In most cases, you will not enter and execute code directly in the console.  Code can be written in a script and then sent directly to the console when you're ready to run it.  The key difference here is that a script can be saved and shared.

Open a new script from the File menu...

![](figure/rstudio_script.jpg)

### Executing code in RStudio

After you write your script it can be sent to the Console to run the code in R.  Any variables you create in your script will not be available in your working environment until this is done.  There are two ways to send code to the console.  First, you can hit the `Run` button at the top right of the scripting window.  Second, and preferred, you can use `ctrl+enter` (`cmd+enter` on a Mac).  Both approaches will send the selected line to the console, then move to the next line in your script.  You can also highlight and send an entire block of code.

![](figure/rstudio_run.jpg)

### What is the environment?

There are two outcomes when you run code.  First, the code will simply print output directly in the console.  Second, there is no output because you have stored it as a variable (we'll talk about variable assignment later).  Output that is stored is actually saved in the `environment`.  The environment is the collection of named objects that are stored in memory for your current R session.  Anything stored in memory will be accessible by it's name without running the original script that was used to create it.  

## Exercise 1

This exercise will make sure R and RStudio are working and that you can get around the basics in RStudio.  Use the blue stickies when you have completed, and red stickies if you are running into problems.

1. Start RStudio: To start both R and RStudio requires only firing up RStudio.  RStudio should be available from All Programs at the Start Menu.  Fire up RStudio. 

1. Create a new project.  Name it "cabw_r_workshop".  We will use this for the rest of the workshop.

1. Create a new "R Script" in the Source Pane, save that file into your newly created project and name it "cabw_script.R". It'll just be a blank text file at this point.

1. Add in a comment line to separate this section.  It should look something like: `# Exercise 1: Just Getting used to RStudio and Scripts`.

1. Lastly, we need to get this project set up with some example data for our exercises.  You should have downloaded this already, but if not, the data are available [here](https://SCCWRP.github.io/CABW2018_R_training/data/datazip.zip).  The data are in a zipped folder.  Download the file to your computer (anywhere).  Create a folder in your new project named `data` and extract the files into this location.  

## R language fundamentals

The basic syntax of a function follows the form: `function_name(arg1, arg2, ...)`.  

With the base install, you will gain access to many functions (`r pkgs <- search(); pkgs <- pkgs[grep("package:",pkgs)]; length(unlist(sapply(pkgs,lsf.str)))`, to be exact). 
Some examples:

```{r}
# print
print('hello world!')

# sequence
seq(1, 10)

# random numbers
rnorm(100, mean = 10, sd = 2)

# average 
mean(rnorm(100))

# sum
sum(rnorm(100))
```

Very often you will see functions used like this:

```{r}
my_random_sum <- sum(rnorm(100))
```

In this case the first part of the line is the name of an object.  You make this up.  Ideally it should have some meaning, but the only rules are that it can't start with a number and must not have any spaces.  The second bit, `<-`, is the assignment operator.  This tells R to take the result of `sum(rnorm(100))` and store it in an object named, `my_random_sum`.  It is stored in the environment and can be used by just executing it's name in the console.

```{r}
my_random_sum
```

With this, you have the very basics of how we write R code and save objects that can be used later.

### Packages

The base install of R is quite powerful, but you will soon have a need or desire to go beyond this.  Packages provide this ability.  They are a standardized way of extending R with new methods, techniques, and programming functionality.  There is a lot to say about packages regarding finding them, using them, etc., but for now let's focus just on the basics.  

### CRAN

One of the reasons for R's popularity is CRAN, [The Comprehensive R Archive Network](http://cran.r-project.org/).  This is where you download R and also where most will gain access to packages (there are other places, but that is for later).  Not much else to say about this now other than to be aware of it.  As of `r format(Sys.time(), "%Y-%m-%d")`, there are `r nrow(available.packages())` on CRAN! 

### Installing packages

When a package gets installed, that means the source code is downloaded and put into your library.  A default library location is set for you so no need to worry about that.  In fact, on Windows most of this is pretty automatic.  Let's give it a shot.

## Exercise 2

We're going to install some packages from CRAN that will give us the tools for our workshop today.  We'll use the tidyverse, sf, mapview, viridis, and USAboundaries packages.  Later, we'll explain in detail what each of these packages provide.  

1. At the top of the script you just created, type the following functions.

    ```{r install_package, eval=FALSE}
# install packages from CRAN
install.packages("tidyverse")
install.packages("sf")
install.packages("mapview")
install.packages("viridis")
install.packages("USAboundaries")
    ```

1. Select all the lines by clicking and dragging the mouse pointer over the text.  

1. Send all the commands to the console using `ctrl+enter`.  You should see some text output on the console about the installation process.  The installation may take a few minutes so don't be alarmed.  

1. After the packages are done installing, verify that there were no errors during the process (this should be pretty obvious, i.e., error text in big scary red letters).

1. Load the packages after they've installed. 

    ```{r eval=FALSE}
library("tidyverse")
library("sf")
library("mapview")
library("viridis")
library("USAboundaries")
    ```

### Getting Help

Being able to find help and interpret that help is probably one of the most important skills for learning a new language.  R is no different. Help on functions and packages can be accessed directly from R, can be found on CRAN and other official R resources, searched on Google, found on StackOverflow, or from any number of fantastic online resources. I will cover a few of these here. 

### Help from the console

Getting help from the console is straightforward and can be done numerous ways.

```{r help_from_console, eval=FALSE}
# Using the help command/shortcut
# When you know the name of a function
help("print") # Help on the print command
?print # Help on the print command using the `?` shortcut

# When you know the name of the package
help(package = "sf") # Help on the package `dplyr`

# Don't know the exact name or just part of it
apropos("print") # Returns all available functions with "print" in the name
??print # shortcut, but also searches demos and vignettes in a formatted page
```

### Official R Resources

In addition to help from within R itself, CRAN and the R-Project have many resources available for support.  Two of the most notable are the mailing lists and the [task views](http://cran.r-project.org/web/views/).

- [R Help Mailing List](https://stat.ethz.ch/mailman/listinfo/r-help): The main mailing list for R help.  Can be a bit daunting and some (although not most) senior folks can be, um, curmudgeonly...
- [R-sig-ecology](https://stat.ethz.ch/mailman/listinfo/r-sig-ecology): A special interest group for use of R in ecology.  Less daunting than the main help with participation from some big names in ecological modelling and statistics (e.g., Ben Bolker, Gavin Simpson, and Phil Dixon).
- [Environmetrics Task View](http://cran.r-project.org/web/views/Environmetrics.html): Task views are great in that they provide an annotated list of packages relevant to a particular field.  This one is maintained by Gavin Simpson and has great info on packages relevant to much of the work at EPA.
- [Spatial Analysis Task View](http://cran.r-project.org/web/views/Spatial.html): One I use a lot that lists all the relevant packages for spatial analysis, GIS, and Remote Sensing in R. 

### Google and StackOverflow

While the resources already mentioned are useful, often the quickest way is to just turn to Google.  However, a search for "R" is a bit challenging.  A few ways around this.  Google works great if you search for a given package or function name.  You can also search for mailing lists directly (i.e. "R-sig-geo"), although Google often finds results from these sources.

Blind googling can require a bit of strategy to get the info you want.  Some pointers:

* Always preface the search with "r"
* Understand which sources are reliable
* Take note of the number of hits and date of a web page
* When in doubt, search with the exact error message (see here for [details](https://cran.r-project.org/doc/manuals/R-lang.html#Exception-handling) about warnings vs errors)

One specific resource that I use quite a bit is [StackOverflow with the 'r' tag](http://stackoverflow.com/questions/tagged/r).  StackOverflow is a discussion forum for all things related to programming.  You can then use this tag and the search functions in StackOverflow and find answers to almost anything you can think of. However, these forums are also very strict and I typically use them to find answers not to ask questions.  

### Other Resources

As I mentioned earlier, there are TOO many resources to list here and everyone has their favorites.  Below are just a few that I like.

- [R For Cats](http://rforcats.net/): Basic introduction site, meant to be a gentle and light-hearted introduction
- [Advanced R](http://adv-r.had.co.nz/): Web home of Hadley Wickham's new book.  Gets into more advanced topics, but also covers the basics in a great way.
- [CRAN Cheatsheets](http://cran.r-project.org/doc/contrib/Short-refcard.pdf): A good cheat sheet from the official source
- [RStudio Cheatsheets](http://www.rstudio.com/resources/cheatsheets/): Additional cheat sheets from RStudio.  I am especially fond of the data wrangling one.

## Data structures in R

Now that you know how to get started in R and where to find resources, we can begin talking about R data structures. Simply put, a data structure is a way for programming languages to handle information storage.

There is a bewildering amount of formats for storing data and R is no exception. Understanding the basic building blocks that make up data types is essential.  All functions in R require specific types of input data and the key to using functions is knowing how these types relate to each other.  

### Vectors (one-dimensional data)

The basic data format in R is a vector - a one-dimensional grouping of elements that have the same type.  These are all vectors and they are created with the `c` function:

```{r}
dbl_var <- c(1, 2.5, 4.5)
int_var <- c(1L, 6L, 10L)
log_var <- c(TRUE, FALSE, T, F)
chr_var <- c("a", "b", "c")
```

The four types of atomic vectors (think atoms that make up a molecule aka vector) are `double` (or numeric), `integer`, `logical`, and `character`.  For most purposes you can ignore the `integer` class, so there are basically three types.  Each type has some useful properties:

```{r}
class(dbl_var)
length(log_var)
```

These properties are useful for not only describing an object, but they define limits on which functions or types of operations that can be used.  That is, some functions require a character string input while others require a numeric input. Similarly, vectors of different types or properties may not play well together. Let's look at some examples:

```{r eval = F}
# taking the mean of a character vector
mean(chr_var)

# adding two numeric vectors of different lengths
vec1 <- c(1, 2, 3, 4)
vec2 <- c(2, 3, 5)
vec1 + vec2
```

### 2-dimensional data

A collection of vectors represented as a single data object are often described as two-dimensional data.  A more common way of storing two-dimensional data is in a data frame (i.e., `data.frame`).  Think of them like your standard spreadsheet, where each column describes a variable and rows link observations between columns.  Here's a simple example:

```{r}
ltrs <- c('a', 'b', 'c')
nums <- c(1, 2, 3)
logs <- c(T, F, T)
mydf <- data.frame(ltrs, nums, logs)
mydf
```

The only constraints required to make a data frame are:

1. Each column contains the same type of data

1. The number of observations in each column is equal.

## Getting your data into R

It is the rare case when you manually enter your data in R, not to mention impractical for most datasets.  Most data analysis workflows typically begin with importing a dataset from an external source.  Literally, this means committing a dataset to memory (i.e., storing it as a variable) as one of R's data structure formats.

Flat data files (text only, rectangular format) present the least complications on import because there is very little to assume about the structure of the data. On import, R tries to guess the data type for each column and this is fairly unambiguous with flat files.  The base installation of R comes with some easy to use functions for importing flat files, such as `read.table()` and `read.csv()`.

### Exercise 3

Now that we have the data downloaded and extracted to our data folder, we'll use `read.csv` to import two files.

1. Type the following in your script.  Note the use of *relative* file paths within your project.

    ```{r}
cscidat <- read.csv('data/cscidat.csv', stringsAsFactors = F)
ascidat <- read.csv('data/ascidat.csv', stringsAsFactors = F)
    ```

1. Send the commands to the console with `ctrl+enter`.

1. Verify that the data imported correctly by viewing the first six rows of each dataset.  Use the `head()` function directly in the console, e.g., `head(cscidat)`

Let's explore the datasets a bit.  There are many useful functions for exploring the characteristics of a dataset.  This is always a good idea when you first import something.  

```{r}
# get the dimensions
dim(cscidat)
dim(ascidat)

# get the column names
names(cscidat)
names(ascidat)

# see the first six rows
head(cscidat)
head(ascidat)

# get the overall structure
str(cscidat)
str(ascidat)
```

You can also view each dataset in a spreadsheet style in the scripting window:

```{r, eval = F}
View(cscidat)
View(ascidat)
```

### Other ways to import data

More often you will probably have an Excel spreadsheet to import. In the old days, importing spreadsheets into R was almost impossible given the proprietary data structure of Excel.  The tools available in R have since matured and it's now pretty painless to import a spreadsheet.  The `readxl` package is the most recent and by far most flexible data import package for Excel files. It comes with the `tidyverse` family of packages.

Once installed, we can load it to access the import functions.

```{r eval = F}
library(readxl)
dat <- read_excel('location/of/excel/file.xlsx')
```

## Summary

In this lesson we learned about R and Rstudio, some of the basic syntax and data structures in R, and how to import files.  We've just imported some provisional data for the California Stream Condition Index (CSCI) and the Algal Stream Condition Index (ASCI) that we'll continue to use for the rest of the workshop.  These data represent a portion of the sampling sites that were used to develop each index.  Next we'll learn how to process and plot these data to gain insight into bioassessment patterns throughout the state.