05-dplyr.Rmd

---
title: "Dataframe Manipulation with dplyr"
teaching: 40
exercises: 15
questions:
- "How can I manipulate dataframes without repeating myself?"
objectives:
- "Display basic properties of data frames including size and class of the columns, names, and first few rows."
- " To be able to use the six main dataframe manipulation 'verbs' with pipes in  `dplyr`."
- " To understand how `group_by()` and `summarize()` can be combined to summarize datasets."
- " Be able to analyze a subset of data using logical filtering."
keypoints:
- "Use `str()`, `summary()`, `nrow()`, `ncol()`, `dim()`, `colnames()`, `rownames()`, `head()`, and `typeof()` to understand the structure of a data frame."
- "Read in a csv file using `readr::read_csv()`."
- "Use the `dplyr` package to manipulate dataframes."
- "Use `select()` to choose variables from a dataframe."
- "Use `filter()` to choose data based on values."
- "Use `group_by()` and `summarize()` to work with subsets of data."
- "Use `mutate()` to create new variables."
source: Rmd
---

# Manipulating Data Frames with dplyr

```{r, include=FALSE}
source("bin/chunk-options.R")
knitr_fig_path("05-")
```

At this point, you've seen it all: in the last lesson, we toured all the basic data types and data structures in R. Everything you do will be a manipulation of those tools. But most of the time, the star of the show is the data frame—the table that we created by loading information from a csv file. In this lesson, we'll learn a few more things about working with data frames including how you might summarize a dataframe by a grouping variable, including only certain observations in your summary.  We will do this with the popular package `dplyr`.

```{r}
library(readr)
cats <- read_csv("data/feline-data.csv")
```

We can find the size and various other attributes of our data with a few handy functions:

```{r}
nrow(cats)
ncol(cats)
dim(cats)
```

```{r}
names(cats)
str(cats)
```

A way to get a nice overall view of a data frame is to use the `summary` function. This function can be used on various objects in R. For data frames, `summary` yields a numeric, tabular, or descriptive summary of each column. Factor columns are summarized by the number of items in each level, numeric or integer columns by the descriptive statistics (quartiles and mean), and character columns by its length, class, and mode.  

```{r}
summary(cats)
```

A really nice feature of RStudio is that it allows us to view our data in a 
very readable and familiar way. Go ahead and click on the name of the `cats`
data frame in the environment pane.

Note that this is a read-only view, it's not editable (which is a good thing!)

## Introducing the Gapminder data

So far, you have seen the basics of manipulating data frames with our cat data;
now let's use those skills to digest a more realistic dataset. Let's read in the
`gapminder` dataset that we downloaded previously and assign it to a data frame called `gapminder`:

```{r}
gapminder <- read_csv("data/gapminder_data.csv")
```


### Challenge 1 (5 minutes)
>
> How many rows and columns does `gapminder` have?
>
> What kind of data is in the `pop` and `continent` columns?
>
> <details>
> 
> <summary>
> Solution to challenge 1
> </summary>
> 
> <br />
> 
> ```{r}
> dim(gapminder)
> nrow(gapminder)
> ncol(gapminder)
> str(gapminder)
> summary(gapminder)
> ```
> </details>

The first thing we should always do is check out what the data looks like with
`str()` or `summary()`:

```{r}
str(gapminder)
```


```{r}
summary(gapminder$country)
```

::: {.rmdtip}

**Miscellaneous Tips**

* Files can also be downloaded directly from the Internet into a local
folder of your choice onto your computer using the `download.file` function.
The `read_csv` function can then be executed to read the downloaded file from the download location, for example,
```{r eval=FALSE, echo=TRUE}
download.file("https://raw.githubusercontent.com/swcarpentry/r-novice-gapminder/gh-pages/_episodes_rmd/data/gapminder_data.csv", destfile = "data/gapminder_data.csv")
gapminder <- read_csv("data/gapminder_data.csv")
```

* Alternatively, you can also read in files directly into R from the Internet by replacing the file paths with a web address in `read_csv`. One should note that in doing this no local copy of the csv file is first saved onto your computer. For example,
```{r eval=FALSE, echo=TRUE}
gapminder <- read_csv("https://raw.githubusercontent.com/swcarpentry/r-novice-gapminder/gh-pages/_episodes_rmd/data/gapminder_data.csv")
```

* You can read directly from excel spreadsheets without
converting them to plain text first by using the [readxl](https://cran.r-project.org/package=readxl) package.

:::


We'll also likely want to know what the titles of all the columns are, so we can
ask for them:

```{r}
names(gapminder)
```

At this stage, it's important to ask ourselves if the structure R is reporting
matches our intuition or expectations; do the basic data types reported for each
column make sense? If not, we need to sort any problems out now before they turn
into bad surprises down the road, using what we've learned about how R
interprets data, and the importance of *strict consistency* in how we record our
data.

Once we're happy that the data types and structures seem reasonable, it's time
to start digging into our data proper. Check out the first few lines:

```{r}
head(gapminder)
```

Another very helpful function for looking at your data is the `unique`()
function, to see the unique values in a particular column:

```{r}
unique(gapminder$continent)
```

Manipulation of dataframes means many things to many researchers, we often
select certain observations (rows) or variables (columns), we often group the
data by a certain variable(s), or we even calculate summary statistics. We can
do these types of operations using the normal base R operations:

```{r}
mean(gapminder$gdpPercap)
```

The question here is how might you summarize and 
split by some group you might be interested in.

## The `dplyr` package

Luckily, the [`dplyr`](https://cran.r-project.org/package=dplyr)
package provides a number of very useful functions for manipulating dataframes
in a way that will reduce repetition, reduce the probability of making
errors, and probably even save you some typing. As an added bonus, you might
even find the `dplyr` grammar easier to read.

Here we're going to cover 5 of the most commonly used functions as well as using
pipes (`%>%`) to combine them.

1. `select()`
2. `filter()`
3. `group_by()`
4. `summarize()`
5. `mutate()`

If you have have not installed this package earlier, please do so:

```{r,eval=FALSE}
install.packages('dplyr')
```

Now let's load the package:

```{r,message=FALSE}
library("dplyr")
```

## Using select()

If, for example, we wanted to move forward with only a few of the variables in
our dataframe we could use the `select()` function. This will keep only the
variables you select.

```{r}
year_country_gdp <- select(gapminder, year, country, gdpPercap)
```

![](fig/07-dplyr-fig1.png){width=100%}

## Using filter()

To select on some subset of rows:
```{r}
filter(gapminder, continent == "Europe")
```

Or select another subset of rows:

```{r}
filter(gapminder, pop > 1000000)
```


Above we used 'normal' grammar, but the strengths of `dplyr` lie in combining 
several functions using pipes. This 
[description](https://twitter.com/WeAreRLadies/status/1172576445794803713) is very useful:

```{r, eval=FALSE}
I woke up %>% 
  showered %>% 
  dressed %>% 
  had coffee %>% 
  came to an R workshop
```

Since the pipes grammar is unlike anything we've seen in R before, 
let's repeat what we've done above using pipes.

```{r}
year_country_gdp <- gapminder %>% 
  select(year, country, gdpPercap)
```

To help you understand why we wrote that in that way, let's walk through it step
by step. First we summon the gapminder dataframe and pass it on, using the pipe
symbol `%>%`, to the next step, which is the `select()` function. In this case
we don't specify which data object we use in the `select()` function since in
gets that from the previous pipe. **Fun Fact**: There is a good chance you have
encountered pipes before in the shell. In R, a pipe symbol is `%>%` while in the
shell it is `|` but the concept is the same! 

## Using filter() with pipes

If we now wanted to move forward with the above, but only with European
countries, we can combine `select` and `filter`

```{r}
year_country_gdp_euro <- gapminder %>%
    filter(continent == "Europe") %>%
    select(year, country, gdpPercap)
```

### Challenge 2 (10 minutes)
> Write a single command (which can span multiple lines and includes pipes) that
> will produce a dataframe that has the African values for `lifeExp`, `country`
> and `year`, but not for other Continents.  How many rows does your dataframe
> have and why?
>
> <details>
> 
> <summary>
> Solution to challenge 2
> </summary>
> 
> <br />
> ```{r}
> year_country_lifeExp_Africa <- gapminder %>%
>                            filter(continent == "Africa") %>%
>                            select(year, country, lifeExp)
> ```
> </details>


As with last time, first we pass the gapminder dataframe to the `filter()`
function, then we pass the filtered version of the gapminder dataframe to the
`select()` function. **Note:** The order of operations is very important in this
case. If we used 'select' first, filter would not be able to find the variable
continent since we would have removed it in the previous step.

## Using group_by() and summarize()

Now, we were supposed to be reducing the error prone repetitiveness of what can
be done with base R, but up to now we haven't done that since we would have to
repeat the above for each continent. Instead of `filter()`, which will only pass
observations that meet your criteria (in the above: `continent=="Europe"`), we
can use `group_by()`, which will essentially use every unique criteria that you
could have used in filter.

```{r}
gapminder

gapminder %>% group_by(continent)
```
You will notice that the structure of the dataframe where we used `group_by()`
(`grouped_df`) is not the same as the original `gapminder` (`data.frame`). A
`grouped_df` can be thought of as a `list` where each item in the `list` is a
`data.frame` which contains only the rows that correspond to the a particular
value `continent` (at least in the example above).

![](fig/07-dplyr-fig2.png){width=100%}

## Using summarize()

The above was a bit on the uneventful side but `group_by()` is much more
exciting in conjunction with `summarize()`. This will allow us to create new
variable(s) by using functions that repeat for each of the continent-specific
data frames. That is to say, using the `group_by()` function, we split our
original dataframe into multiple pieces, then we can run functions
(e.g. `mean()` or `sd()`) within `summarize()`.

```{r}
gdp_bycontinents <- gapminder %>%
    group_by(continent) %>%
    summarize(mean_gdpPercap = mean(gdpPercap))
```

![](fig/07-dplyr-fig3.png){width=100%}

```{r,echo=FALSE}
gdp_bycontinents
```

That allowed us to calculate the mean gdpPercap for each continent, but it gets
even better.

### Challenge 3 (10 minutes)
> Calculate the average life expectancy per country. Which has the longest average life
> expectancy and which has the shortest average life expectancy?
>
> <details>
> 
> <summary>
> Solution to challenge 3
> </summary>
> 
> <br />
>```{r}
>lifeExp_bycountry <- gapminder %>%
>    group_by(country) %>%
>    summarize(mean_lifeExp = mean(lifeExp))
>lifeExp_bycountry %>%
>    filter(mean_lifeExp == min(mean_lifeExp) | mean_lifeExp == max(mean_lifeExp))
>```
Another way to do this is to use the `dplyr` function `arrange()`, which
arranges the rows in a data frame according to the order of one or more
variables from the data frame.  It has similar syntax to other functions from
the `dplyr` package. You can use `desc()` inside `arrange()` to sort in
descending order.
>```{r}
>lifeExp_bycountry %>%
>    arrange(mean_lifeExp) %>%
>    head(1)
>lifeExp_bycountry %>%
>    arrange(desc(mean_lifeExp)) %>%
>    head(1)
>```
> </details>


The function `group_by()` allows us to group by multiple variables. Let's group by `year` and `continent`.


```{r}
gdp_bycontinents_byyear <- gapminder %>%
    group_by(continent, year) %>%
    summarize(mean_gdpPercap = mean(gdpPercap))
```

That is already quite powerful, but it gets even better! You're not limited to defining 1 new variable in `summarize()`.

```{r}
gdp_pop_bycontinents_byyear <- gapminder %>%
    group_by(continent, year) %>%
    summarize(mean_gdpPercap = mean(gdpPercap),
              sd_gdpPercap = sd(gdpPercap),
              mean_pop = mean(pop),
              sd_pop = sd(pop))
```

## Using count()

A very common operation is to count the number of observations for each
group. The `dplyr` package comes with a function that helps with this.

For instance, if we wanted to check the number of countries included in the
dataset for the year 2002, we can use the `count()` function. It takes the name
of one or more columns that contain the groups we are interested in, and we can
optionally sort the results in descending order by adding `sort=TRUE`:

```{r}
gapminder %>%
    filter(year == 2002) %>%
    count(continent, sort = TRUE)
```


## Using mutate()

We can also create new variables prior to (or even after) summarizing information using `mutate()`.

```{r}
gdp_pop_bycontinents_byyear <- gapminder %>%
    mutate(gdp_billion = gdpPercap*pop/10^9) %>%
    group_by(continent,year) %>%
    summarize(mean_gdpPercap = mean(gdpPercap),
              sd_gdpPercap = sd(gdpPercap),
              mean_pop = mean(pop),
              sd_pop = sd(pop),
              mean_gdp_billion = mean(gdp_billion),
              sd_gdp_billion = sd(gdp_billion))
```

### Challenge 4 (10 minutes)
> Calculate the total GDP per contintent and year. Which continent, and in what year,
> had the highest total GDP?
>
> <details>
> 
> <summary>
> Solution to challenge 4
> </summary>
> 
> <br />
>```{r}
> gdp_by_continent_by_year <- gapminder %>% 
>     mutate(gdp = gdpPercap * pop) %>%
>     group_by(continent, year) %>%
>     summarise(total_gdp = sum(gdp)) %>% 
>     arrange(desc(total_gdp))
>```
> </details>

## Connect mutate with logical groupings: ifelse

When creating new variables, we can hook this with a logical condition. A simple combination of
`mutate()` and `ifelse()` facilitates filtering right where it is needed: in the moment of creating something new.
This easy-to-read statement is a fast and powerful way of discarding certain data (even though the overall dimension
of the data frame will not change) or for updating values depending on this given condition.

```{r}
## keeping all data but "filtering" after a certain condition
# calculate GDP only for people with a life expectation above 25
gdp_pop_bycontinents_byyear_above25 <- gapminder %>%
    mutate(gdp_billion = ifelse(lifeExp > 25, gdpPercap * pop / 10^9, NA)) %>%
    group_by(continent, year) %>%
    summarize(mean_gdpPercap = mean(gdpPercap),
              sd_gdpPercap = sd(gdpPercap),
              mean_pop = mean(pop),
              sd_pop = sd(pop),
              mean_gdp_billion = mean(gdp_billion),
              sd_gdp_billion = sd(gdp_billion))

## updating only if certain condition is fullfilled
# for life expectations above 40 years, the gpd to be expected in the future is scaled
gdp_future_bycontinents_byyear_high_lifeExp <- gapminder %>%
    mutate(gdp_futureExpectation = ifelse(lifeExp > 40, gdpPercap * 1.5, gdpPercap)) %>%
    group_by(continent, year) %>%
    summarize(mean_gdpPercap = mean(gdpPercap),
              mean_gdpPercap_expected = mean(gdp_futureExpectation))
```


## Other great resources

* [R for Data Science](http://r4ds.had.co.nz/)
* [Data Wrangling Cheat sheet](https://www.rstudio.com/wp-content/uploads/2015/02/data-wrangling-cheatsheet.pdf)
* [Introduction to dplyr](https://dplyr.tidyverse.org/)
* [Data wrangling with R and RStudio](https://www.rstudio.com/resources/webinars/data-wrangling-with-r-and-rstudio/)