Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggest/EDA.qmd merge two sections of unusual values #1661

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

Suggest/EDA.qmd merge two sections of unusual values #1661

Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
134d5e7
probably a typo
mitsuoxv May 29, 2024
319ac91
`y` denotes width, and xlim is an argument.
mitsuoxv May 29, 2024
61ecccc
make it clearer to be missing, and `y` is width.
mitsuoxv May 29, 2024
5418951
probably careless mistakes
mitsuoxv May 29, 2024
9bbf7f9
combine two sections of unsual values into one
mitsuoxv May 29, 2024
7711b0b
divide exercises
mitsuoxv May 29, 2024
5396bc1
probably careless mistakes
mitsuoxv May 30, 2024
86ac422
probably a careless mistake
mitsuoxv May 30, 2024
8d5ae7c
a typo, and some points are smaller than 1000.
mitsuoxv May 30, 2024
564f4b4
a careless mistake
mitsuoxv May 30, 2024
df07ab8
same notation of color and cut
mitsuoxv May 30, 2024
351590f
a typo
mitsuoxv May 30, 2024
1b4562d
a careless mistake, and "quite similar" contradicts with the sentence…
mitsuoxv May 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 41 additions & 43 deletions EDA.qmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ You can see variation easily in real life; if you measure any continuous variabl
This is true even if you measure quantities that are constant, like the speed of light.
Each of your measurements will include a small amount of error that varies from measurement to measurement.
Variables can also vary if you measure across different subjects (e.g., the eye colors of different people) or at different times (e.g., the energy levels of an electron at different moments).
Every variable has its own pattern of variation, which can reveal interesting information about how that it varies between measurements on the same observation as well as across observations.
Every variable has its own pattern of variation, which can reveal interesting information about how it varies between measurements on the same observation as well as across observations.
The best way to understand that pattern is to visualize the distribution of the variable's values, which you've learned about in @sec-data-visualization.

We'll start our exploration by visualizing the distribution of weights (`carat`) of \~54,000 diamonds from the `diamonds` dataset.
Expand Down Expand Up @@ -151,7 +151,21 @@ Some of these questions can be answered with the data while some will require do
Many of them will prompt you to explore a relationship *between* variables, for example, to see if the values of one variable can explain the behavior of another variable.
We'll get to that shortly.

### Unusual values
#### Exercises

1. Explore the distribution of each of the `x`, `y`, and `z` variables in `diamonds`.
What do you learn?
Think about a diamond and how you might decide which dimension is the length, width, and depth.

2. Explore the distribution of `price`.
Do you discover anything unusual or surprising?
(Hint: Carefully think about the `binwidth` and make sure you try a wide range of values.)

3. How many diamonds are 0.99 carat?
How many are 1 carat?
What do you think is the cause of the difference?

### Unusual values {#sec-unusual-values-eda}

Outliers are observations that are unusual; data points that don't seem to fit the pattern.
Sometimes outliers are data entry errors, sometimes they are simply values at the extremes that happened to be observed in this data collection, and other times they suggest important new discoveries.
Expand All @@ -161,7 +175,7 @@ The only evidence of outliers is the unusually wide limits on the x-axis.

```{r}
#| fig-alt: |
#| A histogram of lengths of diamonds. The x-axis ranges from 0 to 60 and
#| A histogram of widths of diamonds. The x-axis ranges from 0 to 60 and
#| the y-axis ranges from 0 to 12000. There is a peak around 5, and the
#| data appear to be completely clustered around the peak.

Expand All @@ -174,7 +188,7 @@ To make it easy to see the unusual values, we need to zoom to small values of th

```{r}
#| fig-alt: |
#| A histogram of lengths of diamonds. The x-axis ranges from 0 to 60 and the
#| A histogram of widths of diamonds. The x-axis ranges from 0 to 60 and the
#| y-axis ranges from 0 to 50. There is a peak around 5, and the data
#| appear to be completely clustered around the peak. Other than those data,
#| there is one bin at 0 with a height of about 8, one a little over 30 with
Expand All @@ -185,7 +199,7 @@ ggplot(diamonds, aes(x = y)) +
coord_cartesian(ylim = c(0, 50))
```

`coord_cartesian()` also has an `xlim()` argument for when you need to zoom into the x-axis.
`coord_cartesian()` also has an `xlim` argument for when you need to zoom into the x-axis.
ggplot2 also has `xlim()` and `ylim()` functions that work slightly differently: they throw away the data outside the limits.

This allows us to see that there are three unusual values: 0, \~30, and \~60.
Expand All @@ -212,36 +226,16 @@ options(old)
```

The `y` variable measures one of the three dimensions of these diamonds, in mm.
We know that diamonds can't have a width of 0mm, so these values must be incorrect.
We know that diamonds can't have a width of 0mm, so these must be missing values.
By doing EDA, we have discovered missing data that was coded as 0, which we never would have found by simply searching for `NA`s.
Going forward we might choose to re-code these values as `NA`s in order to prevent misleading calculations.
We might also suspect that measurements of 32mm and 59mm are implausible: those diamonds are over an inch long, but don't cost hundreds of thousands of dollars!
We might also suspect that measurements of 32mm and 59mm are implausible: those diamonds are over an inch wide, but don't cost hundreds of thousands of dollars!

It's good practice to repeat your analysis with and without the outliers.
If they have minimal effect on the results, and you can't figure out why they're there, it's reasonable to omit them, and move on.
However, if they have a substantial effect on your results, you shouldn't drop them without justification.
You'll need to figure out what caused them (e.g., a data entry error) and disclose that you removed them in your write-up.

### Exercises

1. Explore the distribution of each of the `x`, `y`, and `z` variables in `diamonds`.
What do you learn?
Think about a diamond and how you might decide which dimension is the length, width, and depth.

2. Explore the distribution of `price`.
Do you discover anything unusual or surprising?
(Hint: Carefully think about the `binwidth` and make sure you try a wide range of values.)

3. How many diamonds are 0.99 carat?
How many are 1 carat?
What do you think is the cause of the difference?

4. Compare and contrast `coord_cartesian()` vs. `xlim()` or `ylim()` when zooming in on a histogram.
What happens if you leave `binwidth` unset?
What happens if you try and zoom so only half a bar shows?

## Unusual values {#sec-unusual-values-eda}

If you've encountered unusual values in your dataset, and simply want to move on to the rest of your analysis, you have two options.

1. Drop the entire row with the strange values:
Expand Down Expand Up @@ -299,7 +293,7 @@ You can do this by making a new variable, using `is.na()` to check if `dep_time`
#| fig-alt: |
#| A frequency polygon of scheduled departure times of flights. Two lines
#| represent flights that are cancelled and not cancelled. The x-axis ranges
#| from 0 to 25 minutes and the y-axis ranges from 0 to 10000. The number of
#| from 0 to 25 hours and the y-axis ranges from 0 to 10000. The number of
#| flights not cancelled are much higher than those cancelled.

nycflights13::flights |>
Expand All @@ -316,17 +310,21 @@ nycflights13::flights |>
However this plot isn't great because there are many more non-cancelled flights than cancelled flights.
In the next section we'll explore some techniques for improving this comparison.

### Exercises
#### Exercises

1. Compare and contrast `coord_cartesian()` vs. `xlim()` or `ylim()` when zooming in on a histogram.
What happens if you leave `binwidth` unset?
What happens if you try and zoom so only half a bar shows?

1. What happens to missing values in a histogram?
2. What happens to missing values in a histogram?
What happens to missing values in a bar chart?
Why is there a difference in how missing values are handled in histograms and bar charts?

2. What does `na.rm = TRUE` do in `mean()` and `sum()`?
3. What does `na.rm = TRUE` do in `mean()` and `sum()`?

3. Recreate the frequency plot of `scheduled_dep_time` colored by whether the flight was cancelled or not.
4. Recreate the frequency plot of `scheduled_dep_time` colored by whether the flight was cancelled or not.
Also facet by the `cancelled` variable.
Experiment with different values of the `scales` variable in the faceting function to mitigate the effect of more non-cancelled flights than cancelled flights.
Experiment with different values of the `scales` argument in the faceting function to mitigate the effect of more non-cancelled flights than cancelled flights.

## Covariation

Expand All @@ -340,9 +338,9 @@ For example, let's explore how the price of a diamond varies with its quality (m

```{r}
#| fig-alt: |
#| A frequency polygon of prices of diamonds where each cut of carat (Fair,
#| A frequency polygon of prices of diamonds where each cut of diamond (Fair,
#| Good, Very Good, Premium, and Ideal) is represented with a different color
#| line. The x-axis ranges from 0 to 30000 and the y-axis ranges from 0 to
#| line. The x-axis ranges from 0 to 20000 and the y-axis ranges from 0 to
#| 5000. The lines overlap a great deal, suggesting similar frequency
#| distributions of prices of diamonds. One notable feature is that
#| Ideal diamonds have the highest peak around 1500.
Expand All @@ -362,7 +360,7 @@ Instead of displaying count, we'll display the **density**, which is the count s
```{r}
#| fig-alt: |
#| A frequency polygon of densities of prices of diamonds where each cut of
#| carat (Fair, Good, Very Good, Premium, and Ideal) is represented with a
#| diamond (Fair, Good, Very Good, Premium, and Ideal) is represented with a
#| different color line. The x-axis ranges from 0 to 20000. The lines overlap
#| a great deal, suggesting similar density distributions of prices of
#| diamonds. One notable feature is that all but Fair diamonds have high peaks
Expand Down Expand Up @@ -470,9 +468,9 @@ One way to do that is to rely on the built-in `geom_count()`:
#| fig-alt: |
#| A scatterplot of color vs. cut of diamonds. There is one point for each
#| combination of levels of cut (Fair, Good, Very Good, Premium, and Ideal)
#| and color (D, E, F, G, G, I, and J). The sizes of the points represent
#| and color (D, E, F, G, H, I, and J). The sizes of the points represent
#| the number of observations for that combination. The legend indicates
#| that these sizes range between 1000 and 4000.
#| that these sizes mainly range between 1000 and 4000.

ggplot(diamonds, aes(x = cut, y = color)) +
geom_count()
Expand All @@ -496,7 +494,7 @@ Then visualize with `geom_tile()` and the fill aesthetic:
#| cut/color combination and tiles are colored according to the number of
#| observations in each tile. There are more Ideal diamonds than other cuts,
#| with the highest number being Ideal diamonds with color G. Fair diamonds
#| and diamonds with color I are the lowest in frequency.
#| and diamonds with color J are the lowest in frequency.

diamonds |>
count(color, cut) |>
Expand All @@ -511,7 +509,7 @@ For larger plots, you might want to try the heatmaply package, which creates int

1. How could you rescale the count dataset above to more clearly show the distribution of cut within color, or color within cut?

2. What different data insights do you get with a segmented bar chart if color is mapped to the `x` aesthetic and `cut` is mapped to the `fill` aesthetic?
2. What different data insights do you get with a segmented bar chart if `color` is mapped to the `x` aesthetic and `cut` is mapped to the `fill` aesthetic?
Calculate the counts that fall into each of the segments.

3. Use `geom_tile()` together with dplyr to explore how average flight departure delays vary by destination and month of year.
Expand Down Expand Up @@ -545,7 +543,7 @@ You've already seen one way to fix the problem: using the `alpha` aesthetic to a
#| fig-alt: |
#| A scatterplot of price vs. carat. The relationship is positive, somewhat
#| strong, and exponential. The points are transparent, showing clusters where
#| the number of points is higher than other areas, The most obvious clusters
#| the number of points is higher than other areas. The most obvious clusters
#| are for diamonds with 1, 1.5, and 2 carats.

ggplot(smaller, aes(x = carat, y = price)) +
Expand Down Expand Up @@ -696,8 +694,8 @@ Once you've removed the strong relationship between carat and price, you can see
```{r}
#| fig-alt: |
#| Side-by-side box plots of residuals by cut. The x-axis displays the various
#| cuts (Fair to Ideal), the y-axis ranges from 0 to almost 5. The medians are
#| quite similar, between roughly 0.75 to 1.25. Each of the distributions of
#| cuts (Fair to Ideal), the y-axis ranges from 0 to almost 4. The medians are
#| between roughly 0.75 to 1.25, and increase in better cuts. Each of the distributions of
#| residuals is right skewed, with many outliers on the higher end.

ggplot(diamonds_aug, aes(x = cut, y = .resid)) +
Expand Down
Loading