evaluating-prediction-accuracy.qmd

# Evaluation of Prediction/Forecasting Accuracy {#sec-predictionAccuracy}

> "Nothing ruins fantasy like reality." – Renee Miller

## Getting Started {#sec-predictionAccuracyGettingStarted}

### Load Packages {#sec-predictionAccuracyLoadPackages}

```{r}
library("petersenlab")
library("tidyverse")
library("pROC")
library("magrittr")
library("viridis")
```

## Overview {#sec-predictionAccuracyOverview}

Predictions can come in different types.
Some predictions involve categorical data, whereas other predictions involve continuous data.
When dealing with a dichotomous ([nominal data](#sec-nominal) that are binary) [predictor](#sec-correlationalStudy) and [outcome](#sec-correlationalStudy) variable (or continuous data that have been dichotomized using a cutoff), we can evaluate predictions using a 2x2 table known as a confusion matrix (see INSERT), or with logistic regression models.
When dealing with a continuous [outcome variable](#sec-correlationalStudy) (e.g., [ordinal](#sec-ordinal), [interval](#sec-interval), or [ratio](#sec-ratio) data), we can evaluate predictions using [multiple regression](#sec-multipleRegression) or similar variants such as structural equation modeling and mixed models.

In fantasy football, we most commonly predict continuous [outcome variables](#sec-correlationalStudy) (e.g., fantasy points, rushing yards).
Nevertheless, it is also important to understand principles in the prediction of categorical [outcomes variables](#sec-correlationalStudy).

In any domain, it is important to evaluate the accuracy of predictions, so we can know how (in)accurate we are, and we can strive to continually improve our predictions.
Fantasy performance—and human behavior more general—is incredibly challenging to predict.
Indeed, many things in the world, in particular long-term trends, are unpredictable [@Kahneman2011].
In fantasy football, there is considerable luck/chance/randomness.
There are relatively few (i.e. 17) games, and there is a sizeable injury risk for each player in a given game.
These and other factors combine to render fantasy football predictions not highly accurate.
Domains with high uncertainty and unpredictability are considered "low-validity environments" [@Kahneman2011, p. 223].
But, first, let's learn about the various ways we can evaluate the accuracy of predictions.

## Types of Accuracy {#sec-accuracyTypes}

There are two primary dimensions of accuracy: (1) [discrimination](#sec-discrimination) and (2) [calibration](#sec-calibration).
[Discrimination](#sec-discrimination) and [calibration](#sec-calibration) are distinct forms of accuracy.
Just because predictions are high in one form of accuracy does not mean that they will be high in the other form of accuracy.
As described by @Lindhiem2020, predictions can follow any of the following configurations (and anywhere in between):

- high [discrimination](#sec-discrimination), high [calibration](#sec-calibration)
- high [discrimination](#sec-discrimination), low [calibration](#sec-calibration)
- low [discrimination](#sec-discrimination), high [calibration](#sec-calibration)
- low [discrimination](#sec-discrimination), low [calibration](#sec-calibration)

Some general indexes of accuracy combine discrimination and calibration, as described in @sec-generalAccuracy.

In addition, accuracy indices can be [threshold-dependent](#sec-thresholdDependentAccuracy) or [-independent](#sec-thresholdIndependentAccuracy) and can be scale-dependent or -independent.
[Threshold-dependent accuracy indices](#sec-thresholdDependentAccuracy) differ based on the cutoff (i.e., threshold), whereas [threshold-independent accuracy indices](#sec-thresholdIndependentAccuracy) do not.
Thus, raising or lowering the cutoff will change [threshold-dependent](#sec-thresholdDependentAccuracy) accuracy indices.
Scale-dependent accuracy indices depend on the metric/scale of the data, whereas scale-independent accuracy indices do not.
Thus, scale-dependent accuracy indices cannot be directly compared when using measures of differing scales, whereas scale-independent accuracy indices can be compared across data of differing scales.

### Discrimination {#sec-discrimination}

When dealing with a categorical outcome, discrimination is the ability to separate events from non-events.
When dealing with a continuous outcome, discrimination is the strength of the association between the predictor and the outcome.
Aspects of discrimination at a particular cutoff (e.g., sensitivity, specificity, area under the ROC curve) are described in INSERT.

### Calibration {#sec-calibration}

When dealing with a categorical outcome, calibration is the degree to which a probabilistic estimate of an event reflects the true underlying probability of the event.
When dealing with a continuous outcome, calibration is the degree to which the predicted values are close in value to the outcome values.
The importance of examining calibration (in addition to discrimination) is described by @Lindhiem2020.

Calibration is relevant to all kinds of predictions, including weather forecasts.
For instance, on the days that the meteorologist says there is a 60% chance of rain, it should rain about 60% of the time.
Calibration is also important for fantasy football predictions.
When projections state that a group of players is each expected to score 200 points, their projections would be miscalibrated if those players scored only 150 points on average.

There are four general patterns of miscalibration: overextremity, underextremity, overprediction, and underprediction (see @fig-miscalibration).
*Overextremity* exists when the predicted probabilites are too close to the extremes (zero or one).
*Underextremity* exists when the predicted probabilities are too far away from the extremes.
*Overprediction* exists when the predicted probabilities are consistently greater than the observed probabilities.
*Underprediction* exists when the predicted probabilities are consistently less than the observed probabilities.
For a more thorough description of these types of miscalibration, see @Lindhiem2020.

Indices for evaluating calibration are described in @sec-calibrationIndices.

### General Accuracy {#sec-generalAccuracy}

General accuracy indices combine estimates of [discrimination](#sec-discrimination) and [calibration](#sec-calibration).

## Prediction of Categorical Outcomes {#sec-predictionCategorical}

To evaluate the accuracy of our predictions for categorical outcome variables (e.g., binary, dichotomous, or [nominal](#sec-nominal) data), we can use either [threshold-dependent](#sec-thresholdDependentAccuracy) or [threshold-independent](#sec-thresholdIndependentAccuracy) accuracy indices.

## Prediction of Continuous Outcomes {#sec-predictionContinuous}

To evaluate the accuracy of our predictions for continuous outcome variables (e.g., [ordinal](#sec-ordinal), [interval](#sec-interval), or [ratio](#sec-ratio) data), the outcome variable does not have cutoffs, so we would use [threshold-independent accuracy indices](#sec-thresholdIndependentAccuracy).

## Threshold-Dependent Accuracy Indices {#sec-thresholdDependentAccuracy}

### Decision Outcomes {#sec-decisionOutcomes}

To consider how we can evaluate the accuracy of predictions for a categorical outcome, consider an example adapted from @Meehl1955.
The military conducts a test of its prospective members to screen out applicants who would likely fail basic training.
To evaluate the accuracy of our predictions using the test, we can examine a confusion matrix.
A confusion matrix is a matrix that presents the predicted outcome on one dimension and the actual outcome (truth) on the other dimension.
If the predictions and outcomes are dichotomous, the confusion matrix is a 2x2 matrix with two rows and two columns that represent four possible predicted-actual combinations (decision outcomes): true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN).

When discussing the four decision outcomes, "true" means an accurate judgment, whereas "false" means an inaccurate judgment.
"Positive" means that the judgment was that the person has the characteristic of interest, whereas "negative" means that the judgment was that the person does not have the characteristic of interest.
A *true positive* is a correct judgment (or prediction) where the judgment was that the person has (or will have) the characteristic of interest, and, in truth, they actually have (or will have) the characteristic.
A *true negative* is a correct judgment (or prediction) where the judgment was that the person does not have (or will not have) the characteristic of interest, and, in truth, they actually do not have (or will not have) the characteristic.
A *false positive* is an incorrect judgment (or prediction) where the judgment was that the person has (or will have) the characteristic of interest, and, in truth, they actually do not have (or will not have) the characteristic.
A *false negative* is an incorrect judgment (or prediction) where the judgment was that the person does not have (or will not have) the characteristic of interest, and, in truth, they actually do have (or will have) the characteristic.

An example of a confusion matrix is in INSERT.

With the information in the confusion matrix, we can calculate the marginal sums and the proportion of people in each cell (in parentheses), as depicted in INSERT.

That is, we can sum across the rows and columns to identify how many people actually showed poor adjustment ($n = 100$) versus good adjustment ($n = 1,900$), and how many people were selected to reject ($n = 508$) versus retain ($n = 1,492$).
If we sum the column of predicted marginal sums ($508 + 1,492$) or the row of actual marginal sums ($100 + 1,900$), we get the total number of people ($N = 2,000$).

Based on the marginal sums, we can compute the [marginal probabilities](#sec-baseRate), as depicted in INSERT.

The [marginal probability](#sec-baseRate) of the person having the characteristic of interest (i.e., showing poor adjustment) is called the [*base rate*](#baseRate) (BR).
That is, the [base rate](#baseRate) is the proportion of people who have the characteristic.
It is calculated by dividing the number of people with poor adjustment ($n = 100$) by the total number of people ($N = 2,000$): $BR = \frac{FN + TP}{N}$.
Here, the [base rate](#baseRate) reflects the prevalence of poor adjustment.
In this case, the [base rate](#baseRate) is .05, so there is a 5% chance that an applicant will be poorly adjusted.
The [marginal probability](#baseRate) of good adjustment is equal to 1 minus the [base rate](#baseRate) of poor adjustment.

The [marginal probability](#baseRate) of predicting that a person has the characteristic (i.e., rejecting a person) is called the *selection ratio* (SR).
The selection ratio is the proportion of people who will be selected (in this case, rejected rather than retained); i.e., the proportion of people who are identified as having the characteristic.
The selection ratio is calculated by dividing the number of people selected to reject ($n = 508$) by the total number of people ($N = 2,000$): $SR = \frac{TP + FP}{N}$.
In this case, the selection ratio is .25, so 25% of people are rejected.
The [marginal probability](#baseRate) of not selecting someone to reject (i.e., the [marginal probability](#baseRate) of retaining) is equal to 1 minus the selection ratio.

The selection ratio might be something that the test dictates according to its cutoff score.
Or, the selection ratio might be imposed by external factors that place limits on how many people you can assign a positive test value.
For instance, when deciding whether to treat a client, the selection ratio may depend on how many therapists are available and how many cases can be treated.

### Percent Accuracy {#sec-percentAccuracyOverview}

Based on the confusion matrix, we can calculate the prediction accuracy based on the percent accuracy of the predictions.
The percent accuracy is the number of correct predictions divided by the total number of predictions, and multiplied by 100.
In the context of a confusion matrix, this is calculated as: $100\% \times \frac{\text{TP} + \text{TN}}{N}$.
In this case, our percent accuracy was 78%—that is, 78% of our predictions were accurate, and 22% of our predictions were inaccurate.

### Percent Accuracy by Chance {#sec-accuracyByChance}

78% sounds pretty accurate.
And it is much higher than 50%, so we are doing a pretty good job, right?
Well, it is important to compare our accuracy to what accuracy we would expect to get by chance alone, if predictions were made by a random process rather than using a test's scores.
Our selection ratio was 25.4%.
How accurate would we be if we randomly selected 25.4% of people to reject?
To determine what accuracy we could get by chance alone given the selection ratio and the base rate, we can calculate the chance probability of true positives and the chance probability of true negatives.
The probability of a given cell in the confusion matrix is a [joint probability](#sec-jointProbability)—the probability of two events occurring simultaneously.
To calculate a [joint probability](#sec-jointProbability), we multiply the probability of each event.

So, to get the chance expectancies of true positives, we would multiply the respective [marginal probabilities](#baseRate), as in @eq-truePositivesByChanceExample:

$$
\begin{aligned}
  P(TP) &= P(\text{Poor adjustment}) \times P(\text{Reject})\\
   &= BR \times SR \\
  &= .05 \times .254 \\
  &= .0127
\end{aligned}
$$ {#eq-truePositivesByChanceExample}

To get the chance expectancies of true negatives, we would multiply the respective [marginal probabilities](#sec-baseRate), as in @eq-trueNegativesByChanceExample:

$$
\begin{aligned}
  P(TN) &= P(\text{Good adjustment}) \times P(\text{Retain})\\
   &= (1 - BR) \times (1 - SR) \\
  &= .95 \times .746 \\
  &= .7087
\end{aligned}
$$ {#eq-trueNegativesByChanceExample}

To get the percent accuracy by chance, we sum the chance expectancies for the correct predictions (TP and TN): $.0127 + .7087 = .7214$.
Thus, the percent accuracy you can get by chance alone is 72%.
This is because most of our predictions are to retain people, and the [base rate](#sec-baseRate) of poor adjustment is quite low (.05).
Our measure with 78% accuracy provides only a 6% increment in correct predictions.
Thus, you cannot judge how good your judgment or prediction is until you know how you would do by random chance.

The chance expectancies for each cell of the confusion matrix are in INSERT

### Predicting from the Base Rate {#sec-predictingFromBaseRate}

Now, let us consider how well you would do if you were to predict from the [base rate](#sec-baseRate).
Predicting from the [base rate](#sec-baseRate) is also called "betting from the [base rate](#sec-baseRate)", and it involves setting the selection ratio by taking advantage of the [base rate](#sec-baseRate) so that you go with the most likely outcome in every prediction.
Because the [base rate](#sec-baseRate) is quite low (.05), we could predict from the [base rate](#sec-baseRate) by selecting no one to reject (i.e., setting the selection ratio at zero).
Our percent accuracy by chance if we predict from the [base rate](#sec-baseRate) would be calculated by multiplying the [marginal probabilities](#sec-baseRate), as we did above, but with a new selection ratio, as in @eq-predictingFromBaseRateExample:

$$
\begin{aligned}
  P(TP) &= P(\text{Poor adjustment}) \times P(\text{Reject})\\
   &= BR \times SR \\
  &= .05 \times 0 \\
  &= 0 \\ \\
  P(TN) &= P(\text{Good adjustment}) \times P(\text{Retain})\\
   &= (1 - BR) \times (1 - SR) \\
  &= .95 \times 1 \\
  &= .95
\end{aligned}
$$ {#eq-predictingFromBaseRateExample}

We sum the chance expectancies for the correct predictions (TP and TN): $0 + .95 = .95$.
Thus, our percent accuracy by predicting from the [base rate](#sec-baseRate) is 95%.
This is damning to our measure because it is a much higher accuracy than the accuracy of our measure.
That is, we can be much more accurate than our measure simply by predicting from the [base rate](#sec-baseRate) and selecting no one to reject.

Going with the most likely outcome in every prediction (predicting from the [base rate](#sec-baseRate)) can be highly accurate (in terms of percent accuracy) as noted by @Meehl1955, especially when the [base rate](#sec-baseRate) is very low or very high.
This should serve as an important reminder that we need to compare the accuracy of our measures to the accuracy by (1) random chance and (2) predicting from the [base rate](#sec-baseRate).
There are several important implications of the impact of [base rates](#sec-baseRate) on prediction accuracy.
One implication is that using the same test in different settings with different [base rates](#sec-baseRate) will markedly change the accuracy of the test.
Oftentimes, using a test will actually *decrease* the predictive accuracy when the [base rate](#sec-baseRate) deviates greatly from .50.
But percent accuracy is not everything.
Percent accuracy treats different kinds of errors as if they are equally important.
However, the value we place on different kinds of errors may be different, as described next.

### Different Kinds of Errors Have Different Costs {#sec-differentErrorsDifferentCosts}

Some errors have a high cost, and some errors have a low cost.
Among the four decision outcomes, there are two types of errors: false positives and false negatives.
The extent to which false positives and false negatives are costly depends on the prediction problem.
So, even though you can often be most accurate by going with the [base rate](#sec-baseRate), it may be advantageous to use a screening instrument despite lower overall accuracy because of the huge difference in costs of false positives versus false negatives in some cases.

Consider the example of a screening instrument for HIV.
False positives would be cases where we said that someone is at high risk of HIV when they are not, whereas false negatives are cases where we said that someone is not at high risk when they actually are.
The costs of false positives include a shortage of blood, some follow-up testing, and potentially some anxiety, but that is about it.
The costs of false negatives may be people getting HIV.
In this case, the costs of false negatives greatly outweigh the costs of false positives, so we use a screening instrument to try to identify the cases at high risk for HIV because of the important consequences of failing to do so, even though using the screening instrument will lower our overall accuracy level.

Another example is when the Central Intelligence Agency (CIA) used a screen for protective typists during wartime to try to detect spies.
False positives would be cases where the CIA believes that a person is a spy when they are not, and the CIA does not hire them.
False negatives would be cases where the CIA believes that a person is not a spy when they actually are, and the CIA hires them.
In this case, a false positive would be fine, but a false negative would be really bad.

How you weigh the costs of different errors depends considerably on the domain and context.
Possible costs of false positives to society include: unnecessary and costly treatment with side effects and sending an innocent person to jail (despite our presumption of innocence in the United States criminal justice system that a person is innocent until proven guilty).
Possible costs of false negatives to society include: setting a guilty person free, failing to detect a bomb or tumor, and preventing someone from getting treatment who needs it.

The differential costs of different errors also depend on how much flexibility you have in the selection ratio in being able to set a stringent versus loose selection ratio.
Consider if there is a high cost of getting rid of people during the selection process.
For example, if you must hire 100 people and only 100 people apply for the position, you cannot lose people, so you need to hire even high-risk people.
However, if you do not need to hire many people, then you can hire more conservatively.

Any time the selection ratio differs from the [base rate](#sec-baseRate), you will make errors.
For example, if you reject 25% of applicants, and the [base rate](#sec-baseRate) of poor adjustment is 5%, then you are making errors of over-rejecting (false positives).
By contrast, if you reject 1% of applicants and the [base rate](#sec-baseRate) of poor adjustment is 5%, then you are making errors of under-rejecting or over-accepting (false negatives).

A low [base rate](#sec-baseRate) makes it harder to make predictions, and tends to lead to less accurate predictions.
For instance, it is very challenging to predict low [base rate](#sec-baseRate) behaviors, including suicide [@Kessler2020].
For this reason, it is likely much more challenging to predict touchdowns—which happen relatively less often—than it is to predict passing/rushing/receiving yards—which are more frequent and continuously distributed.

[EVALUATE EMPIRICALLY]

### Sensitivity, Specificity, PPV, and NPV {#sec-sensitivitySpecificityPPVnpv}

As described earlier, percent accuracy is not the only important aspect of accuracy.
Percent accuracy can be misleading because it is highly influenced by [base rates](#sec-baseRate).
You can have a high percent accuracy by predicting from the base rate and saying that no one has the condition (if the [base rate](#sec-baseRate) is low) or that everyone has the condition (if the [base rate](#sec-baseRate) is high).
Thus, it is also important to consider other aspects of accuracy, including sensitivity (SN), specificity (SP), positive predictive value (PPV), and negative predictive value (NPV).
We want our predictions to be sensitive to be able to detect the characteristic but also to be specific so that we classify only people actually with the characteristic as having the characteristic.

Let us return to the confusion matrix in INSERT.
If we know the frequency of each of the four predicted-actual combinations of the confusion matrix (TP, TN, FP, FN), we can calculate sensitivity, specificity, PPV, and NPV.

Sensitivity is the proportion of those with the characteristic ($\text{TP} + \text{FN}$) that we identified with our measure ($\text{TP}$): $\frac{\text{TP}}{\text{TP} + \text{FN}} = \frac{86}{86 + 14} = .86$.
Specificity is the proportion of those who do not have the characteristic ($\text{TN} + \text{FP}$) that we correctly classify as not having the characteristic ($\text{TN}$): $\frac{\text{TN}}{\text{TN} + \text{FP}} = \frac{1,478}{1,478 + 422} = .78$.
PPV is the proportion of those who we classify as having the characteristic ($\text{TP} + \text{FP}$) who actually have the characteristic ($\text{TP}$): $\frac{\text{TP}}{\text{TP} + \text{FP}} = \frac{86}{86 + 422} = .17$.
NPV is the proportion of those we classify as not having the characteristic ($\text{TN} + \text{FN}$) who actually do not have the characteristic ($\text{TN}$): $\frac{\text{TN}}{\text{TN} + \text{FN}} = \frac{1,478}{1,478 + 14} = .99$.

Sensitivity, specificity, PPV, and NPV are proportions, and their values therefore range from 0 to 1, where higher values reflect greater accuracy.
With sensitivity, specificity, PPV, and NPV, we have a good snapshot of how accurate the measure is at a given cutoff.
In our case, our measure is good at finding whom to reject (high sensitivity), but it is rejecting too many people who do not need to be rejected (lower PPV due to many FPs).
Most people whom we classify as having the characteristic do not actually have the characteristic.
However, the fact that we are over-rejecting could be okay depending on our goals, for instance, if we do not care about over-dropping (i.e., the PPV being low).

#### Some Accuracy Estimates Depend on the Cutoff {#sec-accuracyCutoff}

Sensitivity, specificity, PPV, and NPV differ based on the cutoff (i.e., threshold) for classification.
Consider the following example.
Aliens visit Earth, and they develop a test to determine whether a berry is edible or inedible.

@fig-classificationDistributions depicts the distributions of scores by berry type.
Note how there are clearly two distinct distributions.
However, the distributions overlap to some degree.
Thus, any cutoff will have at least some inaccurate classifications.
The extent of overlap of the distributions reflects the amount of measurement error of the measure with respect to the characteristic of interest.

```{r}
#| label: fig-classificationDistributions
#| fig-cap: "Distribution of Test Scores by Berry Type."
#| fig-alt: "Distribution of Test Scores by Berry Type."
#| code-fold: true

#No Cutoff
sampleSize <- 1000

edibleScores <- rnorm(sampleSize, 50, 15)
inedibleScores <- rnorm(sampleSize, 100, 15)

edibleData <- data.frame(
  score = c(
    edibleScores,
    inedibleScores),
  type = c(
    rep("edible", sampleSize),
    rep("inedible", sampleSize)))

cutoff <- 75

hist_edible <- density(
  edibleScores,
  from = 0,
  to = 150) %$% # exposition pipe magrittr::`%$%`
  data.frame(
    x = x,
    y = y) %>%
  mutate(area = x >= cutoff)

hist_edible$type[hist_edible$area == TRUE] <- "edible_FP"
hist_edible$type[hist_edible$area == FALSE] <- "edible_TN"

hist_inedible <- density(
  inedibleScores,
  from = 0,
  to = 150) %$% # exposition pipe magrittr::`%$%`
  data.frame(
    x = x,
    y = y) %>%
  mutate(area = x < cutoff)

hist_inedible$type[hist_inedible$area == TRUE] <- "inedible_FN"
hist_inedible$type[hist_inedible$area == FALSE] <- "inedible_TP"

density_data <- bind_rows(
  hist_edible,
  hist_inedible)

density_data$type <- factor(
  density_data$type,
  levels = c(
    "edible_TN",
    "inedible_TP",
    "edible_FP",
    "inedible_FN"))

ggplot(
  data = edibleData,
  aes(
    x = score,
    ymin = 0,
    fill = type)) +
  geom_density(alpha = .5) +
  scale_fill_manual(
    name = "Berry Type",
    values = c(
      viridis::viridis(2)[1],
      viridis::viridis(2)[2])) +
  scale_y_continuous(name = "Frequency") +
  theme_bw() +
  theme(
    axis.text.y = element_blank(),
    axis.ticks.y = element_blank())
```

@fig-classificationStandardCutoff depicts the distributions of scores by berry type with a cutoff.
The red line indicates the cutoff—the level above which berries are classified by the test as inedible.
There are errors on each side of the cutoff.
Below the cutoff, there are some false negatives (blue): inedible berries that are inaccurately classified as edible.
Above the cutoff, there are some false positives (green): edible berries that are inaccurately classified as inedible.
Costs of false negatives could include sickness or death from eating the inedible berries.
Costs of false positives could include taking longer to find food, finding insufficient food, and starvation.

```{r}
#| label: fig-classificationStandardCutoff
#| fig-cap: "Classifications Based on a Cutoff. Note that some true negatives and true positives are hidden behind the false positives and false negatives."
#| fig-alt: "Classifications Based on a Cutoff. Note that some true negatives and true positives are hidden behind the false positives and false negatives."
#| code-fold: true

#Standard Cutoff
ggplot(
  data = density_data,
  aes(
    x = x,
    ymin = 0,
    ymax = y,
    fill = type)) +
  geom_ribbon(alpha = 1) +
  scale_fill_manual(
    name = "Berry Type",
    values = c(
      viridis::viridis(4)[4],
      viridis::viridis(4)[1],
      viridis::viridis(4)[3],
      viridis::viridis(4)[2]),
    breaks = c("edible_TN","inedible_TP","edible_FP","inedible_FN"),
    labels = c("Edible: TN","Inedible: TP","Edible: FP","Inedible: FN")) +
  geom_line(aes(y = y)) +
  geom_vline(
    xintercept = cutoff,
    color = "red",
    linewidth = 2) +
  scale_x_continuous(name = "score") +
  scale_y_continuous(name = "Frequency") +
  theme_bw() +
  theme(
    axis.text.y = element_blank(),
    axis.ticks.y = element_blank())
```

Based on our assessment goals, we might use a different selection ratio by changing the cutoff.
@fig-classificationRaiseCutoff depicts the distributions of scores by berry type when we raise the cutoff.
There are now more false negatives (blue) and fewer false positives (green).
If we raise the cutoff (to be more conservative), the number of false negatives increases and the number of false positives decreases.
Consequently, as the cutoff increases, sensitivity and NPV decrease (because we have more false negatives), whereas specificity and PPV increase (because we have fewer false positives).
A higher cutoff could be optimal if the costs of false positives are considered greater than the costs of false negatives.
For instance, if the aliens cannot risk eating the inedible berries because the berries are fatal, and there are sufficient edible berries that can be found to feed the alien colony.

```{r}
#| label: fig-classificationRaiseCutoff
#| fig-cap: "Classifications Based on Raising the Cutoff. Note that some true negatives and true positives are hidden behind the false positives and false negatives."
#| fig-alt: "Classifications Based on Raising the Cutoff. Note that some true negatives and true positives are hidden behind the false positives and false negatives."
#| code-fold: true

#Raise the cutoff
cutoff <- 85

hist_edible <- density(
  edibleScores,
  from = 0,
  to = 150) %$% # exposition pipe magrittr::`%$%`
  data.frame(
    x = x,
    y = y) %>%
  mutate(area = x >= cutoff)

hist_edible$type[hist_edible$area == TRUE] <- "edible_FP"
hist_edible$type[hist_edible$area == FALSE] <- "edible_TN"

hist_inedible <- density(
  inedibleScores,
  from = 0,
  to = 150) %$% # exposition pipe magrittr::`%$%`
  data.frame(
    x = x,
    y = y) %>%
  mutate(area = x < cutoff)

hist_inedible$type[hist_inedible$area == TRUE] <- "inedible_FN"
hist_inedible$type[hist_inedible$area == FALSE] <- "inedible_TP"

density_data <- bind_rows(
  hist_edible,
  hist_inedible)

density_data$type <- factor(
  density_data$type,
  levels = c(
    "edible_TN",
    "inedible_TP",
    "edible_FP",
    "inedible_FN"))

ggplot(
  data = density_data,
  aes(
    x = x,
    ymin = 0,
    ymax = y,
    fill = type)) +
  geom_ribbon(alpha = 1) +
  scale_fill_manual(
    name = "Berry Type",
    values = c(
      viridis::viridis(4)[4],
      viridis::viridis(4)[1],
      viridis::viridis(4)[3],
      viridis::viridis(4)[2]),
    breaks = c("edible_TN","inedible_TP","edible_FP","inedible_FN"),
    labels = c("Edible: TN","Inedible: TP","Edible: FP","Inedible: FN")) +
  geom_line(aes(y = y)) +
  geom_vline(
    xintercept = cutoff,
    color = "red",
    linewidth = 2) +
  scale_x_continuous(name = "score") +
  scale_y_continuous(name = "Frequency") +
  theme_bw() +
  theme(
    axis.text.y = element_blank(),
    axis.ticks.y = element_blank())
```

@fig-classificationLowerCutoff depicts the distributions of scores by berry type when we lower the cutoff.
There are now fewer false negatives (blue) and more false positives (green).
If we lower the cutoff (to be more liberal), the number of false negatives decreases and the number of false positives increases.
Consequently, as the cutoff decreases, sensitivity and NPV increase (because we have fewer false negatives), whereas specificity and PPV decrease (because we have more false positives).
A lower cutoff could be optimal if the costs of false negatives are considered greater than the costs of false positives.
For instance, if the aliens cannot risk missing edible berries because they are in short supply relative to the size of the alien colony, and eating the inedible berries would, at worst, lead to minor, temporary discomfort.

```{r}
#| label: fig-classificationLowerCutoff
#| fig-cap: "Classifications Based on Lowering the Cutoff. Note that some true negatives and true positives are hidden behind the false positives and false negatives."
#| fig-alt: "Classifications Based on Lowering the Cutoff. Note that some true negatives and true positives are hidden behind the false positives and false negatives."
#| code-fold: true

#Lower the cutoff
cutoff <- 65

hist_edible <- density(
  edibleScores,
  from = 0,
  to = 150) %$% # exposition pipe magrittr::`%$%`
  data.frame(
    x = x,
    y = y) %>%
  mutate(area = x >= cutoff)

hist_edible$type[hist_edible$area == TRUE] <- "edible_FP"
hist_edible$type[hist_edible$area == FALSE] <- "edible_TN"

hist_inedible <- density(
  inedibleScores,
  from = 0,
  to = 150) %$% # exposition pipe magrittr::`%$%`
  data.frame(
    x = x,
    y = y) %>%
  mutate(area = x < cutoff)

hist_inedible$type[hist_inedible$area == TRUE] <- "inedible_FN"
hist_inedible$type[hist_inedible$area == FALSE] <- "inedible_TP"

density_data <- bind_rows(
  hist_edible,
  hist_inedible)

density_data$type <- factor(
  density_data$type,
  levels = c(
    "edible_TN",
    "inedible_TP",
    "edible_FP",
    "inedible_FN"))

ggplot(
  data = density_data,
  aes(
    x = x,
    ymin = 0,
    ymax = y,
    fill = type)) +
  geom_ribbon(alpha = 1) +
  scale_fill_manual(
    name = "Berry Type",
    values = c(
      viridis::viridis(4)[4],
      viridis::viridis(4)[1],
      viridis::viridis(4)[3],
      viridis::viridis(4)[2]),
    breaks = c("edible_TN","inedible_TP","edible_FP","inedible_FN"),
    labels = c("Edible: TN","Inedible: TP","Edible: FP","Inedible: FN")) +
  geom_line(aes(y = y)) +
  geom_vline(
    xintercept = cutoff,
    color = "red",
    linewidth = 2) +
  scale_x_continuous(name = "score") +
  scale_y_continuous(name = "Frequency") +
  theme_bw() +
  theme(
    axis.text.y = element_blank(),
    axis.ticks.y = element_blank())
```

In sum, sensitivity and specificity differ based on the cutoff for classification.
If we raise the cutoff, sensitivity and PPV increase (due to fewer false positives), whereas sensitivity and NPV decrease (due to more false negatives).
If we lower the cutoff, sensitivity and NPV increase (due to fewer false negatives), whereas specificity and PPV decrease (due to more false positives).
Thus, the optimal cutoff depends on how costly each type of error is: false negatives and false positives.
If false negatives are more costly than false positives, we would set a low cutoff.
If false positives are more costly than false negatives, we would set a high cutoff.

### Signal Detection Theory {#sec-sdt}

Signal detection theory (SDT) is a probability-based theory for the detection of a given stimulus (signal) from a stimulus set that includes non-target stimuli (noise).
SDT arose through the development of radar (**RA**dio **D**etection **A**nd **R**anging) and sonar (**SO**und **N**avigation **A**nd **R**anging) in World War II based on research on sensory-perception research.
The military wanted to determine which objects on radar/sonar were enemy aircraft/submarines, and which were noise (e.g., different object in the environment or even just the weather itself).
SDT allowed determining how many errors operators made (how accurate they were) and decomposing errors into different kinds of errors.
SDT distinguishes between sensitivity and bias.
In SDT, *sensitivity* (or discriminability) is how well an assessment distinguishes between a target stimulus and non-target stimuli (i.e., how well the assessment detects the target stimulus amid non-target stimuli).
*Bias* is the extent to which the probability of a selection decision from the assessment is higher or lower than the true rate of the target stimulus.

Some radar/sonar operators were not as sensitive to the differences between signal and noise, due to factors such as age, ability to distinguish gradations of a signal, etc.
People who showed low sensitivity (i.e., who were not as successful at distinguishing between signal and noise) were screened out because the military perceived sensitivity as a skill that was not easily taught.
By contrast, other operators could distinguish signal from noise, but their threshold was too low or high—they could take in information, but their decisions tended to be wrong due to systematic bias or poor calibration.
That is, they systematically over-rejected or under-rejected stimuli.
Over-rejecting leads to many false negatives (i.e., saying that a stimulus is safe when it is not).
Under-rejecting leads to many false positives (i.e., saying that a stimulus is harmful when it is not).
A person who showed good sensitivity but systematic bias was considered more teach-able than a person who showed low sensitivity.
Thus, radar and sonar operators were selected based on their sensitivity to distinguish signal from noise, and then were trained to improve the calibration so they reduce their systematic bias and do not systematically over- or under-reject.

Although SDT was originally developed for use in World War II, it now plays an important role in many areas of science and medicine.
A medical application of SDT is tumor detection in radiology.
Another application of SDT in society is using x-ray to detect bombs or other weapons.
An example of applying SDT to fantasy football could be in the prediction (and evaluation) of whether or not a player scores a touchdown in a game.

SDT metrics of sensitivity include $d'$ ("$d$-prime"), $A$ (or $A'$), and the area under the receiver operating characteristic (ROC) curve.
SDT metrics of bias include $\beta$ (beta), $c$, and $b$.

#### Receiver Operating Characteristic (ROC) Curve {#sec-roc}

The x-axis of the ROC curve is the false alarm rate or false positive rate ($1 -$ specificity).
The y-axis is the hit rate or true positive rate (sensitivity).
We can trace the ROC curve as the combination between sensitivity and specificity at every possible cutoff.
At a cutoff of zero (top right of ROC curve), we calculate sensitivity (1.0) and specificity (0) and plot it.
At a cutoff of zero, the assessment tells us to make an action for every stimulus (i.e., it is the most liberal).
We then gradually increase the cutoff, and plot sensitivity and specificity at each cutoff.
As the cutoff increases, sensitivity decreases and specificity increases.
We end at the highest possible cutoff, where the sensitivity is 0 and the specificity is 1.0 (i.e., we never make an action; i.e., it is the most conservative).
Each point on the ROC curve corresponds to a pair of hit and false alarm rates (sensitivity and specificity) resulting from a specific cutoff value.
Then, we can draw lines or a curve to connect the points.

INSERT depicts an empirical ROC plot where lines are drawn to connect the hit and false alarm rates.

INSERT depicts an ROC curve where a smoothed and fitted curve is drawn to connect the hit and false alarm rates.

##### Area Under the ROC Curve {#sec-auc}

ROC methods can be used to compare and compute the discriminative power of measurement devices free from the influence of selection ratios, base rates, and costs and benefits.
An ROC analysis yields a quantitative index of how well an index predicts a signal of interest or can discriminate between different signals.
ROC analysis can help tell us how often our assessment would be correct.
If we randomly pick two observations, and we were right once and wrong once, we were 50% accurate.
But this would be a useless measure because it reflects chance responding.

The geometrical area under the ROC curve reflects the discriminative accuracy of the measure.
The index is called the **a**rea **u**nder the **c**urve (AUC) of an ROC curve.
AUC quantifies the discriminative power of an assessment.
AUC is the probability that a randomly selected target and a randomly selected non-target is ranked correctly by the assessment method.
AUC values range from 0.0 to 1.0, where chance accuracy is 0.5 as indicated by diagonal line in the ROC curve.
That is, a measure can be useful to the extent that its ROC curve is above the diagonal line (i.e., its discriminative accuracy is above chance).

AUC is a [threshold-independent accuracy index](#sec-thresholdIndependentAccuracy) that applies across all possible cutoff values.

@fig-aucRange depicts ROC curves with a range of AUC values.

```{r}
#| label: fig-aucRange
#| fig-cap: "Receiver Operating Characteristic (ROC) Curves for Various Levels of Area Under The ROC Curve (AUC) for Various Measures."
#| fig-alt: "Receiver Operating Characteristic (ROC) Curves for Various Levels of Area Under The ROC Curve (AUC) for Various Measures."
#| fig-width: 8
#| fig-height: 8
#| code-fold: true

set.seed(52242)

auc60 <- petersenlab::simulateAUC(.60, 50000)
auc70 <- petersenlab::simulateAUC(.70, 50000)
auc80 <- petersenlab::simulateAUC(.80, 50000)
auc90 <- petersenlab::simulateAUC(.90, 50000)
auc95 <- petersenlab::simulateAUC(.95, 50000)
auc99 <- petersenlab::simulateAUC(.99, 50000)

plot(
  pROC::roc(
    y ~ x,
    auc60,
    smooth = TRUE),
  legacy.axes = TRUE,
  print.auc = TRUE,
  print.auc.x = .52,
  print.auc.y = .61,
  print.auc.pattern = "%.2f")

plot(
  pROC::roc(
    y ~ x,
    auc70,
    smooth = TRUE),
  legacy.axes = TRUE,
  print.auc = TRUE,
  print.auc.x = .6,
  print.auc.y = .67,
  print.auc.pattern = "%.2f",
  add = TRUE)

plot(
  pROC::roc(
    y ~ x,
    auc80,
    smooth = TRUE),
  legacy.axes = TRUE,
  print.auc = TRUE,
  print.auc.x = .695,
  print.auc.y = .735,
  print.auc.pattern = "%.2f",
  add = TRUE)

plot(
  pROC::roc(
    y ~ x,
    auc90,
    smooth = TRUE),
  legacy.axes = TRUE,
  print.auc = TRUE,
  print.auc.x = .805,
  print.auc.y = .815,
  print.auc.pattern = "%.2f",
  add = TRUE)

plot(
  pROC::roc(
    y ~ x,
    auc95,
    smooth = TRUE),
  legacy.axes = TRUE,
  print.auc = TRUE,
  print.auc.x = .875,
  print.auc.y = .865,
  print.auc.pattern = "%.2f",
  add = TRUE)

plot(
  pROC::roc(
    y ~ x,
    auc99,
    smooth = TRUE),
  legacy.axes = TRUE,
  print.auc = TRUE,
  print.auc.x = .94,
  print.auc.y = .94,
  print.auc.pattern = "%.2f",
  add = TRUE)
```

As an example, given an AUC of .75, this says that the overall score of an individual who has the characteristic in question will be higher 75% of the time than the overall score of an individual who does not have the characteristic.
In lay terms, AUC provides the probability that we will classify correctly based on our instrument if we were to randomly pick one good and one bad outcome.
AUC is a stronger index of accuracy than percent accuracy, because you can have high percent accuracy just by going with the base rate.
AUC tells us how much better than chance a measure is at discriminating outcomes.
AUC is useful as a measure of general discriminative accuracy, and it tells us how accurate a measure is at all possible cutoffs.
Knowing the accuracy of a measure at all possible cutoffs can be helpful for selecting the optimal cutoff, given the goals of the assessment.
In reality, however, we may not be interested in all cutoffs because not all errors are equal in their costs.

If we lower the base rate, we would need a larger sample to get enough people to classify into each group.
SDT/ROC methods are traditionally about dichotomous decisions (yes/no), not graded judgments.
SDT/ROC methods can get messy with ordinal data that are more graded because you would have an AUC curve for each ordinal grouping.

### Accuracy Indices {#sec-accuracyIndicesCategorical}

There are various accuracy indices we can use to evaluate the accuracy of predictions for categorical outcome variables.
We have already described several accuracy indices, including percent accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and area under the ROC curve.
We describe these and other indices in greater detail below.

The [`petersenlab`](https://github.com/DevPsyLab/petersenlab) package [@R-petersenlab] contains the `accuracyAtCutoff()` function that computes many accuracy indices for the prediction of categorical outcome variables.

```{r}
#petersenlab::accuracyAtCutoff()
```

The [`petersenlab`](https://github.com/DevPsyLab/petersenlab) package [@R-petersenlab] contains the `accuracyAtEachCutoff()` function that computes many accuracy indices for the prediction of categorical outcome variables at each possible cutoff.

```{r}
#petersenlab::accuracyAtEachCutoff()
```

There are also test calculators available online:

- <http://araw.mede.uic.edu/cgi-bin/testcalc.pl>
- <https://dlrs.shinyapps.io/shinyDLRs>

#### Confusion Matrix aka 2x2 Accuracy Table aka Cross-Tabulation aka Contingency Table {#sec-confusionMatrix}

A confusion matrix (aka 2x2 accuracy table, cross-tabulation table, or contigency table) is a matrix for categorical data that presents the predicted outcome on one dimension and the actual outcome (truth) on the other dimension.
If the predictions and outcomes are dichotomous, the confusion matrix is a 2x2 matrix with two rows and two columns that represent four possible predicted-actual combinations ([decision outcomes](#sec-decisionOutcomes)).
In such a case, the confusion matrix provides a tabular count of each type of accurate cases ([true positives](#sec-truePositive) and [true negatives](#sec-trueNegative)) versus the number of each type of error ([false positives](#sec-falsePositive) and [false negatives](#sec-falseNegative)), as shown in INSERT.
An example of a confusion matrix is in INSERT.

##### Number {#sec-confusionMatrix-number}

```{r}
#table(mydata$diagnosisFactor, mydata$diseaseFactor)
```

##### Number with margins added {#sec-confusionMatrix-numberMargins}

```{r}
#addmargins(table(mydata$diagnosisFactor, mydata$diseaseFactor))
```

##### Proportions {#sec-confusionMatrix-numberMarginsProportions}

```{r}
#prop.table(table(mydata$diagnosisFactor, mydata$diseaseFactor))
```

##### Proportions with margins added {#sec-confusionMatrix-numberMarginsProportionsMargins}

```{r}
#addmargins(prop.table(table(mydata$diagnosisFactor, mydata$diseaseFactor)))
```

#### True Positives (TP) {#sec-truePositive}

True positives (TPs) are instances in which a positive classification (e.g., stating that a disease is present for a person) is correct—that is, the test says that a classification is present, and the classification is present.
True positives are also called valid positives (VPs) or hits.
Higher values reflect greater accuracy.
The formula for true positives is in @eq-truePositive:

$$
\begin{aligned}
  \text{TP} &= \text{BR} \times \text{SR} \times N
\end{aligned}
$$ {#eq-truePositive}

#### True Negatives (TN) {#sec-trueNegative}

True negatives (TNs) are instances in which a negative classification (e.g., stating that a disease is absent for a person) is correct—that is, the test says that a classification is not present, and the classification is actually not present.
True negatives are also called valid negatives (VNs) or correct rejections.
Higher values reflect greater accuracy.
The formula for true negatives is in @eq-trueNegative:

$$
\begin{aligned}
  \text{TN} &= (1 - \text{BR}) \times (1 - \text{SR}) \times N
\end{aligned}
$$ {#eq-trueNegative}

#### False Positives (FP) {#sec-falsePositive}

False positives (FPs) are instances in which a positive classification (e.g., stating that a disease is present for a person) is incorrect—that is, the test says that a classification is present, and the classification is not present.
False positives are also called false alarms (FAs).
Lower values reflect greater accuracy.
The formula for false positives is in Equation @eq-falsePositive:

$$
\begin{aligned}
  \text{FP} &= (1 - \text{BR}) \times \text{SR} \times N
\end{aligned}
$$ {#eq-falsePositive}

#### False Negatives (FN) {#sec-falseNegative}

False negatives (FNs) are instances in which a negative classification (e.g., stating that a disease is absent for a person) is incorrect—that is, the test says that a classification is not present, and the classification is present.
False negatives are also called misses.
Lower values reflect greater accuracy.
The formula for false negatives is in @eq-falseNegative:

$$
\begin{aligned}
  \text{FN} &= \text{BR} \times (1 - \text{SR}) \times N
\end{aligned}
$$ {#eq-falseNegative}

#### Selection Ratio (SR) {#sec-selectionRatio}

The selection ratio (SR) is the marginal probability of selection, independent of other things: $P(R_i)$.
It is not an index of accuracy, per se.
In medicine, the selection ratio is the proportion of people who test positive for the disease.
In fantasy football, the selection ratio is the proportion of players who you predict will show a given outcome.
For example, if you are trying to predict the players who will score a touchdown in a game, the selection ratio is the proportion of players who you predict will score a touchdown.
The formula for calculating the selection ratio is in @eq-selectionRatio.

$$
\begin{aligned}
  \text{SR} &= P(R_i) \\
  &= \frac{\text{TP} + \text{FP}}{N}
\end{aligned}
$$ {#eq-selectionRatio}

#### Base Rate (BR) {#sec-pretestProbability}

The [base rate](#sec-baseRate) (BR) of a classification is its [marginal probability](#sec-baseRate), independent of other things: $P(C_i)$.
It is not an index of accuracy, per se.
In medicine, the base rate of a disease is its prevalence in the population, as in @eq-baseRate.
Without additional information, the [base rate](#sec-baseRate) is used as the initial *pretest probability*.
In fantasy football, the [base rate](#sec-baseRate) is the proportion of players who actually show the particular outcome.
For example, if you are trying to predict the players who will score a touchdown in a game, the [base rate](#sec-baseRate) is the proportion of players who actually score a touchdown in the game.
The formula for calculating the selection ratio is in @eq-baseRate.

$$
\begin{aligned}
  \text{BR} &= P(C_i) \\
  &= \frac{\text{TP} + \text{FN}}{N}
\end{aligned}
$$ {#eq-baseRate}

#### Pretest Odds {#sec-pretestOdds}

The pretest odds of a classification can be estimated using the pretest probability (i.e., [base rate](#sec-baseRate)).
To convert a probability to odds, divide the probability by one minus that probability, as in @eq-pretestOdds.

$$
\begin{aligned}
  \text{pretest odds} &= \frac{\text{pretest probability}}{1 - \text{pretest probability}} \\
\end{aligned}
$$ {#eq-pretestOdds}

#### Percent Accuracy {#sec-percentAccuracy}

Percent Accuracy is also called overall accuracy.
Higher values reflect greater accuracy.
The formula for percent accuracy is in @eq-percentAccuracy.
Percent accuracy has several problems.
First, it treats all errors ([FP](#sec-falsePositive) and [FN](#sec-falseNegative)) as equally important.
However, in practice, it is rarely the case that [false positives](#sec-falsePositive) and [false negatives](#sec-falseNegative) are equally important.
Second, percent accuracy can be misleading because it is highly influenced by [base rates](#sec-baseRate).
You can have a high percent accuracy by predicting from the [base rate](#sec-baseRate) and saying that no one has the characteristic (if the [base rate](#sec-baseRate) is low) or that everyone has the characteristic (if the [base rate](#sec-baseRate) is high).
Thus, it is also important to consider other aspects of accuracy.

$$
\text{Percent Accuracy} = 100\% \times \frac{\text{TP} + \text{TN}}{N}
$$ {#eq-percentAccuracy}

#### Percent Accuracy by Chance {#sec-percentAccuracyByChance}

The formula for calculating percent accuracy by chance is in @eq-percentAccuracyByChance.

$$
\begin{aligned}
  \text{Percent Accuracy by Chance} &= 100\% \times [P(\text{TP}) + P(\text{TN})] \\
  &= 100\% \times \{(\text{BR} \times {\text{SR}}) + [(1 - \text{BR}) \times (1 - \text{SR})]\}
\end{aligned}
$$ {#eq-percentAccuracyByChance}

#### Percent Accuracy Predicting from the Base Rate {#sec-percentAccuracyPredictingFromBaseRate}

[*Predicting from the base rate*](#sec-predictingFromBaseRate) is going with the most likely outcome in every prediction.
If the [base rate](#sec-baseRate) is less than .50, it would involve predicting that the condition is absent for every case.
If the [base rate](#sec-baseRate) is .50 or above, it would involve predicting that the condition is present for every case.
[Predicting from the base rate](#sec-predictingFromBaseRate) is a special case of [percent accuracy by chance](#sec-percentAccuracyByChance) when the [selection ratio](#sec-selectionRatio) is set to either one (if the [base rate](#sec-baseRate) $\geq$ .5) or zero (if the [base rate](#sec-baseRate) < .5).

#### Relative Improvement Over Chance (RIOC) {#sec-relativeImprovementOverChance}

Relative improvement over chance (RIOC) is a prediction's improvement over chance as a proportion of the maximum possible improvement over chance, as described by @Farrington1989.
Higher values reflect greater accuracy.
The formula for calculating RIOC is in @eq-relativeImprovementOverChance.

$$
\begin{aligned}
  \text{relative improvement over chance (RIOC)} &= \frac{\text{total correct} - \text{chance correct}}{\text{maximum correct} - \text{chance correct}} \\
\end{aligned}
$$ {#eq-relativeImprovementOverChance}

#### Relative Improvement Over Predicting from the Base Rate {#sec-relativeImprovementOverPredictingFromBaseRate}

Relative improvement over [predicting from the base rate](#sec-predictingFromBaseRate) is a prediction's improvement over [predicting from the base rate](#sec-predictingFromBaseRate) as a proportion of the maximum possible improvement over [predicting from the base rate](#sec-predictingFromBaseRate).
Higher values reflect greater accuracy.
The formula for calculating relative improvement over predicting from the base rate is in @eq-relativeImprovementOverPredictingFromBaseRate.

$$
\scriptsize
\begin{aligned}
  \text{relative improvement over predicting from base rate} &= \frac{\text{total correct} - \text{correct by predicting from base rate}}{\text{maximum correct} - \text{correct by predicting from base rate}} \\
\end{aligned}
$$ {#eq-relativeImprovementOverPredictingFromBaseRate}

#### Sensitivity (SN) {#sec-sensitivity}

Sensitivity (SN) is also called true positive rate (TPR), hit rate (HR), or recall.
Sensitivity is the [conditional probability](#sec-conditionalProbability) of a positive test given that the person has the condition: $P(R|C)$.
Higher values reflect greater accuracy.
The formula for calculating sensitivity is in @eq-sensitivity.
As described in Section @sec-accuracyCutoff, as the cutoff increases (becomes more conservative), sensitivity decreases.
As the cutoff decreases, sensitivity increases.

$$
\begin{aligned}
  \text{sensitivity (SN)} &= P(R|C) \\
  &= \frac{\text{TP}}{\text{TP} + \text{FN}} = \frac{\text{TP}}{N \times \text{BR}} = 1 - \text{FNR}
\end{aligned}
$$ {#eq-sensitivity}

#### Specificity (SP) {#sec-specificity}

Specificity (SP) is also called true negative rate (TNR) or selectivity.
Specificity is the [conditional probability](#sec-conditionalProbability) of a negative test given that the person does not have the condition: $P(\text{not } R|\text{not } C)$.
Higher values reflect greater accuracy.
The formula for calculating specificity is in @eq-specificity.
As described in Section @sec-accuracyCutoff, as the cutoff increases (becomes more conservative), specificity increases.
As the cutoff decreases, specificity decreases.

$$
\begin{aligned}
  \text{specificity (SP)} &= P(\text{not } R|\text{not } C) \\
  &= \frac{\text{TN}}{\text{TN} + \text{FP}} = \frac{\text{TN}}{N (1 - \text{BR})} = 1 - \text{FPR}
\end{aligned}
$$ {#eq-specificity}

#### False Negative Rate (FNR) {#sec-falseNegativeRate}

The false negative rate (FNR) is also called the miss rate.
The false negative rate is the [conditional probability](#sec-conditionalProbability) of a negative test given that the person has the condition: $P(\text{not } R|C)$.
Lower values reflect greater accuracy.
The formula for calculating false negative rate is in @eq-falseNegativeRate.

$$
\begin{aligned}
  \text{false negative rate (FNR)} &= P(\text{not } R|C) \\
  &= \frac{\text{FN}}{\text{FN} + \text{TP}} = \frac{\text{FN}}{N \times \text{BR}} = 1 - \text{TPR}
\end{aligned}
$$ {#eq-falseNegativeRate}

#### False Positive Rate (FPR) {#sec-falsePositiveRate}

The false positive rate (FPR) is also called the false alarm rate (FAR) or fall-out.
The false positive rate is the [conditional probability](#sec-conditionalProbability) of a positive test given that the person does not have the condition: $P(R|\text{not } C)$.
Lower values reflect greater accuracy.
The formula for calculating false positive rate is in @eq-falsePositiveRate:

$$
\begin{aligned}
  \text{false positive rate (FPR)} &= P(R|\text{not } C) \\
  &= \frac{\text{FP}}{\text{FP} + \text{TN}} = \frac{\text{FP}}{N (1 - \text{BR})} = 1 - \text{TNR}
\end{aligned}
$$ {#eq-falsePositiveRate}

#### Positive Predictive Value (PPV) {#sec-ppv}

The positive predictive value (PPV) is also called the positive predictive power (PPP) or precision.
Many people confuse [sensitivity](#sec-sensitivity) ($P(R|C)$) with its inverse [conditional probability](#sec-conditionalProbability), PPV ($P(C|R)$).
PPV is the [conditional probability](#sec-conditionalProbability) of having the condition given a positive test: $P(C|R)$.
Higher values reflect greater accuracy.
The formula for calculating positive predictive value is in @eq-positivePredictiveValue.

PPV can be low even when [sensitivity](#sec-sensitivity) is high because it depends not only on [sensitivity](#sec-sensitivity), but also on [specificity](#sec-specificity) and the [base rate](#sec-baseRate).
Because PPV depends on the [base rate](#sec-baseRate), PPV is not an intrinsic property of a measure.
The same measure will have a different PPV in different contexts with different [base rates](#sec-baseRate) [@Treat2023].
As described in Section @sec-accuracyCutoff, as the [base rate](#sec-baseRate) increases, PPV increases.
As the [base rate](#sec-baseRate) decreases, PPV decreases.
PPV also differs as a function of the cutoff.
As described in Section @sec-accuracyCutoff, as the cutoff increases (becomes more conservative), PPV increases.
As the cutoff decreases (becomes more liberal), PPV decreases.

$$
\small
\begin{aligned}
  \text{positive predictive value (PPV)} &= P(C|R) \\
  &= \frac{\text{TP}}{\text{TP} + \text{FP}} = \frac{\text{TP}}{N \times \text{SR}}\\
  &= \frac{\text{sensitivity} \times {\text{BR}}}{\text{sensitivity} \times {\text{BR}} + [(1 - \text{specificity}) \times (1 - \text{BR})]}
\end{aligned}
$$ {#eq-positivePredictiveValue}

#### Negative Predictive Value (NPV) {#sec-npv}

The negative predictive value (NPV) is also called the negative predictive power (NPP).
Many people confuse [specificity](#sec-specificity) ($P(\text{not } R|\text{not } C)$) with its inverse [conditional probability](#sec-conditionalProbability), NPV ($P(\text{not } C| \text{not } R)$).
NPV is the [conditional probability](#sec-conditionalProbability) of not having the condition given a negative test: $P(\text{not } C| \text{not } R)$.
Higher values reflect greater accuracy.
The formula for calculating negative predictive value is in @eq-negativePredictiveValue.

NPV can be low even when [specificity](#sec-specificity) is high because it depends not only on [specificity](#sec-specificity), but also on [sensitivity](#sec-sensitivity) and the [base rate](#sec-baseRate).
Because NPV depends on the [base rate](#sec-baseRate), NPV is not an intrinsic property of a measure.
The same measure will have a different NPV in different contexts with different [base rates](#sec-baseRate) [@Treat2023].
As described in Section @sec-accuracyCutoff, as the [base rate](#sec-baseRate) increases, NPV decreases.
As the [base rate](#sec-baseRate) decreases, NPV increases.
NPV also differs as a function of the cutoff.
As described in Section @sec-accuracyCutoff, as the cutoff increases (becomes more conservative), NPV decreases.
As the cutoff decreases (becomes more liberal), NPV decreases.

$$
\small
\begin{aligned}
  \text{negative predictive value (NPV)} &= P(\text{not } C|\text{not } R) \\
  &= \frac{\text{TN}}{\text{TN} + \text{FN}} = \frac{\text{TN}}{N(\text{1 - SR})}\\
  &= \frac{\text{specificity} \times (1-{\text{BR}})}{\text{specificity} \times (1-{\text{BR}}) + [(1 - \text{sensitivity}) \times \text{BR})]}
\end{aligned}
$$ {#eq-negativePredictiveValue}

#### False Discovery Rate (FDR) {#sec-falseDiscoveryRate}

Many people confuse the false positive rate ($P(R|\text{not } C)$) with its inverse [conditional probability](#sec-conditionalProbability), the false discovery rate ($P(\text{not } C| R)$).
The false discovery rate (FDR) is the [conditional probability](#sec-conditionalProbability) of not having the condition given a positive test: $P(\text{not } C| R)$.
Lower values reflect greater accuracy.
The formula for calculating false discovery rate is in @eq-falseDiscoveryRate.

$$
\begin{aligned}
  \text{false discovery rate (FDR)} &= P(\text{not } C|R) \\
  &= \frac{\text{FP}}{\text{FP} + \text{TP}} = 1 - \text{PPV}
\end{aligned}
$$ {#eq-falseDiscoveryRate}

#### False Omission Rate (FOR) {#sec-falseOmissionRate}

Many people confuse the false negative rate ($P(\text{not } R|C)$) with its inverse [conditional probability](#sec-conditionalProbability), the false omission rate ($P(C|\text{not } R)$).
The false omission rate (FOR) is the conditional probability of having the condition given a negative test: $P(C|\text{not } R)$.
Lower values reflect greater accuracy.
The formula for calculating false omission rate is in @sec-falseOmissionRate.

$$
\begin{aligned}
  \text{false omission rate (FOR)} &= P(C|\text{not } R) \\
  &= \frac{\text{FN}}{\text{FN} + \text{TN}} = 1 - \text{NPV}
\end{aligned}
$$ {#eq-falseOmissionRate}

#### Youden's J Statistic {#sec-youdenJ-example}

Youden's J statistic is also called Youden's Index or informedness.
Youden's J statistic is the sum of [sensitivity](#sec-sensitivity) and [specificity](#sec-specificity) (and subtracting one).
Higher values reflect greater accuracy.
The formula for calculating Youden's J statistic is in @eq-youdenIndex.

$$
\begin{aligned}
  \text{Youden's J statistic} &= \text{sensitivity} + \text{specificity} - 1
\end{aligned}
$$ {#eq-youdenIndex}

#### Balanced Accuracy {#sec-balancedAccuracy}

Balanced accuracy is the average of [sensitivity](#sec-sensitivity) and [specificity](#sec-specificity).
Higher values reflect greater accuracy.
The formula for calculating balanced accuracy is in @eq-balancedAccuracy.

$$
\begin{aligned}
  \text{balanced accuracy} &= \frac{\text{sensitivity} + \text{specificity}}{2}
\end{aligned}
$$ {#eq-balancedAccuracy}

#### F-Score {#sec-fScore}

The F-score combines [precision](#sec-ppv) ([positive predictive value](#sec-ppv)) and [recall](#sec-sensitivity) ([sensitivity](#sec-sensitivity)), where $\beta$ indicates how many times more important [sensitivity](#sec-sensitivity) is than the [positive predictive value](#sec-ppv).
If [sensitivity](#sec-sensitivity) and the [positive predictive value](#sec-ppv) are equally important, $\beta = 1$, and the F-score is called the $F_1$ score.
Higher values reflect greater accuracy.
The formula for calculating the F-score is in @eq-FScore.

$$
\begin{aligned}
  F_\beta &= (1 + \beta^2) \cdot \frac{\text{positive predictive value} \cdot \text{sensitivity}}{(\beta^2 \cdot \text{positive predictive value}) + \text{sensitivity}} \\
  &= \frac{(1 + \beta^2) \cdot \text{TP}}{(1 + \beta^2) \cdot \text{TP} + \beta^2 \cdot \text{FN} + \text{FP}}
\end{aligned}
$$ {#eq-FScore}

The formula for calculating the $F_1$ score is in @eq-F1Score.

$$
\begin{aligned}
  F_1 &= \frac{2 \cdot \text{positive predictive value} \cdot \text{sensitivity}}{(\text{positive predictive value}) + \text{sensitivity}} \\
  &= \frac{2 \cdot \text{TP}}{2 \cdot \text{TP} + \text{FN} + \text{FP}}
\end{aligned}
$$ {#eq-F1Score}

#### Matthews Correlation Coefficient (MCC) {#sec-matthewsCorrelationCoefficient}

The Matthews correlation coefficient (MCC) is also called the phi coefficient.
It is a correlation coefficient between predicted and observed values from a binary classification.
Higher values reflect greater accuracy.
The formula for calculating the MCC is in @eq-matthewsCorrelationCoefficient.

$$
\begin{aligned}
  \text{MCC} &= \frac{\text{TP} \times \text{TN} - \text{FP} \times \text{FN}}{\sqrt{(\text{TP} + \text{FP})(\text{TP} + \text{FN})(\text{TN} + \text{FP})(\text{TN} + \text{FN})}}
\end{aligned}
$$ {#eq-matthewsCorrelationCoefficient}

#### Diagnostic Odds Ratio {#sec-diagnosticOddsRatio}

The diagnostic odds ratio is the odds of a positive test among people with the condition relative to the odds of a positive test among people without the condition.
Higher values reflect greater accuracy.
The formula for calculating the diagnostic odds ratio is in @eq-diagnosticOddsRatio.
If the predictor is bad, the diagnostic odds ratio could be less than one, and values can go up from there.
If the diagnostic odds ratio is greater than 2, we take the odds ratio seriously because we are twice as likely to predict accurately than inaccurately.
However, the diagnostic odds ratio ignores/hides [base rates](#sec-baseRate).
When interpreting the diagnostic odds ratio, it is important to keep in mind the [practical significance](#sec-practicalSignificance), because otherwise it is not very meaningful.
Consider a risk factor that has a diagnostic odds ratio of 3 for tuberculosis, i.e., it puts you at 3 times as likely to develop tuberculosis.
The prevalence of tuberculosis is relatively low.
Assuming the prevalence of tuberculosis is less than 1/10th of 1%, your risk of developing tuberculosis is still very low even if the risk factor (with a diagnostic odds ratio of 3) is present.

$$
\begin{aligned}
  \text{diagnostic odds ratio} &= \frac{\text{TP} \times \text{TN}}{\text{FP} \times \text{FN}} \\
  &= \frac{\text{sensitivity} \times \text{specificity}}{(1 - \text{sensitivity}) \times (1 - \text{specificity})} \\
  &= \frac{\text{PPV} \times \text{NPV}}{(1 - \text{PPV}) \times (1 - \text{NPV})} \\
  &= \frac{\text{LR+}}{\text{LR}-}
\end{aligned}
$$ {#eq-diagnosticOddsRatio}

#### Diagnostic Likelihood Ratio {#sec-diagnosticLikelihoodRatio2}

The diagnostic likelihood ratio is described in @sec-diagnosticLikelihoodRatio.
There are two types of diagnostic likelihood ratios: the [positive likelihood ratio](#sec-positiveLikelihoodRatio) and the [negative likelihood ratio](#sec-negativeLikelihoodRatio).

##### Positive Likelihood Ratio (LR+) {#sec-positiveLikelihoodRatio2}

The positive likelihood ratio (LR+) is described in @sec-positiveLikelihoodRatio.
The formula for calculating the positive likelihood ratio is in @eq-positiveLikelihoodRatio.

##### Negative Likelihood Ratio (LR−) {#sec-negativeLikelihoodRatio2}

The negative likelihood ratio (LR−) is described in @sec-negativeLikelihoodRatio.
The formula for calculating the negative likelihood ratio is in @eq-positiveLikelihoodRatio.

#### Posttest Odds {#sec-posttestOdds}

As presented in @eq-bayes5, the posttest (or posterior) odds are equal to the [pretest odds](#sec-pretestOdds) multiplied by the [likelihood ratio](#sec-diagnosticLikelihoodRatio).
The posttest odds and [posttest probability](#sec-posttestProbability) can be useful to calculate when the [pretest probability](#sec-baseRate) is different from the [pretest probability](#sec-baseRate) (or prevalence) of the classification.
For instance, you might use a different [pretest probability](#sec-baseRate) if a test result is already known and you want to know the updated [posttest probability](#sec-posttestProbability) after conducting a second test.
The formula for calculating posttest odds is in @eq-posttestOdds.

$$
\begin{aligned}
  \text{posttest odds} &= \text{pretest odds} \times \text{likelihood ratio} \\
\end{aligned}
$$ {#eq-posttestOdds}

For calculating the posttest odds of a [true positive](#sec-truePositive) compared to a [false positive](#sec-falsePositive), we use the [positive likelihood ratio](#sec-positiveLikelihoodRatio) below.
We would use the [negative likelihood ratio](#sec-negativeLikelihoodRatio) if we wanted to calculate the posttest odds of a [false negative](#sec-falseNegative) compared to a [true negative](#sec-trueNegative).

#### Posttest Probability {#sec-posttestProbability}

The posttest probability is the probability of having the characteristic given a test result.
When the [base rate](#sec-baseRate) is used as the [pretest probability](#sec-baseRate), the posttest probability given a positive test is equal to [positive predictive value](#sec-ppv).
To convert odds to a probability, divide the odds by one plus the odds, as is in @eq-posttestProbability.

$$
\begin{aligned}
  \text{posttest probability} &= \frac{\text{posttest odds}}{1 + \text{posttest odds}}
\end{aligned}
$$ {#eq-posttestProbability}

#### Mean Difference Between Predicted and Observed Values {#sec-miscalibration}

The mean difference between predicted values versus observed values at a given cutoff is an index of [miscalibration](#sec-calibration) of predictions at that cutoff.
It is called "calibration-in-the-small" (as opposed to calibration-in-the-large, which spans all cutoffs).
Values closer to zero reflect greater accuracy.
Values above zero indicate that the predicted values are, on average, greater than the observed values.
Values below zero indicate that the observed values are, on average, greater than the predicted values.

## Threshold-Independent Accuracy Indices {#sec-thresholdIndependentAccuracy}

This section describes threshold-independent indexes of accuracy.
That is, each index of accuracy described in this section provides a single numerical index of accuracy that aggregates the accuracy across all possible cutoffs.
The [`petersenlab`](https://github.com/DevPsyLab/petersenlab) package [@R-petersenlab] contains the `accuracyOverall()` function that computes many threshold-independent accuracy indices.

### General Prediction Accuracy {#sec-generalPredictionAccuracy}

There are many metrics of general prediction accuracy.
When thinking about which metric(s) may be best for a given problem, it is important to consider the purpose of the assessment.
The estimates of general prediction accuracy are separated below into [scale-dependent](#sec-scaleDependentAccuracy) and [scale-independent](#sec-scaleIndependentAccuracy) accuracy estimates.

#### Scale-Dependent Accuracy Estimates {#sec-scaleDependentAccuracy}

The estimates of prediction accuracy described in this section are scale-dependent.
These accuracy estimates depend on the unit of measurement and therefore cannot be compared across measures with different scales or across data sets.

##### Mean Error {#sec-meanError}

Here, "error" ($e$) is the difference between the predicted and observed value for a given individual ($i$).
Mean error (ME; also known as bias) is the mean difference between the predicted and observed values across individuals ($i$), that is, the mean of the errors across individuals ($e_i$).
Values closer to zero reflect greater accuracy.
If mean error is above zero, it indicates that predicted values are, on average, greater than observed values (i.e., overestimating errors).
If mean error is below zero, it indicates that predicted values are, on average, less than observed values (i.e., underestimating errors).
If both over-estimating and under-estimating errors are present, however, they can cancel each other out.
As a result, even with a mean error of zero, there can still be considerable error present.
Thus, although mean error can be helpful for examining whether predictions systematically under- or over-estimate the actual scores, other forms of accuracy are necessary to examine the *extent* of error.
The formula for mean error is in @eq-meanError:

$$
\begin{aligned}
  \text{mean error} &= \frac{\sum\limits_{i = 1}^n(\text{predicted}_i - \text{observed}_i)}{n} \\
  &= \text{mean}(e_i)
\end{aligned}
$$ {#eq-meanError}

##### Mean Absolute Error (MAE) {#sec-meanAbsoluteError}

Mean absolute error (MAE) is the mean of the absolute value of differences between the predicted and observed values across individuals, that is, the mean of the absolute value of errors.
Smaller MAE values (closer to zero) reflect greater accuracy.
MAE is preferred over [root mean squared error](#sec-rootMeanSquaredError) (RMSE) when you want to give equal weight to all errors and when the outliers have considerable impact.
The formula for MAE is in @eq-meanAbsoluteError:

$$
\begin{aligned}
  \text{mean absolute error (MAE)} &= \frac{\sum\limits_{i = 1}^n|\text{predicted}_i - \text{observed}_i|}{n} \\
  &= \text{mean}(|e_i|)
\end{aligned}
$$ {#eq-meanAbsoluteError}

##### Mean Squared Error (MSE) {#sec-meanSquaredError}

Mean squared error (MSE) is the mean of the square of the differences between the predicted and observed values across individuals, that is, the mean of the squared value of errors.
Smaller MSE values (closer to zero) reflect greater accuracy.
MSE penalizes larger errors more heavily than smaller errors (unlike [MAE](#sec-meanAbsoluteError)).
However, MSE is sensitive to outliers and can be impacted if the errors are skewed.
The formula for MSE is in @eq-meanSquaredError:

$$
\begin{aligned}
  \text{mean squared error (MSE)} &= \frac{\sum\limits_{i = 1}^n(\text{predicted}_i - \text{observed}_i)^2}{n} \\
  &= \text{mean}(e_i^2)
\end{aligned}
$$ {#eq-meanSquaredError}

##### Root Mean Squared Error (RMSE) {#sec-rootMeanSquaredError}

Root mean squared error (RMSE) is the square root of the mean of the square of the differences between the predicted and observed values across individuals, that is, the root mean squared value of errors.
Smaller RMSE values (closer to zero) reflect greater accuracy.
RMSE penalizes larger errors more heavily than smaller errors (unlike [MAE](#sec-meanAbsoluteError)).
However, RMSE is sensitive to outliers and can be impacted if the errors are skewed.
The formula for RMSE is in @eq-rootMeanSquaredError:

$$
\begin{aligned}
  \text{root mean squared error (RMSE)} &= \sqrt{\frac{\sum\limits_{i = 1}^n(\text{predicted}_i - \text{observed}_i)^2}{n}} \\
  &= \sqrt{\text{mean}(e_i^2)}
\end{aligned}
$$ {#eq-rootMeanSquaredError}

#### Scale-Independent Accuracy Estimates {#sec-scaleIndependentAccuracy}

The estimates of prediction accuracy described in this section are intended to be scale-*independent* (unit-free) so the accuracy estimates can be compared across measures with different scales or across data sets [@Hyndman2021].

##### Mean Percentage Error (MPE) {#sec-meanPercentageError}

Mean percentage error (MPE) values closer to zero reflect greater accuracy.
The formula for percentage error is in @eq-percentageError:

$$
\begin{aligned}
  \text{percentage error }(p_i) = \frac{100\% \times (\text{observed}_i - \text{predicted}_i)}{\text{observed}_i}
\end{aligned}
$$ {#eq-percentageError}

We then take the mean of the percentage errors to get MPE.
The formula for MPE is in @eq-meanPercentageError:

$$
\begin{aligned}
  \text{mean percentage error (MPE)} &= \frac{100\%}{n} \sum\limits_{i = 1}^n \frac{\text{observed}_i - \text{predicted}_i}{\text{observed}_i} \\
  &= \text{mean(percentage error)} \\
  &= \text{mean}(p_i)
\end{aligned}
$$ {#eq-meanPercentageError}

Note: MPE is undefined when one or more of the observed values equals zero, due to division by zero.
The `accuracyOverall()` function of the [`petersenlab`](https://github.com/DevPsyLab/petersenlab) package [@R-petersenlab] provides the option in the function to drop undefined values so you can still generate an estimate of accuracy despite undefined values.

##### Mean Absolute Percentage Error (MAPE) {#sec-meanAbsolutePercentageError}

Smaller mean absolute percentage error (MAPE) values (closer to zero) reflect greater accuracy.
The formula for MAPE is in @eq-meanAbsolutePercentageError.
MAPE is asymmetric because it overweights underestimates and underweights overestimates.
MAPE can be preferable to [symmetric mean absolute percentage error](#sec-symmetricMeanAbsolutePercentageError) (sMAPE) if there are no observed values of zero and if you want to emphasize the importance of underestimates (relative to overestimates).

$$
\begin{aligned}
  \text{mean absolute percentage error (MAPE)} &= \frac{100\%}{n} \sum\limits_{i = 1}^n \Bigg|\frac{\text{observed}_i - \text{predicted}_i}{\text{observed}_i}\Bigg| \\
  &= \text{mean(|percentage error|)} \\
  &= \text{mean}(|p_i|)
\end{aligned}
$$ {#eq-meanAbsolutePercentageError}

Note: MAPE is undefined when one or more of the observed values equals zero, due to division by zero.
The `accuracyOverall()` function of the [`petersenlab`](https://github.com/DevPsyLab/petersenlab) package [@R-petersenlab] provides the option in the function to drop undefined values so you can still generate an estimate of accuracy despite undefined values.

##### Symmetric Mean Absolute Percentage Error (sMAPE) {#sec-symmetricMeanAbsolutePercentageError}

Unlike MAPE, symmetric mean absolute percentage error (sMAPE) is symmetric because it equally weights underestimates and overestimates.
Smaller sMAPE values (closer to zero) reflect greater accuracy.
The formula for sMAPE is in @eq-symmetricMeanAbsolutePercentageError:

$$
\small
\begin{aligned}
  \text{symmetric mean absolute percentage error (sMAPE)} = \frac{100\%}{n} \sum\limits_{i = 1}^n \frac{|\text{predicted}_i - \text{observed}_i|}{|\text{predicted}_i| + |\text{observed}_i|}
\end{aligned}
$$ {#eq-symmetricMeanAbsolutePercentageError}

Note: sMAPE is undefined when one or more of the individuals has a prediction–observed combination such that the sum of the absolute value of the predicted value and the absolute value of the observed value equals zero ($|\text{predicted}_i| + |\text{observed}_i|$), due to division by zero.
The `accuracyOverall()` function of the [`petersenlab`](https://github.com/DevPsyLab/petersenlab) package [@R-petersenlab] provides the option in the function to drop undefined values so you can still generate an estimate of accuracy despite undefined values.

##### Mean Absolute Scaled Error (MASE) {#sec-meanAbsoluteScaledError}

Mean absolute scaled error (MASE) is described by [@Hyndman2021].
Values closer to zero reflect greater accuracy.

The adapted formula for MASE with non-time series data is described [here (https://stats.stackexchange.com/a/108963/20338)](https://stats.stackexchange.com/a/108963/20338) (archived at https://perma.cc/G469-8NAJ).
Scaled errors are calculated using @eq-scaledError:

$$
\begin{aligned}
  \text{scaled error}(q_i) &= \frac{\text{observed}_i - \text{predicted}_i}{\text{scaling factor}} \\
  &= \frac{\text{observed}_i - \text{predicted}_i}{\frac{1}{n} \sum\limits_{i = 1}^n |\text{observed}_i - \overline{\text{observed}}|}
\end{aligned}
$$ {#eq-scaledError}

Then, we calculate the mean of the absolute value of the scaled errors to get MASE, as in @eq-meanAbsoluteScaledError:

$$
\begin{aligned}
  \text{mean absolute scaled error (MASE)} &= \frac{1}{n} \sum\limits_{i = 1}^n |q_i| \\
  &= \text{mean(|scaled error|)} \\
  &= \text{mean}(|q_i|)
\end{aligned}
$$ {#eq-meanAbsoluteScaledError}

Note: MASE is undefined when the scaling factor is zero, due to division by zero.
With non-time series data, the scaling factor is the average of the absolute value of individuals' observed scores minus the average observed score ($\frac{1}{n} \sum\limits_{i = 1}^n |\text{observed}_i - \overline{\text{observed}}|$).

##### Root Mean Squared Log Error (RMSLE) {#sec-rootMeanSquaredLogError}

The squared log of the accuracy ratio is described by @Tofallis2015.
The accuracy ratio is in @eq-accuracyRatio1:

$$
\begin{aligned}
  \text{accuracy ratio} &= \frac{\text{predicted}_i}{\text{observed}_i}
\end{aligned}
$$ {#eq-accuracyRatio1}

However, the accuracy ratio is undefined with observed or predicted values of zero, so it is common to modify it by adding 1 to the predictor and denominator, as in @eq-accuracyRatio2:

$$
\begin{aligned}
  \text{accuracy ratio} &= \frac{\text{predicted}_i + 1}{\text{observed}_i + 1}
\end{aligned}
$$ {#eq-accuracyRatio2}

Squaring the log values keeps the values positive, such that smaller values (values closer to zero) reflect greater accuracy.
Then we take the mean of the squared log values, which keeps the values positive, and calculate the square root of the mean squared log values to put them back on the (pre-squared) log metric.
This is known as the root mean squared log error (RMSLE).
Division inside the log is equal to subtraction outside the log.
So, the formula can be reformulated with the subtraction of two logs, as in @eq-rootMeanSquaredLogError:

$$
\scriptsize
\begin{aligned}
  \text{root mean squared log error (RMSLE)} &= \sqrt{\sum\limits_{i = 1}^n log\bigg(\frac{\text{predicted}_i + 1}{\text{observed}_i + 1}\bigg)^2} \\
  &= \sqrt{\text{mean}\Bigg[log\bigg(\frac{\text{predicted}_i + 1}{\text{observed}_i + 1}\bigg)^2\Bigg]} \\
  &= \sqrt{\text{mean}\big[log(\text{accuracy ratio})^2\big]} = \sqrt{\text{mean}\Big\{\big[log(\text{predicted}_i + 1) - log(\text{actual}_i + 1)\big]^2\Big\}}
\end{aligned}
$$ {#eq-rootMeanSquaredLogError}

RMSLE can be preferable when the scores have a wide range of values and are skewed.
RMSLE can help to reduce the impact of outliers.
RMSLE gives more weight to smaller errors in the prediction of small observed values, while also penalizing larger errors in the prediction of larger observed values.
It overweights underestimates and underweights overestimates.

There are other variations of prediction accuracy metrics that use the log of the accuracy ratio.
One variation makes it similar to median symmetric percentage error [@Morley2018].

Note: Root mean squared log error is undefined when one or more predicted values or actual values equals −1.
When predicted or actual values are -1, this leads to $log(0)$, which is undefined.
The `accuracyOverall()` function of the [`petersenlab`](https://github.com/DevPsyLab/petersenlab) package [@R-petersenlab] provides the option in the function to drop undefined values so you can still generate an estimate of accuracy despite undefined values.

##### Coefficient of Determination ($R^2$) {#sec-coefficientOfDetermination}

The [coefficient of determination](#sec-multipleRegressionRSquared) ($R^2$) reflects the proportion of variance in the outcome (dependent) variable that is explained by the model predictions: $R^2 = \frac{\text{variance explained in }Y}{\text{total variance in }Y}$.
Larger values indicate greater accuracy.

$R^2$ is commonly estimated in [multiple regression](#sec-multipleRegression), in which multiple predictors are allowed to predict one outcome.

###### Adjusted $R^2$ ($R^2_{adj}$) {#sec-adjustedRsquared}

Adjusted $R^2$ is similar to the [coefficient of determination](#sec-coefficientOfDetermination), but it accounts for the number of predictors included in the regression model to penalize [overfitting](#sec-overfitting).
Adjusted $R^2$ reflects the proportion of variance in the outcome (dependent) variable that is explained by the model predictions over and above what would be expected to be accounted for by chance, given the number of predictors in the model.
Larger values indicate greater accuracy.
The formula for adjusted $R^2$ is in @eq-adjustedRSquared.
Adjusted $R^2$ is described further in @sec-multipleRegressionRSquared.

###### Predictive $R^2$ {#sec-predictiveRsquared}

Predictive $R^2$ is described here: <https://tomhopper.me/2014/05/16/can-we-do-better-than-r-squared/> (archived at <https://perma.cc/BK8J-HFUK>).
Predictive $R^2$ penalizes [overfitting](#sec-overfitting), unlike traditional $R^2$.
Larger values indicate greater accuracy.

### Discrimination {#sec-discriminationIndices}

When dealing with a categorical outcome, discrimination is the ability to separate events from non-events.
When dealing with a continuous outcome, discrimination is the strength of the association between the predictor and the outcome.
[Threshold-dependent](#sec-thresholdDependentAccuracy) aspects of [discrimination](#sec-discrimination) at a particular cutoff (e.g., [sensitivity](#sec-sensitivity), [specificity](#sec-specificity)) are described in @sec-thresholdDependentAccuracy.

#### Area under the ROC curve (AUC) {#sec-aucROC}

The [area under the ROC curve (AUC)](#sec-auc) is a general index of discrimination accuracy for a categorical outcome.
It is also called the concordance ($c$) statistic.
Larger values reflect greater [discrimination accuracy](#sec-discrimination).
[AUC](#auc) was estimated using the `pROC` package [@R-pROC].

#### Effect Size ($\beta$) of Regression {#sec-standardizedRegressionCoefficient}

The effect size of a predictor, i.e., the standardized regression coefficient is called a beta ($\beta$) coefficient, is a general index of [discrimination accuracy](#sec-discrimination) for a continuous outcome.
Larger values reflect greater accuracy.
We can obtain standardized regression coefficients by standardizing the predictors and outcome using the `scale()` function in `R`.

### Calibration {#sec-calibrationIndices}

When dealing with a categorical outcome, calibration is the degree to which a probabilistic estimate of an event reflects the true underlying probability of the event.
When dealing with a continuous outcome, calibration is the degree to which the predicted values are close in value to the outcome values.
The importance of examining calibration (in addition to [discrimination](#sec-discrimination)) is described by @Lindhiem2020.
Calibration can be examined in several ways, including [Spiegelhalter's $z$](#sec-spiegelhalterZ) (see @sec-spiegelhalterZ), and the [mean difference between predicted and observed values](#sec-miscalibration) at different binned thresholds as depicted graphically with a [calibration plot](#sec-calibrationPlot) (see @fig-miscalibration).

#### Calibration Plot {#sec-calibrationPlot}

Calibration plots can be helpful for identifying miscalibration.
A calibration plot depicts the predicted probability of an event on the x-axis, and the actual (observed) probability of the event on the y-axis.
The predictions are binned into a certain number of groups (commonly 10).
The diagonal line reflects predictions that are perfectly calibrated.
To the extent that predictions deviate from the diagonal line, the predictions are miscalibrated.

Well-calibrated predictions are depicted in @fig-calibrationPlot:

```{r}
#| label: fig-calibrationPlot
#| fig-cap: "Predictions that are Well-Calibrated. That is, the predicted values are close to the observed values."
#| fig-alt: "Predictions that are Well-Calibrated. That is, the predicted values are close to the observed values."
#| fig-width: 8
#| fig-height: 8
#| code-fold: true

# Specify data
examplePredictionsWellCalibrated <- seq(from = 0, to = 1, by = .1)
exampleOutcomesWellCalibrated <- seq(from = 0, to = 1, by = .1)

# Plot
plot(
  examplePredictionsWellCalibrated,
  exampleOutcomesWellCalibrated,
  xlim = c(0,1),
  ylim = c(0,1),
  xlab = "Predicted Probability",
  ylab = "Observed Proportion",
  bty = "l",
  type = "n")

lines(
  c(0,1),
  c(0,1),
  lwd = 2,
  col = "#377eb8")

points(
  examplePredictionsWellCalibrated,
  exampleOutcomesWellCalibrated,
  cex = 1.5,
  col = "#e41a1c",
  lwd = 2,
  type = "p")
```

The various types of general miscalibration are depicted in @fig-miscalibration:

```{r}
#| label: fig-miscalibration
#| fig-cap: "Types of Miscalibration. From @Petersen2024a and @PetersenPrinciplesPsychAssessment."
#| fig-alt: "Types of Miscalibration. From @Petersen2024a and @PetersenPrinciplesPsychAssessment."
#| fig-width: 8
#| fig-height: 8
#| code-fold: true

# Specify data
examplePredictions <- seq(from = 0, to = 1, by = .1)
exampleOutcomes <- c(0, .15, .3, .4, .45, .5, .55, .6, .7, .85, 1)

overPrediction <- c(0, .02, .05, .1, .15, .2, .3, .4, .5, .7, 1)
underPrediction <- c(0, .3, .5, .6, .7, .8, .85, .9, .95, .98, 1)
overExtremity <- c(0, .3, .38, .42, .47, .5, .53, .58, .62, .7, 1)
underExtremity <- c(0, .05, .08, .11, .2, .5, .8, .89, .92, .95, 1)

# Plot
par(
  mfrow = c(2,2),
  mar = c(5,4,1,1) + 0.1) #margins: bottom, left, top, right

plot(
  examplePredictions,
  overExtremity,
  xlim = c(0,1),
  ylim = c(0,1),
  main = "Overextremity",
  xlab = "Predicted Probability",
  ylab = "Observed Proportion",
  bty = "l",
  cex = 1.5,
  col = "#e41a1c",
  type = "o")

lines(
  c(0,1),
  c(0,1),
  lwd = 2,
  col = "#377eb8")

plot(
  examplePredictions,
  underExtremity,
  xlim = c(0,1),
  ylim = c(0,1),
  main = "Underextremity",
  xlab = "Predicted Probability",
  ylab = "Observed Proportion",
  bty = "l",
  cex = 1.5,
  col = "#e41a1c",
  type = "o")

lines(
  c(0,1),
  c(0,1),
  lwd = 2,
  col = "#377eb8")

plot(
  examplePredictions,
  overPrediction,
  xlim = c(0,1),
  ylim = c(0,1),
  main = "Overprediction",
  xlab = "Predicted Probability",
  ylab = "Observed Proportion",
  bty = "l",
  cex = 1.5,
  col = "#e41a1c",
  type = "o")

lines(
  c(0,1),
  c(0,1),
  lwd = 2,
  col = "#377eb8")

plot(
  examplePredictions,
  underPrediction,
  xlim = c(0,1),
  ylim = c(0,1),
  main = "Underprediction",
  xlab = "Predicted Probability",
  ylab = "Observed Proportion",
  bty = "l",
  cex = 1.5,
  col = "#e41a1c",
  type = "o")

lines(
  c(0,1),
  c(0,1),
  lwd = 2,
  col = "#377eb8")
```

However, predictions could also be miscalibrated in more specific ways.
For instance, predictions could be well-calibrated at all predicted probabilities except for a given predicted probability (e.g., 20%).
Or, the predictions could be miscalibrated but not systematically over- or underpredicted.
Thus, it is important to evaluate a calibration plot to evaluate the extent to which the predictions are miscalibrated and the pattern of that miscalibration.

#### Spiegelhalter's *z* {#sec-spiegelhalterZ}

Spiegelhalter's *z* was calculated using the `rms` package [@R-rms].
Smaller *z* values (and larger associated *p*-values) reflect greater [calibration accuracy](#calibration).
A statistically significant Spiegelhalter's *z* (*p* < .05) indicates a significant degree of miscalibration.

#### Calibration for predicting a continuous outcome {#sec-calibrationContinuous}

When predicting a continuous outcome, [calibration](#sec-calibration) of the predicted values in relation to the outcome values can be examined in multiple ways including:

- in a [calibration plot](#sec-calibrationPlot), the extent to which the intercept is near zero and the slope is near one
- in a [calibration plot](#sec-calibrationPlot), the extent to which the 95% confidence interval of the observed value, across all values of the predicted values, includes the diagonal reference line with an intercept of zero and a slope of one
- [mean error](#sec-meanError)
- [mean absolute error](#sec-meanAbsoluteError)
- [mean squared error](#sec-meanSquaredError)
- [root mean squared error](#sec-rootMeanSquaredError)

With a plot of the predictions on the x-axis, and the outcomes on the y-axis (i.e., a [calibration plot](#sec-calibrationPlot)), [calibration](#sec-calibration) can be examined graphically as the extent to which the best-fit regression line has an intercept (alpha) close to zero and a slope (beta) close to one [@Steyerberg2014; @Stevens2020].
The intercept is also called "calibration-in-the-large", whereas "calibration-in-the-small" refers to the extent to which the predicted values match the observed values at a specific predicted value (e.g., when the weather forecaster says that there is a 10% chance of rain, does it actually rain 10% of the time?).
For predictions to be well [calibrated](#sec-calibration), the intercept should be close to zero and the slope should be close to one.
If the slope is close to one but the intercept is not close to zero (or the intercept is close to zero but the slope is not close to one), the predictions would not be considered well [calibrated](#sec-calibration).
The 95% confidence interval of the observed value, across all values of the predicted values, should include the diagonal reference line whose intercept is zero and whose slope is one.

For instance, based on the intercept and slope of the [calibration plot](#sec-calibrationPlot) in Figure INSERT, the predictions are not well calibrated, despite having a slope near one, because the 95% confidence interval of the intercept does not include zero.
The best-fit line is the yellow line.
The intercept from the best-fit line is positive, as shown in the regression equation.
This is a case of underprediction, where the predicted values are consistently less than the observed values.
The confidence interval of the observed value (i.e., the purple band) is the interval within which we have 95% confidence that the true observed value would lie for a given predicted value, based on the model
The 95% prediction interval of the observed value (i.e., the dashed red lines) is the interval within which we would expect that 95% of future observations would lie for a given predicted value.
The black diagonal line indicates the reference line with an intercept of zero and a slope of one.
The predictions would be significantly [miscalibrated](#sec-calibration) at a given level of the predicted values if the 95% confidence interval of the observed value does not include the reference line at that level of the predicted value.
In this case, the 95% confidence interval of the observed value does not include the reference line (i.e., the actual observed value) at lower levels of the predicted values, so the predictions are [miscalibrated](#sec-calibration) lower levels of the predicted values.

Gold-standard recommendations include examining the predicted values in relation to the observed values using locally estimated scatterplot smoothing (LOESS) [@Austin2014], such as in Figure INSERT.
We can examine whether the LOESS-based 95% confidence interval of the observed value at every level of the predicted values includes the diagonal reference line (i.e., the actual observed value).
In this case, the 95% confidence interval of the observed value does not include the reference line at lower levels of the predicted values, so the predictions are [miscalibrated](#sec-calibration) at lower levels of the predicted values.

## Integrating the Accuracy Indices {#sec-integratingAccuracy}

After computing the accuracy indices of [discrimination](#sec-discrimination) and (2) [calibration](#sec-calibration), it is then the task to integrate the indices to determine (a) which are the most accurate predictions for the given goals, and (b) whether additional improvements and refinements to the predictions need to be made.
Each of the accuracy indices is computed differently and thus reward (and penalize) predictive (in)accuracy differently.
Sometimes, the the accuracy indices will paint a consistent picture regarding which predictions are the most accurate.
Other times, the accuracy indices may disagree about which predictions are most accurate.

In fantasy football, when evaluating the accuracy of seasonal projections, we care most about accurately distinguishing between higher levels of points (e.g., 200 vs 150) as opposed to lower levels of points (e.g., 0 vs 10).
Thus, it can be helpful to punish larger errors more heavily than smaller errors, as [RMSE](#sec-rootMeanSquaredError) (unlike [MAE](#sec-meanAbsoluteError)).

Thus, we would emphasize the following metrics:

- [discrimination](#sec-discrimination):
  - [adjusted $R^2$](#sec-adjustedRsquared)
- [calibration](#sec-calibration):
  - [calibration plot](#sec-calibrationPlot)
- [general accuracy](#sec-calibration):
  - [MAE](#sec-meanAbsoluteError)
  - [RMSE](#sec-rootMeanSquaredError)

If you focus on only one accuracy index, [MAE](#sec-meanAbsoluteError) or [RMSE](#sec-rootMeanSquaredError) would be a good choice.
However, I would also examine a [calibration plot](#sec-calibrationPlot) to evaluate whether predictions are poorly calibrated at higher levels of points.
I would also examine [ME](#sec-meanError)—not to compare the accuracy of various predictions per se—but to determine whether predictions are systematically under- or overestimating actual points.
If so, predictions may be able to be refined by adding or subtracting a constant to the predictions (or to a subset of the predictions); however, this could worsen other accuracy indices, so it is important to conduct an iterative process of modifying then evaluating, then further modifying and evaluating, etc.
It may also be valuable to evaluate the accuracy of various subsets of the predictions.
For instance, you might examine the predictive accuracy of players whose projected points are greater than 100, to evaluate the accuracy of predictions specifically to distinguish between players at higher levels of points, which is one of the key goals when selecting which players to draft.

If we are making predictions about a categorical variable, we would emphasize the following metrics:

- [discrimination](#sec-discrimination):
  - [area under the receiver operating curve](#sec-auc)
  - and, secondarily—depending on the particulary cutoff and the relative costs of [false positives](#sec-falsePositive) versus [false negatives](#sec-falseNegative):
    - [sensitivity](#sec-sensitivity)
    - [specificicity](#sec-specificity)
    - [positive predictive value](#sec-ppv)
    - [negative predictive value](#sec-npv)
- [calibration](#sec-calibration):
  - [calibration plot](#sec-calibrationPlot)
  - [Spiegelhalter's *z*](#sec-spiegelhalterZ)
  - [Mean difference between observed and predicted values](#sec-miscalibration)

## Theory Versus Empiricism {#sec-theoryVsEmpiricism}

One question that inevitably arises when making predictions is the extent to which one should leverage theory versus empiricism.
Theory involves conceptual claims of understanding how the causal system works (i.e., what influences what).
For example, use of theory in prediction might involve specification of the causal system that influences player performance, measurement of those factors, and the integration of that information to make a prediction.
Empiricism involves "letting the data speak for themselves" and is an atheoretical approach.
For example, empiricism might involve examining how thousands of variables are associated with the criterion of interest (e.g., fantasy points) and developing the best-fitting model based on those thousands of predictor variables.

Although the atheoretical approach can perform reasonably well, it can be improved by making better use of theory.
An empirical result (e.g., a correlation) might not necessarily have a lot of meaning associated with it.
As the maxim goes, [correlation does not imply causation](#sec-correlationCausality).
Moreover, empiricism can lead to [overfitting](#sec-overfitting).
So, empiricism is often not enough.

As @Silver2012 notes, "The numbers have no way of speaking for themselves.
We speak for them.
We imbue them with meaning." (p. 9).
If we *understand* the variables in the system and how they influence each other, we can predict things more accurately than predicting for the sake of predicting.
For instance, [we have made great strides in the last decades when it comes to more accurate weather forecasts](https://www.npr.org/sections/money/2023/07/11/1186458991/should-we-invest-more-in-weather-forecasting-it-may-save-your-life) (archived at <https://perma.cc/PF8P-BT3D>), including extreme weather events like hurricanes.
These great strides have more to do with a better causal understanding of the weather system and the ability to conduct simulations of the atmosphere than merely because of big data [@Silver2012].
By contrast, other events are still incredibly difficult to predict, including earthquakes, in large part because we do not have a strong understanding of the system (and because we do not have ways of precisely measuring those causes because they occur at a depth below which we are realistically able to drill) [@Silver2012].

At the same time, in the social and behavioral sciences, our theories of the causal processes that influence outcomes are not yet very strong.
Indeed, I have misgivings calling them theories because they do not meet the traditional scientific standard for a theory.
A scientific theory is an explanation of the natural world that is testable and falsifiable, and that has withstood rigorous scientific testing and scrutiny.
In psychology (and other areas of social and behavioral sciences), our "theories" are more like conceptual frameworks.
And these conceptual frameworks are often vague, do not make specific predictions of effects *and* noneffects, and do not hold up consistently when rigorously tested.
As described by @Meehl1978:

> I consider it unnecessary to persuade you that most so-called "theories" in the soft areas of psychology (clinical, counseling, social, personality, community, and school psychology) are scientifically unimpressive and technologically worthless ... Perhaps the easiest way to convince yourself is by scanning the literature of soft psychology over the last 30 years and noticing what happens to theories.
> Most of them suffer the fate that General MacArthur ascribed to old generals—They never die, they just slowly fade away.
> In the developed sciences, theories tend either to become widely accepted and built into the larger edifice of well-tested human knowledge or else they suffer destruction in the face of recalcitrant facts and are abandoned, perhaps regretfully as a "nice try."
> But in fields like personology and social psychology, this seems not to happen.
> There is a period of enthusiasm about a new theory, a period of attempted application to several fact domains, a period of disillusionment as the negative data come in, a growing bafflement about inconsistent and unreplicable empirical results, multiple resort to ad hoc excuses, and then finally people just sort of lose interest in the thing and pursue other endeavors.
> 
> Meehl [-@Meehl1978, pp. 806–807]

Even if we had strong theoretical understanding of the causal system that influences behavior, we would likely still have difficulty making accurate predictions because the field has largely relied on relatively crude instruments.
According to one philosophical perspective known as LaPlace's demon, if we were able to know the exact conditions of everything in the universe, we would be able to know how the conditions would be in the future.
This is an example of scientific determinism, where if you know the initial conditions, you also know the future.
Other perspectives, such as quantum mechanics and chaos theory, would say that, even if we knew the initial conditions with 100% certainty, there would still be uncertainty in our understanding of the future.
But assume, for a moment, that LaPlace's demon is true.
A challenge in the social and behavioral sciences is that we have a relatively poor understanding of the initial conditions of the universe.
Thus, our predictions would necessarily be probabilistic, similar to weather forecasts.
Despite having a strong understanding of how weather systems behave, we have imperfect understanding of the initial conditions (e.g., the position and movement of all molecules) [@Silver2012].

Theories tend to make grand conceptual claims that one observed variable influences another observed variable through a complex chain of intervening processes that are unobservable.
Empiricism provides rich lower-level information, but lacks the broader picture.
So, it seems, that we need both theory *and* empiricism.
Theory and empiricism can—and should—inform each other.

## Test Bias {#sec-testBias}

Test bias refers to systematic error (in measurement, prediction, etc.) as a function of group membership that leads the same score to have different meaning for different groups.
For instance, if the Wonderlic Contemporary Cognitive Ability Test is a strong predictor of performance for Quarterbacks but not for Running Backs, the test is biased.
Test bias, including how to identify and address it, is described in [@PetersenPrinciplesPsychAssessment](https://isaactpetersen.github.io/Principles-Psychological-Assessment/bias.html).

## Ways to Improve Prediction Accuracy {#sec-waysToImprovePredictionAccuracy}

On the whole, experts' predictions are inaccurate.
Experts' predictions from many different domains tend to be inaccurate, including political scientists [@Tetlock2017], physicians [@Koehler2002], clinical psychologists [@Oskamp1965], stock market traders and corporate financial officers [@Skala2008], seismologists' predictions of earthquakes [@Hough2016], economists' predictions about the economy [@Makridakis2009], lawyers [@Koehler2002], and business managers [@Russo1992].
Thus, I would not put much confidence in the predictions by fantasy football fundits.
The most common pattern of experts' predictions is that they show overextremity, that is, their predictions have probability judgments that tend to be too extreme, as described in Section @sec-calibration.
Overextremity of experts' predictions reflects the [overprecision](#sec-cognitiveBiasesOverconfidence) type of [overconfidence bias](#sec-cognitiveBiasesOverconfidence).
The degree of confidence of a person's predictions is often not a good indicator of the accuracy of their predictions [and confidence and prediction accuracy are sometimes inversely associated; @Silver2012].
[Heuristics](#sec-heuristics) such as the [anchoring and adjustment heuristic](#sec-heuristicsAnchoringAdjustment), [cognitive biases](#sec-cognitiveBiases) such as [confirmation bias](#sec-cognitiveBiasesConfirmation) [@Hoch1985; @Koriat1980], [fallacies](#sec-fallacies) such as the [base rate fallacy](#sec-fallaciesBaseRate) [@Eddy1982; @Koehler2002] could contribute to overconfidence of predictions.
[Poorly calibrated](#sec-calibration) predictions are especially likely when the [base rate](#sec-baseRate) is very low (e.g., suicide) or when the [base rate](#sec-baseRate) is very high [@Koehler2002].

Nevertheless, there are some domains that have shown greater predictive accuracy, from which we may learn what practices may lead to greater accuracy.
For instance, experts have shown stronger predictive accuracy in weather forecasting [@Murphy1984], horse race betting [@Johnson2001], and playing the card game of bridge [@Keren1987], but see @Koehler2002 for exceptions.

Here are some potential ways to improve the accuracy (and honesty) of predictions and judgments:

- Provide appropriate [anchoring of your predictions to the base rate](#sec-accountForBaseRates) of the phenomenon you are predicting.
To the extent that the [base rate](#sec-baseRate) of the event you are predicting is low, more extreme evidence should be necessary to consistently and accurately predict that the event will occur.
Applying [actuarial formulas](#sec-actuarialPrediction) and [Bayes' theorem](#sec-bayesTheorem) can help you appropriately weigh the [base rate](#sec-baseRate) and evidence.
- Include multiple predictors, ideally from different measures and measurement methods.
Include the predictors with the strongest validity based on theory of the causal process and based on [criterion-related validity](#sec-criterionValidity).
- When possible, aggregate multiple perspectives of predictions, especially predictions made independently (from different people/methods/etc.).
The "wisdom of the crowd" is often more accurate than individuals' predictions, including predictions by so-called "experts" [@Silver2012].
- A goal of prediction is to capture as much signal as possible and as little noise (error) as possible [@Silver2012].
Parsimony (i.e., not having too many predictors) can help reduce the amount of error variance captured by the prediction model.
However, to accurately model complex systems like human behavior, complex models may be necessary.
However, strong theory of the causal processes and dynamics may be necessary to develop accurate complex models.
- Although incorporating theory can be helpful, provide more weight to empiricism than to theory, until our theories and measures are stronger.
Ideally, we would use theory to design a model that mirrors the causal system, with accurate measures of each process in the system, so we could make accurate predictions.
However, as described in @sec-theoryVsEmpiricism, our psychological theories of the causal processes that influence behavior are not yet very strong.
Until we have stronger theories that specify the causal process for a given outcome, and until we have accurate measures of those causal processes, [actuarial approaches](#sec-actuarialPrediction) are likely to be most accurate, as discussed in @sec-judgmentVsActuarial.
At the same time, keep in mind that measures involving human behavior, and their resulting data, are often noisy.
As a result, theoretically (conceptually) informed empirical approaches may lead to more accuracy than empiricism alone.
- Use an empirically validated and cross-validated [statistical algorithm](#sec-actuarialPrediction) to combine information from the predictors in a formalized way.
Give each predictor appropriate weight in the statistical algorithm, according to its strength of association with the outcome.
Use measures with strong [reliability](#sec-reliability) and [validity](#sec-validity) for assessing these processes to be used in the algorithm.
Cross-validation will help reduce the likelihood that your model is fitting to noise and will maximize the likelihood that the model predicts accurately when applied to new data (i.e., the model's predictions accurately generalize), as described in @sec-bestActuarialApproaches.
- When presenting your predictions, acknowledge what you do not know.
- Express your predictions in terms of probabilistic estimates and present the uncertainty in your predictions with confidence intervals [even though bolder, more extreme predictions tend to receive stronger television ratings; @Silver2012].
- Qualify your predictions by identifying and noting counter-examples that would not be well fit by your prediction model, such as extreme cases, edge cases, and "broken leg" [@Meehl1957] cases.
- Provide clear, consistent, and timely feedback on the outcomes of the predictions to the people making the predictions [@Bolger2004].
- Be self-critical about your predictions.
Update your judgments based on their accuracy, rather than trying to confirm your beliefs [@Atanasov2020].
- In addition to considering the accuracy of the prediction, consider the quality of the prediction *process*, especially when random chance is involved to a degree, such as in poker and fantasy football [@Silver2012].
- Work to identify and mitigate potential blindspots; be aware of [cognitive biases](#sec-cognitiveBiases) and [fallacies](#sec-fallacies), such as [confirmation bias](#sec-cognitiveBiasesConfirmation) and the [base rate fallacy](#sec-fallaciesBaseRate).
- Evaluate for the possibility of [test bias](#sec-testBias).
Correct for any [test bias](#sec-testBias).

## Conclusion {#sec-predictionAccuracyConclusion}

When the [base rate](#baseRate) of a behavior is very low or very high, you can be highly accurate in predicting the behavior by [predicting from the base rate](#sec-predictingFromBaseRate).
Thus, you cannot judge how accurate your prediction is until you know how accurate your predictions would be by [random chance](#sec-accuracyByChance).
Moreover, maximizing [percent accuracy](#sec-percentAccuracy) may not be the ultimate goal because [different errors have different costs](#sec-differentErrorsDifferentCosts).
Though there are many indices of accuracy, there are two general types of accuracy: [discrimination](#sec-discrimination) and [calibration](#sec-calibration).
[Discrimination](#sec-discrimination) accuracy is frequently evaluated with the [area under the receiver operating characteristic curve](#sec-auc), or with [sensitivity](#sec-sensitivity) and [specificity](#sec-specificity), or with [standardized regression coefficients](#sec-standardizedRegressionCoefficient) or the [coefficient of determination](#sec-multipleRegressionRSquared).
[Calibration](#sec-calibration) accuracy is frequently evaluated graphically and with various indices.
[Sensitivity](#sec-sensitivity) and [specificity](#sec-specificity) [depend on the cutoff](#sec-accuracyCutoff).
It is important to evaluate both [discrimination](#sec-discrimination) and [calibration](#sec-calibration) when evaluating prediction accuracy.

::: {.content-visible when-format="html"}

## Session Info {#sec-predictionAccuracySessionInfo}

```{r}
sessionInfo()
```

:::