README.Rmd

---
title: "Code and Data for: Reproduce to Validate: a Comprehensive Study on the Reproducibility of Economics Research"
author:
  - Sylverie Herbert
  - Hautahi Kingi
  - Flavio Stanchi
  - Lars Vilhuber
date: "`r Sys.Date()`"
doi: "10.5683/SP3/GJVVLI"
output:
  html_document: 
    keep_md: yes
    number_sections: yes
    toc: yes
    toc_depth: 1
  word_document: default
  pdf_document: 
    toc: true
    number_sections: true
    toc_depth: 1
editor_options: 
  chunk_output_type: console
bibliography: 
  - text/data.bib
  - text/includes/paper.bib
  - grateful-refs.bib
csl: _readme/chicago-author-date.csl
---

> File prepared for **Canadian Journal of Economics**.


[![DOI:10.5683/SP3/GJVVLI](https://zenodo.org/badge/DOI/10.5683/SP3/GJVVLI.svg)](https://doi.org/10.5683/SP3/GJVVLI)

# Cite as

> Herbert, Sylvérie; Kingi, Hautahi; Stanchi, Flavio; Vilhuber, Lars, 2024, "Replication Data and Code for: Reproduce to Validate: a Comprehensive Study on the Reproducibility of Economics Research", https://doi.org/10.5683/SP3/GJVVLI, Borealis, V1, UNF:6:KAT43A8y80Ct+4PyDB3XYw== [fileUNF] 


```{r setup,echo=FALSE, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
knitr::opts_chunk$set(eval=FALSE)

source(file.path(rprojroot::find_rstudio_root_file(),"pathconfig.R"),echo=FALSE)

we.are.README <- TRUE

source(file.path(basepath,"global-libraries.R"),echo=FALSE)
#source(file.path(programs,"libraries.R"), echo=FALSE)
source(file.path(programs,"config.R"), echo=FALSE)

# we need two packages just for the Readme
source(file.path(basepath,"readme-libraries.R"), echo=FALSE)


library(knitr)
library(readr)
library(dplyr)
library(tidyr)
library(readxl)
library(stringr)

```


# Overview

The code in this replication package constructs the analysis file from the four data sources [@raw-data-2019; @openalex-data; @webofscience2018; @crossref-data]. The code is in R, but also uses public APIs to download some data. The replication package also includes PDF copies the survey used to query the replicators. The original Google Forms surveys cannot be exported. 

# Data Availability and Provenance Statements

All data are public, and can be redistributed. The key data [@raw-data-2019] are provided in a separate repository ([https://doi.org/10.5281/zenodo.`r zenodo.id`](https://doi.org/10.5281/zenodo.`r zenodo.id`)), but are downloaded automatically. Data pulled via APIs are provided as of the date last downloaded, with no guarantees that the code to access the API still works.

## Statement about Rights

- [x] I certify that the author(s) of the manuscript have legitimate access to and permission to use the data used in this manuscript. 
- [x] I certify that the author(s) of the manuscript have documented permission to redistribute/publish the data contained within this replication package. Appropriate permission are documented in the [LICENSE.txt](LICENSE.txt) file.


## License for Data

- Original data [@raw-data-2019] (not included) are licensed as [CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/legalcode).
- Data from @openalex-data are obtained under a [CC0](https://creativecommons.org/publicdomain/zero/1.0/) Public Domain attribution.
- Data from @webofscience2018 are proprietary, but redistribution of small extracts is permitted.
- Data fom @crossref-data is "is open and available for reuse without restriction" ([https://www.crossref.org/documentation/retrieve-metadata/](https://www.crossref.org/documentation/retrieve-metadata/))

All derivative data contained herein, if not otherwise encumbered, is available under a [CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/legalcode) license. Usage by commercial entities is permitted, reselling the data is not.


## Summary of Availability

- [x] All data **are** publicly available.
- [ ] Some data **cannot be made** publicly available.
- [ ] **No data can be made** publicly available.


```{r summary_data,echo=FALSE,eval=TRUE}
datafiles <- read_excel("_readme/datafiles.xlsx") %>%
  arrange(Order,`Data Source`) %>%
# find files present
  mutate(Provided.real = file.exists(Filename),
         Provided.norm = (Provided == "Yes"),
         congruence   = Provided.real == Provided.norm)

table(datafiles$congruence) %>% as.data.frame() -> check.files

check.files.true <- check.files %>% filter(Var1=="TRUE") %>% pull(Freq)
check.files.failure <- nrow(datafiles) - check.files.true
if ( is.numeric(check.files.failure) & check.files.failure > 0 ) {
  warning(paste0(check.files.failure," files fail check - are present but should be absent"))
}
check.files.duplicates <- datafiles %>% distinct(Filename) 
check.files.duplicates.n <- nrow(datafiles) - nrow(check.files.duplicates)

if ( is.numeric(check.files.duplicates.n) & check.files.duplicates.n > 0 ) {
  warning(paste0(check.files.duplicates.n," files are duplicates - error!"))
}

```
```{r print_check,echo=FALSE,results='asis',eval=TRUE}


if ( is.numeric(check.files.failure) & check.files.failure > 0 ) {
   cat('### Files failing check')
   cat(' ')
   datafiles %>% 
    filter(!congruence) %>%
    select(`Data Source`,Filename,Provided.real,Provided.norm) %>%
  kable()
}

```

```{r print_check_dups,echo=FALSE,results='asis',eval=TRUE}

if ( is.numeric(check.files.duplicates.n) & check.files.duplicates.n > 0 ) {
  cat('### Files that are duplicates')
   cat(' ')
   datafiles %>% 
     group_by(Filename) %>%
     summarise(count=n()) %>%
     filter(count>1) %>%
     left_join(datafiles %>% select(`Data Source`,Filename)) %>%
    select(`Data Source`,Filename,count) %>%
  kable()
}
```

```{r show_table,echo=FALSE,results='asis',eval=TRUE}


datafiles %>% select(`Data Source`,Filename,Provided = Provided.real) %>%
  kable()
```



## Details on each Data Source

### @raw-data-2019 

The data were collected through the methods described in the paper. Data were collected over several years, and deposited at @raw-data-2019 in 2019, with replicator names replaced by random identifiers. 

```{r own_data,echo=FALSE,eval=TRUE,results='asis'}

datafiles %>% 
  filter(Order==1|Order==11) %>%
  select(`Data Source`,Filename,Provided = Provided.real) %>%
  kable()
```


### Crossref data

Crossref were extracted as needed to obtain bibliographic information (author names, article titles, publication dates) [@crossref-data], using the `rcrossref` package to query the API `. The database itself is free to access. More information about it can be read in @crossref-paper . Note that data can and is updated, so running the query again 

```{r crossref_data,echo=FALSE,eval=TRUE,results='asis'}

datafiles %>% 
  filter(Order==2|Order==12) %>%
  select(`Data Source`,Filename,Provided = Provided.real) %>%
  kable()
```


### openAlex data

Data were accessed in 2023 to increase the time series covered by the bibliometric analysis [@openalex-data]. The openAlex database is described in @openalex2022. OpenAlex data are accessed via an API, and computed statistics, such as the h-index, are limited to the past 10 years. We therefore had to recompute some numbers. We also saved our extract, as future extracts may have different numbers, due to improvements in entity disambiguation (author names) and other factors outside of our control.

Not all data elements are complete in the openAlex data. We output data for various quality checks, and did manual research to "impute" attributes such as affiliations. All "overrides" are captured in a separate file. 

Thus, the openAlex data has (a) the raw data as downloaded; (b) the problematic data, as output for manual review (c) the edited/imputed data as used to complement the downloaded data.

```{r openalex_data,echo=FALSE,eval=TRUE,results='asis'}

datafiles %>% 
  filter(Order==3|Order==13) %>%
  select(`Data Source`,Filename,Provided = Provided.real) %>%
  kable()
```

### Web of Science data

The bibliometric information, in particular on h-index, was originally extracted from Web of Science [@WebofScience;@webofscience2018], and was used in the early working papers [@kingi2018;@herbert2021]. In response to referee comments, the primary bibliometric analysis shifted to the use of openAlex data (described next). OpenAlex data is compared to WoS data in the appendix.

WoS is proprietary data and requires a subscription. We accessed the interface thanks to Cornell's subscription.

Data were manually extracted from the WoS web interface, with RAs following guidance on what data to extract (see [`data/h_index_data/README.md`](data/h_index_data/README.md)). The collated data is provided in XLSX and CSV formats.


```{r wos_data,echo=FALSE,eval=TRUE,results='asis'}

datafiles %>% 
  filter(Order==4|Order==14) %>%
  select(`Data Source`,Filename,Provided = Provided.real) %>%
  kable()
```


# Computational requirements


## Software Requirements

```{r prepare_sys,echo=FALSE,eval=TRUE}
# Read the rocker image
readLines(file.path("Dockerfile")) %>% as.data.frame() -> dockerfile
names(dockerfile) <- "line"
dockerfile %>% filter(str_detect(line,"FROM")) %>% separate(line,into=c("key","value"),sep=" ") %>% pull(value) -> docker.image

# find libraries
as.data.frame(list.files(".",pattern="*.R$",recursive=TRUE)) -> library.files
names(library.files) <- "file"



```

- [x] The replication package contains one or more programs to install all dependencies and set up the necessary directory structure. 

- `r version$version.string` on `r version$system`
  - Docker image is used (see appendix), with system libraries defined by the relevant image (`r docker.image`) (optional, but recommended)
  - RSPM (now [Posit Package Manager, PPM](https://packagemanager.posit.co/client/)) is used, set to `r mran.date`. All libraries are installed from that time-stamped repository.


```{r list_libs,echo=FALSE,eval=TRUE,results='asis'}
library.files %>% filter(str_detect(file,"librar")) %>% rename(`Libraries are defined in` = file) %>% kable()

```

### Software citations

Only directly loaded libraries are cited. For all libraries used in the runtime environment, see the Appendix.

```{r create_r_cites,echo=FALSE,results='asis',eval=TRUE,warning=FALSE}
# This needs to be run once by hand, and will be fine in the second round.
cite_packages(output = "table", out.format=c("md"), out.dir = ".") %>% kable()
```


## Controlled Randomness



- [ ] Random seed is set at line _____ of program ______
- [x] No Pseudo random generator is used in the analysis described here.

Note that re-running the API queries (turned off by default) **will** generate different data, which is likely to affect the regression output.

## Memory, Runtime, Storage Requirements

### Summary

Approximate time needed to reproduce the analyses on a standard (CURRENT YEAR) desktop machine (when not running API queries):

- [ ] <10 minutes
- [x] 10-60 minutes
- [ ] 1-2 hours
- [ ] 2-8 hours
- [ ] 8-24 hours
- [ ] 1-3 days
- [ ] 3-14 days
- [ ] > 14 days

API queries can take a long time, are not guaranteed to work, and are not guaranteed to return the same results. All API queries are stored as of the last run, and made available in the replication package.

Approximate storage space needed:

- [ ] < 25 MBytes
- [x] 25 MB - 250 MB
- [ ] 250 MB - 2 GB
- [ ] 2 GB - 25 GB
- [ ] 25 GB - 250 GB
- [ ] > 250 GB

- [ ] Not feasible to run on a desktop machine, as described below.

### Details

The code was last run on 

- OS: "openSUSE Leap 15.5"
- Processor:  AMD Ryzen 9 3900X 12-Core Processor, 24 cores
- Memory available: 31GB memory
- Docker version 24.0.7-ce, build 311b9ff0aa93
- `r version$version.string` on `r version$system`

as well as a Macbook with 16GB of memory. At least one program failed when run with less than 16GB of memory.


# Description of programs/code

Each numbered R program can be run independently, in the sequence implied by the numbering scheme. 
A convenience bash main script (`run.sh`) to run all programs is provided in the root of the project, and will run all data cleaning and analysis programs.

Of note, several programs leverage APIs, which can yield different results, and might take  long time. The programs will detect previously downloaded data files, and skip the download part if those files are present. To start with a fresh download, delete the following files:

- `r paste0(crossref.file,".Rds")` - Crossref extract.
- `r bibtex_all` - bibtex file of articles analyzed
- `r openalex.Rds` - extract of articles analyzed from openAlex
- `r openalex.authors.Rds` - bibliometric information from openAlex on all the authors of articles analyzed
- `r doi.file.Rds` - bibliographic information from Crossref for later analysis

Internet access is required to run the programs, 

- to download the key input files [@raw-data-2019]
- when re-running queries to the APIs

The programs can be run without internet access if manually downloaded files from [https://doi.org/10.5281/zenodo.`r zenodo.id`](https://doi.org/10.5281/zenodo.`r zenodo.id`) are placed into `dataloc`.

##  License for Code


The code (all files ending in `.R`, `.Rmd`, and `.sh`) is licensed under a BSD license. See [LICENSE.txt](LICENSE.txt) for details.

# Instructions to Replicators

If using Docker image on Linux or macOS system:

- run `start_rstudio.sh` and connect to [https://localhost:8787](https://localhost:8787)
- in the "Terminal" of the RStudio app, run `bash ./run.sh`

or equivalently,

- in the terminal of a computer with Docker installed, run `bash ./run_docker.sh ./run.sh`

Alternative ways to run this (these were not tested):

- optionally, before running project code, run `Rscript -e "renv::init()"` (on Windows, `Rscript.exe -e "renv::init()"` ) to isolate the project libraries from your system (assumes `renv` is installed, see [renv](https://rstudio.github.io/renv/articles/renv.html)).
- Using the same R version as described above, run each program individually as desired, in the order indicated above.

## Details

```{r program_details, echo=FALSE,eval=TRUE,results='asis'}
programs_info <- read_excel("_readme/programs-info.xlsx")
kable(programs_info)
```


# List of tables and programs


The provided code reproduces:

- [x] All numbers provided in text in the paper
- [x] All tables and figures in the paper
- [ ] Selected tables and figures in the paper, as explained and justified below.

The code also produces numerous tables which were not included in the paper. Note that following a request from copy-editors, all tables used in the paper were subsumed into the main document. Earlier versions used `\input` statements in LaTeX.

```{r table_details, echo=FALSE,eval=TRUE,results='asis',warning=FALSE,message=FALSE}
# we constructed this in two takes
# - parse all Latex files for their label and filename
# - parse all R code for the word "table", to get the filename
#  'grep table_ */*.R | sed 's+out=file.path(Outputs,"+text/analysis/+' | sed 's+=file.path(TexIncludes,"+text/includes/+' | sed 's+"))++' | sed 's+")++' | sed 's+:+,+' | sed 's+outtext+text+' > _readme/tables_programs.csv'
# - some hand-cleaning

tables_programs <- read_csv("_readme/tables_programs.csv", 
    col_names = FALSE) 
names(tables_programs) <- c("Program","LaTeX file")

table_info <- read_excel("_readme/table-mapping.xlsx") %>%
  mutate(number = as.numeric(str_remove(`Table number`,"A")),
         appendix = str_detect(`Table number`,"A")) %>%
  filter(!is.na(`Table number`)) %>%
  group_by(appendix)%>%
  arrange(number,.by_group = TRUE) %>%
  ungroup()

programs_table_info <- left_join(table_info,tables_programs,by="LaTeX file") %>%
  select(`Table number`,Program,`LaTeX file`)

kable(programs_table_info )
```

---

# Acknowledgements

This README based on the template created by @template-readme. 

# References


<div id="refs"></div>

#  Appendix {-} 

## Appendix: System and package info


```{r sysinfo,eval=TRUE,echo=FALSE}
# we don't want to capture what the Rmarkdown session is running, but rather, what the computational session is running
if (! file.exists(file.path(programs,"99_zz_info.txt"))) {
system("cd programs && R CMD BATCH 99_zz_info.R")
}
```

```{r includetext,echo=FALSE,eval=TRUE}
cat(readLines(file.path(programs,"99_zz_info.txt")), sep = '\n')

```

## Appendix: Dockerfile


```{r includedocker,echo=FALSE,eval=TRUE}
cat(readLines(file.path("Dockerfile")), sep = '\n')
```