To do for Module

[abogdanoski]

Book

• Replace "output" with "display item" throughout; clarify the distinction between display item, claim, and estimate @fhoces
• “Closed for reproductions”: clarify what happens if the author later on releases the data? @fhoces
• Number the stages to imply chronological order of each
• Reproducibility scale: restructure scores around three reference points: L1=no data, no code -- L5 CRR -- L10 CRA
• Reproducibility scale: translate levels for admin/restricted-access data @fhoces
• Set up a survey/voting system for reproducibility level, but highlight that our value judgment is that data is more important than code for reproducibility
• Write an email to authors requestion reproduction material (line 48) @abogdanoski
• Capitalize ACRE Diagram Builder throughout
• **Spell out abbreviations and acronyms throughout, particularly decision tree in 1.3, proprietary and confidential data levels
• clarify that last tree diagram is done manually @fhoces
• verify survey + spreadsheet links throughout the guidelines @joelferg
• Write template language for authors to respond to requests (line 253) @abogdanoski
• convert tables into HTML + test that formatting works across browsers @joelferg

Survey 1

• add a question about email ("we will use this to send your Survey 2").
• add a question about contacting authors with simple text entry
• Add a question to prespecify robustness checks and extensions, if applicable
• Add “other” as an option for the context of reproduction exercise in Survey 1.
• Add other in Q7
• Increase the number of claims in Q8. Add an example of a claim.
• Add “doing all claims in the paper” in Q9.
• In Q10 make sure units of measure at the same.
• Switch 13 and 14, and note that the general population may be the same as the one to which estimates apply.
• Q18 use “preferred analytical specification”
• Q19 change to “choose of up to 5” specifications.
• Q17 and Q18 put together in the same table/page. Add “other” field. Include appendix, supplementary materials, etc.
• Add info about which specification is the preferred one. Or pick one as the reproducer.
• 19 move to the beginning of the paper.
• Add a question at the beginning on “to what extent are you familiar with paper? Have you evaluated it before?”
• Q5.2 Add level 4
• Q5.4 add language that the number of claims requires some subjective assessment
• Q6.8 add a 4th row to enter statistic othen than S.E.
• add back buttons throughout @joelferg
• Q5.6 add an option to say more than 6 but not all claims
• Q6.1 make sure than # is filled out with the actual number of the claim
• set up a survey completion report at the end of the survey + instructions on how to access Survey 2 @joelferg
• set up a confirmation email with completion report at the end of the survey + instructions on how to access Survey 2 @joelferg

Survey 2:
From 4.10 meeting:

• Add a question about the reproducer's name at the beginning.
• Q8.3 -- Add "other" as a possible improvement (e.g. an entire file may contain the input for a display item). @fhoces @joelferg
• Q8.4 -- Remove semicolon input requirement
• Q8.5 -- Wording is a bit confusing. Change to "Relative to the original repro package, have you been able to include raw or proprietary data? Add explanation for context (e.g. many times there’s restricted access that authors can’t share). Possible answers: 1) Data is restricted access; 2) Data is now public; 3) Data is restricted access, but I was able to reproduce it privately. @fhoces
• Q8.7 -- Add: “ Given the improvements that you have added …”
• Robustness introduction -- Reword to emphasize it's about meaningful/important analytical choices (provide example). This is a subjective judgment, and you can choose to focus on just the important stuff, but can also analyze all. @fhoces
• Q9.1 -- Add a note: Your time spent will not factor in your grade.
• Comments in final section -- Add comments about own reproduction package + feedback for overall ACRE exercise and materials.

--

From 4.8 meeting:

• Introduction: change due time to 2 pm
• Introduction: add "You may not be able to navigate back from certain loops. This is a programming issue that we hope to resolve in future iterations of the exercise"
• Q.2.1-3 add/fix link to assessment spreadsheet
• make sure all links to assessment spreadsheets open up in a new tab/window (otherwise, navigating back removes all inputted data) @fhoces @joelferg
• replace "output" with "display item" throughout the survey
• replace "best practices" with "reproducibility tools and practices" throughout Survey 2 and the Guidelines
• Q2.8 add explanation of what constitutes a "display item"
• Q2.8 make it possible to enter more than 10 outputs/display items @fhoces
• Q3.3 change to “For tables and figures, you would use the title of the table or figure”.
• Q3.7 clarify that tree diagram is for students' reference and is not required to be submitted
• Q5.1 reword answers @abogdanoski
• Q5.4 combine first two answers into one
• Q5.5 add 3 answer for time spent doing other minor corrections
• Q5.7 explain subjectivity of assessment
• Q6.1 Reword message from economics at large to specific paper
• Q3.7 explain how students can build the tree if they need to do it themselves @fhoces
• Update guidelines to reflect the change in terminology of "display item"
• Q7.2 Clarify that this is about improvements made by the reproducer + give examples of improvements possible

--

• integrate mapping spreadsheet into Survey 2 @joelferg
• Change "minimal effort" definition to one hour or less
• Make it a bit more obvious when assigning the score that it output-level, rather than paper level
• Q14.7: Spell out acronyms from the reproducibility scale
• Q13.5: Make it possible to select multiple answers, correct grammar (remove "did")
• Q14.1: Change "table 1" to "output 1"
• Q14.6 Change answers to more elaborate statements, e.g. "Yes, exactly the same;" "No, but quite similar" + add a short text box to describe differences
• draft brief instructions on how to complete mapping part of the exercise using the new tool @joelferg

Survey 3:

• on hold for now, check with Ted if robustness checks will be a part of the exercise

General:

• confirm with Ted that papers will be pre-selected
• Survey 1 is timestamped when submitted and non-modifiable after. Survey 2 should be possible to go back to until you’ve submitted it.
• re-order Q numbering in surveys once done with editing
• draft emails to distribute the surveys
• confirm point person for questions from students during class
• make Robustness optional and figure out a way to record students' work for it @fhoces
• create a glossary of terms (e.g. computational reproducibility, raw + analytic data, data source + data file, cleaning + analysis code, reproducer, original author, etc.).

Timeline: Survey 1 to be posted 3/30, due 4/10; Survey 2 posted 4/13 (or earlier if ready) due 4/24