-
Notifications
You must be signed in to change notification settings - Fork 17
/
review-expectations.Rmd
129 lines (97 loc) · 6.42 KB
/
review-expectations.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
# Review Expectations {#review-expectation}
Anyone from the Bioconductor community that has a package in Bioconductor can
volunteer to become a community reviewer to assess incoming packages submitted
through the [New Submission Tracker][Packages Under Review]. The following
overviews what is expected of community reviewers.
- [Qualifications](#review-qualifications)
- [Time Commitment](#review-time-commitment)
- [Assignments](#review-assignments)
- [Expectations and deadlines](#review-expectations-and-deadlines)
- [Accept and Decline Packages](#acceptdecline)
- [Requesting reassignment and temporary leave of assignment](#review-reassignment-and-temporary-leave)
- [On-boarding](#review-onboarding)
- [Recognition](#review-recognition)
- [Volunteer](#review-volunteer)
## Qualifications {.unnumbered #review-qualifications}
Actively maintaining a Bioconductor package
## Time commitment {.unnumbered #review-time-commitment}
Package reviews can vary depending on the quality and organization of the
submitted package. On average a package review for a single package is between
30 min - 1.5 hour.
We also plan to have a quarterly “all-hands” meeting of all currently active
reviewers. This meeting will be to acquaint reviewers with any new
requirements, classes, checks activated in CRAN or BiocCheck, etc. It will also
be a place to raise questions or comments to the larger group and perhaps point
out frequently used comments (that perhaps could be implemented as an automated
check). Everyone ends up having slightly different review styles and it could
be beneficial to discuss what everyone finds a priority. Every effort should be
made to attend these meetings and we would expect a reviewer to attend at least
once a year.
## Assignments {.unnumbered #review-assignments}
Packages will be assigned evenly and randomly, the more community review
engagement the more distributed but currently anticipate a package assigned once
every week or every 2 weeks. More during the weeks leading up to a Bioconductor
release.
'Additional Package'/'Circular dependency' packages are submitted occasionally.
Submitters are allowed to submit related packages under the same issue. When
this occurs, the assigned reviewer is responsible for both reviews and should
not accept the issue until both packages are in an accepted state.
## Expectations and deadlines {.unnumbered #review-expectations-and-deadlines}
Package[s] will be expected to be reviewed within 3 weeks of assignment and
produce a clean build report from the build machine. We plan to make a hard
limit for 3 weeks. If the delay is on the submitter, the issue will be closed
for inactivity; it will be reopened when a submitter can commit to timely
updates and engagement of the review process. If the delay is in the reviewer,
it will be reassigned to someone else. We will keep track of how many
reassignments occur and if there are frequent reassignments (3-4 unexplained) we
will remove the reviewer and the person must reapply to be
reactivated. Active reviewers are expected to review at least 1 package per
Bioconductor release cycle. Otherwise they will be removed from being an active
reviewer.
## Accept and Decline Packages {.unnumbered #acceptdecline}
In general we try to work with submitters to get their package in a
state that would be acceptable for inclusion in Bioconductor regardless of how
long that takes. If they are willing to take critiques and be open minded about
Bioconductor requirements; we work with them. If a reviewer truly feels that a
package does not qualify for Bioconductor (because of applicability, coding
style, etc) or encounters a difficult submitter (unwilling to make changes,
aggressive, etc) , please reach out to the package review administrators. The
administrators will make a final determination before an official rejection or
take over if the review becomes uncomfortable.
## Requesting reassignment and temporary leave of assignment {.unnumbered #review-reassignment-and-temporary-leave}
If a temporary leave of assigning packages needs to occur (vacation,
particularly busy period of time, etc), a reviewer should reach out to a
package review committee administrator. The administrator will temporarily stop
assignments and can discuss if currently assigned packages need re-assignment.
There is a [reviewers slack channel][] for package reviewers to ask questions and if necessary
ask to swap/switch/volunteer to review packages. Requests can also be made by
directly contacting the package review administrator.
Reviewers should not request reassignment simply because of lack of interest or
lack of expertise in a particular area. The review should focus on ease-of-use
of the package, documentation, well written code, and interoperability. If a
reviewer knows of a particular ‘expert’ in the field relating to the submitted
package they can request someone offer additional comments (including those not
officially part of the package review list) without explicitly asking for
reassignment.
Frequent requests for reassignment should consider a longer leave of absence
agreement with the package review administrator. If frequent requests are being
made for reassignement the reviewer will be notified and at risk for being
removed from the active review process. If a reviewer is removed, they must
reapply to become an active reviewer.
## On-boarding {.unnumbered #review-onboarding}
New volunteers will undergo an on-boarding process before becoming fully active
reviewers. The on-boarding process involves doing at least one paired review
with a currently active reviewing member. Coordination of the paired review
should be communicated between the mentor and mentee. The recommended practice
will be that each do a separate review of a currently submitted package and
compare the results. The currently active member will then decide if another
paired review is necessary or if the review was sufficient to continue on with
solo reviews.
## Recognition {.unnumbered #review-recognition}
Feel free to advertise on CV and resumes as being a Bioconductor reviewer.
Reviewers will also be listed on the [Community Reviewers Webpage][reviewer-list]. Reviewers
may optionally opt-out of being listed on the webpage by contacting a package
review administrator. Ideally, we plan to have badges for notable achievements
and milestones, and a sticker is in progress.
## Volunteer {.unnumbered #review-volunteer}
Refer to the chapter [Volunteer to Review](#review-volunteer-chapter) for more details.