-
Notifications
You must be signed in to change notification settings - Fork 1
/
index.qmd
132 lines (76 loc) · 6.9 KB
/
index.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
---
title: "Risk Assessment of R Packages at Novartis"
format:
html:
embed-resources: true
toc: true
editor: source
---
This repository contains insights into to the risk-assessment process for validating R packages at Novartis prior to their use on our SCE (Scientific Computing Environment) for analysis and reporting in clinical trials, i.e. GxP activities.
The version of R and associated set of packages are updated periodically as part of operational releases. The validation process prevents the need for double programming of the functionality of the package but users remain accountable for understanding the implementation of the methodology (e.g. solvers and defaults).
The outcome of the package validation is stored in the `./data` directory and the most recent released data is rendered below:
```{r}
DT::datatable(
readr::read_csv("./data/validated_packages_september_2024.csv")
)
```
## Risk assessment timing
As part of the software update cycle, packages are risk assessed and if their risk level is deemed acceptable, included in the next software release on our SCE.
The risk assessment dates shown above capture 2 scenarios:
1. Risk assessments carried out as part of the previous release
2. New risk assessments carried out on the lead up to the latest release
The package versions reported here are the versions of the package installed in the R module on the SCE, which may be a slightly later version of the package than that originally assessed.
We conducted an analysis of changes in risk assessment ratings over time and found that the risk rating changed in only minority of packages (7.4%). In these small number of cases the changes were primarily non-impactful, e.g. a package transitioning from medium risk to low risk, with changes primarily driven by fluctuation in download rates. These data helped inform our strategy for the risk assessment of existing packages - we re-assess the risk for major milestones like updates to the R version but for more incremental releases we carry over the risk assessment from the previous release.
More information about this analysis can be found in [these slides, presented at an R Validation Hub community meeting on 20th August 2024](https://github.com/Novartis/risk_assessments/blob/main/slides/ratings_over_time.pdf).
## Risk ratings
Risk assessment calculations are mostly automated, with some intervention from reviewers when metrics require manual verification or calculation (e.g. trusted source type metrics), and to give overall approval of a package's risk assessment.
Low or medium risk packages are automatically approved for installation on our SCE.
If a package has been deemed to be high risk, the requester is required to perform additional checks to verify the functionality of the package through performance qualification (PQ) tests.
If a package receives a rating of "ineligible", it has failed all basic pre-requisites and cannot be installed as part of this process.
## Risk rating criteria
Each risk rating has a set of defined criteria. If a package meets at least one of these criteria, then it is assigned that particular risk rating.
In the section below, we briefly describe the criteria at a high-level, and in the next section we expand on the details of some of the more complex ones.
### Prerequisites
Does the package fulfill the following criteria?
* Source code is publicly available (e.g., via CRAN, Bioconductor, GitHub, ...)
* Package has not been archived more than 180 days ago on CRAN or Bioconductor
If the answer is "no" to both of these questions, then the package is ineligible.
### Low risk packages
The package meets at least one of these low-risk criteria:
1. Does the package belong to any of the following groups?
* base or recommended package
* tidyverse package
* imports/suggestions of a base or recommended package
2. Does the package come from a recognizable source or reputable institution?
3. Does the package have at least 50,000 downloads per month? Use the average over the last 6 months or since release
4. Does the package have at least 5 reverse dependencies?
5. Is the package a "low risk support package", e.g., one devoted solely to format, display or visualize the output or is simply a data package?
### Medium risk packages
The package meets at least one of these medium-risk criteria:
6. Does the package have at least 1,000 downloads per month?
7. Are there any peer-reviewed publications about the package in reputable journals?
8. Is there unit testing covering more than 60%?
### High risk packages
The package meets at least one of these high-risk criteria:
9. Was the first CRAN / Bioconductor production release of the package more than 1 year ago?
10. Does the package contain any additional documentation? (vignettes, manuals, website, newsfeed)
## Acceptance criteria guidelines
Questions 2, 5, and 7 rely on more complex decision criteria for determining a package's risk rating, and more details of these are found below.
### Trusted sources and institutions
We maintain a list of trusted sources, consisting of individuals who are considered to be experienced developers with numerous contributions to the open source community, and institutions who are considered to be well-established.
Users can request that new sources are added to this list, and each request is evaluated individually. As a minimum, an individual must fulfill one of the following criteria to be included:
* a h-index greater than or equal to 20
* have 5 or more active packages on CRAN or Bioconductor
For institutions, the both of the following criteria must be met:
* have existed for 5 or more years
* have 5 or more active packages on CRAN or Bioconductor
For both institutions and individuals, the criteria above are a minimum, and meeting these criteria does not guarantee that the request for their addition will be granted.
### Journal publication criteria
For a package to be deemed medium-risk due to having an associated journal article publication, the journal must be one from a list of approved journals. A journal must fulfill all of the following minimum criteria:
* publisher maturity of 5 or more years
* peer review is an obligatory part of the publishing process
In addition, a journal must fulfill at least one of the following conditions:
* A trustworthy journal should have an above average citation potential (>= 1.0 on the SCImago Journal Rank)
* Impact factor of 2 or more
### Low-risk support package
Low-risk support packages are those that do not have the capacity to alter the scientific analysis or conclusion. This includes pure display or data packages. Packages that merely provide some limited level of interactivity (e.g., zoom, hover text) may still be considered low-risk support packages, but packages that contain complex or impactful mathematical operations, generate or manipulate code used in scientific analysis, or do any kind of computations cannot be considered to be low-risk support packages.