-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME.Rmd
91 lines (62 loc) · 3.62 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
---
output: github_document
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r, include=FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "Readme_files/",
eval = FALSE
)
devtools::load_all()
```
[](https://github.com/difuture-lmu/dsCalibration/actions) [](https://www.gnu.org/licenses/lgpl-3.0) [](https://codecov.io/gh/difuture-lmu/dsCalibration)
# Calibration Functions for DataSHIELD
The package provides functionality to assess calibration for a given vector of predictions for binary classes on decentralized data. The basis is the DataSHIELD](https://www.datashield.org/) infrastructure for distributed computing. This package provides the calculation of the [__Brier score__](https://en.wikipedia.org/wiki/Brier_score) as well as [__calibration curves__](https://medium.com/analytics-vidhya/calibration-in-machine-learning-e7972ac93555). In order to calculate the Brier score or calibration curves it is necessary to have prediction values on the server. For instructions how to push and predict models see the package [`dsPredictBase`](https://github.com/difuture-lmu/dsPredictBase). Note that DataSHIELD uses an option `datashield.privacyLevel` to indicate the minimal amount of numbers required to be allowed to share an aggregated value of these numbers. Instead of setting the option, we directly retrieve the privacy level from the [`DESCRIPTION`](https://github.com/difuture-lmu/dsCalibration/blob/master/DESCRIPTION) file each time a function calls for it. This options is set to 5 by default.
## Installation
At the moment, there is no CRAN version available. Install the development version from GitHub:
```{r,eval=FALSE}
remotes::install_github("difuture-lmu/dsCalibration")
```
#### Register aggregate methods
It is necessary to register the aggregate methods in the OPAL administration. The assign methods are:
- `brierScore`
- `calibrationCurve`
These methods are registered automatically when publishing the package on OPAL (see [`DESCRIPTION`](https://github.com/difuture/dsPredictBase/blob/master/DESCRIPTION)).
Note that the package needs to be installed at both locations, the server and the analysts machine.
## Usage
```{r, results="hide", warnings=FALSE}
library(DSI)
library(DSOpal)
library(DSLite)
library(dsBaseClient)
library(dsCalibration)
builder = DSI::newDSLoginBuilder()
builder$append(
server = "ibe",
url = "******",
user = "***",
password = "******",
table = "ProVal.KUM"
)
logindata = builder$build()
connections = DSI::datashield.login(logins = logindata, assign = TRUE, symbol = "D",
opts = list(ssl_verifyhost = 0, ssl_verifypeer = 0))
### Get available tables:
DSI::datashield.symbols(connections)
### Test data with same structure as data on test server:
dat = read.csv("data/test-kum.csv")
### Model we want to upload:
mod = glm(gender ~ age + height, family = "binomial", data = dat)
### Upload model to DataSHIELD server
pushObject(connections, mod)
predictModel(connections, mod, "pred", "D", predict_fun = "predict(mod, newdata = D, type = 'response')")
DSI::datashield.symbols(connections)
### Calculate brier score:
dsBrierScore(connections, "D$gender", "pred")
### Calculate and plot calibration curve:
cc = dsCalibrationCurve(connections, "D$gender", "pred", 10, 3)
plotCalibrationCurve(cc)
DSI::datashield.logout(conns = connections, save = FALSE)
```