A new performance measure, relative bias, can now be calculated along with its Monte Carlo error (#41). More details on the formulae are in the introductory vignette, which has been updated accordingly.
diff --git a/docs/pkgdown.yml b/docs/pkgdown.yml
index ede0940..ba4126d 100644
--- a/docs/pkgdown.yml
+++ b/docs/pkgdown.yml
@@ -8,7 +8,7 @@ articles:
D-nlp: D-nlp.html
E-custom-inputs: E-custom-inputs.html
F-rsimsumtidyverse: F-rsimsumtidyverse.html
-last_built: 2023-05-27T09:07Z
+last_built: 2023-05-27T10:43Z
urls:
reference: https://ellessenne.github.io/rsimsum/reference
article: https://ellessenne.github.io/rsimsum/articles
diff --git a/docs/search.json b/docs/search.json
index d60eeca..660bad9 100644
--- a/docs/search.json
+++ b/docs/search.json
@@ -1 +1 @@
-[{"path":[]},{"path":"https://ellessenne.github.io/rsimsum/CODE_OF_CONDUCT.html","id":"our-pledge","dir":"","previous_headings":"","what":"Our Pledge","title":"Contributor Covenant Code of Conduct","text":"interest fostering open welcoming environment, contributors maintainers pledge making participation project community harassment-free experience everyone, regardless age, body size, disability, ethnicity, sex characteristics, gender identity expression, level experience, education, socio-economic status, nationality, personal appearance, race, religion, sexual identity orientation.","code":""},{"path":"https://ellessenne.github.io/rsimsum/CODE_OF_CONDUCT.html","id":"our-standards","dir":"","previous_headings":"","what":"Our Standards","title":"Contributor Covenant Code of Conduct","text":"Examples behavior contributes creating positive environment include: Using welcoming inclusive language respectful differing viewpoints experiences Gracefully accepting constructive criticism Focusing best community Showing empathy towards community members Examples unacceptable behavior participants include: use sexualized language imagery unwelcome sexual attention advances Trolling, insulting/derogatory comments, personal political attacks Public private harassment Publishing others’ private information, physical electronic address, without explicit permission conduct reasonably considered inappropriate professional setting","code":""},{"path":"https://ellessenne.github.io/rsimsum/CODE_OF_CONDUCT.html","id":"our-responsibilities","dir":"","previous_headings":"","what":"Our Responsibilities","title":"Contributor Covenant Code of Conduct","text":"Project maintainers responsible clarifying standards acceptable behavior expected take appropriate fair corrective action response instances unacceptable behavior. Project maintainers right responsibility remove, edit, reject comments, commits, code, wiki edits, issues, contributions aligned Code Conduct, ban temporarily permanently contributor behaviors deem inappropriate, threatening, offensive, harmful.","code":""},{"path":"https://ellessenne.github.io/rsimsum/CODE_OF_CONDUCT.html","id":"scope","dir":"","previous_headings":"","what":"Scope","title":"Contributor Covenant Code of Conduct","text":"Code Conduct applies within project spaces public spaces individual representing project community. Examples representing project community include using official project e-mail address, posting via official social media account, acting appointed representative online offline event. Representation project may defined clarified project maintainers.","code":""},{"path":"https://ellessenne.github.io/rsimsum/CODE_OF_CONDUCT.html","id":"enforcement","dir":"","previous_headings":"","what":"Enforcement","title":"Contributor Covenant Code of Conduct","text":"Instances abusive, harassing, otherwise unacceptable behavior may reported contacting project team alessandro.gasparini@ki.se. complaints reviewed investigated result response deemed necessary appropriate circumstances. project team obligated maintain confidentiality regard reporter incident. details specific enforcement policies may posted separately. Project maintainers follow enforce Code Conduct good faith may face temporary permanent repercussions determined members project’s leadership.","code":""},{"path":"https://ellessenne.github.io/rsimsum/CODE_OF_CONDUCT.html","id":"attribution","dir":"","previous_headings":"","what":"Attribution","title":"Contributor Covenant Code of Conduct","text":"Code Conduct adapted Contributor Covenant, version 1.4, available https://www.contributor-covenant.org/version/1/4/code--conduct.html answers common questions code conduct, see https://www.contributor-covenant.org/faq","code":""},{"path":"https://ellessenne.github.io/rsimsum/CONTRIBUTING.html","id":null,"dir":"","previous_headings":"","what":"Contributing","title":"Contributing","text":"want contribute project make better, help welcome. small, simple changes fixing typos can edit file directly GitHub clicking Edit button opening . complicated changes, manually create pull request (PR) forking repository. See next section information. submit non-trivial pull request (e.g. just fixing typo), may add name Authors@R field contributor (ctb) R package DESCRIPTION file wish.","code":""},{"path":"https://ellessenne.github.io/rsimsum/CONTRIBUTING.html","id":"pull-request-workflow","dir":"","previous_headings":"","what":"Pull Request Workflow","title":"Contributing","text":"Create personal fork project Github, clone fork local machine; Create new branch work ; Implement/fix feature, comment code; Follow code style project, including indentation; Run tests using devtools: devtools::test(); Write adapt tests needed; Add change documentation needed. Please run roxygen2, include changes .Rd files pull request - re-roxygenise documentation ; Push branch fork Github; fork open pull request correct branch. step--step workflow adapted https://github.com/MarcDiethelm/contributing. Working first Pull Request? can learn free series Contribute Open Source Project GitHub.","code":""},{"path":"https://ellessenne.github.io/rsimsum/CONTRIBUTING.html","id":"code-of-conduct","dir":"","previous_headings":"","what":"Code of Conduct","title":"Contributing","text":"contributing project agree adhere Contributors Code Conduct: please read CONDUCT.md proposing change.","code":""},{"path":"https://ellessenne.github.io/rsimsum/articles/A-introduction.html","id":"rsimsum","dir":"Articles","previous_headings":"","what":"rsimsum","title":"Introduction to rsimsum","text":"rsimsum R package can compute summary statistics simulation studies. inspired user-written command simsum Stata (White .R., 2010). aim rsimsum helping reporting simulation studies, including understanding role chance results simulation studies. Specifically, rsimsum can compute Monte Carlo standard errors summary statistics, defined standard deviation estimated summary statistic; reported default. Formula summary statistics Monte Carlo standard errors presented next section. Note terms summary statistic performance measure used interchangeably.","code":""},{"path":"https://ellessenne.github.io/rsimsum/articles/A-introduction.html","id":"notation","dir":"Articles","previous_headings":"","what":"Notation","title":"Introduction to rsimsum","text":"use th following notation throughout vignette: \\(\\theta\\): estimand, true value \\(n_{\\text{sim}}\\): number simulations \\(= 1, \\dots, n_{\\text{sim}}\\): indexes given simulation \\(\\hat{\\theta}_i\\): estimated value \\(\\theta\\) \\(^{\\text{th}}\\) replication \\(\\widehat{\\text{Var}}(\\hat{\\theta}_i)\\): estimated variance \\(\\text{Var}(\\hat{\\theta}_i)\\) \\(\\hat{\\theta}_i\\) \\(^{\\text{th}}\\) replication \\(\\text{Var}(\\hat{\\theta})\\): empirical variance \\(\\hat{\\theta}\\) \\(\\alpha\\): nominal significance level","code":""},{"path":"https://ellessenne.github.io/rsimsum/articles/A-introduction.html","id":"performance-measures","dir":"Articles","previous_headings":"","what":"Performance measures","title":"Introduction to rsimsum","text":"first performance measure interest bias, quantifies whether estimator targets true value \\(\\theta\\) average. Bias calculated : \\[\\text{Bias} = \\frac{1}{n_{\\text{sim}}} \\sum_{= 1} ^ {n_{\\text{sim}}} \\hat{\\theta}_i - \\theta\\] Monte Carlo standard error bias calculated : \\[\\text{MCSE(Bias)} = \\sqrt{\\frac{\\frac{1}{n_{\\text{sim}} - 1} \\sum_{= 1} ^ {n_{\\text{sim}}} (\\hat{\\theta}_i - \\bar{\\theta}) ^ 2}{n_{\\text{sim}}}}\\] rsimsum can also compute relative bias (relative true value \\(\\theta\\)), can interpreted similarly bias, relative terms rather absolute. calculated : \\[\\text{Relative Bias} = \\frac{1}{n_{\\text{sim}}} \\sum_{= 1} ^ {n_{\\text{sim}}} \\frac{\\hat{\\theta}_i - \\theta}{\\theta}\\] Monte Carlo standard error calculated : \\[ \\text{MCSE(Relative Bias)} = \\sqrt{\\frac{1}{n_{\\text{sim}} (n_{\\text{sim}} - 1)} \\sum_i^{n_{\\text{sim} \\left[ \\frac{\\hat{\\theta}_i - \\theta}{\\theta} - \\widehat{\\text{Relative Bias}} \\right]^2} \\] empirical standard error \\(\\theta\\) depends \\(\\hat{\\theta}\\) require knowledge \\(\\theta\\). estimates standard deviation \\(\\hat{\\theta}\\) \\(n_{\\text{sim}}\\) replications: \\[\\text{Empirical SE} = \\sqrt{\\frac{1}{n_{\\text{sim}} - 1} \\sum_{= 1} ^ {n_{\\text{sim}}} (\\hat{\\theta}_i - \\bar{\\theta}) ^ 2}\\] Monte Carlo standard error calculated : \\[\\text{MCSE(Emp. SE)} = \\frac{\\widehat{\\text{Emp. SE}}}{\\sqrt{2 (n_{\\text{sim}} - 1)}}\\] comparing different methods, relative precision given method B reference method computed : \\[\\text{Relative % increase precision} = 100 \\left[ \\left( \\frac{\\widehat{\\text{Emp. SE}}_A}{\\widehat{\\text{Emp. SE}}_B} \\right) ^ 2 - 1 \\right]\\] (approximated) Monte Carlo standard error : \\[\\text{MCSE(Relative % increase precision)} \\simeq 200 \\left( \\frac{\\widehat{\\text{Emp. SE}}_A}{\\widehat{\\text{Emp. SE}}_B} \\right)^2 \\sqrt{\\frac{1 - \\rho^2_{AB}}{n_{\\text{sim}} - 1}}\\] \\(\\rho^2_{AB}\\) correlation \\(\\hat{\\theta}_A\\) \\(\\hat{\\theta}_B\\). measure takes account precision accuracy method mean squared error, sum squared bias variance \\(\\hat{\\theta}\\): \\[\\text{MSE} = \\frac{1}{n_{\\text{sim}}} \\sum_{= 1} ^ {n_{\\text{sim}}} (\\hat{\\theta}_i - \\theta) ^ 2\\] Monte Carlo standard error : \\[\\text{MCSE(MSE)} = \\sqrt{\\frac{\\sum_{= 1} ^ {n_{\\text{sim}}} \\left[ (\\hat{\\theta}_i - \\theta) ^2 - \\text{MSE} \\right] ^ 2}{n_{\\text{sim}} (n_{\\text{sim}} - 1)}}\\] model based standard error computed averaging estimated standard errors replication: \\[\\text{Model SE} = \\sqrt{\\frac{1}{n_{\\text{sim}}} \\sum_{= 1} ^ {n_{\\text{sim}}} \\widehat{\\text{Var}}(\\hat{\\theta}_i)}\\] (approximated) Monte Carlo standard error computed : \\[\\text{MCSE(Model SE)} \\simeq \\sqrt{\\frac{\\text{Var}[\\widehat{\\text{Var}}(\\hat{\\theta}_i)]}{4 n_{\\text{sim}} \\widehat{\\text{Model SE}}}}\\] model standard error targets empirical standard error. Hence, relative error model standard error informative performance measure: \\[\\text{Relative % error model SE} = 100 \\left( \\frac{\\text{Model SE}}{\\text{Empirical SE}} - 1\\right)\\] Monte Carlo standard error computed : \\[\\text{MCSE(Relative % error model SE)} = 100 \\left( \\frac{\\text{Model SE}}{\\text{Empirical SE}} \\right) \\sqrt{\\frac{\\text{Var}[\\widehat{\\text{Var}}(\\hat{\\theta}_i)]}{4 n_{\\text{sim}} \\widehat{\\text{Model SE}} ^ 4} + \\frac{1}{2(n_{\\text{sim}} - 1)}}\\] Coverage another key property estimator. defined probability confidence interval contains true value \\(\\theta\\), computed : \\[\\text{Coverage} = \\frac{1}{n_{\\text{sim}}} \\sum_{= 1} ^ {n_{\\text{sim}}} (\\hat{\\theta}_{, \\text{low}} \\le \\theta \\le \\hat{\\theta}_{, \\text{upp}})\\] \\((\\cdot)\\) indicator function. Monte Carlo standard error computed : \\[\\text{MCSE(Coverage)} = \\sqrt{\\frac{\\text{Coverage} \\times (1 - \\text{Coverage})}{n_{\\text{sim}}}}\\] coverage expected : \\(\\text{Bias} \\ne 0\\), \\(\\text{Models SE} < \\text{Empirical SE}\\), distribution \\(\\hat{\\theta}\\) normal intervals constructed assuming normality, \\(\\widehat{\\text{Var}}(\\hat{\\theta}_i)\\) variable coverage occurs result \\(\\text{Models SE} > \\text{Empirical SE}\\). coverage may result bias, another useful summary statistic bias-eliminated coverage: \\[\\text{Bias-eliminated coverage} = \\frac{1}{n_{\\text{sim}}} \\sum_{= 1} ^ {n_{\\text{sim}}} (\\hat{\\theta}_{, \\text{low}} \\le \\bar{\\theta} \\le \\hat{\\theta}_{, \\text{upp}}) \\] Monte Carlo standard error analogously coverage: \\[\\text{MCSE(Bias-eliminated coverage)} = \\sqrt{\\frac{\\text{Bias-eliminated coverage} \\times (1 - \\text{Bias-eliminated coverage})}{n_{\\text{sim}}}}\\] Finally, power significance test \\(\\alpha\\) level defined : \\[\\text{Power} = \\frac{1}{n_{\\text{sim}}} \\sum_{= 1} ^ {n_{\\text{sim}}} \\left[ |\\hat{\\theta}_i| \\ge z_{\\alpha/2} \\times \\sqrt{\\widehat{\\text{Var}}(\\hat{\\theta_i})} \\right]\\] Monte Carlo standard error analogously coverage: \\[\\text{MCSE(Power)} = \\sqrt{\\frac{\\text{Power} \\times (1 - \\text{Power})}{n_{\\text{sim}}}}\\] information summary statistics simulation studies can found White (2010) Morris, White, Crowther (2019).","code":""},{"path":"https://ellessenne.github.io/rsimsum/articles/A-introduction.html","id":"example-1-simulation-study-on-missing-data","dir":"Articles","previous_headings":"","what":"Example 1: Simulation study on missing data","title":"Introduction to rsimsum","text":"example dataset included rsimsum aim summarise simulation study comparing different ways handle missing covariates fitting Cox model (White Royston, 2009). One thousand datasets simulated, containing normally distributed covariates \\(x\\) \\(z\\) time--event outcome. covariates \\(20\\%\\) values deleted independently variables data became missing completely random (Little Rubin, 2002). simulated dataset analysed three ways. Cox model fit complete cases (CC). two methods multiple imputation using chained equations (van Buuren, Boshuizen, Knook, 1999) used. MI_LOGT method multiply imputes missing values \\(x\\) \\(z\\) outcome included \\(\\log(t)\\) \\(d\\), \\(t\\) survival time \\(d\\) event indicator. MI_T method except \\(\\log(t)\\) replaced \\(t\\) imputation model. load data usual way: Let’s look first 10 rows dataset: included variables : dataset, number simulated dataset; method, method used (CC, MI_LOGT MI_T); b, point estimate; se, standard error point estimate. summarise results simulation study method using simsum function: set true = 0.50 true value point estimate b - data simulated - 0.50. select CC reference method consider complete cases analysis reference method benchmark ; set reference method, simsum picks one automatically. Using default settings, Monte Carlo standard errors computed returned. Summarising simsum object, obtain following output: output begins brief overview setting simulation study (e.g. method variable, unique methods, etc.), continues summary statistic method (defined, case). values reported point estimates Monte Carlo standard errors brackets; however, also possible require confidence intervals based Monte Carlo standard errors reported instead: Highlighting points interest summary results : CC method small-sample bias away null (point estimate 0.0168, 95% confidence interval: 0.0074 - 0.0261); CC inefficient compared MI_LOGT MI_T: relative gain precision two methods 1.3105% 1.2637% compared CC, respectively; Model-based standard errors close empirical standard errors; Coverage nominal 95% confidence intervals also seems fine, surprising view generally low (lack ) bias good model-based standard errors; CC lower power compared MI_LOGT MI_T, surprising view inefficiency.","code":"library(rsimsum) data(\"MIsim\", package = \"rsimsum\") head(MIsim, n = 10) #> # A tibble: 10 × 4 #> dataset method b se #>