3.10.0

* small updates into new dev branch

* Fix bug for when `RhoConfig['Beta1']=5` ...

... which wasn't implemented properly in the CPP.

* Update test-random_beta_against_glmer.R

* further adding `skip_on_ci`

* adding weird .yml edit

following suggestion from @k-doering-NOAA

* removing FishStatsUtils from DESCRIPTION ...

 ... to see if that forces an updated version and fixes MacOS CI Action

* add back "test-platform" CI check

* using `Additional_respositories` in DESCRIPTION ...

... copying Kelli Johnson's edits to SpatialDeltaGLMM

* removing SpatialDeltaGLMM from dependencies

* adding back some integrated tests

* add back more CIs

* re-adding three more integrated tests

* removing one re-added integrated-test

* re-removing multivariate CI tests

* moving dependency versions from .onAttach to DESCRIPTION

* change FishStatsUtils to `Imports`

* further experiment with DESCRIPTION

* revert from last experiment

* further small changes to DESCRIPTION

* housekeeping: remove .onAttach action

* remove all integrated-tests requiring data ...

... given that dev branch of FishStatsUtils isn't being installed properly via Actions

* experiment with Actions loading of FishStatsUtils

* Experiment: Move FishStatsUtils install back to .onAttach

* small fix in last commit

* reverting last experiment

* explore change for `R-CMD-check.yml` ...

... following advise from Kathryn Doering

* add back BS pollock integrated test

* Adding deviance calculation for Tweedie distribution

* expand User Manual ...

... to provide more details about abundance vs. density-weighted indices

* small updates

* adding test for new epsilon feature

* test CI fix ...

... for exploring new TMB features not available in CRAN version

* add new bias-correction option

* adding `apply_epsilon`

* Update test-Tweedie-against-mgcv.R

* disable newton::tag feature for now

* Update zzz.R

* update H info for manual

* adding project_model

* fix project_model to return simulated stuff properly

* add demo snippet and labels to output

* update NEWS

* remove junk code from testing

* bug-fix for unconditional simulation of beta1/beta2

* fixing EOF to work with unconditional simulation of beta/epsilon

* update NEWS

* fix typo in last commit

* update docs for `project_model`

* fix `historical_uncertainty == "random"`

* Another fix to historical_uncertainty = "random"

* further update project_model docs ...

... h/t Andrew Allyn for continuing to point out implications of alternative estimation-model settings

* re-add devtools to fix Ubuntu CI Action

* fix project_model to work with explicit units

* improve defaults and docs for density-dependent feature

* Now excluding category-specific intercepts in Q1_formula / Q2_formula

h/t Yumi Arimitsu for suggesting the idea.

* fix bug in last commit

h/t Yumi for flagging

* fix bug in last commit ...

... occurring in catchbaility-covariate intercept detection in univariate models

* fix bug in EOF for v14_0_1

* revert to main-branch versions

* re-update dev branch

* Update version number

* Add EOF CI and disable MICE and combined-data CIs

* update EOF CI to avoid label-switching issue

* fix typo in EOF CI

* test fix for EOF CI for Ubuntu

* adding auto-error check for catchability_data ...

... for cases where it contains NA, NaN, Inf values.  H/t Anna Marcout for flagging the issue

* another auto-check for catchability inputs

* fix bug in last commit ...

... where is.nan(.) and is.infinite(.) aren't designed for data-frames

* A few typos in `project_model.R` (#324)

Plus rebuild roxygen docs

* adding `reload_model`

* improve docs for `make_data`

* Fixing backwards compatibility by ...

1.  Removing CPP versions prior to V8.0.0
2.  adding an integrated test for installed CPPs
3.  fixing compilation and build issues with CPP versions <10.0.0

* turn off CI for 'test-past-versions' integrated test

* fix bug in effects.R

* Update reload_model.R

* adding new option for Expansion_cz

* fix bug in last commit

* small updates

* fix bug in Barrier-SPDE to keep up with release 3.9.1

* updates to keep up with release 3.9.1

* switching to framework = TMBad by default

* experimenting with speedup to project_model ...

... whereby the user can specify `n_samples` and avoid recompiling objects for each sample run

* fix typo in last commit

* add option to subset variables in `project_model`

* generalize `apply_epsilon` to work ..

... when other variables are also being epsilon-corrected

* bug fix ...

... h/t Maurice Goodman

* update v_i to allow NA for observations without vessel effects

* Update make_data.R

* fix v_i change

* doc updates

* update NEWS to unify main and dev

Co-authored-by: Sean Anderson <seananderson@users.noreply.github.com>

.github/workflows/R-CMD-check.yml

@@ -23,6 +23,7 @@ jobs:
     env:
       R_REMOTES_NO_ERRORS_FROM_WARNINGS: true
       RSPM: ${{ matrix.config.rspm }}
+      GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
 
     steps:
       - uses: actions/checkout@v2
@@ -64,8 +65,6 @@ jobs:
         run: |
           remotes::install_deps(dependencies=TRUE)
           remotes::install_cran("rcmdcheck")
-          remove.packages('FishStatsUtils')
-          remotes::install_github('James-Thorson-NOAA/FishStatsUtils')
         shell: Rscript {0}
 
       - name: Check Ubuntu

DESCRIPTION

@@ -1,8 +1,8 @@
 Package: VAST
 Type: Package
 Title: Vector-Autoregressive Spatio-Temporal (VAST) Model
-Version: 3.9.1
-Date: 2022-07-07
+Version: 3.10.0
+Date: 2022-11-18
 Authors@R: 
     c(person(given = "James",
              family = "Thorson",
@@ -20,35 +20,34 @@ Description: VAST is an R package for conducting spatio-temporal analysis
     factor-analysis or autoregressive correlations among categories, with derived
     calculation of abundance indices, center-of-gravity, and area-occupied.
 Imports:
-    deldir,
-    stats,
-    devtools,
     abind,
-    MatrixModels,
-    ThorsonUtilities,
     effects,
     INLA,
-    SpatialDeltaGLMM
+    stats,
+    ThorsonUtilities,
+    utils,
+    remotes,
+    devtools
 Depends:
-    TMB,
-    R (>= 3.5.0),
+    TMB (>= 1.8.0),
+    FishStatsUtils,
+    R (>= 3.5.0)
 Suggests:
     testthat,
     knitr,
     rmarkdown,
-    FishStatsUtils,
     pscl,
     tweedie,
     mgcv,
     lme4
 Remotes:
     james-thorson-NOAA/FishStatsUtils,
-    james-thorson/utilities,
-    nwfsc-assess/geostatistical_delta-GLMM
+    james-thorson/utilities
+Additional_repositories: https://inla.r-inla-download.org/R/stable
 License: file LICENSE
 LazyData: true
 Encoding: UTF-8
-RoxygenNote: 7.2.0
+RoxygenNote: 7.2.2
 ByteCompile: true
 URL: http://github.com/James-Thorson-NOAA/VAST
 BugReports: http://github.com/James-Thorson-NOAA/VAST/issues

NAMESPACE

@@ -10,11 +10,14 @@ export(Param_Fn)
 export(Plot_Overdispersion)
 export(Plot_factors)
 export(Summarize_Covariance)
+export(apply_epsilon)
 export(check_fit)
 export(make_data)
 export(make_map)
 export(make_model)
 export(make_parameters)
+export(project_model)
+export(reload_model)
 importFrom(INLA,inla.as.dgTMatrix)
 importFrom(grDevices,colorRampPalette)
 importFrom(grDevices,dev.off)

R/VAST.R

@@ -13,9 +13,10 @@
 #' Features are built to be compatible among model types, e.g., by allowing catchability and density covariates to be included in EOF analysis.
 #'
 #' See \code{\link[FishStatsUtils]{fit_model}} for a simple example of high-level wrapper functions for using VAST.
-#' Also see the wiki \url{https://github.com/James-Thorson-NOAA/VAST/wiki} for examples documenting many different use-cases and features.
 #'
 #' @seealso \code{\link[VAST]{VAST}} for general documentation, \code{\link[FishStatsUtils]{make_settings}} for generic settings, \code{\link[FishStatsUtils]{fit_model}} for model fitting, and \code{\link[FishStatsUtils]{plot_results}} for generic plots
+#' @seealso VAST wiki \url{https://github.com/James-Thorson-NOAA/VAST/wiki} for examples documenting many different use-cases and features.
+#' @seealso GitHub mainpage \url{https://github.com/James-Thorson-NOAA/VAST#description} for a list of user resources and publications documenting features
 #' @docType package
 #' @name VAST
 #' @importFrom utils packageVersion packageDescription

R/apply_epsilon.R

@@ -0,0 +1,83 @@
+
+#' Custom epsilon-correct method
+#'
+#' \code{apply_epsilon} uses updates to TMB to implement a faster calculation for epsilon-correction
+#'
+#' @inheritParams TMB::MakeADFun
+#' @param fit output from \code{\link[FishStatsUtils]{fit_model}}, specifically using
+#'        slotes \code{tmb_list}, \code{input_args}, \code{parameter_estimates$SD}
+#' @param ADREPORT_name string indicating name of ADREPORT'ed variable
+#' @param eps_name string indicating name of PARAMETER used internally by TMB
+#'        for calculating desired gradient
+#'
+#' @return Standard output from \code{\link[TMB]{sdreport}}, but with slot
+#'         \code{x$unbiased} added if needed, and adding or replacing values for
+#'         \code{x$unbiased$value} corresponding to \code{ADREPORT_name}
+#'
+#' @export
+apply_epsilon <-
+function( fit,
+          ADREPORT_name = "Index_ctl",
+          eps_name = "eps_Index_ctl",
+          inner.control = list(sparse=TRUE, lowrank=TRUE, trace=FALSE) ){
+
+  # Extract stuff
+  Obj = fit$tmb_list$Obj
+  if(is.null(fit$Report)) fit$Report = Obj$report()
+  if(is.null(fit$ParHat)) fit$ParHat = Obj$env$parList()
+
+  # Error checks
+  if( !all(c("tmb_list","input_args","parameter_estimates") %in% names(fit)) ) stop("Check `fit` in `apply_epsilon`: function is only designed to work with output from VAST using `fit_model`")
+  if( !(ADREPORT_name %in% names(fit$Report)) ) stop("Check `ADREPORT_name` in `apply_epsilon`")
+  if( fit$input_args$model_args_input$framework!="TMBad" ) stop("`apply_epsilon` requires that the CPP be compiled using framework=`TMBad`")
+  if( is.null(fit$parameter_estimates$SD) ) stop("Please re-run with `getsd=TRUE`")
+
+  # Simple extractions
+  Data = Obj$env$data
+  Map = Obj$env$map
+  Random = fit$tmb_list$Random
+
+  # Extract and modify parameters
+  New_params = fit$ParHat
+  New_params[[eps_name]] = array(0, dim=dim(fit$Report[[ADREPORT_name]]) )
+
+  # Change MLE
+  fixed = fit$parameter_estimates$par
+  new_values = rep( 0, prod(dim(New_params[[eps_name]])) )
+  names(new_values) = rep( eps_name, length(new_values))
+  fixed = c( fixed, new_values )
+
+  # detect sparse + lowrank hessian ... appears to freeze with lowrank=FALSE
+  obj = MakeADFun( data = Data,
+                    parameters = New_params,
+                    map = Map,
+                    random = Random,
+                    intern = TRUE,
+                    inner.control = inner.control )
+  obj$env$beSilent()
+  gradient = obj$gr(fixed)
+
+  # Expand SD
+  SD = fit$parameter_estimates$SD
+  if( is.null(SD$unbiased) ){
+    SD$unbiased = list( "value"=SD$value, "sd"=NA, "cov"=array(NA,c(1,1)) )
+    SD$unbiased$value[] = NA
+  }else{
+    if( any(!is.na(SD$unbiased$value[ADREPORT_name])) ){
+      warning( paste0("it appears that `", ADREPORT_name,"` is already bias-corrected; using `apply_epsilon` seems inefficient and will replace existing values") )
+    }
+  }
+
+  # Merge gradient into SD
+  i = which( names(SD$value) == ADREPORT_name )
+  j = which( names(obj$par) == eps_name )
+  if( length(i)==length(j) ){
+    SD$unbiased$value[i] = gradient[j]
+  }else{
+    warning("Check `apply_epsilon` for bugs")
+  }
+
+  # return SD
+  class(SD) = "sdreport"
+  return(SD)
+}

R/check_fit.R

@@ -32,7 +32,8 @@
 #' @export
 #' @md
 # Using https://cran.r-project.org/web/packages/roxygen2/vignettes/rd-formatting.html for guidance on markdown-enabled documentation
-check_fit = function( parameter_estimates,
+check_fit <-
+function( parameter_estimates,
           check_gradients=FALSE,
           quiet=FALSE ){
 

R/effects.R

@@ -7,16 +7,24 @@
 #' @param which_formula which formula to use e.g., \code{"X1"}
 #' @param category_number which category code{c_i} to use when plotting density covariates
 #'
+#' If getting the error \code{non-conformable arguments}, consider exploring \code{pad_values}
+#'   The error arises in when constructing the linear predictor without an intercept,
+#'   and the default \code{pad_values = 1} attempts to insert a dummy intercept with estimate and SE
+#'   equal to zero.
+#'
 #' @rawNamespace S3method(effects::Effect, fit_model)
 #' @export
 Effect.fit_model <-
 function( focal.predictors,
           mod,
           which_formula = "X1",
-          pad_values = c(),
+          pad_values = c(1),
           category_number = NULL,
           ...) {
 
+  #
+  message("please read `?Effect.fit_model` for details")
+
   # Error checks
   #if( mod$data_list$n_c>1 & which_formula%in%c("X1","X2") ){
   #  stop("`Effect.fit_model` is not currently designed for multivariate models using density covariates")
@@ -65,7 +73,7 @@ function( focal.predictors,
   # Identify which parameters to extract from par and cov
   whichnum = which( names(mod$parameter_estimates$par) == parname )
   map_indices = mod$tmb_list$Parameters[[parname]]
-  if( "parname" %in% names(mod$tmb_list$Obj$env$map) ){
+  if( parname %in% names(mod$tmb_list$Obj$env$map) ){
     map_indices[] = mod$tmb_list$Obj$env$map[[parname]]
     if( any(table(map_indices)>1) ) stop("`Effects.fit_model` not designed to work with mapping of duplicate values")
   }else{
@@ -110,12 +118,18 @@ function( focal.predictors,
   #rownames(mod$covhat) = colnames(mod$covhat) = names(mod$parhat)
 
   # Augment stuff
-  formula_full = stats::update.formula(formula_orig, linear_predictor~.+0)
+  formula_full = stats::update.formula(formula_orig, linear_predictor~.+1)
   mod$coefficients = mod$parhat
   mod$vcov = mod$covhat
   mod$formula = formula_full
   mod$family = stats::gaussian(link = "identity")
 
+  #
+  if( FALSE ){
+    formula_full = update.formula(mod$formula, linear_predictor~.+0)
+    mod$call = lm( formula_full, data=catchability_data_full)$call
+  }
+
   if( FALSE ){
     Tmp = model.matrix( formula_full, data=fit$effects$catchability_data )
   }
@@ -142,15 +156,15 @@ function( focal.predictors,
   }
 
   # Bundle arguments
-  args = list(call = mod$call,
+  args = list( call = mod$call,
                coefficients = mod$coefficients,
                vcov = mod$vcov,
                family = fam,
                formula = formula_full)
 
   # Do call
-  effects::Effect.default(focal.predictors,
+  tmp = effects::Effect.default(focal.predictors,
     mod,
-    ...,
+    #...,
     sources = args)
 }