Skip to content

Conversation

@dylanpieper
Copy link

@dylanpieper dylanpieper commented Aug 16, 2025

💥 tidyREDCap gains a big data friendly duckdb backend! 💥

This PR introduces version 1.2.0 with major updates to import_instruments() and new codebook functions:

Adds parameters to import_instruments():

  • filter_instrument and filter_function for lazy data filtering
  • return_list for returning a list of instrument data.frames
  • labels for adding/removing column labels

Currently, you need to install the dev version of redquack: pak::pak("dylanpieper/redquack").

Todos

  • Document in roxygen and examples
  • Document in description, readme, vignettes, and news
  • Add/update unit tests
  • Merge updates with Update useAPI.Rmd #74
  • Submit redquack updates to CRAN

@dylanpieper dylanpieper changed the title import_instruments(): Use redquack for REDCap API export with filtering and list support import_instruments(): Use redquack for REDCap API export with filtering and list support Aug 16, 2025
dylanpieper and others added 6 commits August 17, 2025 07:44
- Bump version from 1.1.2 to 1.2 in DESCRIPTION
- Update NEWS.md with comprehensive 1.2 changelog
- Update README.md with new v1.2 feature highlights
- Switch example URLs from Miami to bbmc.ouhsc.edu (Miami token no longer works)
- Improve vignette formatting and documentation clarity
- Enhance import_instruments() with filtering and list return features
- Add comprehensive test coverage for new parameters
- Preserve record_id labels when drop_blank=TRUE and labels=TRUE
- Keep empty data.frame structure instead of setting to NULL

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
… keyring docs

- Enhanced drop_label() function to support multiple variable selection using tidyselect helpers
- Updated function calls to use new drop_label() interface with multiple variables
- Fixed keyring documentation formatting in useAPI.Rmd vignette
- Regenerated documentation for drop_label() function

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@dylanpieper dylanpieper changed the title import_instruments(): Use redquack for REDCap API export with filtering and list support import_instruments(): Use redquack for REDCap API, add parameters, and update to 1.2 Aug 29, 2025
dylanpieper and others added 3 commits August 30, 2025 14:46
- Replace labelVector dependency with labelled package in DESCRIPTION and NAMESPACE
- Update all label handling functions to use labelled::var_label syntax
- Add Dylan Pieper as package author with ORCID
- Fix label cleaning to remove numbered suffixes (e.g., "Complete?...10" -> "Complete?")
- Remove deprecated dropLabels vignette and related HTML documentation
- Add new sample data files for vignettes
- Update README with clearer package description and functionality
- Update test expectations to match cleaned labels

Fixes #75

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Removed outdated redcapAPI example and enhanced keyring section with practical code examples showing secure token storage and retrieval.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@dylanpieper
Copy link
Author

dylanpieper commented Aug 30, 2025

@RaymondBalise: I updated the PR with substantive changes that should fix a few additional issues, and I made some heavy handed edits to the docs. For example, the REDCap instance for the nacho anxiety examples did not work for me, and it made more sense for me to use the University of Oklahoma's instance, as it's used in the tests. Let me know if you have any concerns, and when you might have time to review.

@dylanpieper
Copy link
Author

dylanpieper commented Aug 31, 2025

🤖 Claude's summary of additional changes:

Functionality:

  • Two-step filtering process: filter one instrument to get record IDs, then apply those IDs to filter all other instruments
  • Memory management includes informative messaging for large projects (25M+ and 100M+ data elements)
  • drop_label() function now supports multiple variables and tidyselect syntax (e.g., drop_label(df, starts_with("redcap_")))
  • Updated all label handling to use modern labelled package methods instead of deprecated labelVector

Testing & Documentation:

  • Refreshed all vignette example data and improved practical filtering examples
  • Updated security recommendations to include keyring package for encrypted token storage
  • Added extensive test suite covering all new parameter combinations and edge cases
  • Added Dylan Pieper as package author for his contributions

These improvements make the package significantly more robust for production use with large REDCap projects.

@dylanpieper dylanpieper changed the title import_instruments(): Use redquack for REDCap API, add parameters, and update to 1.2 import_instruments(): Use redquack for REDCap API, add parameters, and update to 1.2.0 Aug 31, 2025
dylanpieper and others added 7 commits August 31, 2025 18:32
Replace undefined 'redcap' object references with 'sample_data' and update
ingredient variables to use available race variables in sample data.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Add /doc/ and /Meta/ to gitignore and fix bullet point formatting
in import_instruments.Rmd for better readability.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Add simulation of missing data to properly demonstrate the make_instrument
function's ability to filter out completely empty records. Also fix minor
text reference in makeChooseOneTable vignette.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Implement codebook() S3 generic with data.frame and default methods
- Add value label parsing for REDCap categorical variables
- Update NAMESPACE with new exports and CLI imports
- Document new functionality in NEWS.md and README.md
- Add codebook vignette section with usage examples
- Include comprehensive test suite for codebook functionality

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Remove magrittr pipe operator and related files (utils-pipe.R, pipe.Rd)
- Replace %>% with |> throughout codebase
- Add DBI, duckdb, redquack as new dependencies
- Remove magrittr dependency
- Update imports and add missing import declarations
- Add .claude to .Rbuildignore

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@dylanpieper dylanpieper changed the title import_instruments(): Use redquack for REDCap API, add parameters, and update to 1.2.0 import_instruments(): Use redquack for REDCap API, add parameters, add codebook(), and update to 1.2.0 Sep 9, 2025
… labels

- Add codebook_convert() with S3 methods for data.frame and default
- Add has_redcap_values() helper function for identifying labeled columns
- Document functions in README, NEWS, and vignettes with usage examples
- Add comprehensive test coverage for various data types and edge cases
- Support both data.frame$column and data.frame["column"] syntax patterns

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@dylanpieper dylanpieper changed the title import_instruments(): Use redquack for REDCap API, add parameters, add codebook(), and update to 1.2.0 Use redquack for REDCap API, add parameters to import_instruments(), add codebook functions, and update to 1.2.0 Sep 9, 2025
dylanpieper and others added 7 commits September 9, 2025 14:50
- Add comprehensive error checking for REDCapR API calls
- Extract meaningful error messages from REDCap server responses
- Provide specific guidance for common issues (403, 404, network errors)
- Improve user experience with informative error messages instead of generic failures
- Add robust error handling tests covering token validation, API permissions, and network issues
- Maintain original function behavior (invisible return for environment mode, list for list mode)

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Update make_yes_no() and make_yes_no_unknown() functions to preserve
variable labels during factor conversion, addressing issue #50. Both
functions now store original labels before transformation and restore
them on the result using the labelled package.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Replace detailed custom error messages with generic API error messages
- Use outcome_message from REDCapR for metadata and labels read failures
- Add error handling for redcap_to_db data import step
- Update test expectations to match simplified error messages

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Replace apply() with for loops to prevent stack overflow on large datasets. Resolves issue #68 where 1750+ row datasets caused C stack usage errors.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@dylanpieper
Copy link
Author

Added error catching to import functions.
Also fixed bugs in #50 and #68.

Happy to split these into multiple PRs too when we are ready to implement.

@RaymondBalise
Copy link
Owner

RaymondBalise commented Sep 10, 2025 via email

@dylanpieper
Copy link
Author

dylanpieper commented Sep 12, 2025

@RaymondBalise Have fun at REDCapConf! After next week, I'll be out until Oct 6th for my wedding / honeymoon!

I think we should hold off on this PR until I get some functions sorted for redquack's redcap_to_db(). They include tidying the collected data format (i.e., single df, list, and assign to global env) and column and coded value labeling and conversion (by reading metadata using httr2 and storing it in the temporary database). This should streamline a lot of the code for an improved import_instruments() and remove the dependency on REDCapR.

Feel free to browse what I have done here, but we should break up this work into smaller PRs.

@dylanpieper dylanpieper closed this Oct 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

consider replacing {labelVector} with the {labelled} package import_instruments fail after dplyr Handle redcap labels

2 participants