The Efficacy and Predictive Power of the Diagnostic Assessment and Achievement of College Skills on Academic Success Indicators

Jason M. Bryer¹, Heidi L. Andrade², Timothy J. Cleary³, Angela M. Lui¹, David W. Franklin Jr.¹, and Diana Akhmedjanova²

¹ City University of New York, New York, New York.
² University at Albany, State University of New York, Albany, New York
³ Rutgers, The State University of New Jersey, New Brunswick, New Jersey

Click here for preprint manuscript. Forthcoming in Educational Technology Research and Development journal.

Abstract

The purpose of this study was to examine the effects and predictive power of the Diagnostic Assessment and Achievement of College Skills (DAACS) on student success. DAACS is an open-source diagnostic assessment tool designed to measure newly enrolled college students’ reading, writing, mathematics, and self-regulated learning skills, and to provide individualized feedback and learning resources that students can use to become better prepared for college. A randomized control trial was performed at two online colleges (n = 23,467) to test the effects of DAACS on credit acquisition and retention. The results indicate an overall null effect of treatment, but post hoc analyses reveal two important findings: 1) Students who not only received the assessment results but also accessed the feedback were significantly more likely to earn credits and be retained for a second term than students who only accessed the assessment results; 2) some students who only accessed the assessment results without reading the feedback, particularly those with low scores on the assessments, low self-efficacy, or high test anxiety, had worse outcomes than the control group. We speculate that feedback mitigates the potentially negative effects of testing on student success. In addition, an examination of the predictive power of DAACS indicated that DAACS data significantly strengthen predictions of academic outcomes.

Keywords: college readiness, diagnostic assessment, remediation, predicting student success, developmental education

Author Note: Correspondence concerning this article should be addressed to Jason M. Bryer, City University of New York, School of Professional Studies, 119 W 31st Street, New York, NY 10001. Email: jason.bryer@cuny.edu

Funding: This work was supported by the U.S. Department of Education under Grant under grants #P116F150077 and #R305A210269. Opinions expressed herein are those of the authors and do not necessarily represent the views of the Institute or the U.S. Department of Education.

Ethics Approval and Consent: The authors assert that approval was obtained from an ethics review board (IRB) at Excelsior College in Albany, NY, and Western Governors University in Salt Lake City, Utah. Consent was obtained from participants of the study.

Data, Materials and/or Code Availability: Analysis scripts and supplemental materials are available at https://github.com/daacs/FIPSE-Efficacy.

Author Contributions: The first three authors contributed to the study conception and design. All authors contributed to material preparation. Data collection and analysis were performed by Jason Bryer and Angela Lui. All authors contributed to the writing of the first draft of the manuscript, commented on previous versions of the manuscript, and read and approved the final manuscript.

Acknowledgements: We would like to thank the faculty, staff, and students at Excelsior College and Western Governors University who participated in this study. We would also like to acknowledge our advisory committee members who provided valuble resources and feedback on the DAACS system. List of past and current adivors is available at https://daacs.net/about/.

Repository Structure

• data/ - Final data. This is available upon request.
• R/ - Analysis scripts.
  • R/DataSetup.R - This is a shared script used for each part of the analysis. This script calculates student usage (dosage) of DAACS, creates a variable TreatLevels which classifies students into treatment groups by dosage, and imputes missing values using the mice package.
  • R/descriptives.R - This script generates the descriptive statistics included in the apendicies.
  • R/Analysis_PSA.R - This script runs the propenisty score anlayses.
  • R/Intent_to_Treat.R - This script runs the analysis for research question one analyzing based upon intent-to-treat.
  • R/merge_mids.R - This defines a function that merges the reesults of the imputation with the original dataset which includes a shadow matrix used to test the missing at random assumption.
• Figures/ - Generated figures.
• Tables/ - Generated tables.
• Manuscript/ - Manuscript.