A reliability generalization study of the STEM-CIS scale: Exploring moderator effects

Abstract

A reliability generalization (RG) study is crucial for assessing and improving scale reliability across contexts, guiding future research, and ensuring valid results. Therefore, our study aim was to conduct an RG of the STEM (Science, Technology, Engineering, and Mathematics) Career Interest Scale (STEM-CIS) to assess its reliability across contexts and explore factors influencing reliability. Assuming a random-effects model, we found a strong reliability coefficient of.92 (95% CI [.91,.93]) for the STEM-CIS by analyzing data from 39 studies using the transformed reliability coefficient values from Bonett's formula. Both the overall scale and its subscales (Science, Technology, Engineering, and Mathematics) demonstrated substantial heterogeneity, indicating variability in internal consistency across studies. ANOVA and meta-regression analyses were conducted to investigate variability of Cronbach's alpha estimates. Results showed the most important moderators were sample size, language, country, test version (original or adapted scale), and school level. The final predictive model consisted of these important moderators. Results showed these five moderators significantly affected variation of Cronbach's alpha values. Despite notable heterogeneity found among the STEM-CIS subscales, the scale demonstrated satisfactory reliability overall. Researchers should report reliability metrics from their specific datasets to ensure accurate interpretation and application of the scale in varying contexts.