Faculty Confidence and Engagement Survey: Validation and Analysis with the Rasch Model

Hill, Yao Zhang

Faculty Confidence and Engagement Survey: Validation and Analysis with the Rasch Model

Files

Hill2013_05_AIR_Forum_FaCES_RaschAnalysis.pdf (721.33 KB)

Date

2013-05

Authors

Hill, Yao Zhang

Abstract

This presentation explains how the Item Response Theory Rasch model was used to validate and analyze the Faculty Confidence and Engagement Survey (FaCES) administered at a large urban community college. The presentaton illustrates the advantages of using the Rasch model to analyze survey ratings, the major steps in the analysis, and the important statistics to be used for instrument revision, interpretation, and reporting purposes. The presenter shares major findings related to the survey’s construct validity and findings on faculty/staff’s level of confidence and engagement in different aspects.

Description

The study explored using the Item Response Theory (IRT) Rasch model to validate and analyze the Faculty Confidence and Engagement Survey (FaCES) used in a large urban community college. The FaCES survey was developed collaboratively by the professional development leaders on campus. It aimed to gauge faculty/staff’s confidence and engagement in areas such as teaching practices, primary professional responsibilities, professional development, collaboration, scholarship, willingness to help and seek help, sense of trust, institutional responsibilities, and etc. The survey utilized 36 five-point Likert scale items. The data used in the analysis came from 194 respondents in the Spring 2012 survey administration. Using IRT Rasch Model analysis advances the technique in survey rating data validation and analysis for the following reasons: 1. The traditional factor analysis technique in survey construct validation can be misleading because it treats survey ratings as interval scale rather than ordinal scale, and because the resulting factors were treated as equivalent in data interpretation, while statistically they contribute differently to the variance in the data. 2. Rasch IRT analysis allows the evaluators and researchers to discover the “true” ratings, and thus, more accurate answers about these ratings. In Rasch analysis, using probability estimation, an item’s endorsability/difficulty and a person’s level of confidence can be estimated simultaneously using logit scores. The resulting logit scores are on a true interval scale. 3. “Not all items contribute to the same latent trait, and of those items that do, they do not all contribute equally” (Bond & Fox, 2007). The presenter will share the major steps in the Rasch model analysis, and the important statistics that should be used for instrument revision, interpretation, and reporting purposes. The major findings of the study are as follows: (1) the instrument satisfactorily measured the holistic construct of faculty/staff’s confidence and engagement; (2) the engagement items related to teaching practices and duties/responsibilities within one’s locus of control were more easily endorsed and the items related to scholarship, collaboration, duties/responsibilities outside one’s locus of control, or involving support from administration were more difficult to endorse.

Keywords

Rasch models, faculty confidence and engagement, survey analysis method, evaluation of faculty professional development

Citation

Hill, Y. Z. (2013, May). Faculty confidence and engagement survey: Validation and analysis with the Rasch model. Paper presented at the Association for Institutional Research Annual Forum, Long Beach, CA.

URI

http://hdl.handle.net/10125/48504

Collections

Workshops and Presentations

Full item page

Email libraryada-l@lists.hawaii.edu if you need this content in ADA-compliant format.