Faculty Confidence and Engagement Survey: Validation and Analysis with the Rasch Model
Date
2013-05
Authors
Contributor
Advisor
Department
Instructor
Depositor
Speaker
Researcher
Consultant
Interviewer
Narrator
Transcriber
Annotator
Journal Title
Journal ISSN
Volume Title
Publisher
Volume
Number/Issue
Starting Page
Ending Page
Alternative Title
Abstract
This presentation explains how the Item Response Theory Rasch model was used to validate and analyze the Faculty Confidence and Engagement Survey (FaCES) administered at a large urban community college. The presentaton illustrates the advantages of using the Rasch model to analyze survey ratings, the major steps in the analysis, and the important statistics to be used for instrument revision, interpretation, and reporting purposes. The presenter shares major findings related to the survey’s construct validity and findings on faculty/staff’s level of confidence and engagement in different aspects.
Description
The study explored using the Item Response Theory (IRT) Rasch model to validate and analyze the Faculty Confidence and Engagement Survey (FaCES) used in a large urban community college. The FaCES survey was developed collaboratively by the professional development leaders on campus. It aimed to gauge faculty/staff’s confidence and engagement in areas such as teaching practices, primary professional responsibilities, professional development, collaboration, scholarship, willingness to help and seek help, sense of trust, institutional responsibilities, and etc. The survey utilized 36 five-point Likert scale items. The data used in the analysis came from 194 respondents in the Spring 2012 survey administration.
Using IRT Rasch Model analysis advances the technique in survey rating data validation and analysis for the following reasons:
1. The traditional factor analysis technique in survey construct validation can be misleading because it treats survey ratings as interval scale rather than ordinal scale, and because the resulting factors were treated as equivalent in data interpretation, while statistically they contribute differently to the variance in the data.
2. Rasch IRT analysis allows the evaluators and researchers to discover the “true” ratings, and thus, more accurate answers about these ratings. In Rasch analysis, using probability estimation, an item’s endorsability/difficulty and a person’s level of confidence can be estimated simultaneously using logit scores. The resulting logit scores are on a true interval scale.
3. “Not all items contribute to the same latent trait, and of those items that do, they do not all contribute equally” (Bond & Fox, 2007). The presenter will share the major steps in the Rasch model analysis, and the important statistics that should be used for instrument revision, interpretation, and reporting purposes.
The major findings of the study are as follows: (1) the instrument satisfactorily measured the holistic construct of faculty/staff’s confidence and engagement; (2) the engagement items related to teaching practices and duties/responsibilities within one’s locus of control were more easily endorsed and the items related to scholarship, collaboration, duties/responsibilities outside one’s locus of control, or involving support from administration were more difficult to endorse.
Keywords
Rasch models, faculty confidence and engagement, survey analysis method, evaluation of faculty professional development
Citation
Hill, Y. Z. (2013, May). Faculty confidence and engagement survey: Validation and analysis with the Rasch model. Paper presented at the Association for Institutional Research Annual Forum, Long Beach, CA.
Extent
Format
Geographic Location
Time Period
Related To
Related To (URI)
Table of Contents
Rights
Rights Holder
Local Contexts
Collections
Email libraryada-l@lists.hawaii.edu if you need this content in ADA-compliant format.