Synthetic Construct Validity: A Transformational Solution

Abstract

In the information systems (IS) discipline, ensuring the accuracy of psychometric measurement through construct validation is paramount for theoretical advancement. Construct validity delineates the degree to which an instrument genuinely measures the concept it purports to assess. Traditional methods like principal components analysis can bias results by employing the same dataset to validate the instrument and test the proposed theory. In contrast, our research introduces a novel computational tool for evaluating construct validity, which validates the instrument separately from the data collected to test the theory. By finetuning a RoBERTa-based transformer model, we estimate the likelihood of indicator pairs being associated with the same construct. These probabilities serve as measures of indicator-to-construct concordance. Our empirical findings reveal significant congruence between our method's probabilistic evaluations and established factor loading measures. The proposed approach improves construct validity assessments and refines IS measurement tools, paving the way for more precise and reliable theoretical model development.