Please use this identifier to cite or link to this item: http://hdl.handle.net/10125/70666

Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol

File Size Format  
0046.pdf 1.42 MB Adobe PDF View/Open

Item Summary

Title:Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol
Authors:Pearson, Ross
Fraser, Hannah
Bush, Martin
Mody, Fallon
Widjaja, Ivo
show 7 moreHead, Andy
Wilkinson, David Peter
Sinnott, Richard
Wintle, Bonnie
Burgman, Mark
Fidler, Fiona
Vesk, Peter
show less
Keywords:Design, Development, and Evaluation of Collaboration Technologies
expert elicitation
delphi process
idea protocol
replicability
Date Issued:05 Jan 2021
Abstract:In recent years there has been increased interest in replicating prior research. One of the biggest challenges to assessing replicability is the cost in resources and time that it takes to repeat studies. Thus there is an impetus to develop rapid elicitation protocols that can, in a practical manner, estimate the likelihood that research findings will successfully replicate. We employ a novel implementation of the IDEA (‘Investigate’, ‘Discuss’, ‘Estimate’ and ‘Aggregate) protocol, realised through the repliCATS platform. The repliCATS platform is designed to scalably elicit expert opinion about replicability of social and behavioural science research. The IDEA protocol provides a structured methodology for eliciting judgements and reasoning from groups. This paper describes the repliCATS platform as a multi-user cloud-based software platform featuring (1) a technical implementation of the IDEA protocol for eliciting expert opinion on research replicability, (2) capture of consent and demographic data, (3) on-line training on replication concepts, and (4) exporting of completed judgements. The platform has, to date, evaluated 3432 social and behavioural science research claims from 637 participants.
Pages/Duration:10 pages
URI:http://hdl.handle.net/10125/70666
ISBN:978-0-9981331-4-0
DOI:10.24251/HICSS.2021.055
Rights:Attribution-NonCommercial-NoDerivatives 4.0 International
https://creativecommons.org/licenses/by-nc-nd/4.0/
Appears in Collections: Design, Development, and Evaluation of Collaboration Technologies


Please email libraryada-l@lists.hawaii.edu if you need this content in ADA-compliant format.

This item is licensed under a Creative Commons License Creative Commons