Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol

Simple item page
dc.contributor.authorPearson, Ross
dc.contributor.authorFraser, Hannah
dc.contributor.authorBush, Martin
dc.contributor.authorMody, Fallon
dc.contributor.authorWidjaja, Ivo
dc.contributor.authorHead, Andy
dc.contributor.authorWilkinson, David Peter
dc.contributor.authorSinnott, Richard
dc.contributor.authorWintle, Bonnie
dc.contributor.authorBurgman, Mark
dc.contributor.authorFidler, Fiona
dc.contributor.authorVesk, Peter
dc.date.accessioned2020-12-24T19:03:41Z
dc.date.available2020-12-24T19:03:41Z
dc.date.issued2021-01-05
dc.description.abstractIn recent years there has been increased interest in replicating prior research. One of the biggest challenges to assessing replicability is the cost in resources and time that it takes to repeat studies. Thus there is an impetus to develop rapid elicitation protocols that can, in a practical manner, estimate the likelihood that research findings will successfully replicate. We employ a novel implementation of the IDEA (‘Investigate’, ‘Discuss’, ‘Estimate’ and ‘Aggregate) protocol, realised through the repliCATS platform. The repliCATS platform is designed to scalably elicit expert opinion about replicability of social and behavioural science research. The IDEA protocol provides a structured methodology for eliciting judgements and reasoning from groups. This paper describes the repliCATS platform as a multi-user cloud-based software platform featuring (1) a technical implementation of the IDEA protocol for eliciting expert opinion on research replicability, (2) capture of consent and demographic data, (3) on-line training on replication concepts, and (4) exporting of completed judgements. The platform has, to date, evaluated 3432 social and behavioural science research claims from 637 participants.
dc.format.extent10 pages
dc.identifier.doi10.24251/HICSS.2021.055
dc.identifier.isbn978-0-9981331-4-0
dc.identifier.urihttp://hdl.handle.net/10125/70666
dc.language.isoEnglish
dc.relation.ispartofProceedings of the 54th Hawaii International Conference on System Sciences
dc.rightsAttribution-NonCommercial-NoDerivatives 4.0 International
dc.rights.urihttps://creativecommons.org/licenses/by-nc-nd/4.0/
dc.subjectDesign, Development, and Evaluation of Collaboration Technologies
dc.subjectexpert elicitation
dc.subjectdelphi process
dc.subjectidea protocol
dc.subjectreplicability
dc.titleEliciting group judgements about replicability: a technical implementation of the IDEA Protocol
prism.startingpage461

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
0046.pdf
Size:
1.39 MB
Format:
Adobe Portable Document Format
Download

Collections

Design, Development, and Evaluation of Collaboration Technologies