5 − 4 ≠ 4 − 3: On the Uneven Gaps between Different Levels of Graded User Satisfaction in Interactive Information Retrieval Evaluation

Simple item page
dc.contributor.authorLiu, Jiqun
dc.contributor.authorHan, Fangyuan
dc.date.accessioned2022-12-27T19:12:04Z
dc.date.available2022-12-27T19:12:04Z
dc.date.issued2023-01-03
dc.description.abstractSimilar to other ground truth measures, graded user satisfaction has been frequently employed as a continuous variable in information retrieval evaluation based on the assumption that intervals between adjacent grades are quantitatively equal. To examine the validity of equal-gap assumption and explore dynamic perceptual thresholds triggering grade changes in search evaluation, we investigate the extent to which users are sensitive to changes in search efforts and outcomes across different gaps of graded satisfaction. Experiments on four user study datasets (15,337 queries) indicate that 1) User satisfaction sensitivity, especially to offline evaluation metrics, changes significantly across gaps in satisfaction scale; 2) the size and direction of changes in sensitivity vary across study settings, search types, and intentions, especially within “3-5” scale subrange. This study speaks to the fundamentals of user-centered evaluation and advances the knowledge of heterogeneity in satisfaction sensitivity to search efforts and gains and implicit changes in evaluation thresholds.
dc.format.extent10
dc.identifier.doi10.24251/HICSS.2023.509
dc.identifier.isbn978-0-9981331-6-4
dc.identifier.urihttps://hdl.handle.net/10125/103141
dc.language.isoeng
dc.relation.ispartofProceedings of the 56th Hawaii International Conference on System Sciences
dc.rightsAttribution-NonCommercial-NoDerivatives 4.0 International
dc.rights.urihttps://creativecommons.org/licenses/by-nc-nd/4.0/
dc.subjectIntelligent Information Access and Retrieval
dc.subjectgraded user satisfaction
dc.subjectinformation retrieval evaluation
dc.subjectuser satisfaction sensitivity
dc.title5 − 4 ≠ 4 − 3: On the Uneven Gaps between Different Levels of Graded User Satisfaction in Interactive Information Retrieval Evaluation
dc.type.dcmitext
prism.startingpage4171

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
0407.pdf
Size:
785.21 KB
Format:
Adobe Portable Document Format
Download

Collections

Intelligent Information Access and Retrieval