Date of Original Version

4-2011

Type

Conference Proceeding

Journal Title

Lecture Notes in Computer Science

Volume

6611

First Page

141

Last Page

152

Rights Management

The final publication is available at Springer via http://dx.doi.org/10.1007/978-3-642-20161-5_15

Abstract or Description

Aggregated search is the task of incorporating results from different specialized search services, or verticals, into Web search results. While most prior work focuses on deciding which verticals to present, the task of deciding where in the Web results to embed the vertical results has received less attention. We propose a methodology for evaluating an aggregated set of results. Our method elicits a relatively small number of human judgements for a given query and then uses these to facilitate a metric-based evaluation of any possible presentation for the query. An extensive user study with 13 verticals confirms that, when users prefer one presentation of results over another, our metric agrees with the stated preference. By using Amazon’s Mechanical Turk, we show that reliable assessments can be obtained quickly and inexpensively.

DOI

10.1007/978-3-642-20161-5_15

Share

COinS
 

Published In

Lecture Notes in Computer Science, 6611, 141-152.