Active Ordering of Interactive Prediction Tasks

Date of Original Version



Conference Proceeding

Journal Title

Proceedings of the SIAM International Conference on Data Mining

First Page


Last Page


Rights Management

Copyright 2010 SIAM. The version of record is available online at http://dx.doi.org/10.1137/1.9781611972801.47

Abstract or Description

Many applications involve a set of prediction tasks that must be accomplished sequentially through user interaction. If the tasks are interdependent, the order in which they are posed may have a significant impact on the effective utilization of user feedback by the prediction systems, affecting their overall performance. This paper presents a novel approach for dynamically ordering a series of prediction tasks by taking into account the effect of user feedback on the performance of multiple prediction systems. The proposed approach represents a general strategy for learning incrementally during test phase when the system interacts with the end-user, who expects good performance instead of merely providing correct labels to the system. Therefore, the system must balance system benefit against user benefit when selecting items for user’s attention. We apply the proposed approach to two practical applications that involve interactive trouble report generation and document annotation, respectively. Our experiments show significant improvements in prediction performance (in terms of Mean Average Precision) using the proposed active ordering approach, as compared to baseline approaches that either determine a task order offline and hold it fixed during test phase, or do not optimize the order at all.



This document is currently not available here.



Published In

Proceedings of the SIAM International Conference on Data Mining, 537-547.