Modern Information Retrieval Chapter 10: User Interfaces and Visualization

Contents

     
1. Interfaces for Standard Relevance Feedback

relevance feedback interfaces!standard interfaces

A standard interface for relevance feedback consists of a list of titles with checkboxes beside the titles that allow the user to mark relevant documents. This can imply either that unmarked documents are not relevant or that no opinion has been made about unmarked documents, depending on the system. Another option is to provide a choice among several checkboxes indicating relevant or not relevant (with no selection implying no opinion). In some cases users are allowed to indicate a value on a relevance scale [#!belew96!#]. Standard relevance feedback algorithms usually do not perform better given negative relevance judgement evidence [#!dunlop97!#], but machine learning algorithms can take advantage of negative feedback [#!pazzani96!#,#!kozierok93!#].

After the user has made a set of relevance judgements and issued a search command, the system can either automatically reweight the query and re-execute the search, or generate a list of terms for the user to select from in order to augment the original query. (See Figure , taken from [#!koenemann96!#].) Systems usually do not suggest terms to remove from the query.

  

Figure: An example of an interface for relevance feedback [#!koenemann96!#].

After the query is re-executed, a new list of titles is shown. It can be helpful to retain an indicator such as a marked checkbox beside the documents that the user has already judged. A difficult design decision concerns whether or not to show documents that the user has already viewed towards the top of the ranked list [#!aalbersberg92!#]. Repeatedly showing the same set of documents at the top may inconvenience a user who is trying to create a large set of relevant documents, but at the same time, this can serve as feedback indicating that the revised query does not downgrade the ranking of those documents that have been found especially important. One solution is to retain a separate window that shows the rankings of only the documents that have not been retrieved or ranked highly previously. Another solution is to use smaller fonts or gray-out color for the titles of documents already seen.

relevance feedback interfaces!multiple vs. single judgements

Creating multiple relevance judgements is an effortful task, and the notion of relevance feedback is unfamiliar to most users. To circumvent these problems, Web-based search engines have adopted the terminology of `more like this' as a simpler way to indicate that the user is requesting documents similar to the selected one. This `one-click' interaction method is simpler than standard relevance feedback dialog which requires users to rate a small number of documents and then request a reranking. Unfortunately, in most cases relevance feedback requires many relevance judgements in order to work well. To partly alleviate this problem, Aalbersberg [#!aalbersberg92!#] proposes incremental relevance feedback which works well given only one relevant document at a time and thus can be used to hide the two-step procedure from the user.