Overview

The main motivation for the INEX iTrack is twofold. First, to investigate the behaviour of users when interacting with elements in richly structured XML documents, and secondly to investigate and develop approaches to element retrieval which are effective in user-based environments.

The INEX interactive track has been run from 2004 onwards. Although there has been variations in task content and focus, some fundamental premises has been in force throughout:

a common subject recruiting procedure
a common set of user tasks and data collection instruments such as interview guides and questionnaires
a common logging procedure for user/system interaction
an understanding that the data collected should be made available to all participants for analysis.

This has ensured that through a manageable effort, participant institutions have had access to a rich and comparable set of data on user background and user behaviour, of sufficient size and level of detail to allow both qualitative and quantitative analysis. This material has already been the source of a number of papers and conference presentations.

In 2010 we intend to continue experiments on the 2009 data collection, a crawl of 2 million records from the book database of the online bookseller Amazon.com, consolidated with corresponding bibliographic records from the cooperative book cataloguing tool LibraryThing. The records present book descriptions on a number of levels: formalized author, title and publisher data; subject descriptions and user tags; book cover images; full text reviews and content descriptions. The data base will enable investigation of research questions concerning, for instance

What is the basis for judgements on relevance in a richly structured and diverse material? What fields / how much descriptive text do users make use of / chose to see to be able to judge relevance?
How do users understand and make use of structure (e.g. representing different levels of description, from highly formalized bibliographic data to free text with varying degrees of authority) in their search development?
How do users construct and change their queries during search (sources of terms, use and understanding of tags, query development strategies ..)?

Participants are invited to design their own investigations, under certain restrictions which will be agreed on, such as:

A version of the Daffodil IR system, used in previous years' iTracks and extended to accommodate the special features of the Amazon data, will be made available for participants to use either alone or in comparison with their own system(s).
Each participating site will be responsible for recruiting test persons to participate in the study as searchers. A minimum number of test persons will be defined so that the obtained results can be statistically meaningful.
Participants must make their data available to all participating groups, and describe their collection process and experimental procedure so that the data can be interpretable for others.
Logging procedures are prepared by the organizers with the aim of making search logs comparable.
Participants must use, but not necessarily be confined to, a common set of topics and searcher tasks.

Links

iTrack 2009 Website
iTrack 2010 Website

Schedule

June 2010	Development of the format of the track, including search themes and topics, user questionnaires etc.
Late Aug 2010	System and experimental design for the Interactive track made available.
Aug-Oct 2010	Search experiments.
15 Oct 2010	Submission deadline for shareable Interactive Track results from participants.
15 Nov 2010	Release of Interactive Track data to participants.

Please notify us of your interest in participating in the track through e-mail to Ragnar Nordlie

Track organizers

Ragnar Nordlie
Nils Pharo
Norbert Fuhr
Thomas Beckers
Khairun Nisa Fachry