Summary
Although the speed of computers and volume of information has grown exponentially since the first search engines were created in the 1950s, nearly 6 decades on, the methods used to test state of the art information access systems have changed little. However, there is growing scientific evidence of a need for a complete testing overhaul: producing novel evaluation methods that take a user-oriented perspective on assessing the effectiveness of information access systems. The proposed network will establish a research and training programme for the evaluation of information access systems which will define a new measurement paradigm based on so-called living laboratories. This paradigm involves (i) exploitation of novel market places and forums where large numbers of users are recruited into early stage evaluation experiments to test a particular aspect of an information access system; and (ii) using operational systems as experimental platforms on which to conduct user-based experiments at scale. The network will achieve an evaluation methodology directed at user-oriented interactive evaluation, a new and tested set of interactive evaluation metrics, infrastructure and test suites based on these and an ongoing collaborative forum with evaluation cycles with a focus on evaluating information access systems, as well as producing a European network of new researchers trained in the improved methodologies.
More information can be found on the programme's own website.
Keywords
valuation methodology, Living laboratories, Information access, Interactive information retrieval, Information seeking
Duration
5 years: June 2011 - June 2016
