Task 3. User-Centred Health Information Retrieval (IR)
Laypeople have different search behaviour, which can be seen both in the formulation of queries and expectations for retrieved documents. To support these needs, our goal is to develop methods and resources for the evaluation of Information Retrieval (IR) from patients’ perspective. Towards this, this year's Task 3 is split into two parts:
Task 3a - monolingual information retrieval (IR) task
Task 3a is a standard TREC-style IR task using (a) the 2012 crawl of approximately one million medical documents made available by the EU-FP7 Khresmoi project (http://www.khresmoi.eu/) in plain text form which was used in CLEF eHealth 2013's Task 3 (this year we provide an improved cleaned version of the document collection which was distributed last year) and (b) a new 2014 set of English general public queries that individuals may realistically pose based on the content of their discharge summaries (this year, as distinct from last year, we use the main disorder diagnosed in the discharge summary [last year a random disorder was selected from the discharge summary]). Queries are generated from the discharge summaries used in Tasks 2. The goal of Task 3a is to retrieve the relevant documents for the user queries.
Task 3b - multilingual IR task
Task 3b extends Task 3a by providing a translation of the queries from Task 3a into German, French and Czech. The goal in Task 3b is to develop techniques to translate these queries into English and then apply them to the retrieval task 3a.
Task 3a and 3b will operate by distributing the test collection (document set, sample development queries (in English, German, French, Czech), and result set) to registered task participants. Participants will have one month to explore the collection and develop retrieval techniques, after which test queries for the task will be released. Post-submission relevance assessment will be conducted using a pool of the submitted runs. Result sets for the task and performance measures will be distributed to participants.
Participants are free to take part in Task 3a or Task 3b or both.