department of informatics

Extending Relevance Judgements for AOR at SemSearch 2010 and 2011



In this page we collect our extended relevance judgements for the AOR task as evaluated at SemSearch 2010 and 2011.
Starting from the original assessment files available at the evaluation webpages:
SemSearch 2010
SemSearch 2011
we extended them by crowdsourcing additional relevance judgements for unjudged entities retrieved by our system in top 10 results.
The resulting extended assessment files are:

Extended assessment file for 2010 
Extended assessment file for 2011 

Row data

The extended assessment files were obtained by processing the output of MTurk judgements:
Original MTurk output files for 2010 
Original MTurk output files for 2011 

With the following Java program:
Script to extend an assessment file with additional judgements