Dataset Collection for Evaluating Semantic Relatedness

I am researching algorithms to calculate semantic relatedness. Since the purpose of the algorithms is to approximate human judgment as close as possible, it is necessary to compare the algorithm results to human intuition of the same pairs. For that, there exist several datasets which are more or less widely used in current literature.

I uploaded a tar file with some evaluation datasets in a pandas readable format with headers, which can be downloaded under http://www.thomas-niebler.de/downloads/eval_datasets_pandas.tar.gz. The file contains the following datasets:

  • Miller&Charles [1]
  • Rubenstein&Goodenough [2]
  • MEN dataset [3]
  • MTurk dataset [4]
  • WordSimilarity353 [5]

For the WordSimilarity353 dataset, I also included the distinct scores of each rater.

I also published a BitBucket repository with a small toolbox written in python. This repository also contains the data at hand.

Have fun using these data!

Thomas

[1]: http://www.bibsonomy.org/bibtex/22c87f7793dba056ca6b4825aedd0b843/thoni
[2]: http://www.bibsonomy.org/bibtex/266442505e9fd26df63769a44fa13bcd5/thoni
[3]: http://www.bibsonomy.org/bibtex/2db36b22fa6e6b2ab0a60f910651cea44/thoni
[4]: http://www.bibsonomy.org/bibtex/2399d253e5a509b27db83b8f5ad51b901/thoni
[5]: http://www.bibsonomy.org/bibtex/29295f683b417386a11112579f4d61290/thoni

 

 

Leave a Reply