Results from our human intuition experiment at the WWW’13 conference, Rio de Janeiro

Hello all,

at the WWW’13 conference in Rio de Janeiro, we presented our poster on “Calculating Semantic Relatedness from Human Navigational Paths on Wikipedia” (see my BibSonomy profile). We also collected some user intuitions about the semantic relatedness of “bread” and “butter”, two exemplary words from the WordSimilarity353 dataset.

The WordSimilarity353 dataset contains 353 pairs of english words or names, combined with a human judgment of semantic similarity between these word pairs. For all word pairs, at least 10 different persons rated the similarity by assigning a value between 0 and 10, where 0 represents no relation and 10 represents high relation. Some basic statistical data are given right here:

WordSimilarity353 ratings

Rating Score between 0 and 1065.589687564439
The ratings of 13 human judges of the similarity between "bread" and "butter"

On a Google Form, we asked some visitors to give us their intuition about the semantic similarity between “bread” and “butter”. The results are shown below:

WWW'13 visitors' ratings

Rating Score between 0 and 1010689878910377778
The ratings from the WWW'13 visitors.

We calculated the mean and the standard deviation of both results:

Mean and Standard Deviation

 MeanStandard Deviation
WordSim3536.191.93
WWW'137.61.41
The mean and standard deviation of both ratings.

As you can see, the visitors of the conference see a quite higher relatedness score than the judges from the WordSimilarity353 creators’ team. The WS353 ratings show a higher standard deviation from the average rating than the WWW’13 visitors. One might only guess why the scientists rated the relationship so much higher; It could have something to do with the time we acquired the ratings – right before dinner.

Leave a Reply