at the WWW’13 conference in Rio de Janeiro, we presented our poster on “Calculating Semantic Relatedness from Human Navigational Paths on Wikipedia” (see my BibSonomy profile). We also collected some user intuitions about the semantic relatedness of “bread” and “butter”, two exemplary words from the WordSimilarity353 dataset.
The WordSimilarity353 dataset contains 353 pairs of english words or names, combined with a human judgment of semantic similarity between these word pairs. For all word pairs, at least 10 different persons rated the similarity by assigning a value between 0 and 10, where 0 represents no relation and 10 represents high relation. Some basic statistical data are given right here:
|Rating Score between 0 and 10||6||5.5||8||9||6||8||7||5||6||4||4||3||9|
On a Google Form, we asked some visitors to give us their intuition about the semantic similarity between “bread” and “butter”. The results are shown below:
WWW'13 visitors' ratings
|Rating Score between 0 and 10||10||6||8||9||8||7||8||9||10||3||7||7||7||7||8|
We calculated the mean and the standard deviation of both results:
Mean and Standard Deviation
As you can see, the visitors of the conference see a quite higher relatedness score than the judges from the WordSimilarity353 creators’ team. The WS353 ratings show a higher standard deviation from the average rating than the WWW’13 visitors. One might only guess why the scientists rated the relationship so much higher; It could have something to do with the time we acquired the ratings – right before dinner.