Abstract

In this paper, we study and compare three different but related types of “metadata” about web documents: social annotations provided by readers of web documents, hyperlink anchor text provided by authors of web documents, and search queries of users trying to find web documents. We introduce a large research data set called CABS120k08 which we have created for this study from a variety of information sources such as AOL500k, the Open Directory Project, del.icio.us/Yahoo!, Google and the WWW in general. We use this data set to investigate several characteristics of said metadata including length, novelty, diversity, and similarity and discuss theoretical and practical implications.

CABS120k08, a large research data set about Web metadata based on a sample of 120,000 web documents with data retrieved from the Open Directory Project, the AOL Search query log corpus AOL500k, Google PageRank, Delicious.com, and anchor text from incoming hyperlinks

DMOZ100k06, a large research data set about document metadata based on a random sample of 100,000 web documents