Contents
1. Popularity of the Websites within the Common Crawl
In order to get an impression about the coverage of the Common Crawl, we matched the the Alexa top 1 million sites list with the pay-level-domains (PLDs) covered by the Common Crawl. In addition, we calculated the percentages of the popular websites that contain structured data. The table also shows that frequently visited websites are more likely to contain structured data than less popular ones. For instance, nearly 50% of the Top 10,000 websites embed structured data.
| Alexa PLDs in CC | # Alexa PLDs in | % PLDs from WDC included in CC included in Alexa | |||||
| Top # PLDs Alexa List | # | % | WDC Aug 2012 | Overall | RDFa | Microdata | Microformats |
| 100 | 99 | 99.00 | 74 | 74.75 | 34.34 | 55.56 | 68.69 |
| 1,000 | 963 | 96.30 | 603 | 62.62 | 40.08 | 31.67 | 46.11 |
| 10,000 | 9,294 | 92.94 | 4,400 | 47.34 | 30.47 | 15.55 | 29.75 |
| 100,000 | 85,058 | 85.01 | 27,167 | 31.94 | 16.46 | 7.20 | 20.07 |
| 1,000,000 | 734,882 | 73.49 | 151,076 | 20.56 | 7.55 | 3.04 | 14.18 |
2. Distribution by Top-Level-Domain
The following table shows the distribution of the number of websites (Pay-Level-Domains, PLD) for the Top 20 Top-Level-Domain (TLDs) within the Common Crawl (CC). The table also contains the absolute and relative number of PLDs serving structured data as well as using specific formats. The table shows that RDFa, Microdata, and Microformats are used within all TLDs. The deployment on generic TLDs, such as 'com', 'net' or 'info', seem to be slightly stronger than the deployment on country-specific TLDs.
| PLDs per TLD in CC | PLD publishing | structured data | RDFa | Microdata | Microformats | |||||
| TLD | in CC # | # | % | # | % | # | % | # | % | |
| 1 | com | 19,950,689 | 1,317,757 | 6.61 | 292,186 | 1.46 | 75,955 | 0.38 | 1,043,272 | 5.23 |
| 2 | de | 2,810,040 | 79,366 | 2.82 | 15,769 | 0.56 | 4,760 | 0.17 | 64,358 | 2.29 |
| 3 | net | 2,203,474 | 145,547 | 6.61 | 27,041 | 1.23 | 7,958 | 0.36 | 120,339 | 5.46 |
| 4 | org | 2,064,960 | 152,977 | 7.41 | 30,231 | 1.46 | 5,854 | 0.28 | 125,490 | 6.08 |
| 5 | co.uk | 1,448,245 | 64,043 | 4.42 | 14,456 | 1.00 | 4,658 | 0.32 | 48,963 | 3.38 |
| 6 | nl | 951,484 | 28,820 | 3.03 | 8,413 | 0.88 | 2,718 | 0.29 | 20,352 | 2.14 |
| 7 | ru | 699,275 | 27,496 | 3.93 | 4,144 | 0.59 | 2,202 | 0.31 | 22,439 | 3.21 |
| 8 | info | 663,451 | 46,633 | 7.03 | 5,590 | 0.84 | 2,225 | 0.34 | 40,711 | 6.14 |
| 9 | it | 620,726 | 20,068 | 3.23 | 6,478 | 1.04 | 1,647 | 0.27 | 13,905 | 2.24 |
| 10 | com.br | 501,720 | 20,418 | 4.07 | 5,335 | 1.06 | 1,288 | 0.26 | 15,534 | 3.10 |
| 11 | com.au | 428,164 | 21,400 | 5.00 | 6,474 | 1.51 | 2,157 | 0.50 | 14,194 | 3.32 |
| 12 | fr | 425,204 | 29,794 | 7.01 | 6,441 | 1.51 | 3,397 | 0.80 | 22,073 | 5.19 |
| 13 | ch | 390,336 | 8,659 | 2.22 | 2,245 | 0.58 | 486 | 0.12 | 6,437 | 1.65 |
| 14 | pl | 382,670 | 15,524 | 4.06 | 3,951 | 1.03 | 1,414 | 0.37 | 11,309 | 2.96 |
| 15 | cz | 368,429 | 11,271 | 3.06 | 4,970 | 1.35 | 693 | 0.19 | 6,361 | 1.73 |
| 16 | ca | 319,055 | 20,938 | 6.56 | 5,247 | 1.64 | 1,632 | 0.51 | 15,343 | 4.81 |
| 17 | jp | 288,267 | 14,248 | 4.94 | 6,206 | 2.15 | 415 | 0.14 | 8,403 | 2.92 |
| 18 | se | 286,740 | 16,649 | 5.81 | 3,614 | 1.26 | 1,112 | 0.39 | 13,082 | 4.56 |
| 19 | eu | 259,105 | 11,429 | 4.41 | 2,960 | 1.14 | 726 | 0.28 | 8,484 | 3.27 |
| 20 | dk | 247,693 | 10,766 | 4.35 | 2,638 | 1.07 | 1,180 | 0.48 | 7,663 | 3.09 |
3. Main Topical Areas of the Data
Altogether we discovered structured data within 369 million of the 3 billion pages contained in the Common Crawl corpus (12.3%). The pages containing structured data originate from 2.29 million among the 40.5 million websites (PLDs) contained in the corpus (5.65%). Approximately 519 thousand websites use RDFa, while only 140 thousand websites use Microdata. Microformats are used on 1.7 million websites. It is interesting to see that Microformats are used by approximately 2.5 times as many websites as RDFa and Microdata together.
The statistics about the Top Classes by domain count show that the most popular topical areas of the published data (in terms of number of websites providing data about a topic) are:
- persons and organizations (mainly represented using the hCard Microformat used on 1.5 million websites, but also RDFa and Microdata to a smaller extent)
- blog- and CMS-related metadata (represented using RDFa and Microdata)
- navigational metadata (breadcrumbs represented using RDFa and Microdata)
- product and review data (represented using all three formats)
- event data (represented using the hcalendar Microformat).
Additional topical domains with smaller but significant adoption include job postings (represented using Microdata) and recipes (represented using a Microformat).
The topics of the data, as well as the formats and vocabularies used to represent the data, seem to be largely determined by the major consumers the data is targeted at: Google, Facebook, Yahoo!, and Bing. For instance, the examined RDFa data is dominated by the Open Graph Protocol promoted by Facebook, while the examined Microdata is dominated by the vocabularies promoted by Google, Yahoo!, and Bing via schema.org.
In the following, we will analyze the main topical areas in more detail. The analyses are based on the basis statistics of the August 2012 Web Data Commons corpus and the Website-Class-Property Matrixes.
3.1 Product-related Data
RDFa, Microdata and Microformats are all used for the representation of product related-data .
Products in RDFa
We identified three RDFa classes, og:"product", dv:Product, and gr:Offering, that are used each on at least 500 different websites for describing products. og:"product" is the most popular class, being used by more than 19,000 websites.
Products in Microdata
Reviewing all Microdata classes that are used in more than 100 different websites, we could identify four classes, schema:Product, schema:Offer, datavoc:Product, and datavoc:Offer, that are frequently used to describe products or offers.
The following table shows the co-occurences of these classes with other product-related classes on the same website. For instance, 4,308 websites provide product data together with aggregate ratings for these products.
| Class co-occurrence | # PLDs |
|---|---|
| {schema:Product} | 16,612 |
| {schema:Product, schema:Offer} | 7,194 |
| {schema:Product, schema:AggregateRating} | 4,308 |
| {schema:Product, schema:Offer,schema:AggregateRating} | 3,226 |
| {schema:Product, datavoc:Product} | 2,810 |
| {schema:Product, schema:Offer,datavoc:Product, datavoc:Offer-aggregate} | 2,701 |
In addition to the class co-occurrences, we analyzed which properties are frequently used to describe schema:Products. The table below shows that schema:Product/name, schema:Product/description, schema:Product/image, and schema:Product/offers are the most frequently used properties.
| PLDs Total | ||
| Property | # | % |
| schema:Product/name | 14,342 | 86.34 |
| schema:Product/description | 10,297 | 61.99 |
| schema:Product/image | 8,093 | 48.72 |
| schema:Product/offers | 7,545 | 45.42 |
| schema:Offer/price | 6,894 | 41.50 |
| schema:AggregateRating | 4,308 | 25.93 |
| schema:AggregateRating/ratingValue | 3,990 | 24.02 |
| schema:PostalAddress/streetAddress | 3,723 | 22.41 |
Additionaly we analyzed the types of products that are offered by e-shops using Microdata markup.
Products in Microformats
The hListing microformat is used by 4,030 websites to represent product data. Around 80% of these websites also offer information about the price. Around 70% use the properties itemUrl and itemPhoto. Overall we found almost 3,000 websites which offer detailed hListing information (listing, lister, item, price, itemUrl and itemPhoto) within our data set.
3.2. Blog and CMS Metadata
Blog and CMS Data in RDFa
Frequently used classes within the RDFa corpus that are related to blogs and CMS are og:"article", foaf:Document, sioc:Item, sioc:Post, sioct:BlogPost, and sioc:Comment. We analyzed the co-occurence of those classes which is shown in the following table.
| Class co-occurence | # PLDs |
|---|---|
| {foaf:document} | 49,252 |
| {foaf:document, foaf:image} | 36,860 |
| {foaf:document,sioc:item} | 33,132 |
| {foaf:image, sioc:item} | 28,799 |
| {foaf:document, foaf:image, sioc:item} | 28,799 |
| {foaf:document, foaf:image, sioc:item, sioc:useraccount} | 15,180 |
| {foaf:document, foaf:image, sioc:item, sioc:useraccount, skos:concept} | 7,981 |
Blog and CMS Data in Microdata
The class schema:BlogPosting is used by 25,235 websites making it the single most used Microdata class in our data set.
3.3. Open Graph Protocol (OGP)
The most frequently used RDFa classes are dominated by the Open Graph Protocol (OGP). For instance, og:"article" is used by 183,046 websites. OGP is developed and promoted by Facebook in order to ease the integration of external content into the social networking platform. In contrast to RDFa, OGP allows the usage of literals instead of URIs to identify classes. For this reason we added the 'og:'-prefix in the table and mark the literals with quotes.
Deployment of Facebook Classes
We could identify all 9 pre-defined Facebook classes to be used within our corpus. The table below shows the number of pay-level-domains where the mentioned Facebook classes could be found. The first 3 classes are widely used by different websites. Beside those classes the Facebook websites defines 6 other classes, which are not that widely adopted yet.
| Class | # PLD | |
|---|---|---|
| 1 | og:"article" | 183,046 |
| 2 | og:"blog" | 58,971 |
| 3 | og:"website" | 56,573 |
| 4 | og:"book" | 742 |
| 5 | og:"profile" | 394 |
| 6 | og:"video.movie" | 50 |
| 7 | og:"video.other" | 47 |
| 8 | og:"video.episode" | 20 |
| 9 | og:"video.tv_show" | 18 |
Facebook Class co-occurence
To determine if Facebook classes are used in combination with non-Facebook classes, we analyzed the co-occurrence of the three most frequent OGP classes together with other classes. The table below shows that only a very small fraction of the websites use Facebook classes together with other classes from non-Facebook vocabularies (FOAF being used together with OGP by 2,452 out of 183,046 websites).
| Class co-occurrence | #PLD | |
|---|---|---|
| 1 | {og:"article"} | 183,046 |
| 2 | {og:"article", og:"blog"} | 34,978 |
| 3 | {og:"article", og:"website"} | 20,706 |
| 4 | {og:"article", foaf:document} | 2,452 |
| 5 | {og:"article", og:"blog", og:"website"} | 2,443 |
| 6 | {og:"article", foaf:image} | 2,126 |
3.4. Job Postings
We identified 552 websites using schema:JobPostings. Based on the co-occurrence analysis between this class and co-occurring properties we see that around 50% of the PLDs using schema:JobPostings also use properties to describe the title, description and the location of the job. This means our corpus includes around 250 websites which provide comprehensive job descriptions and could potentially be good starting points for focused deeper crawling.
3.5. Recipes
We identified 3,278 websites offering structured data about recipes in general. Over 80% do list ingredients for their recipes and 20% of those sites offer additional information like durations and nutrition information. Besides the hRecipe classes, around 40% of the websites use in addition hCard to include information about the author. This means our corpus includes over 600 pages which contain comprehensive information about recipes and could potentially be good starting points for crawling.