Web Data Commons - Microdata, RDFa, JSON-LD, and Microformat Data Sets

As the charts show a large fraction of websites make already use of embedded JSON-LD. In most cases (>90%) the websites use the syntax to enable Google to create a search box within the search results, as annonced by Google in September 2014. A interesting discussion about the topic can be found in the Google+ Posting by Aaron Bradley. >

3.10. Extraction Results from the December 2014 Common Crawl Corpus

3.11. Extraction Results from the November 2013 Common Crawl Corpus

3.12. Extraction Results from the August 2012 Common Crawl Corpus

3.12b. Extraction Results from the February 2012 Common Crawl Corpus

3.13. Extraction Results from the 2009/2010 Common Crawl Corpus

Extraction Costs

The costs for parsing the 28.9 Terabytes of compressed input data of the 2009/2010 Common Crawl corpus, extracting the RDF data and storing the extracted data on S3 totaled 576 EUR (excluding VAT) in Amazon EC2 fees. We used 100 spot instances of type c1.xlarge for the extraction which altogether required 3,537 machine hours.

4. Example Data

For each data format, we provide a small subset of the extracted data below for testing purposes. The data is encoded as N-Quads, with the forth element used to represent the provenance of each triple (the URL of the page the triple was extracted from). Be advised to use a parser which is able to skip invalid lines, since they could present in the data files.

• html-rdfa
• html-microdata
• html-embedded-jsonld
• html-mf-hcard
• html-mf-hcalendar
• html-mf-xfn
• html-mf-geo
• html-mf-hlisting
• html-mf-hrecipe
• html-mf-hresume
• html-mf-hreview
• html-mf-species

5. Note about the N-Quads Download Files

It is important to note that the N-Quads download files do not conform completely with the N-Quads specification concerning blank node identifiers. The specification requires labels of distinct blank nodes to be unique with respect to the complete N-Quads document. In our N-Quads files, the blank node labels are unique only with respect to the HTML page from which the data was extracted. This means that different blank nodes in a download file may have the same label. For distinguishing between these nodes, the blank node label needs to be considered together with the URL of the page from which the data was extracted (the fourth element of the quad). This issue is due to 100 machines working in parallel on the extraction of the data from the web corpus without communicating with each other. We may fix this issue in upcoming WDC releases by renaming the blank nodes.

6. Conversion to Other Formats

We provide the extracted data for download using a variation of the N-Quads format. For users who prefer other formats, we provide code for converting the download files into the CSV and JSON formats, which are supported by a wider range of spreadsheet applications, relational databases are data mining tools.

The conversion tool takes the following parameters:

Parameter Name	Description
out	Folder where the outputfile(s) are written to
in	Folder containing WDC download files in N-Quads format
threads	Number of threads used for the conversion
convert	Indicates the output format. Supported formats: JSON, CSV
density	Indicates the minimum density of properties in order be included in the output file. Range: 0 - 1. A density of 0.2 indicates that the properties that have more than 20% non-null values will be included in the output .
multiplePropValues	Indicates if the converted result should contain all property values for a certain property of a subject or if one value per property is enough. Range: [true/false]

Below you can find an example command which transforms the files found in the input directory to JSON files using 5 threads and density as well as property value filtering.

java -cp  StatsCreator-0.0.2-SNAPSHOT-jar-with-dependencies.jar org.webdatacommons.structureddata.stats.WDCQuadConverter  -out "output_convert"
            
-in "input_convert" -threads 5 -tp "http://www.w3.org/1999/02/22-rdf-syntax-ns#type,http://ogp.me/ns#type,http://opengraphprotocol.org/schema/type"
            
-convert "JSON" -density 0.15 -multiplePropValues true

File Structure

CSV file format

Each file starts with three fixed headers [graph, subject, type] followed by the property set of headers. Every line after the header represents one entity. You can find a sample CSV file with the structure of the conversion output here.

JSON file format

Each file contains a list of JSON objects with three fixed properties [graph, subject, type] followed by the property set describing the concrete entity. Every JSON object in the file represents one entity. You can find a sample JSON file with the structure of the conversion output here.

Conversion Process

In the following, we document the conversion process that is performed by the tool: The first step of the conversion process is to sort the input .nq file by subject and URL. For this purpose a temporary file containing the sorted entities is created and deleted by the end of the conversion. The size of the temporary file is equal to the size of the input .nq file. Next, the retrieved entities are written in the converted file. In the case of the CSV file format, all the distinct predicates are stored during parsing for the header row to be filled. In the case of the JSON file format the entities are transformed from Java objects to JSON ones with the help of the Gson library. The provided tool supports parallel execution on the directory level, meaning that multiple files can be converted simultaneously. In addition, the conversion tool provides density and property value filtering. In the case of density filtering the user can set a density threshold in order to filter uncommon properties. Please note that in the average case the maximum property density is calculated to be 35%, so a relatively high threshold could lead to empty property value results. In the case of property value filtering, the user can choose if the converted file should keep track of all the multiple values of a certain property belonging to a certain subject or if one value is enough for his/ her purposes.

7. Extraction Process

Since the Common Crawl data sets are stored in the AWS Simple Storage Service (S3), it made sense to perform the extraction in the Amazon cloud (EC2). The main criteria here is the cost to achieve a certain task. Instead of using the ubiquitous Hadoop framework, we found using the Simple Queue Service (SQS) for our extraction process increased efficiency. SQS provides a message queue implementation, which we use to co-ordinate the extraction nodes. The Common Crawl dataset is readily partitioned into compressed files of around 100MB each. We add the identifiers of each of these files as messages to the queue. A number of EC2 nodes monitor this queue, and take file identifiers from it. The corresponding file is then downloaded from S3. Using the ARC file parser from the Common Crawl codebase, the file is split into individual web pages. On each page, we run our RDF extractor based on the Anything To Triples (Any23) library. The resulting RDF triples are then written back to S3 together with the extraction statistics, which are later collected. The advantage of this queue is that messages have to be explicitly marked as processed, which is done after the entire file has been extracted. Should any error occur, the message is requeued after some time and processed again.

Any23 parses web pages for structured data by building a DOM tree and then evaluates XPath expressions to find structured data. While profiling, we found this tree generation to account for much of the parsing cost, and we have thus searched for a way to reduce the number of times this tree is built. Our solution is to run (Java) regular expressions against each webpages prior to extraction, which detect the presence of a microformat in a HTML page, and then only run the Any23 extractor when the regular expression find potentional matches. The formats html-mf-hcard, html-mf-hcalendar, html-mf-hlisting, html-mf-hresume, html-mf-hreview and html-mf-recipe define unique enough class names, so that the presence of the class name in the HTML document is ample indication of the Microformat being present. For the remaining formats, the following table shows the used regular expressions.

Format	Regular Expression
html-rdfa	(property|typeof|about|resource)\\s*=
html-microdata	(itemscope|itemprop\\s*=)
html-mf-xfn	<a[^>]*rel\\s*=\\s*(\"|')[^\"']*(contact|acquaintance|friend|met|co-worker|colleague|co-resident|neighbor|child|parent|sibling|spouse|kin|muse|crush|date|sweetheart|me)
html-mf-geo	class\\s*=\\s*(\"|')[^\"']*geo
html-mf-species	class\\s*=\\s*(\"|')[^\"']*species

8. Source Code

The source code can be checked out from our Github repository. Afterwards, create your own configuration by copying src/main/resources/ccrdf.properties.dist to src/main/resources/ccrdf.properties, then fill in your AWS authentication information and bucket names. Compilation is performed using Maven, thus changing into the source root directory and typing mvn install should be sufficient to create a build. In order to run the extractor on more than 10 EC2 nodes, you will have to request an EC2 instance limit increase for your AWS account. More information about running the extractor is provided in the file readme.txt .

9. License

The Web Data Commons extraction framework can be used under the terms of the Apache Software License.

10. Feedback

Please send questions and feedback to the Web Data Commons mailing list or post them in our Web Data Commons Google Group.

More information about Web Data Commons is found here.

11. Credits

Web Data Commons has started as a joint effort of the Freie Univeristät Berlin and the Institute AIFB at the Karlsruhe Institute of Technology in early 2012. Now it is mainly maintained by the Research Group Data and Web Science at the University of Mannheim.

We thank our former contributors for the help and support:

• Hannes Mühleisen, now working at CWI in Amsterdam for the initial version of the extraction code and the first two releases.
• Andreas Harth and Steffen Stadtmüller, working at Institute AIFB at KIT for their support during the first releases in 2012 and the hosting parts of the data.
• Michael Schuhmacher, Johanna Völker working at the DWS Group and Kai Eckert, now working at the Hochschule der Medien in Stuttgart for their support creating the expressive statistics of the release in 2012.
• Petar Petrovski for extracting the data and creating the statistics for the 2013 release.

Also lots of thanks to

• the Common Crawl project for providing their great web crawl and thus enabling Web Data Commons.
• the Any23 project for providing their great library of structured data parsers.

Web Data Commons was supported by the PlanetData and LOD2 research projects.

The extraction and analysis of the October 2016 and the November 2017 corpora was supported by the ViCE research project and the Ministry of Economy, Research and Arts of Baden - Württemberg .

   

12. References

• Alexander Brinkmann, Anna Primpeli, Christian Bizer: The Web Data Commons Schema.org Data Set Series. In Companion Proceedings of the ACM Web Conference 2023 (WWW ’23 Companion), Austin, Texas, USA, April 2023.
• Robert Meusel, Petar Petrovski, Christian Bizer: The WebDataCommons Microdata, RDFa and Microformat Dataset Series. In Proceedings of the 13th International Semantic Web Conference: Replication, Benchmark, Data and Software Track (ISWC2014).
• Robert Meusel, Christian Bizer, Heiko Paulheim: A Web-scale Study of the Adoption and Evolution of the schema.org Vocabulary over Time. 5th International Conference on Web Intelligence, Mining and Semantics (WIMS2015), Limassol, Cyprus, July 2015.
• Robert Meusel and Heiko Paulheim: Heuristics for Fixing Common Errors in Deployed schema.org Microdata, To appear in: Proceedings of the 12th Extended Semantic Web Conference (ESWC 2015), Portoroz, Slovenia, May 2015.
• Christian Bizer, Kai Eckert, Robert Meusel, Hannes Mühleisen, Michael Schuhmacher, and Johanna Völker: Deployment of RDFa, Microdata, and Microformats on the Web - A Quantitative Analysis. In Proceedings of the 12th International Semantic Web Conference, Part II: In-Use Track, pp.17-32 (ISWC2013).
• Hannes Mühleisen, Christian Bizer: Web Data Commons - Extracting Structured Data from Two Large Web Corpora. In Proceedings of the WWW2012 Workshop on Linked Data on the Web (LDOW2012).
• Peter Mika, Tim Potter: Metadata Statistics for a Large Web Corpus. In Proceedings of the WWW2012 Workshop on Linked Data on the Web (LDOW2012).
• Peter Mika: Microformats and RDFa deployment across the Web. Blog Post.
• Class Statistics from the Sindice data search engine.