Summary

Like other software artifacts, questions and answers on Stack Overflow evolve over time, for example when bugs in code snippets are fixed or text surrounding a code snippet is edited for clarity. To be able to analyze how Stack Overflow posts evolves, we built SOTorrent, an open dataset based on the official Stack Overflow data dump. SOTorrent provides access to the version history of Stack Overflow content at the level of whole posts and individual text or code blocks. It connects Stack Overflow posts to other platforms by aggregating URLs from text blocks and comments, and by collecting references from GitHub files to Stack Overflow posts.
More information about SOTorrent and the results of a first analysis using the dataset can be found in this blog post and in the corresponding research paper. The source code of the tools used to build and analyze SOTorrent is available on GitHub.

Database Layout

SOTorrent contains all tables from the official Stack Overflow data dump (see schema documentation here).
Additionally, the dataset provides the following tables:

PostVersion: Version history on the level of whole Stack Overflow posts.PostBlockVersion: Version history on the level of post blocks, which are either text (1) or code (2) blocks.PostBlockDiff: Line-based difference between post block versions.TitleVersion: Version history of question titles.PostVersionUrl: URLs extracted from text block versions.CommentUrl: URLs extracted from post comments.PostReferenceGH: URLs extracted from GitHub repositories pointing to Stack Overflow posts (information about included projects can be found here).

In the following database schema (based on version 2018-08-28), the tables from the offical Stack Overflow data dump are marked gray, the additional tables are marked blue. Please note that not all foreign key constraints are shown.

SOTorrent Dataset Versions

The changelog for each dataset version is available here. Please note that newline characters are escaped as &#xD;&#xA; in the BigQuery versions of the dataset.