Blog – Common Crawl

November/December 2020 crawl archive now available

December 10, 2020Sebastian Nagel

The crawl archive for November/December 2020 is now available! The data was crawled between November 23 and December 6 and contains 2.64 billion web pages or 270 TiB of uncompressed content. It includes page captures of 1.4 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The November/December crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-50/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line, you end up with the S3 and HTTP paths respectively.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-50/segment.paths.gz	100
WARC files	CC-MAIN-2020-50/warc.paths.gz	72000	59.95
WAT files	CC-MAIN-2020-50/wat.paths.gz	72000	17.82
WET files	CC-MAIN-2020-50/wet.paths.gz	72000	7.89
Robots.txt files	CC-MAIN-2020-50/robotstxt.paths.gz	72000	0.2
Non-200 responses files	CC-MAIN-2020-50/non200responses.paths.gz	72000	1.71
URL index files	CC-MAIN-2020-50/cc-index.paths.gz	302	0.2

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-50/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

October 2020 crawl archive now available

November 7, 2020Sebastian Nagel

The crawl archive for October 2020 is now available! The data was crawled between October 19th and November 1st and contains 2.71 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.5 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The October crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-45/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-45/segment.paths.gz	100
WARC files	CC-MAIN-2020-45/warc.paths.gz	72000	63.79
WAT files	CC-MAIN-2020-45/wat.paths.gz	72000	18.39
WET files	CC-MAIN-2020-45/wet.paths.gz	72000	8.23
Robots.txt files	CC-MAIN-2020-45/robotstxt.paths.gz	72000	0.2
Non-200 responses files	CC-MAIN-2020-45/non200responses.paths.gz	72000	1.75
URL index files	CC-MAIN-2020-45/cc-index.paths.gz	302	0.21

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-45/. Also the columnar index has been updated to contain this crawl.

Interactive Webgraph Statistics Notebook Released

October 28, 2020Alex Xue

We are pleased to announce the release of an interactive Jupyter notebook that is used to provide:

Visualization of web graph statistics
An interface for interacting with the webgraph

The visualization of the web graph statistics is done by leveraging the WebGraph framework, which provides means of gathering many interesting data points of a web graph, such as the frequency distribution of indegrees/outdegrees in the graph, or size distributions of the connected components. We then are able to use pandas and matplotlib to provide a visualization for the data provided by WebGraph. This effort was largely inspired by the Topology of the 2012 WDC Hyperlink Graph document. Further details of WebGraph tool installation/usage, and the data visualization may be found in the cc-notebooks repository.

The interface for interacting with the webgraph is done by using pyWebGraph, a front end that interfaces Jython with WebGraph. First, before using this interface we must re-build the string maps, in order to create a mapping between the node ID (a numerical value), to domain name (and vice versa). Once this is established we are able to simply load up the graph into pyWebGraph, and you will be able to traverse the graph interactively.

Further details of pyWebGraph installation/usage, and how to rebuild the string maps may be found in interactive webgraph README of the cc-notebooks repository.

The Jupyter notebook is available on Github in the same repository. More details about how to navigate the repository can be found in the notebook itself, as well as in the README.

We hope that users will be able to use these notebooks to gain more insight into the web graph in a numerical and practical sense.

We are grateful for WebGraph for providing extremely useful tools for processing the web graph itself, and Massimo Santini for developing pyWebGraph.

Host- and Domain-Level Web Graphs Jul/Aug/Sep 2020

October 16, 2020Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August and September 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

Host-level graph

The graph consists of 539 million nodes and 3.02 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 467 million dangling nodes (86.7%) and the largest strongly connected component contains 46 million (8.5%) nodes.

You can download the graph and the ranks of all 539 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2020-jul-aug-sep/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2020-jul-aug-sep/host/ as prefix to access the files from everywhere.

Size	File	Description
3.32 GB	cc-main-2020-jul-aug-sep-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 12 vertices files
13.7 GB	cc-main-2020-jul-aug-sep-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 24 edges files

5.95 GB	cc-main-2020-jul-aug-sep-host.graph	graph in BVGraph format
2 kB	cc-main-2020-jul-aug-sep-host.properties

6.76 GB	cc-main-2020-jul-aug-sep-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2020-jul-aug-sep-host-t.properties

1 kB	cc-main-2020-jul-aug-sep-host.stats	WebGraph statistics

7.77 GB	cc-main-2020-jul-aug-sep-host-ranks.txt.gz	harmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 89 million nodes and 1.71 billion edges. 51% or 45 million nodes are dangling nodes, the largest strongly connected component covers 35 million or 39% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2020-jul-aug-sep/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2020-jul-aug-sep/domain/.

Download files of the Common Crawl Jul/Aug/Sep 2020 domain-level webgraph

Size	File	Description
0.61 GB	cc-main-2020-jul-aug-sep-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
6.80 GB	cc-main-2020-jul-aug-sep-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.75 GB	cc-main-2020-jul-aug-sep-domain.graph	graph in BVGraph format
2 kB	cc-main-2020-jul-aug-sep-domain.properties

3.69 GB	cc-main-2020-jul-aug-sep-domain-t.graph	transpose of the graph
2 kB	cc-main-2020-jul-aug-sep-domain-t.properties

1 kB	cc-main-2020-jul-aug-sep-domain.stats	WebGraph statistics

1.91 GB	cc-main-2020-jul-aug-sep-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 89 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Jul/Aug/Sep 2020)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	32027928	1	0.018888	com.googleapis
2	30312944	3	0.012001	com.facebook
3	29025948	2	0.013237	com.google
4	26560472	4	0.007343	org.w
5	26516534	5	0.007172	com.twitter
6	26016464	6	0.006600	com.youtube
7	24614190	9	0.004795	com.instagram
8	24220712	8	0.005190	org.gmpg
9	23572970	7	0.005599	com.googletagmanager
10	23188190	11	0.003202	com.linkedin
11	22457894	15	0.002590	com.gravatar
12	22451350	10	0.003967	com.cloudflare
13	22364152	14	0.002726	com.gstatic
14	22350042	12	0.003105	org.wordpress
15	21926906	22	0.001505	com.pinterest
16	21699168	21	0.001752	com.wordpress
17	21599006	26	0.001181	org.wikipedia
18	21538264	16	0.002431	com.bootstrapcdn
19	21497526	18	0.001836	com.apple
20	21314410	30	0.001106	com.vimeo
21	21248994	41	0.000830	be.youtu
22	21186566	20	0.001794	com.jquery
23	21081822	23	0.001444	com.microsoft
24	21073240	45	0.000773	com.blogspot
25	20994964	39	0.000952	com.amazonaws
26	20975988	46	0.000732	gl.goo
27	20971574	25	0.001384	com.wp
28	20921220	47	0.000723	com.amazon
29	20788608	72	0.000439	com.tumblr
30	20716256	19	0.001804	com.adobe
31	20694562	67	0.000535	ly.bit
32	20675418	34	0.001018	com.google-analytics
33	20627694	53	0.000673	org.mozilla
34	20618998	17	0.001975	com.github
35	20617620	31	0.001059	net.cloudfront
36	20579928	71	0.000449	com.yahoo
37	20571130	29	0.001127	com.googlesyndication
38	20570586	60	0.000612	eu.europa
39	20562028	52	0.000679	com.flickr
40	20560188	42	0.000818	net.jsdelivr
41	20526264	97	0.000347	com.googleusercontent
42	20481758	62	0.000606	co.t
43	20480218	109	0.000313	com.reddit
44	20451670	24	0.001419	com.fontawesome
45	20436180	83	0.000389	com.weebly
46	20387228	56	0.000628	com.paypal
47	20375802	40	0.000910	com.macromedia
48	20372972	70	0.000450	com.medium
49	20370180	43	0.000808	com.addthis
50	20360678	28	0.001156	ru.yandex
51	20338498	27	0.001156	me.wp
52	20331252	64	0.000559	org.w3
53	20326560	79	0.000411	io.github
54	20292836	138	0.000223	com.nytimes
55	20275824	76	0.000414	org.creativecommons
56	20274244	59	0.000615	org.schema
57	20255326	150	0.000192	com.forbes
58	20246068	173	0.000151	com.imgur
59	20227930	36	0.000979	net.doubleclick
60	20219612	194	0.000133	uk.co.bbc
61	20210924	114	0.000285	com.soundcloud
62	20171070	66	0.000548	com.vk
63	20155222	195	0.000133	com.cnn
64	20142696	44	0.000803	org.apache
65	20134806	63	0.000587	com.whatsapp
66	20129582	314	0.000082	edu.mit
67	20123032	180	0.000146	com.imdb
68	20118310	208	0.000124	net.slideshare
69	20116626	243	0.000101	com.wsj
70	20115768	197	0.000128	org.wikimedia
71	20089462	85	0.000388	com.shopify
72	20082204	215	0.000120	edu.stanford
73	20076684	154	0.000181	gov.cdc
74	20075632	328	0.000079	com.wired
75	20069724	268	0.000094	com.techcrunch
76	20057066	255	0.000096	edu.harvard
77	20051336	353	0.000076	com.appspot
78	20051292	207	0.000124	net.sourceforge
79	20051264	257	0.000096	com.oracle
80	20051250	155	0.000177	int.who
81	20050888	206	0.000124	com.businessinsider
82	20046050	137	0.000227	org.archive
83	20038198	230	0.000113	com.washingtonpost
84	20035810	250	0.000097	com.live
85	20029940	164	0.000163	com.bing
86	20028210	549	0.000054	com.livejournal
87	20027622	424	0.000069	com.go
88	20024666	456	0.000066	com.msn
89	20019992	407	0.000072	uk.co.telegraph
90	20009306	170	0.000154	com.theguardian
91	20002514	527	0.000056	edu.cornell
92	19997146	199	0.000128	org.ietf
93	19996714	486	0.000063	gov.nasa
94	19995476	259	0.000096	com.android
95	19986252	302	0.000084	com.reuters
96	19983946	51	0.000702	net.fbcdn
97	19974890	240	0.000102	com.bloomberg
98	19966464	162	0.000164	com.giphy
99	19960428	77	0.000414	com.list-manage
100	19959046	520	0.000057	com.googleblog
101	19956558	269	0.000093	com.bbc
102	19955204	409	0.000071	com.slack
103	19942056	143	0.000205	com.spotify
104	19938828	591	0.000049	com.zdnet
105	19936894	48	0.000721	net.facebook
106	19935010	586	0.000050	com.quora
107	19931072	126	0.000265	com.ytimg
108	19922774	444	0.000067	com.myspace
109	19922046	757	0.000038	edu.umich
110	19920178	715	0.000040	edu.upenn
111	19917482	151	0.000185	gov.nih
112	19907886	344	0.000077	com.usatoday
113	19903896	654	0.000045	com.economist
114	19903722	313	0.000082	com.cnbc
115	19902700	308	0.000083	com.example
116	19896552	525	0.000056	com.pixabay
117	19895014	418	0.000070	net.researchgate
118	19882790	449	0.000066	com.latimes
119	19881164	188	0.000138	com.blogger
120	19870046	387	0.000075	org.python
121	19864804	65	0.000555	com.wix
122	19860760	433	0.000068	com.githubusercontent
123	19858732	693	0.000042	org.ieee
124	19854254	499	0.000061	com.mashable
125	19850918	571	0.000052	edu.berkeley
126	19847554	135	0.000241	com.youtube-nocookie
127	19845130	160	0.000167	com.issuu
128	19843068	218	0.000118	org.acm
129	19839736	834	0.000036	org.chromium
130	19839550	235	0.000106	uk.co.google
131	19835790	551	0.000054	org.arxiv
132	19833020	246	0.000099	net.behance
133	19832682	291	0.000086	org.npr
134	19831994	108	0.000320	com.unpkg
135	19831136	884	0.000034	com.arstechnica
136	19826840	213	0.000121	com.unsplash
137	19822884	341	0.000078	com.outlook
138	19822670	110	0.000303	de.google
139	19812430	54	0.000654	com.googleadservices
140	19810872	347	0.000077	com.prnewswire
141	19806458	678	0.000043	edu.columbia
142	19805382	171	0.000153	me.t
143	19804886	297	0.000085	com.dribbble
144	19804142	256	0.000096	com.squarespace
145	19799032	139	0.000215	gov.privacyshield
146	19798806	306	0.000083	com.huffingtonpost
147	19797964	260	0.000096	com.bandcamp
148	19795112	398	0.000074	com.time
149	19793874	37	0.000975	com.baidu
150	19792082	616	0.000048	com.gitlab
151	19790406	334	0.000079	com.nationalgeographic
152	19788214	443	0.000067	com.nature
153	19785178	794	0.000037	com.stackexchange
154	19782114	179	0.000147	gle.forms
155	19781676	258	0.000096	org.ampproject
156	19778534	548	0.000054	com.fortune
157	19777902	813	0.000036	com.git-scm
158	19776608	33	0.001030	com.wixstatic
159	19774030	771	0.000038	com.qz
160	19772390	281	0.000089	com.wiley
161	19772268	646	0.000046	au.net.abc
162	19770930	638	0.000046	edu.yale
163	19769582	428	0.000068	com.meetup
164	19767876	468	0.000064	com.ted
165	19761386	1160	0.000026	com.hatenablog
166	19759052	448	0.000066	com.patreon
167	19757472	283	0.000089	com.disqus
168	19756748	936	0.000032	edu.ucla
169	19753998	147	0.000195	com.dropbox
170	19753380	168	0.000158	com.yelp
171	19750678	271	0.000093	org.un
172	19746384	212	0.000122	com.twimg
173	19743118	254	0.000096	org.drupal
174	19741474	689	0.000042	org.bitbucket
175	19736540	422	0.000069	com.statista
176	19735440	903	0.000033	uk.ac.cam
177	19731940	718	0.000040	com.evernote
178	19731916	682	0.000043	com.newyorker
179	19725638	603	0.000049	com.buzzfeed
180	19719544	606	0.000049	me.about
181	19718654	722	0.000040	com.mysql
182	19716804	850	0.000035	com.thenextweb
183	19715420	495	0.000061	com.theatlantic
184	19710920	279	0.000091	com.sciencedirect
185	19710826	403	0.000073	com.getpocket
186	19705326	669	0.000043	uk.co.blogspot
187	19702126	1293	0.000023	com.tinypic
188	19696730	450	0.000066	com.booking
189	19695652	514	0.000058	com.xinhuanet
190	19694904	743	0.000039	org.weforum
191	19694268	247	0.000098	gov.ca
192	19692322	602	0.000049	gov.loc
193	19690998	1282	0.000023	org.postgresql
194	19689908	828	0.000036	edu.princeton
195	19687954	239	0.000103	uk.co.amazon
196	19685942	480	0.000063	com.dailymotion
197	19679672	1452	0.000021	ru.narod
198	19678926	189	0.000138	com.xing
199	19675914	879	0.000034	edu.jhu
200	19673670	500	0.000060	gov.whitehouse
201	19671846	665	0.000044	org.worldbank
202	19668706	1365	0.000022	org.eclipse
203	19667770	400	0.000073	com.springer
204	19667684	445	0.000067	com.nypost
205	19665872	316	0.000081	com.ft
206	19660930	61	0.000606	com.fb
207	19658986	204	0.000125	com.feedburner
208	19658394	826	0.000036	org.cambridge
209	19654762	476	0.000063	uk.co.dailymail
210	19654386	766	0.000038	edu.washington
211	19654242	496	0.000061	org.eff
212	19653044	32	0.001054	com.qq
213	19650144	473	0.000064	com.goodreads
214	19649524	264	0.000095	org.doi
215	19649502	512	0.000058	com.w3schools
216	19641242	1311	0.000023	edu.virginia
217	19641212	440	0.000067	com.googlecode
218	19638348	633	0.000047	com.vice
219	19633128	506	0.000059	com.force
220	19632976	723	0.000040	com.trello
221	19632780	836	0.000035	com.about
222	19630562	523	0.000056	com.inc
223	19629482	453	0.000066	com.scribd
224	19629368	2053	0.000016	com.wikidot
225	19628436	619	0.000048	org.semver
226	19614496	607	0.000049	com.cbsnews
227	19607794	651	0.000045	com.withgoogle
228	19605512	146	0.000196	me.line
229	19603410	2089	0.000016	com.googlesource
230	19601476	219	0.000118	org.iana
231	19601452	546	0.000054	gov.usda
232	19599800	309	0.000083	com.tinyurl
233	19598290	1090	0.000027	com.techradar
234	19597674	858	0.000035	com.dropboxusercontent
235	19597446	384	0.000076	com.ibm
236	19595200	1284	0.000023	co.elastic
237	19594024	289	0.000087	com.squareup
238	19593336	1434	0.000021	org.linuxfoundation
239	19592388	1134	0.000026	org.coursera
240	19589830	1027	0.000029	gov.fbi
241	19588284	1158	0.000026	edu.unc
242	19586008	705	0.000041	com.vox
243	19583350	193	0.000134	de.amazon
244	19583096	550	0.000054	uk.co.independent
245	19580554	1423	0.000021	ms.1drv
246	19578950	383	0.000076	com.digg
247	19567612	1393	0.000022	org.kernel
248	19563948	113	0.000287	com.sharethis
249	19563468	751	0.000039	org.d3js
250	19557490	801	0.000037	gov.fcc
251	19557292	1026	0.000029	com.hollywoodreporter
252	19556258	1369	0.000022	com.howstuffworks
253	19553700	430	0.000068	com.cnet
254	19552068	804	0.000037	com.foxnews
255	19547134	152	0.000183	com.addtoany
256	19547006	644	0.000046	com.indiatimes
257	19546928	995	0.000029	com.steamcommunity
258	19546864	1105	0.000026	cn.com.chinadaily
259	19545628	584	0.000050	com.psychologytoday
260	19544130	823	0.000036	uk.co.guardian
261	19543920	1463	0.000021	it.scoop
262	19543754	133	0.000247	com.mailchimp
263	19542234	837	0.000035	com.slate
264	19542214	153	0.000182	com.opera
265	19538412	589	0.000050	com.mckinsey
266	19536816	1020	0.000029	com.sap
267	19536418	2605	0.000013	org.wikiquote
268	19534334	307	0.000083	com.bitly
269	19533308	627	0.000047	com.mozilla
270	19533054	262	0.000095	jp.ameblo
271	19531260	735	0.000039	org.sciencemag
272	19528246	116	0.000284	com.paypalobjects
273	19528108	2345	0.000014	org.wikibooks
274	19527104	176	0.000151	com.amazon-adsystem
275	19526948	688	0.000042	gov.noaa
276	19524868	305	0.000083	com.netdna-ssl
277	19524544	310	0.000083	com.nbcnews
278	19523330	989	0.000030	com.target
279	19522776	1523	0.000020	com.instructables
280	19517526	975	0.000030	edu.umn
281	19516530	965	0.000031	com.merriam-webster
282	19516260	1431	0.000021	hk.com.google
283	19514852	185	0.000140	com.tripadvisor
284	19514608	2377	0.000014	com.diigo
285	19503916	497	0.000061	ca.google
286	19499262	236	0.000106	com.wpengine
287	19499246	1029	0.000028	com.sun
288	19496562	1189	0.000025	com.digitaltrends
289	19496340	391	0.000075	com.stumbleupon
290	19491846	115	0.000284	com.weibo
291	19491638	1626	0.000019	com.ign
292	19491210	1314	0.000023	com.mercurynews
293	19490964	1352	0.000022	de.zeit
294	19490636	229	0.000114	com.etsy
295	19489106	797	0.000037	uk.ac.ox
296	19487454	284	0.000089	com.optimizely
297	19485106	73	0.000425	net.akamaihd
298	19484368	1207	0.000025	net.speedtest
299	19484284	1522	0.000020	org.greenpeace
300	19483622	1553	0.000020	net.seesaa
301	19479450	720	0.000040	au.com.google
302	19478604	904	0.000033	de.spiegel
303	19476336	1077	0.000027	com.podbean
304	19475142	628	0.000047	org.pbs
305	19474722	516	0.000058	com.gofundme
306	19474484	416	0.000070	com.kickstarter
307	19473590	1340	0.000022	com.urbandictionary
308	19472422	472	0.000064	org.pewresearch
309	19471320	519	0.000057	com.bigcommerce
310	19467912	2137	0.000015	de.bild
311	19467240	231	0.000112	com.eepurl
312	19465300	515	0.000058	com.theverge
313	19464792	273	0.000092	com.stackoverflow
314	19464598	926	0.000032	com.politico
315	19463036	811	0.000036	co.ibb
316	19462394	332	0.000079	it.google
317	19462162	2110	0.000016	ly.visual
318	19461840	955	0.000031	org.unicef
319	19460932	2020	0.000016	org.tensorflow
320	19457592	1688	0.000018	com.itv
321	19457150	1013	0.000029	com.lifehacker
322	19456512	106	0.000334	com.stripe
323	19456272	1349	0.000022	edu.msu
324	19455412	312	0.000083	net.windows
325	19453374	805	0.000037	edu.academia
326	19450284	1391	0.000022	com.storify
327	19449638	1257	0.000024	com.crunchbase
328	19449386	595	0.000049	com.tandfonline
329	19449132	1958	0.000017	com.lego
330	19444682	1187	0.000025	com.jetbrains
331	19443796	677	0.000043	gov.senate
332	19443664	855	0.000035	com.chicagotribune
333	19443234	2301	0.000014	com.rottentomatoes
334	19440224	770	0.000038	ca.cbc
335	19439934	205	0.000125	com.eventbrite
336	19439496	1273	0.000023	hk.hku
337	19436402	1035	0.000028	edu.wisc
338	19436104	691	0.000042	com.libsyn
339	19435742	1051	0.000028	edu.northwestern
340	19433212	944	0.000031	com.scientificamerican
341	19432798	1043	0.000028	edu.uchicago
342	19431182	1288	0.000023	uk.co.wired
343	19425546	190	0.000137	jp.co.google
344	19424346	2002	0.000016	org.maven
345	19423732	1030	0.000028	com.mediafire
346	19423350	415	0.000070	me.telegram
347	19418440	396	0.000074	com.criteo
348	19417208	357	0.000076	fr.google
349	19417038	664	0.000044	us.icio
350	19416402	1477	0.000020	com.deadline
351	19415808	640	0.000046	com.sagepub
352	19414256	730	0.000039	com.ecwid
353	19413466	1275	0.000023	org.aclu
354	19413258	576	0.000051	com.typepad
355	19412168	471	0.000064	com.photobucket
356	19407294	533	0.000055	com.oup
357	19407168	1199	0.000025	com.reverbnation
358	19406968	1514	0.000020	de.mpg
359	19405330	1389	0.000022	edu.rutgers
360	19404790	1067	0.000027	com.scmp
361	19403976	81	0.000392	net.jsfiddle
362	19403692	421	0.000069	com.calendly
363	19403618	844	0.000035	com.sciencedaily
364	19403468	727	0.000039	gov.justice
365	19400830	575	0.000051	gov.hhs
366	19398258	919	0.000032	com.theconversation
367	19397596	991	0.000030	com.apnews
368	19397442	938	0.000032	com.huffpost
369	19394934	1518	0.000020	com.newscientist
370	19394656	608	0.000049	org.openstreetmap
371	19393300	1287	0.000023	com.aljazeera
372	19393230	216	0.000119	com.hubspot
373	19390018	645	0.000046	gov.house
374	19388118	2682	0.000012	uk.co.timesonline
375	19388034	2564	0.000013	com.space
376	19383910	700	0.000041	com.pinimg
377	19383504	432	0.000068	page.g
378	19381990	1241	0.000024	com.sky
379	19381844	866	0.000035	gov.congress
380	19381026	912	0.000033	com.500px
381	19380632	1217	0.000024	org.wiktionary
382	19380340	958	0.000031	com.ssrn
383	19379742	1709	0.000018	edu.bu
384	19377640	1757	0.000018	gov.cia
385	19375740	214	0.000120	org.bbb
386	19375634	1438	0.000021	com.foxbusiness
387	19371814	624	0.000047	ru.gov
388	19371056	1598	0.000019	ca.mcgill
389	19367926	790	0.000037	com.qualtrics
390	19366054	1290	0.000023	org.semanticscholar
391	19365778	761	0.000038	site.business
392	19365760	267	0.000094	ru.ok
393	19363798	977	0.000030	edu.si
394	19363758	887	0.000034	br.com.google
395	19363688	847	0.000035	co.g
396	19363204	1021	0.000029	uk.co.thetimes
397	19362122	2663	0.000012	com.discovermagazine
398	19359920	182	0.000142	us.zoom
399	19359492	889	0.000034	org.fao
400	19359352	683	0.000043	org.change
401	19357866	1469	0.000020	com.salon
402	19356650	228	0.000114	com.aliyuncs
403	19356280	997	0.000029	com.thehill
404	19354818	973	0.000030	gov.usgs
405	19351584	298	0.000085	com.ebay
406	19350988	1222	0.000024	com.nikkei
407	19350142	338	0.000078	com.rawgit
408	19349660	578	0.000051	it.placehold
409	19348824	157	0.000173	com.wixsite
410	19348122	1238	0.000024	com.smithsonianmag
411	19346552	758	0.000038	org.oecd
412	19346514	1088	0.000027	ee.linktr
413	19345254	3312	0.000011	com.openai
414	19342288	1048	0.000028	uk.co.mirror
415	19341656	679	0.000043	com.deviantart
416	19341332	1576	0.000019	org.phys
417	19340598	413	0.000070	tv.twitch
418	19340138	404	0.000072	com.mapbox
419	19335246	1546	0.000020	ca.sfu
420	19332464	2754	0.000012	com.instapaper
421	19330656	244	0.000100	org.gnu
422	19330504	2115	0.000016	au.edu.unimelb
423	19328724	1044	0.000028	int.coe
424	19328320	2078	0.000016	org.nobelprize
425	19328286	667	0.000043	pl.google
426	19327680	1333	0.000022	com.irishtimes
427	19327578	293	0.000086	com.office
428	19327536	1962	0.000017	org.torproject
429	19324936	484	0.000063	net.imgix
430	19324628	1281	0.000023	uk.ac.ucl
431	19320926	1054	0.000028	org.ohchr
432	19318772	1213	0.000025	com.strikingly
433	19315502	509	0.000059	org.hbr
434	19315040	1411	0.000021	uk.co.metro
435	19314304	123	0.000270	com.statcounter
436	19313468	972	0.000030	gov.dhs
437	19313380	287	0.000088	com.thedailybeast
438	19313234	1811	0.000017	com.bankofamerica
439	19312534	1265	0.000024	com.buzzsprout
440	19311940	863	0.000035	gov.nps
441	19309868	2426	0.000014	au.com.theage
442	19307472	933	0.000032	com.aweber
443	19306766	1557	0.000020	blog.home
444	19305448	848	0.000035	gov.bls
445	19305296	490	0.000062	edu.nyu
446	19304346	2087	0.000016	com.oxforddictionaries
447	19304074	1162	0.000025	gov.nyc
448	19303568	93	0.000356	org.reactjs
449	19302778	1382	0.000022	au.com.news
450	19300882	2291	0.000014	sg.edu.nus
451	19299900	1429	0.000021	com.flipboard
452	19299896	481	0.000063	com.scorecardresearch
453	19298010	2517	0.000013	com.dummies
454	19295840	2465	0.000013	org.rsc
455	19295472	1010	0.000029	com.britannica
456	19294984	714	0.000040	gov.state
457	19294216	1700	0.000018	org.gutenberg
458	19292892	3565	0.000010	fm.ask
459	19290866	2970	0.000011	com.pearltrees
460	19289990	793	0.000037	com.zapier
461	19286494	2562	0.000013	com.mystrikingly
462	19284092	876	0.000034	com.cctv
463	19283500	816	0.000036	com.healthline
464	19283044	1955	0.000017	com.chrome
465	19282638	1484	0.000020	com.rt
466	19282550	967	0.000031	com.newsweek
467	19280538	2362	0.000014	com.biography
468	19279646	1005	0.000029	ch.google
469	19270504	1412	0.000021	com.ifttt
470	19270238	1584	0.000019	com.axios
471	19270042	466	0.000065	es.google
472	19269658	882	0.000034	au.gov.nsw
473	19267444	3483	0.000010	hk.edu.cuhk
474	19267150	862	0.000035	com.stitcher
475	19267000	2520	0.000013	com.boredpanda
476	19265582	1192	0.000025	fr.lemonde
477	19263992	554	0.000053	com.steampowered
478	19263878	1055	0.000028	org.jstor
479	19262150	1335	0.000022	org.imf
480	19261918	873	0.000034	com.venturebeat
481	19261196	825	0.000036	org.poynter
482	19259574	1684	0.000018	com.straitstimes
483	19259452	3390	0.000010	com.chosun
484	19259322	1502	0.000020	edu.asu
485	19258762	2351	0.000014	io.gitlab
486	19256810	956	0.000031	ru.google
487	19255996	952	0.000031	sg.com.google
488	19253798	1331	0.000022	uk.co.standard
489	19252906	612	0.000048	de.gesetze-im-internet
490	19251516	948	0.000031	gov.archives
491	19250270	2385	0.000014	th.co.google
492	19249730	423	0.000069	io.codepen
493	19248930	3033	0.000011	com.nola
494	19248894	2023	0.000016	edu.gmu
495	19245246	2836	0.000012	app.netlify
496	19245158	1116	0.000026	com.wikia
497	19242656	1353	0.000022	com.history
498	19242160	1007	0.000029	com.thelancet
499	19241830	2918	0.000011	com.coca-colacompany
500	19240640	2654	0.000012	google.ai
501	19240600	856	0.000035	com.freepik
502	19240430	1548	0.000020	com.buzzfeednews
503	19238648	2894	0.000012	org.cato
504	19237700	431	0.000068	net.datatables
505	19237456	501	0.000060	com.rackcdn
506	19236168	1590	0.000019	gov.supremecourt
507	19233302	2534	0.000013	edu.byu
508	19233268	642	0.000046	fr.amazon
509	19232920	2872	0.000012	tw.blogspot
510	19231944	803	0.000037	in.co.google
511	19231530	1977	0.000017	org.edx
512	19231228	1309	0.000023	com.tunein
513	19231156	1779	0.000018	org.ocks
514	19230478	522	0.000057	nl.google
515	19228370	555	0.000053	com.gmail
516	19227068	2398	0.000014	com.nationalpost
517	19226910	1867	0.000017	edu.ucsb
518	19226418	2383	0.000014	edu.nd
519	19226392	1372	0.000022	com.dw
520	19226256	127	0.000262	com.jimdo
521	19225860	2412	0.000014	no.uio
522	19225400	1006	0.000029	google.blog
523	19222398	1409	0.000021	cn.cntv
524	19222164	3285	0.000011	cn.org.china
525	19221136	1639	0.000019	org.unwomen
526	19218950	946	0.000031	com.airtable
527	19217788	2510	0.000013	edu.uoregon
528	19215376	2172	0.000015	org.britishcouncil
529	19214674	2668	0.000012	org.icrc
530	19214462	951	0.000031	com.gallup
531	19213378	2265	0.000015	ru.kremlin
532	19212894	1332	0.000022	com.globalsign
533	19210850	875	0.000034	gov.uspto
534	19210492	959	0.000031	edu.psu
535	19210022	1509	0.000020	com.penguinrandomhouse
536	19209318	1345	0.000022	com.netdna-cdn
537	19208686	3269	0.000011	is.archive
538	19208344	1531	0.000020	uk.ac.lse
539	19207952	2503	0.000013	fi.helsinki
540	19207620	2042	0.000016	edu.pitt
541	19207236	2170	0.000015	net.openid
542	19206256	1155	0.000026	edu.brookings
543	19205290	786	0.000037	com.imageshack
544	19204770	172	0.000152	com.npmjs
545	19204486	3290	0.000011	de.diplo
546	19204380	1956	0.000017	edu.unl
547	19203832	1544	0.000020	edu.georgetown
548	19203210	2125	0.000015	org.metmuseum
549	19202750	1240	0.000024	org.nejm
550	19202244	726	0.000040	com.adage
551	19200434	1990	0.000017	com.channel4
552	19200290	1511	0.000020	com.findlaw
553	19200030	2224	0.000015	com.france24
554	19198938	282	0.000089	net.php
555	19198698	1784	0.000017	com.csmonitor
556	19197866	419	0.000069	com.proofpoint
557	19195320	192	0.000135	com.iubenda
558	19194372	1011	0.000029	gov.treasury
559	19194028	1708	0.000018	com.euronews
560	19191446	2286	0.000014	com.thoughtco
561	19190136	3742	0.000009	com.doodlekit
562	19189862	107	0.000320	com.godaddy
563	19189334	1298	0.000023	edu.duke
564	19188652	2071	0.000016	com.foreignpolicy
565	19185118	1996	0.000017	org.documentcloud
566	19183756	1300	0.000023	com.livescience
567	19183706	2508	0.000013	com.upi
568	19183104	2085	0.000016	com.gq
569	19182260	178	0.000148	com.zendesk
570	19182074	3020	0.000011	com.authorstream
571	19182074	3915	0.000009	com.mysanantonio
572	19181694	4133	0.000008	tw.edu.sinica
573	19177894	2719	0.000012	org.wikisource
574	19177382	2220	0.000015	com.insider
575	19177180	851	0.000035	gov.nist
576	19177000	1625	0.000019	com.thestar
577	19176642	181	0.000145	jp.co.yahoo
578	19174546	1304	0.000023	au.com.smh
579	19174028	2025	0.000016	org.ncsl
580	19173800	4252	0.000008	hk.edu.cityu
581	19173744	3349	0.000010	com.sina
582	19173108	2197	0.000015	ie.independent
583	19172266	2156	0.000015	edu.uky
584	19171704	96	0.000349	me.ogp
585	19170936	3413	0.000010	uk.ac.sussex
586	19170792	1755	0.000018	gov.doc
587	19170704	131	0.000250	org.networkadvertising
588	19169566	320	0.000080	io.shields
589	19168058	649	0.000045	gov.usa
590	19166990	4291	0.000008	org.china-embassy
591	19166810	3137	0.000011	com.udn
592	19163774	161	0.000166	ru.mail
593	19163712	3474	0.000010	com.worldatlas
594	19163522	505	0.000060	com.netflix
595	19163254	857	0.000035	com.thinkwithgoogle
596	19162356	1441	0.000021	gov.defense
597	19161952	1318	0.000023	tw.com.google
598	19160826	1604	0.000019	org.hrw
599	19159812	1495	0.000020	com.asahi
600	19159570	785	0.000037	io.readthedocs
601	19158768	2688	0.000012	org.freedomhouse
602	19158654	1413	0.000021	tv.ustream
603	19157822	893	0.000034	org.mediawiki
604	19156446	1715	0.000018	org.pypi
605	19151800	3028	0.000011	org.adb
606	19151406	2099	0.000016	fr.leparisien
607	19151152	2615	0.000013	com.abc7news
608	19150650	2063	0.000016	com.voanews
609	19150048	1019	0.000029	com.pcmag
610	19148698	447	0.000067	org.nodejs
611	19148554	4288	0.000008	com.theundefeated
612	19147816	3860	0.000009	org.gephi
613	19147176	1327	0.000023	org.undp
614	19146462	3277	0.000011	org.iucnredlist
615	19146454	2583	0.000013	com.sacbee
616	19146204	1594	0.000019	com.treehugger
617	19145608	2292	0.000014	no.google
618	19144462	2471	0.000013	co.ello
619	19143354	1986	0.000017	com.msnbc
620	19143354	252	0.000097	com.myshopify
621	19142810	981	0.000030	uk.parliament
622	19142520	2287	0.000014	co.pcdn
623	19141942	1255	0.000024	gov.uscourts
624	19141896	1422	0.000021	co.lpages
625	19140780	2344	0.000014	org.fas
626	19139768	781	0.000037	com.intel
627	19138740	807	0.000036	com.marketwatch
628	19136914	2047	0.000016	com.infogram
629	19133848	2538	0.000013	com.sputniknews
630	19133704	2430	0.000014	ie.google
631	19132582	1344	0.000022	se.google
632	19131798	990	0.000030	com.netlify
633	19131000	925	0.000032	com.jekyllrb
634	19130612	3055	0.000011	int.interpol
635	19130308	524	0.000056	fr.free
636	19130180	1198	0.000025	be.google
637	19129750	1575	0.000019	uk.co.huffingtonpost
638	19129310	2323	0.000014	ly.rebrand
639	19129104	1504	0.000020	link.page
640	19128704	1794	0.000017	com.sched
641	19127724	2218	0.000015	jp.co.japantimes
642	19127254	2829	0.000012	org.tigris
643	19127152	2839	0.000012	org.pri
644	19127006	2319	0.000014	nz.co.nzherald
645	19125622	1204	0.000025	at.google
646	19125464	5292	0.000007	org.arkive
647	19125326	222	0.000116	com.salesforce
648	19123296	650	0.000045	br.com.uol
649	19121018	4242	0.000008	kr.co.kbs
650	19119374	1665	0.000018	com.thebalance
651	19119126	1455	0.000021	org.oxfordjournals
652	19118638	3738	0.000009	com.encyclopedia
653	19117262	2204	0.000015	org.eji
654	19116506	2818	0.000012	org.heritage
655	19116298	2371	0.000014	com.popsci
656	19114518	2199	0.000015	com.snopes
657	19114098	2601	0.000013	org.oas
658	19113348	156	0.000174	com.aspnetcdn
659	19112712	1031	0.000028	org.ilo
660	19109654	2263	0.000015	com.insidehighered
661	19108980	1587	0.000019	gov.usembassy
662	19108932	1622	0.000019	dk.google
663	19108040	3392	0.000010	org.jenkins-ci
664	19107388	2827	0.000012	org.project-syndicate
665	19106556	1963	0.000017	com.justia
666	19104120	1563	0.000019	gov.govinfo
667	19103152	1699	0.000018	com.firebaseapp
668	19102068	2093	0.000016	edu.uga
669	19102028	3678	0.000010	edu.wm
670	19101614	3284	0.000011	com.cgtn
671	19101596	1881	0.000017	org.worldcat
672	19101226	900	0.000033	com.zoho
673	19100590	392	0.000074	com.atlassian
674	19100290	2676	0.000012	org.transparency
675	19099776	1317	0.000023	org.aarp
676	19099686	1675	0.000018	org.americanbar
677	19099164	2239	0.000015	com.timeshighereducation
678	19097964	3270	0.000011	com.pastemagazine
679	19095902	2598	0.000013	org.csis
680	19094342	629	0.000047	com.samsung
681	19094058	774	0.000038	com.pexels
682	19093374	1964	0.000017	com.washingtontimes
683	19092714	2016	0.000016	gov.usaid
684	19090166	1334	0.000022	org.heart
685	19088764	191	0.000136	com.automattic
686	19088428	865	0.000035	com.verisign
687	19087660	2108	0.000016	com.motherjones
688	19087034	2944	0.000011	org.vim
689	19086498	2062	0.000016	edu.nap
690	19086172	924	0.000032	com.webs
691	19084778	1593	0.000019	org.amnesty
692	19084344	2101	0.000016	ua.com.google
693	19083552	3988	0.000009	org.globalnetworkinitiative
694	19083196	2546	0.000013	org.globalcitizen
695	19082500	1754	0.000018	com.surveygizmo
696	19082058	2262	0.000015	org.wbur
697	19081048	2353	0.000014	uk.gov.companieshouse
698	19080398	2468	0.000013	jp.mainichi
699	19080286	3181	0.000011	com.podomatic
700	19078116	1751	0.000018	org.unhcr
701	19076276	2118	0.000016	ca.ctvnews
702	19075310	2565	0.000013	uk.co.bbci
703	19073812	968	0.000031	uk.gov.legislation
704	19071522	2681	0.000012	com.nationalreview
705	19070832	2523	0.000013	com.cleveland
706	19070474	3814	0.000009	org.neocities
707	19069884	1073	0.000027	ly.snip
708	19068864	438	0.000067	com.herokuapp
709	19068510	656	0.000045	com.oreilly
710	19066730	1154	0.000026	cz.google
711	19066464	2164	0.000015	org.nrdc
712	19065768	2671	0.000012	org.thinkprogress
713	19065654	1795	0.000017	ca.globalnews
714	19065106	270	0.000093	jp.co.amazon
715	19062840	1328	0.000023	org.altervista
716	19061732	3119	0.000011	uk.ac.nottingham
717	19061168	1267	0.000024	uk.gov.nationalarchives
718	19060934	2106	0.000016	au.edu.anu
719	19060236	3035	0.000011	com.intensedebate
720	19060102	2734	0.000012	de.hu-berlin
721	19059802	736	0.000039	com.airbnb
722	19059800	2326	0.000014	de.auswaertiges-amt
723	19059376	2316	0.000014	nz.co.google
724	19059170	2672	0.000012	org.unenvironment
725	19058978	3132	0.000011	org.rsf
726	19057932	4110	0.000008	com.koreaherald
727	19057778	1960	0.000017	org.pewtrusts
728	19057678	2867	0.000012	com.techinasia
729	19057488	2276	0.000014	com.thecut
730	19056174	3700	0.000009	com.viki
731	19056068	2724	0.000012	org.gnupg
732	19054590	2469	0.000013	ro.google
733	19054394	2057	0.000016	edu.gwu
734	19054116	3057	0.000011	com.bangkokpost
735	19053626	2572	0.000013	fr.rfi
736	19052868	414	0.000070	com.pubmatic
737	19051906	2309	0.000014	com.tutsplus
738	19051648	1079	0.000027	tr.com.google
739	19051516	248	0.000098	com.getbootstrap
740	19050908	4424	0.000008	com.wonderhowto
741	19050626	3619	0.000010	com.upworthy
742	19050496	2883	0.000012	org.sonatype
743	19050382	288	0.000087	com.typeform
744	19049574	2806	0.000012	il.co.google
745	19049384	2739	0.000012	uk.ac.leeds
746	19048116	201	0.000127	to.amzn
747	19047986	2703	0.000012	vn.com.google
748	19047578	274	0.000092	com.surveymonkey
749	19047380	922	0.000032	int.wipo
750	19046288	1057	0.000028	com.gizmodo
751	19046144	874	0.000034	com.box
752	19045578	2298	0.000014	com.oregonlive
753	19044916	547	0.000054	gg.discord
754	19044444	3356	0.000010	com.theepochtimes
755	19044400	2480	0.000013	ar.com.google
756	19044144	2943	0.000011	bg.google
757	19043632	2061	0.000016	com.squarespace-cdn
758	19043400	3479	0.000010	io.soup
759	19042778	2545	0.000013	com.webbyawards
760	19042384	2744	0.000012	io.fabric
761	19042298	1588	0.000019	com.speakerdeck
762	19041684	136	0.000232	info.aboutads
763	19040606	907	0.000033	com.docker
764	19038814	1817	0.000017	com.miamiherald
765	19037924	3191	0.000011	ph.com.google
766	19037762	2463	0.000013	com.channelnewsasia
767	19037556	3198	0.000011	uk.co.vogue
768	19037554	2619	0.000013	edu.fsu
769	19035870	485	0.000063	com.staticflickr
770	19035284	2495	0.000013	za.co.google
771	19033678	2696	0.000012	com.thejakartapost
772	19032442	1236	0.000024	edu.ucsd
773	19032258	487	0.000062	com.fc2
774	19032038	5415	0.000007	com.armorgames
775	19031944	2155	0.000015	fi.google
776	19031234	3885	0.000009	com.alamy
777	19030868	2221	0.000015	id.co.google
778	19030462	2794	0.000012	com.rd
779	19029712	2951	0.000011	com.cartodb
780	19029584	2092	0.000016	com.newrepublic
781	19029348	3436	0.000010	com.benzinga
782	19028364	661	0.000044	com.entrepreneur
783	19027960	5376	0.000007	org.gwtproject
784	19026660	2988	0.000011	com.sciencealert
785	19026538	2763	0.000012	org.iaea
786	19026402	2376	0.000014	com.thenation
787	19023692	3411	0.000010	si.google
788	19023046	2400	0.000014	pt.google
789	19020124	2965	0.000011	au.gov.nla
790	19019838	3513	0.000010	com.dailykos
791	19019756	494	0.000061	com.aol
792	19019128	2519	0.000013	edu.emory
793	19019012	3573	0.000010	com.inhabitat
794	19018956	3415	0.000010	uk.ac.soas
795	19018402	666	0.000044	com.deloitte
796	19018230	1185	0.000025	com.today
797	19016838	978	0.000030	com.windowsphone
798	19016186	3659	0.000010	org.cpj
799	19016164	2119	0.000016	kr.co.google
800	19015906	2981	0.000011	se.lu
801	19015780	2774	0.000012	org.cfr
802	19014856	429	0.000068	me.fb
803	19013678	3288	0.000011	com.joins
804	19012980	4264	0.000008	sa.com.google
805	19012878	2814	0.000012	com.politifact
806	19012292	964	0.000031	com.alexa
807	19011442	4131	0.000008	edu.utm
808	19011068	2735	0.000012	com.law360
809	19010546	983	0.000030	com.engadget
810	19008662	3583	0.000010	hr.google
811	19008538	2146	0.000015	hu.google
812	19006860	631	0.000047	fm.last
813	19006540	2476	0.000013	eu.politico
814	19006248	4047	0.000009	com.chinatimes
815	19006116	2521	0.000013	mx.com.google
816	19006060	3141	0.000011	com.jezebel
817	19005942	3868	0.000009	com.iconarchive
818	19005318	3471	0.000010	com.ogilvy
819	19004866	2399	0.000014	gr.google
820	19004086	2816	0.000012	com.monday
821	19003252	2738	0.000012	com.digitaljournal
822	19003248	3149	0.000011	com.nyt
823	19003220	3300	0.000011	audio.breaker
824	19002640	2823	0.000012	uk.co.guim
825	19002384	625	0.000047	com.cisco
826	19002038	3391	0.000010	cn.globaltimes
827	19001808	2648	0.000012	com.instructure
828	19000646	3321	0.000011	com.crashlytics
829	18999720	2723	0.000012	au.com.businessinsider
830	18999338	3430	0.000010	org.grist
831	18998280	1209	0.000025	com.pastebin
832	18998118	315	0.000082	ai.shortpixel
833	18998078	3990	0.000009	org.constitutioncenter
834	18997960	4842	0.000007	jp.hatenadiary
835	18996780	3770	0.000009	edu.ttu
836	18996076	2997	0.000011	uk.ac.york
837	18995936	1671	0.000018	com.eater
838	18995084	90	0.000364	com.livestream
839	18995036	2772	0.000012	com.bepress
840	18994752	2898	0.000012	org.wri
841	18992262	2043	0.000016	my.com.thestar
842	18991122	3775	0.000009	com.minds
843	18990592	2352	0.000014	mp.j
844	18990570	3708	0.000009	app.web
845	18990062	3410	0.000010	org.carnegieendowment
846	18989786	3645	0.000010	tr.com.aa
847	18989418	711	0.000041	gov.sec
848	18987746	3812	0.000009	com.hyperallergic
849	18987282	3408	0.000010	com.foreignaffairs
850	18986640	3797	0.000009	au.edu.uts
851	18985392	470	0.000064	com.fastcompany
852	18985032	3560	0.000010	org.hypotheses
853	18984468	3896	0.000009	com.japantoday
854	18982752	3507	0.000010	edu.wayne
855	18982048	3713	0.000009	uk.ac.kent
856	18981988	3697	0.000009	rs.google
857	18980532	4071	0.000009	org.sourcewatch
858	18979366	832	0.000036	com.symantec
859	18978424	2539	0.000013	fr.paris
860	18977996	2942	0.000011	com.prweek
861	18977902	1765	0.000018	ch.ipcc
862	18976960	2217	0.000015	com.kinstacdn
863	18976262	1046	0.000028	edu.cmu
864	18975462	2039	0.000016	int.unfccc
865	18975062	4196	0.000008	eg.com.google
866	18974804	3180	0.000011	org.nationalgeographic
867	18974548	2643	0.000013	gov.doi
868	18973940	3406	0.000010	de.uni-frankfurt
869	18973494	4243	0.000008	by.google
870	18972022	5050	0.000007	com.symbaloo
871	18971010	3417	0.000010	nl.wur
872	18969950	2328	0.000014	org.unodc
873	18968430	1599	0.000019	com.routledge
874	18968412	4509	0.000008	com.ipsos-mori
875	18966962	3658	0.000010	ae.google
876	18966152	4482	0.000008	com.etymonline
877	18965888	4982	0.000007	build.bazel
878	18965566	3320	0.000011	org.brainpickings
879	18964544	3143	0.000011	com.scotsman
880	18963796	4295	0.000008	com.oilprice
881	18963380	3597	0.000010	uk.ac.westminster
882	18963266	4545	0.000008	lk.google
883	18962576	1260	0.000024	fr.blogspot
884	18961360	3412	0.000010	org.rferl
885	18961310	3173	0.000011	org.epi
886	18959900	4115	0.000008	lv.google
887	18959812	3909	0.000009	au.edu.griffith
888	18959422	4219	0.000008	kr.ac.snu
889	18957280	1312	0.000023	com.upwork
890	18957076	2436	0.000014	com.html5rocks
891	18956714	5493	0.000007	me.nimbusweb
892	18956502	2940	0.000011	fr.archives-ouvertes
893	18956398	4293	0.000008	com.delawareonline
894	18955462	1792	0.000017	ru.rbc
895	18954968	745	0.000039	com.gartner
896	18954930	1127	0.000026	edu.utexas
897	18953642	2526	0.000013	net.noscript
898	18953466	2717	0.000012	ae.thenational
899	18953336	3380	0.000010	com.study
900	18953092	427	0.000068	com.hp
901	18953074	3641	0.000010	uk.co.spectator
902	18952762	3869	0.000009	com.cleantechnica
903	18952208	2803	0.000012	org.unctad
904	18951200	4255	0.000008	com.teslamotors
905	18950118	1614	0.000019	com.billboard
906	18949366	3074	0.000011	com.theculturetrip
907	18947896	2454	0.000013	com.multiscreensite
908	18947738	704	0.000041	com.visualstudio
909	18947588	3985	0.000009	uk.ac.plymouth
910	18947454	2660	0.000012	sk.google
911	18947312	3811	0.000009	net.aljazeera
912	18947110	2413	0.000014	com.theintercept
913	18946556	3421	0.000010	uk.ac.exeter
914	18946494	3332	0.000010	social.mastodon
915	18945876	2828	0.000012	com.euractiv
916	18945864	3635	0.000010	com.db
917	18942736	4447	0.000008	org.mises
918	18942316	4680	0.000008	ng.com.google
919	18942016	2795	0.000012	org.panda
920	18941622	2466	0.000013	uk.gov.justice
921	18941430	5602	0.000007	net.chinadialogue
922	18940924	4118	0.000008	cat.uab
923	18940746	4227	0.000008	com.spokesman
924	18940082	3523	0.000010	co.com.google
925	18939230	4473	0.000008	lu.google
926	18938996	4189	0.000008	pe.com.google
927	18938618	3366	0.000010	com.nybooks
928	18938606	4381	0.000008	uk.ac.core
929	18938206	2228	0.000015	com.termsfeed
930	18938194	1669	0.000018	com.pcworld
931	18938112	3846	0.000009	kr.co.yna
932	18938002	4793	0.000007	com.gust
933	18937788	3880	0.000009	org.cgiar
934	18937300	4231	0.000008	pk.com.google
935	18936530	3575	0.000010	net.inquirer
936	18936008	3083	0.000011	ru.lenta
937	18934000	1468	0.000020	com.nokia
938	18933676	2932	0.000011	tw.com.pchome
939	18933496	1223	0.000024	com.ycombinator
940	18933350	2911	0.000011	nl.volkskrant
941	18933194	78	0.000411	com.oculus
942	18932612	3455	0.000010	cl.google
943	18931862	3949	0.000009	org.polymer-project
944	18930888	2637	0.000013	com.washingtonexaminer
945	18930622	3945	0.000009	sk.sme
946	18930534	3389	0.000010	edu.monash
947	18930086	918	0.000032	com.canva
948	18929552	454	0.000066	org.opensource
949	18929398	3977	0.000009	com.rappler
950	18928630	4000	0.000009	org.plan-international
951	18926518	4561	0.000008	cr.co.google
952	18926412	3587	0.000010	lt.google
953	18925832	3810	0.000009	ca.macleans
954	18925646	817	0.000036	net.adform
955	18925046	4873	0.000007	com.blogto
956	18924952	3508	0.000010	uk.ac.nhm
957	18924928	3211	0.000011	edu.ua
958	18923554	2815	0.000012	com.articulate
959	18923288	249	0.000098	com.sxsw
960	18922866	3993	0.000009	org.wilsoncenter
961	18922676	4082	0.000009	edu.lehigh
962	18922336	417	0.000070	com.skype
963	18921546	4699	0.000008	com.out
964	18920714	1085	0.000027	com.redhat
965	18920680	3266	0.000011	my.com.google
966	18919064	2031	0.000016	gov.ecfr
967	18918900	4585	0.000008	org.nsidc
968	18918778	412	0.000070	net.secureservercdn
969	18918112	4536	0.000008	kz.google
970	18917590	3295	0.000011	org.osce
971	18917562	557	0.000053	org.whatwg
972	18917418	4096	0.000009	com.wsoctv
973	18917380	2587	0.000013	uk.org.nationaltrust
974	18917220	3201	0.000011	uk.gov.london
975	18917048	1973	0.000017	scot.gov
976	18916982	3865	0.000009	uk.ac.qub
977	18916460	3807	0.000009	com.governing
978	18916430	528	0.000056	com.businesswire
979	18916300	2253	0.000015	wales.gov
980	18915066	3422	0.000010	com.afp
981	18914982	3080	0.000011	uk.ac.qmul
982	18914878	5154	0.000007	com.ingress
983	18914540	4596	0.000008	com.webcindario
984	18914316	3402	0.000010	org.psychiatryonline
985	18913230	4148	0.000008	org.marxists
986	18913096	4073	0.000009	me.thinglink
987	18912970	1660	0.000018	com.css-tricks
988	18912858	4732	0.000008	ie.nuigalway
989	18912514	4348	0.000008	com.asiaone
990	18912368	3354	0.000010	com.kaspersky-labs
991	18912110	1249	0.000024	com.smashingmagazine
992	18912064	3787	0.000009	org.nationalinterest
993	18911848	556	0.000053	com.adweek
994	18911436	4498	0.000008	ec.com.google
995	18911404	4722	0.000008	bd.com.google
996	18910006	4846	0.000007	uy.com.google
997	18909998	4233	0.000008	com.match
998	18909746	4021	0.000009	ee.google
999	18909688	3962	0.000009	com.adn
1000	18909474	4310	0.000008	com.wnd

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

September 2020 crawl archive now available

October 7, 2020Sebastian Nagel

The crawl archive for September 2020 is now available! The data was crawled between September 18th and October 2nd and contains 3.45 billion web pages or 345 TiB of uncompressed content. It includes page captures of 1.5 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The September crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-40/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-40/segment.paths.gz	100
WARC files	CC-MAIN-2020-40/warc.paths.gz	79600	81.8
WAT files	CC-MAIN-2020-40/wat.paths.gz	79600	23.14
WET files	CC-MAIN-2020-40/wet.paths.gz	79600	10.28
Robots.txt files	CC-MAIN-2020-40/robotstxt.paths.gz	79600	0.22
Non-200 responses files	CC-MAIN-2020-40/non200responses.paths.gz	79600	2.36
URL index files	CC-MAIN-2020-40/cc-index.paths.gz	302	0.27

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-40/. Also the columnar index has been updated to contain this crawl.

August 2020 crawl archive now available

August 19, 2020Sebastian Nagel

The crawl archive for August 2020 is now available! It contains 2.45 billion web pages or 235 TiB of uncompressed content, crawled between August 2nd and 15th. It includes page captures of 940 million URLs unknown in any of our prior crawl archives.

Archive Location and Download

The August crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-34/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-34/segment.paths.gz	100
WARC files	CC-MAIN-2020-34/warc.paths.gz	60000	48.9
WAT files	CC-MAIN-2020-34/wat.paths.gz	60000	16.9
WET files	CC-MAIN-2020-34/wet.paths.gz	60000	7.56
Robots.txt files	CC-MAIN-2020-34/robotstxt.paths.gz	60000	0.19
Non-200 responses files	CC-MAIN-2020-34/non200responses.paths.gz	60000	1.94
URL index files	CC-MAIN-2020-34/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-34/. Also the columnar index has been updated to contain this crawl.

July 2020 crawl archive now available

July 20, 2020Sebastian Nagel

The crawl archive for July 2020 is now available! It contains 3.14 billion web pages or 300 TiB of uncompressed content, crawled between July 2nd and 16th. It includes page captures of 1.1 billion URLs unknown in any of our prior crawl archives.

Bug Fixes and Improvements

The URL index fields "redirect" and "mime" haven’t been filled if the corresponding HTTP headers Location and Content-Type are written in lower-case letters or any other variant not matching case. This bug has been detected during the crawl and was fixed for 90 out of 100 segments. It also affects the columnar index and the fields "fetch_redirect" resp. "content_mime_type". To a minor extend it may affect the detection of character set and content language as the value of the Content-Type header is used as additional hint for the detection. Additional information about this bug fix is given in the corresponding issue report.

Archive Location and Download

The July crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-29/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-29/segment.paths.gz	100
WARC files	CC-MAIN-2020-29/warc.paths.gz	60000	62.64
WAT files	CC-MAIN-2020-29/wat.paths.gz	60000	22.23
WET files	CC-MAIN-2020-29/wet.paths.gz	60000	9.87
Robots.txt files	CC-MAIN-2020-29/robotstxt.paths.gz	60000	0.21
Non-200 responses files	CC-MAIN-2020-29/non200responses.paths.gz	60000	2.52
URL index files	CC-MAIN-2020-29/cc-index.paths.gz	302	0.24

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-29/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs Feb/Mar/May 2020

June 16, 2020Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February, March/April and May/June 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

What’s new?

The host-level graph now includes hosts visited by the crawler but not linking to any other host. Why is this possible – isn’t any host found via links the crawler is following? Yes, but some links were already detected in a prior crawl, not in one of the 3 crawls used to build the web graphs. More details about the issue are given in cc-pyspark#15. The impact of this fix on the graph size is minimal: the recent crawl now includes 1 million nodes (0.1% of all nodes) which are not connected to any other node.

Host-level graph

The graph consists of 927 million nodes and 3.88 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 857 million dangling nodes (92.5%) and the largest strongly connected component contains 47 million (5.1%) nodes.

You can download the graph and the ranks of all 927 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2020-feb-mar-may/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2020-feb-mar-may/host/ as prefix to access the files from everywhere.

Download files of the Common Crawl Feb/Mar/May 2020 host-level webgraph

Size	File	Description
5.67 GB	cc-main-2020-feb-mar-may-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 12 vertices files
17.26 GB	cc-main-2020-feb-mar-may-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 24 edges files

7.40 GB	cc-main-2020-feb-mar-may-host.graph	graph in BVGraph format
2 kB	cc-main-2020-feb-mar-may-host.properties

8.57 GB	cc-main-2020-feb-mar-may-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2020-feb-mar-may-host-t.properties

1 kB	cc-main-2020-feb-mar-may-host.stats	WebGraph statistics

12.16 GB	cc-main-2020-feb-mar-may-host-ranks.txt.gz	harmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 91 million nodes and 1.96 billion edges. 51% or 46 million nodes are dangling nodes, the largest strongly connected component covers 36 million or 39% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2020-feb-mar-may/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2020-feb-mar-may/domain/.

Download files of the Common Crawl Feb/Mar/May 2020 domain-level webgraph

Size	File	Description
0.62 GB	cc-main-2020-feb-mar-may-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
7.79 GB	cc-main-2020-feb-mar-may-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

4.23 GB	cc-main-2020-feb-mar-may-domain.graph	graph in BVGraph format
2 kB	cc-main-2020-feb-mar-may-domain.properties

4.16 GB	cc-main-2020-feb-mar-may-domain-t.graph	transpose of the graph
2 kB	cc-main-2020-feb-mar-may-domain-t.properties

1 kB	cc-main-2020-feb-mar-may-domain.stats	WebGraph statistics

1.96 GB	cc-main-2020-feb-mar-may-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 91 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Feb/Mar/May 2020)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	32667618	1	0.018180	com.googleapis
2	30552772	3	0.011873	com.facebook
3	29569088	2	0.013789	com.google
4	26920460	4	0.007145	com.twitter
5	26883128	5	0.007106	org.w
6	26360448	6	0.006483	com.youtube
7	24719396	9	0.004210	com.instagram
8	24251942	8	0.005125	org.gmpg
9	23841332	7	0.005329	com.googletagmanager
10	23606890	13	0.002940	com.linkedin
11	22741292	10	0.003621	com.cloudflare
12	22732960	12	0.002974	org.wordpress
13	22661910	14	0.002515	com.gravatar
14	22577680	15	0.002438	com.gstatic
15	22378134	22	0.001529	com.pinterest
16	22196962	27	0.001192	org.wikipedia
17	22189650	19	0.001864	com.wordpress
18	22066028	16	0.002404	com.bootstrapcdn
19	21967760	18	0.001884	com.apple
20	21751768	20	0.001863	com.jquery
21	21589606	24	0.001461	com.microsoft
22	21568908	44	0.000785	be.youtu
23	21568474	43	0.000806	com.blogspot
24	21533280	31	0.001104	com.vimeo
25	21415938	46	0.000761	gl.goo
26	21399120	35	0.001040	com.amazonaws
27	21358048	53	0.000665	com.amazon
28	21331634	21	0.001737	com.adobe
29	21324666	23	0.001506	com.wp
30	21209012	70	0.000452	com.tumblr
31	21184360	17	0.001949	com.github
32	21150652	37	0.001008	com.google-analytics
33	21110976	30	0.001152	com.baidu
34	21096692	87	0.000387	com.yahoo
35	21081268	59	0.000547	ly.bit
36	21060360	33	0.001072	com.macromedia
37	21046916	36	0.001035	net.cloudfront
38	21036258	45	0.000763	com.flickr
39	20997926	32	0.001101	com.googlesyndication
40	20993476	26	0.001277	me.wp
41	20980462	97	0.000340	com.googleusercontent
42	20966446	56	0.000624	eu.europa
43	20960242	42	0.000807	net.jsdelivr
44	20959910	52	0.000677	co.t
45	20901872	29	0.001163	ru.yandex
46	20846092	50	0.000742	net.doubleclick
47	20843032	41	0.000869	com.addthis
48	20823518	69	0.000457	io.github
49	20817952	76	0.000433	com.medium
50	20810030	25	0.001287	com.fontawesome
51	20809120	139	0.000189	com.forbes
52	20796434	61	0.000510	org.w3
53	20759102	55	0.000640	com.paypal
54	20757266	109	0.000282	com.soundcloud
55	20754514	90	0.000368	org.creativecommons
56	20747472	57	0.000619	com.vk
57	20711184	54	0.000658	org.mozilla
58	20710182	88	0.000382	com.weebly
59	20698442	84	0.000410	com.wix
60	20675372	102	0.000317	com.weibo
61	20663930	58	0.000604	org.schema
62	20650202	164	0.000151	com.imgur
63	20644452	147	0.000177	org.apache
64	20642282	178	0.000138	uk.co.bbc
65	20625560	129	0.000210	org.archive
66	20610354	274	0.000089	com.ibm
67	20609614	154	0.000169	com.bing
68	20602380	191	0.000125	net.sourceforge
69	20579012	130	0.000207	com.nytimes
70	20578626	150	0.000174	int.who
71	20571012	183	0.000131	com.cnn
72	20561674	174	0.000140	net.slideshare
73	20547634	158	0.000164	gov.cdc
74	20542546	202	0.000116	com.android
75	20527230	228	0.000104	com.wsj
76	20518548	194	0.000122	edu.stanford
77	20505546	205	0.000115	com.businessinsider
78	20495034	254	0.000095	com.oracle
79	20489434	34	0.001049	net.fbcdn
80	20488868	373	0.000067	com.msn
81	20488282	261	0.000093	edu.harvard
82	20483384	310	0.000080	com.go
83	20478152	99	0.000335	com.shopify
84	20471424	267	0.000093	com.bbc
85	20464434	297	0.000083	edu.mit
86	20461340	330	0.000076	com.myspace
87	20458776	62	0.000497	com.whatsapp
88	20457206	289	0.000085	com.appspot
89	20454466	307	0.000080	com.wired
90	20446300	292	0.000085	com.reuters
91	20442004	101	0.000323	com.godaddy
92	20435550	171	0.000147	com.theguardian
93	20417770	143	0.000182	gov.nih
94	20412536	196	0.000120	org.ietf
95	20401330	388	0.000065	gov.nasa
96	20397298	423	0.000061	com.theverge
97	20394736	149	0.000175	com.giphy
98	20394276	382	0.000066	net.researchgate
99	20384930	270	0.000092	com.bloomberg
100	20377778	108	0.000285	com.unpkg
101	20376394	114	0.000271	com.reddit
102	20373856	337	0.000075	com.xinhuanet
103	20366736	215	0.000108	org.gnu
104	20363506	318	0.000079	com.usatoday
105	20352660	813	0.000037	org.chromium
106	20344996	356	0.000071	com.springer
107	20343678	98	0.000335	de.google
108	20342420	28	0.001184	com.qq
109	20341824	345	0.000073	com.example
110	20336510	744	0.000041	edu.psu
111	20324536	468	0.000055	edu.cornell
112	20324378	184	0.000131	com.blogger
113	20314024	60	0.000516	net.akamaihd
114	20304242	375	0.000067	org.hbr
115	20302310	750	0.000040	com.git-scm
116	20300014	937	0.000032	com.wikia
117	20298546	137	0.000191	com.spotify
118	20296012	485	0.000053	edu.yale
119	20295516	113	0.000271	com.jimdo
120	20293140	554	0.000047	com.cbsnews
121	20291946	717	0.000043	com.economist
122	20290574	214	0.000109	com.washingtonpost
123	20288504	140	0.000188	jp.co.yahoo
124	20286470	285	0.000086	com.huffingtonpost
125	20284558	316	0.000080	org.un
126	20281874	410	0.000063	fr.free
127	20279946	473	0.000054	edu.berkeley
128	20275446	287	0.000086	com.cnbc
129	20273280	245	0.000099	com.dribbble
130	20271584	576	0.000046	org.arxiv
131	20269716	151	0.000172	com.issuu
132	20257038	545	0.000047	com.mysql
133	20256262	160	0.000157	com.twimg
134	20252532	107	0.000285	com.statcounter
135	20251682	338	0.000075	uk.co.telegraph
136	20247478	305	0.000081	com.w3schools
137	20246682	561	0.000047	com.gitlab
138	20242210	802	0.000038	edu.columbia
139	20240978	524	0.000049	gov.noaa
140	20230666	122	0.000230	com.ytimg
141	20229900	119	0.000233	com.youtube-nocookie
142	20227656	731	0.000042	org.ieee
143	20227126	333	0.000075	org.npr
144	20225528	729	0.000042	io.readthedocs
145	20225206	286	0.000086	org.acm
146	20222314	339	0.000074	com.time
147	20220430	1180	0.000025	org.eclipse
148	20220382	241	0.000100	org.ampproject
149	20218616	344	0.000074	com.fc2
150	20215730	142	0.000185	com.wixsite
151	20213692	755	0.000040	edu.washington
152	20210122	421	0.000061	com.force
153	20209864	276	0.000089	com.prnewswire
154	20209130	500	0.000052	com.buzzfeed
155	20207136	434	0.000060	com.nationalgeographic
156	20206402	403	0.000063	com.nature
157	20203826	200	0.000118	gle.forms
158	20202490	799	0.000038	org.sciencemag
159	20201144	428	0.000061	com.theatlantic
160	20200104	871	0.000035	com.stackexchange
161	20198142	280	0.000088	com.sciencedirect
162	20185400	332	0.000075	com.staticflickr
163	20184528	495	0.000052	uk.co.independent
164	20182256	263	0.000093	gov.ca
165	20180972	687	0.000043	org.worldbank
166	20175994	435	0.000060	com.mozilla
167	20175400	734	0.000041	com.marketwatch
168	20168098	1087	0.000027	com.hatenablog
169	20167040	364	0.000069	com.nypost
170	20164016	646	0.000043	org.bitbucket
171	20161192	219	0.000107	com.ft
172	20151116	463	0.000056	com.pixabay
173	20143796	354	0.000071	jp.co.rakuten
174	20142652	743	0.000041	edu.upenn
175	20140126	277	0.000089	org.doi
176	20139376	966	0.000031	jp.livedoor
177	20136546	198	0.000120	uk.co.google
178	20134932	407	0.000063	uk.co.dailymail
179	20134404	724	0.000042	org.pbs
180	20133936	258	0.000094	net.behance
181	20132914	192	0.000124	org.wikimedia
182	20127860	917	0.000033	edu.jhu
183	20127828	454	0.000057	gov.whitehouse
184	20122352	856	0.000035	org.weforum
185	20122170	416	0.000062	com.dailymotion
186	20117054	1487	0.000020	com.warnerbros
187	20111898	326	0.000077	org.opensource
188	20110798	1091	0.000027	cn.com.chinadaily
189	20109916	548	0.000047	me.about
190	20109820	232	0.000103	jp.ameblo
191	20108940	558	0.000047	com.oup
192	20103428	325	0.000077	com.digg
193	20097418	455	0.000056	com.entrepreneur
194	20095108	631	0.000044	com.vice
195	20094142	749	0.000040	com.qz
196	20092692	1259	0.000024	com.discovery
197	20091154	444	0.000058	com.goodreads
198	20091052	447	0.000057	gg.discord
199	20082910	1109	0.000027	com.sap
200	20082186	353	0.000071	com.scribd
201	20079412	188	0.000128	com.feedburner
202	20076146	466	0.000055	com.fortune
203	20075556	580	0.000045	com.gartner
204	20072598	1012	0.000029	com.500px
205	20072136	458	0.000056	jp.ne.sakura
206	20067400	176	0.000139	com.imdb
207	20060950	732	0.000042	uk.co.blogspot
208	20059054	1735	0.000018	com.amd
209	20058228	947	0.000032	edu.princeton
210	20056666	890	0.000034	org.cambridge
211	20056572	51	0.000714	com.fb
212	20056272	848	0.000036	com.evernote
213	20054472	144	0.000180	com.dropbox
214	20053532	39	0.000951	com.wixstatic
215	20051662	617	0.000044	org.unesco
216	20050940	1461	0.000020	com.fandom
217	20048152	294	0.000084	com.wiley
218	20046134	768	0.000039	com.withgoogle
219	20039426	1015	0.000029	org.altervista
220	20039010	2337	0.000014	com.wolfram
221	20037920	798	0.000038	com.slate
222	20031484	1201	0.000025	org.kernel
223	20028164	1049	0.000028	edu.purdue
224	20025282	569	0.000046	page.g
225	20021340	786	0.000038	com.trello
226	20017018	230	0.000103	com.disqus
227	20012796	757	0.000040	org.eff
228	20010430	951	0.000031	com.merriam-webster
229	20004686	493	0.000052	gov.usda
230	20004240	981	0.000030	com.netlify
231	20003994	2179	0.000015	com.diigo
232	20002918	807	0.000038	com.vox
233	20002690	180	0.000135	org.allaboutcookies
234	20002220	1206	0.000025	com.jetbrains
235	19999418	1416	0.000021	edu.arizona
236	19994384	542	0.000047	com.tandfonline
237	19993030	844	0.000036	com.foxnews
238	19992184	291	0.000085	com.live
239	19991142	175	0.000140	com.xing
240	19989874	909	0.000033	com.politico
241	19988570	320	0.000079	com.outlook
242	19985036	1135	0.000026	jp.ne.goo
243	19983340	754	0.000040	au.net.abc
244	19982680	1945	0.000016	com.wikidot
245	19977934	793	0.000038	com.investopedia
246	19977574	1066	0.000028	edu.uchicago
247	19976820	1009	0.000029	edu.wisc
248	19975922	197	0.000120	com.eepurl
249	19972560	1039	0.000028	com.bostonglobe
250	19972096	775	0.000039	org.semver
251	19969594	619	0.000044	com.sagepub
252	19969182	497	0.000052	gov.fda
253	19968442	347	0.000073	net.windows
254	19968084	1568	0.000019	edu.osu
255	19965386	319	0.000079	com.nbcnews
256	19963946	244	0.000099	com.myshopify
257	19962892	585	0.000045	cn.google
258	19962530	608	0.000044	site.business
259	19961066	832	0.000036	com.sciencedaily
260	19960380	1044	0.000028	com.strikingly
261	19956366	1236	0.000024	edu.unc
262	19956268	1446	0.000021	edu.virginia
263	19956034	1204	0.000025	co.elastic
264	19952960	1194	0.000025	com.nymag
265	19950500	2206	0.000015	com.renren
266	19950490	742	0.000041	gov.house
267	19950448	2163	0.000015	sg.edu.nus
268	19947976	2285	0.000014	org.wikibooks
269	19947284	1961	0.000016	com.googlesource
270	19940598	235	0.000103	com.wpengine
271	19940158	323	0.000078	com.googlecode
272	19939212	761	0.000040	gov.senate
273	19938008	513	0.000051	com.herokuapp
274	19937738	452	0.000057	org.pewresearch
275	19937492	567	0.000046	org.iana
276	19936954	1093	0.000027	com.podbean
277	19935818	982	0.000030	com.alexa
278	19934742	1629	0.000019	gd.is
279	19933804	103	0.000301	com.paypalobjects
280	19932740	805	0.000038	org.unicef
281	19932416	718	0.000043	com.newyorker
282	19930858	969	0.000031	uk.co.thetimes
283	19929324	404	0.000063	com.patreon
284	19928266	1060	0.000028	com.lifehacker
285	19925940	381	0.000066	com.criteo
286	19924524	997	0.000030	com.huffpost
287	19922576	303	0.000081	com.squareup
288	19922510	839	0.000036	ca.cbc
289	19921808	1145	0.000026	org.wiktionary
290	19918844	146	0.000178	com.addtoany
291	19918174	201	0.000117	com.optimizely
292	19918052	1342	0.000022	edu.msu
293	19915986	1371	0.000022	com.history
294	19913384	418	0.000062	com.calendly
295	19905860	1181	0.000025	com.udemy
296	19903364	809	0.000037	uk.ac.ox
297	19902920	172	0.000145	com.amazon-adsystem
298	19899332	49	0.000743	com.googleadservices
299	19896924	155	0.000167	com.opera
300	19890970	887	0.000034	org.fao
301	19890832	1017	0.000029	com.ecwid
302	19890826	476	0.000054	com.googleblog
303	19887142	211	0.000110	com.stackoverflow
304	19886190	1419	0.000021	uk.ac.lse
305	19885312	360	0.000070	com.getpocket
306	19884456	1667	0.000018	org.maven
307	19883800	915	0.000033	uk.co.guardian
308	19883358	169	0.000148	org.bbb
309	19881084	1337	0.000022	com.aljazeera
310	19880790	255	0.000095	com.aliyuncs
311	19879938	2723	0.000013	net.pixnet
312	19874384	3180	0.000011	net.hinet
313	19869028	1170	0.000025	com.smithsonianmag
314	19868832	1347	0.000022	edu.ucdavis
315	19868258	894	0.000034	gov.congress
316	19867190	1320	0.000023	edu.illinois
317	19865168	1120	0.000026	com.theglobeandmail
318	19863306	1036	0.000029	gov.archives
319	19862414	492	0.000052	it.placehold
320	19861934	93	0.000359	net.facebook
321	19861376	1615	0.000019	hk.com.google
322	19860922	1473	0.000020	ca.sfu
323	19856352	1676	0.000018	blog.home
324	19855290	1073	0.000027	com.apnews
325	19854892	963	0.000031	com.ssrn
326	19853682	3383	0.000010	com.wizards
327	19851102	1997	0.000016	com.nabble
328	19851032	760	0.000040	com.chinaz
329	19850412	3667	0.000010	cn.edu.sjtu
330	19848140	1484	0.000020	com.urbandictionary
331	19844436	1136	0.000026	com.scmp
332	19842326	1489	0.000020	ms.1drv
333	19841796	4361	0.000008	tw.com.gamer
334	19838582	1392	0.000021	com.flipboard
335	19838166	919	0.000033	co.g
336	19837542	547	0.000047	com.gofundme
337	19836996	2097	0.000015	com.france24
338	19835636	1405	0.000021	jp.geocities
339	19833654	1370	0.000022	com.ibtimes
340	19831362	581	0.000045	com.biomedcentral
341	19830056	1128	0.000026	com.britannica
342	19829420	2174	0.000015	com.oregonlive
343	19827062	412	0.000062	com.kickstarter
344	19826214	962	0.000031	com.adjust
345	19824188	867	0.000035	gov.fcc
346	19824048	715	0.000043	uk.co.mirror
347	19823266	589	0.000045	us.icio
348	19823172	1129	0.000026	com.mediafire
349	19821768	1432	0.000021	edu.tamu
350	19821310	587	0.000045	com.usnews
351	19820442	1314	0.000023	org.greenpeace
352	19820252	985	0.000030	edu.academia
353	19819486	1381	0.000021	com.livescience
354	19815972	1684	0.000018	gov.cia
355	19814564	1325	0.000023	com.akamai
356	19813266	930	0.000032	com.chicagotribune
357	19811538	156	0.000167	com.npmjs
358	19811100	1429	0.000021	net.seesaa
359	19810120	329	0.000076	es.google
360	19809710	1238	0.000024	com.reverbnation
361	19809490	550	0.000047	com.quora
362	19808314	3481	0.000010	com.proboards
363	19806268	1040	0.000028	com.thehill
364	19803840	321	0.000078	org.python
365	19801476	1132	0.000026	org.jstor
366	19801018	1722	0.000018	ca.mcgill
367	19799982	167	0.000149	com.zendesk
368	19792890	999	0.000030	com.thelancet
369	19792246	1094	0.000027	com.jamanetwork
370	19788594	1935	0.000016	uk.ac.manchester
371	19785214	540	0.000048	com.udacity
372	19783328	1372	0.000021	ca.utoronto
373	19783082	579	0.000046	com.bigcartel
374	19782230	2487	0.000013	org.wikiquote
375	19781186	1357	0.000022	edu.rutgers
376	19780028	896	0.000034	org.apa
377	19779718	439	0.000059	com.newsweek
378	19778538	920	0.000033	com.healthline
379	19777982	2204	0.000015	com.knowyourmeme
380	19775610	328	0.000077	com.tinyurl
381	19775558	726	0.000042	gov.state
382	19775092	216	0.000108	com.unsplash
383	19773702	1708	0.000018	ca.ualberta
384	19772378	406	0.000063	com.githubusercontent
385	19771900	1471	0.000020	com.asahi
386	19771220	259	0.000094	org.nodejs
387	19769436	475	0.000054	com.latimes
388	19769258	1027	0.000029	com.timeanddate
389	19768686	432	0.000060	com.slack
390	19768410	769	0.000039	jp.shinobi
391	19767976	1674	0.000018	com.buzzfeednews
392	19765038	415	0.000062	com.elsevier
393	19764722	1335	0.000022	edu.gatech
394	19764298	2861	0.000012	com.youdao
395	19761256	895	0.000034	com.brightcove
396	19759730	1774	0.000017	com.bankofamerica
397	19759530	2569	0.000013	edu.byu
398	19758760	1918	0.000016	com.voanews
399	19757586	3164	0.000011	com.opendns
400	19756816	1425	0.000021	com.sky
401	19755780	2336	0.000014	com.slides
402	19754462	1373	0.000021	com.dw
403	19754458	1158	0.000026	com.nikkei
404	19752590	904	0.000033	com.cbslocal
405	19748766	2236	0.000014	net.earthlink
406	19748678	391	0.000064	com.cnet
407	19748150	1642	0.000018	com.xrea
408	19747430	1354	0.000022	uk.co.huffingtonpost
409	19746424	182	0.000133	com.eventbrite
410	19746370	1071	0.000027	com.nydailynews
411	19744090	1305	0.000023	me.vk
412	19743194	918	0.000033	gov.bls
413	19741542	1458	0.000020	org.ap
414	19740936	384	0.000066	net.imgix
415	19739860	2414	0.000014	org.aclweb
416	19739750	1641	0.000018	com.axios
417	19738940	987	0.000030	com.wattpad
418	19737530	1713	0.000018	com.straitstimes
419	19737412	474	0.000054	com.ted
420	19736874	1294	0.000023	edu.brookings
421	19728634	967	0.000031	int.coe
422	19727580	212	0.000109	com.etsy
423	19727112	2392	0.000014	com.biography
424	19726080	865	0.000035	gov.va
425	19725710	217	0.000107	com.typepad
426	19724628	1932	0.000016	com.cocolog-nifty
427	19723580	1608	0.000019	com.reference
428	19720740	553	0.000047	com.livejournal
429	19717406	2096	0.000015	ru.kremlin
430	19716354	815	0.000037	uk.gov.service
431	19715378	298	0.000083	com.techcrunch
432	19712358	2462	0.000013	org.wikisource
433	19712296	1553	0.000019	com.foxbusiness
434	19711620	1281	0.000023	mil.army
435	19711244	1761	0.000017	com.itv
436	19710260	733	0.000041	com.deviantart
437	19705952	1311	0.000023	de.mpg
438	19705288	845	0.000036	gov.justice
439	19704574	1993	0.000016	cn.people
440	19703248	1262	0.000024	au.com.smh
441	19701656	1763	0.000017	org.tensorflow
442	19701634	1223	0.000024	org.ohchr
443	19701000	568	0.000046	ru.gov
444	19700136	400	0.000064	com.technorati
445	19699596	2134	0.000015	jp.co.japantimes
446	19697954	83	0.000413	com.list-manage
447	19697088	1068	0.000028	com.thedrum
448	19696754	1538	0.000019	uk.co.standard
449	19695430	185	0.000131	com.rawgit
450	19694216	2120	0.000015	com.oxforddictionaries
451	19693006	2241	0.000014	com.shutterfly
452	19692082	3147	0.000011	tw.edu.ntu
453	19691564	2550	0.000013	com.smashwords
454	19689862	1862	0.000016	edu.unl
455	19688768	2402	0.000014	org.fas
456	19688646	296	0.000084	uk.org.ico
457	19688138	2710	0.000013	tv.blip
458	19686066	957	0.000031	com.bandsintown
459	19684448	3516	0.000010	cn.org.china
460	19682960	1550	0.000019	uk.co.express
461	19679708	1082	0.000027	jp.jugem
462	19679158	3656	0.000010	info.webry
463	19678730	1403	0.000021	gov.uscourts
464	19677944	2157	0.000015	au.edu.unimelb
465	19675766	92	0.000363	com.wsimg
466	19674868	283	0.000086	ru.rambler
467	19673738	1921	0.000016	com.washingtontimes
468	19671754	351	0.000072	com.proofpoint
469	19669412	74	0.000441	net.jsfiddle
470	19668352	788	0.000038	org.mediawiki
471	19668158	2851	0.000012	jp.blog
472	19667740	1479	0.000020	com.firebaseapp
473	19667418	1618	0.000019	com.webnode
474	19665940	2173	0.000015	com.pbworks
475	19665748	3374	0.000011	com.patheos
476	19665684	3135	0.000011	uk.co.timesonline
477	19663980	2171	0.000015	google.ai
478	19663354	233	0.000103	com.squarespace
479	19662188	2904	0.000012	fr.rfi
480	19660984	1454	0.000020	gov.supremecourt
481	19659200	1889	0.000016	int.unfccc
482	19658534	331	0.000076	com.office
483	19656526	577	0.000046	pl.google
484	19654098	991	0.000030	gov.wa
485	19652796	804	0.000038	gov.sba
486	19652626	1267	0.000023	com.cognitoforms
487	19650066	2207	0.000015	org.csis
488	19649008	366	0.000068	io.codepen
489	19648750	2344	0.000014	com.kobo
490	19646512	110	0.000281	com.mailchimp
491	19643428	1671	0.000018	edu.wustl
492	19642572	2734	0.000013	edu.kit
493	19642334	1480	0.000020	org.hrw
494	19642276	953	0.000031	edu.umich
495	19641856	1389	0.000021	com.dictionary
496	19641544	836	0.000036	com.mapquest
497	19640836	1747	0.000017	org.worldcat
498	19640276	3621	0.000010	net.aljazeera
499	19640144	357	0.000071	com.photobucket
500	19639948	2046	0.000015	net.cnki
501	19638510	1705	0.000018	com.secondlife
502	19638416	2421	0.000014	int.wmo
503	19637888	1089	0.000027	org.ilo
504	19637450	1100	0.000027	google.blog
505	19636692	378	0.000067	com.meetup
506	19634634	995	0.000030	uk.co.pinterest
507	19633770	3397	0.000010	com.freehostia
508	19630412	3256	0.000011	com.doodlekit
509	19629746	936	0.000032	com.arstechnica
510	19628370	3730	0.000009	com.colourlovers
511	19628356	1696	0.000018	ru.ucoz
512	19628298	952	0.000031	com.thenextweb
513	19624458	2286	0.000014	org.unep
514	19622342	2252	0.000014	org.icrc
515	19621808	1424	0.000021	com.findlaw
516	19621134	2334	0.000014	com.similarweb
517	19620696	481	0.000054	com.gmail
518	19619304	3040	0.000012	io.soup
519	19616246	1437	0.000021	com.imageshack
520	19615956	2785	0.000013	com.sputniknews
521	19614078	3080	0.000012	com.smore
522	19613232	3246	0.000011	org.iucnredlist
523	19611766	3117	0.000011	com.kinja
524	19611760	1883	0.000016	com.csmonitor
525	19611604	145	0.000180	ru.mail
526	19610088	1339	0.000022	gov.uscis
527	19608554	446	0.000058	net.secureservercdn
528	19606314	3004	0.000012	sh.now
529	19605748	427	0.000061	tv.twitch
530	19604994	1580	0.000019	link.app
531	19600814	440	0.000059	com.statista
532	19599160	3676	0.000010	jp.hatenablog
533	19595550	4356	0.000008	com.coroflot
534	19595264	3177	0.000011	org.jenkins-ci
535	19595158	1757	0.000017	gov.oregon
536	19593130	3200	0.000011	li.paper
537	19593106	3847	0.000009	com.pixar
538	19589878	3095	0.000011	com.shell
539	19588194	4035	0.000009	com.scienceblogs
540	19586188	1625	0.000019	org.amnesty
541	19584824	892	0.000034	com.thedailybeast
542	19582464	1767	0.000017	org.pypi
543	19582346	2149	0.000015	com.foreignpolicy
544	19580310	2849	0.000012	com.instapaper
545	19579672	2910	0.000012	org.accessnow
546	19578614	1602	0.000019	com.surveygizmo
547	19577780	1733	0.000018	ca.globalnews
548	19576200	3175	0.000011	de.uni-koeln
549	19576198	239	0.000101	io.shields
550	19576184	3377	0.000011	org.lds
551	19575902	2238	0.000014	org.rand
552	19574790	207	0.000114	com.salesforce
553	19574544	3438	0.000010	net.mootools
554	19574428	2357	0.000014	at.ac.univie
555	19574182	4050	0.000009	org.marxists
556	19571664	2860	0.000012	org.panda
557	19571194	2806	0.000013	com.oprah
558	19568576	1874	0.000016	com.justia
559	19567970	3471	0.000010	org.avaaz
560	19567854	2880	0.000012	com.openai
561	19567764	3597	0.000010	org.neocities
562	19567260	3753	0.000009	cn.edu.sdu
563	19564960	762	0.000040	com.netflix
564	19564120	498	0.000052	com.oreilly
565	19563086	4405	0.000008	com.yam
566	19562248	227	0.000105	uk.co.amazon
567	19562204	866	0.000035	com.zoho
568	19560956	629	0.000044	com.zdnet
569	19559966	1298	0.000023	ly.snip
570	19558790	1790	0.000017	ch.ipcc
571	19558664	993	0.000030	uk.parliament
572	19558508	3787	0.000009	com.nestle
573	19556304	1254	0.000024	se.google
574	19556292	2997	0.000012	com.treehugger
575	19555184	1011	0.000029	net.nocookie
576	19555096	4644	0.000008	com.x0
577	19553368	3631	0.000010	org.tvtropes
578	19550992	1141	0.000026	org.sphinx-doc
579	19549994	2122	0.000015	ru.mos
580	19548820	3044	0.000012	es.csic
581	19548530	2913	0.000012	uk.gov.companieshouse
582	19546576	1034	0.000029	com.engadget
583	19546230	1183	0.000025	com.here
584	19545492	5060	0.000007	com.dbs
585	19545438	4103	0.000009	br.ufrj
586	19544204	2159	0.000015	edu.colostate
587	19543398	2706	0.000013	de.uni-heidelberg
588	19540500	3059	0.000012	com.pearltrees
589	19539268	2176	0.000015	net.openid
590	19537880	2600	0.000013	com.mystrikingly
591	19537844	3880	0.000009	com.chinatimes
592	19535834	2400	0.000014	link.page
593	19534182	2354	0.000014	com.real
594	19533432	1836	0.000017	org.ncsl
595	19532288	301	0.000082	com.surveymonkey
596	19531930	362	0.000070	com.hp
597	19531412	1193	0.000025	org.js
598	19530700	2135	0.000015	com.123formbuilder
599	19528842	2426	0.000014	org.vim
600	19528104	3205	0.000011	pl.wp
601	19528018	2602	0.000013	au.com.sbs
602	19526780	170	0.000148	com.yelp
603	19526216	2499	0.000013	uk.ac.kcl
604	19524346	1338	0.000022	org.aarp
605	19523692	2621	0.000013	th.co.google
606	19523156	1006	0.000029	uk.gov.legislation
607	19523042	260	0.000094	com.getbootstrap
608	19522856	3663	0.000010	com.magcloud
609	19522274	3990	0.000009	com.zynga
610	19521942	1268	0.000023	tw.com.google
611	19521922	2829	0.000013	com.kaggle
612	19520130	948	0.000031	gov.gpo
613	19519742	946	0.000032	com.about
614	19519714	3273	0.000011	org.rsf
615	19518740	2976	0.000012	org.tigris
616	19518224	2727	0.000013	uk.ac.leeds
617	19515512	3535	0.000010	de.dw
618	19515434	3019	0.000012	org.cfr
619	19514574	3253	0.000011	de.uni-freiburg
620	19513570	3640	0.000010	de.uni-konstanz
621	19512714	3881	0.000009	ua.at
622	19511254	2117	0.000015	info.worldometers
623	19510314	4657	0.000008	com.embarcadero
624	19509370	2999	0.000012	vn.zing
625	19509134	3229	0.000011	com.bangkokpost
626	19508804	3615	0.000010	ly.rebrand
627	19508548	2008	0.000016	gov.ky
628	19508426	4009	0.000009	org.wilsoncenter
629	19506774	4059	0.000009	jp.hatenadiary
630	19506284	4374	0.000008	com.musictoday
631	19505388	3824	0.000009	org.constitutioncenter
632	19505186	372	0.000067	com.booking
633	19504402	2579	0.000013	com.eiseverywhere
634	19503800	4038	0.000009	com.itsnicethat
635	19503776	3331	0.000011	il.ac.tau
636	19502096	2359	0.000014	mx.com.google
637	19500806	3736	0.000009	com.db
638	19498928	312	0.000080	com.ebay
639	19498588	3578	0.000010	jp.hateblo
640	19498166	3348	0.000011	org.democracynow
641	19497296	3975	0.000009	edu.odu
642	19496812	2815	0.000013	dk.au
643	19496626	4220	0.000008	com.etymonline
644	19496184	2885	0.000012	uk.gov.metoffice
645	19495756	361	0.000070	com.skype
646	19495566	3570	0.000010	com.hsbc
647	19494844	2228	0.000015	com.bankrate
648	19494104	2240	0.000014	gov.wi
649	19493352	1815	0.000017	fi.google
650	19493306	4426	0.000008	com.x10host
651	19492136	3224	0.000011	org.royalsociety
652	19491096	817	0.000037	com.pexels
653	19490358	532	0.000048	com.mashable
654	19490282	4614	0.000008	com.epochtimes
655	19490018	1174	0.000025	edu.ucla
656	19489656	3226	0.000011	cc.reurl
657	19489414	3430	0.000010	com.dailykos
658	19489360	3742	0.000009	uk.ac.uea
659	19488050	3705	0.000010	ca.shaw
660	19486104	1968	0.000016	uk.gov.tfl
661	19485988	3434	0.000010	uk.ac.nhm
662	19485032	3060	0.000012	com.ipage
663	19484754	2498	0.000013	com.prweek
664	19484598	1819	0.000017	gov.usembassy
665	19483966	4861	0.000007	am.do
666	19483636	3086	0.000011	com.viki
667	19483518	3252	0.000011	se.liu
668	19482718	3066	0.000012	com.coca-colacompany
669	19482580	4232	0.000008	br.ufrgs
670	19482498	3639	0.000010	de.uni-kiel
671	19481340	1453	0.000020	com.speakerdeck
672	19480718	3077	0.000012	net.openreview
673	19480660	2208	0.000015	de.auswaertiges-amt
674	19480248	208	0.000113	com.hubspot
675	19479762	2026	0.000016	com.lexisnexis
676	19478700	2106	0.000015	net.ucoz
677	19477552	3494	0.000010	com.iconarchive
678	19477532	819	0.000037	com.steampowered
679	19477286	756	0.000040	com.xiti
680	19477132	2486	0.000013	com.post-gazette
681	19476898	3369	0.000011	com.eklablog
682	19476632	2937	0.000012	uk.co.bbci
683	19476378	1911	0.000016	hu.google
684	19476160	4399	0.000008	com.jacobinmag
685	19475974	3323	0.000011	uk.ac.sussex
686	19474368	3068	0.000012	uk.ac.qmul
687	19474212	3930	0.000009	nf.co
688	19473014	4114	0.000009	com.collinsdictionary
689	19472896	5215	0.000007	com.evaair
690	19472846	2572	0.000013	com.marketwire
691	19472580	3138	0.000011	au.com.telstra
692	19472114	3916	0.000009	it.unitn
693	19471646	898	0.000034	com.visualstudio
694	19471330	3807	0.000009	in.ernet
695	19470994	2906	0.000012	nl.rug
696	19468708	5297	0.000007	org.arkive
697	19468252	252	0.000096	org.drupal
698	19467050	3460	0.000010	ca.dal
699	19467046	3693	0.000010	com.canada
700	19465642	1451	0.000021	com.tinypic
701	19465304	3136	0.000011	org.wri
702	19465034	3698	0.000010	com.la-croix
703	19464108	4557	0.000008	com.mitsubishielectric
704	19463828	4748	0.000008	com.gamejolt
705	19462976	2789	0.000013	gr.google
706	19462882	4882	0.000007	cz.webgarden
707	19462404	3079	0.000012	my.com.thestar
708	19461830	269	0.000092	net.php
709	19461640	4329	0.000008	au.gov.fairwork
710	19460770	2279	0.000014	co.pcdn
711	19460176	3943	0.000009	uk.ac.essex
712	19459984	121	0.000231	org.networkadvertising
713	19459684	3396	0.000010	org.rferl
714	19459068	4211	0.000008	com.sc
715	19459020	3292	0.000011	com.blogfa
716	19458794	3382	0.000010	ca.yelp
717	19457580	4102	0.000009	edu.utm
718	19457248	5694	0.000007	com.anghami
719	19456532	5210	0.000007	su.clan
720	19456144	4095	0.000009	it.justpaste
721	19456006	414	0.000062	com.sxsw
722	19455914	3258	0.000011	com.waterstones
723	19454602	3960	0.000009	com.jigsy
724	19454516	838	0.000036	com.intel
725	19454394	4032	0.000009	ee.ut
726	19453242	916	0.000033	com.docker
727	19452988	738	0.000041	com.samsung
728	19451802	3422	0.000010	es.ucm
729	19450718	2503	0.000013	com.washingtonexaminer
730	19450342	3951	0.000009	tl.page
731	19450206	2209	0.000015	org.wbur
732	19449036	4112	0.000009	site.negocio
733	19448922	2773	0.000013	com.yell
734	19448516	3988	0.000009	com.fatcow
735	19448266	3282	0.000011	pl.poznan
736	19448198	135	0.000194	com.youku
737	19447930	2878	0.000012	ae.thenational
738	19447766	4705	0.000008	id.co.kaskus
739	19447668	3407	0.000010	com.afp
740	19447602	5336	0.000007	net.manilatimes
741	19446734	419	0.000062	com.caniuse
742	19446168	1470	0.000020	com.pastebin
743	19445910	3387	0.000010	uk.org.rspb
744	19445736	765	0.000039	com.moz
745	19444376	4027	0.000009	lv.draugiem
746	19441604	2508	0.000013	gov.dni
747	19440874	2593	0.000013	ro.google
748	19440144	2946	0.000012	com.broadwayworld
749	19439574	3750	0.000009	ru.msu
750	19439374	3766	0.000009	pl.cba
751	19439332	4137	0.000009	org.rfa
752	19439280	5562	0.000007	org.bukkit
753	19439086	2013	0.000016	scot.gov
754	19438868	133	0.000200	com.constantcontact
755	19438826	5638	0.000007	org.adbusters
756	19438094	4517	0.000008	google.design
757	19437654	4154	0.000008	com.macobserver
758	19437088	1649	0.000018	fr.pagesjaunes
759	19437020	2502	0.000013	com.thenation
760	19436776	3973	0.000009	com.bbcamerica
761	19434556	4857	0.000007	com.orgfree
762	19433810	2978	0.000012	com.channelnewsasia
763	19432506	735	0.000041	gov.sec
764	19432502	4008	0.000009	com.teamspeak
765	19432430	2800	0.000013	org.gnupg
766	19432260	3780	0.000009	com.the-scientist
767	19432252	3015	0.000012	com.laweekly
768	19431446	2921	0.000012	au.edu.sydney
769	19430084	3577	0.000010	uk.co.yougov
770	19430000	3140	0.000011	vn.com.google
771	19429942	4417	0.000008	com.50webs
772	19429004	3124	0.000011	org.repec
773	19428938	3215	0.000011	org.ourworldindata
774	19427890	3506	0.000010	com.tradingeconomics
775	19427352	3102	0.000011	tw.com.pchome
776	19426582	3332	0.000011	com.monday
777	19426556	3556	0.000010	org.project-syndicate
778	19425552	2331	0.000014	com.amebaownd
779	19424890	1596	0.000019	org.whatbrowser
780	19424750	1956	0.000016	org.americanbar
781	19424680	3739	0.000009	ie.thejournal
782	19424152	104	0.000298	com.stripe
783	19424140	4014	0.000009	com.hatenadiary
784	19424060	2933	0.000012	org.thinkprogress
785	19423712	3073	0.000012	uk.gov.london
786	19423054	3927	0.000009	com.thesaurus
787	19423006	3475	0.000010	net.webself
788	19422964	3432	0.000010	io.pantheon
789	19421712	3420	0.000010	uk.ac.exeter
790	19421508	4343	0.000008	com.appledaily
791	19421118	3528	0.000010	com.bravesites
792	19420816	5178	0.000007	com.bambuser
793	19420592	3379	0.000011	com.foreignaffairs
794	19419378	2432	0.000013	com.instructables
795	19416388	2185	0.000015	vn.vietnamnet
796	19414736	3994	0.000009	com.webcindario
797	19414328	2823	0.000013	org.ewg
798	19413934	4534	0.000008	ws.nimb
799	19413778	2833	0.000013	org.fullfact
800	19413352	256	0.000095	us.zoom
801	19412556	3685	0.000010	com.encyclopedia
802	19412474	3897	0.000009	de.uni-erlangen
803	19410822	5341	0.000007	net.boards
804	19409598	341	0.000074	com.histats
805	19409534	4201	0.000008	is.pse
806	19409436	748	0.000040	fm.last
807	19407808	3661	0.000010	com.mongabay
808	19407040	3220	0.000011	me.site123
809	19406338	3436	0.000010	com.seetickets
810	19405550	5838	0.000007	com.gamigo
811	19404400	1666	0.000018	com.materialdesignicons
812	19404108	5140	0.000007	bd.com.google
813	19403242	790	0.000038	com.venturebeat
814	19401218	4601	0.000008	uk.org.phrases
815	19400780	3213	0.000011	com.instructure
816	19400298	2817	0.000013	gov.arkansas
817	19399890	72	0.000444	com.livestream
818	19399554	4081	0.000009	cat.uab
819	19399486	3546	0.000010	org.lacity
820	19399372	3612	0.000010	com.heraldscotland
821	19398370	1499	0.000020	com.teachable
822	19396672	2895	0.000012	com.foodandwine
823	19395752	1233	0.000024	com.createjs
824	19394274	2266	0.000014	com.ajc
825	19394172	3950	0.000009	com.rappler
826	19394030	2355	0.000014	net.noscript
827	19393982	4140	0.000009	jp.doorblog
828	19392882	2873	0.000012	com.timeshighereducation
829	19392238	275	0.000089	com.bandcamp
830	19389332	3969	0.000009	jp.ne.hi-ho
831	19388094	3629	0.000010	net.inquirer
832	19387882	552	0.000047	com.cisco
833	19387318	4076	0.000009	pl.lublin
834	19386370	1657	0.000018	com.pcworld
835	19383404	266	0.000093	com.typeform
836	19382886	203	0.000116	com.naver
837	19382698	3723	0.000010	gov.bts
838	19382192	1816	0.000017	jp.makeshop
839	19382102	4462	0.000008	com.tor
840	19382072	4513	0.000008	com.weightwatchers
841	19381346	1438	0.000021	org.khanacademy
842	19381274	954	0.000031	com.thinkwithgoogle
843	19381020	3385	0.000010	uk.ac.jisc
844	19380238	4088	0.000009	ly.genial
845	19379986	4007	0.000009	com.themoscowtimes
846	19378500	3272	0.000011	com.nyt
847	19378434	3760	0.000009	com.springernature
848	19378356	3390	0.000010	int.cbd
849	19377854	6045	0.000006	es.xurl
850	19376898	1756	0.000017	com.netsolhost
851	19376598	3852	0.000009	au.edu.griffith
852	19376054	4740	0.000008	co.edu.unal
853	19376040	4074	0.000009	kr.co.koreatimes
854	19374588	727	0.000042	com.deloitte
855	19374300	4986	0.000007	org.edc
856	19373940	4149	0.000008	vn.tienphong
857	19373476	3515	0.000010	com.thediplomat
858	19372932	4099	0.000009	uk.ac.lancs
859	19372798	5006	0.000007	com.inoreader
860	19372746	4922	0.000007	com.ueuo
861	19372594	1585	0.000019	tv.ustream
862	19372576	3234	0.000011	com.tapatalk
863	19372356	3416	0.000010	nl.wur
864	19372106	4848	0.000007	net.hypermart
865	19371636	2293	0.000014	org.kff
866	19369356	398	0.000064	com.pubmatic
867	19368982	3625	0.000010	org.grist
868	19368480	3088	0.000011	tw.gov.cdc
869	19368288	3389	0.000010	com.gothamist
870	19368130	1106	0.000027	com.gizmodo
871	19368116	4101	0.000009	com.globalpost
872	19367676	814	0.000037	gov.nist
873	19367536	4563	0.000008	org.globalsecurity
874	19366454	4547	0.000008	build.bazel
875	19366384	3782	0.000009	us.ms.state
876	19365878	4256	0.000008	gr.ntua
877	19365776	4444	0.000008	se.thelocal
878	19365372	2963	0.000012	com.politifact
879	19365128	1317	0.000023	com.ensighten
880	19363588	5097	0.000007	ru.my1
881	19362680	3468	0.000010	com.rabbitmq
882	19359698	4138	0.000009	com.elasticbeanstalk
883	19359574	1364	0.000022	com.billboard
884	19359122	4766	0.000008	cc.dict
885	19358774	5687	0.000007	fi.mbnet
886	19357390	879	0.000035	com.aliexpress
887	19356918	210	0.000111	to.amzn
888	19355668	4275	0.000008	edu.ohio
889	19355546	3452	0.000010	com.thejakartapost
890	19355350	3277	0.000011	vn.com.dantri
891	19355080	5285	0.000007	com.galvanize
892	19354880	3484	0.000010	jp.go.ndl
893	19354790	4710	0.000008	com.kiwibox
894	19354514	2140	0.000015	org.linuxfoundation
895	19354500	4801	0.000007	ru.nnov
896	19353166	4288	0.000008	gr.auth
897	19352970	2257	0.000014	net.vnexpress
898	19351770	2900	0.000012	com.crashlytics
899	19351594	1045	0.000028	com.dropboxusercontent
900	19350828	3439	0.000010	com.scotusblog
901	19350712	4090	0.000009	org.carnegieendowment
902	19350278	395	0.000064	com.atlassian
903	19349726	3465	0.000010	com.study
904	19348724	350	0.000072	com.mapbox
905	19348532	1046	0.000028	com.redhat
906	19347886	1799	0.000017	com.bravenet
907	19347460	4284	0.000008	uk.org.npg
908	19347152	4463	0.000008	com.btplc
909	19347148	5289	0.000007	ru.drom
910	19346542	2430	0.000013	com.vimeopro
911	19345900	4419	0.000008	edu.marquette
912	19345644	426	0.000061	com.adweek
913	19345144	914	0.000033	com.shutterstock
914	19345090	1016	0.000029	com.ubuntu
915	19341960	5712	0.000007	in.ac.nptel
916	19341488	1227	0.000024	com.msdn
917	19340714	4707	0.000008	com.vocabulary
918	19340680	3929	0.000009	edu.uaf
919	19339658	3919	0.000009	com.atavist
920	19339456	3201	0.000011	com.healthgrades
921	19339092	2546	0.000013	com.kinstacdn
922	19338384	2345	0.000014	com.gazhall
923	19337938	5398	0.000007	com.asmallorange
924	19337800	3797	0.000009	com.generalmills
925	19336176	4585	0.000008	vn.vtc
926	19335908	1519	0.000020	cn.gov.mofcom
927	19333778	797	0.000038	com.box
928	19333606	3966	0.000009	si.uni-lj
929	19333322	4170	0.000008	az.president
930	19333194	1788	0.000017	org.reactjs
931	19332412	3605	0.000010	com.postaffiliatepro
932	19331922	5192	0.000007	edu.uah
933	19331280	3599	0.000010	org.openedition
934	19330696	4838	0.000007	com.kapook
935	19330382	4153	0.000008	org.caringbridge
936	19330374	483	0.000053	com.aol
937	19329614	2303	0.000014	org.nfpa
938	19329538	5956	0.000006	com.glosbe
939	19329194	4124	0.000009	com.mcall
940	19327622	4289	0.000008	ru.tmweb
941	19326876	4126	0.000009	uk.co.liverpoolecho
942	19326422	4244	0.000008	com.atwebpages
943	19325980	1067	0.000028	com.freepik
944	19324790	4085	0.000009	org.specialolympics
945	19323868	4845	0.000007	net.freeforums
946	19323676	4744	0.000008	uk.ac.westminster
947	19323532	4092	0.000009	com.tok2
948	19323460	1025	0.000029	com.elpais
949	19323150	4946	0.000007	tw.com.sina
950	19322508	3296	0.000011	com.wowza
951	19322306	317	0.000079	com.webs
952	19322024	4697	0.000008	com.warriorplus
953	19321918	3414	0.000010	com.cityam
954	19321812	4482	0.000008	org.fee
955	19321520	4854	0.000007	tw.edu.ntnu
956	19321296	4962	0.000007	com.sparknotes
957	19320202	4516	0.000008	com.newspapers
958	19319634	2192	0.000015	com.tutsplus
959	19319600	5868	0.000007	com.ananova
960	19319274	3818	0.000009	org.opensecrets
961	19319134	633	0.000044	gov.uspto
962	19318722	5680	0.000007	su.moy
963	19318366	1013	0.000029	com.uk
964	19318266	4936	0.000007	ru.pr-cy
965	19318058	3827	0.000009	cz.centrum
966	19317780	4158	0.000008	edu.niu
967	19315320	1665	0.000018	org.webkit
968	19315014	4692	0.000008	pl.edu.amu
969	19314084	5186	0.000007	com.artfire
970	19313894	3800	0.000009	org.ascd
971	19312106	3801	0.000009	edu.scu
972	19311742	4307	0.000008	com.taipeitimes
973	19311568	4351	0.000008	edu.whoi
974	19310854	5949	0.000006	com.voatiengviet
975	19310748	3100	0.000011	com.broadcastingcable
976	19310720	4655	0.000008	hk.rthk
977	19310246	5703	0.000007	com.enotes
978	19309910	488	0.000053	com.indiatimes
979	19309660	860	0.000035	com.playstation
980	19309040	4866	0.000007	com.brothersoft
981	19308948	2708	0.000013	uk.gov.defra
982	19307606	231	0.000103	org.whatwg
983	19307178	4451	0.000008	com.batchgeo
984	19307118	751	0.000040	com.psychologytoday
985	19306368	4263	0.000008	uk.co.lrb
986	19306350	5034	0.000007	ca.pe.gov
987	19305884	4159	0.000008	com.ecowatch
988	19303820	4195	0.000008	com.williamhill
989	19303548	5767	0.000007	pt.ipp
990	19302972	4843	0.000007	uk.org.38degrees
991	19301624	1303	0.000023	com.technologyreview
992	19301464	4091	0.000009	org.spie
993	19301068	959	0.000031	com.libsyn
994	19300572	4795	0.000007	com.storeboard
995	19300548	3260	0.000011	de.bmel
996	19299448	4749	0.000008	net.onlinewebshop
997	19299274	3872	0.000009	ru.1gb
998	19298654	279	0.000088	com.automattic
999	19298502	3870	0.000009	com.piie
1000	19297440	5306	0.000007	com.allthatsinteresting

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

May/June 2020 crawl archive now available

June 10, 2020Sebastian Nagel

The crawl archive for May/June 2020 is now available! It contains 2.75 billion web pages or 255 TiB of uncompressed content, crawled between May 24th and June 7th. It includes page captures of 1.2 billion URLs unknown in any of our prior crawl archives.

Starting with this crawl the WET files indicate the natural language(s) a text is written in. The language is detected using Compact Language Detector 2 (CLD2) and was made available since August 2018 only in WARC and WAT files and URL indexes. It is now also provided in WET files in the WARC header "WARC-Identified-Content-Language". Up to three language(s) are detected per document and given as comma-separated list of ISO-639-3 codes, here one example WET record fragment:

...
WARC-Identified-Content-Language: isl,eng
Content-Type: text/plain
Content-Length: 10494

Bananabrauð með Nutella – Ljúfmeti og lekkerheit
...

Additional information about this improvement is given in the corresponding issue report.

Archive Location and Download

The May/June crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-24/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-24/segment.paths.gz	100
WARC files	CC-MAIN-2020-24/warc.paths.gz	60000	53.16
WAT files	CC-MAIN-2020-24/wat.paths.gz	60000	19.02
WET files	CC-MAIN-2020-24/wet.paths.gz	60000	8.42
Robots.txt files	CC-MAIN-2020-24/robotstxt.paths.gz	60000	0.22
Non-200 responses files	CC-MAIN-2020-24/non200responses.paths.gz	60000	2.77
URL index files	CC-MAIN-2020-24/cc-index.paths.gz	302	0.22

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-24/. Also the columnar index has been updated to contain this crawl.

March/April 2020 crawl archive now available

April 14, 2020Sebastian Nagel

The crawl archive for March/April 2020 is now available! It contains 2.85 billion web pages or 280 TiB of uncompressed content, crawled between March 28th and April 10th. It includes page captures of 1 billion URLs unknown in any of our prior crawl archives.

Archive Location and Download

The March/April crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-16/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-16/segment.paths.gz	100
WARC files	CC-MAIN-2020-16/warc.paths.gz	56000	62.67
WAT files	CC-MAIN-2020-16/wat.paths.gz	56000	20.37
WET files	CC-MAIN-2020-16/wet.paths.gz	56000	8.97
Robots.txt files	CC-MAIN-2020-16/robotstxt.paths.gz	56000	0.19
Non-200 responses files	CC-MAIN-2020-16/non200responses.paths.gz	56000	1.39
URL index files	CC-MAIN-2020-16/cc-index.paths.gz	302	0.21

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-16/. Also the columnar index has been updated to contain this crawl.