Blog – Common Crawl

August 2020 crawl archive now available

August 19, 2020Sebastian Nagel

The crawl archive for August 2020 is now available! It contains 2.45 billion web pages or 235 TiB of uncompressed content, crawled between August 2nd and 15th. It includes page captures of 940 million URLs unknown in any of our prior crawl archives.

Archive Location and Download

The August crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-34/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line, you end up with the S3 and HTTP paths respectively.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-34/segment.paths.gz	100
WARC files	CC-MAIN-2020-34/warc.paths.gz	60000	48.9
WAT files	CC-MAIN-2020-34/wat.paths.gz	60000	16.9
WET files	CC-MAIN-2020-34/wet.paths.gz	60000	7.56
Robots.txt files	CC-MAIN-2020-34/robotstxt.paths.gz	60000	0.19
Non-200 responses files	CC-MAIN-2020-34/non200responses.paths.gz	60000	1.94
URL index files	CC-MAIN-2020-34/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-34/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

July 2020 crawl archive now available

July 20, 2020Sebastian Nagel

The crawl archive for July 2020 is now available! It contains 3.14 billion web pages or 300 TiB of uncompressed content, crawled between July 2nd and 16th. It includes page captures of 1.1 billion URLs unknown in any of our prior crawl archives.

Bug Fixes and Improvements

The URL index fields "redirect" and "mime" haven’t been filled if the corresponding HTTP headers Location and Content-Type are written in lower-case letters or any other variant not matching case. This bug has been detected during the crawl and was fixed for 90 out of 100 segments. It also affects the columnar index and the fields "fetch_redirect" resp. "content_mime_type". To a minor extend it may affect the detection of character set and content language as the value of the Content-Type header is used as additional hint for the detection. Additional information about this bug fix is given in the corresponding issue report.

Archive Location and Download

The July crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-29/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-29/segment.paths.gz	100
WARC files	CC-MAIN-2020-29/warc.paths.gz	60000	62.64
WAT files	CC-MAIN-2020-29/wat.paths.gz	60000	22.23
WET files	CC-MAIN-2020-29/wet.paths.gz	60000	9.87
Robots.txt files	CC-MAIN-2020-29/robotstxt.paths.gz	60000	0.21
Non-200 responses files	CC-MAIN-2020-29/non200responses.paths.gz	60000	2.52
URL index files	CC-MAIN-2020-29/cc-index.paths.gz	302	0.24

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-29/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs Feb/Mar/May 2020

June 16, 2020Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February, March/April and May/June 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

What’s new?

The host-level graph now includes hosts visited by the crawler but not linking to any other host. Why is this possible – isn’t any host found via links the crawler is following? Yes, but some links were already detected in a prior crawl, not in one of the 3 crawls used to build the web graphs. More details about the issue are given in cc-pyspark#15. The impact of this fix on the graph size is minimal: the recent crawl now includes 1 million nodes (0.1% of all nodes) which are not connected to any other node.

Host-level graph

The graph consists of 927 million nodes and 3.88 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 857 million dangling nodes (92.5%) and the largest strongly connected component contains 47 million (5.1%) nodes.

You can download the graph and the ranks of all 927 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2020-feb-mar-may/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2020-feb-mar-may/host/ as prefix to access the files from everywhere.

Download files of the Common Crawl Feb/Mar/May 2020 host-level webgraph

Size	File	Description
5.67 GB	cc-main-2020-feb-mar-may-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 12 vertices files
17.26 GB	cc-main-2020-feb-mar-may-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 24 edges files

7.40 GB	cc-main-2020-feb-mar-may-host.graph	graph in BVGraph format
2 kB	cc-main-2020-feb-mar-may-host.properties

8.57 GB	cc-main-2020-feb-mar-may-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2020-feb-mar-may-host-t.properties

1 kB	cc-main-2020-feb-mar-may-host.stats	WebGraph statistics

12.16 GB	cc-main-2020-feb-mar-may-host-ranks.txt.gz	harmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 91 million nodes and 1.96 billion edges. 51% or 46 million nodes are dangling nodes, the largest strongly connected component covers 36 million or 39% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2020-feb-mar-may/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2020-feb-mar-may/domain/.

Download files of the Common Crawl Feb/Mar/May 2020 domain-level webgraph

Size	File	Description
0.62 GB	cc-main-2020-feb-mar-may-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
7.79 GB	cc-main-2020-feb-mar-may-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

4.23 GB	cc-main-2020-feb-mar-may-domain.graph	graph in BVGraph format
2 kB	cc-main-2020-feb-mar-may-domain.properties

4.16 GB	cc-main-2020-feb-mar-may-domain-t.graph	transpose of the graph
2 kB	cc-main-2020-feb-mar-may-domain-t.properties

1 kB	cc-main-2020-feb-mar-may-domain.stats	WebGraph statistics

1.96 GB	cc-main-2020-feb-mar-may-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 91 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Feb/Mar/May 2020)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	32667618	1	0.018180	com.googleapis
2	30552772	3	0.011873	com.facebook
3	29569088	2	0.013789	com.google
4	26920460	4	0.007145	com.twitter
5	26883128	5	0.007106	org.w
6	26360448	6	0.006483	com.youtube
7	24719396	9	0.004210	com.instagram
8	24251942	8	0.005125	org.gmpg
9	23841332	7	0.005329	com.googletagmanager
10	23606890	13	0.002940	com.linkedin
11	22741292	10	0.003621	com.cloudflare
12	22732960	12	0.002974	org.wordpress
13	22661910	14	0.002515	com.gravatar
14	22577680	15	0.002438	com.gstatic
15	22378134	22	0.001529	com.pinterest
16	22196962	27	0.001192	org.wikipedia
17	22189650	19	0.001864	com.wordpress
18	22066028	16	0.002404	com.bootstrapcdn
19	21967760	18	0.001884	com.apple
20	21751768	20	0.001863	com.jquery
21	21589606	24	0.001461	com.microsoft
22	21568908	44	0.000785	be.youtu
23	21568474	43	0.000806	com.blogspot
24	21533280	31	0.001104	com.vimeo
25	21415938	46	0.000761	gl.goo
26	21399120	35	0.001040	com.amazonaws
27	21358048	53	0.000665	com.amazon
28	21331634	21	0.001737	com.adobe
29	21324666	23	0.001506	com.wp
30	21209012	70	0.000452	com.tumblr
31	21184360	17	0.001949	com.github
32	21150652	37	0.001008	com.google-analytics
33	21110976	30	0.001152	com.baidu
34	21096692	87	0.000387	com.yahoo
35	21081268	59	0.000547	ly.bit
36	21060360	33	0.001072	com.macromedia
37	21046916	36	0.001035	net.cloudfront
38	21036258	45	0.000763	com.flickr
39	20997926	32	0.001101	com.googlesyndication
40	20993476	26	0.001277	me.wp
41	20980462	97	0.000340	com.googleusercontent
42	20966446	56	0.000624	eu.europa
43	20960242	42	0.000807	net.jsdelivr
44	20959910	52	0.000677	co.t
45	20901872	29	0.001163	ru.yandex
46	20846092	50	0.000742	net.doubleclick
47	20843032	41	0.000869	com.addthis
48	20823518	69	0.000457	io.github
49	20817952	76	0.000433	com.medium
50	20810030	25	0.001287	com.fontawesome
51	20809120	139	0.000189	com.forbes
52	20796434	61	0.000510	org.w3
53	20759102	55	0.000640	com.paypal
54	20757266	109	0.000282	com.soundcloud
55	20754514	90	0.000368	org.creativecommons
56	20747472	57	0.000619	com.vk
57	20711184	54	0.000658	org.mozilla
58	20710182	88	0.000382	com.weebly
59	20698442	84	0.000410	com.wix
60	20675372	102	0.000317	com.weibo
61	20663930	58	0.000604	org.schema
62	20650202	164	0.000151	com.imgur
63	20644452	147	0.000177	org.apache
64	20642282	178	0.000138	uk.co.bbc
65	20625560	129	0.000210	org.archive
66	20610354	274	0.000089	com.ibm
67	20609614	154	0.000169	com.bing
68	20602380	191	0.000125	net.sourceforge
69	20579012	130	0.000207	com.nytimes
70	20578626	150	0.000174	int.who
71	20571012	183	0.000131	com.cnn
72	20561674	174	0.000140	net.slideshare
73	20547634	158	0.000164	gov.cdc
74	20542546	202	0.000116	com.android
75	20527230	228	0.000104	com.wsj
76	20518548	194	0.000122	edu.stanford
77	20505546	205	0.000115	com.businessinsider
78	20495034	254	0.000095	com.oracle
79	20489434	34	0.001049	net.fbcdn
80	20488868	373	0.000067	com.msn
81	20488282	261	0.000093	edu.harvard
82	20483384	310	0.000080	com.go
83	20478152	99	0.000335	com.shopify
84	20471424	267	0.000093	com.bbc
85	20464434	297	0.000083	edu.mit
86	20461340	330	0.000076	com.myspace
87	20458776	62	0.000497	com.whatsapp
88	20457206	289	0.000085	com.appspot
89	20454466	307	0.000080	com.wired
90	20446300	292	0.000085	com.reuters
91	20442004	101	0.000323	com.godaddy
92	20435550	171	0.000147	com.theguardian
93	20417770	143	0.000182	gov.nih
94	20412536	196	0.000120	org.ietf
95	20401330	388	0.000065	gov.nasa
96	20397298	423	0.000061	com.theverge
97	20394736	149	0.000175	com.giphy
98	20394276	382	0.000066	net.researchgate
99	20384930	270	0.000092	com.bloomberg
100	20377778	108	0.000285	com.unpkg
101	20376394	114	0.000271	com.reddit
102	20373856	337	0.000075	com.xinhuanet
103	20366736	215	0.000108	org.gnu
104	20363506	318	0.000079	com.usatoday
105	20352660	813	0.000037	org.chromium
106	20344996	356	0.000071	com.springer
107	20343678	98	0.000335	de.google
108	20342420	28	0.001184	com.qq
109	20341824	345	0.000073	com.example
110	20336510	744	0.000041	edu.psu
111	20324536	468	0.000055	edu.cornell
112	20324378	184	0.000131	com.blogger
113	20314024	60	0.000516	net.akamaihd
114	20304242	375	0.000067	org.hbr
115	20302310	750	0.000040	com.git-scm
116	20300014	937	0.000032	com.wikia
117	20298546	137	0.000191	com.spotify
118	20296012	485	0.000053	edu.yale
119	20295516	113	0.000271	com.jimdo
120	20293140	554	0.000047	com.cbsnews
121	20291946	717	0.000043	com.economist
122	20290574	214	0.000109	com.washingtonpost
123	20288504	140	0.000188	jp.co.yahoo
124	20286470	285	0.000086	com.huffingtonpost
125	20284558	316	0.000080	org.un
126	20281874	410	0.000063	fr.free
127	20279946	473	0.000054	edu.berkeley
128	20275446	287	0.000086	com.cnbc
129	20273280	245	0.000099	com.dribbble
130	20271584	576	0.000046	org.arxiv
131	20269716	151	0.000172	com.issuu
132	20257038	545	0.000047	com.mysql
133	20256262	160	0.000157	com.twimg
134	20252532	107	0.000285	com.statcounter
135	20251682	338	0.000075	uk.co.telegraph
136	20247478	305	0.000081	com.w3schools
137	20246682	561	0.000047	com.gitlab
138	20242210	802	0.000038	edu.columbia
139	20240978	524	0.000049	gov.noaa
140	20230666	122	0.000230	com.ytimg
141	20229900	119	0.000233	com.youtube-nocookie
142	20227656	731	0.000042	org.ieee
143	20227126	333	0.000075	org.npr
144	20225528	729	0.000042	io.readthedocs
145	20225206	286	0.000086	org.acm
146	20222314	339	0.000074	com.time
147	20220430	1180	0.000025	org.eclipse
148	20220382	241	0.000100	org.ampproject
149	20218616	344	0.000074	com.fc2
150	20215730	142	0.000185	com.wixsite
151	20213692	755	0.000040	edu.washington
152	20210122	421	0.000061	com.force
153	20209864	276	0.000089	com.prnewswire
154	20209130	500	0.000052	com.buzzfeed
155	20207136	434	0.000060	com.nationalgeographic
156	20206402	403	0.000063	com.nature
157	20203826	200	0.000118	gle.forms
158	20202490	799	0.000038	org.sciencemag
159	20201144	428	0.000061	com.theatlantic
160	20200104	871	0.000035	com.stackexchange
161	20198142	280	0.000088	com.sciencedirect
162	20185400	332	0.000075	com.staticflickr
163	20184528	495	0.000052	uk.co.independent
164	20182256	263	0.000093	gov.ca
165	20180972	687	0.000043	org.worldbank
166	20175994	435	0.000060	com.mozilla
167	20175400	734	0.000041	com.marketwatch
168	20168098	1087	0.000027	com.hatenablog
169	20167040	364	0.000069	com.nypost
170	20164016	646	0.000043	org.bitbucket
171	20161192	219	0.000107	com.ft
172	20151116	463	0.000056	com.pixabay
173	20143796	354	0.000071	jp.co.rakuten
174	20142652	743	0.000041	edu.upenn
175	20140126	277	0.000089	org.doi
176	20139376	966	0.000031	jp.livedoor
177	20136546	198	0.000120	uk.co.google
178	20134932	407	0.000063	uk.co.dailymail
179	20134404	724	0.000042	org.pbs
180	20133936	258	0.000094	net.behance
181	20132914	192	0.000124	org.wikimedia
182	20127860	917	0.000033	edu.jhu
183	20127828	454	0.000057	gov.whitehouse
184	20122352	856	0.000035	org.weforum
185	20122170	416	0.000062	com.dailymotion
186	20117054	1487	0.000020	com.warnerbros
187	20111898	326	0.000077	org.opensource
188	20110798	1091	0.000027	cn.com.chinadaily
189	20109916	548	0.000047	me.about
190	20109820	232	0.000103	jp.ameblo
191	20108940	558	0.000047	com.oup
192	20103428	325	0.000077	com.digg
193	20097418	455	0.000056	com.entrepreneur
194	20095108	631	0.000044	com.vice
195	20094142	749	0.000040	com.qz
196	20092692	1259	0.000024	com.discovery
197	20091154	444	0.000058	com.goodreads
198	20091052	447	0.000057	gg.discord
199	20082910	1109	0.000027	com.sap
200	20082186	353	0.000071	com.scribd
201	20079412	188	0.000128	com.feedburner
202	20076146	466	0.000055	com.fortune
203	20075556	580	0.000045	com.gartner
204	20072598	1012	0.000029	com.500px
205	20072136	458	0.000056	jp.ne.sakura
206	20067400	176	0.000139	com.imdb
207	20060950	732	0.000042	uk.co.blogspot
208	20059054	1735	0.000018	com.amd
209	20058228	947	0.000032	edu.princeton
210	20056666	890	0.000034	org.cambridge
211	20056572	51	0.000714	com.fb
212	20056272	848	0.000036	com.evernote
213	20054472	144	0.000180	com.dropbox
214	20053532	39	0.000951	com.wixstatic
215	20051662	617	0.000044	org.unesco
216	20050940	1461	0.000020	com.fandom
217	20048152	294	0.000084	com.wiley
218	20046134	768	0.000039	com.withgoogle
219	20039426	1015	0.000029	org.altervista
220	20039010	2337	0.000014	com.wolfram
221	20037920	798	0.000038	com.slate
222	20031484	1201	0.000025	org.kernel
223	20028164	1049	0.000028	edu.purdue
224	20025282	569	0.000046	page.g
225	20021340	786	0.000038	com.trello
226	20017018	230	0.000103	com.disqus
227	20012796	757	0.000040	org.eff
228	20010430	951	0.000031	com.merriam-webster
229	20004686	493	0.000052	gov.usda
230	20004240	981	0.000030	com.netlify
231	20003994	2179	0.000015	com.diigo
232	20002918	807	0.000038	com.vox
233	20002690	180	0.000135	org.allaboutcookies
234	20002220	1206	0.000025	com.jetbrains
235	19999418	1416	0.000021	edu.arizona
236	19994384	542	0.000047	com.tandfonline
237	19993030	844	0.000036	com.foxnews
238	19992184	291	0.000085	com.live
239	19991142	175	0.000140	com.xing
240	19989874	909	0.000033	com.politico
241	19988570	320	0.000079	com.outlook
242	19985036	1135	0.000026	jp.ne.goo
243	19983340	754	0.000040	au.net.abc
244	19982680	1945	0.000016	com.wikidot
245	19977934	793	0.000038	com.investopedia
246	19977574	1066	0.000028	edu.uchicago
247	19976820	1009	0.000029	edu.wisc
248	19975922	197	0.000120	com.eepurl
249	19972560	1039	0.000028	com.bostonglobe
250	19972096	775	0.000039	org.semver
251	19969594	619	0.000044	com.sagepub
252	19969182	497	0.000052	gov.fda
253	19968442	347	0.000073	net.windows
254	19968084	1568	0.000019	edu.osu
255	19965386	319	0.000079	com.nbcnews
256	19963946	244	0.000099	com.myshopify
257	19962892	585	0.000045	cn.google
258	19962530	608	0.000044	site.business
259	19961066	832	0.000036	com.sciencedaily
260	19960380	1044	0.000028	com.strikingly
261	19956366	1236	0.000024	edu.unc
262	19956268	1446	0.000021	edu.virginia
263	19956034	1204	0.000025	co.elastic
264	19952960	1194	0.000025	com.nymag
265	19950500	2206	0.000015	com.renren
266	19950490	742	0.000041	gov.house
267	19950448	2163	0.000015	sg.edu.nus
268	19947976	2285	0.000014	org.wikibooks
269	19947284	1961	0.000016	com.googlesource
270	19940598	235	0.000103	com.wpengine
271	19940158	323	0.000078	com.googlecode
272	19939212	761	0.000040	gov.senate
273	19938008	513	0.000051	com.herokuapp
274	19937738	452	0.000057	org.pewresearch
275	19937492	567	0.000046	org.iana
276	19936954	1093	0.000027	com.podbean
277	19935818	982	0.000030	com.alexa
278	19934742	1629	0.000019	gd.is
279	19933804	103	0.000301	com.paypalobjects
280	19932740	805	0.000038	org.unicef
281	19932416	718	0.000043	com.newyorker
282	19930858	969	0.000031	uk.co.thetimes
283	19929324	404	0.000063	com.patreon
284	19928266	1060	0.000028	com.lifehacker
285	19925940	381	0.000066	com.criteo
286	19924524	997	0.000030	com.huffpost
287	19922576	303	0.000081	com.squareup
288	19922510	839	0.000036	ca.cbc
289	19921808	1145	0.000026	org.wiktionary
290	19918844	146	0.000178	com.addtoany
291	19918174	201	0.000117	com.optimizely
292	19918052	1342	0.000022	edu.msu
293	19915986	1371	0.000022	com.history
294	19913384	418	0.000062	com.calendly
295	19905860	1181	0.000025	com.udemy
296	19903364	809	0.000037	uk.ac.ox
297	19902920	172	0.000145	com.amazon-adsystem
298	19899332	49	0.000743	com.googleadservices
299	19896924	155	0.000167	com.opera
300	19890970	887	0.000034	org.fao
301	19890832	1017	0.000029	com.ecwid
302	19890826	476	0.000054	com.googleblog
303	19887142	211	0.000110	com.stackoverflow
304	19886190	1419	0.000021	uk.ac.lse
305	19885312	360	0.000070	com.getpocket
306	19884456	1667	0.000018	org.maven
307	19883800	915	0.000033	uk.co.guardian
308	19883358	169	0.000148	org.bbb
309	19881084	1337	0.000022	com.aljazeera
310	19880790	255	0.000095	com.aliyuncs
311	19879938	2723	0.000013	net.pixnet
312	19874384	3180	0.000011	net.hinet
313	19869028	1170	0.000025	com.smithsonianmag
314	19868832	1347	0.000022	edu.ucdavis
315	19868258	894	0.000034	gov.congress
316	19867190	1320	0.000023	edu.illinois
317	19865168	1120	0.000026	com.theglobeandmail
318	19863306	1036	0.000029	gov.archives
319	19862414	492	0.000052	it.placehold
320	19861934	93	0.000359	net.facebook
321	19861376	1615	0.000019	hk.com.google
322	19860922	1473	0.000020	ca.sfu
323	19856352	1676	0.000018	blog.home
324	19855290	1073	0.000027	com.apnews
325	19854892	963	0.000031	com.ssrn
326	19853682	3383	0.000010	com.wizards
327	19851102	1997	0.000016	com.nabble
328	19851032	760	0.000040	com.chinaz
329	19850412	3667	0.000010	cn.edu.sjtu
330	19848140	1484	0.000020	com.urbandictionary
331	19844436	1136	0.000026	com.scmp
332	19842326	1489	0.000020	ms.1drv
333	19841796	4361	0.000008	tw.com.gamer
334	19838582	1392	0.000021	com.flipboard
335	19838166	919	0.000033	co.g
336	19837542	547	0.000047	com.gofundme
337	19836996	2097	0.000015	com.france24
338	19835636	1405	0.000021	jp.geocities
339	19833654	1370	0.000022	com.ibtimes
340	19831362	581	0.000045	com.biomedcentral
341	19830056	1128	0.000026	com.britannica
342	19829420	2174	0.000015	com.oregonlive
343	19827062	412	0.000062	com.kickstarter
344	19826214	962	0.000031	com.adjust
345	19824188	867	0.000035	gov.fcc
346	19824048	715	0.000043	uk.co.mirror
347	19823266	589	0.000045	us.icio
348	19823172	1129	0.000026	com.mediafire
349	19821768	1432	0.000021	edu.tamu
350	19821310	587	0.000045	com.usnews
351	19820442	1314	0.000023	org.greenpeace
352	19820252	985	0.000030	edu.academia
353	19819486	1381	0.000021	com.livescience
354	19815972	1684	0.000018	gov.cia
355	19814564	1325	0.000023	com.akamai
356	19813266	930	0.000032	com.chicagotribune
357	19811538	156	0.000167	com.npmjs
358	19811100	1429	0.000021	net.seesaa
359	19810120	329	0.000076	es.google
360	19809710	1238	0.000024	com.reverbnation
361	19809490	550	0.000047	com.quora
362	19808314	3481	0.000010	com.proboards
363	19806268	1040	0.000028	com.thehill
364	19803840	321	0.000078	org.python
365	19801476	1132	0.000026	org.jstor
366	19801018	1722	0.000018	ca.mcgill
367	19799982	167	0.000149	com.zendesk
368	19792890	999	0.000030	com.thelancet
369	19792246	1094	0.000027	com.jamanetwork
370	19788594	1935	0.000016	uk.ac.manchester
371	19785214	540	0.000048	com.udacity
372	19783328	1372	0.000021	ca.utoronto
373	19783082	579	0.000046	com.bigcartel
374	19782230	2487	0.000013	org.wikiquote
375	19781186	1357	0.000022	edu.rutgers
376	19780028	896	0.000034	org.apa
377	19779718	439	0.000059	com.newsweek
378	19778538	920	0.000033	com.healthline
379	19777982	2204	0.000015	com.knowyourmeme
380	19775610	328	0.000077	com.tinyurl
381	19775558	726	0.000042	gov.state
382	19775092	216	0.000108	com.unsplash
383	19773702	1708	0.000018	ca.ualberta
384	19772378	406	0.000063	com.githubusercontent
385	19771900	1471	0.000020	com.asahi
386	19771220	259	0.000094	org.nodejs
387	19769436	475	0.000054	com.latimes
388	19769258	1027	0.000029	com.timeanddate
389	19768686	432	0.000060	com.slack
390	19768410	769	0.000039	jp.shinobi
391	19767976	1674	0.000018	com.buzzfeednews
392	19765038	415	0.000062	com.elsevier
393	19764722	1335	0.000022	edu.gatech
394	19764298	2861	0.000012	com.youdao
395	19761256	895	0.000034	com.brightcove
396	19759730	1774	0.000017	com.bankofamerica
397	19759530	2569	0.000013	edu.byu
398	19758760	1918	0.000016	com.voanews
399	19757586	3164	0.000011	com.opendns
400	19756816	1425	0.000021	com.sky
401	19755780	2336	0.000014	com.slides
402	19754462	1373	0.000021	com.dw
403	19754458	1158	0.000026	com.nikkei
404	19752590	904	0.000033	com.cbslocal
405	19748766	2236	0.000014	net.earthlink
406	19748678	391	0.000064	com.cnet
407	19748150	1642	0.000018	com.xrea
408	19747430	1354	0.000022	uk.co.huffingtonpost
409	19746424	182	0.000133	com.eventbrite
410	19746370	1071	0.000027	com.nydailynews
411	19744090	1305	0.000023	me.vk
412	19743194	918	0.000033	gov.bls
413	19741542	1458	0.000020	org.ap
414	19740936	384	0.000066	net.imgix
415	19739860	2414	0.000014	org.aclweb
416	19739750	1641	0.000018	com.axios
417	19738940	987	0.000030	com.wattpad
418	19737530	1713	0.000018	com.straitstimes
419	19737412	474	0.000054	com.ted
420	19736874	1294	0.000023	edu.brookings
421	19728634	967	0.000031	int.coe
422	19727580	212	0.000109	com.etsy
423	19727112	2392	0.000014	com.biography
424	19726080	865	0.000035	gov.va
425	19725710	217	0.000107	com.typepad
426	19724628	1932	0.000016	com.cocolog-nifty
427	19723580	1608	0.000019	com.reference
428	19720740	553	0.000047	com.livejournal
429	19717406	2096	0.000015	ru.kremlin
430	19716354	815	0.000037	uk.gov.service
431	19715378	298	0.000083	com.techcrunch
432	19712358	2462	0.000013	org.wikisource
433	19712296	1553	0.000019	com.foxbusiness
434	19711620	1281	0.000023	mil.army
435	19711244	1761	0.000017	com.itv
436	19710260	733	0.000041	com.deviantart
437	19705952	1311	0.000023	de.mpg
438	19705288	845	0.000036	gov.justice
439	19704574	1993	0.000016	cn.people
440	19703248	1262	0.000024	au.com.smh
441	19701656	1763	0.000017	org.tensorflow
442	19701634	1223	0.000024	org.ohchr
443	19701000	568	0.000046	ru.gov
444	19700136	400	0.000064	com.technorati
445	19699596	2134	0.000015	jp.co.japantimes
446	19697954	83	0.000413	com.list-manage
447	19697088	1068	0.000028	com.thedrum
448	19696754	1538	0.000019	uk.co.standard
449	19695430	185	0.000131	com.rawgit
450	19694216	2120	0.000015	com.oxforddictionaries
451	19693006	2241	0.000014	com.shutterfly
452	19692082	3147	0.000011	tw.edu.ntu
453	19691564	2550	0.000013	com.smashwords
454	19689862	1862	0.000016	edu.unl
455	19688768	2402	0.000014	org.fas
456	19688646	296	0.000084	uk.org.ico
457	19688138	2710	0.000013	tv.blip
458	19686066	957	0.000031	com.bandsintown
459	19684448	3516	0.000010	cn.org.china
460	19682960	1550	0.000019	uk.co.express
461	19679708	1082	0.000027	jp.jugem
462	19679158	3656	0.000010	info.webry
463	19678730	1403	0.000021	gov.uscourts
464	19677944	2157	0.000015	au.edu.unimelb
465	19675766	92	0.000363	com.wsimg
466	19674868	283	0.000086	ru.rambler
467	19673738	1921	0.000016	com.washingtontimes
468	19671754	351	0.000072	com.proofpoint
469	19669412	74	0.000441	net.jsfiddle
470	19668352	788	0.000038	org.mediawiki
471	19668158	2851	0.000012	jp.blog
472	19667740	1479	0.000020	com.firebaseapp
473	19667418	1618	0.000019	com.webnode
474	19665940	2173	0.000015	com.pbworks
475	19665748	3374	0.000011	com.patheos
476	19665684	3135	0.000011	uk.co.timesonline
477	19663980	2171	0.000015	google.ai
478	19663354	233	0.000103	com.squarespace
479	19662188	2904	0.000012	fr.rfi
480	19660984	1454	0.000020	gov.supremecourt
481	19659200	1889	0.000016	int.unfccc
482	19658534	331	0.000076	com.office
483	19656526	577	0.000046	pl.google
484	19654098	991	0.000030	gov.wa
485	19652796	804	0.000038	gov.sba
486	19652626	1267	0.000023	com.cognitoforms
487	19650066	2207	0.000015	org.csis
488	19649008	366	0.000068	io.codepen
489	19648750	2344	0.000014	com.kobo
490	19646512	110	0.000281	com.mailchimp
491	19643428	1671	0.000018	edu.wustl
492	19642572	2734	0.000013	edu.kit
493	19642334	1480	0.000020	org.hrw
494	19642276	953	0.000031	edu.umich
495	19641856	1389	0.000021	com.dictionary
496	19641544	836	0.000036	com.mapquest
497	19640836	1747	0.000017	org.worldcat
498	19640276	3621	0.000010	net.aljazeera
499	19640144	357	0.000071	com.photobucket
500	19639948	2046	0.000015	net.cnki
501	19638510	1705	0.000018	com.secondlife
502	19638416	2421	0.000014	int.wmo
503	19637888	1089	0.000027	org.ilo
504	19637450	1100	0.000027	google.blog
505	19636692	378	0.000067	com.meetup
506	19634634	995	0.000030	uk.co.pinterest
507	19633770	3397	0.000010	com.freehostia
508	19630412	3256	0.000011	com.doodlekit
509	19629746	936	0.000032	com.arstechnica
510	19628370	3730	0.000009	com.colourlovers
511	19628356	1696	0.000018	ru.ucoz
512	19628298	952	0.000031	com.thenextweb
513	19624458	2286	0.000014	org.unep
514	19622342	2252	0.000014	org.icrc
515	19621808	1424	0.000021	com.findlaw
516	19621134	2334	0.000014	com.similarweb
517	19620696	481	0.000054	com.gmail
518	19619304	3040	0.000012	io.soup
519	19616246	1437	0.000021	com.imageshack
520	19615956	2785	0.000013	com.sputniknews
521	19614078	3080	0.000012	com.smore
522	19613232	3246	0.000011	org.iucnredlist
523	19611766	3117	0.000011	com.kinja
524	19611760	1883	0.000016	com.csmonitor
525	19611604	145	0.000180	ru.mail
526	19610088	1339	0.000022	gov.uscis
527	19608554	446	0.000058	net.secureservercdn
528	19606314	3004	0.000012	sh.now
529	19605748	427	0.000061	tv.twitch
530	19604994	1580	0.000019	link.app
531	19600814	440	0.000059	com.statista
532	19599160	3676	0.000010	jp.hatenablog
533	19595550	4356	0.000008	com.coroflot
534	19595264	3177	0.000011	org.jenkins-ci
535	19595158	1757	0.000017	gov.oregon
536	19593130	3200	0.000011	li.paper
537	19593106	3847	0.000009	com.pixar
538	19589878	3095	0.000011	com.shell
539	19588194	4035	0.000009	com.scienceblogs
540	19586188	1625	0.000019	org.amnesty
541	19584824	892	0.000034	com.thedailybeast
542	19582464	1767	0.000017	org.pypi
543	19582346	2149	0.000015	com.foreignpolicy
544	19580310	2849	0.000012	com.instapaper
545	19579672	2910	0.000012	org.accessnow
546	19578614	1602	0.000019	com.surveygizmo
547	19577780	1733	0.000018	ca.globalnews
548	19576200	3175	0.000011	de.uni-koeln
549	19576198	239	0.000101	io.shields
550	19576184	3377	0.000011	org.lds
551	19575902	2238	0.000014	org.rand
552	19574790	207	0.000114	com.salesforce
553	19574544	3438	0.000010	net.mootools
554	19574428	2357	0.000014	at.ac.univie
555	19574182	4050	0.000009	org.marxists
556	19571664	2860	0.000012	org.panda
557	19571194	2806	0.000013	com.oprah
558	19568576	1874	0.000016	com.justia
559	19567970	3471	0.000010	org.avaaz
560	19567854	2880	0.000012	com.openai
561	19567764	3597	0.000010	org.neocities
562	19567260	3753	0.000009	cn.edu.sdu
563	19564960	762	0.000040	com.netflix
564	19564120	498	0.000052	com.oreilly
565	19563086	4405	0.000008	com.yam
566	19562248	227	0.000105	uk.co.amazon
567	19562204	866	0.000035	com.zoho
568	19560956	629	0.000044	com.zdnet
569	19559966	1298	0.000023	ly.snip
570	19558790	1790	0.000017	ch.ipcc
571	19558664	993	0.000030	uk.parliament
572	19558508	3787	0.000009	com.nestle
573	19556304	1254	0.000024	se.google
574	19556292	2997	0.000012	com.treehugger
575	19555184	1011	0.000029	net.nocookie
576	19555096	4644	0.000008	com.x0
577	19553368	3631	0.000010	org.tvtropes
578	19550992	1141	0.000026	org.sphinx-doc
579	19549994	2122	0.000015	ru.mos
580	19548820	3044	0.000012	es.csic
581	19548530	2913	0.000012	uk.gov.companieshouse
582	19546576	1034	0.000029	com.engadget
583	19546230	1183	0.000025	com.here
584	19545492	5060	0.000007	com.dbs
585	19545438	4103	0.000009	br.ufrj
586	19544204	2159	0.000015	edu.colostate
587	19543398	2706	0.000013	de.uni-heidelberg
588	19540500	3059	0.000012	com.pearltrees
589	19539268	2176	0.000015	net.openid
590	19537880	2600	0.000013	com.mystrikingly
591	19537844	3880	0.000009	com.chinatimes
592	19535834	2400	0.000014	link.page
593	19534182	2354	0.000014	com.real
594	19533432	1836	0.000017	org.ncsl
595	19532288	301	0.000082	com.surveymonkey
596	19531930	362	0.000070	com.hp
597	19531412	1193	0.000025	org.js
598	19530700	2135	0.000015	com.123formbuilder
599	19528842	2426	0.000014	org.vim
600	19528104	3205	0.000011	pl.wp
601	19528018	2602	0.000013	au.com.sbs
602	19526780	170	0.000148	com.yelp
603	19526216	2499	0.000013	uk.ac.kcl
604	19524346	1338	0.000022	org.aarp
605	19523692	2621	0.000013	th.co.google
606	19523156	1006	0.000029	uk.gov.legislation
607	19523042	260	0.000094	com.getbootstrap
608	19522856	3663	0.000010	com.magcloud
609	19522274	3990	0.000009	com.zynga
610	19521942	1268	0.000023	tw.com.google
611	19521922	2829	0.000013	com.kaggle
612	19520130	948	0.000031	gov.gpo
613	19519742	946	0.000032	com.about
614	19519714	3273	0.000011	org.rsf
615	19518740	2976	0.000012	org.tigris
616	19518224	2727	0.000013	uk.ac.leeds
617	19515512	3535	0.000010	de.dw
618	19515434	3019	0.000012	org.cfr
619	19514574	3253	0.000011	de.uni-freiburg
620	19513570	3640	0.000010	de.uni-konstanz
621	19512714	3881	0.000009	ua.at
622	19511254	2117	0.000015	info.worldometers
623	19510314	4657	0.000008	com.embarcadero
624	19509370	2999	0.000012	vn.zing
625	19509134	3229	0.000011	com.bangkokpost
626	19508804	3615	0.000010	ly.rebrand
627	19508548	2008	0.000016	gov.ky
628	19508426	4009	0.000009	org.wilsoncenter
629	19506774	4059	0.000009	jp.hatenadiary
630	19506284	4374	0.000008	com.musictoday
631	19505388	3824	0.000009	org.constitutioncenter
632	19505186	372	0.000067	com.booking
633	19504402	2579	0.000013	com.eiseverywhere
634	19503800	4038	0.000009	com.itsnicethat
635	19503776	3331	0.000011	il.ac.tau
636	19502096	2359	0.000014	mx.com.google
637	19500806	3736	0.000009	com.db
638	19498928	312	0.000080	com.ebay
639	19498588	3578	0.000010	jp.hateblo
640	19498166	3348	0.000011	org.democracynow
641	19497296	3975	0.000009	edu.odu
642	19496812	2815	0.000013	dk.au
643	19496626	4220	0.000008	com.etymonline
644	19496184	2885	0.000012	uk.gov.metoffice
645	19495756	361	0.000070	com.skype
646	19495566	3570	0.000010	com.hsbc
647	19494844	2228	0.000015	com.bankrate
648	19494104	2240	0.000014	gov.wi
649	19493352	1815	0.000017	fi.google
650	19493306	4426	0.000008	com.x10host
651	19492136	3224	0.000011	org.royalsociety
652	19491096	817	0.000037	com.pexels
653	19490358	532	0.000048	com.mashable
654	19490282	4614	0.000008	com.epochtimes
655	19490018	1174	0.000025	edu.ucla
656	19489656	3226	0.000011	cc.reurl
657	19489414	3430	0.000010	com.dailykos
658	19489360	3742	0.000009	uk.ac.uea
659	19488050	3705	0.000010	ca.shaw
660	19486104	1968	0.000016	uk.gov.tfl
661	19485988	3434	0.000010	uk.ac.nhm
662	19485032	3060	0.000012	com.ipage
663	19484754	2498	0.000013	com.prweek
664	19484598	1819	0.000017	gov.usembassy
665	19483966	4861	0.000007	am.do
666	19483636	3086	0.000011	com.viki
667	19483518	3252	0.000011	se.liu
668	19482718	3066	0.000012	com.coca-colacompany
669	19482580	4232	0.000008	br.ufrgs
670	19482498	3639	0.000010	de.uni-kiel
671	19481340	1453	0.000020	com.speakerdeck
672	19480718	3077	0.000012	net.openreview
673	19480660	2208	0.000015	de.auswaertiges-amt
674	19480248	208	0.000113	com.hubspot
675	19479762	2026	0.000016	com.lexisnexis
676	19478700	2106	0.000015	net.ucoz
677	19477552	3494	0.000010	com.iconarchive
678	19477532	819	0.000037	com.steampowered
679	19477286	756	0.000040	com.xiti
680	19477132	2486	0.000013	com.post-gazette
681	19476898	3369	0.000011	com.eklablog
682	19476632	2937	0.000012	uk.co.bbci
683	19476378	1911	0.000016	hu.google
684	19476160	4399	0.000008	com.jacobinmag
685	19475974	3323	0.000011	uk.ac.sussex
686	19474368	3068	0.000012	uk.ac.qmul
687	19474212	3930	0.000009	nf.co
688	19473014	4114	0.000009	com.collinsdictionary
689	19472896	5215	0.000007	com.evaair
690	19472846	2572	0.000013	com.marketwire
691	19472580	3138	0.000011	au.com.telstra
692	19472114	3916	0.000009	it.unitn
693	19471646	898	0.000034	com.visualstudio
694	19471330	3807	0.000009	in.ernet
695	19470994	2906	0.000012	nl.rug
696	19468708	5297	0.000007	org.arkive
697	19468252	252	0.000096	org.drupal
698	19467050	3460	0.000010	ca.dal
699	19467046	3693	0.000010	com.canada
700	19465642	1451	0.000021	com.tinypic
701	19465304	3136	0.000011	org.wri
702	19465034	3698	0.000010	com.la-croix
703	19464108	4557	0.000008	com.mitsubishielectric
704	19463828	4748	0.000008	com.gamejolt
705	19462976	2789	0.000013	gr.google
706	19462882	4882	0.000007	cz.webgarden
707	19462404	3079	0.000012	my.com.thestar
708	19461830	269	0.000092	net.php
709	19461640	4329	0.000008	au.gov.fairwork
710	19460770	2279	0.000014	co.pcdn
711	19460176	3943	0.000009	uk.ac.essex
712	19459984	121	0.000231	org.networkadvertising
713	19459684	3396	0.000010	org.rferl
714	19459068	4211	0.000008	com.sc
715	19459020	3292	0.000011	com.blogfa
716	19458794	3382	0.000010	ca.yelp
717	19457580	4102	0.000009	edu.utm
718	19457248	5694	0.000007	com.anghami
719	19456532	5210	0.000007	su.clan
720	19456144	4095	0.000009	it.justpaste
721	19456006	414	0.000062	com.sxsw
722	19455914	3258	0.000011	com.waterstones
723	19454602	3960	0.000009	com.jigsy
724	19454516	838	0.000036	com.intel
725	19454394	4032	0.000009	ee.ut
726	19453242	916	0.000033	com.docker
727	19452988	738	0.000041	com.samsung
728	19451802	3422	0.000010	es.ucm
729	19450718	2503	0.000013	com.washingtonexaminer
730	19450342	3951	0.000009	tl.page
731	19450206	2209	0.000015	org.wbur
732	19449036	4112	0.000009	site.negocio
733	19448922	2773	0.000013	com.yell
734	19448516	3988	0.000009	com.fatcow
735	19448266	3282	0.000011	pl.poznan
736	19448198	135	0.000194	com.youku
737	19447930	2878	0.000012	ae.thenational
738	19447766	4705	0.000008	id.co.kaskus
739	19447668	3407	0.000010	com.afp
740	19447602	5336	0.000007	net.manilatimes
741	19446734	419	0.000062	com.caniuse
742	19446168	1470	0.000020	com.pastebin
743	19445910	3387	0.000010	uk.org.rspb
744	19445736	765	0.000039	com.moz
745	19444376	4027	0.000009	lv.draugiem
746	19441604	2508	0.000013	gov.dni
747	19440874	2593	0.000013	ro.google
748	19440144	2946	0.000012	com.broadwayworld
749	19439574	3750	0.000009	ru.msu
750	19439374	3766	0.000009	pl.cba
751	19439332	4137	0.000009	org.rfa
752	19439280	5562	0.000007	org.bukkit
753	19439086	2013	0.000016	scot.gov
754	19438868	133	0.000200	com.constantcontact
755	19438826	5638	0.000007	org.adbusters
756	19438094	4517	0.000008	google.design
757	19437654	4154	0.000008	com.macobserver
758	19437088	1649	0.000018	fr.pagesjaunes
759	19437020	2502	0.000013	com.thenation
760	19436776	3973	0.000009	com.bbcamerica
761	19434556	4857	0.000007	com.orgfree
762	19433810	2978	0.000012	com.channelnewsasia
763	19432506	735	0.000041	gov.sec
764	19432502	4008	0.000009	com.teamspeak
765	19432430	2800	0.000013	org.gnupg
766	19432260	3780	0.000009	com.the-scientist
767	19432252	3015	0.000012	com.laweekly
768	19431446	2921	0.000012	au.edu.sydney
769	19430084	3577	0.000010	uk.co.yougov
770	19430000	3140	0.000011	vn.com.google
771	19429942	4417	0.000008	com.50webs
772	19429004	3124	0.000011	org.repec
773	19428938	3215	0.000011	org.ourworldindata
774	19427890	3506	0.000010	com.tradingeconomics
775	19427352	3102	0.000011	tw.com.pchome
776	19426582	3332	0.000011	com.monday
777	19426556	3556	0.000010	org.project-syndicate
778	19425552	2331	0.000014	com.amebaownd
779	19424890	1596	0.000019	org.whatbrowser
780	19424750	1956	0.000016	org.americanbar
781	19424680	3739	0.000009	ie.thejournal
782	19424152	104	0.000298	com.stripe
783	19424140	4014	0.000009	com.hatenadiary
784	19424060	2933	0.000012	org.thinkprogress
785	19423712	3073	0.000012	uk.gov.london
786	19423054	3927	0.000009	com.thesaurus
787	19423006	3475	0.000010	net.webself
788	19422964	3432	0.000010	io.pantheon
789	19421712	3420	0.000010	uk.ac.exeter
790	19421508	4343	0.000008	com.appledaily
791	19421118	3528	0.000010	com.bravesites
792	19420816	5178	0.000007	com.bambuser
793	19420592	3379	0.000011	com.foreignaffairs
794	19419378	2432	0.000013	com.instructables
795	19416388	2185	0.000015	vn.vietnamnet
796	19414736	3994	0.000009	com.webcindario
797	19414328	2823	0.000013	org.ewg
798	19413934	4534	0.000008	ws.nimb
799	19413778	2833	0.000013	org.fullfact
800	19413352	256	0.000095	us.zoom
801	19412556	3685	0.000010	com.encyclopedia
802	19412474	3897	0.000009	de.uni-erlangen
803	19410822	5341	0.000007	net.boards
804	19409598	341	0.000074	com.histats
805	19409534	4201	0.000008	is.pse
806	19409436	748	0.000040	fm.last
807	19407808	3661	0.000010	com.mongabay
808	19407040	3220	0.000011	me.site123
809	19406338	3436	0.000010	com.seetickets
810	19405550	5838	0.000007	com.gamigo
811	19404400	1666	0.000018	com.materialdesignicons
812	19404108	5140	0.000007	bd.com.google
813	19403242	790	0.000038	com.venturebeat
814	19401218	4601	0.000008	uk.org.phrases
815	19400780	3213	0.000011	com.instructure
816	19400298	2817	0.000013	gov.arkansas
817	19399890	72	0.000444	com.livestream
818	19399554	4081	0.000009	cat.uab
819	19399486	3546	0.000010	org.lacity
820	19399372	3612	0.000010	com.heraldscotland
821	19398370	1499	0.000020	com.teachable
822	19396672	2895	0.000012	com.foodandwine
823	19395752	1233	0.000024	com.createjs
824	19394274	2266	0.000014	com.ajc
825	19394172	3950	0.000009	com.rappler
826	19394030	2355	0.000014	net.noscript
827	19393982	4140	0.000009	jp.doorblog
828	19392882	2873	0.000012	com.timeshighereducation
829	19392238	275	0.000089	com.bandcamp
830	19389332	3969	0.000009	jp.ne.hi-ho
831	19388094	3629	0.000010	net.inquirer
832	19387882	552	0.000047	com.cisco
833	19387318	4076	0.000009	pl.lublin
834	19386370	1657	0.000018	com.pcworld
835	19383404	266	0.000093	com.typeform
836	19382886	203	0.000116	com.naver
837	19382698	3723	0.000010	gov.bts
838	19382192	1816	0.000017	jp.makeshop
839	19382102	4462	0.000008	com.tor
840	19382072	4513	0.000008	com.weightwatchers
841	19381346	1438	0.000021	org.khanacademy
842	19381274	954	0.000031	com.thinkwithgoogle
843	19381020	3385	0.000010	uk.ac.jisc
844	19380238	4088	0.000009	ly.genial
845	19379986	4007	0.000009	com.themoscowtimes
846	19378500	3272	0.000011	com.nyt
847	19378434	3760	0.000009	com.springernature
848	19378356	3390	0.000010	int.cbd
849	19377854	6045	0.000006	es.xurl
850	19376898	1756	0.000017	com.netsolhost
851	19376598	3852	0.000009	au.edu.griffith
852	19376054	4740	0.000008	co.edu.unal
853	19376040	4074	0.000009	kr.co.koreatimes
854	19374588	727	0.000042	com.deloitte
855	19374300	4986	0.000007	org.edc
856	19373940	4149	0.000008	vn.tienphong
857	19373476	3515	0.000010	com.thediplomat
858	19372932	4099	0.000009	uk.ac.lancs
859	19372798	5006	0.000007	com.inoreader
860	19372746	4922	0.000007	com.ueuo
861	19372594	1585	0.000019	tv.ustream
862	19372576	3234	0.000011	com.tapatalk
863	19372356	3416	0.000010	nl.wur
864	19372106	4848	0.000007	net.hypermart
865	19371636	2293	0.000014	org.kff
866	19369356	398	0.000064	com.pubmatic
867	19368982	3625	0.000010	org.grist
868	19368480	3088	0.000011	tw.gov.cdc
869	19368288	3389	0.000010	com.gothamist
870	19368130	1106	0.000027	com.gizmodo
871	19368116	4101	0.000009	com.globalpost
872	19367676	814	0.000037	gov.nist
873	19367536	4563	0.000008	org.globalsecurity
874	19366454	4547	0.000008	build.bazel
875	19366384	3782	0.000009	us.ms.state
876	19365878	4256	0.000008	gr.ntua
877	19365776	4444	0.000008	se.thelocal
878	19365372	2963	0.000012	com.politifact
879	19365128	1317	0.000023	com.ensighten
880	19363588	5097	0.000007	ru.my1
881	19362680	3468	0.000010	com.rabbitmq
882	19359698	4138	0.000009	com.elasticbeanstalk
883	19359574	1364	0.000022	com.billboard
884	19359122	4766	0.000008	cc.dict
885	19358774	5687	0.000007	fi.mbnet
886	19357390	879	0.000035	com.aliexpress
887	19356918	210	0.000111	to.amzn
888	19355668	4275	0.000008	edu.ohio
889	19355546	3452	0.000010	com.thejakartapost
890	19355350	3277	0.000011	vn.com.dantri
891	19355080	5285	0.000007	com.galvanize
892	19354880	3484	0.000010	jp.go.ndl
893	19354790	4710	0.000008	com.kiwibox
894	19354514	2140	0.000015	org.linuxfoundation
895	19354500	4801	0.000007	ru.nnov
896	19353166	4288	0.000008	gr.auth
897	19352970	2257	0.000014	net.vnexpress
898	19351770	2900	0.000012	com.crashlytics
899	19351594	1045	0.000028	com.dropboxusercontent
900	19350828	3439	0.000010	com.scotusblog
901	19350712	4090	0.000009	org.carnegieendowment
902	19350278	395	0.000064	com.atlassian
903	19349726	3465	0.000010	com.study
904	19348724	350	0.000072	com.mapbox
905	19348532	1046	0.000028	com.redhat
906	19347886	1799	0.000017	com.bravenet
907	19347460	4284	0.000008	uk.org.npg
908	19347152	4463	0.000008	com.btplc
909	19347148	5289	0.000007	ru.drom
910	19346542	2430	0.000013	com.vimeopro
911	19345900	4419	0.000008	edu.marquette
912	19345644	426	0.000061	com.adweek
913	19345144	914	0.000033	com.shutterstock
914	19345090	1016	0.000029	com.ubuntu
915	19341960	5712	0.000007	in.ac.nptel
916	19341488	1227	0.000024	com.msdn
917	19340714	4707	0.000008	com.vocabulary
918	19340680	3929	0.000009	edu.uaf
919	19339658	3919	0.000009	com.atavist
920	19339456	3201	0.000011	com.healthgrades
921	19339092	2546	0.000013	com.kinstacdn
922	19338384	2345	0.000014	com.gazhall
923	19337938	5398	0.000007	com.asmallorange
924	19337800	3797	0.000009	com.generalmills
925	19336176	4585	0.000008	vn.vtc
926	19335908	1519	0.000020	cn.gov.mofcom
927	19333778	797	0.000038	com.box
928	19333606	3966	0.000009	si.uni-lj
929	19333322	4170	0.000008	az.president
930	19333194	1788	0.000017	org.reactjs
931	19332412	3605	0.000010	com.postaffiliatepro
932	19331922	5192	0.000007	edu.uah
933	19331280	3599	0.000010	org.openedition
934	19330696	4838	0.000007	com.kapook
935	19330382	4153	0.000008	org.caringbridge
936	19330374	483	0.000053	com.aol
937	19329614	2303	0.000014	org.nfpa
938	19329538	5956	0.000006	com.glosbe
939	19329194	4124	0.000009	com.mcall
940	19327622	4289	0.000008	ru.tmweb
941	19326876	4126	0.000009	uk.co.liverpoolecho
942	19326422	4244	0.000008	com.atwebpages
943	19325980	1067	0.000028	com.freepik
944	19324790	4085	0.000009	org.specialolympics
945	19323868	4845	0.000007	net.freeforums
946	19323676	4744	0.000008	uk.ac.westminster
947	19323532	4092	0.000009	com.tok2
948	19323460	1025	0.000029	com.elpais
949	19323150	4946	0.000007	tw.com.sina
950	19322508	3296	0.000011	com.wowza
951	19322306	317	0.000079	com.webs
952	19322024	4697	0.000008	com.warriorplus
953	19321918	3414	0.000010	com.cityam
954	19321812	4482	0.000008	org.fee
955	19321520	4854	0.000007	tw.edu.ntnu
956	19321296	4962	0.000007	com.sparknotes
957	19320202	4516	0.000008	com.newspapers
958	19319634	2192	0.000015	com.tutsplus
959	19319600	5868	0.000007	com.ananova
960	19319274	3818	0.000009	org.opensecrets
961	19319134	633	0.000044	gov.uspto
962	19318722	5680	0.000007	su.moy
963	19318366	1013	0.000029	com.uk
964	19318266	4936	0.000007	ru.pr-cy
965	19318058	3827	0.000009	cz.centrum
966	19317780	4158	0.000008	edu.niu
967	19315320	1665	0.000018	org.webkit
968	19315014	4692	0.000008	pl.edu.amu
969	19314084	5186	0.000007	com.artfire
970	19313894	3800	0.000009	org.ascd
971	19312106	3801	0.000009	edu.scu
972	19311742	4307	0.000008	com.taipeitimes
973	19311568	4351	0.000008	edu.whoi
974	19310854	5949	0.000006	com.voatiengviet
975	19310748	3100	0.000011	com.broadcastingcable
976	19310720	4655	0.000008	hk.rthk
977	19310246	5703	0.000007	com.enotes
978	19309910	488	0.000053	com.indiatimes
979	19309660	860	0.000035	com.playstation
980	19309040	4866	0.000007	com.brothersoft
981	19308948	2708	0.000013	uk.gov.defra
982	19307606	231	0.000103	org.whatwg
983	19307178	4451	0.000008	com.batchgeo
984	19307118	751	0.000040	com.psychologytoday
985	19306368	4263	0.000008	uk.co.lrb
986	19306350	5034	0.000007	ca.pe.gov
987	19305884	4159	0.000008	com.ecowatch
988	19303820	4195	0.000008	com.williamhill
989	19303548	5767	0.000007	pt.ipp
990	19302972	4843	0.000007	uk.org.38degrees
991	19301624	1303	0.000023	com.technologyreview
992	19301464	4091	0.000009	org.spie
993	19301068	959	0.000031	com.libsyn
994	19300572	4795	0.000007	com.storeboard
995	19300548	3260	0.000011	de.bmel
996	19299448	4749	0.000008	net.onlinewebshop
997	19299274	3872	0.000009	ru.1gb
998	19298654	279	0.000088	com.automattic
999	19298502	3870	0.000009	com.piie
1000	19297440	5306	0.000007	com.allthatsinteresting

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

May/June 2020 crawl archive now available

June 10, 2020Sebastian Nagel

The crawl archive for May/June 2020 is now available! It contains 2.75 billion web pages or 255 TiB of uncompressed content, crawled between May 24th and June 7th. It includes page captures of 1.2 billion URLs unknown in any of our prior crawl archives.

Starting with this crawl the WET files indicate the natural language(s) a text is written in. The language is detected using Compact Language Detector 2 (CLD2) and was made available since August 2018 only in WARC and WAT files and URL indexes. It is now also provided in WET files in the WARC header "WARC-Identified-Content-Language". Up to three language(s) are detected per document and given as comma-separated list of ISO-639-3 codes, here one example WET record fragment:

...
WARC-Identified-Content-Language: isl,eng
Content-Type: text/plain
Content-Length: 10494

Bananabrauð með Nutella – Ljúfmeti og lekkerheit
...

Additional information about this improvement is given in the corresponding issue report.

Archive Location and Download

The May/June crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-24/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-24/segment.paths.gz	100
WARC files	CC-MAIN-2020-24/warc.paths.gz	60000	53.16
WAT files	CC-MAIN-2020-24/wat.paths.gz	60000	19.02
WET files	CC-MAIN-2020-24/wet.paths.gz	60000	8.42
Robots.txt files	CC-MAIN-2020-24/robotstxt.paths.gz	60000	0.22
Non-200 responses files	CC-MAIN-2020-24/non200responses.paths.gz	60000	2.77
URL index files	CC-MAIN-2020-24/cc-index.paths.gz	302	0.22

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-24/. Also the columnar index has been updated to contain this crawl.

March/April 2020 crawl archive now available

April 14, 2020Sebastian Nagel

The crawl archive for March/April 2020 is now available! It contains 2.85 billion web pages or 280 TiB of uncompressed content, crawled between March 28th and April 10th. It includes page captures of 1 billion URLs unknown in any of our prior crawl archives.

Archive Location and Download

The March/April crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-16/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-16/segment.paths.gz	100
WARC files	CC-MAIN-2020-16/warc.paths.gz	56000	62.67
WAT files	CC-MAIN-2020-16/wat.paths.gz	56000	20.37
WET files	CC-MAIN-2020-16/wet.paths.gz	56000	8.97
Robots.txt files	CC-MAIN-2020-16/robotstxt.paths.gz	56000	0.19
Non-200 responses files	CC-MAIN-2020-16/non200responses.paths.gz	56000	1.39
URL index files	CC-MAIN-2020-16/cc-index.paths.gz	302	0.21

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-16/. Also the columnar index has been updated to contain this crawl.

February 2020 crawl archive now available

March 4, 2020Sebastian Nagel

The crawl archive for February 2020 is now available! It contains 2.6 billion web pages or 240 TiB of uncompressed content, crawled between February 16th and 29th. It includes page captures of 1 billion URLs unknown in any of our prior crawl archives.

Improvements and Fixes

The HTTP headers in WARC response records have been fixed: the HTTP response status line now has a white space following the status code if the reason-phrase is empty. E.g., if a server sends an empty message (instead of “OK”), the status line will include a trailing space character: “HTTP/1.1 200 ”. Following RFC 7230 the white space between status code and message is mandatory. Please refer to the bug report NUTCH-2763 for further details.

Archive Location and Download

The February crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-10/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-10/segment.paths.gz	100
WARC files	CC-MAIN-2020-10/warc.paths.gz	56000	49.28
WAT files	CC-MAIN-2020-10/wat.paths.gz	56000	17.98
WET files	CC-MAIN-2020-10/wet.paths.gz	56000	7.97
Robots.txt files	CC-MAIN-2020-10/robotstxt.paths.gz	56000	0.22
Non-200 responses files	CC-MAIN-2020-10/non200responses.paths.gz	56000	2.21
URL index files	CC-MAIN-2020-10/cc-index.paths.gz	302	0.2

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-10/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs Nov/Dec/Jan 2019 – 2020

February 10, 2020Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, December 2019 and January 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

Host-level graph

The graph consists of 1.24 billion nodes and 4.54 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 1.17 billion dangling nodes (95%) and the largest strongly connected component contains 45 million (3.6%) nodes.

You can download the graph and the ranks of all 1.24 billion hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-20-nov-dec-jan/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-20-nov-dec-jan/host/ as prefix to access the files from everywhere.

Download files of the Common Crawl Nov/Dec/Jan 2019-20 domain-level webgraph

Size	File	Description
7.23 GB	cc-main-2019-20-nov-dec-jan-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 12 vertices files
20.16 GB	cc-main-2019-20-nov-dec-jan-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 24 edges files

8.42 GB	cc-main-2019-20-nov-dec-jan-host.graph	graph in BVGraph format
2 kB	cc-main-2019-20-nov-dec-jan-host.properties

10.80 GB	cc-main-2019-20-nov-dec-jan-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2019-20-nov-dec-jan-host-t.properties

1 kB	cc-main-2019-20-nov-dec-jan-host.stats	WebGraph statistics

16.32 GB	cc-main-2019-20-nov-dec-jan-host-ranks.txt.gz	harmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 85.8 million nodes and 1.9 billion edges. 51% or 44 million nodes are dangling nodes, the largest strongly connected component covers 34 million or 39% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-20-nov-dec-jan/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-20-nov-dec-jan/domain/.

Download files of the Common Crawl Nov/Dec/Jan 2019-20 domain-level webgraph

Size	File	Description
0.59 GB	cc-main-2019-20-nov-dec-jan-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
7.65 GB	cc-main-2019-20-nov-dec-jan-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

4.10 GB	cc-main-2019-20-nov-dec-jan-domain.graph	graph in BVGraph format
2 kB	cc-main-2019-20-nov-dec-jan-domain.properties

4.13 GB	cc-main-2019-20-nov-dec-jan-domain-t.graph	transpose of the graph
2 kB	cc-main-2019-20-nov-dec-jan-domain-t.properties

1 kB	cc-main-2019-20-nov-dec-jan-domain.stats	WebGraph statistics

1.86 GB	cc-main-2019-20-nov-dec-jan-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 86 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Nov/Dec/Jan 2019-2020)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	30598398	1	0.019072	com.googleapis
2	29113136	3	0.012214	com.facebook
3	27475138	2	0.013236	com.google
4	25610480	4	0.007452	com.twitter
5	24947126	5	0.007174	org.w
6	24904712	6	0.006611	com.youtube
7	23281504	9	0.004269	com.instagram
8	22446296	7	0.005561	org.gmpg
9	22154750	8	0.005033	com.googletagmanager
10	22107784	13	0.003001	com.linkedin
11	21307220	10	0.003433	org.wordpress
12	21290688	20	0.001717	com.gravatar
13	21096944	11	0.003266	com.cloudflare
14	21086168	23	0.001516	com.pinterest
15	20869868	16	0.002242	com.gstatic
16	20855286	15	0.002366	com.wordpress
17	20713268	25	0.001234	org.wikipedia
18	20641712	17	0.002130	com.apple
19	20584368	14	0.002460	com.bootstrapcdn
20	20371046	33	0.001102	com.vimeo
21	20336732	42	0.000833	com.blogspot
22	20237058	18	0.001787	com.jquery
23	20220572	50	0.000732	be.youtu
24	20218806	32	0.001130	com.microsoft
25	20119820	49	0.000737	com.wp
26	20059828	19	0.001764	com.adobe
27	20029544	52	0.000709	com.amazon
28	20013346	44	0.000784	gl.goo
29	19972608	35	0.001020	com.amazonaws
30	19941934	67	0.000471	com.tumblr
31	19937390	64	0.000518	ly.bit
32	19862906	29	0.001164	com.macromedia
33	19817494	30	0.001151	com.baidu
34	19816382	38	0.000958	com.google-analytics
35	19808066	31	0.001142	com.googlesyndication
36	19781460	34	0.001092	net.cloudfront
37	19770754	24	0.001254	ru.yandex
38	19749302	53	0.000693	com.flickr
39	19700112	22	0.001568	com.github
40	19698814	80	0.000368	com.yahoo
41	19676534	58	0.000644	eu.europa
42	19632506	115	0.000287	com.reddit
43	19603190	41	0.000918	com.addthis
44	19561006	72	0.000403	com.weebly
45	19559908	43	0.000823	org.w3
46	19550116	63	0.000524	me.wp
47	19546614	108	0.000313	com.googleusercontent
48	19531876	45	0.000777	io.github
49	19523224	182	0.000140	org.wikimedia
50	19521602	70	0.000422	com.medium
51	19516098	47	0.000743	org.schema
52	19496222	46	0.000754	net.jsdelivr
53	19495662	76	0.000374	org.creativecommons
54	19472610	173	0.000153	com.imgur
55	19459848	36	0.000989	net.doubleclick
56	19451512	51	0.000711	com.wix
57	19416054	186	0.000138	uk.co.bbc
58	19408312	149	0.000181	com.forbes
59	19405426	93	0.000332	com.weibo
60	19404990	60	0.000601	co.t
61	19398622	28	0.001192	com.fontawesome
62	19382484	48	0.000741	com.paypal
63	19375914	210	0.000114	com.cnn
64	19372292	144	0.000190	org.archive
65	19368680	61	0.000583	org.mozilla
66	19348182	189	0.000137	net.sourceforge
67	19304774	315	0.000079	edu.mit
68	19302936	174	0.000152	com.theguardian
69	19293782	278	0.000089	edu.harvard
70	19283372	179	0.000144	com.bing
71	19271390	117	0.000286	com.jimdo
72	19270368	140	0.000202	com.nytimes
73	19260752	27	0.001195	com.qq
74	19259406	237	0.000103	com.wsj
75	19257754	21	0.001581	org.apache
76	19256254	56	0.000658	com.googleadservices
77	19241110	215	0.000111	com.washingtonpost
78	19237816	269	0.000092	com.bloomberg
79	19237164	259	0.000094	com.techcrunch
80	19210562	544	0.000049	com.deviantart
81	19205946	162	0.000162	org.ietf
82	19198346	247	0.000098	com.oracle
83	19194696	279	0.000089	com.android
84	19193448	69	0.000430	com.list-manage
85	19174968	397	0.000065	com.ted
86	19167820	321	0.000078	com.reuters
87	19161964	312	0.000080	com.wired
88	19158768	154	0.000173	com.wixsite
89	19154874	341	0.000073	com.ft
90	19152068	330	0.000076	uk.co.telegraph
91	19151666	427	0.000060	com.theverge
92	19145078	155	0.000172	gov.nih
93	19143886	272	0.000091	com.myspace
94	19142172	377	0.000068	gov.nasa
95	19136022	291	0.000085	com.bbc
96	19126138	339	0.000074	com.example
97	19119916	239	0.000103	org.python
98	19117868	82	0.000361	com.whatsapp
99	19111550	122	0.000267	com.unpkg
100	19110346	188	0.000138	uk.co.google
101	19089504	707	0.000041	com.economist
102	19088766	255	0.000095	com.appspot
103	19082300	384	0.000067	uk.co.dailymail
104	19081904	209	0.000115	org.gnu
105	19081708	262	0.000093	com.githubusercontent
106	19080182	131	0.000232	com.ytimg
107	19077156	320	0.000078	org.un
108	19075806	163	0.000162	com.giphy
109	19068860	398	0.000065	com.latimes
110	19067606	169	0.000157	com.twimg
111	19066404	431	0.000060	com.googleblog
112	19056162	176	0.000148	com.blogger
113	19054310	232	0.000104	com.dribbble
114	19052824	207	0.000115	com.npmjs
115	19050524	564	0.000047	org.arxiv
116	19045194	666	0.000042	edu.upenn
117	19042770	171	0.000154	com.eventbrite
118	19036612	379	0.000068	com.springer
119	19032422	277	0.000090	org.ampproject
120	19031354	557	0.000047	com.gitlab
121	19025616	596	0.000045	com.vice
122	19025236	206	0.000116	com.disqus
123	19023978	1036	0.000031	com.hatenablog
124	19023406	835	0.000039	edu.columbia
125	19018670	818	0.000040	io.readthedocs
126	19008280	205	0.000116	me.t
127	19005842	390	0.000066	com.w3schools
128	19004030	941	0.000034	org.chromium
129	19003900	418	0.000062	com.nature
130	19001452	716	0.000041	com.slate
131	19000862	157	0.000171	jp.co.yahoo
132	18997566	325	0.000077	com.time
133	18997164	430	0.000060	com.statista
134	18990336	744	0.000040	com.ubuntu
135	18985466	158	0.000167	com.yelp
136	18982550	632	0.000043	org.worldbank
137	18982428	143	0.000191	com.spotify
138	18981078	347	0.000072	com.skype
139	18978660	935	0.000034	com.playstation
140	18976524	306	0.000082	com.fc2
141	18973312	1304	0.000024	org.coursera
142	18969438	121	0.000281	com.stripe
143	18968562	816	0.000040	com.qz
144	18968168	617	0.000044	com.git-scm
145	18966448	486	0.000053	uk.co.independent
146	18965810	199	0.000126	com.eepurl
147	18964444	961	0.000034	com.500px
148	18964342	405	0.000063	net.researchgate
149	18962654	241	0.000101	com.bandcamp
150	18959274	55	0.000669	net.facebook
151	18956478	389	0.000066	com.outlook
152	18955490	229	0.000105	com.unsplash
153	18950460	631	0.000043	com.mysql
154	18949330	419	0.000062	com.theatlantic
155	18948456	116	0.000286	com.soundcloud
156	18948364	180	0.000143	com.amazon-adsystem
157	18947670	123	0.000259	org.networkadvertising
158	18940768	571	0.000046	org.bitbucket
159	18940298	1163	0.000027	com.jetbrains
160	18936474	408	0.000063	com.mozilla
161	18936104	498	0.000052	com.nationalgeographic
162	18932906	316	0.000079	com.usatoday
163	18930890	439	0.000059	com.criteo
164	18927342	837	0.000039	uk.ac.ox
165	18925560	468	0.000054	com.fortune
166	18924956	466	0.000055	com.pixabay
167	18922224	1278	0.000024	uk.co.thesun
168	18921476	230	0.000104	net.behance
169	18916700	1547	0.000019	com.amd
170	18915574	822	0.000039	com.evernote
171	18909918	40	0.000932	com.vk
172	18909318	799	0.000040	com.about
173	18907910	505	0.000051	uk.co.blogspot
174	18904000	1193	0.000026	se.haxx
175	18903686	251	0.000097	gle.forms
176	18900132	719	0.000041	com.docker
177	18900020	967	0.000033	uk.co.guardian
178	18899576	303	0.000082	org.doi
179	18898330	497	0.000052	me.about
180	18896862	525	0.000050	gg.discord
181	18894432	1383	0.000022	com.instructables
182	18891582	147	0.000189	com.dropbox
183	18888432	1000	0.000032	com.scientificamerican
184	18885228	332	0.000076	jp.co.rakuten
185	18881548	903	0.000036	google.blog
186	18875704	194	0.000130	com.feedburner
187	18870758	1081	0.000030	org.altervista
188	18869068	634	0.000043	org.unesco
189	18868966	1082	0.000030	org.eclipse
190	18868488	275	0.000090	gov.ca
191	18866552	875	0.000037	jp.livedoor
192	18865164	1490	0.000020	org.phys
193	18864674	292	0.000085	com.sciencedirect
194	18860674	212	0.000113	jp.ameblo
195	18855700	572	0.000046	gov.loc
196	18854892	993	0.000033	org.cambridge
197	18845512	403	0.000064	ca.google
198	18842558	663	0.000042	edu.washington
199	18836168	71	0.000408	net.slideshare
200	18832808	543	0.000049	com.cisco
201	18829838	1364	0.000023	edu.rutgers
202	18828808	304	0.000082	com.nbcnews
203	18827294	243	0.000099	ru.rambler
204	18826594	823	0.000039	au.net.abc
205	18826034	998	0.000032	uk.co.thetimes
206	18825554	1472	0.000021	com.bankofamerica
207	18821912	62	0.000549	com.fb
208	18821670	863	0.000037	org.sciencemag
209	18813844	887	0.000036	com.speakerdeck
210	18810986	412	0.000063	jp.ne.sakura
211	18806748	220	0.000109	org.iana
212	18802112	1258	0.000025	com.wikidot
213	18799706	1650	0.000018	pt.sapo
214	18792880	905	0.000036	uk.co.mirror
215	18790620	218	0.000110	edu.stanford
216	18787734	1143	0.000028	org.kernel
217	18786526	450	0.000057	com.elsevier
218	18784420	1452	0.000021	edu.osu
219	18783446	1622	0.000019	com.googlesource
220	18782426	1395	0.000022	com.vogue
221	18781902	74	0.000401	net.akamaihd
222	18781258	842	0.000038	gov.fcc
223	18780926	1240	0.000025	ms.1drv
224	18779106	1377	0.000022	edu.asu
225	18778828	254	0.000095	com.businessinsider
226	18778534	844	0.000038	co.ibb
227	18778386	1843	0.000016	com.wolfram
228	18771090	721	0.000041	com.trello
229	18769148	106	0.000323	com.paypalobjects
230	18766298	301	0.000083	net.windows
231	18762238	1216	0.000026	jp.geocities
232	18762150	587	0.000045	com.box
233	18761840	876	0.000037	com.sciencedaily
234	18758802	227	0.000106	com.wpengine
235	18752986	493	0.000052	com.herokuapp
236	18752026	1030	0.000032	edu.princeton
237	18748814	915	0.000035	edu.academia
238	18748232	284	0.000087	com.googlecode
239	18746146	1177	0.000027	com.asahi
240	18740044	1375	0.000022	com.newscientist
241	18733718	1428	0.000021	blog.home
242	18732458	314	0.000080	com.tinyurl
243	18732010	516	0.000051	com.udacity
244	18731740	2103	0.000014	com.wizards
245	18729906	386	0.000067	com.cnet
246	18729644	1421	0.000022	com.ndtv
247	18729370	335	0.000075	com.getpocket
248	18729276	1349	0.000023	com.fandom
249	18726412	1241	0.000025	net.seesaa
250	18723376	167	0.000158	com.imdb
251	18718440	333	0.000076	org.debian
252	18715552	699	0.000041	site.business
253	18713638	261	0.000093	com.live
254	18711498	1059	0.000031	jp.ne.goo
255	18710778	1504	0.000020	io.itch
256	18704016	1155	0.000027	org.greenpeace
257	18702254	1047	0.000031	com.netlify
258	18701568	1808	0.000017	net.pixnet
259	18700552	402	0.000064	com.squareup
260	18699852	1145	0.000028	co.elastic
261	18699756	310	0.000081	com.ibm
262	18698784	203	0.000118	com.stackoverflow
263	18698146	494	0.000052	com.indiatimes
264	18696312	3420	0.000010	com.armorgames
265	18693148	265	0.000092	com.aliyuncs
266	18692378	226	0.000106	com.optimizely
267	18685944	1777	0.000017	uk.co.timesonline
268	18679570	975	0.000033	com.mixcloud
269	18677530	1618	0.000019	com.itv
270	18674460	198	0.000128	org.bbb
271	18674340	57	0.000648	net.fbcdn
272	18674148	2236	0.000014	com.opendns
273	18672682	3302	0.000010	tw.com.gamer
274	18672646	406	0.000063	com.go
275	18672486	437	0.000059	com.msn
276	18672432	37	0.000966	com.wixstatic
277	18667606	1485	0.000021	org.archlinux
278	18666546	448	0.000058	org.pewresearch
279	18664590	87	0.000345	com.shopify
280	18662914	838	0.000039	jp.shinobi
281	18662550	995	0.000032	com.bmj
282	18660516	1579	0.000019	com.diigo
283	18659688	160	0.000167	com.opera
284	18659332	1822	0.000017	com.youdao
285	18657686	1259	0.000025	com.angelfire
286	18657376	906	0.000036	jp.naver
287	18656934	1038	0.000031	com.thelancet
288	18653610	1585	0.000019	uk.bl
289	18652186	864	0.000037	br.com.google
290	18649914	490	0.000053	com.bigcartel
291	18648796	1007	0.000032	com.sky
292	18647596	1408	0.000022	net.daringfireball
293	18642592	1779	0.000017	uk.ac.kcl
294	18641168	1752	0.000017	org.maven
295	18639544	534	0.000049	me.m
296	18639298	1107	0.000029	com.reverbnation
297	18637290	1665	0.000018	net.cnki
298	18634102	1089	0.000030	com.theconversation
299	18630504	460	0.000056	it.placehold
300	18629064	1117	0.000029	com.podbean
301	18628660	881	0.000036	org.fao
302	18626956	1144	0.000028	co.g
303	18626264	1370	0.000023	com.dw
304	18623646	119	0.000283	com.mailchimp
305	18621260	1631	0.000019	jp.ne.so-net
306	18620264	1346	0.000023	com.livescience
307	18620110	2071	0.000015	edu.kit
308	18611900	1308	0.000024	ca.utoronto
309	18610618	1310	0.000024	com.webnode
310	18608388	957	0.000034	au.gov.nsw
311	18606964	1411	0.000022	com.citrix
312	18603704	1031	0.000032	jp.jugem
313	18602374	1115	0.000029	gov.wa
314	18600002	458	0.000056	com.quora
315	18598840	99	0.000326	com.godaddy
316	18597740	1334	0.000023	com.bloglovin
317	18596834	1194	0.000026	com.serving-sys
318	18595976	900	0.000036	gov.dhs
319	18595120	1630	0.000019	org.edx
320	18592696	244	0.000099	me.wa
321	18592582	1830	0.000017	com.pearltrees
322	18592022	1576	0.000019	com.twitpic
323	18591744	1776	0.000017	cn.people
324	18590350	1171	0.000027	com.britannica
325	18589380	1893	0.000016	sg.edu.nus
326	18588130	1791	0.000017	com.kinja
327	18587404	2296	0.000013	com.authorstream
328	18585922	1566	0.000019	ca.mcgill
329	18585132	380	0.000068	com.kickstarter
330	18581636	1237	0.000025	com.lulu
331	18576948	2591	0.000012	com.colourlovers
332	18575064	1919	0.000016	com.hm
333	18574242	337	0.000075	com.rackcdn
334	18567994	2261	0.000013	uk.ac.sussex
335	18563954	1945	0.000016	org.vim
336	18562946	1063	0.000031	com.healthline
337	18562860	1911	0.000016	org.wikibooks
338	18562650	1867	0.000016	io.soup
339	18560052	1704	0.000018	nl.blogspot
340	18555740	509	0.000051	com.mashable
341	18553730	264	0.000093	com.typepad
342	18553486	1071	0.000031	com.adjust
343	18551456	396	0.000065	com.photobucket
344	18544528	1741	0.000017	org.bitcoin
345	18542952	2227	0.000014	tw.edu.ntu
346	18541072	1100	0.000030	com.ecwid
347	18540798	1820	0.000017	com.indianexpress
348	18540288	1889	0.000016	co.ello
349	18538870	575	0.000046	edu.berkeley
350	18537372	2026	0.000015	com.upi
351	18537264	132	0.000232	com.squarespace
352	18535682	280	0.000089	uk.org.ico
353	18534924	1136	0.000028	com.ssrn
354	18534382	2152	0.000014	com.viki
355	18533818	1219	0.000025	it.scoop
356	18532606	270	0.000092	com.surveymonkey
357	18532016	1601	0.000019	com.fastcodesign
358	18530620	1782	0.000017	org.unep
359	18529588	1057	0.000031	uk.parliament
360	18527356	1966	0.000016	org.haskell
361	18527140	224	0.000107	com.etsy
362	18527064	1442	0.000021	com.shutterfly
363	18525388	1569	0.000019	uk.org.tate
364	18524530	2862	0.000011	co.electrek
365	18523134	2693	0.000011	jp.doorblog
366	18522838	156	0.000171	com.issuu
367	18519148	2018	0.000015	com.dezeen
368	18517910	2430	0.000013	sh.now
369	18517530	1157	0.000027	com.tradedoubler
370	18515028	1173	0.000027	gov.weather
371	18513616	1109	0.000029	com.imageshack
372	18512682	1693	0.000018	com.channel4
373	18512134	1116	0.000029	gov.dot
374	18511018	2703	0.000011	cn.edu.sdu
375	18510554	1164	0.000027	com.wikia
376	18509244	282	0.000088	com.huffingtonpost
377	18509124	953	0.000034	uk.co.pinterest
378	18508676	924	0.000035	com.arstechnica
379	18507156	271	0.000091	com.rawgit
380	18505812	484	0.000053	tv.twitch
381	18505722	1917	0.000016	th.co.google
382	18503134	2390	0.000013	uk.ac.nhm
383	18502360	1764	0.000017	com.netvibes
384	18501156	1871	0.000016	edu.emory
385	18500964	918	0.000035	in.amazon
386	18500252	963	0.000034	com.strikingly
387	18499224	1773	0.000017	net.bplaced
388	18497786	3356	0.000010	tw.edu.ntnu
389	18495692	1811	0.000017	edu.iu
390	18494542	833	0.000039	com.brightcove
391	18491624	225	0.000107	com.hubspot
392	18491366	1470	0.000021	com.wattpad
393	18490914	1476	0.000021	gov.michigan
394	18489222	1916	0.000016	nl.tudelft
395	18488064	1436	0.000021	org.c-span
396	18487708	394	0.000065	com.meetup
397	18483412	2110	0.000014	com.kaggle
398	18481662	1299	0.000024	edu.brookings
399	18478490	86	0.000345	net.jsfiddle
400	18478420	2621	0.000012	sh.surge
401	18475530	2248	0.000014	com.rsa
402	18475220	1781	0.000017	gov.ahrq
403	18474128	825	0.000039	org.mediawiki
404	18473922	346	0.000072	edu.yale
405	18472664	826	0.000039	com.intel
406	18472288	1512	0.000020	gov.faa
407	18471926	1975	0.000015	io.material
408	18471732	1073	0.000031	com.thenextweb
409	18471706	1847	0.000016	net.earthlink
410	18469610	1908	0.000016	jp.blog
411	18469092	831	0.000039	com.pexels
412	18464760	1029	0.000032	uk.gov.nationalarchives
413	18460916	1973	0.000016	com.smashwords
414	18459088	939	0.000034	org.ieee
415	18457420	2185	0.000014	com.smore
416	18456724	345	0.000072	com.livejournal
417	18456360	3366	0.000010	hk.edu.hkbu
418	18453542	414	0.000063	com.nypost
419	18453464	1785	0.000017	com.business-standard
420	18453440	2907	0.000011	com.yam
421	18451248	1233	0.000025	org.aarp
422	18450408	1999	0.000015	com.oprah
423	18449942	2053	0.000015	org.jpn
424	18449880	1503	0.000020	org.amnesty
425	18449790	1651	0.000018	com.avvo
426	18449648	2540	0.000012	com.cleantechnica
427	18449504	522	0.000050	edu.cornell
428	18448124	2586	0.000012	com.mysanantonio
429	18447594	473	0.000054	io.shields
430	18447544	1444	0.000021	org.hrw
431	18444240	2487	0.000012	org.neocities
432	18442810	1943	0.000016	com.care2
433	18440702	1714	0.000018	com.snopes
434	18440194	1148	0.000027	com.gizmodo
435	18440058	2474	0.000012	com.googledrive
436	18439272	2594	0.000012	com.iflscience
437	18437524	1594	0.000019	org.pypi
438	18437266	268	0.000092	net.php
439	18436902	1927	0.000016	org.rsc
440	18436288	1389	0.000022	com.pbworks
441	18435978	2735	0.000011	com.itsnicethat
442	18435362	2092	0.000015	ae.thenational
443	18435326	2461	0.000012	com.hsbc
444	18434642	338	0.000074	com.hp
445	18432636	1354	0.000023	uk.co.standard
446	18431764	1807	0.000017	com.instapaper
447	18431596	465	0.000055	io.codepen
448	18431390	553	0.000047	com.buzzfeed
449	18431020	1910	0.000016	com.secondlife
450	18430258	2425	0.000013	jp.go.ndl
451	18429756	2106	0.000014	io.gitlab
452	18428432	373	0.000069	int.who
453	18427128	2300	0.000013	org.lds
454	18426976	2471	0.000012	uk.mod
455	18426954	1711	0.000018	google.ai
456	18426290	96	0.000330	de.google
457	18423898	1427	0.000021	com.thehindu
458	18423724	1749	0.000017	com.curbed
459	18422902	1638	0.000019	no.google
460	18421738	340	0.000074	com.cnbc
461	18420686	1061	0.000031	com.thedrum
462	18419780	165	0.000160	com.ebay
463	18418790	627	0.000043	com.zdnet
464	18418454	2330	0.000013	pl.cba
465	18416392	2441	0.000013	com.minds
466	18413488	201	0.000125	com.salesforce
467	18413252	1574	0.000019	com.moonfruit
468	18412358	1156	0.000027	com.mixpanel
469	18411670	2818	0.000011	tl.page
470	18409196	1982	0.000015	com.name
471	18409080	2282	0.000013	jp.hateblo
472	18407830	2507	0.000012	org.tvtropes
473	18407318	2580	0.000012	jp.hatenadiary
474	18406348	2510	0.000012	de.dw
475	18405634	1854	0.000016	com.googlegroups
476	18405508	1876	0.000016	mx.com.google
477	18405134	2198	0.000014	org.aiga
478	18403404	2883	0.000011	uk.co.birminghammail
479	18403340	367	0.000069	com.booking
480	18401602	2314	0.000013	vn.com.google
481	18401550	1729	0.000018	gov.pa
482	18399972	1666	0.000018	org.hrc
483	18399618	882	0.000036	gov.nist
484	18398472	2742	0.000011	com.exxonmobil
485	18397376	1841	0.000016	ar.com.google
486	18396356	989	0.000033	net.clickbank
487	18395660	976	0.000033	com.matterport
488	18392402	2429	0.000013	ua.at
489	18390522	2011	0.000015	uk.ac.leeds
490	18387444	309	0.000081	gov.cdc
491	18386528	1558	0.000019	int.unfccc
492	18386408	2342	0.000013	com.eklablog
493	18385700	459	0.000056	com.gmail
494	18385598	401	0.000064	org.npr
495	18384832	1672	0.000018	gov.maryland
496	18384390	357	0.000070	com.office
497	18383950	2240	0.000014	se.liu
498	18383810	2067	0.000015	com.discovermagazine
499	18383400	2204	0.000014	com.ipage
500	18381626	1110	0.000029	com.stackexchange
501	18381594	2418	0.000013	it.justpaste
502	18380974	449	0.000058	fr.free
503	18380682	1718	0.000018	sg.com.google
504	18379672	1060	0.000031	com.engadget
505	18378238	2421	0.000013	my.com.thestar
506	18377282	1273	0.000024	dk.google
507	18377136	2210	0.000014	org.biorxiv
508	18377062	1861	0.000016	com.weheartit
509	18374194	1598	0.000019	uk.gov.tfl
510	18371274	508	0.000051	gov.whitehouse
511	18369330	1723	0.000018	ly.snip
512	18369006	1809	0.000017	com.yourstory
513	18366356	3154	0.000011	com.bonanza
514	18365650	2833	0.000011	com.scienceblogs
515	18365484	1431	0.000021	com.ebayimg
516	18365436	1774	0.000017	gov.ky
517	18363500	858	0.000038	com.venturebeat
518	18362924	1160	0.000027	se.google
519	18362454	1350	0.000023	com.firebaseapp
520	18362026	178	0.000147	com.zendesk
521	18360150	2004	0.000015	uk.gov.metoffice
522	18359990	928	0.000035	com.windowsphone
523	18359750	2336	0.000013	com.rediff
524	18358388	518	0.000051	com.alibaba
525	18355256	2225	0.000014	com.blogfa
526	18355232	415	0.000063	com.fastcompany
527	18353212	1426	0.000021	com.surveygizmo
528	18352352	2021	0.000015	au.com.telstra
529	18351454	1134	0.000028	org.sphinx-doc
530	18350502	2048	0.000015	ro.google
531	18350126	1904	0.000016	org.tigris
532	18349524	2835	0.000011	be.lesoir
533	18349430	2698	0.000011	cz.centrum
534	18349372	2047	0.000015	link.page
535	18349260	479	0.000054	org.nodejs
536	18349028	1960	0.000016	com.marketwire
537	18347672	2242	0.000014	com.mystrikingly
538	18347018	2260	0.000013	ch.unige
539	18346850	2753	0.000011	cat.uab
540	18346818	2889	0.000011	com.zynga
541	18345164	1510	0.000020	us.mn.state
542	18341622	2275	0.000013	com.articulate
543	18340012	991	0.000033	edu.psu
544	18339422	2141	0.000014	com.thecvf
545	18339020	2150	0.000014	es.csic
546	18338922	2880	0.000011	co.carrd
547	18337380	1611	0.000019	gov.mo
548	18337360	2297	0.000013	com.newatlas
549	18335690	3908	0.000009	jp.rdy
550	18334630	1990	0.000015	org.iea
551	18333598	2565	0.000012	com.db
552	18332716	2310	0.000013	com.webstarts
553	18332584	2488	0.000012	jp.hatenablog
554	18331976	2331	0.000013	ly.rebrand
555	18331344	370	0.000069	com.mapbox
556	18331228	485	0.000053	com.livechatinc
557	18325352	1998	0.000015	org.mozillazine
558	18324944	2271	0.000013	de.uni-freiburg
559	18324472	1372	0.000023	com.tinypic
560	18324252	883	0.000036	com.steampowered
561	18323842	2072	0.000015	uk.ac.york
562	18322186	1097	0.000030	com.thinkwithgoogle
563	18320582	2589	0.000012	ru.msu
564	18320156	2458	0.000012	org.kotlinlang
565	18319540	1629	0.000019	gov.oregon
566	18318914	3507	0.000010	com.ingress
567	18318120	1806	0.000017	gov.wi
568	18318056	541	0.000049	com.aol
569	18318040	1969	0.000016	gr.google
570	18317824	2741	0.000011	lv.draugiem
571	18316720	2305	0.000013	org.iucnredlist
572	18315822	2035	0.000015	com.broadwayworld
573	18314076	134	0.000221	com.youtube-nocookie
574	18313954	1511	0.000020	net.openid
575	18313704	168	0.000158	com.tripadvisor
576	18313516	435	0.000059	com.dailymotion
577	18313398	1548	0.000019	net.leadpages
578	18313360	3389	0.000010	com.brother
579	18313186	2755	0.000011	com.webcindario
580	18313116	3161	0.000011	es.usal
581	18312962	2338	0.000013	bg.google
582	18312728	907	0.000036	com.xiti
583	18312338	2273	0.000013	us.oh.state
584	18310760	720	0.000041	fm.last
585	18310700	1662	0.000018	net.ucoz
586	18306936	353	0.000071	org.acm
587	18304074	3931	0.000009	com.worldlingo
588	18303254	3379	0.000010	com.embarcadero
589	18303110	1789	0.000017	com.eiseverywhere
590	18302838	2230	0.000014	org.wri
591	18302746	395	0.000065	com.pubmatic
592	18302516	470	0.000054	com.goodreads
593	18300826	2212	0.000014	com.thehindubusinessline
594	18300196	2289	0.000013	com.mihanblog
595	18299988	2037	0.000015	com.intensedebate
596	18298184	3230	0.000011	com.hellomagazine
597	18297298	3474	0.000010	net.hypermart
598	18294990	235	0.000103	uk.co.amazon
599	18294428	2711	0.000011	nf.co
600	18294164	73	0.000401	me.fb
601	18294088	542	0.000049	com.entrepreneur
602	18293062	2543	0.000012	com.futurelearn
603	18292644	2389	0.000013	com.iconarchive
604	18292306	1275	0.000024	com.cognitoforms
605	18292138	1730	0.000018	org.khanacademy
606	18291072	2041	0.000015	com.financialpost
607	18291000	1744	0.000017	us.pa.state
608	18288698	2653	0.000012	com.fatcow
609	18288564	372	0.000069	com.staticflickr
610	18288272	1851	0.000016	io.bower
611	18287878	3657	0.000010	nz.govt.tepapa
612	18285000	2199	0.000014	org.prlog
613	18284980	2743	0.000011	ca.shaw
614	18284736	2104	0.000014	com.bravesites
615	18283030	2675	0.000012	de.uni-erlangen
616	18282568	2446	0.000012	org.lacity
617	18282024	1534	0.000020	fi.google
618	18282018	2267	0.000013	de.uni-koeln
619	18280510	2464	0.000012	uk.co.spectator
620	18279326	334	0.000076	com.typeform
621	18279324	2725	0.000011	is.good
622	18279290	3399	0.000010	com.114la
623	18278842	3277	0.000010	net.freeforums
624	18278384	920	0.000035	com.zoho
625	18273898	2465	0.000012	uk.ac.jisc
626	18273404	2489	0.000012	com.mnn
627	18273352	2393	0.000013	ca.dal
628	18272740	114	0.000290	com.statcounter
629	18272722	480	0.000054	com.netflix
630	18272318	1567	0.000019	com.flashtalking
631	18272212	1903	0.000016	com.prweek
632	18270806	3115	0.000011	site.negocio
633	18270794	2080	0.000015	org.lung
634	18270506	2727	0.000011	com.mouser
635	18270340	2569	0.000012	uk.co.profilebusiness
636	18269840	3616	0.000010	uk.gov.number10
637	18268146	3806	0.000009	net.dead
638	18267734	3375	0.000010	jp.ac.kobe-u
639	18267628	1925	0.000016	uk.org.nice
640	18267512	88	0.000343	com.oculus
641	18267198	3196	0.000011	build.bazel
642	18266546	1878	0.000016	org.gentoo
643	18266166	2181	0.000014	ie.thejournal
644	18266148	109	0.000310	com.sharethis
645	18265976	1906	0.000016	org.gnupg
646	18264728	148	0.000186	ru.mail
647	18263376	1860	0.000016	com.doodlekit
648	18262238	1948	0.000016	com.crashlytics
649	18262156	1831	0.000017	org.alz
650	18261954	2549	0.000012	us.ms.state
651	18261162	2459	0.000012	com.instructure
652	18260540	820	0.000040	com.cbsnews
653	18259844	2877	0.000011	ee.ut
654	18259826	1211	0.000026	com.msdn
655	18259610	777	0.000040	com.samsung
656	18257004	1338	0.000023	com.emailmeform
657	18254934	549	0.000048	edu.cmu
658	18254822	2496	0.000012	uk.co.osoo
659	18254762	83	0.000354	com.livestream
660	18254656	2226	0.000014	com.atavist
661	18252876	2208	0.000014	fr.archives-ouvertes
662	18252282	2792	0.000011	com.cnsnews
663	18252018	2348	0.000013	io.pantheon
664	18251148	898	0.000036	com.createjs
665	18251026	1755	0.000017	us.fl.state
666	18250730	2321	0.000013	com.rabbitmq
667	18250628	2712	0.000011	uk.co.newmedianow
668	18248576	1422	0.000022	com.123formbuilder
669	18247032	2086	0.000015	gov.nh
670	18243504	2233	0.000014	org.crossref
671	18242314	2229	0.000014	us.nm.state
672	18242254	296	0.000084	com.scribd
673	18241366	3254	0.000010	ca.qc.montreal
674	18240908	3285	0.000010	uk.co.lrb
675	18240828	135	0.000215	com.youku
676	18239750	517	0.000051	com.slack
677	18239658	2677	0.000012	com.hatenadiary
678	18239656	2292	0.000013	com.itsmyurls
679	18237636	2671	0.000012	uk.org.oxonaa
680	18236902	246	0.000099	com.constantcontact
681	18236862	3348	0.000010	com.outlookindia
682	18235854	3893	0.000009	in.ac.nptel
683	18235540	2681	0.000012	uk.org.oxfam
684	18235236	2344	0.000013	com.yext
685	18233812	256	0.000094	com.getbootstrap
686	18233324	2107	0.000014	org.jenkins-ci
687	18230584	2055	0.000015	com.broadcastingcable
688	18230478	1686	0.000018	uk.gov.direct
689	18230416	2663	0.000012	com.wmtransfer
690	18230374	1977	0.000015	gov.mt
691	18230164	2821	0.000011	uk.ac.stir
692	18228540	1052	0.000031	com.marketwatch
693	18227744	2266	0.000013	com.tmcnet
694	18227440	3136	0.000011	uk.co.hsbc
695	18227086	1798	0.000017	org.nfpa
696	18226792	2939	0.000011	com.batchgeo
697	18225844	3275	0.000010	com.weightwatchers
698	18225636	234	0.000103	to.amzn
699	18224632	3574	0.000010	com.orgfree
700	18223778	1355	0.000023	org.whatbrowser
701	18221814	2843	0.000011	com.adn
702	18221276	1190	0.000026	org.weforum
703	18220506	481	0.000054	org.hbr
704	18219880	2820	0.000011	au.edu.deakin
705	18219734	1455	0.000021	org.js
706	18219118	2445	0.000013	in.ernet
707	18217962	2854	0.000011	hu.elte
708	18217516	3025	0.000011	pl.edu.uw
709	18217274	2367	0.000013	uk.org.rspb
710	18216528	2220	0.000014	com.healthgrades
711	18216264	2779	0.000011	org.carbonbrief
712	18214214	366	0.000069	com.prnewswire
713	18213956	2088	0.000015	com.tapatalk
714	18213180	2431	0.000013	org.grist
715	18212750	3423	0.000010	id.co.kaskus
716	18210638	456	0.000057	com.oreilly
717	18210106	3587	0.000010	com.skepticalscience
718	18209950	539	0.000049	gov.sec
719	18209922	3081	0.000011	com.deccanherald
720	18209668	1905	0.000016	tl.we
721	18208770	2311	0.000013	us.ma.state
722	18206860	1101	0.000030	uk.ac.cam
723	18205994	3630	0.000010	ua.meta
724	18205738	3526	0.000010	app.web
725	18204462	2398	0.000013	uk.co.zoopla
726	18201966	3210	0.000011	org.oceanconservancy
727	18199630	3421	0.000010	org.atsjournals
728	18198962	3532	0.000010	ru.my1
729	18198444	3162	0.000011	com.mozello
730	18195600	1562	0.000019	com.pastebin
731	18194580	2867	0.000011	de.freenet
732	18193414	1137	0.000028	edu.ucla
733	18193100	3052	0.000011	com.telegraphindia
734	18193026	2857	0.000011	com.chagasi
735	18192758	937	0.000034	br.com.uol
736	18188994	2630	0.000012	com.atwebpages
737	18188626	3036	0.000011	com.remind
738	18187922	1132	0.000028	com.redhat
739	18187748	608	0.000044	com.wikihow
740	18187658	3377	0.000010	edu.utep
741	18187264	3455	0.000010	ru.nnov
742	18186834	1881	0.000016	uk.gov.defra
743	18186568	2359	0.000013	net.portfoliobox
744	18185624	2610	0.000012	com.blogsky
745	18185434	3856	0.000009	uk.co.mailonsunday
746	18185432	2723	0.000011	jp.xxxxxxxx
747	18184122	1425	0.000021	edu.ucsd
748	18183962	1449	0.000021	com.digitaltrends
749	18183738	196	0.000130	jp.ne.hatena
750	18182464	2563	0.000012	uk.co.inews
751	18181728	2313	0.000013	gov.la
752	18181656	1266	0.000024	ly.ow
753	18180360	3441	0.000010	gr.sch
754	18179802	3055	0.000011	com.sc
755	18178628	3373	0.000010	com.cummins
756	18177566	2363	0.000013	com.activerain
757	18176026	3801	0.000009	com.kazeo
758	18176002	2901	0.000011	net.onlinewebshop
759	18175422	3689	0.000010	com.galvanize
760	18174902	3473	0.000010	ru.pr-cy
761	18174826	503	0.000052	com.dmca
762	18173528	3328	0.000010	com.kaywa
763	18173348	821	0.000040	com.psychologytoday
764	18172118	2853	0.000011	uk.co.heatall
765	18171416	84	0.000350	me.ogp
766	18168128	2601	0.000012	gov.ks
767	18167782	1516	0.000020	ca.blogspot
768	18167558	2170	0.000014	com.cityam
769	18167284	3604	0.000010	gov.cabq
770	18166436	1813	0.000017	org.reactjs
771	18166052	3283	0.000010	org.escardio
772	18165734	1064	0.000031	com.foxnews
773	18165680	1897	0.000016	com.fifa
774	18164860	204	0.000117	com.naver
775	18164404	3761	0.000009	com.carscoops
776	18162680	2928	0.000011	com.ecowatch
777	18162390	1507	0.000020	com.literatumonline
778	18161998	535	0.000049	net.2mdn
779	18161800	476	0.000054	com.force
780	18160578	159	0.000167	gov.privacyshield
781	18160270	1896	0.000016	com.pcworld
782	18160192	2986	0.000011	com.theyworkforyou
783	18159730	81	0.000365	com.messenger
784	18159700	3939	0.000009	com.anghami
785	18159426	424	0.000061	edu.nyu
786	18157990	1294	0.000024	com.indiegogo
787	18157828	1869	0.000016	kr.or.kisa
788	18157816	364	0.000070	com.discordapp
789	18157014	3186	0.000011	uk.org.38degrees
790	18156850	3628	0.000010	com.insideevs
791	18155496	1488	0.000020	com.placeholder
792	18155072	3250	0.000010	google.design
793	18155044	3764	0.000009	gle.goo
794	18154462	454	0.000057	com.walmart
795	18153360	428	0.000060	com.flipboard
796	18152044	2902	0.000011	pl.lublin
797	18151952	422	0.000062	com.wufoo
798	18151198	1123	0.000029	com.shutterstock
799	18150684	2537	0.000012	org.iihs
800	18149446	2788	0.000011	in.businessworld
801	18148636	981	0.000033	com.pinimg
802	18147760	2407	0.000013	jp.e-shops
803	18147734	2250	0.000014	com.codecademy
804	18146340	2642	0.000012	com.zx2c4
805	18146328	129	0.000243	info.aboutads
806	18145944	2138	0.000014	ca.ubc
807	18145538	2874	0.000011	com.bnef
808	18144354	3240	0.000011	uk.ac.rcplondon
809	18144254	3718	0.000009	com.wsoctv
810	18143902	3950	0.000009	com.monbiot
811	18143342	3463	0.000010	com.droppages
812	18143148	2366	0.000013	gov.arts
813	18142454	2644	0.000012	us.wi.state
814	18142046	3477	0.000010	org.usatf
815	18140878	1624	0.000019	com.nvidia
816	18138866	3636	0.000010	com.elmercurio
817	18138838	1538	0.000020	com.businessweek
818	18138462	2176	0.000014	com.tutsplus
819	18138382	554	0.000047	com.atlassian
820	18137356	1184	0.000026	com.searchengineland
821	18137278	3594	0.000010	com.glu
822	18137124	3645	0.000010	es.consumer
823	18135974	240	0.000102	cn.com.sina
824	18135596	3948	0.000009	com.allmyfaves
825	18135342	3446	0.000010	com.businessgreen
826	18133642	350	0.000072	com.163
827	18133268	3292	0.000010	org.jython
828	18133230	471	0.000054	com.smugmug
829	18132816	3864	0.000009	org.thechicagocouncil
830	18132126	3576	0.000010	gov.azdot
831	18130470	1176	0.000027	com.ycombinator
832	18129838	3339	0.000010	org.transportenvironment
833	18128538	2993	0.000011	gov.ferc
834	18127910	936	0.000034	com.aliexpress
835	18126154	356	0.000070	com.wiley
836	18125790	696	0.000042	com.moz
837	18124996	2756	0.000011	uk.gov.environment-agency
838	18124886	3012	0.000011	org.zsl
839	18124136	3704	0.000009	org.ssireview
840	18123520	2378	0.000013	uk.gov.scotland
841	18122978	1595	0.000019	tv.ustream
842	18122522	3096	0.000011	org.dailystrength
843	18122038	598	0.000045	com.caniuse
844	18120996	2485	0.000012	net.privacypolicytemplate
845	18120866	768	0.000040	gov.noaa
846	18120818	1573	0.000019	jp.makeshop
847	18120518	3040	0.000011	org.rspo
848	18119946	2303	0.000013	com.seetickets
849	18119454	2183	0.000014	com.ign
850	18118896	404	0.000064	mp.mailchi
851	18118000	311	0.000081	com.digg
852	18118000	2855	0.000011	gov.txdot
853	18117366	3412	0.000010	uk.ac.ceh
854	18117164	1479	0.000021	com.crunchbase
855	18117074	1127	0.000029	com.highcharts
856	18115870	2645	0.000012	com.9to5mac
857	18114648	1090	0.000030	com.withgoogle
858	18114314	889	0.000036	com.webs
859	18114072	2481	0.000012	uk.co.streetmap
860	18112508	3865	0.000009	com.pushwoosh
861	18111708	3204	0.000011	ca.uwaterloo
862	18111130	817	0.000040	com.shinystat
863	18111078	305	0.000082	fr.google
864	18111050	3467	0.000010	com.baomoi
865	18110974	3916	0.000009	uk.ac.tyndall
866	18110396	1766	0.000017	com.webmasterplan
867	18110180	3686	0.000010	dk.bloggersdelight
868	18109908	3401	0.000010	uk.gov.hm-treasury
869	18109262	1793	0.000017	uk.org.cqc
870	18108948	1248	0.000025	com.smashingmagazine
871	18108138	331	0.000076	com.automattic
872	18107572	1530	0.000020	com.ning
873	18106984	2829	0.000011	com.linkwithin
874	18106522	3002	0.000011	uk.org.greenpeace
875	18103768	956	0.000034	com.libsyn
876	18103538	1239	0.000025	com.sap
877	18102956	2091	0.000015	edu.uci
878	18102564	628	0.000043	com.patreon
879	18102020	3503	0.000010	com.climatechangenews
880	18101758	409	0.000063	com.xinhuanet
881	18101336	3464	0.000010	com.kapook
882	18100618	885	0.000036	com.newyorker
883	18100474	3640	0.000010	com.spruz
884	18100196	478	0.000054	com.inc
885	18100062	2676	0.000012	jp.aikotoba
886	18099268	914	0.000035	org.eff
887	18098794	3662	0.000010	com.platts
888	18098556	3535	0.000010	org.c2es
889	18098550	2747	0.000011	com.mykaratestore
890	18096786	1770	0.000017	com.ikea
891	18096394	1423	0.000022	com.billboard
892	18095092	1070	0.000031	com.hootsuite
893	18094948	3525	0.000010	com.jkp
894	18093496	2824	0.000011	org.mcsuk
895	18092622	1254	0.000025	es.agpd
896	18092438	3349	0.000010	net.edie
897	18092358	533	0.000050	com.ea
898	18092112	376	0.000068	org.opensource
899	18091568	2903	0.000011	ru.drom
900	18090162	2639	0.000012	com.yelloyello
901	18089968	2544	0.000012	uk.co.intersol
902	18089740	139	0.000202	com.alicdn
903	18089422	4051	0.000009	com.mforos
904	18086990	1473	0.000021	com.fiverr
905	18086352	934	0.000035	com.foursquare
906	18085918	1737	0.000017	org.freecsstemplates
907	18084810	4142	0.000009	uk.org.indymedia
908	18084674	2049	0.000015	uk.gov.education
909	18083694	3843	0.000009	com.thinkbroadband
910	18082164	231	0.000104	jp.co.amazon
911	18080114	3868	0.000009	org.sciencenewsforstudents
912	18080034	221	0.000108	org.drupal
913	18079726	1096	0.000030	com.variety
914	18078666	290	0.000086	com.stumbleupon
915	18078038	3269	0.000010	net.scienceontheweb
916	18077582	1756	0.000017	com.nba
917	18077452	2561	0.000012	org.webring
918	18076502	1033	0.000031	com.visualstudio
919	18075958	4005	0.000009	io.raindrop
920	18074544	2744	0.000011	jp.zouri
921	18073766	3904	0.000009	org.corporateeurope
922	18072470	1402	0.000022	com.storify
923	18071436	375	0.000069	gov.ftc
924	18071372	1603	0.000019	net.with2
925	18070926	1448	0.000021	com.nike
926	18070222	4048	0.000009	io.dataquest
927	18070066	1255	0.000025	org.unicef
928	18069672	3567	0.000010	bnpparibas.group
929	18069172	3685	0.000010	com.thestatesman
930	18068866	3427	0.000010	uk.org.rya
931	18067508	383	0.000068	com.airbnb
932	18067204	1635	0.000019	de.zeit
933	18067190	2555	0.000012	com.hackernoon
934	18066274	3451	0.000010	ca.pe.gov
935	18065266	4031	0.000009	com.raamdev
936	18064388	2438	0.000013	io.postach
937	18064126	1487	0.000020	edu.purdue
938	18063508	407	0.000063	com.tripod
939	18063228	1228	0.000025	gov.fbi
940	18063154	1369	0.000023	com.lifehacker
941	18063130	1069	0.000031	com.uk
942	18061878	3432	0.000010	in.gov.mhrd
943	18061130	3527	0.000010	org.gmplib
944	18060100	3879	0.000009	com.gitimmersion
945	18059578	2807	0.000011	jp.at-ninja
946	18059004	3010	0.000011	com.shichihuku
947	18058826	3629	0.000010	com.h2database
948	18057736	3482	0.000010	uk.org.rcn
949	18057640	3737	0.000009	com.writetothem
950	18056592	1366	0.000023	com.parsiblog
951	18056586	984	0.000033	com.dropboxusercontent
952	18055950	1306	0.000024	com.prweb
953	18055628	3695	0.000009	com.websiteseguro
954	18055104	1118	0.000029	com.vox
955	18054272	1397	0.000022	us.imageshack
956	18053964	2032	0.000015	com.howstuffworks
957	18052920	1531	0.000020	com.yoast
958	18052280	1298	0.000024	com.pcmag
959	18051398	3008	0.000011	uk.org.woodlandtrust
960	18050936	3523	0.000010	gle.posts
961	18050800	3838	0.000009	org.priceofoil
962	18049580	1614	0.000019	com.ccbill
963	18049066	3750	0.000009	com.fourfour
964	18047214	945	0.000034	gov.census
965	18046486	1328	0.000023	edu.wisc
966	18045876	151	0.000179	jp.co.google
967	18045710	1220	0.000025	com.blackberry
968	18045414	1103	0.000030	edu.umich
969	18045390	1952	0.000016	com.w3layouts
970	18043894	146	0.000190	me.line
971	18043816	1593	0.000019	edu.usc
972	18042356	2842	0.000011	com.zatunen
973	18042240	500	0.000052	com.nasdaq
974	18042130	567	0.000046	net.daum
975	18041570	3118	0.000011	vn.tuoitre
976	18040556	2573	0.000012	com.hisupplier
977	18039444	2023	0.000015	com.nfl
978	18039370	927	0.000035	com.ggpht
979	18039324	1549	0.000019	com.vmware
980	18039020	3827	0.000009	com.realtytimes
981	18038362	3261	0.000010	net.batcave
982	18038116	3341	0.000010	org.mygamesonline
983	18037866	734	0.000040	com.mckinsey
984	18037674	3983	0.000009	org.eia-international
985	18037604	258	0.000094	com.sohu
986	18037594	3700	0.000009	io.dropwizard
987	18037394	1026	0.000032	gov.nps
988	18037244	2131	0.000014	au.com.news
989	18036608	3652	0.000010	de.epubli
990	18034198	1381	0.000022	com.unity3d
991	18034072	2992	0.000011	net.nend
992	18033048	4098	0.000009	com.easyhits4u
993	18031890	1162	0.000027	com.steamcommunity
994	18031622	1451	0.000021	edu.uchicago
995	18031570	1086	0.000030	com.uber
996	18031470	5306	0.000007	com.plurk
997	18030490	597	0.000045	com.adweek
998	18030182	3635	0.000010	com.jal
999	18029670	1786	0.000017	com.techradar
1000	18029244	1271	0.000024	com.ifttt

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

January 2020 crawl archive now available

February 3, 2020Sebastian Nagel

The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th. It includes page captures of 960 million URLs not contained in any crawl archive before.

Improvements and Fixes

date time values in the column "fetch_time" of the columnar index are now stored using the "int64" data type. For details and compatibility issues please see cc-index-table#7
WARC request records now show the HTTP protocol version sent with the HTTP request which can be different from the version received in the HTTP response message, cf. NUTCH-2760

Archive Location and Download

The January crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-05/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-05/segment.paths.gz	100
WARC files	CC-MAIN-2020-05/warc.paths.gz	56000	59.94
WAT files	CC-MAIN-2020-05/wat.paths.gz	56000	22.3
WET files	CC-MAIN-2020-05/wet.paths.gz	56000	10
Robots.txt files	CC-MAIN-2020-05/robotstxt.paths.gz	56000	0.25
Non-200 responses files	CC-MAIN-2020-05/non200responses.paths.gz	56000	2.28
URL index files	CC-MAIN-2020-05/cc-index.paths.gz	302	0.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-05/. Also the columnar index has been updated to contain this crawl.

December 2019 crawl archive now available

December 19, 2019Sebastian Nagel

The crawl archive for December 2019 is now available! It contains 2.45 billion web pages or 234 TiB of uncompressed content, crawled between December 5th and 16th. It includes page captures of 850 million URLs not contained in any crawl archive before.

Archive Location and Download

The December crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-51/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-51/segment.paths.gz	100
WARC files	CC-MAIN-2019-51/warc.paths.gz	56000	47.47
WAT files	CC-MAIN-2019-51/wat.paths.gz	56000	17.6
WET files	CC-MAIN-2019-51/wet.paths.gz	56000	8.06
Robots.txt files	CC-MAIN-2019-51/robotstxt.paths.gz	56000	0.26
Non-200 responses files	CC-MAIN-2019-51/non200responses.paths.gz	56000	3.5
URL index files	CC-MAIN-2019-51/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-51/. Also the columnar index has been updated to contain this crawl.

November 2019 crawl archive now available

November 27, 2019Sebastian Nagel

The crawl archive for November 2019 is now available! It contains 2.55 billion web pages or 250 TiB of uncompressed content, crawled between November 11th and 23rd with a short operational break on Nov 16th. It includes page captures of 1.1 billion URLs not contained in any crawl archive before.

What’s new?

We’ve added two new fields to the URL indexes (CDX and columnar):

the redirect target location is stored in the CDX JSON field "redirect" resp. the column "fetch_redirect". The value is extracted from HTTP header field "Location" if the HTTP status code indicates a HTTP redirect. A relative URL path is converted to an absolute URL using the page URL as base URL. The key is absent (resp. the field value is null) in case the "Location" value is missing, not a valid URL or not a valid relative URL path.
truncation of the WARC record payload is indicated by the key "truncated" resp. the column "content_truncated". The reason for the truncation is given only for truncated records following the WARC header field "WARC-Truncated".

Additional details and examples can be found in the corresponding PR #15.

We’ve fixed a bug affecting the capture time (WARC-Date) in the the robots.txt subset which has been extracted from the HTTP "Date" field of the HTTP header and appeared to be occasionally wrong. Please see issue #14 for further details.

Archive Location and Download

The November crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-47/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-47/segment.paths.gz	100
WARC files	CC-MAIN-2019-47/warc.paths.gz	56000	53.95
WAT files	CC-MAIN-2019-47/wat.paths.gz	56000	18.50
WET files	CC-MAIN-2019-47/wet.paths.gz	56000	8.34
Robots.txt files	CC-MAIN-2019-47/robotstxt.paths.gz	56000	0.24
Non-200 responses files	CC-MAIN-2019-47/non200responses.paths.gz	56000	3.05
URL index files	CC-MAIN-2019-47/cc-index.paths.gz	302	0.20

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-47/. Also the columnar index has been updated to contain this crawl.