Blog – Common Crawl

April 2021 crawl archive now available

April 27, 2021Sebastian Nagel

The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content. It includes page captures of 1.35 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The April crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-17/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line, you end up with the S3 and HTTP paths respectively.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-17/segment.paths.gz	100
WARC files	CC-MAIN-2021-17/warc.paths.gz	64000	69.78
WAT files	CC-MAIN-2021-17/wat.paths.gz	64000	21.05
WET files	CC-MAIN-2021-17/wet.paths.gz	64000	9.25
Robots.txt files	CC-MAIN-2021-17/robotstxt.paths.gz	64000	0.16
Non-200 responses files	CC-MAIN-2021-17/non200responses.paths.gz	64000	1.76
URL index files	CC-MAIN-2021-17/cc-index.paths.gz	302	0.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-17/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

February/March 2021 crawl archive now available

March 14, 2021Sebastian Nagel

The crawl archive for February/March 2021 is now available! The data was crawled between February 24th and March 9th and contains 2.7 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.2 billion new URLs, not visited in any of our prior crawls.

Improvements and Fixes

The ISO639-3 code for the Hmong language was updated to "hmn" – the code "blu" used so far was already deprecated in 2008. Crawl archives prior to this crawl will still use the code "blu". More details about this update are found here.

Archive Location and Download

The February/March crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-10/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-10/segment.paths.gz	100
WARC files	CC-MAIN-2021-10/warc.paths.gz	64000	62.51
WAT files	CC-MAIN-2021-10/wat.paths.gz	64000	18.44
WET files	CC-MAIN-2021-10/wet.paths.gz	64000	8.06
Robots.txt files	CC-MAIN-2021-10/robotstxt.paths.gz	64000	0.2
Non-200 responses files	CC-MAIN-2021-10/non200responses.paths.gz	64000	1.58
URL index files	CC-MAIN-2021-10/cc-index.paths.gz	302	0.21

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-10/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs October, November/December 2020 and January 2021

February 10, 2021Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of October, November/December 2020 and January 2021. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of webgraph notebooks.

Host-level graph

The graph consists of 490 million nodes and 2.57 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 414 million dangling nodes (84.4%) and the largest strongly connected component contains 42.6 million (8.7%) nodes.

Host names in the graph are in reverse domain name notation and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 490 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2020-21-oct-nov-jan/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2020-21-oct-nov-jan/host/ as prefix to access the files from everywhere.

Please note that the text representation of the host-level graph is shipped in 36 gzip-compressed files listed in two path listings – one for the nodes, one for the edges. First, download the paths listing and uncompress it using “gzip”. By adding the prefix s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line in the path listing you get the list of URLs to download the entire graph.

Download files of the Common Crawl Oct/Nov/Jan 2020-2021 host-level webgraph

Size	File	Description
3.08 GB	cc-main-2020-21-oct-nov-jan-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 12 vertices files
11.76 GB	cc-main-2020-21-oct-nov-jan-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 24 edges files

5.18 GB	cc-main-2020-21-oct-nov-jan-host.graph	graph in BVGraph format
2 kB	cc-main-2020-21-oct-nov-jan-host.properties

5.63 GB	cc-main-2020-21-oct-nov-jan-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2020-21-oct-nov-jan-host-t.properties

1 kB	cc-main-2020-21-oct-nov-jan-host.stats	WebGraph statistics

7.04 GB	cc-main-2020-21-oct-nov-jan-host-ranks.txt.gz	harmonic centrality and pagerank

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 86 million nodes and 1.47 billion edges. 50% or 43 million nodes are dangling nodes, the largest strongly connected component covers 34 million or 39% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2020-21-oct-nov-jan/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2020-21-oct-nov-jan/domain/.

Download files of the Common Crawl Oct/Nov/Jan 2020-2021 domain-level webgraph

Size	File	Description
0.59 GB	cc-main-2020-21-oct-nov-jan-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
6.00 GB	cc-main-2020-21-oct-nov-jan-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.40 GB	cc-main-2020-21-oct-nov-jan-domain.graph	graph in BVGraph format
2 kB	cc-main-2020-21-oct-nov-jan-domain.properties

3.26 GB	cc-main-2020-21-oct-nov-jan-domain-t.graph	transpose of the graph
2 kB	cc-main-2020-21-oct-nov-jan-domain-t.properties

1 kB	cc-main-2020-21-oct-nov-jan-domain.stats	WebGraph statistics

1.85 GB	cc-main-2020-21-oct-nov-jan-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 86 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Oct/Nov/Jan 2020-2021)

harmonic centrality rank	hc value	page rank	page rank value	reversed domain name
1	30355566	1	0.017956	com.googleapis
2	29427164	3	0.012871	com.facebook
3	28173562	2	0.012899	com.google
4	25702812	5	0.007348	com.twitter
5	25628314	4	0.007628	org.w
6	25297808	6	0.007231	com.youtube
7	24195466	9	0.005352	com.instagram
8	23355356	8	0.005532	org.gmpg
9	23233674	7	0.006500	com.googletagmanager
10	22492432	11	0.003277	com.linkedin
11	21576402	10	0.004076	com.cloudflare
12	21468510	14	0.002649	com.gravatar
13	21395642	13	0.003020	org.wordpress
14	21353798	22	0.001726	com.pinterest
15	20946722	30	0.001242	org.wikipedia
16	20926308	19	0.001834	com.wordpress
17	20877776	16	0.002056	com.gstatic
18	20799472	15	0.002451	com.bootstrapcdn
19	20795402	18	0.001943	com.apple
20	20626472	32	0.001165	com.vimeo
21	20527986	41	0.000886	be.youtu
22	20419038	21	0.001769	com.jquery
23	20391686	28	0.001246	com.microsoft
24	20327544	24	0.001500	com.wp
25	20314602	45	0.000769	com.blogspot
26	20231490	37	0.001025	com.amazonaws
27	20208912	51	0.000691	com.amazon
28	20199388	47	0.000740	gl.goo
29	20093688	71	0.000448	com.tumblr
30	20070176	35	0.001070	com.google-analytics
31	20050256	61	0.000598	ly.bit
32	20030452	20	0.001794	com.adobe
33	19998314	17	0.002005	com.github
34	19989010	50	0.000715	org.mozilla
35	19962834	58	0.000639	eu.europa
36	19945306	34	0.001103	net.cloudfront
37	19849112	52	0.000682	com.flickr
38	19843288	40	0.000909	net.jsdelivr
39	19833032	91	0.000369	com.googleusercontent
40	19823560	105	0.000347	com.yahoo
41	19752300	56	0.000650	co.t
42	19722088	33	0.001114	com.googlesyndication
43	19712406	23	0.001517	com.fontawesome
44	19708354	81	0.000392	com.weebly
45	19706054	55	0.000653	com.paypal
46	19695288	109	0.000308	com.reddit
47	19641534	31	0.001231	me.wp
48	19640398	73	0.000435	com.medium
49	19635162	67	0.000491	io.github
50	19590444	137	0.000225	com.nytimes
51	19587880	121	0.000280	com.soundcloud
52	19585192	27	0.001262	ru.yandex
53	19583494	43	0.000786	com.addthis
54	19582250	44	0.000776	com.macromedia
55	19560416	66	0.000504	org.w3
56	19549714	70	0.000451	com.shopify
57	19518672	146	0.000201	com.forbes
58	19502448	144	0.000205	org.archive
59	19496300	90	0.000371	org.creativecommons
60	19490348	194	0.000131	uk.co.bbc
61	19482926	59	0.000630	org.schema
62	19479528	39	0.000910	com.baidu
63	19464572	36	0.001035	net.doubleclick
64	19459966	200	0.000129	com.cnn
65	19451100	53	0.000677	com.whatsapp
66	19449068	60	0.000611	com.vk
67	19444966	206	0.000126	net.slideshare
68	19443956	158	0.000169	com.bing
69	19419878	174	0.000152	com.imdb
70	19385956	186	0.000140	com.imgur
71	19372520	236	0.000112	com.washingtonpost
72	19371076	176	0.000150	com.theguardian
73	19356952	254	0.000102	com.wsj
74	19356474	210	0.000123	org.wikimedia
75	19352128	219	0.000117	com.businessinsider
76	19347698	209	0.000123	com.stackoverflow
77	19342712	409	0.000065	com.msn
78	19326654	327	0.000079	com.appspot
79	19324334	157	0.000172	int.who
80	19321112	216	0.000119	edu.stanford
81	19316796	179	0.000148	org.apache
82	19310390	333	0.000078	com.ibm
83	19309354	337	0.000077	edu.mit
84	19304938	225	0.000116	net.sourceforge
85	19292932	116	0.000288	com.ytimg
86	19287812	57	0.000649	net.fbcdn
87	19282486	285	0.000091	com.techcrunch
88	19276500	269	0.000094	com.bbc
89	19275480	155	0.000181	com.wixsite
90	19275222	152	0.000189	gov.nih
91	19275200	220	0.000117	com.livejournal
92	19270650	233	0.000113	uk.co.google
93	19270610	440	0.000062	gov.nasa
94	19263354	54	0.000666	com.googleadservices
95	19243404	262	0.000097	edu.harvard
96	19243154	270	0.000094	com.oracle
97	19243126	276	0.000093	org.acm
98	19238650	218	0.000117	org.ietf
99	19238450	185	0.000142	com.blogger
100	19238426	223	0.000116	gov.ca
101	19234630	465	0.000059	fr.free
102	19232058	259	0.000098	com.bloomberg
103	19221844	275	0.000093	com.android
104	19218636	304	0.000085	com.live
105	19210812	126	0.000271	com.jimdo
106	19208896	169	0.000159	com.issuu
107	19205802	166	0.000162	com.giphy
108	19194156	438	0.000062	com.ted
109	19190178	348	0.000075	com.huffingtonpost
110	19187782	130	0.000254	com.weibo
111	19186862	154	0.000186	us.zoom
112	19185794	252	0.000103	org.gnu
113	19176332	403	0.000066	com.myspace
114	19162122	1039	0.000030	com.wikia
115	19152582	373	0.000071	net.researchgate
116	19150058	343	0.000075	com.usatoday
117	19148332	309	0.000084	com.reuters
118	19143988	400	0.000067	uk.co.telegraph
119	19141202	446	0.000061	com.latimes
120	19130976	372	0.000071	com.example
121	19129552	345	0.000075	com.githubusercontent
122	19127344	93	0.000366	com.unpkg
123	19127116	384	0.000069	com.nature
124	19125396	336	0.000077	com.wired
125	19124320	25	0.001485	com.wixstatic
126	19114842	299	0.000087	org.npr
127	19111018	308	0.000084	com.cnbc
128	19107772	328	0.000079	com.ebay
129	19103704	293	0.000088	com.wiley
130	19102814	111	0.000299	de.google
131	19097732	191	0.000135	com.npmjs
132	19095454	344	0.000075	com.hp
133	19088550	539	0.000050	com.cisco
134	19084048	932	0.000034	com.stackexchange
135	19081736	132	0.000251	com.youtube-nocookie
136	19080638	134	0.000250	com.ft
137	19078814	213	0.000120	org.ampproject
138	19077232	532	0.000051	com.steampowered
139	19074638	365	0.000072	com.patreon
140	19072918	455	0.000061	com.theatlantic
141	19072880	476	0.000057	com.gitlab
142	19072344	890	0.000035	com.pcmag
143	19068436	195	0.000131	com.unsplash
144	19065494	877	0.000036	edu.psu
145	19063926	376	0.000070	com.time
146	19061142	208	0.000125	com.twimg
147	19061064	164	0.000165	com.yelp
148	19059332	873	0.000036	edu.washington
149	19057196	533	0.000051	edu.cornell
150	19054152	148	0.000197	com.dropbox
151	19051738	603	0.000046	org.arxiv
152	19047626	379	0.000070	com.statista
153	19043050	324	0.000080	org.un
154	19042602	249	0.000104	com.bandcamp
155	19040914	824	0.000038	com.venturebeat
156	19040684	75	0.000432	me.fb
157	19039882	841	0.000037	org.chromium
158	19033464	65	0.000519	com.wix
159	19026244	284	0.000092	com.sciencedirect
160	19019766	629	0.000045	edu.yale
161	19016326	584	0.000047	com.pexels
162	19015230	826	0.000038	org.bitbucket
163	19010452	832	0.000038	org.ieee
164	19007636	388	0.000068	com.springer
165	19001810	765	0.000041	com.evernote
166	18997506	855	0.000037	edu.upenn
167	18994926	258	0.000098	jp.ameblo
168	18993772	149	0.000195	me.t
169	18992834	416	0.000065	org.hbr
170	18992028	296	0.000088	com.outlook
171	18985954	168	0.000160	jp.co.yahoo
172	18983238	577	0.000048	com.cbsnews
173	18982546	792	0.000040	me.about
174	18981228	891	0.000035	com.git-scm
175	18980336	829	0.000038	com.economist
176	18980328	150	0.000193	com.opera
177	18978056	138	0.000223	me.line
178	18974996	450	0.000061	com.goodreads
179	18973364	645	0.000044	com.mysql
180	18973114	842	0.000037	com.docker
181	18969708	562	0.000048	com.buzzfeed
182	18969566	565	0.000048	com.mashable
183	18968398	587	0.000047	com.mozilla
184	18964540	951	0.000034	com.about
185	18962632	797	0.000040	org.worldbank
186	18956128	815	0.000039	com.newyorker
187	18954668	342	0.000076	com.dribbble
188	18954236	265	0.000096	net.behance
189	18951876	390	0.000068	com.theverge
190	18951838	501	0.000054	gov.whitehouse
191	18950142	456	0.000061	uk.co.dailymail
192	18943890	347	0.000075	com.xinhuanet
193	18942812	320	0.000080	com.w3schools
194	18941124	378	0.000070	com.fc2
195	18936488	1151	0.000027	edu.wisc
196	18935074	764	0.000041	gov.noaa
197	18932396	294	0.000088	com.disqus
198	18931228	1337	0.000023	co.elastic
199	18927646	38	0.000956	com.qq
200	18926694	448	0.000061	com.bigcommerce
201	18926442	624	0.000045	gov.loc
202	18925620	156	0.000179	gov.cdc
203	18924632	929	0.000035	gov.fcc
204	18922816	136	0.000228	info.aboutads
205	18921630	821	0.000039	com.qz
206	18921308	2295	0.000015	com.wikidot
207	18919240	385	0.000069	com.scribd
208	18915104	748	0.000042	org.unesco
209	18914418	959	0.000033	com.apnews
210	18912426	375	0.000070	com.digg
211	18911082	779	0.000040	com.vox
212	18910370	180	0.000147	com.amazon-adsystem
213	18910110	272	0.000094	com.squareup
214	18907410	495	0.000054	uk.co.independent
215	18906224	256	0.000100	org.iana
216	18905608	1251	0.000025	edu.uchicago
217	18901398	420	0.000064	com.force
218	18898702	646	0.000044	com.usnews
219	18898108	647	0.000044	com.gartner
220	18894918	295	0.000088	com.nbcnews
221	18890160	470	0.000058	com.dailymotion
222	18883488	1004	0.000031	com.dropboxusercontent
223	18878276	617	0.000045	org.pbs
224	18876454	181	0.000147	jp.co.google
225	18876164	113	0.000292	com.sharethis
226	18875824	467	0.000059	com.nationalgeographic
227	18874112	811	0.000039	uk.co.blogspot
228	18873340	844	0.000037	au.net.abc
229	18868000	934	0.000034	com.foxnews
230	18865322	1559	0.000020	org.eclipse
231	18859464	399	0.000067	com.getpocket
232	18859228	947	0.000034	com.slate
233	18859062	266	0.000095	org.doi
234	18858866	63	0.000541	com.fb
235	18856638	968	0.000033	com.politico
236	18849992	907	0.000035	com.playstation
237	18849334	600	0.000046	org.semver
238	18848468	1565	0.000020	gd.is
239	18847004	1311	0.000024	edu.unc
240	18846758	1523	0.000021	org.kernel
241	18846310	839	0.000037	org.sciencemag
242	18846038	257	0.000099	com.typepad
243	18844998	1152	0.000027	com.hatenablog
244	18844004	1981	0.000018	com.googlesource
245	18842180	202	0.000128	com.naver
246	18840548	248	0.000104	com.feedburner
247	18839830	1028	0.000030	edu.umn
248	18837518	421	0.000064	com.ecwid
249	18833048	332	0.000078	net.windows
250	18831042	914	0.000035	com.trello
251	18829176	554	0.000049	com.tandfonline
252	18829172	1369	0.000023	cn.com.chinadaily
253	18828382	189	0.000138	org.allaboutcookies
254	18825844	746	0.000042	gov.senate
255	18823946	119	0.000286	com.paypalobjects
256	18819980	1005	0.000031	ly.ow
257	18818724	2014	0.000017	org.tensorflow
258	18818710	901	0.000035	edu.umich
259	18817936	291	0.000089	com.tinyurl
260	18817212	479	0.000056	org.pewresearch
261	18815000	76	0.000423	com.list-manage
262	18811132	239	0.000111	com.wpengine
263	18806908	834	0.000038	ca.cbc
264	18805144	740	0.000043	co.ibb
265	18804044	477	0.000057	gov.fda
266	18802934	222	0.000117	com.eepurl
267	18802462	318	0.000081	it.google
268	18798744	79	0.000413	net.facebook
269	18797046	2019	0.000017	com.instructables
270	18795562	1200	0.000026	edu.northwestern
271	18794710	752	0.000042	org.change
272	18793610	394	0.000068	es.google
273	18793484	893	0.000035	org.cambridge
274	18790202	251	0.000103	com.calendly
275	18784862	962	0.000033	gov.congress
276	18784862	1022	0.000030	uk.co.guardian
277	18782014	555	0.000049	com.bigcartel
278	18777808	1348	0.000023	org.semanticscholar
279	18776340	1006	0.000031	com.gumroad
280	18775690	637	0.000044	org.plos
281	18774956	1341	0.000023	com.nikkei
282	18773712	313	0.000083	com.optimizely
283	18772988	405	0.000066	com.googlecode
284	18766674	896	0.000035	gov.justice
285	18764788	1044	0.000029	com.huffpost
286	18764312	153	0.000186	com.addtoany
287	18763408	398	0.000067	me.m
288	18761658	80	0.000403	com.wsimg
289	18760046	411	0.000065	com.tripod
290	18754884	957	0.000033	ee.linktr
291	18754526	1021	0.000030	gov.usgs
292	18753164	1459	0.000021	uk.co.wired
293	18752728	338	0.000077	fr.google
294	18751846	1059	0.000029	com.500px
295	18751636	452	0.000061	ca.google
296	18749418	1996	0.000017	com.amd
297	18744444	1944	0.000018	com.azure
298	18742964	777	0.000040	au.com.google
299	18742506	481	0.000056	com.163
300	18741292	1091	0.000028	com.ssrn
301	18740758	1065	0.000029	com.newsweek
302	18734910	1688	0.000019	ca.utoronto
303	18734620	139	0.000218	com.spotify
304	18731112	744	0.000042	cn.com.people
305	18730384	334	0.000078	page.g
306	18730074	2751	0.000012	com.nabble
307	18728400	1454	0.000021	com.howstuffworks
308	18722938	2107	0.000016	com.lego
309	18719762	1675	0.000019	com.storify
310	18719332	1140	0.000027	uk.co.thetimes
311	18717930	801	0.000039	site.business
312	18717726	884	0.000036	uk.ac.ox
313	18716206	311	0.000083	com.bitly
314	18715060	1218	0.000026	com.scmp
315	18713618	798	0.000040	com.adage
316	18713552	654	0.000044	com.indiatimes
317	18712564	1908	0.000018	de.mpg
318	18712368	1057	0.000029	com.thehill
319	18705466	519	0.000052	com.criteo
320	18704754	1078	0.000028	org.ohchr
321	18704474	1531	0.000020	com.aljazeera
322	18703348	802	0.000039	uk.gov.service
323	18701482	1545	0.000020	org.greenpeace
324	18699064	331	0.000078	com.netdna-ssl
325	18698378	967	0.000033	ch.google
326	18693994	784	0.000040	us.icio
327	18693690	1153	0.000027	int.coe
328	18692556	933	0.000034	org.d3js
329	18690456	1499	0.000021	com.history
330	18689794	1018	0.000030	com.netlify
331	18688064	1320	0.000023	com.nymag
332	18687064	1363	0.000023	org.wiktionary
333	18684868	287	0.000091	ru.ok
334	18683792	1293	0.000024	com.intuit
335	18682796	1419	0.000022	uk.co.standard
336	18681388	1995	0.000017	edu.arizona
337	18679058	944	0.000034	gov.archives
338	18678794	953	0.000034	ru.google
339	18677084	1054	0.000029	sg.com.google
340	18675890	900	0.000035	br.com.google
341	18674402	85	0.000385	co.g
342	18674068	1975	0.000018	com.wattpad
343	18673754	526	0.000051	ru.gov
344	18673370	1351	0.000023	com.ikea
345	18668598	1461	0.000021	com.reverbnation
346	18668444	2681	0.000013	edu.drexel
347	18668276	1121	0.000027	edu.si
348	18666994	1174	0.000027	uk.co.mirror
349	18666846	2572	0.000013	org.maven
350	18666724	412	0.000065	com.cnet
351	18664542	580	0.000048	org.openstreetmap
352	18663710	1373	0.000023	com.jetbrains
353	18663688	1032	0.000030	com.theconversation
354	18663542	1921	0.000018	com.newscientist
355	18661472	847	0.000037	gov.state
356	18661146	1572	0.000020	ms.1drv
357	18660152	2644	0.000013	com.mystrikingly
358	18655360	973	0.000032	org.fao
359	18654458	590	0.000047	cn.google
360	18653472	235	0.000112	com.etsy
361	18652232	1485	0.000021	com.flipboard
362	18651820	767	0.000041	com.deviantart
363	18651514	1375	0.000023	com.thedailybeast
364	18651404	1220	0.000026	org.jstor
365	18649024	1270	0.000024	com.strikingly
366	18647422	2045	0.000017	blog.home
367	18646812	634	0.000044	com.zdnet
368	18644828	325	0.000079	tv.twitch
369	18642272	2781	0.000012	com.diigo
370	18640482	1123	0.000027	com.britannica
371	18639254	1904	0.000018	ca.ubc
372	18638840	367	0.000072	com.jotform
373	18635188	1959	0.000018	com.gettyimages
374	18634254	1685	0.000019	com.channel4
375	18631278	1494	0.000021	org.pypi
376	18630386	813	0.000039	in.co.google
377	18627814	417	0.000064	com.ssl-images-amazon
378	18626978	161	0.000166	gle.forms
379	18623310	1982	0.000018	org.hrw
380	18623132	281	0.000092	com.cloudinary
381	18618612	1382	0.000022	au.com.smh
382	18617234	1566	0.000020	uk.co.metro
383	18617180	2031	0.000017	hk.com.google
384	18617072	1599	0.000020	edu.ufl
385	18613590	2332	0.000015	ly.rebrand
386	18612786	457	0.000061	net.imgix
387	18609746	418	0.000064	com.webflow
388	18609050	2311	0.000015	com.shutterfly
389	18607782	568	0.000048	com.feedly
390	18603850	538	0.000050	gov.epa
391	18602470	104	0.000348	com.stripe
392	18601118	83	0.000391	net.jsfiddle
393	18599796	3423	0.000010	org.aclweb
394	18597166	2348	0.000014	com.yarnpkg
395	18596278	69	0.000461	net.akamaihd
396	18596202	1907	0.000018	gov.supremecourt
397	18595244	2344	0.000014	com.thefreedictionary
398	18593816	468	0.000058	nl.google
399	18592072	1578	0.000020	com.dw
400	18588294	2955	0.000012	com.upi
401	18587932	981	0.000032	com.thelancet
402	18587926	425	0.000064	com.slack
403	18587680	396	0.000067	com.kickstarter
404	18587378	787	0.000040	com.urldefense
405	18585950	1713	0.000019	ca.sfu
406	18583582	460	0.000060	com.livechatinc
407	18581082	623	0.000045	com.quora
408	18580964	428	0.000063	com.rackcdn
409	18580620	1967	0.000018	com.euronews
410	18580552	451	0.000061	com.go
411	18580130	1368	0.000023	com.tunein
412	18578076	594	0.000046	ru.liveinternet
413	18576712	475	0.000057	com.googleblog
414	18571776	2597	0.000013	pt.sapo
415	18571212	2109	0.000016	com.itv
416	18570630	1945	0.000018	uk.co.huffingtonpost
417	18570542	1286	0.000024	edu.brookings
418	18570528	4423	0.000008	tl.page
419	18570058	2369	0.000014	com.angelfire
420	18568882	2614	0.000013	org.wikibooks
421	18567302	1692	0.000019	com.ifttt
422	18564134	861	0.000036	com.freepik
423	18563246	2244	0.000015	com.netvibes
424	18562602	133	0.000251	com.mailchimp
425	18562564	364	0.000072	me.telegram
426	18562400	561	0.000048	com.microsoftonline
427	18562224	1976	0.000018	uk.co.express
428	18559206	2888	0.000012	sg.edu.nus
429	18559092	1928	0.000018	io.webflow
430	18557292	772	0.000041	pl.google
431	18555900	480	0.000056	com.meetup
432	18555482	4752	0.000007	com.newgrounds
433	18554944	2397	0.000014	google.ai
434	18554512	2439	0.000014	com.yolasite
435	18553912	2124	0.000016	jp.geocities
436	18552986	3394	0.000011	com.instapaper
437	18551338	362	0.000072	com.proofpoint
438	18548844	1358	0.000023	com.people
439	18546296	64	0.000531	net.typekit
440	18543694	2104	0.000016	org.c-span
441	18541918	159	0.000169	ru.mail
442	18541834	2043	0.000017	com.avg
443	18540650	2249	0.000015	app.netlify
444	18539394	3004	0.000011	com.000webhostapp
445	18539316	485	0.000055	com.elsevier
446	18538008	3494	0.000010	cn.edu.pku
447	18536872	1609	0.000020	com.asahi
448	18535422	876	0.000036	org.worldwildlife
449	18535204	1127	0.000027	uk.parliament
450	18534822	1956	0.000018	uk.gov.ons
451	18533694	188	0.000138	com.iubenda
452	18532790	2113	0.000016	org.documentcloud
453	18532338	3074	0.000011	uk.co.timesonline
454	18531118	264	0.000096	com.office
455	18527764	237	0.000112	com.eventbrite
456	18527012	2699	0.000013	com.self
457	18526172	2511	0.000013	com.foreignpolicy
458	18524804	2421	0.000014	org.sundance
459	18524702	214	0.000120	com.aliyuncs
460	18524140	1213	0.000026	be.google
461	18523242	2200	0.000016	ie.google
462	18523000	1432	0.000022	gov.weather
463	18522694	3136	0.000011	com.openai
464	18522588	879	0.000036	org.mediawiki
465	18521124	2806	0.000012	com.pearltrees
466	18520306	1704	0.000019	com.firebaseapp
467	18516520	3620	0.000010	com.dailycaller
468	18514512	498	0.000054	it.placehold
469	18514168	2695	0.000013	com.france24
470	18513026	644	0.000044	edu.berkeley
471	18512138	492	0.000055	cn.360
472	18511428	2296	0.000015	com.msnbc
473	18510986	2089	0.000017	com.thestar
474	18510258	3732	0.000009	me.site123
475	18509392	2133	0.000016	com.gfycat
476	18508906	341	0.000076	com.rawgit
477	18507920	521	0.000052	com.gmail
478	18507686	1952	0.000018	org.ocks
479	18506872	2739	0.000012	org.rsc
480	18504310	2486	0.000014	edu.hawaii
481	18503766	2366	0.000014	de.br
482	18503250	2447	0.000014	edu.colostate
483	18502578	171	0.000154	com.zendesk
484	18501424	2222	0.000015	org.nobelprize
485	18501096	3293	0.000011	net.pixnet
486	18500188	1528	0.000020	net.seesaa
487	18500164	2471	0.000014	com.motherjones
488	18499720	756	0.000042	com.vice
489	18499378	4234	0.000008	com.masslive
490	18496634	2355	0.000014	com.cision
491	18495058	101	0.000361	com.godaddy
492	18492104	886	0.000036	gov.nist
493	18491956	1249	0.000025	org.ilo
494	18490654	2070	0.000017	com.surveygizmo
495	18490628	3378	0.000011	com.minds
496	18490576	635	0.000044	com.matterport
497	18489858	2656	0.000013	ph.com.google
498	18488106	369	0.000071	org.python
499	18487032	980	0.000032	gov.va
500	18485800	1166	0.000027	at.google
501	18485152	1318	0.000023	se.google
502	18483644	1961	0.000018	ru.ucoz
503	18482996	2401	0.000014	com.freep
504	18482190	3874	0.000009	com.wizards
505	18481738	3583	0.000010	edu.uvm
506	18478142	3711	0.000010	org.tvtropes
507	18476988	1506	0.000021	com.cognitoforms
508	18476516	1493	0.000021	gov.uscourts
509	18476024	3530	0.000010	org.oxfam
510	18473992	2235	0.000015	cn.t
511	18473054	4331	0.000008	fm.ask
512	18473034	1708	0.000019	dk.google
513	18470526	3122	0.000011	de.dw
514	18467204	2009	0.000017	ua.com.google
515	18467126	3935	0.000009	com.youdao
516	18464016	128	0.000262	org.networkadvertising
517	18462968	1031	0.000030	com.arstechnica
518	18462674	2310	0.000015	int.unfccc
519	18461844	3323	0.000011	ch.nzz
520	18460156	123	0.000276	com.statcounter
521	18460126	3757	0.000009	net.hinet
522	18460018	2484	0.000014	com.washingtontimes
523	18459778	3391	0.000011	edu.miami
524	18459648	5025	0.000007	tw.com.gamer
525	18459120	4313	0.000008	ch.qos
526	18458774	788	0.000040	com.intel
527	18456584	2220	0.000015	mx.com.google
528	18455734	2241	0.000015	gov.ky
529	18455504	3426	0.000010	com.nwsource
530	18454948	856	0.000037	io.readthedocs
531	18453730	2187	0.000016	gov.cisa
532	18451988	2256	0.000015	com.straitstimes
533	18449466	371	0.000071	io.codepen
534	18447006	361	0.000072	com.prnewswire
535	18446224	4097	0.000009	com.smore
536	18446132	2188	0.000016	pt.google
537	18445920	2719	0.000012	net.bplaced
538	18445802	5349	0.000007	net.wargaming
539	18445232	3272	0.000011	org.csis
540	18444732	1435	0.000022	org.aarp
541	18444080	289	0.000090	net.php
542	18443758	2282	0.000015	no.google
543	18443228	3924	0.000009	com.steemit
544	18443146	1304	0.000024	tw.com.google
545	18442018	314	0.000083	com.squarespace
546	18440872	743	0.000043	com.oreilly
547	18440596	199	0.000130	com.hubspot
548	18439354	4877	0.000007	com.bonanza
549	18438802	2020	0.000017	co.lpages
550	18438606	1079	0.000028	net.ovh
551	18438208	835	0.000037	com.imageshack
552	18437874	4023	0.000009	com.doodlekit
553	18436818	2425	0.000014	com.voanews
554	18436680	358	0.000073	ru.rambler
555	18436048	2805	0.000012	com.nationalpost
556	18435420	4534	0.000008	by.google
557	18435256	614	0.000045	org.nodejs
558	18435200	397	0.000067	com.onesignal
559	18434470	3374	0.000011	fr.rfi
560	18434466	463	0.000060	gov.irs
561	18434444	2584	0.000013	com.snopes
562	18434230	1899	0.000018	link.page
563	18434190	3637	0.000010	org.vim
564	18434018	2240	0.000015	th.co.google
565	18433782	3395	0.000010	org.scala-lang
566	18432434	3142	0.000011	com.inquirer
567	18430898	2887	0.000012	org.ballotpedia
568	18430888	3324	0.000011	com.real
569	18428600	649	0.000044	br.com.uol
570	18428004	513	0.000052	com.pixabay
571	18426658	2142	0.000016	uk.co.which
572	18426634	4070	0.000009	com.viki
573	18425674	1038	0.000030	com.thenextweb
574	18424302	3146	0.000011	org.aps
575	18424050	2764	0.000012	com.post-gazette
576	18423516	2499	0.000014	net.openid
577	18422702	2627	0.000013	edu.usf
578	18421138	82	0.000391	com.livestream
579	18420414	961	0.000033	jp.shinobi
580	18420272	956	0.000033	int.wipo
581	18417146	4450	0.000008	com.bravesites
582	18415542	2881	0.000012	ru.aif
583	18414574	2906	0.000012	io.gitlab
584	18414284	3387	0.000011	org.pri
585	18414276	1932	0.000018	gov.ct
586	18413984	2602	0.000013	il.co.google
587	18413906	1910	0.000018	org.oxfordjournals
588	18413218	4664	0.000008	com.ucoz
589	18412422	566	0.000048	com.photobucket
590	18412344	2191	0.000016	com.xrea
591	18412198	2234	0.000015	nz.co.google
592	18410920	2088	0.000017	net.cnki
593	18410828	2847	0.000012	com.webbyawards
594	18410164	433	0.000063	com.staticflickr
595	18409934	3675	0.000010	org.heritage
596	18408908	1993	0.000018	tr.com.google
597	18408574	2053	0.000017	com.treehugger
598	18406062	1695	0.000019	net.leadpages
599	18405282	2112	0.000016	fi.google
600	18402764	5153	0.000007	kz.google
601	18402708	211	0.000121	to.amzn
602	18402670	569	0.000048	com.deloitte
603	18402662	1100	0.000028	cz.google
604	18402526	4562	0.000008	com.freehostia
605	18402334	2156	0.000016	gov.faa
606	18402326	2724	0.000012	com.detroitnews
607	18402220	2774	0.000012	com.slidesharecdn
608	18402102	346	0.000075	com.adnxs
609	18396726	812	0.000039	com.thinkwithgoogle
610	18392816	1471	0.000021	com.trustwave
611	18392376	2640	0.000013	org.iea
612	18392262	2883	0.000012	jp.blog
613	18391148	4426	0.000008	com.goal
614	18390184	3284	0.000011	com.financialpost
615	18389140	3636	0.000010	net.alarabiya
616	18389082	3570	0.000010	org.neocities
617	18388580	3784	0.000009	co.ello
618	18388256	207	0.000126	com.salesforce
619	18386478	3500	0.000010	com.archdaily
620	18385984	4517	0.000008	com.alamy
621	18385924	2297	0.000015	gr.google
622	18385398	160	0.000168	gov.privacyshield
623	18385020	2569	0.000013	org.kqed
624	18383196	277	0.000093	org.drupal
625	18382110	354	0.000074	com.snapchat
626	18381496	2338	0.000015	ro.google
627	18381392	3367	0.000011	uk.ac.leeds
628	18381316	271	0.000094	com.mapbox
629	18380144	3907	0.000009	uk.gov.scotland
630	18379620	1946	0.000018	hu.google
631	18378224	4399	0.000008	co.aeon
632	18377446	374	0.000070	com.cdninstagram
633	18376062	3545	0.000010	gov.fec
634	18376022	3312	0.000011	com.virgin
635	18375628	2219	0.000015	ar.com.google
636	18375060	4128	0.000009	cn.globaltimes
637	18374688	4333	0.000008	com.corel
638	18374066	464	0.000059	com.herokuapp
639	18373200	4062	0.000009	jp.go.ndl
640	18373110	791	0.000040	google.blog
641	18372316	2208	0.000016	com.justia
642	18372216	2320	0.000015	za.co.google
643	18370616	2216	0.000016	ru.ria
644	18370232	3694	0.000010	com.intensedebate
645	18369794	3793	0.000009	com.visualcapitalist
646	18369094	2722	0.000012	si.google
647	18368512	4182	0.000008	com.rediff
648	18367604	3834	0.000009	ca.uvic
649	18367236	2577	0.000013	ru.rosminzdrav
650	18365918	439	0.000062	com.nypost
651	18365880	4678	0.000008	org.wikimapia
652	18365350	3439	0.000010	com.nationalreview
653	18364962	2134	0.000016	uk.org.asa
654	18364282	3850	0.000009	tw.edu.ntu
655	18363974	598	0.000046	com.samsung
656	18363190	2703	0.000012	is.google
657	18362598	3869	0.000009	com.podomatic
658	18361242	316	0.000082	cn.bshare
659	18360424	3484	0.000010	org.wri
660	18360028	4160	0.000009	uk.co.spectator
661	18359858	1711	0.000019	ly.cutt
662	18358316	4989	0.000007	to.gplus
663	18358086	4908	0.000007	com.atwebpages
664	18357826	177	0.000150	com.tripadvisor
665	18357438	5003	0.000007	org.scala-sbt
666	18356488	4276	0.000008	ru.msu
667	18356450	1161	0.000027	com.udemy
668	18355358	2973	0.000011	com.timesofisrael
669	18352506	5213	0.000007	edu.csulb
670	18351622	4744	0.000007	com.authorstream
671	18350944	4127	0.000009	gy.rb
672	18350110	3204	0.000011	us.ny.state
673	18349876	3644	0.000010	com.linuxquota
674	18349798	3563	0.000010	com.udn
675	18349578	3845	0.000009	org.jenkins-ci
676	18349508	1686	0.000019	com.pcworld
677	18349104	2481	0.000014	uk.ac.imperial
678	18348784	5238	0.000007	com.etymonline
679	18348026	3492	0.000010	eg.com.google
680	18347774	3363	0.000011	uk.co.bbci
681	18347338	2386	0.000014	com.name
682	18346938	3745	0.000009	com.novell
683	18345924	1487	0.000021	com.digitaloceanspaces
684	18345376	6040	0.000006	net.vingle
685	18345350	2615	0.000013	us.pa.state
686	18345040	642	0.000044	com.xiti
687	18345006	2302	0.000015	fr.pagesjaunes
688	18344246	4604	0.000008	by.tut
689	18341982	78	0.000417	com.messenger
690	18341502	1672	0.000019	id.co.google
691	18341492	4012	0.000009	com.donaldjtrump
692	18339724	2359	0.000014	co.pcdn
693	18338674	606	0.000046	com.indeed
694	18338446	459	0.000060	com.sxsw
695	18337870	2379	0.000014	sk.google
696	18337126	246	0.000105	uk.co.amazon
697	18336826	351	0.000074	com.atlassian
698	18336810	1225	0.000025	com.dell
699	18336442	4947	0.000007	fr.online
700	18336226	1933	0.000018	com.law
701	18335648	3783	0.000009	com.wmtransfer
702	18335422	2242	0.000015	kr.co.google
703	18335402	4709	0.000008	edu.odu
704	18335130	2971	0.000011	cl.google
705	18335024	4300	0.000008	il.ac.huji
706	18334782	4271	0.000008	tw.gov.cdc
707	18333794	2886	0.000012	my.com.google
708	18333014	3385	0.000011	com.scotsman
709	18332864	3322	0.000011	com.instructure
710	18332832	4563	0.000008	com.hackaday
711	18332194	2131	0.000016	gov.pa
712	18332054	627	0.000045	com.withgoogle
713	18331108	1997	0.000017	scot.gov
714	18330912	3178	0.000011	com.broadwayworld
715	18330804	858	0.000036	com.canva
716	18330694	4525	0.000008	com.mongabay
717	18329802	4508	0.000008	com.macobserver
718	18329686	3725	0.000010	org.sonatype
719	18328118	2391	0.000014	gov.wi
720	18327736	2683	0.000013	org.usgbc
721	18327662	4113	0.000009	gov.peacecorps
722	18327624	4652	0.000008	cn.tianya
723	18326710	3495	0.000010	pk.com.google
724	18326302	870	0.000036	com.marketwatch
725	18326164	1490	0.000021	com.billboard
726	18324976	107	0.000316	net.gandi
727	18324878	2845	0.000012	com.thecut
728	18324686	89	0.000372	me.ogp
729	18323980	4585	0.000008	io.meduza
730	18323898	2827	0.000012	uk.org.nationaltrust
731	18323758	3911	0.000009	au.edu.adelaide
732	18323398	4766	0.000007	de.uni-erlangen
733	18322482	3759	0.000009	uk.org.rspb
734	18322376	3773	0.000009	cv.google
735	18321256	5135	0.000007	cat.bcn
736	18319736	3728	0.000009	com.ipage
737	18319726	5311	0.000007	com.brother
738	18318148	2410	0.000014	my.com.thestar
739	18317872	3401	0.000010	uk.ac.york
740	18317504	3315	0.000011	com.politifact
741	18317408	3128	0.000011	ee.google
742	18317178	3326	0.000011	org.thinkprogress
743	18317034	2102	0.000016	se.haxx
744	18316764	4554	0.000008	au.edu.rmit
745	18316272	2959	0.000011	hr.google
746	18315296	5212	0.000007	com.selfridges
747	18315244	3772	0.000009	au.com.telstra
748	18313746	1436	0.000022	com.fiverr
749	18313044	3420	0.000010	de.hu-berlin
750	18311516	3572	0.000010	com.nola
751	18311094	3458	0.000010	sa.com.google
752	18310436	4145	0.000009	ca.dal
753	18310126	6237	0.000006	org.arkive
754	18309422	2759	0.000012	bg.google
755	18308696	3429	0.000010	com.monday
756	18308664	4635	0.000008	at.tugraz
757	18308432	3508	0.000010	com.eiseverywhere
758	18308298	3764	0.000009	uk.co.cfdr
759	18308102	3298	0.000011	org.iucn
760	18307444	3571	0.000010	app.web
761	18306932	3702	0.000010	org.iucnredlist
762	18306908	292	0.000088	com.surveymonkey
763	18306390	3806	0.000009	gi.com.google
764	18306038	5056	0.000007	ec.com.google
765	18305962	3875	0.000009	de.uni-freiburg
766	18305528	4244	0.000008	au.com.heraldsun
767	18305222	515	0.000052	io.shields
768	18304914	610	0.000046	org.eff
769	18304878	3829	0.000009	com.psmag
770	18304506	4721	0.000007	ua.at
771	18302798	930	0.000034	gov.uspto
772	18302648	190	0.000137	com.automattic
773	18301286	3948	0.000009	com.mozello
774	18300612	1108	0.000028	com.gizmodo
775	18300418	3596	0.000010	pl.wp
776	18300322	3471	0.000010	org.royalsociety
777	18299622	2819	0.000012	org.unep
778	18299452	3606	0.000010	com.realclearpolitics
779	18298298	3531	0.000010	jp.coocan
780	18298296	2613	0.000013	vn.com.google
781	18298218	4434	0.000008	jp.hatenablog
782	18297896	4281	0.000008	com.waitrose
783	18297876	4676	0.000008	info.webry
784	18297852	4427	0.000008	net.inquirer
785	18297704	4274	0.000008	jp.gree
786	18297178	4611	0.000008	org.nationalinterest
787	18296330	2981	0.000011	edu.uconn
788	18295610	946	0.000034	edu.columbia
789	18295554	5531	0.000006	org.mises
790	18295452	1274	0.000024	com.smashingmagazine
791	18295224	3303	0.000011	uk.gov.companieshouse
792	18294866	4442	0.000008	gov.ourdocuments
793	18294666	3894	0.000009	sl.com.google
794	18292912	6218	0.000006	com.rhino3d
795	18292842	3435	0.000010	org.cfr
796	18292780	790	0.000040	com.airbnb
797	18292712	283	0.000092	jp.co.amazon
798	18291570	413	0.000065	com.pubmatic
799	18290920	878	0.000036	com.box
800	18290426	5610	0.000006	com.coroflot
801	18290346	4348	0.000008	com.thediplomat
802	18286902	4066	0.000009	com.inhabitat
803	18286668	3277	0.000011	com.bp
804	18286522	4592	0.000008	cat.uab
805	18283480	3827	0.000009	uk.co.villiers-london
806	18283014	4140	0.000009	org.grist
807	18282452	4016	0.000009	com.foreignaffairs
808	18281324	1081	0.000028	com.tapad
809	18280378	1347	0.000023	org.altervista
810	18280358	382	0.000069	com.skype
811	18280324	4349	0.000008	com.worldsecuresystems
812	18279680	2409	0.000014	com.volusion
813	18279516	2907	0.000012	ru.nethouse
814	18279480	3527	0.000010	pe.com.google
815	18279438	4779	0.000007	be.lesoir
816	18278874	3288	0.000011	co.com.google
817	18278816	3885	0.000009	de.uni-koeln
818	18278778	2910	0.000012	org.gnupg
819	18278022	4656	0.000008	com.mihanblog
820	18277554	3360	0.000011	org.panda
821	18277186	3440	0.000010	lv.google
822	18276674	5300	0.000007	lu.google
823	18276442	484	0.000055	com.inc
824	18275676	5103	0.000007	cn.com.caijing
825	18275134	3331	0.000011	uk.gov.metoffice
826	18274258	68	0.000471	com.oculus
827	18273732	2364	0.000014	org.donorbox
828	18273312	3038	0.000011	rs.google
829	18273256	1197	0.000026	com.merriam-webster
830	18271448	5051	0.000007	ee.ut
831	18271060	2519	0.000013	com.amebaownd
832	18270922	4482	0.000008	com.marksandspencer
833	18270780	6447	0.000006	su.clan
834	18269948	4096	0.000009	ru.interfax
835	18269620	3852	0.000009	org.rferl
836	18268756	2904	0.000012	gov.nd
837	18267994	548	0.000049	com.fortune
838	18267776	4693	0.000008	it.unitn
839	18267714	5665	0.000006	am.google
840	18266762	3502	0.000010	org.iaea
841	18263748	3893	0.000009	pr.com.google
842	18262158	5045	0.000007	com.tok2
843	18261938	1901	0.000018	ch.ethz
844	18261922	3342	0.000011	gov.la
845	18261182	4507	0.000008	org.democracynow
846	18261176	2593	0.000013	net.noscript
847	18260216	836	0.000037	com.mix
848	18259862	408	0.000066	net.adform
849	18259608	5208	0.000007	tn.google
850	18257978	4212	0.000008	jp.hateblo
851	18257888	6029	0.000006	hk.edu.hkbu
852	18257680	3884	0.000009	nl.wur
853	18257594	5009	0.000007	gr.auth
854	18257406	997	0.000031	com.webs
855	18256760	4512	0.000008	com.mnn
856	18256702	5759	0.000006	ru.nnov
857	18256238	3954	0.000009	com.afp
858	18255744	1365	0.000023	com.format
859	18255662	5209	0.000007	nf.co
860	18253954	329	0.000079	com.getbootstrap
861	18252988	4961	0.000007	jp.hatenadiary
862	18252154	4728	0.000007	hk.com.hkex
863	18251258	1193	0.000026	com.redhat
864	18250974	5600	0.000006	com.gust
865	18250088	1067	0.000029	com.symantec
866	18249466	2562	0.000013	net.ucoz
867	18249320	268	0.000095	com.typeform
868	18248694	6327	0.000006	com.x10host
869	18248332	3547	0.000010	uk.co.saveourschools
870	18247898	2934	0.000012	com.squarespace-cdn
871	18247292	2977	0.000011	lt.google
872	18246872	525	0.000051	com.adweek
873	18246844	4295	0.000008	com.scienceblogs
874	18246472	4848	0.000007	de.uni-konstanz
875	18245562	6362	0.000006	com.ueuo
876	18245048	3856	0.000009	uk.gov.data
877	18244756	4005	0.000009	tr.com.hurriyet
878	18243652	3070	0.000011	ae.google
879	18243570	1891	0.000019	com.speakerdeck
880	18243330	5079	0.000007	com.blogsky
881	18243134	2044	0.000017	tv.ustream
882	18240374	6711	0.000006	su.moy
883	18239298	761	0.000041	gov.copyright
884	18239096	5292	0.000007	ru.novayagazeta
885	18239044	2789	0.000012	gov.nh
886	18238990	4057	0.000009	org.hathitrust
887	18238948	3648	0.000010	org.annualreviews
888	18238932	1154	0.000027	pl.home
889	18238882	3815	0.000009	com.businesscatalyst
890	18237740	472	0.000058	com.ea
891	18237726	3087	0.000011	uk.gov.hmrc
892	18236940	3930	0.000009	cc.uxdesign
893	18236894	6015	0.000006	com.artfire
894	18236704	366	0.000072	org.opensource
895	18236530	3467	0.000010	it.beniculturali
896	18236132	2507	0.000014	gov.mn
897	18236076	1019	0.000030	com.engadget
898	18235902	3682	0.000010	ve.co.google
899	18235452	4973	0.000007	com.teslamotors
900	18234038	7475	0.000005	com.hangame
901	18233966	427	0.000063	com.fastcompany
902	18233600	4263	0.000008	com.hsbc
903	18233074	2462	0.000014	com.netsolhost
904	18232582	5556	0.000006	me.google
905	18232344	5643	0.000006	mu.google
906	18231570	5529	0.000006	com.yam
907	18231242	3969	0.000009	tz.co.google
908	18230998	974	0.000032	com.verisign
909	18230916	3364	0.000011	tw.com.pchome
910	18230662	7293	0.000005	com.addr
911	18230628	2636	0.000013	com.shell
912	18230602	6599	0.000006	com.dropmark
913	18229708	5635	0.000006	li.google
914	18229116	5002	0.000007	com.gab
915	18229106	4493	0.000008	com.tapatalk
916	18228194	1325	0.000023	edu.ucla
917	18227958	3557	0.000010	uk.co.newmedianow
918	18227938	4988	0.000007	edu.whoi
919	18227810	3738	0.000009	ng.com.google
920	18227630	5444	0.000007	ni.com.google
921	18226076	4110	0.000009	uk.co.sainsburys
922	18225458	4412	0.000008	com.iconarchive
923	18225080	5380	0.000007	gr.ntua
924	18224924	6152	0.000006	com.epochtimes
925	18224716	5198	0.000007	org.birdlife
926	18224610	3532	0.000010	uk.co.intersol
927	18224178	5615	0.000006	id.co.kaskus
928	18223762	950	0.000034	com.zoho
929	18223166	5403	0.000007	cr.co.google
930	18223046	5695	0.000006	sv.com.google
931	18222882	4074	0.000009	vn.zing
932	18222714	4537	0.000008	uk.co.zoopla
933	18222480	4039	0.000009	uk.ac.jisc
934	18221034	3836	0.000009	com.prweek
935	18220422	3098	0.000011	int.wmo
936	18220410	5466	0.000006	mz.co.google
937	18220202	4966	0.000007	edu.umb
938	18220196	1290	0.000024	uk.co.freeukbusinessdirectory
939	18220068	1476	0.000021	org.owasp
940	18219726	6669	0.000006	net.comunidades
941	18218976	4141	0.000009	com.scotusblog
942	18218840	5636	0.000006	com.cyberlink
943	18218738	3828	0.000009	do.com.google
944	18218672	2966	0.000011	io.termly
945	18218262	4735	0.000007	com.fatcow
946	18218172	3851	0.000009	mt.com.google
947	18218110	3589	0.000010	uk.org.oxonaa
948	18217958	3774	0.000009	gt.com.google
949	18216908	3737	0.000009	com.solidworks
950	18216782	3641	0.000010	uk.co.profilebusiness
951	18216270	3625	0.000010	uk.co.heatall
952	18216034	4506	0.000008	com.theringer
953	18215388	2558	0.000013	nl.jouwweb
954	18215320	800	0.000039	com.wikihow
955	18215060	5953	0.000006	com.symbaloo
956	18214768	5171	0.000007	pl.cba
957	18214162	5740	0.000006	kg.google
958	18213594	2321	0.000015	com.freeprivacypolicy
959	18212850	1222	0.000026	com.att
960	18212680	5203	0.000007	pl.lublin
961	18212672	1541	0.000020	edu.umd
962	18212174	5485	0.000006	uk.org.labour
963	18212074	4288	0.000008	us.ms.state
964	18211828	3449	0.000010	com.wantedly
965	18211570	4396	0.000008	org.ametsoc
966	18211542	3701	0.000010	uy.com.google
967	18211486	5553	0.000006	jp.ifdef
968	18211438	5218	0.000007	es.usal
969	18211398	769	0.000041	com.netflix
970	18211196	6329	0.000006	org.cgsociety
971	18210854	3897	0.000009	hn.google
972	18210544	5602	0.000006	org.svoboda
973	18207828	4432	0.000008	org.ascd
974	18207784	4500	0.000008	uk.co.dailystar
975	18207712	3651	0.000010	uk.co.articlelistings
976	18207370	503	0.000054	com.dmca
977	18207114	916	0.000035	com.ggpht
978	18207032	5199	0.000007	com.curseforge
979	18206432	5265	0.000007	org.nsidc
980	18206340	1520	0.000021	com.technologyreview
981	18205908	5668	0.000006	ug.co.google
982	18205822	4030	0.000009	org.lacity
983	18205348	4843	0.000007	com.cbn
984	18204716	434	0.000063	com.businesswire
985	18204712	5860	0.000006	mn.google
986	18204394	6868	0.000005	kr.ac.postech
987	18204332	5613	0.000006	it.unige
988	18203526	3314	0.000011	uk.gov.food
989	18203314	6353	0.000006	com.skepticalscience
990	18203052	909	0.000035	org.weforum
991	18202434	4907	0.000007	com.globalpost
992	18202416	5172	0.000007	com.weightwatchers
993	18202000	3403	0.000010	com.lexology
994	18200738	5944	0.000006	tt.google
995	18200210	5282	0.000007	com.betfair
996	18199968	5428	0.000007	py.com.google
997	18198928	4815	0.000007	com.abcnews
998	18198698	763	0.000041	com.psychologytoday
999	18198512	6974	0.000005	org.toile-libre
1000	18198414	3291	0.000011	net.vnexpress

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

January 2021 crawl archive now available

February 2, 2021Sebastian Nagel

The crawl archive for January 2021 is now available! The data was crawled between January 15th and 28th and contains 3.4 billion web pages or 350 TiB of uncompressed content. It includes page captures of 1.15 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The January crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-04/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-04/segment.paths.gz	100
WARC files	CC-MAIN-2021-04/warc.paths.gz	79840	78.98
WAT files	CC-MAIN-2021-04/wat.paths.gz	79840	22.92
WET files	CC-MAIN-2021-04/wet.paths.gz	79840	10.04
Robots.txt files	CC-MAIN-2021-04/robotstxt.paths.gz	79840	0.23
Non-200 responses files	CC-MAIN-2021-04/non200responses.paths.gz	79840	2.11
URL index files	CC-MAIN-2021-04/cc-index.paths.gz	302	0.26

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-04/. Also the columnar index has been updated to contain this crawl.

November/December 2020 crawl archive now available

December 10, 2020Sebastian Nagel

The crawl archive for November/December 2020 is now available! The data was crawled between November 23 and December 6 and contains 2.64 billion web pages or 270 TiB of uncompressed content. It includes page captures of 1.4 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The November/December crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-50/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-50/segment.paths.gz	100
WARC files	CC-MAIN-2020-50/warc.paths.gz	72000	59.95
WAT files	CC-MAIN-2020-50/wat.paths.gz	72000	17.82
WET files	CC-MAIN-2020-50/wet.paths.gz	72000	7.89
Robots.txt files	CC-MAIN-2020-50/robotstxt.paths.gz	72000	0.2
Non-200 responses files	CC-MAIN-2020-50/non200responses.paths.gz	72000	1.71
URL index files	CC-MAIN-2020-50/cc-index.paths.gz	302	0.2

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-50/. Also the columnar index has been updated to contain this crawl.

October 2020 crawl archive now available

November 7, 2020Sebastian Nagel

The crawl archive for October 2020 is now available! The data was crawled between October 19th and November 1st and contains 2.71 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.5 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The October crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-45/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-45/segment.paths.gz	100
WARC files	CC-MAIN-2020-45/warc.paths.gz	72000	63.79
WAT files	CC-MAIN-2020-45/wat.paths.gz	72000	18.39
WET files	CC-MAIN-2020-45/wet.paths.gz	72000	8.23
Robots.txt files	CC-MAIN-2020-45/robotstxt.paths.gz	72000	0.2
Non-200 responses files	CC-MAIN-2020-45/non200responses.paths.gz	72000	1.75
URL index files	CC-MAIN-2020-45/cc-index.paths.gz	302	0.21

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-45/. Also the columnar index has been updated to contain this crawl.

Interactive Webgraph Statistics Notebook Released

October 28, 2020Alex Xue

We are pleased to announce the release of an interactive Jupyter notebook that is used to provide:

Visualization of web graph statistics
An interface for interacting with the webgraph

The visualization of the web graph statistics is done by leveraging the WebGraph framework, which provides means of gathering many interesting data points of a web graph, such as the frequency distribution of indegrees/outdegrees in the graph, or size distributions of the connected components. We then are able to use pandas and matplotlib to provide a visualization for the data provided by WebGraph. This effort was largely inspired by the Topology of the 2012 WDC Hyperlink Graph document. Further details of WebGraph tool installation/usage, and the data visualization may be found in the cc-notebooks repository.

The interface for interacting with the webgraph is done by using pyWebGraph, a front end that interfaces Jython with WebGraph. First, before using this interface we must re-build the string maps, in order to create a mapping between the node ID (a numerical value), to domain name (and vice versa). Once this is established we are able to simply load up the graph into pyWebGraph, and you will be able to traverse the graph interactively.

Further details of pyWebGraph installation/usage, and how to rebuild the string maps may be found in interactive webgraph README of the cc-notebooks repository.

The Jupyter notebook is available on Github in the same repository. More details about how to navigate the repository can be found in the notebook itself, as well as in the README.

We hope that users will be able to use these notebooks to gain more insight into the web graph in a numerical and practical sense.

We are grateful for WebGraph for providing extremely useful tools for processing the web graph itself, and Massimo Santini for developing pyWebGraph.

Host- and Domain-Level Web Graphs Jul/Aug/Sep 2020

October 16, 2020Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August and September 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

Host-level graph

The graph consists of 539 million nodes and 3.02 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 467 million dangling nodes (86.7%) and the largest strongly connected component contains 46 million (8.5%) nodes.

You can download the graph and the ranks of all 539 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2020-jul-aug-sep/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2020-jul-aug-sep/host/ as prefix to access the files from everywhere.

Size	File	Description
3.32 GB	cc-main-2020-jul-aug-sep-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 12 vertices files
13.7 GB	cc-main-2020-jul-aug-sep-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 24 edges files

5.95 GB	cc-main-2020-jul-aug-sep-host.graph	graph in BVGraph format
2 kB	cc-main-2020-jul-aug-sep-host.properties

6.76 GB	cc-main-2020-jul-aug-sep-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2020-jul-aug-sep-host-t.properties

1 kB	cc-main-2020-jul-aug-sep-host.stats	WebGraph statistics

7.77 GB	cc-main-2020-jul-aug-sep-host-ranks.txt.gz	harmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 89 million nodes and 1.71 billion edges. 51% or 45 million nodes are dangling nodes, the largest strongly connected component covers 35 million or 39% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2020-jul-aug-sep/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2020-jul-aug-sep/domain/.

Download files of the Common Crawl Jul/Aug/Sep 2020 domain-level webgraph

Size	File	Description
0.61 GB	cc-main-2020-jul-aug-sep-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
6.80 GB	cc-main-2020-jul-aug-sep-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.75 GB	cc-main-2020-jul-aug-sep-domain.graph	graph in BVGraph format
2 kB	cc-main-2020-jul-aug-sep-domain.properties

3.69 GB	cc-main-2020-jul-aug-sep-domain-t.graph	transpose of the graph
2 kB	cc-main-2020-jul-aug-sep-domain-t.properties

1 kB	cc-main-2020-jul-aug-sep-domain.stats	WebGraph statistics

1.91 GB	cc-main-2020-jul-aug-sep-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 89 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Jul/Aug/Sep 2020)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	32027928	1	0.018888	com.googleapis
2	30312944	3	0.012001	com.facebook
3	29025948	2	0.013237	com.google
4	26560472	4	0.007343	org.w
5	26516534	5	0.007172	com.twitter
6	26016464	6	0.006600	com.youtube
7	24614190	9	0.004795	com.instagram
8	24220712	8	0.005190	org.gmpg
9	23572970	7	0.005599	com.googletagmanager
10	23188190	11	0.003202	com.linkedin
11	22457894	15	0.002590	com.gravatar
12	22451350	10	0.003967	com.cloudflare
13	22364152	14	0.002726	com.gstatic
14	22350042	12	0.003105	org.wordpress
15	21926906	22	0.001505	com.pinterest
16	21699168	21	0.001752	com.wordpress
17	21599006	26	0.001181	org.wikipedia
18	21538264	16	0.002431	com.bootstrapcdn
19	21497526	18	0.001836	com.apple
20	21314410	30	0.001106	com.vimeo
21	21248994	41	0.000830	be.youtu
22	21186566	20	0.001794	com.jquery
23	21081822	23	0.001444	com.microsoft
24	21073240	45	0.000773	com.blogspot
25	20994964	39	0.000952	com.amazonaws
26	20975988	46	0.000732	gl.goo
27	20971574	25	0.001384	com.wp
28	20921220	47	0.000723	com.amazon
29	20788608	72	0.000439	com.tumblr
30	20716256	19	0.001804	com.adobe
31	20694562	67	0.000535	ly.bit
32	20675418	34	0.001018	com.google-analytics
33	20627694	53	0.000673	org.mozilla
34	20618998	17	0.001975	com.github
35	20617620	31	0.001059	net.cloudfront
36	20579928	71	0.000449	com.yahoo
37	20571130	29	0.001127	com.googlesyndication
38	20570586	60	0.000612	eu.europa
39	20562028	52	0.000679	com.flickr
40	20560188	42	0.000818	net.jsdelivr
41	20526264	97	0.000347	com.googleusercontent
42	20481758	62	0.000606	co.t
43	20480218	109	0.000313	com.reddit
44	20451670	24	0.001419	com.fontawesome
45	20436180	83	0.000389	com.weebly
46	20387228	56	0.000628	com.paypal
47	20375802	40	0.000910	com.macromedia
48	20372972	70	0.000450	com.medium
49	20370180	43	0.000808	com.addthis
50	20360678	28	0.001156	ru.yandex
51	20338498	27	0.001156	me.wp
52	20331252	64	0.000559	org.w3
53	20326560	79	0.000411	io.github
54	20292836	138	0.000223	com.nytimes
55	20275824	76	0.000414	org.creativecommons
56	20274244	59	0.000615	org.schema
57	20255326	150	0.000192	com.forbes
58	20246068	173	0.000151	com.imgur
59	20227930	36	0.000979	net.doubleclick
60	20219612	194	0.000133	uk.co.bbc
61	20210924	114	0.000285	com.soundcloud
62	20171070	66	0.000548	com.vk
63	20155222	195	0.000133	com.cnn
64	20142696	44	0.000803	org.apache
65	20134806	63	0.000587	com.whatsapp
66	20129582	314	0.000082	edu.mit
67	20123032	180	0.000146	com.imdb
68	20118310	208	0.000124	net.slideshare
69	20116626	243	0.000101	com.wsj
70	20115768	197	0.000128	org.wikimedia
71	20089462	85	0.000388	com.shopify
72	20082204	215	0.000120	edu.stanford
73	20076684	154	0.000181	gov.cdc
74	20075632	328	0.000079	com.wired
75	20069724	268	0.000094	com.techcrunch
76	20057066	255	0.000096	edu.harvard
77	20051336	353	0.000076	com.appspot
78	20051292	207	0.000124	net.sourceforge
79	20051264	257	0.000096	com.oracle
80	20051250	155	0.000177	int.who
81	20050888	206	0.000124	com.businessinsider
82	20046050	137	0.000227	org.archive
83	20038198	230	0.000113	com.washingtonpost
84	20035810	250	0.000097	com.live
85	20029940	164	0.000163	com.bing
86	20028210	549	0.000054	com.livejournal
87	20027622	424	0.000069	com.go
88	20024666	456	0.000066	com.msn
89	20019992	407	0.000072	uk.co.telegraph
90	20009306	170	0.000154	com.theguardian
91	20002514	527	0.000056	edu.cornell
92	19997146	199	0.000128	org.ietf
93	19996714	486	0.000063	gov.nasa
94	19995476	259	0.000096	com.android
95	19986252	302	0.000084	com.reuters
96	19983946	51	0.000702	net.fbcdn
97	19974890	240	0.000102	com.bloomberg
98	19966464	162	0.000164	com.giphy
99	19960428	77	0.000414	com.list-manage
100	19959046	520	0.000057	com.googleblog
101	19956558	269	0.000093	com.bbc
102	19955204	409	0.000071	com.slack
103	19942056	143	0.000205	com.spotify
104	19938828	591	0.000049	com.zdnet
105	19936894	48	0.000721	net.facebook
106	19935010	586	0.000050	com.quora
107	19931072	126	0.000265	com.ytimg
108	19922774	444	0.000067	com.myspace
109	19922046	757	0.000038	edu.umich
110	19920178	715	0.000040	edu.upenn
111	19917482	151	0.000185	gov.nih
112	19907886	344	0.000077	com.usatoday
113	19903896	654	0.000045	com.economist
114	19903722	313	0.000082	com.cnbc
115	19902700	308	0.000083	com.example
116	19896552	525	0.000056	com.pixabay
117	19895014	418	0.000070	net.researchgate
118	19882790	449	0.000066	com.latimes
119	19881164	188	0.000138	com.blogger
120	19870046	387	0.000075	org.python
121	19864804	65	0.000555	com.wix
122	19860760	433	0.000068	com.githubusercontent
123	19858732	693	0.000042	org.ieee
124	19854254	499	0.000061	com.mashable
125	19850918	571	0.000052	edu.berkeley
126	19847554	135	0.000241	com.youtube-nocookie
127	19845130	160	0.000167	com.issuu
128	19843068	218	0.000118	org.acm
129	19839736	834	0.000036	org.chromium
130	19839550	235	0.000106	uk.co.google
131	19835790	551	0.000054	org.arxiv
132	19833020	246	0.000099	net.behance
133	19832682	291	0.000086	org.npr
134	19831994	108	0.000320	com.unpkg
135	19831136	884	0.000034	com.arstechnica
136	19826840	213	0.000121	com.unsplash
137	19822884	341	0.000078	com.outlook
138	19822670	110	0.000303	de.google
139	19812430	54	0.000654	com.googleadservices
140	19810872	347	0.000077	com.prnewswire
141	19806458	678	0.000043	edu.columbia
142	19805382	171	0.000153	me.t
143	19804886	297	0.000085	com.dribbble
144	19804142	256	0.000096	com.squarespace
145	19799032	139	0.000215	gov.privacyshield
146	19798806	306	0.000083	com.huffingtonpost
147	19797964	260	0.000096	com.bandcamp
148	19795112	398	0.000074	com.time
149	19793874	37	0.000975	com.baidu
150	19792082	616	0.000048	com.gitlab
151	19790406	334	0.000079	com.nationalgeographic
152	19788214	443	0.000067	com.nature
153	19785178	794	0.000037	com.stackexchange
154	19782114	179	0.000147	gle.forms
155	19781676	258	0.000096	org.ampproject
156	19778534	548	0.000054	com.fortune
157	19777902	813	0.000036	com.git-scm
158	19776608	33	0.001030	com.wixstatic
159	19774030	771	0.000038	com.qz
160	19772390	281	0.000089	com.wiley
161	19772268	646	0.000046	au.net.abc
162	19770930	638	0.000046	edu.yale
163	19769582	428	0.000068	com.meetup
164	19767876	468	0.000064	com.ted
165	19761386	1160	0.000026	com.hatenablog
166	19759052	448	0.000066	com.patreon
167	19757472	283	0.000089	com.disqus
168	19756748	936	0.000032	edu.ucla
169	19753998	147	0.000195	com.dropbox
170	19753380	168	0.000158	com.yelp
171	19750678	271	0.000093	org.un
172	19746384	212	0.000122	com.twimg
173	19743118	254	0.000096	org.drupal
174	19741474	689	0.000042	org.bitbucket
175	19736540	422	0.000069	com.statista
176	19735440	903	0.000033	uk.ac.cam
177	19731940	718	0.000040	com.evernote
178	19731916	682	0.000043	com.newyorker
179	19725638	603	0.000049	com.buzzfeed
180	19719544	606	0.000049	me.about
181	19718654	722	0.000040	com.mysql
182	19716804	850	0.000035	com.thenextweb
183	19715420	495	0.000061	com.theatlantic
184	19710920	279	0.000091	com.sciencedirect
185	19710826	403	0.000073	com.getpocket
186	19705326	669	0.000043	uk.co.blogspot
187	19702126	1293	0.000023	com.tinypic
188	19696730	450	0.000066	com.booking
189	19695652	514	0.000058	com.xinhuanet
190	19694904	743	0.000039	org.weforum
191	19694268	247	0.000098	gov.ca
192	19692322	602	0.000049	gov.loc
193	19690998	1282	0.000023	org.postgresql
194	19689908	828	0.000036	edu.princeton
195	19687954	239	0.000103	uk.co.amazon
196	19685942	480	0.000063	com.dailymotion
197	19679672	1452	0.000021	ru.narod
198	19678926	189	0.000138	com.xing
199	19675914	879	0.000034	edu.jhu
200	19673670	500	0.000060	gov.whitehouse
201	19671846	665	0.000044	org.worldbank
202	19668706	1365	0.000022	org.eclipse
203	19667770	400	0.000073	com.springer
204	19667684	445	0.000067	com.nypost
205	19665872	316	0.000081	com.ft
206	19660930	61	0.000606	com.fb
207	19658986	204	0.000125	com.feedburner
208	19658394	826	0.000036	org.cambridge
209	19654762	476	0.000063	uk.co.dailymail
210	19654386	766	0.000038	edu.washington
211	19654242	496	0.000061	org.eff
212	19653044	32	0.001054	com.qq
213	19650144	473	0.000064	com.goodreads
214	19649524	264	0.000095	org.doi
215	19649502	512	0.000058	com.w3schools
216	19641242	1311	0.000023	edu.virginia
217	19641212	440	0.000067	com.googlecode
218	19638348	633	0.000047	com.vice
219	19633128	506	0.000059	com.force
220	19632976	723	0.000040	com.trello
221	19632780	836	0.000035	com.about
222	19630562	523	0.000056	com.inc
223	19629482	453	0.000066	com.scribd
224	19629368	2053	0.000016	com.wikidot
225	19628436	619	0.000048	org.semver
226	19614496	607	0.000049	com.cbsnews
227	19607794	651	0.000045	com.withgoogle
228	19605512	146	0.000196	me.line
229	19603410	2089	0.000016	com.googlesource
230	19601476	219	0.000118	org.iana
231	19601452	546	0.000054	gov.usda
232	19599800	309	0.000083	com.tinyurl
233	19598290	1090	0.000027	com.techradar
234	19597674	858	0.000035	com.dropboxusercontent
235	19597446	384	0.000076	com.ibm
236	19595200	1284	0.000023	co.elastic
237	19594024	289	0.000087	com.squareup
238	19593336	1434	0.000021	org.linuxfoundation
239	19592388	1134	0.000026	org.coursera
240	19589830	1027	0.000029	gov.fbi
241	19588284	1158	0.000026	edu.unc
242	19586008	705	0.000041	com.vox
243	19583350	193	0.000134	de.amazon
244	19583096	550	0.000054	uk.co.independent
245	19580554	1423	0.000021	ms.1drv
246	19578950	383	0.000076	com.digg
247	19567612	1393	0.000022	org.kernel
248	19563948	113	0.000287	com.sharethis
249	19563468	751	0.000039	org.d3js
250	19557490	801	0.000037	gov.fcc
251	19557292	1026	0.000029	com.hollywoodreporter
252	19556258	1369	0.000022	com.howstuffworks
253	19553700	430	0.000068	com.cnet
254	19552068	804	0.000037	com.foxnews
255	19547134	152	0.000183	com.addtoany
256	19547006	644	0.000046	com.indiatimes
257	19546928	995	0.000029	com.steamcommunity
258	19546864	1105	0.000026	cn.com.chinadaily
259	19545628	584	0.000050	com.psychologytoday
260	19544130	823	0.000036	uk.co.guardian
261	19543920	1463	0.000021	it.scoop
262	19543754	133	0.000247	com.mailchimp
263	19542234	837	0.000035	com.slate
264	19542214	153	0.000182	com.opera
265	19538412	589	0.000050	com.mckinsey
266	19536816	1020	0.000029	com.sap
267	19536418	2605	0.000013	org.wikiquote
268	19534334	307	0.000083	com.bitly
269	19533308	627	0.000047	com.mozilla
270	19533054	262	0.000095	jp.ameblo
271	19531260	735	0.000039	org.sciencemag
272	19528246	116	0.000284	com.paypalobjects
273	19528108	2345	0.000014	org.wikibooks
274	19527104	176	0.000151	com.amazon-adsystem
275	19526948	688	0.000042	gov.noaa
276	19524868	305	0.000083	com.netdna-ssl
277	19524544	310	0.000083	com.nbcnews
278	19523330	989	0.000030	com.target
279	19522776	1523	0.000020	com.instructables
280	19517526	975	0.000030	edu.umn
281	19516530	965	0.000031	com.merriam-webster
282	19516260	1431	0.000021	hk.com.google
283	19514852	185	0.000140	com.tripadvisor
284	19514608	2377	0.000014	com.diigo
285	19503916	497	0.000061	ca.google
286	19499262	236	0.000106	com.wpengine
287	19499246	1029	0.000028	com.sun
288	19496562	1189	0.000025	com.digitaltrends
289	19496340	391	0.000075	com.stumbleupon
290	19491846	115	0.000284	com.weibo
291	19491638	1626	0.000019	com.ign
292	19491210	1314	0.000023	com.mercurynews
293	19490964	1352	0.000022	de.zeit
294	19490636	229	0.000114	com.etsy
295	19489106	797	0.000037	uk.ac.ox
296	19487454	284	0.000089	com.optimizely
297	19485106	73	0.000425	net.akamaihd
298	19484368	1207	0.000025	net.speedtest
299	19484284	1522	0.000020	org.greenpeace
300	19483622	1553	0.000020	net.seesaa
301	19479450	720	0.000040	au.com.google
302	19478604	904	0.000033	de.spiegel
303	19476336	1077	0.000027	com.podbean
304	19475142	628	0.000047	org.pbs
305	19474722	516	0.000058	com.gofundme
306	19474484	416	0.000070	com.kickstarter
307	19473590	1340	0.000022	com.urbandictionary
308	19472422	472	0.000064	org.pewresearch
309	19471320	519	0.000057	com.bigcommerce
310	19467912	2137	0.000015	de.bild
311	19467240	231	0.000112	com.eepurl
312	19465300	515	0.000058	com.theverge
313	19464792	273	0.000092	com.stackoverflow
314	19464598	926	0.000032	com.politico
315	19463036	811	0.000036	co.ibb
316	19462394	332	0.000079	it.google
317	19462162	2110	0.000016	ly.visual
318	19461840	955	0.000031	org.unicef
319	19460932	2020	0.000016	org.tensorflow
320	19457592	1688	0.000018	com.itv
321	19457150	1013	0.000029	com.lifehacker
322	19456512	106	0.000334	com.stripe
323	19456272	1349	0.000022	edu.msu
324	19455412	312	0.000083	net.windows
325	19453374	805	0.000037	edu.academia
326	19450284	1391	0.000022	com.storify
327	19449638	1257	0.000024	com.crunchbase
328	19449386	595	0.000049	com.tandfonline
329	19449132	1958	0.000017	com.lego
330	19444682	1187	0.000025	com.jetbrains
331	19443796	677	0.000043	gov.senate
332	19443664	855	0.000035	com.chicagotribune
333	19443234	2301	0.000014	com.rottentomatoes
334	19440224	770	0.000038	ca.cbc
335	19439934	205	0.000125	com.eventbrite
336	19439496	1273	0.000023	hk.hku
337	19436402	1035	0.000028	edu.wisc
338	19436104	691	0.000042	com.libsyn
339	19435742	1051	0.000028	edu.northwestern
340	19433212	944	0.000031	com.scientificamerican
341	19432798	1043	0.000028	edu.uchicago
342	19431182	1288	0.000023	uk.co.wired
343	19425546	190	0.000137	jp.co.google
344	19424346	2002	0.000016	org.maven
345	19423732	1030	0.000028	com.mediafire
346	19423350	415	0.000070	me.telegram
347	19418440	396	0.000074	com.criteo
348	19417208	357	0.000076	fr.google
349	19417038	664	0.000044	us.icio
350	19416402	1477	0.000020	com.deadline
351	19415808	640	0.000046	com.sagepub
352	19414256	730	0.000039	com.ecwid
353	19413466	1275	0.000023	org.aclu
354	19413258	576	0.000051	com.typepad
355	19412168	471	0.000064	com.photobucket
356	19407294	533	0.000055	com.oup
357	19407168	1199	0.000025	com.reverbnation
358	19406968	1514	0.000020	de.mpg
359	19405330	1389	0.000022	edu.rutgers
360	19404790	1067	0.000027	com.scmp
361	19403976	81	0.000392	net.jsfiddle
362	19403692	421	0.000069	com.calendly
363	19403618	844	0.000035	com.sciencedaily
364	19403468	727	0.000039	gov.justice
365	19400830	575	0.000051	gov.hhs
366	19398258	919	0.000032	com.theconversation
367	19397596	991	0.000030	com.apnews
368	19397442	938	0.000032	com.huffpost
369	19394934	1518	0.000020	com.newscientist
370	19394656	608	0.000049	org.openstreetmap
371	19393300	1287	0.000023	com.aljazeera
372	19393230	216	0.000119	com.hubspot
373	19390018	645	0.000046	gov.house
374	19388118	2682	0.000012	uk.co.timesonline
375	19388034	2564	0.000013	com.space
376	19383910	700	0.000041	com.pinimg
377	19383504	432	0.000068	page.g
378	19381990	1241	0.000024	com.sky
379	19381844	866	0.000035	gov.congress
380	19381026	912	0.000033	com.500px
381	19380632	1217	0.000024	org.wiktionary
382	19380340	958	0.000031	com.ssrn
383	19379742	1709	0.000018	edu.bu
384	19377640	1757	0.000018	gov.cia
385	19375740	214	0.000120	org.bbb
386	19375634	1438	0.000021	com.foxbusiness
387	19371814	624	0.000047	ru.gov
388	19371056	1598	0.000019	ca.mcgill
389	19367926	790	0.000037	com.qualtrics
390	19366054	1290	0.000023	org.semanticscholar
391	19365778	761	0.000038	site.business
392	19365760	267	0.000094	ru.ok
393	19363798	977	0.000030	edu.si
394	19363758	887	0.000034	br.com.google
395	19363688	847	0.000035	co.g
396	19363204	1021	0.000029	uk.co.thetimes
397	19362122	2663	0.000012	com.discovermagazine
398	19359920	182	0.000142	us.zoom
399	19359492	889	0.000034	org.fao
400	19359352	683	0.000043	org.change
401	19357866	1469	0.000020	com.salon
402	19356650	228	0.000114	com.aliyuncs
403	19356280	997	0.000029	com.thehill
404	19354818	973	0.000030	gov.usgs
405	19351584	298	0.000085	com.ebay
406	19350988	1222	0.000024	com.nikkei
407	19350142	338	0.000078	com.rawgit
408	19349660	578	0.000051	it.placehold
409	19348824	157	0.000173	com.wixsite
410	19348122	1238	0.000024	com.smithsonianmag
411	19346552	758	0.000038	org.oecd
412	19346514	1088	0.000027	ee.linktr
413	19345254	3312	0.000011	com.openai
414	19342288	1048	0.000028	uk.co.mirror
415	19341656	679	0.000043	com.deviantart
416	19341332	1576	0.000019	org.phys
417	19340598	413	0.000070	tv.twitch
418	19340138	404	0.000072	com.mapbox
419	19335246	1546	0.000020	ca.sfu
420	19332464	2754	0.000012	com.instapaper
421	19330656	244	0.000100	org.gnu
422	19330504	2115	0.000016	au.edu.unimelb
423	19328724	1044	0.000028	int.coe
424	19328320	2078	0.000016	org.nobelprize
425	19328286	667	0.000043	pl.google
426	19327680	1333	0.000022	com.irishtimes
427	19327578	293	0.000086	com.office
428	19327536	1962	0.000017	org.torproject
429	19324936	484	0.000063	net.imgix
430	19324628	1281	0.000023	uk.ac.ucl
431	19320926	1054	0.000028	org.ohchr
432	19318772	1213	0.000025	com.strikingly
433	19315502	509	0.000059	org.hbr
434	19315040	1411	0.000021	uk.co.metro
435	19314304	123	0.000270	com.statcounter
436	19313468	972	0.000030	gov.dhs
437	19313380	287	0.000088	com.thedailybeast
438	19313234	1811	0.000017	com.bankofamerica
439	19312534	1265	0.000024	com.buzzsprout
440	19311940	863	0.000035	gov.nps
441	19309868	2426	0.000014	au.com.theage
442	19307472	933	0.000032	com.aweber
443	19306766	1557	0.000020	blog.home
444	19305448	848	0.000035	gov.bls
445	19305296	490	0.000062	edu.nyu
446	19304346	2087	0.000016	com.oxforddictionaries
447	19304074	1162	0.000025	gov.nyc
448	19303568	93	0.000356	org.reactjs
449	19302778	1382	0.000022	au.com.news
450	19300882	2291	0.000014	sg.edu.nus
451	19299900	1429	0.000021	com.flipboard
452	19299896	481	0.000063	com.scorecardresearch
453	19298010	2517	0.000013	com.dummies
454	19295840	2465	0.000013	org.rsc
455	19295472	1010	0.000029	com.britannica
456	19294984	714	0.000040	gov.state
457	19294216	1700	0.000018	org.gutenberg
458	19292892	3565	0.000010	fm.ask
459	19290866	2970	0.000011	com.pearltrees
460	19289990	793	0.000037	com.zapier
461	19286494	2562	0.000013	com.mystrikingly
462	19284092	876	0.000034	com.cctv
463	19283500	816	0.000036	com.healthline
464	19283044	1955	0.000017	com.chrome
465	19282638	1484	0.000020	com.rt
466	19282550	967	0.000031	com.newsweek
467	19280538	2362	0.000014	com.biography
468	19279646	1005	0.000029	ch.google
469	19270504	1412	0.000021	com.ifttt
470	19270238	1584	0.000019	com.axios
471	19270042	466	0.000065	es.google
472	19269658	882	0.000034	au.gov.nsw
473	19267444	3483	0.000010	hk.edu.cuhk
474	19267150	862	0.000035	com.stitcher
475	19267000	2520	0.000013	com.boredpanda
476	19265582	1192	0.000025	fr.lemonde
477	19263992	554	0.000053	com.steampowered
478	19263878	1055	0.000028	org.jstor
479	19262150	1335	0.000022	org.imf
480	19261918	873	0.000034	com.venturebeat
481	19261196	825	0.000036	org.poynter
482	19259574	1684	0.000018	com.straitstimes
483	19259452	3390	0.000010	com.chosun
484	19259322	1502	0.000020	edu.asu
485	19258762	2351	0.000014	io.gitlab
486	19256810	956	0.000031	ru.google
487	19255996	952	0.000031	sg.com.google
488	19253798	1331	0.000022	uk.co.standard
489	19252906	612	0.000048	de.gesetze-im-internet
490	19251516	948	0.000031	gov.archives
491	19250270	2385	0.000014	th.co.google
492	19249730	423	0.000069	io.codepen
493	19248930	3033	0.000011	com.nola
494	19248894	2023	0.000016	edu.gmu
495	19245246	2836	0.000012	app.netlify
496	19245158	1116	0.000026	com.wikia
497	19242656	1353	0.000022	com.history
498	19242160	1007	0.000029	com.thelancet
499	19241830	2918	0.000011	com.coca-colacompany
500	19240640	2654	0.000012	google.ai
501	19240600	856	0.000035	com.freepik
502	19240430	1548	0.000020	com.buzzfeednews
503	19238648	2894	0.000012	org.cato
504	19237700	431	0.000068	net.datatables
505	19237456	501	0.000060	com.rackcdn
506	19236168	1590	0.000019	gov.supremecourt
507	19233302	2534	0.000013	edu.byu
508	19233268	642	0.000046	fr.amazon
509	19232920	2872	0.000012	tw.blogspot
510	19231944	803	0.000037	in.co.google
511	19231530	1977	0.000017	org.edx
512	19231228	1309	0.000023	com.tunein
513	19231156	1779	0.000018	org.ocks
514	19230478	522	0.000057	nl.google
515	19228370	555	0.000053	com.gmail
516	19227068	2398	0.000014	com.nationalpost
517	19226910	1867	0.000017	edu.ucsb
518	19226418	2383	0.000014	edu.nd
519	19226392	1372	0.000022	com.dw
520	19226256	127	0.000262	com.jimdo
521	19225860	2412	0.000014	no.uio
522	19225400	1006	0.000029	google.blog
523	19222398	1409	0.000021	cn.cntv
524	19222164	3285	0.000011	cn.org.china
525	19221136	1639	0.000019	org.unwomen
526	19218950	946	0.000031	com.airtable
527	19217788	2510	0.000013	edu.uoregon
528	19215376	2172	0.000015	org.britishcouncil
529	19214674	2668	0.000012	org.icrc
530	19214462	951	0.000031	com.gallup
531	19213378	2265	0.000015	ru.kremlin
532	19212894	1332	0.000022	com.globalsign
533	19210850	875	0.000034	gov.uspto
534	19210492	959	0.000031	edu.psu
535	19210022	1509	0.000020	com.penguinrandomhouse
536	19209318	1345	0.000022	com.netdna-cdn
537	19208686	3269	0.000011	is.archive
538	19208344	1531	0.000020	uk.ac.lse
539	19207952	2503	0.000013	fi.helsinki
540	19207620	2042	0.000016	edu.pitt
541	19207236	2170	0.000015	net.openid
542	19206256	1155	0.000026	edu.brookings
543	19205290	786	0.000037	com.imageshack
544	19204770	172	0.000152	com.npmjs
545	19204486	3290	0.000011	de.diplo
546	19204380	1956	0.000017	edu.unl
547	19203832	1544	0.000020	edu.georgetown
548	19203210	2125	0.000015	org.metmuseum
549	19202750	1240	0.000024	org.nejm
550	19202244	726	0.000040	com.adage
551	19200434	1990	0.000017	com.channel4
552	19200290	1511	0.000020	com.findlaw
553	19200030	2224	0.000015	com.france24
554	19198938	282	0.000089	net.php
555	19198698	1784	0.000017	com.csmonitor
556	19197866	419	0.000069	com.proofpoint
557	19195320	192	0.000135	com.iubenda
558	19194372	1011	0.000029	gov.treasury
559	19194028	1708	0.000018	com.euronews
560	19191446	2286	0.000014	com.thoughtco
561	19190136	3742	0.000009	com.doodlekit
562	19189862	107	0.000320	com.godaddy
563	19189334	1298	0.000023	edu.duke
564	19188652	2071	0.000016	com.foreignpolicy
565	19185118	1996	0.000017	org.documentcloud
566	19183756	1300	0.000023	com.livescience
567	19183706	2508	0.000013	com.upi
568	19183104	2085	0.000016	com.gq
569	19182260	178	0.000148	com.zendesk
570	19182074	3020	0.000011	com.authorstream
571	19182074	3915	0.000009	com.mysanantonio
572	19181694	4133	0.000008	tw.edu.sinica
573	19177894	2719	0.000012	org.wikisource
574	19177382	2220	0.000015	com.insider
575	19177180	851	0.000035	gov.nist
576	19177000	1625	0.000019	com.thestar
577	19176642	181	0.000145	jp.co.yahoo
578	19174546	1304	0.000023	au.com.smh
579	19174028	2025	0.000016	org.ncsl
580	19173800	4252	0.000008	hk.edu.cityu
581	19173744	3349	0.000010	com.sina
582	19173108	2197	0.000015	ie.independent
583	19172266	2156	0.000015	edu.uky
584	19171704	96	0.000349	me.ogp
585	19170936	3413	0.000010	uk.ac.sussex
586	19170792	1755	0.000018	gov.doc
587	19170704	131	0.000250	org.networkadvertising
588	19169566	320	0.000080	io.shields
589	19168058	649	0.000045	gov.usa
590	19166990	4291	0.000008	org.china-embassy
591	19166810	3137	0.000011	com.udn
592	19163774	161	0.000166	ru.mail
593	19163712	3474	0.000010	com.worldatlas
594	19163522	505	0.000060	com.netflix
595	19163254	857	0.000035	com.thinkwithgoogle
596	19162356	1441	0.000021	gov.defense
597	19161952	1318	0.000023	tw.com.google
598	19160826	1604	0.000019	org.hrw
599	19159812	1495	0.000020	com.asahi
600	19159570	785	0.000037	io.readthedocs
601	19158768	2688	0.000012	org.freedomhouse
602	19158654	1413	0.000021	tv.ustream
603	19157822	893	0.000034	org.mediawiki
604	19156446	1715	0.000018	org.pypi
605	19151800	3028	0.000011	org.adb
606	19151406	2099	0.000016	fr.leparisien
607	19151152	2615	0.000013	com.abc7news
608	19150650	2063	0.000016	com.voanews
609	19150048	1019	0.000029	com.pcmag
610	19148698	447	0.000067	org.nodejs
611	19148554	4288	0.000008	com.theundefeated
612	19147816	3860	0.000009	org.gephi
613	19147176	1327	0.000023	org.undp
614	19146462	3277	0.000011	org.iucnredlist
615	19146454	2583	0.000013	com.sacbee
616	19146204	1594	0.000019	com.treehugger
617	19145608	2292	0.000014	no.google
618	19144462	2471	0.000013	co.ello
619	19143354	1986	0.000017	com.msnbc
620	19143354	252	0.000097	com.myshopify
621	19142810	981	0.000030	uk.parliament
622	19142520	2287	0.000014	co.pcdn
623	19141942	1255	0.000024	gov.uscourts
624	19141896	1422	0.000021	co.lpages
625	19140780	2344	0.000014	org.fas
626	19139768	781	0.000037	com.intel
627	19138740	807	0.000036	com.marketwatch
628	19136914	2047	0.000016	com.infogram
629	19133848	2538	0.000013	com.sputniknews
630	19133704	2430	0.000014	ie.google
631	19132582	1344	0.000022	se.google
632	19131798	990	0.000030	com.netlify
633	19131000	925	0.000032	com.jekyllrb
634	19130612	3055	0.000011	int.interpol
635	19130308	524	0.000056	fr.free
636	19130180	1198	0.000025	be.google
637	19129750	1575	0.000019	uk.co.huffingtonpost
638	19129310	2323	0.000014	ly.rebrand
639	19129104	1504	0.000020	link.page
640	19128704	1794	0.000017	com.sched
641	19127724	2218	0.000015	jp.co.japantimes
642	19127254	2829	0.000012	org.tigris
643	19127152	2839	0.000012	org.pri
644	19127006	2319	0.000014	nz.co.nzherald
645	19125622	1204	0.000025	at.google
646	19125464	5292	0.000007	org.arkive
647	19125326	222	0.000116	com.salesforce
648	19123296	650	0.000045	br.com.uol
649	19121018	4242	0.000008	kr.co.kbs
650	19119374	1665	0.000018	com.thebalance
651	19119126	1455	0.000021	org.oxfordjournals
652	19118638	3738	0.000009	com.encyclopedia
653	19117262	2204	0.000015	org.eji
654	19116506	2818	0.000012	org.heritage
655	19116298	2371	0.000014	com.popsci
656	19114518	2199	0.000015	com.snopes
657	19114098	2601	0.000013	org.oas
658	19113348	156	0.000174	com.aspnetcdn
659	19112712	1031	0.000028	org.ilo
660	19109654	2263	0.000015	com.insidehighered
661	19108980	1587	0.000019	gov.usembassy
662	19108932	1622	0.000019	dk.google
663	19108040	3392	0.000010	org.jenkins-ci
664	19107388	2827	0.000012	org.project-syndicate
665	19106556	1963	0.000017	com.justia
666	19104120	1563	0.000019	gov.govinfo
667	19103152	1699	0.000018	com.firebaseapp
668	19102068	2093	0.000016	edu.uga
669	19102028	3678	0.000010	edu.wm
670	19101614	3284	0.000011	com.cgtn
671	19101596	1881	0.000017	org.worldcat
672	19101226	900	0.000033	com.zoho
673	19100590	392	0.000074	com.atlassian
674	19100290	2676	0.000012	org.transparency
675	19099776	1317	0.000023	org.aarp
676	19099686	1675	0.000018	org.americanbar
677	19099164	2239	0.000015	com.timeshighereducation
678	19097964	3270	0.000011	com.pastemagazine
679	19095902	2598	0.000013	org.csis
680	19094342	629	0.000047	com.samsung
681	19094058	774	0.000038	com.pexels
682	19093374	1964	0.000017	com.washingtontimes
683	19092714	2016	0.000016	gov.usaid
684	19090166	1334	0.000022	org.heart
685	19088764	191	0.000136	com.automattic
686	19088428	865	0.000035	com.verisign
687	19087660	2108	0.000016	com.motherjones
688	19087034	2944	0.000011	org.vim
689	19086498	2062	0.000016	edu.nap
690	19086172	924	0.000032	com.webs
691	19084778	1593	0.000019	org.amnesty
692	19084344	2101	0.000016	ua.com.google
693	19083552	3988	0.000009	org.globalnetworkinitiative
694	19083196	2546	0.000013	org.globalcitizen
695	19082500	1754	0.000018	com.surveygizmo
696	19082058	2262	0.000015	org.wbur
697	19081048	2353	0.000014	uk.gov.companieshouse
698	19080398	2468	0.000013	jp.mainichi
699	19080286	3181	0.000011	com.podomatic
700	19078116	1751	0.000018	org.unhcr
701	19076276	2118	0.000016	ca.ctvnews
702	19075310	2565	0.000013	uk.co.bbci
703	19073812	968	0.000031	uk.gov.legislation
704	19071522	2681	0.000012	com.nationalreview
705	19070832	2523	0.000013	com.cleveland
706	19070474	3814	0.000009	org.neocities
707	19069884	1073	0.000027	ly.snip
708	19068864	438	0.000067	com.herokuapp
709	19068510	656	0.000045	com.oreilly
710	19066730	1154	0.000026	cz.google
711	19066464	2164	0.000015	org.nrdc
712	19065768	2671	0.000012	org.thinkprogress
713	19065654	1795	0.000017	ca.globalnews
714	19065106	270	0.000093	jp.co.amazon
715	19062840	1328	0.000023	org.altervista
716	19061732	3119	0.000011	uk.ac.nottingham
717	19061168	1267	0.000024	uk.gov.nationalarchives
718	19060934	2106	0.000016	au.edu.anu
719	19060236	3035	0.000011	com.intensedebate
720	19060102	2734	0.000012	de.hu-berlin
721	19059802	736	0.000039	com.airbnb
722	19059800	2326	0.000014	de.auswaertiges-amt
723	19059376	2316	0.000014	nz.co.google
724	19059170	2672	0.000012	org.unenvironment
725	19058978	3132	0.000011	org.rsf
726	19057932	4110	0.000008	com.koreaherald
727	19057778	1960	0.000017	org.pewtrusts
728	19057678	2867	0.000012	com.techinasia
729	19057488	2276	0.000014	com.thecut
730	19056174	3700	0.000009	com.viki
731	19056068	2724	0.000012	org.gnupg
732	19054590	2469	0.000013	ro.google
733	19054394	2057	0.000016	edu.gwu
734	19054116	3057	0.000011	com.bangkokpost
735	19053626	2572	0.000013	fr.rfi
736	19052868	414	0.000070	com.pubmatic
737	19051906	2309	0.000014	com.tutsplus
738	19051648	1079	0.000027	tr.com.google
739	19051516	248	0.000098	com.getbootstrap
740	19050908	4424	0.000008	com.wonderhowto
741	19050626	3619	0.000010	com.upworthy
742	19050496	2883	0.000012	org.sonatype
743	19050382	288	0.000087	com.typeform
744	19049574	2806	0.000012	il.co.google
745	19049384	2739	0.000012	uk.ac.leeds
746	19048116	201	0.000127	to.amzn
747	19047986	2703	0.000012	vn.com.google
748	19047578	274	0.000092	com.surveymonkey
749	19047380	922	0.000032	int.wipo
750	19046288	1057	0.000028	com.gizmodo
751	19046144	874	0.000034	com.box
752	19045578	2298	0.000014	com.oregonlive
753	19044916	547	0.000054	gg.discord
754	19044444	3356	0.000010	com.theepochtimes
755	19044400	2480	0.000013	ar.com.google
756	19044144	2943	0.000011	bg.google
757	19043632	2061	0.000016	com.squarespace-cdn
758	19043400	3479	0.000010	io.soup
759	19042778	2545	0.000013	com.webbyawards
760	19042384	2744	0.000012	io.fabric
761	19042298	1588	0.000019	com.speakerdeck
762	19041684	136	0.000232	info.aboutads
763	19040606	907	0.000033	com.docker
764	19038814	1817	0.000017	com.miamiherald
765	19037924	3191	0.000011	ph.com.google
766	19037762	2463	0.000013	com.channelnewsasia
767	19037556	3198	0.000011	uk.co.vogue
768	19037554	2619	0.000013	edu.fsu
769	19035870	485	0.000063	com.staticflickr
770	19035284	2495	0.000013	za.co.google
771	19033678	2696	0.000012	com.thejakartapost
772	19032442	1236	0.000024	edu.ucsd
773	19032258	487	0.000062	com.fc2
774	19032038	5415	0.000007	com.armorgames
775	19031944	2155	0.000015	fi.google
776	19031234	3885	0.000009	com.alamy
777	19030868	2221	0.000015	id.co.google
778	19030462	2794	0.000012	com.rd
779	19029712	2951	0.000011	com.cartodb
780	19029584	2092	0.000016	com.newrepublic
781	19029348	3436	0.000010	com.benzinga
782	19028364	661	0.000044	com.entrepreneur
783	19027960	5376	0.000007	org.gwtproject
784	19026660	2988	0.000011	com.sciencealert
785	19026538	2763	0.000012	org.iaea
786	19026402	2376	0.000014	com.thenation
787	19023692	3411	0.000010	si.google
788	19023046	2400	0.000014	pt.google
789	19020124	2965	0.000011	au.gov.nla
790	19019838	3513	0.000010	com.dailykos
791	19019756	494	0.000061	com.aol
792	19019128	2519	0.000013	edu.emory
793	19019012	3573	0.000010	com.inhabitat
794	19018956	3415	0.000010	uk.ac.soas
795	19018402	666	0.000044	com.deloitte
796	19018230	1185	0.000025	com.today
797	19016838	978	0.000030	com.windowsphone
798	19016186	3659	0.000010	org.cpj
799	19016164	2119	0.000016	kr.co.google
800	19015906	2981	0.000011	se.lu
801	19015780	2774	0.000012	org.cfr
802	19014856	429	0.000068	me.fb
803	19013678	3288	0.000011	com.joins
804	19012980	4264	0.000008	sa.com.google
805	19012878	2814	0.000012	com.politifact
806	19012292	964	0.000031	com.alexa
807	19011442	4131	0.000008	edu.utm
808	19011068	2735	0.000012	com.law360
809	19010546	983	0.000030	com.engadget
810	19008662	3583	0.000010	hr.google
811	19008538	2146	0.000015	hu.google
812	19006860	631	0.000047	fm.last
813	19006540	2476	0.000013	eu.politico
814	19006248	4047	0.000009	com.chinatimes
815	19006116	2521	0.000013	mx.com.google
816	19006060	3141	0.000011	com.jezebel
817	19005942	3868	0.000009	com.iconarchive
818	19005318	3471	0.000010	com.ogilvy
819	19004866	2399	0.000014	gr.google
820	19004086	2816	0.000012	com.monday
821	19003252	2738	0.000012	com.digitaljournal
822	19003248	3149	0.000011	com.nyt
823	19003220	3300	0.000011	audio.breaker
824	19002640	2823	0.000012	uk.co.guim
825	19002384	625	0.000047	com.cisco
826	19002038	3391	0.000010	cn.globaltimes
827	19001808	2648	0.000012	com.instructure
828	19000646	3321	0.000011	com.crashlytics
829	18999720	2723	0.000012	au.com.businessinsider
830	18999338	3430	0.000010	org.grist
831	18998280	1209	0.000025	com.pastebin
832	18998118	315	0.000082	ai.shortpixel
833	18998078	3990	0.000009	org.constitutioncenter
834	18997960	4842	0.000007	jp.hatenadiary
835	18996780	3770	0.000009	edu.ttu
836	18996076	2997	0.000011	uk.ac.york
837	18995936	1671	0.000018	com.eater
838	18995084	90	0.000364	com.livestream
839	18995036	2772	0.000012	com.bepress
840	18994752	2898	0.000012	org.wri
841	18992262	2043	0.000016	my.com.thestar
842	18991122	3775	0.000009	com.minds
843	18990592	2352	0.000014	mp.j
844	18990570	3708	0.000009	app.web
845	18990062	3410	0.000010	org.carnegieendowment
846	18989786	3645	0.000010	tr.com.aa
847	18989418	711	0.000041	gov.sec
848	18987746	3812	0.000009	com.hyperallergic
849	18987282	3408	0.000010	com.foreignaffairs
850	18986640	3797	0.000009	au.edu.uts
851	18985392	470	0.000064	com.fastcompany
852	18985032	3560	0.000010	org.hypotheses
853	18984468	3896	0.000009	com.japantoday
854	18982752	3507	0.000010	edu.wayne
855	18982048	3713	0.000009	uk.ac.kent
856	18981988	3697	0.000009	rs.google
857	18980532	4071	0.000009	org.sourcewatch
858	18979366	832	0.000036	com.symantec
859	18978424	2539	0.000013	fr.paris
860	18977996	2942	0.000011	com.prweek
861	18977902	1765	0.000018	ch.ipcc
862	18976960	2217	0.000015	com.kinstacdn
863	18976262	1046	0.000028	edu.cmu
864	18975462	2039	0.000016	int.unfccc
865	18975062	4196	0.000008	eg.com.google
866	18974804	3180	0.000011	org.nationalgeographic
867	18974548	2643	0.000013	gov.doi
868	18973940	3406	0.000010	de.uni-frankfurt
869	18973494	4243	0.000008	by.google
870	18972022	5050	0.000007	com.symbaloo
871	18971010	3417	0.000010	nl.wur
872	18969950	2328	0.000014	org.unodc
873	18968430	1599	0.000019	com.routledge
874	18968412	4509	0.000008	com.ipsos-mori
875	18966962	3658	0.000010	ae.google
876	18966152	4482	0.000008	com.etymonline
877	18965888	4982	0.000007	build.bazel
878	18965566	3320	0.000011	org.brainpickings
879	18964544	3143	0.000011	com.scotsman
880	18963796	4295	0.000008	com.oilprice
881	18963380	3597	0.000010	uk.ac.westminster
882	18963266	4545	0.000008	lk.google
883	18962576	1260	0.000024	fr.blogspot
884	18961360	3412	0.000010	org.rferl
885	18961310	3173	0.000011	org.epi
886	18959900	4115	0.000008	lv.google
887	18959812	3909	0.000009	au.edu.griffith
888	18959422	4219	0.000008	kr.ac.snu
889	18957280	1312	0.000023	com.upwork
890	18957076	2436	0.000014	com.html5rocks
891	18956714	5493	0.000007	me.nimbusweb
892	18956502	2940	0.000011	fr.archives-ouvertes
893	18956398	4293	0.000008	com.delawareonline
894	18955462	1792	0.000017	ru.rbc
895	18954968	745	0.000039	com.gartner
896	18954930	1127	0.000026	edu.utexas
897	18953642	2526	0.000013	net.noscript
898	18953466	2717	0.000012	ae.thenational
899	18953336	3380	0.000010	com.study
900	18953092	427	0.000068	com.hp
901	18953074	3641	0.000010	uk.co.spectator
902	18952762	3869	0.000009	com.cleantechnica
903	18952208	2803	0.000012	org.unctad
904	18951200	4255	0.000008	com.teslamotors
905	18950118	1614	0.000019	com.billboard
906	18949366	3074	0.000011	com.theculturetrip
907	18947896	2454	0.000013	com.multiscreensite
908	18947738	704	0.000041	com.visualstudio
909	18947588	3985	0.000009	uk.ac.plymouth
910	18947454	2660	0.000012	sk.google
911	18947312	3811	0.000009	net.aljazeera
912	18947110	2413	0.000014	com.theintercept
913	18946556	3421	0.000010	uk.ac.exeter
914	18946494	3332	0.000010	social.mastodon
915	18945876	2828	0.000012	com.euractiv
916	18945864	3635	0.000010	com.db
917	18942736	4447	0.000008	org.mises
918	18942316	4680	0.000008	ng.com.google
919	18942016	2795	0.000012	org.panda
920	18941622	2466	0.000013	uk.gov.justice
921	18941430	5602	0.000007	net.chinadialogue
922	18940924	4118	0.000008	cat.uab
923	18940746	4227	0.000008	com.spokesman
924	18940082	3523	0.000010	co.com.google
925	18939230	4473	0.000008	lu.google
926	18938996	4189	0.000008	pe.com.google
927	18938618	3366	0.000010	com.nybooks
928	18938606	4381	0.000008	uk.ac.core
929	18938206	2228	0.000015	com.termsfeed
930	18938194	1669	0.000018	com.pcworld
931	18938112	3846	0.000009	kr.co.yna
932	18938002	4793	0.000007	com.gust
933	18937788	3880	0.000009	org.cgiar
934	18937300	4231	0.000008	pk.com.google
935	18936530	3575	0.000010	net.inquirer
936	18936008	3083	0.000011	ru.lenta
937	18934000	1468	0.000020	com.nokia
938	18933676	2932	0.000011	tw.com.pchome
939	18933496	1223	0.000024	com.ycombinator
940	18933350	2911	0.000011	nl.volkskrant
941	18933194	78	0.000411	com.oculus
942	18932612	3455	0.000010	cl.google
943	18931862	3949	0.000009	org.polymer-project
944	18930888	2637	0.000013	com.washingtonexaminer
945	18930622	3945	0.000009	sk.sme
946	18930534	3389	0.000010	edu.monash
947	18930086	918	0.000032	com.canva
948	18929552	454	0.000066	org.opensource
949	18929398	3977	0.000009	com.rappler
950	18928630	4000	0.000009	org.plan-international
951	18926518	4561	0.000008	cr.co.google
952	18926412	3587	0.000010	lt.google
953	18925832	3810	0.000009	ca.macleans
954	18925646	817	0.000036	net.adform
955	18925046	4873	0.000007	com.blogto
956	18924952	3508	0.000010	uk.ac.nhm
957	18924928	3211	0.000011	edu.ua
958	18923554	2815	0.000012	com.articulate
959	18923288	249	0.000098	com.sxsw
960	18922866	3993	0.000009	org.wilsoncenter
961	18922676	4082	0.000009	edu.lehigh
962	18922336	417	0.000070	com.skype
963	18921546	4699	0.000008	com.out
964	18920714	1085	0.000027	com.redhat
965	18920680	3266	0.000011	my.com.google
966	18919064	2031	0.000016	gov.ecfr
967	18918900	4585	0.000008	org.nsidc
968	18918778	412	0.000070	net.secureservercdn
969	18918112	4536	0.000008	kz.google
970	18917590	3295	0.000011	org.osce
971	18917562	557	0.000053	org.whatwg
972	18917418	4096	0.000009	com.wsoctv
973	18917380	2587	0.000013	uk.org.nationaltrust
974	18917220	3201	0.000011	uk.gov.london
975	18917048	1973	0.000017	scot.gov
976	18916982	3865	0.000009	uk.ac.qub
977	18916460	3807	0.000009	com.governing
978	18916430	528	0.000056	com.businesswire
979	18916300	2253	0.000015	wales.gov
980	18915066	3422	0.000010	com.afp
981	18914982	3080	0.000011	uk.ac.qmul
982	18914878	5154	0.000007	com.ingress
983	18914540	4596	0.000008	com.webcindario
984	18914316	3402	0.000010	org.psychiatryonline
985	18913230	4148	0.000008	org.marxists
986	18913096	4073	0.000009	me.thinglink
987	18912970	1660	0.000018	com.css-tricks
988	18912858	4732	0.000008	ie.nuigalway
989	18912514	4348	0.000008	com.asiaone
990	18912368	3354	0.000010	com.kaspersky-labs
991	18912110	1249	0.000024	com.smashingmagazine
992	18912064	3787	0.000009	org.nationalinterest
993	18911848	556	0.000053	com.adweek
994	18911436	4498	0.000008	ec.com.google
995	18911404	4722	0.000008	bd.com.google
996	18910006	4846	0.000007	uy.com.google
997	18909998	4233	0.000008	com.match
998	18909746	4021	0.000009	ee.google
999	18909688	3962	0.000009	com.adn
1000	18909474	4310	0.000008	com.wnd

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

September 2020 crawl archive now available

October 7, 2020Sebastian Nagel

The crawl archive for September 2020 is now available! The data was crawled between September 18th and October 2nd and contains 3.45 billion web pages or 345 TiB of uncompressed content. It includes page captures of 1.5 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The September crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-40/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-40/segment.paths.gz	100
WARC files	CC-MAIN-2020-40/warc.paths.gz	79600	81.8
WAT files	CC-MAIN-2020-40/wat.paths.gz	79600	23.14
WET files	CC-MAIN-2020-40/wet.paths.gz	79600	10.28
Robots.txt files	CC-MAIN-2020-40/robotstxt.paths.gz	79600	0.22
Non-200 responses files	CC-MAIN-2020-40/non200responses.paths.gz	79600	2.36
URL index files	CC-MAIN-2020-40/cc-index.paths.gz	302	0.27

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-40/. Also the columnar index has been updated to contain this crawl.

August 2020 crawl archive now available

August 19, 2020Sebastian Nagel

The crawl archive for August 2020 is now available! It contains 2.45 billion web pages or 235 TiB of uncompressed content, crawled between August 2nd and 15th. It includes page captures of 940 million URLs unknown in any of our prior crawl archives.

Archive Location and Download

The August crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-34/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-34/segment.paths.gz	100
WARC files	CC-MAIN-2020-34/warc.paths.gz	60000	48.9
WAT files	CC-MAIN-2020-34/wat.paths.gz	60000	16.9
WET files	CC-MAIN-2020-34/wet.paths.gz	60000	7.56
Robots.txt files	CC-MAIN-2020-34/robotstxt.paths.gz	60000	0.19
Non-200 responses files	CC-MAIN-2020-34/non200responses.paths.gz	60000	1.94
URL index files	CC-MAIN-2020-34/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-34/. Also the columnar index has been updated to contain this crawl.