Blog – Common Crawl

March/April 2020 crawl archive now available

April 14, 2020Sebastian Nagel

The crawl archive for March/April 2020 is now available! It contains 2.85 billion web pages or 280 TiB of uncompressed content, crawled between March 28th and April 10th. It includes page captures of 1 billion URLs unknown in any of our prior crawl archives.

Archive Location and Download

The March/April crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-16/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line, you end up with the S3 and HTTP paths respectively.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-16/segment.paths.gz	100
WARC files	CC-MAIN-2020-16/warc.paths.gz	56000	62.67
WAT files	CC-MAIN-2020-16/wat.paths.gz	56000	20.37
WET files	CC-MAIN-2020-16/wet.paths.gz	56000	8.97
Robots.txt files	CC-MAIN-2020-16/robotstxt.paths.gz	56000	0.19
Non-200 responses files	CC-MAIN-2020-16/non200responses.paths.gz	56000	1.39
URL index files	CC-MAIN-2020-16/cc-index.paths.gz	302	0.21

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-16/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

February 2020 crawl archive now available

March 4, 2020Sebastian Nagel

The crawl archive for February 2020 is now available! It contains 2.6 billion web pages or 240 TiB of uncompressed content, crawled between February 16th and 29th. It includes page captures of 1 billion URLs unknown in any of our prior crawl archives.

Improvements and Fixes

The HTTP headers in WARC response records have been fixed: the HTTP response status line now has a white space following the status code if the reason-phrase is empty. E.g., if a server sends an empty message (instead of “OK”), the status line will include a trailing space character: “HTTP/1.1 200 ”. Following RFC 7230 the white space between status code and message is mandatory. Please refer to the bug report NUTCH-2763 for further details.

Archive Location and Download

The February crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-10/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-10/segment.paths.gz	100
WARC files	CC-MAIN-2020-10/warc.paths.gz	56000	49.28
WAT files	CC-MAIN-2020-10/wat.paths.gz	56000	17.98
WET files	CC-MAIN-2020-10/wet.paths.gz	56000	7.97
Robots.txt files	CC-MAIN-2020-10/robotstxt.paths.gz	56000	0.22
Non-200 responses files	CC-MAIN-2020-10/non200responses.paths.gz	56000	2.21
URL index files	CC-MAIN-2020-10/cc-index.paths.gz	302	0.2

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-10/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs Nov/Dec/Jan 2019 – 2020

February 10, 2020Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, December 2019 and January 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

Host-level graph

The graph consists of 1.24 billion nodes and 4.54 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 1.17 billion dangling nodes (95%) and the largest strongly connected component contains 45 million (3.6%) nodes.

You can download the graph and the ranks of all 1.24 billion hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-20-nov-dec-jan/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-20-nov-dec-jan/host/ as prefix to access the files from everywhere.

Download files of the Common Crawl Nov/Dec/Jan 2019-20 domain-level webgraph

Size	File	Description
7.23 GB	cc-main-2019-20-nov-dec-jan-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 12 vertices files
20.16 GB	cc-main-2019-20-nov-dec-jan-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 24 edges files

8.42 GB	cc-main-2019-20-nov-dec-jan-host.graph	graph in BVGraph format
2 kB	cc-main-2019-20-nov-dec-jan-host.properties

10.80 GB	cc-main-2019-20-nov-dec-jan-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2019-20-nov-dec-jan-host-t.properties

1 kB	cc-main-2019-20-nov-dec-jan-host.stats	WebGraph statistics

16.32 GB	cc-main-2019-20-nov-dec-jan-host-ranks.txt.gz	harmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 85.8 million nodes and 1.9 billion edges. 51% or 44 million nodes are dangling nodes, the largest strongly connected component covers 34 million or 39% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-20-nov-dec-jan/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-20-nov-dec-jan/domain/.

Download files of the Common Crawl Nov/Dec/Jan 2019-20 domain-level webgraph

Size	File	Description
0.59 GB	cc-main-2019-20-nov-dec-jan-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
7.65 GB	cc-main-2019-20-nov-dec-jan-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

4.10 GB	cc-main-2019-20-nov-dec-jan-domain.graph	graph in BVGraph format
2 kB	cc-main-2019-20-nov-dec-jan-domain.properties

4.13 GB	cc-main-2019-20-nov-dec-jan-domain-t.graph	transpose of the graph
2 kB	cc-main-2019-20-nov-dec-jan-domain-t.properties

1 kB	cc-main-2019-20-nov-dec-jan-domain.stats	WebGraph statistics

1.86 GB	cc-main-2019-20-nov-dec-jan-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 86 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Nov/Dec/Jan 2019-2020)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	30598398	1	0.019072	com.googleapis
2	29113136	3	0.012214	com.facebook
3	27475138	2	0.013236	com.google
4	25610480	4	0.007452	com.twitter
5	24947126	5	0.007174	org.w
6	24904712	6	0.006611	com.youtube
7	23281504	9	0.004269	com.instagram
8	22446296	7	0.005561	org.gmpg
9	22154750	8	0.005033	com.googletagmanager
10	22107784	13	0.003001	com.linkedin
11	21307220	10	0.003433	org.wordpress
12	21290688	20	0.001717	com.gravatar
13	21096944	11	0.003266	com.cloudflare
14	21086168	23	0.001516	com.pinterest
15	20869868	16	0.002242	com.gstatic
16	20855286	15	0.002366	com.wordpress
17	20713268	25	0.001234	org.wikipedia
18	20641712	17	0.002130	com.apple
19	20584368	14	0.002460	com.bootstrapcdn
20	20371046	33	0.001102	com.vimeo
21	20336732	42	0.000833	com.blogspot
22	20237058	18	0.001787	com.jquery
23	20220572	50	0.000732	be.youtu
24	20218806	32	0.001130	com.microsoft
25	20119820	49	0.000737	com.wp
26	20059828	19	0.001764	com.adobe
27	20029544	52	0.000709	com.amazon
28	20013346	44	0.000784	gl.goo
29	19972608	35	0.001020	com.amazonaws
30	19941934	67	0.000471	com.tumblr
31	19937390	64	0.000518	ly.bit
32	19862906	29	0.001164	com.macromedia
33	19817494	30	0.001151	com.baidu
34	19816382	38	0.000958	com.google-analytics
35	19808066	31	0.001142	com.googlesyndication
36	19781460	34	0.001092	net.cloudfront
37	19770754	24	0.001254	ru.yandex
38	19749302	53	0.000693	com.flickr
39	19700112	22	0.001568	com.github
40	19698814	80	0.000368	com.yahoo
41	19676534	58	0.000644	eu.europa
42	19632506	115	0.000287	com.reddit
43	19603190	41	0.000918	com.addthis
44	19561006	72	0.000403	com.weebly
45	19559908	43	0.000823	org.w3
46	19550116	63	0.000524	me.wp
47	19546614	108	0.000313	com.googleusercontent
48	19531876	45	0.000777	io.github
49	19523224	182	0.000140	org.wikimedia
50	19521602	70	0.000422	com.medium
51	19516098	47	0.000743	org.schema
52	19496222	46	0.000754	net.jsdelivr
53	19495662	76	0.000374	org.creativecommons
54	19472610	173	0.000153	com.imgur
55	19459848	36	0.000989	net.doubleclick
56	19451512	51	0.000711	com.wix
57	19416054	186	0.000138	uk.co.bbc
58	19408312	149	0.000181	com.forbes
59	19405426	93	0.000332	com.weibo
60	19404990	60	0.000601	co.t
61	19398622	28	0.001192	com.fontawesome
62	19382484	48	0.000741	com.paypal
63	19375914	210	0.000114	com.cnn
64	19372292	144	0.000190	org.archive
65	19368680	61	0.000583	org.mozilla
66	19348182	189	0.000137	net.sourceforge
67	19304774	315	0.000079	edu.mit
68	19302936	174	0.000152	com.theguardian
69	19293782	278	0.000089	edu.harvard
70	19283372	179	0.000144	com.bing
71	19271390	117	0.000286	com.jimdo
72	19270368	140	0.000202	com.nytimes
73	19260752	27	0.001195	com.qq
74	19259406	237	0.000103	com.wsj
75	19257754	21	0.001581	org.apache
76	19256254	56	0.000658	com.googleadservices
77	19241110	215	0.000111	com.washingtonpost
78	19237816	269	0.000092	com.bloomberg
79	19237164	259	0.000094	com.techcrunch
80	19210562	544	0.000049	com.deviantart
81	19205946	162	0.000162	org.ietf
82	19198346	247	0.000098	com.oracle
83	19194696	279	0.000089	com.android
84	19193448	69	0.000430	com.list-manage
85	19174968	397	0.000065	com.ted
86	19167820	321	0.000078	com.reuters
87	19161964	312	0.000080	com.wired
88	19158768	154	0.000173	com.wixsite
89	19154874	341	0.000073	com.ft
90	19152068	330	0.000076	uk.co.telegraph
91	19151666	427	0.000060	com.theverge
92	19145078	155	0.000172	gov.nih
93	19143886	272	0.000091	com.myspace
94	19142172	377	0.000068	gov.nasa
95	19136022	291	0.000085	com.bbc
96	19126138	339	0.000074	com.example
97	19119916	239	0.000103	org.python
98	19117868	82	0.000361	com.whatsapp
99	19111550	122	0.000267	com.unpkg
100	19110346	188	0.000138	uk.co.google
101	19089504	707	0.000041	com.economist
102	19088766	255	0.000095	com.appspot
103	19082300	384	0.000067	uk.co.dailymail
104	19081904	209	0.000115	org.gnu
105	19081708	262	0.000093	com.githubusercontent
106	19080182	131	0.000232	com.ytimg
107	19077156	320	0.000078	org.un
108	19075806	163	0.000162	com.giphy
109	19068860	398	0.000065	com.latimes
110	19067606	169	0.000157	com.twimg
111	19066404	431	0.000060	com.googleblog
112	19056162	176	0.000148	com.blogger
113	19054310	232	0.000104	com.dribbble
114	19052824	207	0.000115	com.npmjs
115	19050524	564	0.000047	org.arxiv
116	19045194	666	0.000042	edu.upenn
117	19042770	171	0.000154	com.eventbrite
118	19036612	379	0.000068	com.springer
119	19032422	277	0.000090	org.ampproject
120	19031354	557	0.000047	com.gitlab
121	19025616	596	0.000045	com.vice
122	19025236	206	0.000116	com.disqus
123	19023978	1036	0.000031	com.hatenablog
124	19023406	835	0.000039	edu.columbia
125	19018670	818	0.000040	io.readthedocs
126	19008280	205	0.000116	me.t
127	19005842	390	0.000066	com.w3schools
128	19004030	941	0.000034	org.chromium
129	19003900	418	0.000062	com.nature
130	19001452	716	0.000041	com.slate
131	19000862	157	0.000171	jp.co.yahoo
132	18997566	325	0.000077	com.time
133	18997164	430	0.000060	com.statista
134	18990336	744	0.000040	com.ubuntu
135	18985466	158	0.000167	com.yelp
136	18982550	632	0.000043	org.worldbank
137	18982428	143	0.000191	com.spotify
138	18981078	347	0.000072	com.skype
139	18978660	935	0.000034	com.playstation
140	18976524	306	0.000082	com.fc2
141	18973312	1304	0.000024	org.coursera
142	18969438	121	0.000281	com.stripe
143	18968562	816	0.000040	com.qz
144	18968168	617	0.000044	com.git-scm
145	18966448	486	0.000053	uk.co.independent
146	18965810	199	0.000126	com.eepurl
147	18964444	961	0.000034	com.500px
148	18964342	405	0.000063	net.researchgate
149	18962654	241	0.000101	com.bandcamp
150	18959274	55	0.000669	net.facebook
151	18956478	389	0.000066	com.outlook
152	18955490	229	0.000105	com.unsplash
153	18950460	631	0.000043	com.mysql
154	18949330	419	0.000062	com.theatlantic
155	18948456	116	0.000286	com.soundcloud
156	18948364	180	0.000143	com.amazon-adsystem
157	18947670	123	0.000259	org.networkadvertising
158	18940768	571	0.000046	org.bitbucket
159	18940298	1163	0.000027	com.jetbrains
160	18936474	408	0.000063	com.mozilla
161	18936104	498	0.000052	com.nationalgeographic
162	18932906	316	0.000079	com.usatoday
163	18930890	439	0.000059	com.criteo
164	18927342	837	0.000039	uk.ac.ox
165	18925560	468	0.000054	com.fortune
166	18924956	466	0.000055	com.pixabay
167	18922224	1278	0.000024	uk.co.thesun
168	18921476	230	0.000104	net.behance
169	18916700	1547	0.000019	com.amd
170	18915574	822	0.000039	com.evernote
171	18909918	40	0.000932	com.vk
172	18909318	799	0.000040	com.about
173	18907910	505	0.000051	uk.co.blogspot
174	18904000	1193	0.000026	se.haxx
175	18903686	251	0.000097	gle.forms
176	18900132	719	0.000041	com.docker
177	18900020	967	0.000033	uk.co.guardian
178	18899576	303	0.000082	org.doi
179	18898330	497	0.000052	me.about
180	18896862	525	0.000050	gg.discord
181	18894432	1383	0.000022	com.instructables
182	18891582	147	0.000189	com.dropbox
183	18888432	1000	0.000032	com.scientificamerican
184	18885228	332	0.000076	jp.co.rakuten
185	18881548	903	0.000036	google.blog
186	18875704	194	0.000130	com.feedburner
187	18870758	1081	0.000030	org.altervista
188	18869068	634	0.000043	org.unesco
189	18868966	1082	0.000030	org.eclipse
190	18868488	275	0.000090	gov.ca
191	18866552	875	0.000037	jp.livedoor
192	18865164	1490	0.000020	org.phys
193	18864674	292	0.000085	com.sciencedirect
194	18860674	212	0.000113	jp.ameblo
195	18855700	572	0.000046	gov.loc
196	18854892	993	0.000033	org.cambridge
197	18845512	403	0.000064	ca.google
198	18842558	663	0.000042	edu.washington
199	18836168	71	0.000408	net.slideshare
200	18832808	543	0.000049	com.cisco
201	18829838	1364	0.000023	edu.rutgers
202	18828808	304	0.000082	com.nbcnews
203	18827294	243	0.000099	ru.rambler
204	18826594	823	0.000039	au.net.abc
205	18826034	998	0.000032	uk.co.thetimes
206	18825554	1472	0.000021	com.bankofamerica
207	18821912	62	0.000549	com.fb
208	18821670	863	0.000037	org.sciencemag
209	18813844	887	0.000036	com.speakerdeck
210	18810986	412	0.000063	jp.ne.sakura
211	18806748	220	0.000109	org.iana
212	18802112	1258	0.000025	com.wikidot
213	18799706	1650	0.000018	pt.sapo
214	18792880	905	0.000036	uk.co.mirror
215	18790620	218	0.000110	edu.stanford
216	18787734	1143	0.000028	org.kernel
217	18786526	450	0.000057	com.elsevier
218	18784420	1452	0.000021	edu.osu
219	18783446	1622	0.000019	com.googlesource
220	18782426	1395	0.000022	com.vogue
221	18781902	74	0.000401	net.akamaihd
222	18781258	842	0.000038	gov.fcc
223	18780926	1240	0.000025	ms.1drv
224	18779106	1377	0.000022	edu.asu
225	18778828	254	0.000095	com.businessinsider
226	18778534	844	0.000038	co.ibb
227	18778386	1843	0.000016	com.wolfram
228	18771090	721	0.000041	com.trello
229	18769148	106	0.000323	com.paypalobjects
230	18766298	301	0.000083	net.windows
231	18762238	1216	0.000026	jp.geocities
232	18762150	587	0.000045	com.box
233	18761840	876	0.000037	com.sciencedaily
234	18758802	227	0.000106	com.wpengine
235	18752986	493	0.000052	com.herokuapp
236	18752026	1030	0.000032	edu.princeton
237	18748814	915	0.000035	edu.academia
238	18748232	284	0.000087	com.googlecode
239	18746146	1177	0.000027	com.asahi
240	18740044	1375	0.000022	com.newscientist
241	18733718	1428	0.000021	blog.home
242	18732458	314	0.000080	com.tinyurl
243	18732010	516	0.000051	com.udacity
244	18731740	2103	0.000014	com.wizards
245	18729906	386	0.000067	com.cnet
246	18729644	1421	0.000022	com.ndtv
247	18729370	335	0.000075	com.getpocket
248	18729276	1349	0.000023	com.fandom
249	18726412	1241	0.000025	net.seesaa
250	18723376	167	0.000158	com.imdb
251	18718440	333	0.000076	org.debian
252	18715552	699	0.000041	site.business
253	18713638	261	0.000093	com.live
254	18711498	1059	0.000031	jp.ne.goo
255	18710778	1504	0.000020	io.itch
256	18704016	1155	0.000027	org.greenpeace
257	18702254	1047	0.000031	com.netlify
258	18701568	1808	0.000017	net.pixnet
259	18700552	402	0.000064	com.squareup
260	18699852	1145	0.000028	co.elastic
261	18699756	310	0.000081	com.ibm
262	18698784	203	0.000118	com.stackoverflow
263	18698146	494	0.000052	com.indiatimes
264	18696312	3420	0.000010	com.armorgames
265	18693148	265	0.000092	com.aliyuncs
266	18692378	226	0.000106	com.optimizely
267	18685944	1777	0.000017	uk.co.timesonline
268	18679570	975	0.000033	com.mixcloud
269	18677530	1618	0.000019	com.itv
270	18674460	198	0.000128	org.bbb
271	18674340	57	0.000648	net.fbcdn
272	18674148	2236	0.000014	com.opendns
273	18672682	3302	0.000010	tw.com.gamer
274	18672646	406	0.000063	com.go
275	18672486	437	0.000059	com.msn
276	18672432	37	0.000966	com.wixstatic
277	18667606	1485	0.000021	org.archlinux
278	18666546	448	0.000058	org.pewresearch
279	18664590	87	0.000345	com.shopify
280	18662914	838	0.000039	jp.shinobi
281	18662550	995	0.000032	com.bmj
282	18660516	1579	0.000019	com.diigo
283	18659688	160	0.000167	com.opera
284	18659332	1822	0.000017	com.youdao
285	18657686	1259	0.000025	com.angelfire
286	18657376	906	0.000036	jp.naver
287	18656934	1038	0.000031	com.thelancet
288	18653610	1585	0.000019	uk.bl
289	18652186	864	0.000037	br.com.google
290	18649914	490	0.000053	com.bigcartel
291	18648796	1007	0.000032	com.sky
292	18647596	1408	0.000022	net.daringfireball
293	18642592	1779	0.000017	uk.ac.kcl
294	18641168	1752	0.000017	org.maven
295	18639544	534	0.000049	me.m
296	18639298	1107	0.000029	com.reverbnation
297	18637290	1665	0.000018	net.cnki
298	18634102	1089	0.000030	com.theconversation
299	18630504	460	0.000056	it.placehold
300	18629064	1117	0.000029	com.podbean
301	18628660	881	0.000036	org.fao
302	18626956	1144	0.000028	co.g
303	18626264	1370	0.000023	com.dw
304	18623646	119	0.000283	com.mailchimp
305	18621260	1631	0.000019	jp.ne.so-net
306	18620264	1346	0.000023	com.livescience
307	18620110	2071	0.000015	edu.kit
308	18611900	1308	0.000024	ca.utoronto
309	18610618	1310	0.000024	com.webnode
310	18608388	957	0.000034	au.gov.nsw
311	18606964	1411	0.000022	com.citrix
312	18603704	1031	0.000032	jp.jugem
313	18602374	1115	0.000029	gov.wa
314	18600002	458	0.000056	com.quora
315	18598840	99	0.000326	com.godaddy
316	18597740	1334	0.000023	com.bloglovin
317	18596834	1194	0.000026	com.serving-sys
318	18595976	900	0.000036	gov.dhs
319	18595120	1630	0.000019	org.edx
320	18592696	244	0.000099	me.wa
321	18592582	1830	0.000017	com.pearltrees
322	18592022	1576	0.000019	com.twitpic
323	18591744	1776	0.000017	cn.people
324	18590350	1171	0.000027	com.britannica
325	18589380	1893	0.000016	sg.edu.nus
326	18588130	1791	0.000017	com.kinja
327	18587404	2296	0.000013	com.authorstream
328	18585922	1566	0.000019	ca.mcgill
329	18585132	380	0.000068	com.kickstarter
330	18581636	1237	0.000025	com.lulu
331	18576948	2591	0.000012	com.colourlovers
332	18575064	1919	0.000016	com.hm
333	18574242	337	0.000075	com.rackcdn
334	18567994	2261	0.000013	uk.ac.sussex
335	18563954	1945	0.000016	org.vim
336	18562946	1063	0.000031	com.healthline
337	18562860	1911	0.000016	org.wikibooks
338	18562650	1867	0.000016	io.soup
339	18560052	1704	0.000018	nl.blogspot
340	18555740	509	0.000051	com.mashable
341	18553730	264	0.000093	com.typepad
342	18553486	1071	0.000031	com.adjust
343	18551456	396	0.000065	com.photobucket
344	18544528	1741	0.000017	org.bitcoin
345	18542952	2227	0.000014	tw.edu.ntu
346	18541072	1100	0.000030	com.ecwid
347	18540798	1820	0.000017	com.indianexpress
348	18540288	1889	0.000016	co.ello
349	18538870	575	0.000046	edu.berkeley
350	18537372	2026	0.000015	com.upi
351	18537264	132	0.000232	com.squarespace
352	18535682	280	0.000089	uk.org.ico
353	18534924	1136	0.000028	com.ssrn
354	18534382	2152	0.000014	com.viki
355	18533818	1219	0.000025	it.scoop
356	18532606	270	0.000092	com.surveymonkey
357	18532016	1601	0.000019	com.fastcodesign
358	18530620	1782	0.000017	org.unep
359	18529588	1057	0.000031	uk.parliament
360	18527356	1966	0.000016	org.haskell
361	18527140	224	0.000107	com.etsy
362	18527064	1442	0.000021	com.shutterfly
363	18525388	1569	0.000019	uk.org.tate
364	18524530	2862	0.000011	co.electrek
365	18523134	2693	0.000011	jp.doorblog
366	18522838	156	0.000171	com.issuu
367	18519148	2018	0.000015	com.dezeen
368	18517910	2430	0.000013	sh.now
369	18517530	1157	0.000027	com.tradedoubler
370	18515028	1173	0.000027	gov.weather
371	18513616	1109	0.000029	com.imageshack
372	18512682	1693	0.000018	com.channel4
373	18512134	1116	0.000029	gov.dot
374	18511018	2703	0.000011	cn.edu.sdu
375	18510554	1164	0.000027	com.wikia
376	18509244	282	0.000088	com.huffingtonpost
377	18509124	953	0.000034	uk.co.pinterest
378	18508676	924	0.000035	com.arstechnica
379	18507156	271	0.000091	com.rawgit
380	18505812	484	0.000053	tv.twitch
381	18505722	1917	0.000016	th.co.google
382	18503134	2390	0.000013	uk.ac.nhm
383	18502360	1764	0.000017	com.netvibes
384	18501156	1871	0.000016	edu.emory
385	18500964	918	0.000035	in.amazon
386	18500252	963	0.000034	com.strikingly
387	18499224	1773	0.000017	net.bplaced
388	18497786	3356	0.000010	tw.edu.ntnu
389	18495692	1811	0.000017	edu.iu
390	18494542	833	0.000039	com.brightcove
391	18491624	225	0.000107	com.hubspot
392	18491366	1470	0.000021	com.wattpad
393	18490914	1476	0.000021	gov.michigan
394	18489222	1916	0.000016	nl.tudelft
395	18488064	1436	0.000021	org.c-span
396	18487708	394	0.000065	com.meetup
397	18483412	2110	0.000014	com.kaggle
398	18481662	1299	0.000024	edu.brookings
399	18478490	86	0.000345	net.jsfiddle
400	18478420	2621	0.000012	sh.surge
401	18475530	2248	0.000014	com.rsa
402	18475220	1781	0.000017	gov.ahrq
403	18474128	825	0.000039	org.mediawiki
404	18473922	346	0.000072	edu.yale
405	18472664	826	0.000039	com.intel
406	18472288	1512	0.000020	gov.faa
407	18471926	1975	0.000015	io.material
408	18471732	1073	0.000031	com.thenextweb
409	18471706	1847	0.000016	net.earthlink
410	18469610	1908	0.000016	jp.blog
411	18469092	831	0.000039	com.pexels
412	18464760	1029	0.000032	uk.gov.nationalarchives
413	18460916	1973	0.000016	com.smashwords
414	18459088	939	0.000034	org.ieee
415	18457420	2185	0.000014	com.smore
416	18456724	345	0.000072	com.livejournal
417	18456360	3366	0.000010	hk.edu.hkbu
418	18453542	414	0.000063	com.nypost
419	18453464	1785	0.000017	com.business-standard
420	18453440	2907	0.000011	com.yam
421	18451248	1233	0.000025	org.aarp
422	18450408	1999	0.000015	com.oprah
423	18449942	2053	0.000015	org.jpn
424	18449880	1503	0.000020	org.amnesty
425	18449790	1651	0.000018	com.avvo
426	18449648	2540	0.000012	com.cleantechnica
427	18449504	522	0.000050	edu.cornell
428	18448124	2586	0.000012	com.mysanantonio
429	18447594	473	0.000054	io.shields
430	18447544	1444	0.000021	org.hrw
431	18444240	2487	0.000012	org.neocities
432	18442810	1943	0.000016	com.care2
433	18440702	1714	0.000018	com.snopes
434	18440194	1148	0.000027	com.gizmodo
435	18440058	2474	0.000012	com.googledrive
436	18439272	2594	0.000012	com.iflscience
437	18437524	1594	0.000019	org.pypi
438	18437266	268	0.000092	net.php
439	18436902	1927	0.000016	org.rsc
440	18436288	1389	0.000022	com.pbworks
441	18435978	2735	0.000011	com.itsnicethat
442	18435362	2092	0.000015	ae.thenational
443	18435326	2461	0.000012	com.hsbc
444	18434642	338	0.000074	com.hp
445	18432636	1354	0.000023	uk.co.standard
446	18431764	1807	0.000017	com.instapaper
447	18431596	465	0.000055	io.codepen
448	18431390	553	0.000047	com.buzzfeed
449	18431020	1910	0.000016	com.secondlife
450	18430258	2425	0.000013	jp.go.ndl
451	18429756	2106	0.000014	io.gitlab
452	18428432	373	0.000069	int.who
453	18427128	2300	0.000013	org.lds
454	18426976	2471	0.000012	uk.mod
455	18426954	1711	0.000018	google.ai
456	18426290	96	0.000330	de.google
457	18423898	1427	0.000021	com.thehindu
458	18423724	1749	0.000017	com.curbed
459	18422902	1638	0.000019	no.google
460	18421738	340	0.000074	com.cnbc
461	18420686	1061	0.000031	com.thedrum
462	18419780	165	0.000160	com.ebay
463	18418790	627	0.000043	com.zdnet
464	18418454	2330	0.000013	pl.cba
465	18416392	2441	0.000013	com.minds
466	18413488	201	0.000125	com.salesforce
467	18413252	1574	0.000019	com.moonfruit
468	18412358	1156	0.000027	com.mixpanel
469	18411670	2818	0.000011	tl.page
470	18409196	1982	0.000015	com.name
471	18409080	2282	0.000013	jp.hateblo
472	18407830	2507	0.000012	org.tvtropes
473	18407318	2580	0.000012	jp.hatenadiary
474	18406348	2510	0.000012	de.dw
475	18405634	1854	0.000016	com.googlegroups
476	18405508	1876	0.000016	mx.com.google
477	18405134	2198	0.000014	org.aiga
478	18403404	2883	0.000011	uk.co.birminghammail
479	18403340	367	0.000069	com.booking
480	18401602	2314	0.000013	vn.com.google
481	18401550	1729	0.000018	gov.pa
482	18399972	1666	0.000018	org.hrc
483	18399618	882	0.000036	gov.nist
484	18398472	2742	0.000011	com.exxonmobil
485	18397376	1841	0.000016	ar.com.google
486	18396356	989	0.000033	net.clickbank
487	18395660	976	0.000033	com.matterport
488	18392402	2429	0.000013	ua.at
489	18390522	2011	0.000015	uk.ac.leeds
490	18387444	309	0.000081	gov.cdc
491	18386528	1558	0.000019	int.unfccc
492	18386408	2342	0.000013	com.eklablog
493	18385700	459	0.000056	com.gmail
494	18385598	401	0.000064	org.npr
495	18384832	1672	0.000018	gov.maryland
496	18384390	357	0.000070	com.office
497	18383950	2240	0.000014	se.liu
498	18383810	2067	0.000015	com.discovermagazine
499	18383400	2204	0.000014	com.ipage
500	18381626	1110	0.000029	com.stackexchange
501	18381594	2418	0.000013	it.justpaste
502	18380974	449	0.000058	fr.free
503	18380682	1718	0.000018	sg.com.google
504	18379672	1060	0.000031	com.engadget
505	18378238	2421	0.000013	my.com.thestar
506	18377282	1273	0.000024	dk.google
507	18377136	2210	0.000014	org.biorxiv
508	18377062	1861	0.000016	com.weheartit
509	18374194	1598	0.000019	uk.gov.tfl
510	18371274	508	0.000051	gov.whitehouse
511	18369330	1723	0.000018	ly.snip
512	18369006	1809	0.000017	com.yourstory
513	18366356	3154	0.000011	com.bonanza
514	18365650	2833	0.000011	com.scienceblogs
515	18365484	1431	0.000021	com.ebayimg
516	18365436	1774	0.000017	gov.ky
517	18363500	858	0.000038	com.venturebeat
518	18362924	1160	0.000027	se.google
519	18362454	1350	0.000023	com.firebaseapp
520	18362026	178	0.000147	com.zendesk
521	18360150	2004	0.000015	uk.gov.metoffice
522	18359990	928	0.000035	com.windowsphone
523	18359750	2336	0.000013	com.rediff
524	18358388	518	0.000051	com.alibaba
525	18355256	2225	0.000014	com.blogfa
526	18355232	415	0.000063	com.fastcompany
527	18353212	1426	0.000021	com.surveygizmo
528	18352352	2021	0.000015	au.com.telstra
529	18351454	1134	0.000028	org.sphinx-doc
530	18350502	2048	0.000015	ro.google
531	18350126	1904	0.000016	org.tigris
532	18349524	2835	0.000011	be.lesoir
533	18349430	2698	0.000011	cz.centrum
534	18349372	2047	0.000015	link.page
535	18349260	479	0.000054	org.nodejs
536	18349028	1960	0.000016	com.marketwire
537	18347672	2242	0.000014	com.mystrikingly
538	18347018	2260	0.000013	ch.unige
539	18346850	2753	0.000011	cat.uab
540	18346818	2889	0.000011	com.zynga
541	18345164	1510	0.000020	us.mn.state
542	18341622	2275	0.000013	com.articulate
543	18340012	991	0.000033	edu.psu
544	18339422	2141	0.000014	com.thecvf
545	18339020	2150	0.000014	es.csic
546	18338922	2880	0.000011	co.carrd
547	18337380	1611	0.000019	gov.mo
548	18337360	2297	0.000013	com.newatlas
549	18335690	3908	0.000009	jp.rdy
550	18334630	1990	0.000015	org.iea
551	18333598	2565	0.000012	com.db
552	18332716	2310	0.000013	com.webstarts
553	18332584	2488	0.000012	jp.hatenablog
554	18331976	2331	0.000013	ly.rebrand
555	18331344	370	0.000069	com.mapbox
556	18331228	485	0.000053	com.livechatinc
557	18325352	1998	0.000015	org.mozillazine
558	18324944	2271	0.000013	de.uni-freiburg
559	18324472	1372	0.000023	com.tinypic
560	18324252	883	0.000036	com.steampowered
561	18323842	2072	0.000015	uk.ac.york
562	18322186	1097	0.000030	com.thinkwithgoogle
563	18320582	2589	0.000012	ru.msu
564	18320156	2458	0.000012	org.kotlinlang
565	18319540	1629	0.000019	gov.oregon
566	18318914	3507	0.000010	com.ingress
567	18318120	1806	0.000017	gov.wi
568	18318056	541	0.000049	com.aol
569	18318040	1969	0.000016	gr.google
570	18317824	2741	0.000011	lv.draugiem
571	18316720	2305	0.000013	org.iucnredlist
572	18315822	2035	0.000015	com.broadwayworld
573	18314076	134	0.000221	com.youtube-nocookie
574	18313954	1511	0.000020	net.openid
575	18313704	168	0.000158	com.tripadvisor
576	18313516	435	0.000059	com.dailymotion
577	18313398	1548	0.000019	net.leadpages
578	18313360	3389	0.000010	com.brother
579	18313186	2755	0.000011	com.webcindario
580	18313116	3161	0.000011	es.usal
581	18312962	2338	0.000013	bg.google
582	18312728	907	0.000036	com.xiti
583	18312338	2273	0.000013	us.oh.state
584	18310760	720	0.000041	fm.last
585	18310700	1662	0.000018	net.ucoz
586	18306936	353	0.000071	org.acm
587	18304074	3931	0.000009	com.worldlingo
588	18303254	3379	0.000010	com.embarcadero
589	18303110	1789	0.000017	com.eiseverywhere
590	18302838	2230	0.000014	org.wri
591	18302746	395	0.000065	com.pubmatic
592	18302516	470	0.000054	com.goodreads
593	18300826	2212	0.000014	com.thehindubusinessline
594	18300196	2289	0.000013	com.mihanblog
595	18299988	2037	0.000015	com.intensedebate
596	18298184	3230	0.000011	com.hellomagazine
597	18297298	3474	0.000010	net.hypermart
598	18294990	235	0.000103	uk.co.amazon
599	18294428	2711	0.000011	nf.co
600	18294164	73	0.000401	me.fb
601	18294088	542	0.000049	com.entrepreneur
602	18293062	2543	0.000012	com.futurelearn
603	18292644	2389	0.000013	com.iconarchive
604	18292306	1275	0.000024	com.cognitoforms
605	18292138	1730	0.000018	org.khanacademy
606	18291072	2041	0.000015	com.financialpost
607	18291000	1744	0.000017	us.pa.state
608	18288698	2653	0.000012	com.fatcow
609	18288564	372	0.000069	com.staticflickr
610	18288272	1851	0.000016	io.bower
611	18287878	3657	0.000010	nz.govt.tepapa
612	18285000	2199	0.000014	org.prlog
613	18284980	2743	0.000011	ca.shaw
614	18284736	2104	0.000014	com.bravesites
615	18283030	2675	0.000012	de.uni-erlangen
616	18282568	2446	0.000012	org.lacity
617	18282024	1534	0.000020	fi.google
618	18282018	2267	0.000013	de.uni-koeln
619	18280510	2464	0.000012	uk.co.spectator
620	18279326	334	0.000076	com.typeform
621	18279324	2725	0.000011	is.good
622	18279290	3399	0.000010	com.114la
623	18278842	3277	0.000010	net.freeforums
624	18278384	920	0.000035	com.zoho
625	18273898	2465	0.000012	uk.ac.jisc
626	18273404	2489	0.000012	com.mnn
627	18273352	2393	0.000013	ca.dal
628	18272740	114	0.000290	com.statcounter
629	18272722	480	0.000054	com.netflix
630	18272318	1567	0.000019	com.flashtalking
631	18272212	1903	0.000016	com.prweek
632	18270806	3115	0.000011	site.negocio
633	18270794	2080	0.000015	org.lung
634	18270506	2727	0.000011	com.mouser
635	18270340	2569	0.000012	uk.co.profilebusiness
636	18269840	3616	0.000010	uk.gov.number10
637	18268146	3806	0.000009	net.dead
638	18267734	3375	0.000010	jp.ac.kobe-u
639	18267628	1925	0.000016	uk.org.nice
640	18267512	88	0.000343	com.oculus
641	18267198	3196	0.000011	build.bazel
642	18266546	1878	0.000016	org.gentoo
643	18266166	2181	0.000014	ie.thejournal
644	18266148	109	0.000310	com.sharethis
645	18265976	1906	0.000016	org.gnupg
646	18264728	148	0.000186	ru.mail
647	18263376	1860	0.000016	com.doodlekit
648	18262238	1948	0.000016	com.crashlytics
649	18262156	1831	0.000017	org.alz
650	18261954	2549	0.000012	us.ms.state
651	18261162	2459	0.000012	com.instructure
652	18260540	820	0.000040	com.cbsnews
653	18259844	2877	0.000011	ee.ut
654	18259826	1211	0.000026	com.msdn
655	18259610	777	0.000040	com.samsung
656	18257004	1338	0.000023	com.emailmeform
657	18254934	549	0.000048	edu.cmu
658	18254822	2496	0.000012	uk.co.osoo
659	18254762	83	0.000354	com.livestream
660	18254656	2226	0.000014	com.atavist
661	18252876	2208	0.000014	fr.archives-ouvertes
662	18252282	2792	0.000011	com.cnsnews
663	18252018	2348	0.000013	io.pantheon
664	18251148	898	0.000036	com.createjs
665	18251026	1755	0.000017	us.fl.state
666	18250730	2321	0.000013	com.rabbitmq
667	18250628	2712	0.000011	uk.co.newmedianow
668	18248576	1422	0.000022	com.123formbuilder
669	18247032	2086	0.000015	gov.nh
670	18243504	2233	0.000014	org.crossref
671	18242314	2229	0.000014	us.nm.state
672	18242254	296	0.000084	com.scribd
673	18241366	3254	0.000010	ca.qc.montreal
674	18240908	3285	0.000010	uk.co.lrb
675	18240828	135	0.000215	com.youku
676	18239750	517	0.000051	com.slack
677	18239658	2677	0.000012	com.hatenadiary
678	18239656	2292	0.000013	com.itsmyurls
679	18237636	2671	0.000012	uk.org.oxonaa
680	18236902	246	0.000099	com.constantcontact
681	18236862	3348	0.000010	com.outlookindia
682	18235854	3893	0.000009	in.ac.nptel
683	18235540	2681	0.000012	uk.org.oxfam
684	18235236	2344	0.000013	com.yext
685	18233812	256	0.000094	com.getbootstrap
686	18233324	2107	0.000014	org.jenkins-ci
687	18230584	2055	0.000015	com.broadcastingcable
688	18230478	1686	0.000018	uk.gov.direct
689	18230416	2663	0.000012	com.wmtransfer
690	18230374	1977	0.000015	gov.mt
691	18230164	2821	0.000011	uk.ac.stir
692	18228540	1052	0.000031	com.marketwatch
693	18227744	2266	0.000013	com.tmcnet
694	18227440	3136	0.000011	uk.co.hsbc
695	18227086	1798	0.000017	org.nfpa
696	18226792	2939	0.000011	com.batchgeo
697	18225844	3275	0.000010	com.weightwatchers
698	18225636	234	0.000103	to.amzn
699	18224632	3574	0.000010	com.orgfree
700	18223778	1355	0.000023	org.whatbrowser
701	18221814	2843	0.000011	com.adn
702	18221276	1190	0.000026	org.weforum
703	18220506	481	0.000054	org.hbr
704	18219880	2820	0.000011	au.edu.deakin
705	18219734	1455	0.000021	org.js
706	18219118	2445	0.000013	in.ernet
707	18217962	2854	0.000011	hu.elte
708	18217516	3025	0.000011	pl.edu.uw
709	18217274	2367	0.000013	uk.org.rspb
710	18216528	2220	0.000014	com.healthgrades
711	18216264	2779	0.000011	org.carbonbrief
712	18214214	366	0.000069	com.prnewswire
713	18213956	2088	0.000015	com.tapatalk
714	18213180	2431	0.000013	org.grist
715	18212750	3423	0.000010	id.co.kaskus
716	18210638	456	0.000057	com.oreilly
717	18210106	3587	0.000010	com.skepticalscience
718	18209950	539	0.000049	gov.sec
719	18209922	3081	0.000011	com.deccanherald
720	18209668	1905	0.000016	tl.we
721	18208770	2311	0.000013	us.ma.state
722	18206860	1101	0.000030	uk.ac.cam
723	18205994	3630	0.000010	ua.meta
724	18205738	3526	0.000010	app.web
725	18204462	2398	0.000013	uk.co.zoopla
726	18201966	3210	0.000011	org.oceanconservancy
727	18199630	3421	0.000010	org.atsjournals
728	18198962	3532	0.000010	ru.my1
729	18198444	3162	0.000011	com.mozello
730	18195600	1562	0.000019	com.pastebin
731	18194580	2867	0.000011	de.freenet
732	18193414	1137	0.000028	edu.ucla
733	18193100	3052	0.000011	com.telegraphindia
734	18193026	2857	0.000011	com.chagasi
735	18192758	937	0.000034	br.com.uol
736	18188994	2630	0.000012	com.atwebpages
737	18188626	3036	0.000011	com.remind
738	18187922	1132	0.000028	com.redhat
739	18187748	608	0.000044	com.wikihow
740	18187658	3377	0.000010	edu.utep
741	18187264	3455	0.000010	ru.nnov
742	18186834	1881	0.000016	uk.gov.defra
743	18186568	2359	0.000013	net.portfoliobox
744	18185624	2610	0.000012	com.blogsky
745	18185434	3856	0.000009	uk.co.mailonsunday
746	18185432	2723	0.000011	jp.xxxxxxxx
747	18184122	1425	0.000021	edu.ucsd
748	18183962	1449	0.000021	com.digitaltrends
749	18183738	196	0.000130	jp.ne.hatena
750	18182464	2563	0.000012	uk.co.inews
751	18181728	2313	0.000013	gov.la
752	18181656	1266	0.000024	ly.ow
753	18180360	3441	0.000010	gr.sch
754	18179802	3055	0.000011	com.sc
755	18178628	3373	0.000010	com.cummins
756	18177566	2363	0.000013	com.activerain
757	18176026	3801	0.000009	com.kazeo
758	18176002	2901	0.000011	net.onlinewebshop
759	18175422	3689	0.000010	com.galvanize
760	18174902	3473	0.000010	ru.pr-cy
761	18174826	503	0.000052	com.dmca
762	18173528	3328	0.000010	com.kaywa
763	18173348	821	0.000040	com.psychologytoday
764	18172118	2853	0.000011	uk.co.heatall
765	18171416	84	0.000350	me.ogp
766	18168128	2601	0.000012	gov.ks
767	18167782	1516	0.000020	ca.blogspot
768	18167558	2170	0.000014	com.cityam
769	18167284	3604	0.000010	gov.cabq
770	18166436	1813	0.000017	org.reactjs
771	18166052	3283	0.000010	org.escardio
772	18165734	1064	0.000031	com.foxnews
773	18165680	1897	0.000016	com.fifa
774	18164860	204	0.000117	com.naver
775	18164404	3761	0.000009	com.carscoops
776	18162680	2928	0.000011	com.ecowatch
777	18162390	1507	0.000020	com.literatumonline
778	18161998	535	0.000049	net.2mdn
779	18161800	476	0.000054	com.force
780	18160578	159	0.000167	gov.privacyshield
781	18160270	1896	0.000016	com.pcworld
782	18160192	2986	0.000011	com.theyworkforyou
783	18159730	81	0.000365	com.messenger
784	18159700	3939	0.000009	com.anghami
785	18159426	424	0.000061	edu.nyu
786	18157990	1294	0.000024	com.indiegogo
787	18157828	1869	0.000016	kr.or.kisa
788	18157816	364	0.000070	com.discordapp
789	18157014	3186	0.000011	uk.org.38degrees
790	18156850	3628	0.000010	com.insideevs
791	18155496	1488	0.000020	com.placeholder
792	18155072	3250	0.000010	google.design
793	18155044	3764	0.000009	gle.goo
794	18154462	454	0.000057	com.walmart
795	18153360	428	0.000060	com.flipboard
796	18152044	2902	0.000011	pl.lublin
797	18151952	422	0.000062	com.wufoo
798	18151198	1123	0.000029	com.shutterstock
799	18150684	2537	0.000012	org.iihs
800	18149446	2788	0.000011	in.businessworld
801	18148636	981	0.000033	com.pinimg
802	18147760	2407	0.000013	jp.e-shops
803	18147734	2250	0.000014	com.codecademy
804	18146340	2642	0.000012	com.zx2c4
805	18146328	129	0.000243	info.aboutads
806	18145944	2138	0.000014	ca.ubc
807	18145538	2874	0.000011	com.bnef
808	18144354	3240	0.000011	uk.ac.rcplondon
809	18144254	3718	0.000009	com.wsoctv
810	18143902	3950	0.000009	com.monbiot
811	18143342	3463	0.000010	com.droppages
812	18143148	2366	0.000013	gov.arts
813	18142454	2644	0.000012	us.wi.state
814	18142046	3477	0.000010	org.usatf
815	18140878	1624	0.000019	com.nvidia
816	18138866	3636	0.000010	com.elmercurio
817	18138838	1538	0.000020	com.businessweek
818	18138462	2176	0.000014	com.tutsplus
819	18138382	554	0.000047	com.atlassian
820	18137356	1184	0.000026	com.searchengineland
821	18137278	3594	0.000010	com.glu
822	18137124	3645	0.000010	es.consumer
823	18135974	240	0.000102	cn.com.sina
824	18135596	3948	0.000009	com.allmyfaves
825	18135342	3446	0.000010	com.businessgreen
826	18133642	350	0.000072	com.163
827	18133268	3292	0.000010	org.jython
828	18133230	471	0.000054	com.smugmug
829	18132816	3864	0.000009	org.thechicagocouncil
830	18132126	3576	0.000010	gov.azdot
831	18130470	1176	0.000027	com.ycombinator
832	18129838	3339	0.000010	org.transportenvironment
833	18128538	2993	0.000011	gov.ferc
834	18127910	936	0.000034	com.aliexpress
835	18126154	356	0.000070	com.wiley
836	18125790	696	0.000042	com.moz
837	18124996	2756	0.000011	uk.gov.environment-agency
838	18124886	3012	0.000011	org.zsl
839	18124136	3704	0.000009	org.ssireview
840	18123520	2378	0.000013	uk.gov.scotland
841	18122978	1595	0.000019	tv.ustream
842	18122522	3096	0.000011	org.dailystrength
843	18122038	598	0.000045	com.caniuse
844	18120996	2485	0.000012	net.privacypolicytemplate
845	18120866	768	0.000040	gov.noaa
846	18120818	1573	0.000019	jp.makeshop
847	18120518	3040	0.000011	org.rspo
848	18119946	2303	0.000013	com.seetickets
849	18119454	2183	0.000014	com.ign
850	18118896	404	0.000064	mp.mailchi
851	18118000	311	0.000081	com.digg
852	18118000	2855	0.000011	gov.txdot
853	18117366	3412	0.000010	uk.ac.ceh
854	18117164	1479	0.000021	com.crunchbase
855	18117074	1127	0.000029	com.highcharts
856	18115870	2645	0.000012	com.9to5mac
857	18114648	1090	0.000030	com.withgoogle
858	18114314	889	0.000036	com.webs
859	18114072	2481	0.000012	uk.co.streetmap
860	18112508	3865	0.000009	com.pushwoosh
861	18111708	3204	0.000011	ca.uwaterloo
862	18111130	817	0.000040	com.shinystat
863	18111078	305	0.000082	fr.google
864	18111050	3467	0.000010	com.baomoi
865	18110974	3916	0.000009	uk.ac.tyndall
866	18110396	1766	0.000017	com.webmasterplan
867	18110180	3686	0.000010	dk.bloggersdelight
868	18109908	3401	0.000010	uk.gov.hm-treasury
869	18109262	1793	0.000017	uk.org.cqc
870	18108948	1248	0.000025	com.smashingmagazine
871	18108138	331	0.000076	com.automattic
872	18107572	1530	0.000020	com.ning
873	18106984	2829	0.000011	com.linkwithin
874	18106522	3002	0.000011	uk.org.greenpeace
875	18103768	956	0.000034	com.libsyn
876	18103538	1239	0.000025	com.sap
877	18102956	2091	0.000015	edu.uci
878	18102564	628	0.000043	com.patreon
879	18102020	3503	0.000010	com.climatechangenews
880	18101758	409	0.000063	com.xinhuanet
881	18101336	3464	0.000010	com.kapook
882	18100618	885	0.000036	com.newyorker
883	18100474	3640	0.000010	com.spruz
884	18100196	478	0.000054	com.inc
885	18100062	2676	0.000012	jp.aikotoba
886	18099268	914	0.000035	org.eff
887	18098794	3662	0.000010	com.platts
888	18098556	3535	0.000010	org.c2es
889	18098550	2747	0.000011	com.mykaratestore
890	18096786	1770	0.000017	com.ikea
891	18096394	1423	0.000022	com.billboard
892	18095092	1070	0.000031	com.hootsuite
893	18094948	3525	0.000010	com.jkp
894	18093496	2824	0.000011	org.mcsuk
895	18092622	1254	0.000025	es.agpd
896	18092438	3349	0.000010	net.edie
897	18092358	533	0.000050	com.ea
898	18092112	376	0.000068	org.opensource
899	18091568	2903	0.000011	ru.drom
900	18090162	2639	0.000012	com.yelloyello
901	18089968	2544	0.000012	uk.co.intersol
902	18089740	139	0.000202	com.alicdn
903	18089422	4051	0.000009	com.mforos
904	18086990	1473	0.000021	com.fiverr
905	18086352	934	0.000035	com.foursquare
906	18085918	1737	0.000017	org.freecsstemplates
907	18084810	4142	0.000009	uk.org.indymedia
908	18084674	2049	0.000015	uk.gov.education
909	18083694	3843	0.000009	com.thinkbroadband
910	18082164	231	0.000104	jp.co.amazon
911	18080114	3868	0.000009	org.sciencenewsforstudents
912	18080034	221	0.000108	org.drupal
913	18079726	1096	0.000030	com.variety
914	18078666	290	0.000086	com.stumbleupon
915	18078038	3269	0.000010	net.scienceontheweb
916	18077582	1756	0.000017	com.nba
917	18077452	2561	0.000012	org.webring
918	18076502	1033	0.000031	com.visualstudio
919	18075958	4005	0.000009	io.raindrop
920	18074544	2744	0.000011	jp.zouri
921	18073766	3904	0.000009	org.corporateeurope
922	18072470	1402	0.000022	com.storify
923	18071436	375	0.000069	gov.ftc
924	18071372	1603	0.000019	net.with2
925	18070926	1448	0.000021	com.nike
926	18070222	4048	0.000009	io.dataquest
927	18070066	1255	0.000025	org.unicef
928	18069672	3567	0.000010	bnpparibas.group
929	18069172	3685	0.000010	com.thestatesman
930	18068866	3427	0.000010	uk.org.rya
931	18067508	383	0.000068	com.airbnb
932	18067204	1635	0.000019	de.zeit
933	18067190	2555	0.000012	com.hackernoon
934	18066274	3451	0.000010	ca.pe.gov
935	18065266	4031	0.000009	com.raamdev
936	18064388	2438	0.000013	io.postach
937	18064126	1487	0.000020	edu.purdue
938	18063508	407	0.000063	com.tripod
939	18063228	1228	0.000025	gov.fbi
940	18063154	1369	0.000023	com.lifehacker
941	18063130	1069	0.000031	com.uk
942	18061878	3432	0.000010	in.gov.mhrd
943	18061130	3527	0.000010	org.gmplib
944	18060100	3879	0.000009	com.gitimmersion
945	18059578	2807	0.000011	jp.at-ninja
946	18059004	3010	0.000011	com.shichihuku
947	18058826	3629	0.000010	com.h2database
948	18057736	3482	0.000010	uk.org.rcn
949	18057640	3737	0.000009	com.writetothem
950	18056592	1366	0.000023	com.parsiblog
951	18056586	984	0.000033	com.dropboxusercontent
952	18055950	1306	0.000024	com.prweb
953	18055628	3695	0.000009	com.websiteseguro
954	18055104	1118	0.000029	com.vox
955	18054272	1397	0.000022	us.imageshack
956	18053964	2032	0.000015	com.howstuffworks
957	18052920	1531	0.000020	com.yoast
958	18052280	1298	0.000024	com.pcmag
959	18051398	3008	0.000011	uk.org.woodlandtrust
960	18050936	3523	0.000010	gle.posts
961	18050800	3838	0.000009	org.priceofoil
962	18049580	1614	0.000019	com.ccbill
963	18049066	3750	0.000009	com.fourfour
964	18047214	945	0.000034	gov.census
965	18046486	1328	0.000023	edu.wisc
966	18045876	151	0.000179	jp.co.google
967	18045710	1220	0.000025	com.blackberry
968	18045414	1103	0.000030	edu.umich
969	18045390	1952	0.000016	com.w3layouts
970	18043894	146	0.000190	me.line
971	18043816	1593	0.000019	edu.usc
972	18042356	2842	0.000011	com.zatunen
973	18042240	500	0.000052	com.nasdaq
974	18042130	567	0.000046	net.daum
975	18041570	3118	0.000011	vn.tuoitre
976	18040556	2573	0.000012	com.hisupplier
977	18039444	2023	0.000015	com.nfl
978	18039370	927	0.000035	com.ggpht
979	18039324	1549	0.000019	com.vmware
980	18039020	3827	0.000009	com.realtytimes
981	18038362	3261	0.000010	net.batcave
982	18038116	3341	0.000010	org.mygamesonline
983	18037866	734	0.000040	com.mckinsey
984	18037674	3983	0.000009	org.eia-international
985	18037604	258	0.000094	com.sohu
986	18037594	3700	0.000009	io.dropwizard
987	18037394	1026	0.000032	gov.nps
988	18037244	2131	0.000014	au.com.news
989	18036608	3652	0.000010	de.epubli
990	18034198	1381	0.000022	com.unity3d
991	18034072	2992	0.000011	net.nend
992	18033048	4098	0.000009	com.easyhits4u
993	18031890	1162	0.000027	com.steamcommunity
994	18031622	1451	0.000021	edu.uchicago
995	18031570	1086	0.000030	com.uber
996	18031470	5306	0.000007	com.plurk
997	18030490	597	0.000045	com.adweek
998	18030182	3635	0.000010	com.jal
999	18029670	1786	0.000017	com.techradar
1000	18029244	1271	0.000024	com.ifttt

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

January 2020 crawl archive now available

February 3, 2020Sebastian Nagel

The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th. It includes page captures of 960 million URLs not contained in any crawl archive before.

Improvements and Fixes

date time values in the column "fetch_time" of the columnar index are now stored using the "int64" data type. For details and compatibility issues please see cc-index-table#7
WARC request records now show the HTTP protocol version sent with the HTTP request which can be different from the version received in the HTTP response message, cf. NUTCH-2760

Archive Location and Download

The January crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-05/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-05/segment.paths.gz	100
WARC files	CC-MAIN-2020-05/warc.paths.gz	56000	59.94
WAT files	CC-MAIN-2020-05/wat.paths.gz	56000	22.3
WET files	CC-MAIN-2020-05/wet.paths.gz	56000	10
Robots.txt files	CC-MAIN-2020-05/robotstxt.paths.gz	56000	0.25
Non-200 responses files	CC-MAIN-2020-05/non200responses.paths.gz	56000	2.28
URL index files	CC-MAIN-2020-05/cc-index.paths.gz	302	0.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-05/. Also the columnar index has been updated to contain this crawl.

December 2019 crawl archive now available

December 19, 2019Sebastian Nagel

The crawl archive for December 2019 is now available! It contains 2.45 billion web pages or 234 TiB of uncompressed content, crawled between December 5th and 16th. It includes page captures of 850 million URLs not contained in any crawl archive before.

Archive Location and Download

The December crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-51/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-51/segment.paths.gz	100
WARC files	CC-MAIN-2019-51/warc.paths.gz	56000	47.47
WAT files	CC-MAIN-2019-51/wat.paths.gz	56000	17.6
WET files	CC-MAIN-2019-51/wet.paths.gz	56000	8.06
Robots.txt files	CC-MAIN-2019-51/robotstxt.paths.gz	56000	0.26
Non-200 responses files	CC-MAIN-2019-51/non200responses.paths.gz	56000	3.5
URL index files	CC-MAIN-2019-51/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-51/. Also the columnar index has been updated to contain this crawl.

November 2019 crawl archive now available

November 27, 2019Sebastian Nagel

The crawl archive for November 2019 is now available! It contains 2.55 billion web pages or 250 TiB of uncompressed content, crawled between November 11th and 23rd with a short operational break on Nov 16th. It includes page captures of 1.1 billion URLs not contained in any crawl archive before.

What’s new?

We’ve added two new fields to the URL indexes (CDX and columnar):

the redirect target location is stored in the CDX JSON field "redirect" resp. the column "fetch_redirect". The value is extracted from HTTP header field "Location" if the HTTP status code indicates a HTTP redirect. A relative URL path is converted to an absolute URL using the page URL as base URL. The key is absent (resp. the field value is null) in case the "Location" value is missing, not a valid URL or not a valid relative URL path.
truncation of the WARC record payload is indicated by the key "truncated" resp. the column "content_truncated". The reason for the truncation is given only for truncated records following the WARC header field "WARC-Truncated".

Additional details and examples can be found in the corresponding PR #15.

We’ve fixed a bug affecting the capture time (WARC-Date) in the the robots.txt subset which has been extracted from the HTTP "Date" field of the HTTP header and appeared to be occasionally wrong. Please see issue #14 for further details.

Archive Location and Download

The November crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-47/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-47/segment.paths.gz	100
WARC files	CC-MAIN-2019-47/warc.paths.gz	56000	53.95
WAT files	CC-MAIN-2019-47/wat.paths.gz	56000	18.50
WET files	CC-MAIN-2019-47/wet.paths.gz	56000	8.34
Robots.txt files	CC-MAIN-2019-47/robotstxt.paths.gz	56000	0.24
Non-200 responses files	CC-MAIN-2019-47/non200responses.paths.gz	56000	3.05
URL index files	CC-MAIN-2019-47/cc-index.paths.gz	302	0.20

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-47/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs Aug/Sep/Oct 2019

November 12, 2019Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of August, September and October 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

What’s new?

The following improvements have been made for this webgraph release:

the graphs now also included edges stemming from HTTP 303 “See Other” redirects (in addition to other HTTP redirect status codes)
the Common Crawl robots.txt WARC files are used to get additional host-level redirects including hosts which exclude the entire content in their robots.txt
links from robots.txt files to sitemaps are now extracted directly from the robots.txt WARC files, see the Feb/Mar/Apr 2018 web graph announcement for more details about this type of host-level links

Host-level graph

The graph consists of 820 million nodes and 4.55 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 752 million dangling nodes (92%) and the largest strongly connected component contains 50 million (6%) nodes.

You can download the graph and the ranks of all 820 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-aug-sep-oct/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-aug-sep-oct/host/ as prefix to access the files from everywhere.

Size	File	Description
5.29 GB	cc-main-2019-aug-sep-oct-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 28 vertices files
20.73 GB	cc-main-2019-aug-sep-oct-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 56 edges files

8.15 GB	cc-main-2019-aug-sep-oct-host.graph	graph in BVGraph format
2 kB	cc-main-2019-aug-sep-oct-host.properties

10.00 GB	cc-main-2019-aug-sep-oct-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2019-aug-sep-oct-host-t.properties

1 kB	cc-main-2019-aug-sep-oct-host.stats	WebGraph statistics

11.59 GB	cc-main-2019-aug-sep-oct-host-ranks.txt.gz	harmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 92.7 million nodes and 2.4 billion edges. 52% or 48 million nodes are dangling nodes, the largest strongly connected component covers 36 million or 40% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-aug-sep-oct/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-aug-sep-oct/domain/.

Download files of the Common Crawl Aug/Sep/Oct 2019 domain-level webgraph

Size	File	Description
0.64 GB	cc-main-2019-aug-sep-oct-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
9.06 GB	cc-main-2019-aug-sep-oct-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

4.64 GB	cc-main-2019-aug-sep-oct-domain.graph	graph in BVGraph format
2 kB	cc-main-2019-aug-sep-oct-domain.properties

4.82 GB	cc-main-2019-aug-sep-oct-domain-t.graph	transpose of the graph
2 kB	cc-main-2019-aug-sep-oct-domain-t.properties

1 kB	cc-main-2019-aug-sep-oct-domain.stats	WebGraph statistics

1.97 GB	cc-main-2019-aug-sep-oct-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 92 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Aug/Sept/Oct 2019)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	32193222	1	0.020989	com.googleapis
2	29939490	3	0.012691	com.facebook
3	29202762	2	0.012925	com.google
4	26823584	4	0.007369	com.twitter
5	26304804	5	0.006660	org.w
6	26023106	6	0.006435	com.youtube
7	24330930	9	0.003914	com.instagram
8	23954698	7	0.004993	org.gmpg
9	23537216	8	0.004863	com.googletagmanager
10	23456362	13	0.002913	com.linkedin
11	22601398	12	0.003086	org.wordpress
12	22511672	10	0.003602	com.cloudflare
13	22484028	22	0.001698	com.gravatar
14	22366260	23	0.001509	com.pinterest
15	22337722	19	0.002143	com.wordpress
16	22168804	14	0.002422	com.bootstrapcdn
17	22152656	32	0.001134	org.wikipedia
18	21939876	20	0.001777	com.apple
19	21666694	42	0.000842	com.blogspot
20	21595338	21	0.001736	com.jquery
21	21576018	49	0.000713	be.youtu
22	21574068	35	0.001064	com.vimeo
23	21525328	30	0.001154	com.microsoft
24	21517514	18	0.002164	com.gstatic
25	21444512	17	0.002176	com.adobe
26	21427572	39	0.000964	com.amazonaws
27	21426674	50	0.000695	com.wp
28	21393434	51	0.000681	com.amazon
29	21314258	65	0.000516	com.tumblr
30	21291646	46	0.000767	gl.goo
31	21256658	25	0.001309	com.macromedia
32	21253410	29	0.001173	com.baidu
33	21136800	67	0.000501	ly.bit
34	21087412	27	0.001200	com.flickr
35	21083642	24	0.001391	com.github
36	21068510	89	0.000381	com.yahoo
37	21063868	41	0.000928	com.google-analytics
38	21063204	31	0.001139	com.googlesyndication
39	21053852	57	0.000608	eu.europa
40	21051522	61	0.000541	org.mozilla
41	20997812	106	0.000300	com.reddit
42	20934970	37	0.001002	net.cloudfront
43	20930738	28	0.001184	ru.yandex
44	20907836	38	0.000964	com.addthis
45	20875048	48	0.000734	co.t
46	20863874	47	0.000744	net.doubleclick
47	20860222	70	0.000482	org.w3
48	20822794	98	0.000329	com.googleusercontent
49	20819594	43	0.000814	com.squarespace
50	20815248	75	0.000462	com.medium
51	20812946	91	0.000376	org.creativecommons
52	20811076	175	0.000140	org.wikimedia
53	20788398	86	0.000417	com.weebly
54	20786242	63	0.000534	me.wp
55	20764144	129	0.000221	com.nytimes
56	20754576	88	0.000400	io.github
57	20744406	55	0.000625	com.paypal
58	20731050	165	0.000148	uk.co.bbc
59	20729776	58	0.000557	net.jsdelivr
60	20723608	108	0.000297	com.soundcloud
61	20720614	172	0.000141	com.imgur
62	20696474	130	0.000210	com.dropbox
63	20676226	137	0.000181	com.forbes
64	20641078	173	0.000141	net.slideshare
65	20640886	54	0.000634	org.schema
66	20637016	153	0.000163	com.theguardian
67	20619410	187	0.000136	com.cnn
68	20614482	204	0.000118	com.businessinsider
69	20589376	217	0.000109	com.wsj
70	20575384	281	0.000086	edu.harvard
71	20572806	167	0.000147	com.bing
72	20571044	241	0.000098	com.techcrunch
73	20567292	290	0.000084	edu.mit
74	20557144	285	0.000084	com.reuters
75	20554074	375	0.000067	com.msn
76	20549852	329	0.000075	com.cnet
77	20542104	140	0.000178	org.archive
78	20538104	250	0.000094	com.bloomberg
79	20522694	33	0.001120	com.fontawesome
80	20521374	141	0.000175	gov.nih
81	20513408	93	0.000355	com.shopify
82	20513140	271	0.000089	com.myspace
83	20507048	207	0.000116	edu.stanford
84	20496152	53	0.000647	com.wix
85	20489102	200	0.000120	com.stackoverflow
86	20487938	434	0.000057	com.googleblog
87	20484632	154	0.000163	org.apache
88	20478194	229	0.000102	com.oracle
89	20475392	214	0.000110	com.washingtonpost
90	20472386	260	0.000091	com.android
91	20469298	267	0.000090	com.bbc
92	20466440	194	0.000123	org.ietf
93	20438542	310	0.000079	com.time
94	20436352	298	0.000081	uk.co.telegraph
95	20417976	369	0.000067	com.ted
96	20415662	372	0.000067	gov.nasa
97	20412316	368	0.000067	com.githubusercontent
98	20409888	185	0.000136	com.npmjs
99	20401604	394	0.000063	com.quora
100	20396146	601	0.000042	com.thenextweb
101	20395674	161	0.000156	com.giphy
102	20388778	726	0.000037	com.wikia
103	20380728	343	0.000072	uk.co.dailymail
104	20379652	294	0.000082	com.usatoday
105	20378396	371	0.000067	com.latimes
106	20370212	713	0.000037	org.chromium
107	20369700	306	0.000079	org.un
108	20368148	144	0.000174	com.wixsite
109	20366610	493	0.000050	com.economist
110	20361312	26	0.001226	com.qq
111	20343426	268	0.000090	com.appspot
112	20339266	480	0.000052	com.pixabay
113	20337398	491	0.000050	com.zdnet
114	20328308	315	0.000079	com.example
115	20325422	358	0.000070	com.livejournal
116	20322334	380	0.000066	com.mashable
117	20308200	302	0.000080	com.cnbc
118	20308066	253	0.000093	org.ampproject
119	20306984	442	0.000056	com.nationalgeographic
120	20293426	505	0.000049	com.venturebeat
121	20292380	404	0.000062	com.dailymotion
122	20285502	139	0.000178	com.twimg
123	20284164	476	0.000052	org.bitbucket
124	20282368	547	0.000046	com.pexels
125	20280714	327	0.000075	com.springer
126	20279992	218	0.000108	com.huffingtonpost
127	20279190	94	0.000355	com.whatsapp
128	20277928	459	0.000054	com.cisco
129	20268416	146	0.000170	com.blogger
130	20267684	123	0.000234	com.ytimg
131	20264730	413	0.000061	com.fortune
132	20263014	641	0.000040	uk.ac.ox
133	20262258	231	0.000100	com.getbootstrap
134	20261648	847	0.000035	org.cambridge
135	20261268	629	0.000040	org.weforum
136	20250854	197	0.000123	com.typepad
137	20250698	279	0.000086	com.sciencedirect
138	20250162	512	0.000048	com.about
139	20247192	286	0.000084	com.wired
140	20240130	317	0.000078	com.skype
141	20235202	558	0.000045	org.worldbank
142	20230192	134	0.000186	com.issuu
143	20225004	504	0.000049	com.mysql
144	20220996	650	0.000039	org.sciencemag
145	20220972	531	0.000047	org.arxiv
146	20218296	624	0.000041	uk.co.guardian
147	20216194	407	0.000062	com.nature
148	20214012	127	0.000226	com.unpkg
149	20213638	143	0.000175	com.spotify
150	20195500	824	0.000036	com.playstation
151	20195352	177	0.000139	uk.co.google
152	20195260	440	0.000057	gov.noaa
153	20193574	323	0.000077	com.staticflickr
154	20193512	366	0.000068	com.gmail
155	20191934	1037	0.000028	org.eclipse
156	20191832	395	0.000063	net.researchgate
157	20185934	342	0.000072	com.fc2
158	20179194	603	0.000042	org.ieee
159	20177140	132	0.000201	com.zendesk
160	20177108	383	0.000065	com.theatlantic
161	20173850	590	0.000043	com.git-scm
162	20173722	182	0.000136	me.t
163	20169446	282	0.000085	com.googlecode
164	20167964	212	0.000113	net.behance
165	20166960	364	0.000068	com.w3schools
166	20165408	657	0.000039	com.stackexchange
167	20147566	128	0.000222	com.youtube-nocookie
168	20144266	430	0.000058	com.buzzfeed
169	20143168	573	0.000043	br.com.uol
170	20141222	828	0.000036	ca.blogspot
171	20138528	592	0.000042	com.evernote
172	20137536	854	0.000034	com.scientificamerican
173	20123000	227	0.000102	com.dribbble
174	20122966	495	0.000049	com.vice
175	20119812	180	0.000137	com.feedburner
176	20118786	574	0.000043	net.azurewebsites
177	20113370	536	0.000046	com.alexa
178	20110780	418	0.000059	com.outlook
179	20103382	424	0.000059	com.gitlab
180	20092588	422	0.000059	me.about
181	20092232	409	0.000061	com.goodreads
182	20091842	1102	0.000026	com.nvidia
183	20082450	419	0.000059	com.mozilla
184	20078524	447	0.000056	com.entrepreneur
185	20073740	236	0.000099	com.ft
186	20071534	452	0.000055	com.wikihow
187	20066124	245	0.000096	com.disqus
188	20064942	1092	0.000026	com.jetbrains
189	20063756	1327	0.000023	org.phys
190	20062066	602	0.000042	org.greenpeace
191	20061474	386	0.000065	org.hbr
192	20059468	178	0.000139	com.salesforce
193	20058532	537	0.000046	com.adage
194	20056012	300	0.000080	org.doi
195	20055914	1106	0.000026	org.ap
196	20054068	860	0.000034	com.500px
197	20051824	488	0.000051	gov.loc
198	20051342	957	0.000030	com.sap
199	20050500	626	0.000041	com.marketwatch
200	20049824	1265	0.000024	com.siemens
201	20049584	1173	0.000025	ca.utoronto
202	20049300	428	0.000058	uk.co.independent
203	20048034	222	0.000104	com.hubspot
204	20045788	593	0.000042	com.slate
205	20042018	349	0.000071	gg.discord
206	20024956	1435	0.000021	com.hackernoon
207	20022096	487	0.000051	uk.co.blogspot
208	20012130	1451	0.000021	org.tensorflow
209	20007682	401	0.000062	com.indiatimes
210	20007486	1035	0.000028	org.kernel
211	20001698	530	0.000047	com.trello
212	19999034	666	0.000038	com.searchengineland
213	19997084	1009	0.000029	com.unity3d
214	19996940	473	0.000052	com.computerworld
215	19996232	549	0.000045	com.withgoogle
216	19993078	1369	0.000022	edu.osu
217	19991880	949	0.000030	edu.si
218	19990236	612	0.000041	au.net.abc
219	19988088	1428	0.000021	com.lego
220	19987532	287	0.000084	com.nbcnews
221	19977482	1356	0.000022	com.angelfire
222	19976080	499	0.000049	com.moz
223	19975358	199	0.000122	net.sourceforge
224	19969236	667	0.000038	co.ibb
225	19968114	1618	0.000019	org.edx
226	19967072	515	0.000048	com.box
227	19961458	986	0.000029	com.huffpost
228	19961370	598	0.000042	gov.state
229	19956418	1563	0.000019	blog.home
230	19955608	1678	0.000018	com.oregonlive
231	19954284	631	0.000040	com.pinimg
232	19953180	863	0.000034	gov.usgs
233	19949892	2048	0.000016	com.sputniknews
234	19948950	1047	0.000027	co.elastic
235	19947460	1196	0.000025	edu.rutgers
236	19946614	211	0.000115	com.optimizely
237	19945418	1409	0.000021	org.maven
238	19942668	1373	0.000022	net.seesaa
239	19939512	237	0.000099	com.aliyuncs
240	19939300	291	0.000083	com.tinyurl
241	19939182	188	0.000134	com.eepurl
242	19938152	224	0.000103	com.wpengine
243	19936538	2235	0.000014	com.slides
244	19935586	659	0.000039	com.sciencedaily
245	19933262	136	0.000183	com.addtoany
246	19933088	946	0.000031	com.storify
247	19932194	142	0.000175	com.yimg
248	19927032	354	0.000070	com.getpocket
249	19925642	715	0.000037	com.vox
250	19922530	60	0.000546	com.vk
251	19920994	171	0.000142	org.allaboutcookies
252	19919990	1165	0.000025	com.vogue
253	19918364	335	0.000074	com.wufoo
254	19914676	1282	0.000023	ms.1drv
255	19906484	1481	0.000020	io.itch
256	19906312	834	0.000035	com.techtarget
257	19905162	600	0.000042	org.change
258	19901530	597	0.000042	com.uk
259	19901258	421	0.000059	com.squareup
260	19897576	1408	0.000021	com.itv
261	19896802	954	0.000030	com.thehill
262	19896772	1291	0.000023	com.scmp
263	19894514	1777	0.000017	com.diigo
264	19893192	316	0.000079	es.google
265	19890244	651	0.000039	com.lifehacker
266	19888786	671	0.000038	gov.fcc
267	19886980	739	0.000037	com.chicagotribune
268	19886180	2309	0.000014	com.pearltrees
269	19885516	1554	0.000019	org.unep
270	19881960	313	0.000079	net.windows
271	19881842	248	0.000094	ru.rambler
272	19880642	506	0.000049	us.icio
273	19877580	92	0.000358	com.weibo
274	19876556	109	0.000290	com.paypalobjects
275	19874826	891	0.000033	com.strikingly
276	19873598	1178	0.000025	com.netlify
277	19867654	456	0.000055	gov.epa
278	19866350	292	0.000083	com.criteo
279	19864080	714	0.000037	org.pewresearch
280	19861136	533	0.000047	org.plos
281	19860954	1225	0.000024	com.newscientist
282	19860836	849	0.000035	uk.co.mirror
283	19860700	1010	0.000029	com.mediafire
284	19860298	1072	0.000027	com.sky
285	19859946	928	0.000031	com.buffer
286	19858910	1228	0.000024	com.aljazeera
287	19858168	1339	0.000022	it.scoop
288	19858040	209	0.000116	org.iana
289	19857260	2070	0.000016	com.coca-colacompany
290	19856912	683	0.000038	com.flipboard
291	19853900	1801	0.000017	jp.ac.u-tokyo
292	19853116	1018	0.000028	uk.co.metro
293	19851054	309	0.000079	com.ibm
294	19846968	322	0.000077	com.go
295	19846838	1552	0.000019	uk.bl
296	19841556	1264	0.000024	com.nikkei
297	19840090	52	0.000667	com.fb
298	19839844	2506	0.000013	it.unimi
299	19836858	1595	0.000019	com.googlesource
300	19834504	474	0.000052	com.udacity
301	19834024	835	0.000035	uk.co.thetimes
302	19832262	168	0.000144	com.imdb
303	19831660	843	0.000035	gov.congress
304	19828142	668	0.000038	org.fao
305	19826656	1191	0.000025	org.acs
306	19825238	1728	0.000018	com.toptal
307	19824736	1065	0.000027	edu.duke
308	19823982	621	0.000041	site.business
309	19820920	1133	0.000026	com.trendmicro
310	19817822	955	0.000030	com.theconversation
311	19814258	983	0.000029	co.g
312	19813034	851	0.000034	com.bmj
313	19812202	170	0.000143	com.amazon-adsystem
314	19808398	1045	0.000027	com.searchenginewatch
315	19806128	1376	0.000022	edu.gatech
316	19803474	2207	0.000015	com.viki
317	19803388	1135	0.000026	edu.brookings
318	19803178	971	0.000030	com.reverbnation
319	19798960	1069	0.000027	au.com.smh
320	19797938	44	0.000797	com.googleadservices
321	19796164	475	0.000052	org.freecodecamp
322	19792806	658	0.000039	br.com.google
323	19791896	1766	0.000017	jp.co.japantimes
324	19791234	400	0.000063	me.telegram
325	19790208	1332	0.000022	com.msnbc
326	19789672	1915	0.000016	org.wikibooks
327	19789356	1296	0.000023	com.dw
328	19787622	1366	0.000022	com.hostgator
329	19784154	477	0.000052	com.theverge
330	19780916	1574	0.000019	com.bankofamerica
331	19776986	994	0.000029	com.yoast
332	19775742	997	0.000029	com.socialmediaexaminer
333	19774146	841	0.000035	org.apa
334	19772748	426	0.000058	com.elsevier
335	19771404	458	0.000055	com.bigcartel
336	19770190	2240	0.000014	com.kinja
337	19770024	1701	0.000018	com.mediaplex
338	19769080	1058	0.000027	uk.co.huffingtonpost
339	19766820	1682	0.000018	org.bitcoin
340	19765668	1430	0.000021	com.grammarly
341	19765220	2071	0.000016	com.mathworks
342	19764662	1253	0.000024	com.livescience
343	19764202	249	0.000094	com.live
344	19763516	2265	0.000014	org.biorxiv
345	19762024	1794	0.000017	com.makeuseof
346	19760700	942	0.000031	com.econsultancy
347	19759296	518	0.000047	com.bigcommerce
348	19759088	953	0.000030	com.searchenginejournal
349	19757028	62	0.000537	net.akamaihd
350	19755866	1764	0.000017	com.colourlovers
351	19751232	314	0.000079	com.rackcdn
352	19749162	1834	0.000017	com.sas
353	19746962	223	0.000104	org.gnu
354	19742390	2490	0.000013	com.itsnicethat
355	19741694	2296	0.000014	uk.ac.sussex
356	19739212	820	0.000036	com.neilpatel
357	19738554	162	0.000156	com.opera
358	19738540	951	0.000030	com.gumroad
359	19733434	868	0.000034	com.business2community
360	19731138	909	0.000032	uk.co.pinterest
361	19730570	617	0.000041	uk.parliament
362	19729560	898	0.000032	com.ecwid
363	19729024	526	0.000047	me.m
364	19728302	1186	0.000025	com.thelancet
365	19727476	1677	0.000018	uk.co.timesonline
366	19725568	1662	0.000018	edu.iastate
367	19720948	890	0.000033	com.thedrum
368	19718200	1234	0.000024	com.seattletimes
369	19716772	116	0.000258	com.jimdo
370	19715158	1748	0.000018	org.rsc
371	19713318	318	0.000078	me.wa
372	19713052	2312	0.000014	io.soup
373	19712174	240	0.000098	net.php
374	19710524	996	0.000029	com.healthline
375	19706662	103	0.000317	net.facebook
376	19700662	389	0.000064	com.meetup
377	19698168	1397	0.000021	int.unfccc
378	19697842	2364	0.000014	com.autoblog
379	19697184	1147	0.000026	uk.co.ebay
380	19696284	1512	0.000020	com.channel4
381	19696102	345	0.000072	int.who
382	19695842	856	0.000034	com.photoshelter
383	19693426	297	0.000081	org.python
384	19693168	2103	0.000016	edu.miami
385	19693108	2445	0.000013	com.mysanantonio
386	19693052	1314	0.000023	com.bustle
387	19693004	2416	0.000013	com.smore
388	19690872	1244	0.000024	uk.co.express
389	19689496	1882	0.000016	com.smashwords
390	19689346	1454	0.000021	com.gawker
391	19689266	1492	0.000020	org.hrc
392	19688570	1378	0.000022	uk.gov.blog
393	19688210	266	0.000090	com.rawgit
394	19685386	251	0.000094	uk.org.ico
395	19684372	2229	0.000015	org.vim
396	19683694	2148	0.000015	uk.ac.york
397	19683048	1902	0.000016	com.discovermagazine
398	19682466	2017	0.000016	com.dummies
399	19682270	2811	0.000012	com.iht
400	19678702	1498	0.000020	fr.lesechos
401	19677190	1643	0.000019	org.amnesty
402	19677184	1087	0.000026	org.aarp
403	19675912	836	0.000035	uk.gov.legislation
404	19675666	1582	0.000019	com.pbworks
405	19675228	1197	0.000025	com.cio
406	19675036	1541	0.000020	com.googlegroups
407	19673696	888	0.000033	uk.gov.nationalarchives
408	19671786	489	0.000051	com.nwsource
409	19669190	1344	0.000022	com.thestar
410	19668328	1993	0.000016	com.treehugger
411	19668276	1602	0.000019	com.brainyquote
412	19667868	513	0.000048	com.livechatinc
413	19667262	1195	0.000025	org.heart
414	19666046	259	0.000091	com.unsplash
415	19665938	1475	0.000020	ie.independent
416	19665662	2444	0.000013	org.sciencenews
417	19664008	1478	0.000020	fi.google
418	19662896	1201	0.000025	uk.co.standard
419	19662404	163	0.000156	com.eventbrite
420	19661850	1997	0.000016	com.timesofisrael
421	19661340	1304	0.000023	com.surveygizmo
422	19659778	1245	0.000024	org.ohchr
423	19656716	1989	0.000016	com.nationalreview
424	19654260	2022	0.000016	com.gucci
425	19653254	605	0.000041	org.mediawiki
426	19651234	972	0.000029	com.wordstream
427	19651102	1584	0.000019	com.netvibes
428	19649566	1976	0.000016	org.bitcointalk
429	19648228	2372	0.000014	com.deepmind
430	19648124	1773	0.000017	org.iucn
431	19647904	1496	0.000020	com.startribune
432	19646024	293	0.000082	com.ebay
433	19639388	1355	0.000022	com.convinceandconvert
434	19637100	522	0.000047	edu.yale
435	19636614	384	0.000065	com.kickstarter
436	19635776	100	0.000321	com.godaddy
437	19634912	2157	0.000015	com.instapaper
438	19633830	1767	0.000017	uk.co.ibtimes
439	19631378	1261	0.000024	com.imageshack
440	19630146	110	0.000284	com.mailchimp
441	19627008	2887	0.000011	net.openreview
442	19626924	481	0.000052	gov.whitehouse
443	19626884	1301	0.000023	ch.ipcc
444	19625858	959	0.000030	com.bandsintown
445	19625598	388	0.000064	com.office
446	19624032	2039	0.000016	edu.udel
447	19623636	1818	0.000017	uk.ac.kcl
448	19619866	988	0.000029	org.ilo
449	19618636	1880	0.000016	tl.we
450	19618128	2092	0.000016	io.gitlab
451	19616698	1975	0.000016	com.digitaljournal
452	19615278	84	0.000440	com.list-manage
453	19614194	15	0.002224	com.wixstatic
454	19611928	1791	0.000017	com.secondlife
455	19604998	1171	0.000025	uk.gov.tfl
456	19603646	1994	0.000016	org.peta
457	19602880	1252	0.000024	com.medicalnewstoday
458	19601844	1744	0.000018	com.teenvogue
459	19601126	45	0.000773	net.fbcdn
460	19600768	1813	0.000017	com.upi
461	19600410	205	0.000117	com.etsy
462	19598800	1577	0.000019	no.google
463	19597706	2097	0.000016	com.shell
464	19596732	1535	0.000020	com.quicksprout
465	19596622	406	0.000062	com.fastcompany
466	19596226	1324	0.000023	org.hrw
467	19596164	559	0.000045	edu.berkeley
468	19595736	826	0.000036	com.intel
469	19593408	1911	0.000016	com.tomsguide
470	19592762	1655	0.000018	ca.pinterest
471	19591462	365	0.000068	com.hp
472	19590312	649	0.000039	org.nodejs
473	19589296	2135	0.000015	com.politifact
474	19588516	2400	0.000013	com.towardsdatascience
475	19588356	2292	0.000014	com.dailykos
476	19588058	1749	0.000018	com.oprah
477	19585238	3039	0.000011	org.arkive
478	19584732	859	0.000034	com.engadget
479	19584238	1740	0.000018	com.shareholder
480	19584228	967	0.000030	ly.snip
481	19577646	1359	0.000022	com.smallbiztrends
482	19577604	2384	0.000014	com.hsbc
483	19577414	104	0.000312	com.statcounter
484	19577334	566	0.000044	com.photobucket
485	19576468	2161	0.000015	org.jenkins-ci
486	19574024	1017	0.000028	com.contentmarketinginstitute
487	19569238	2447	0.000013	uk.co.spectator
488	19567958	1966	0.000016	com.thecut
489	19567398	2655	0.000012	uk.ac.mmu
490	19563030	1458	0.000021	net.convio
491	19562626	1897	0.000016	org.project-syndicate
492	19562602	857	0.000034	com.deviantart
493	19562312	1658	0.000018	google.ai
494	19560912	1921	0.000016	com.ogilvy
495	19560528	1775	0.000017	com.csoonline
496	19559434	990	0.000029	com.cognitoforms
497	19558398	2029	0.000016	link.page
498	19557452	2224	0.000015	com.upworthy
499	19555356	1670	0.000018	com.kinsta
500	19551574	393	0.000063	com.getclicky
501	19548794	1907	0.000016	ms.nyti
502	19548294	1951	0.000016	uk.ac.leeds
503	19546822	1247	0.000024	st.po
504	19546690	359	0.000069	com.mapbox
505	19545958	2341	0.000014	com.sciencealert
506	19545120	2387	0.000013	com.instructure
507	19543894	2343	0.000014	org.theiet
508	19543292	2620	0.000012	com.ksl
509	19540054	2168	0.000015	com.webbyawards
510	19537886	2852	0.000011	com.brandyourself
511	19535564	2735	0.000012	jp.hatenablog
512	19534552	2741	0.000012	com.zynga
513	19533780	382	0.000066	org.acm
514	19532322	1841	0.000017	com.cmswire
515	19531950	431	0.000058	io.codepen
516	19531032	1343	0.000022	org.pocoo
517	19530112	2931	0.000011	uk.co.autocar
518	19529900	160	0.000158	com.tripadvisor
519	19529372	234	0.000099	org.drupal
520	19528028	991	0.000029	com.gizmodo
521	19525144	2317	0.000014	org.aei
522	19524148	864	0.000034	com.matterport
523	19523142	1871	0.000017	uk.co.thesundaytimes
524	19521230	1041	0.000027	com.tinypic
525	19520944	813	0.000036	com.netflix
526	19520420	2439	0.000013	com.newatlas
527	19518764	2410	0.000013	com.triplepundit
528	19518666	381	0.000066	com.booking
529	19518320	2978	0.000011	fr.hellocoton
530	19517366	2201	0.000015	org.unfpa
531	19516300	1603	0.000019	pt.google
532	19514002	1715	0.000018	net.openid
533	19511332	3080	0.000011	com.blogsky
534	19511240	1763	0.000017	com.bloglines
535	19508014	272	0.000089	com.adnxs
536	19507232	2106	0.000015	org.royalsociety
537	19506586	2659	0.000012	com.asiaone
538	19504284	2308	0.000014	com.waterstones
539	19503858	2342	0.000014	com.financialexpress
540	19503212	1639	0.000019	uk.org.nationaltrust
541	19502772	1646	0.000019	org.pypi
542	19501202	899	0.000032	com.highcharts
543	19500790	1889	0.000016	org.panda
544	19500702	2898	0.000011	org.ifaw
545	19500700	1828	0.000017	org.thinkprogress
546	19499724	901	0.000032	com.arstechnica
547	19498236	2203	0.000015	com.kaggle
548	19497656	1948	0.000016	org.wri
549	19494804	2693	0.000012	co.electrek
550	19493786	2306	0.000014	uk.org.wwf
551	19493426	2436	0.000013	com.mongabay
552	19493282	3319	0.000010	com.carscoops
553	19492162	1082	0.000027	com.mixpanel
554	19486550	1502	0.000020	io.fabric
555	19486258	1269	0.000023	com.firebaseapp
556	19485830	906	0.000032	edu.psu
557	19484868	1848	0.000017	com.infolinks
558	19484056	1647	0.000018	com.coschedule
559	19481940	1672	0.000018	us.pa.state
560	19480200	2209	0.000015	uk.ac.nhm
561	19479650	1302	0.000023	com.clicky
562	19477726	500	0.000049	tv.twitch
563	19477544	532	0.000047	edu.cornell
564	19477084	872	0.000033	edu.washington
565	19476626	71	0.000478	com.livestream
566	19475600	2307	0.000014	com.autonews
567	19474520	2660	0.000012	pt.publico
568	19474486	1929	0.000016	org.americanprogress
569	19474190	2578	0.000012	com.nordvpn
570	19473972	2206	0.000015	org.sonatype
571	19471930	1457	0.000021	com.activecampaign
572	19471612	625	0.000041	com.samsung
573	19471306	2730	0.000012	com.delawareonline
574	19470860	2848	0.000011	com.topgear
575	19468240	999	0.000029	edu.upenn
576	19465494	1760	0.000017	uk.gov.metoffice
577	19464352	2733	0.000012	com.sc
578	19464298	2573	0.000013	br.inpe
579	19460386	1873	0.000017	com.prweek
580	19460086	2589	0.000012	com.ecowatch
581	19459484	72	0.000477	net.jsfiddle
582	19458590	3293	0.000010	com.algorithmia
583	19457214	2027	0.000016	com.scotsman
584	19457126	429	0.000058	com.slack
585	19455372	1887	0.000016	com.impactbnd
586	19453748	1008	0.000029	uk.ac.cam
587	19453316	2263	0.000014	com.articulate
588	19453140	2780	0.000012	com.nouw
589	19451266	2896	0.000011	com.flock
590	19449038	2571	0.000013	org.globalcitizen
591	19447006	538	0.000046	com.proofpoint
592	19445998	2335	0.000014	com.googledrive
593	19444262	2434	0.000013	nz.co.radionz
594	19444224	2763	0.000012	jp.riken
595	19443690	2388	0.000013	de.greenpeace
596	19443190	119	0.000244	com.youku
597	19442118	174	0.000141	jp.co.yahoo
598	19441598	2831	0.000011	com.mumsnet
599	19439924	1874	0.000017	com.crashlytics
600	19439174	965	0.000030	edu.umich
601	19439028	2114	0.000015	uk.org.rspb
602	19438028	208	0.000116	uk.co.amazon
603	19437448	101	0.000321	de.google
604	19435790	2748	0.000012	com.quickanddirtytips
605	19431834	2668	0.000012	au.com.huffingtonpost
606	19431216	1896	0.000016	uk.gov.london
607	19430698	2541	0.000013	com.thejakartapost
608	19429486	3097	0.000011	com.shanghaidaily
609	19428860	415	0.000061	com.xinhuanet
610	19428614	3069	0.000011	com.theminimalists
611	19428486	1271	0.000023	com.sprinklr
612	19426496	1208	0.000025	org.iea
613	19426466	2512	0.000013	ie.thejournal
614	19426152	1785	0.000017	com.jeffbullas
615	19424902	2979	0.000011	com.art
616	19424640	2837	0.000011	it.polito
617	19423008	1808	0.000017	com.martechtoday
618	19422426	2599	0.000012	uk.co.profilebusiness
619	19421492	2534	0.000013	com.db
620	19420756	2851	0.000011	org.onegreenplanet
621	19418396	2340	0.000014	net.opendemocracy
622	19416952	1869	0.000017	org.iucnredlist
623	19413908	2688	0.000012	uk.org.savethechildren
624	19412614	2379	0.000014	com.theyworkforyou
625	19411666	695	0.000037	com.xiti
626	19409198	2661	0.000012	org.oceanconservancy
627	19408718	2683	0.000012	com.dreamgrow
628	19407976	2254	0.000014	com.rabbitmq
629	19407372	2568	0.000013	com.shoutmeloud
630	19407170	1028	0.000028	com.mcafeesecure
631	19406866	449	0.000055	fr.free
632	19403640	362	0.000069	org.npr
633	19402072	1865	0.000017	com.copyscape
634	19401308	2791	0.000012	com.sitesell
635	19400880	312	0.000079	gov.cdc
636	19399828	2423	0.000013	com.cleantechnica
637	19399686	2809	0.000012	pl.edu.uw
638	19397274	399	0.000063	com.nypost
639	19396828	569	0.000044	com.aol
640	19396446	3167	0.000010	com.seeker
641	19396390	2760	0.000012	uk.org.amnesty
642	19396212	265	0.000090	com.sohu
643	19395962	1613	0.000019	com.flashtalking
644	19395308	2516	0.000013	com.generalmills
645	19393472	2049	0.000016	com.cityam
646	19392474	3380	0.000010	com.dremel
647	19392370	396	0.000063	com.163
648	19391762	3020	0.000011	com.brothersoft
649	19391670	2061	0.000016	org.gnupg
650	19388022	36	0.001003	com.createjs
651	19387660	1027	0.000028	edu.ucla
652	19386630	511	0.000048	com.dmca
653	19385442	1495	0.000020	scot.gov
654	19383806	2347	0.000014	org.grist
655	19383592	2474	0.000013	uk.org.oxfam
656	19381766	2457	0.000013	uk.co.thisismoney
657	19380480	3259	0.000010	org.aqicn
658	19379848	2566	0.000013	uk.org.rspca
659	19379190	1169	0.000025	com.hollywoodreporter
660	19378746	2726	0.000012	org.irena
661	19377826	2908	0.000011	org.kuow
662	19375866	2934	0.000011	eu.i-scoop
663	19375282	3137	0.000011	com.winefolly
664	19374230	244	0.000096	com.bandcamp
665	19373806	1350	0.000022	net.leadpages
666	19371298	1855	0.000017	net.noscript
667	19370726	1438	0.000021	com.pastebin
668	19370120	2692	0.000012	com.targetmarketingmag
669	19368524	3516	0.000010	co.edureka
670	19368376	2773	0.000012	com.ipsos-mori
671	19368284	2546	0.000013	org.zsl
672	19368044	2393	0.000013	com.moodys
673	19367896	1170	0.000025	gov.fbi
674	19367686	2182	0.000015	com.thermofisher
675	19366198	2800	0.000012	uk.ac.ceh
676	19365484	273	0.000089	com.surveymonkey
677	19364456	1703	0.000018	uk.co.which
678	19363118	1431	0.000021	uk.gov.defra
679	19362092	2626	0.000012	com.wikidot
680	19361864	2112	0.000015	com.problogger
681	19361432	2794	0.000012	com.pnsegypt
682	19360486	3132	0.000011	com.hatenadiary
683	19359572	169	0.000143	com.taobao
684	19359506	333	0.000074	com.pubmatic
685	19358770	377	0.000066	com.scribd
686	19358748	2985	0.000011	org.storyofstuff
687	19358106	3168	0.000010	org.heartland
688	19356998	2902	0.000011	com.nationalgrid
689	19355728	352	0.000070	com.wiley
690	19355014	886	0.000033	com.windowsphone
691	19351528	2511	0.000013	uk.gov.forestry
692	19349818	2746	0.000012	org.spie
693	19349596	816	0.000036	com.mobirise
694	19346822	2963	0.000011	uk.ac.mdx
695	19345936	463	0.000054	com.oreilly
696	19345228	2298	0.000014	com.iconarchive
697	19344974	3213	0.000010	edu.uah
698	19344130	893	0.000032	edu.columbia
699	19343846	2196	0.000015	uk.gov.food
700	19342492	2770	0.000012	edu.dukeupress
701	19341928	2518	0.000013	com.wral
702	19337306	1239	0.000024	google.blog
703	19337180	453	0.000055	com.sxsw
704	19337108	686	0.000038	com.steampowered
705	19332972	2891	0.000011	com.almanac
706	19332496	915	0.000031	com.docker
707	19332138	433	0.000057	com.force
708	19330890	913	0.000032	org.reactjs
709	19330434	3158	0.000011	com.dbs
710	19330012	3320	0.000010	uk.org.bornfree
711	19329944	1283	0.000023	uk.org.greenpeace
712	19328328	1100	0.000026	com.redhat
713	19328004	1248	0.000024	com.elpais
714	19327924	785	0.000036	com.webs
715	19324934	3401	0.000010	org.sciencenewsforstudents
716	19324548	3476	0.000010	org.sharktrust
717	19323678	3447	0.000010	uk.org.caat
718	19322218	305	0.000080	com.digg
719	19320384	325	0.000076	com.typeform
720	19320196	2756	0.000012	com.batchgeo
721	19319558	2116	0.000015	com.fifa
722	19317480	2389	0.000013	org.chathamhouse
723	19317116	1322	0.000023	org.whatbrowser
724	19317094	2098	0.000016	org.fsc
725	19316024	1706	0.000018	com.nike
726	19315926	2357	0.000014	uk.co.inews
727	19315824	1362	0.000022	edu.ucsd
728	19315458	3400	0.000010	com.artstation
729	19315386	855	0.000034	org.unesco
730	19315260	2654	0.000012	com.ingress
731	19313414	1561	0.000019	com.technologyreview
732	19312758	2375	0.000014	io.pantheon
733	19311846	2952	0.000011	com.climatechangenews
734	19311082	2981	0.000011	org.c2es
735	19309714	1771	0.000017	com.ikea
736	19309506	3010	0.000011	com.foodsafetynews
737	19306598	2574	0.000012	uk.org.38degrees
738	19305744	2676	0.000012	com.thecvf
739	19305478	2588	0.000012	org.carbonbrief
740	19305458	2990	0.000011	org.sourcewatch
741	19304968	571	0.000043	com.cbsnews
742	19304594	2986	0.000011	com.moneysupermarket
743	19304168	469	0.000053	com.statista
744	19304094	3414	0.000010	me.start
745	19301508	2844	0.000011	com.tiddlywiki
746	19299692	2645	0.000012	com.bnef
747	19298620	3095	0.000011	uk.co.bristolpost
748	19297446	198	0.000122	io.polyfill
749	19297002	3059	0.000011	jp.ac.kobe-u
750	19296802	122	0.000238	org.networkadvertising
751	19296318	502	0.000049	com.atlassian
752	19294076	338	0.000073	com.prnewswire
753	19291522	1128	0.000026	com.canva
754	19288978	3012	0.000011	org.twinery
755	19288828	2737	0.000012	com.adcolony
756	19288458	3117	0.000011	no.forskning
757	19286246	2785	0.000012	com.doctoroz
758	19284850	3556	0.000010	com.cmgdigital
759	19284678	3143	0.000011	com.sunherald
760	19284062	3172	0.000010	com.ibmbigdatahub
761	19283992	3517	0.000010	com.2createawebsite
762	19283716	2996	0.000011	net.organicfacts
763	19282858	2243	0.000014	com.privacypolicies
764	19282122	2905	0.000011	com.winemag
765	19281746	1056	0.000027	com.ubuntu
766	19281512	1419	0.000021	uk.co.thesun
767	19281086	470	0.000053	com.inc
768	19281010	2143	0.000015	org.cites
769	19280990	2290	0.000014	uk.gov.dft
770	19279280	3146	0.000011	com.insideevs
771	19279174	2734	0.000012	de.ksta
772	19278422	2684	0.000012	com.e-activist
773	19278376	1412	0.000021	com.speakerdeck
774	19276894	2747	0.000012	com.chubb
775	19273916	2608	0.000012	org.rspo
776	19273894	964	0.000030	net.2mdn
777	19273142	3265	0.000010	com.jordantimes
778	19272034	319	0.000078	gov.ca
779	19268910	3506	0.000010	com.idt
780	19268426	2757	0.000012	com.theinnovationenterprise
781	19267542	2349	0.000014	uk.gov.environment-agency
782	19267478	3496	0.000010	com.sutori
783	19266406	151	0.000163	ru.mail
784	19266224	164	0.000152	com.yelp
785	19265510	3184	0.000010	com.galvanize
786	19264800	3425	0.000010	com.thewritepractice
787	19264778	3212	0.000010	org.carbontracker
788	19264570	3464	0.000010	org.earthworksaction
789	19263548	1713	0.000018	com.martechseries
790	19262638	981	0.000029	com.visualstudio
791	19262168	3383	0.000010	com.nutraingredients
792	19261694	3222	0.000010	com.quandl
793	19261452	1484	0.000020	uk.co.foe
794	19260924	232	0.000100	to.amzn
795	19260174	1731	0.000018	org.khanacademy
796	19260130	2699	0.000012	com.businessgreen
797	19259920	524	0.000047	com.airbnb
798	19259634	3200	0.000010	com.thedrinksbusiness
799	19258704	3384	0.000010	com.monbiot
800	19258488	2685	0.000012	au.com.mumbrella
801	19257102	3072	0.000011	fr.thelocal
802	19256728	3330	0.000010	org.cnduk
803	19256628	660	0.000039	org.eff
804	19256476	1441	0.000021	com.tutsplus
805	19255922	3090	0.000011	ai.fast
806	19255422	2723	0.000012	com.goinswriter
807	19255170	3358	0.000010	org.thechicagocouncil
808	19253936	3029	0.000011	jp.hatenadiary
809	19252730	2743	0.000012	gov.ferc
810	19252634	1384	0.000022	com.uber
811	19252094	3444	0.000010	com.visitdublin
812	19250954	2582	0.000012	nz.govt.mfat
813	19249844	2223	0.000015	uk.gov.charitycommission
814	19249406	1192	0.000025	edu.utexas
815	19249112	3273	0.000010	com.chemistryworld
816	19248998	3300	0.000010	org.alaskapublic
817	19248984	1418	0.000021	fr.lemonde
818	19248812	3144	0.000011	com.tuck
819	19247226	3156	0.000011	com.marksdailyapple
820	19246284	1005	0.000029	com.americanexpress
821	19246204	579	0.000043	com.patreon
822	19245062	2814	0.000012	com.ing
823	19245032	166	0.000147	jp.co.google
824	19244244	1932	0.000016	uk.gov.education
825	19242896	2753	0.000012	com.webestools
826	19242502	2504	0.000013	com.instructables
827	19242460	1185	0.000025	edu.princeton
828	19240552	3645	0.000010	com.theppk
829	19240536	3305	0.000010	com.machinelearningmastery
830	19238864	1716	0.000018	se.haxx
831	19238712	1149	0.000026	com.digiday
832	19238462	896	0.000032	com.zoho
833	19238268	4669	0.000009	com.9to5mac
834	19237602	3761	0.000010	org.muslimaid
835	19235836	541	0.000046	com.alibaba
836	19235736	2817	0.000012	uk.ac.rcplondon
837	19233882	556	0.000045	gov.sec
838	19232880	3043	0.000011	com.platts
839	19232688	2651	0.000012	com.recyclenow
840	19232618	3441	0.000010	org.thebestschools
841	19231994	3352	0.000010	com.beruby
842	19231826	202	0.000119	com.constantcontact
843	19231002	2354	0.000014	net.privacypolicytemplate
844	19230142	3207	0.000010	com.gpsvisualizer
845	19227774	3104	0.000011	com.rabobank
846	19227216	3306	0.000010	com.seat61
847	19227198	3412	0.000010	uk.co.lep
848	19226122	311	0.000079	com.marriott
849	19224666	239	0.000098	cn.com.sina
850	19224282	753	0.000036	com.css-tricks
851	19223532	246	0.000095	jp.co.amazon
852	19222846	1299	0.000023	gd.is
853	19221828	2350	0.000014	uk.co.vogue
854	19221424	1381	0.000022	com.dell
855	19221118	722	0.000037	fm.last
856	19221104	2009	0.000016	io.getmdl
857	19220430	3756	0.000010	uk.org.stopwar
858	19220196	2627	0.000012	org.ramsar
859	19217988	1987	0.000016	com.instapage
860	19217434	595	0.000042	com.psychologytoday
861	19217202	3592	0.000010	com.fox13memphis
862	19216396	3134	0.000011	uk.org.sja
863	19216342	3538	0.000010	com.breakingenergy
864	19216070	3436	0.000010	com.star2
865	19215784	3103	0.000011	org.scielo
866	19215692	97	0.000332	com.sharethis
867	19215686	881	0.000033	com.aliexpress
868	19215320	3613	0.000010	it.diggita
869	19214836	210	0.000116	jp.ne.hatena
870	19214614	1125	0.000026	com.firefox
871	19214492	634	0.000040	gov.nist
872	19212940	3252	0.000010	org.beatthemicrobead
873	19212372	3603	0.000010	nl.zoom
874	19212310	1232	0.000024	com.convertkit
875	19207820	545	0.000046	uk.co.eventbrite
876	19207334	3145	0.000011	com.abnamro
877	19206384	2904	0.000011	org.wildlifetrusts
878	19206088	1637	0.000019	org.whales
879	19205750	1068	0.000027	com.shutterstock
880	19204676	3981	0.000009	com.visitguatemala
881	19203846	3128	0.000011	uk.org.scope
882	19203288	1030	0.000028	com.foxnews
883	19203148	2675	0.000012	org.soilassociation
884	19202842	1019	0.000028	com.cbslocal
885	19200848	3702	0.000010	no.haugenbok
886	19199586	2914	0.000011	com.ironsrc
887	19199426	952	0.000030	com.variety
888	19199344	2622	0.000012	com.feedreader
889	19198876	517	0.000048	com.ea
890	19198322	3595	0.000010	uk.co.theboltonnews
891	19198052	1190	0.000025	com.globo
892	19196818	2863	0.000011	com.itsma
893	19196098	1564	0.000019	org.freecsstemplates
894	19195978	2078	0.000016	com.hulu
895	19195672	3148	0.000011	com.rebekahradice
896	19195262	289	0.000084	com.discordapp
897	19194762	3543	0.000010	info.e-ir
898	19194546	3364	0.000010	org.swi-prolog
899	19192288	3182	0.000010	com.wpxi
900	19191698	486	0.000051	com.nasdaq
901	19190864	3170	0.000010	uk.co.dennis
902	19190668	3551	0.000010	com.alaskadispatch
903	19190460	1150	0.000026	com.java
904	19190276	230	0.000100	com.googletagservices
905	19189676	3504	0.000010	es.ree
906	19189562	3162	0.000010	com.sgx
907	19188626	3721	0.000010	br.org.imazon
908	19188334	3776	0.000010	com.citymayors
909	19188182	3582	0.000010	au.com.hotfrog
910	19187066	3272	0.000010	uk.org.cat
911	19186280	3279	0.000010	aq.ats
912	19186188	678	0.000038	com.newyorker
913	19184690	4002	0.000009	net.politicalscrapbook
914	19184344	3606	0.000010	com.southernfriedscience
915	19183484	3125	0.000011	app.web
916	19182578	220	0.000106	com.naver
917	19181770	1736	0.000018	com.techrepublic
918	19180978	3632	0.000010	com.theoildrum
919	19180728	3728	0.000010	org.worldnuclearreport
920	19180674	156	0.000162	gov.privacyshield
921	19179118	2871	0.000011	uk.co.realbusiness
922	19177254	1463	0.000021	edu.uchicago
923	19176726	1453	0.000021	tv.ustream
924	19175164	1807	0.000017	com.nba
925	19173352	3271	0.000010	uk.org.cpre
926	19173188	1788	0.000017	org.golang
927	19172402	2955	0.000011	com.writetothem
928	19172368	2041	0.000016	com.howstuffworks
929	19170896	1407	0.000021	uk.co.theregister
930	19170668	464	0.000054	com.adweek
931	19170630	243	0.000096	com.stumbleupon
932	19170480	1579	0.000019	edu.unc
933	19169444	2211	0.000015	edu.virginia
934	19168860	3619	0.000010	com.renewablesnow
935	19168518	1390	0.000022	com.over-blog
936	19167800	1443	0.000021	com.digitaltrends
937	19167782	4073	0.000009	uk.co.moblog
938	19165140	1406	0.000021	us.imageshack
939	19164724	3604	0.000010	com.at0086
940	19164492	2144	0.000015	org.coursera
941	19164428	3799	0.000010	com.avivaromm
942	19162582	984	0.000029	com.thinkwithgoogle
943	19162450	3629	0.000010	com.eremnews
944	19161660	466	0.000053	com.snapchat
945	19159918	1442	0.000021	com.billboard
946	19159904	3394	0.000010	uk.gov.peterborough
947	19159506	3530	0.000010	org.campaigncc
948	19158600	642	0.000039	org.pbs
949	19157590	3299	0.000010	uk.co.siemens
950	19157574	3470	0.000010	org.ilga-europe
951	19156258	978	0.000029	com.dropboxusercontent
952	19154380	894	0.000032	com.uservoice
953	19154252	1578	0.000019	com.ssllabs
954	19153992	3367	0.000010	com.trafficgenerationcafe
955	19152256	1614	0.000019	com.warnerbros
956	19152042	922	0.000031	com.libsyn
957	19151852	3665	0.000010	uk.org.biofuelwatch
958	19151718	3617	0.000010	uk.org.garyhall
959	19151548	2399	0.000013	com.ehow
960	19150820	3771	0.000010	no.universitetsforlaget
961	19148430	3559	0.000010	br.org.idec
962	19148290	839	0.000035	com.qz
963	19148164	2911	0.000011	net.nend
964	19147422	690	0.000038	com.webmd
965	19147238	1694	0.000018	com.codeplex
966	19144852	1374	0.000022	com.fiverr
967	19144584	3922	0.000009	net.kjokkenutstyr
968	19144572	497	0.000049	edu.cmu
969	19144158	4006	0.000009	org.freedom-now
970	19143932	1167	0.000025	com.smashingmagazine
971	19143604	3147	0.000011	uk.org.refill
972	19143372	1971	0.000016	com.invisionapp
973	19142256	2864	0.000011	com.dzone
974	19142158	3490	0.000010	io.dataquest
975	19141538	3984	0.000009	org.alqaws
976	19141230	3242	0.000010	io.dropwizard
977	19140662	3821	0.000010	com.superiorthreads
978	19140420	3398	0.000010	uk.co.firstnews
979	19138994	355	0.000070	org.debian
980	19138154	2134	0.000015	com.w3layouts
981	19134388	877	0.000033	com.foursquare
982	19134040	3035	0.000011	com.vungle
983	19133716	3205	0.000010	org.corporateeurope
984	19133644	866	0.000034	gov.census
985	19133496	3995	0.000009	com.tinnedtomatoes
986	19133382	1400	0.000021	com.blackberry
987	19133336	1335	0.000022	jp.livedoor
988	19132476	3334	0.000010	com.drillordrop
989	19131836	3329	0.000010	com.ovoenergy
990	19131172	3804	0.000010	com.descarteslabs
991	19130778	1064	0.000027	com.politico
992	19128888	3788	0.000010	org.ianfairlie
993	19128728	1866	0.000017	com.nokia
994	19127786	2669	0.000012	in.bbc
995	19127512	2987	0.000011	org.vegsoc
996	19127108	3387	0.000010	com.figure-eight
997	19125818	357	0.000070	gov.ftc
998	19125596	242	0.000097	org.icann
999	19125358	1401	0.000021	com.xkcd
1000	19125216	3609	0.000010	br.com.ambev

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

October 2019 crawl archive now available

October 29, 2019Sebastian Nagel

The crawl archive for October 2019 is now available! It contains 3.0 billion web pages or 280 TiB of uncompressed content, crawled between October 13th and 24th. It includes page captures of 1.1 billion URLs not contained in any crawl archive before.

Archive Location and Download

The October crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-43/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-43/segment.paths.gz	100
WARC files	CC-MAIN-2019-43/warc.paths.gz	56000	59.56
WAT files	CC-MAIN-2019-43/wat.paths.gz	56000	21.7
WET files	CC-MAIN-2019-43/wet.paths.gz	56000	9.94
Robots.txt files	CC-MAIN-2019-43/robotstxt.paths.gz	56000	0.15
Non-200 responses files	CC-MAIN-2019-43/non200responses.paths.gz	56000	1.69
URL index files	CC-MAIN-2019-43/cc-index.paths.gz	302	0.22

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-43/. Also the columnar index has been updated to contain this crawl.

September 2019 crawl archive now available

September 28, 2019Sebastian Nagel

The crawl archive for September 2019 is now available! It contains 2.55 billion web pages or 240 TiB of uncompressed content, crawled between September 15th and 24th. It includes page captures of 1.0 billion URLs not contained in any crawl archive before. The other 1.5 billion pages have been already captured in prior crawls and are now revisited.

Archive Location and Download

The September crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-39/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-39/segment.paths.gz	100
WARC files	CC-MAIN-2019-39/warc.paths.gz	56000	55.99
WAT files	CC-MAIN-2019-39/wat.paths.gz	56000	18.01
WET files	CC-MAIN-2019-39/wet.paths.gz	56000	8.16
Robots.txt files	CC-MAIN-2019-39/robotstxt.paths.gz	56000	0.14
Non-200 responses files	CC-MAIN-2019-39/non200responses.paths.gz	56000	1.63
URL index files	CC-MAIN-2019-39/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-39/. Also the columnar index has been updated to contain this crawl.

August 2019 crawl archive now available

August 30, 2019Sebastian Nagel

The crawl archive for August 2019 is now available! It contains 2.95 billion web pages or 260 TiB of uncompressed content, crawled between August 17th and 26th.

The August crawl contains page captures of 1.1 billion URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks (harmonic centrality) published as part of the May/Jun/Jul 2019 webgraph data set from the following sources:

a random sample of 2.1 billion outlinks extracted from July crawl WAT files
1.8 billion URLs mined in a breadth-first side crawl within a maximum of 6 links (“hops”), started from
- the homepages of the top 60 million hosts and domains and randomly selected samples of
- 2 million human-readable sitemap pages (HTML format)
- 3 million URLs of pages written in 130 less-represented languages (cf. language distributions)
1 billion URLs extracted and sampled from 20 million sitemaps, RSS and Atom feeds

Starting with this crawl the following fixes and improvements are applied to the provided data formats:

reliable marking of WARC records with truncated payload, see issue report “WARC-Truncated header”
improved decoding of XML/HTML character entities in WAT and WET files

Archive Location and Download

The August crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-35/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-35/segment.paths.gz	100
WARC files	CC-MAIN-2019-35/warc.paths.gz	56000	53.53
WAT files	CC-MAIN-2019-35/wat.paths.gz	56000	20.85
WET files	CC-MAIN-2019-35/wet.paths.gz	56000	9.29
Robots.txt files	CC-MAIN-2019-35/robotstxt.paths.gz	56000	0.18
Non-200 responses files	CC-MAIN-2019-35/non200responses.paths.gz	56000	1.79
URL index files	CC-MAIN-2019-35/cc-index.paths.gz	302	0.22

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-35/. Also the columnar index has been updated to contain this crawl.