Blog – Page 5 – Common Crawl

Host- and Domain-Level Web Graphs Feb/Mar/Apr 2019

May 9, 2019Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of February, March and April 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

What’s new?

The software which builds the graph from WAT and WARC files has been extended to extract more links from the HTML <head> element:

more links are taken from <metadata> elements, e.g, the thumbnail meta name, Open Graph or twitter:* properties
links from <script> elements are now included

Note that previous web graph releases already include all kinds of links: not only <a href="..."> but also links to images and multi-media content, links from <form> elements, canonical links, and many more.

While the domain-level graph shows almost the same size and metrics as the previous one released three months ago, the host-level graph has increased in size by 85 million nodes but is less densely connected. The growth in the number of nodes is mainly caused by a link spam cluster of 190 million hosts distributed over 15k domains. Thanks to the webgraph these domains (e.g., 24340.tw) are detected and the crawler is advised not to visit them again.

Host-level graph

The graph consists of 492 million nodes and 3.0 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 426 million dangling nodes (87%) and the largest strongly connected component contains 52 million (10.5%) nodes.

You can download the graph and the ranks of all 492 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-feb-mar-apr/host/. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2019-feb-mar-apr/host/ as prefix to access the files from everywhere.

Download files of the Common Crawl Feb/Mar/Apr 2019 host-level webgraph

Size	File	Description
3.36 GB	cc-main-2019-feb-mar-apr-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 28 vertices files
14.40 GB	cc-main-2019-feb-mar-apr-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 56 edges files

6.33 GB	cc-main-2019-feb-mar-apr-host.graph	graph in BVGraph format
2 kB	cc-main-2019-feb-mar-apr-host.properties

7.02 GB	cc-main-2019-feb-mar-apr-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2019-feb-mar-apr-host-t.properties

1 kB	cc-main-2019-feb-mar-apr-host.stats	WebGraph statistics

7.85 GB	cc-main-2019-feb-mar-apr-host-ranks.txt.gz	harmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 91 million nodes and 1.89 billion edges. 51% or 46 million nodes are dangling nodes, the largest strongly connected component covers 38 million or 42% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-feb-mar-apr/domain/ resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2019-feb-mar-apr/domain/.

Download files of the Common Crawl Feb/Mar/Apr 2019 domain-level webgraph

Size	File	Description
0.63 GB	cc-main-2019-feb-mar-apr-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
7.48 GB	cc-main-2019-feb-mar-apr-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

4.01 GB	cc-main-2019-feb-mar-apr-domain.graph	graph in BVGraph format
2 kB	cc-main-2019-feb-mar-apr-domain.properties

4.02 GB	cc-main-2019-feb-mar-apr-domain-t.graph	transpose of the graph
2 kB	cc-main-2019-feb-mar-apr-domain-t.properties

1 kB	cc-main-2019-feb-mar-apr-domain.stats	WebGraph statistics

1.98 GB	cc-main-2019-feb-mar-apr-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 90 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Feb/Mar/Apr 2019)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	29096444	1	0.020470	com.googleapis
2	28007982	3	0.012308	com.facebook
3	26801494	2	0.013202	com.google
4	24862042	4	0.006929	com.twitter
5	24696428	5	0.006794	com.youtube
6	24174048	6	0.006211	org.w
7	22325176	9	0.003651	com.instagram
8	22277546	7	0.004565	org.gmpg
9	21663616	13	0.002903	com.linkedin
10	21253740	8	0.003880	com.googletagmanager
11	20994008	22	0.001629	com.gravatar
12	20770844	11	0.003144	com.cloudflare
13	20763426	12	0.002915	org.wordpress
14	20723350	15	0.002103	com.wordpress
15	20597380	19	0.001856	com.pinterest
16	20589106	26	0.001344	org.wikipedia
17	20538196	14	0.002455	com.bootstrapcdn
18	20340120	18	0.001857	com.apple
19	20168332	28	0.001208	com.vimeo
20	20100244	40	0.000945	com.blogspot
21	20076842	21	0.001721	com.jquery
22	19900496	44	0.000861	gl.goo
23	19874514	49	0.000756	be.youtu
24	19845858	24	0.001528	com.adobe
25	19808478	27	0.001240	com.microsoft
26	19798758	50	0.000749	com.amazon
27	19710384	60	0.000607	com.tumblr
28	19689216	53	0.000667	com.wp
29	19584236	34	0.001016	com.amazonaws
30	19567898	25	0.001456	com.macromedia
31	19563206	88	0.000433	com.yahoo
32	19557094	51	0.000734	com.flickr
33	19526652	42	0.000880	com.google-analytics
34	19499686	80	0.000502	ly.bit
35	19489648	32	0.001034	com.googlesyndication
36	19472580	62	0.000580	org.mozilla
37	19466578	23	0.001557	com.gstatic
38	19459466	31	0.001093	net.cloudfront
39	19428390	20	0.001823	com.github
40	19302390	66	0.000558	me.wp
41	19278286	39	0.000949	net.doubleclick
42	19253848	46	0.000802	com.paypal
43	19222312	99	0.000316	com.googleusercontent
44	19214440	82	0.000487	com.medium
45	19194374	41	0.000882	com.squarespace
46	19181944	85	0.000440	com.weebly
47	19164390	79	0.000520	org.w3
48	19162880	127	0.000234	com.nytimes
49	19140860	86	0.000440	io.github
50	19138696	102	0.000307	com.reddit
51	19125448	92	0.000375	org.creativecommons
52	19052088	154	0.000166	net.slideshare
53	19050126	162	0.000162	com.theguardian
54	19047964	139	0.000189	com.imgur
55	19010700	57	0.000626	com.bing
56	19007804	136	0.000202	com.forbes
57	18975024	166	0.000158	net.sourceforge
58	18969344	217	0.000110	com.businessinsider
59	18964518	64	0.000566	org.schema
60	18930562	202	0.000115	com.myspace
61	18929412	161	0.000162	com.blogger
62	18929098	206	0.000113	com.techcrunch
63	18929096	188	0.000132	com.android
64	18907716	101	0.000313	com.mailchimp
65	18887038	251	0.000097	com.tinyurl
66	18886912	54	0.000649	com.baidu
67	18881598	249	0.000098	com.wired
68	18879314	91	0.000411	de.google
69	18872146	354	0.000068	com.photobucket
70	18870082	182	0.000140	com.stackoverflow
71	18844718	100	0.000316	org.ampproject
72	18842202	38	0.000953	org.apache
73	18829206	266	0.000090	com.bbc
74	18824202	103	0.000307	com.shopify
75	18821492	361	0.000068	com.quora
76	18818304	315	0.000076	com.appspot
77	18801506	37	0.000974	com.fontawesome
78	18797872	113	0.000275	com.ytimg
79	18796778	36	0.000976	com.addthis
80	18776040	209	0.000112	com.oracle
81	18766484	558	0.000045	org.chromium
82	18761466	353	0.000069	com.googleblog
83	18753718	380	0.000064	com.theverge
84	18728722	526	0.000047	org.ieee
85	18726166	510	0.000048	edu.washington
86	18724898	462	0.000053	com.economist
87	18724374	96	0.000330	com.statcounter
88	18720414	98	0.000317	com.soundcloud
89	18717340	151	0.000171	org.ietf
90	18714060	553	0.000046	edu.yale
91	18706362	319	0.000075	com.githubusercontent
92	18703214	300	0.000078	com.ted
93	18695606	61	0.000589	eu.europa
94	18694532	441	0.000056	com.venturebeat
95	18691888	235	0.000103	com.hubspot
96	18688542	655	0.000043	com.tinypic
97	18680252	144	0.000180	com.spotify
98	18673738	141	0.000185	com.yelp
99	18671830	133	0.000213	com.issuu
100	18662516	395	0.000063	com.cisco
101	18657986	93	0.000354	co.t
102	18652360	95	0.000340	com.sharethis
103	18648440	438	0.000056	com.deviantart
104	18644944	702	0.000040	edu.princeton
105	18644036	265	0.000090	com.sciencedirect
106	18635732	360	0.000068	me.about
107	18631710	460	0.000053	org.arxiv
108	18627650	279	0.000086	org.npr
109	18615844	179	0.000141	org.wikimedia
110	18608530	751	0.000038	google.blog
111	18606402	341	0.000071	com.theatlantic
112	18596892	345	0.000071	com.mozilla
113	18592502	572	0.000044	edu.ucla
114	18587904	454	0.000054	com.mysql
115	18584416	134	0.000211	com.dropbox
116	18581890	963	0.000033	com.jetbrains
117	18579386	121	0.000250	com.whatsapp
118	18576676	294	0.000081	com.example
119	18575790	81	0.000499	net.jsdelivr
120	18574812	271	0.000089	com.fastcompany
121	18567024	331	0.000072	com.typeform
122	18560472	400	0.000062	com.zdnet
123	18556406	468	0.000052	com.wikihow
124	18554544	30	0.001112	ru.yandex
125	18552784	540	0.000046	com.thenextweb
126	18551678	502	0.000049	com.git-scm
127	18550824	1063	0.000030	com.chrome
128	18546650	169	0.000156	com.salesforce
129	18542982	375	0.000065	uk.co.blogspot
130	18537770	443	0.000055	com.about
131	18535658	117	0.000263	org.networkadvertising
132	18535608	488	0.000050	com.pixabay
133	18526084	212	0.000112	com.dribbble
134	18525790	201	0.000116	com.stumbleupon
135	18524658	1434	0.000021	com.diigo
136	18509334	499	0.000049	com.ubuntu
137	18501930	741	0.000038	org.eclipse
138	18497596	505	0.000049	com.slate
139	18497252	208	0.000112	com.googlecode
140	18490168	58	0.000611	com.wix
141	18486604	425	0.000058	com.moz
142	18481186	191	0.000127	com.cnn
143	18475472	122	0.000242	com.stripe
144	18475424	181	0.000141	uk.co.bbc
145	18464340	652	0.000043	com.stackexchange
146	18462864	369	0.000066	com.entrepreneur
147	18458960	275	0.000087	com.nbcnews
148	18453376	253	0.000095	gov.ca
149	18444504	595	0.000044	com.withgoogle
150	18440430	518	0.000048	com.qz
151	18439414	542	0.000046	com.trello
152	18429892	214	0.000111	edu.stanford
153	18426238	1071	0.000030	edu.illinois
154	18424824	1040	0.000030	edu.gatech
155	18424290	293	0.000081	com.foursquare
156	18421718	1509	0.000020	org.wikibooks
157	18418178	573	0.000044	com.searchengineland
158	18417420	516	0.000048	com.unity3d
159	18414768	670	0.000042	org.sciencemag
160	18412040	267	0.000090	com.npmjs
161	18401668	463	0.000053	gov.loc
162	18397856	926	0.000034	com.sap
163	18397328	16	0.002068	com.wixstatic
164	18396224	1097	0.000029	edu.rutgers
165	18394270	156	0.000165	org.bbb
166	18392954	213	0.000111	es.google
167	18392682	661	0.000042	com.variety
168	18391296	155	0.000166	com.twimg
169	18381936	538	0.000046	com.libsyn
170	18380340	546	0.000046	com.evernote
171	18380064	174	0.000152	com.imdb
172	18378896	211	0.000112	com.wsj
173	18377090	33	0.001018	net.fbcdn
174	18373124	142	0.000185	gov.privacyshield
175	18369136	705	0.000040	com.techtarget
176	18368040	45	0.000851	com.fb
177	18365978	1190	0.000026	edu.utah
178	18365960	146	0.000180	org.archive
179	18365486	377	0.000065	com.getpocket
180	18358592	439	0.000056	gov.fda
181	18357482	194	0.000125	com.optimizely
182	18350542	419	0.000060	au.com.google
183	18346246	904	0.000034	com.econsultancy
184	18346130	210	0.000112	net.windows
185	18345860	1315	0.000024	com.douban
186	18345148	449	0.000055	org.freecodecamp
187	18338274	1321	0.000023	com.discogs
188	18338174	613	0.000043	uk.ac.ox
189	18335780	1019	0.000031	com.nike
190	18332678	1207	0.000026	org.tensorflow
191	18325660	75	0.000531	com.vk
192	18324804	287	0.000082	edu.mit
193	18323908	953	0.000033	com.buffer
194	18323654	1203	0.000026	com.aljazeera
195	18321184	1133	0.000028	ca.utoronto
196	18317982	813	0.000036	com.netlify
197	18315894	1079	0.000030	com.nvidia
198	18314254	531	0.000047	net.azurewebsites
199	18311612	350	0.000069	com.msn
200	18310748	984	0.000032	org.kernel
201	18307064	1123	0.000028	it.scoop
202	18305946	94	0.000346	com.paypalobjects
203	18299178	723	0.000039	com.indeed
204	18298870	807	0.000036	com.mixcloud
205	18296584	236	0.000103	com.live
206	18291116	1121	0.000028	org.postgresql
207	18289064	810	0.000036	com.neilpatel
208	18280540	365	0.000067	com.discordapp
209	18269300	1217	0.000026	ms.1drv
210	18269146	942	0.000034	com.business2community
211	18267622	303	0.000078	com.reuters
212	18266166	389	0.000064	gov.nasa
213	18263344	1452	0.000021	com.makeuseof
214	18261880	153	0.000168	gov.nih
215	18261660	444	0.000055	com.udacity
216	18257262	1289	0.000024	com.hostgator
217	18253994	996	0.000032	com.chron
218	18252416	264	0.000091	com.ibm
219	18243366	1024	0.000031	com.socialmediaexaminer
220	18240594	1126	0.000028	com.trendmicro
221	18238016	203	0.000114	com.washingtonpost
222	18236948	1117	0.000028	com.computerworld
223	18235154	568	0.000045	com.images-amazon
224	18232544	180	0.000141	com.etsy
225	18231292	1340	0.000023	io.itch
226	18226226	773	0.000037	co.g
227	18218104	1206	0.000026	edu.osu
228	18215060	968	0.000033	com.yoast
229	18211670	1277	0.000024	com.hbo
230	18210326	190	0.000130	com.ebay
231	18208488	304	0.000077	com.cnet
232	18207542	291	0.000082	edu.harvard
233	18207312	1766	0.000017	com.pearltrees
234	18206394	956	0.000033	com.mediafire
235	18206036	715	0.000039	site.business
236	18201166	56	0.000629	net.akamaihd
237	18200054	1055	0.000030	com.healthline
238	18198490	483	0.000051	com.usnews
239	18196662	196	0.000120	com.huffingtonpost
240	18192554	1144	0.000027	com.bustle
241	18190242	1001	0.000032	com.me
242	18188000	557	0.000045	org.d3js
243	18186074	165	0.000159	com.eventbrite
244	18185850	87	0.000437	com.list-manage
245	18185730	1737	0.000018	com.panoramio
246	18184450	384	0.000064	com.mashable
247	18181712	465	0.000053	edu.berkeley
248	18181104	803	0.000036	co.ibb
249	18180796	277	0.000087	com.bloomberg
250	18177922	760	0.000037	com.adjust
251	18177008	822	0.000036	com.ecwid
252	18174004	349	0.000070	com.mapbox
253	18172194	891	0.000034	gov.wa
254	18172152	1028	0.000031	org.aarp
255	18170224	1158	0.000027	edu.brookings
256	18169502	192	0.000127	org.iana
257	18165772	1250	0.000025	com.dw
258	18165654	1245	0.000025	com.medicalnewstoday
259	18163584	218	0.000109	net.php
260	18163574	420	0.000059	me.telegram
261	18163048	296	0.000081	org.acm
262	18162354	207	0.000112	org.gnu
263	18158198	1476	0.000021	com.sas
264	18152116	611	0.000043	me.paypal
265	18150248	1561	0.000020	com.dezeen
266	18150072	1088	0.000029	com.cio
267	18149100	799	0.000036	co.elastic
268	18148524	1293	0.000024	uk.org.tate
269	18147684	402	0.000062	com.latimes
270	18146412	199	0.000118	uk.co.amazon
271	18146172	455	0.000054	com.bigcommerce
272	18145158	1565	0.000020	be.blogspot
273	18144468	1260	0.000025	com.hackernoon
274	18143080	316	0.000076	uk.co.telegraph
275	18141808	1428	0.000022	com.googlesource
276	18141690	1448	0.000021	edu.iastate
277	18138884	1698	0.000018	org.edublogs
278	18137392	1559	0.000020	com.mathworks
279	18134024	1234	0.000025	gov.michigan
280	18132964	372	0.000066	com.livejournal
281	18132828	1140	0.000028	com.xrea
282	18132092	1623	0.000019	li.paper
283	18128158	47	0.000767	com.qq
284	18127880	1679	0.000018	com.dummies
285	18126586	143	0.000183	com.unpkg
286	18123856	1015	0.000031	com.searchenginejournal
287	18122824	939	0.000034	com.searchenginewatch
288	18120806	1802	0.000017	fr.unblog
289	18119164	286	0.000083	com.go
290	18114598	658	0.000042	com.livechatinc
291	18113078	149	0.000173	com.opera
292	18112360	757	0.000037	au.gov.nsw
293	18110326	1162	0.000027	va.vatican
294	18104616	1474	0.000021	jp.ac.u-tokyo
295	18104360	1016	0.000031	uk.co.pinterest
296	18103034	374	0.000065	com.elsevier
297	18097712	1412	0.000022	com.activecampaign
298	18097614	311	0.000076	com.meetup
299	18097106	1925	0.000016	com.jigsy
300	18094268	1084	0.000029	uk.gov.nationalarchives
301	18092632	1219	0.000026	us.mn.state
302	18091718	1246	0.000025	com.firebaseapp
303	18091416	1247	0.000025	com.convinceandconvert
304	18090158	1367	0.000023	us.fl.state
305	18083166	1700	0.000018	org.emojipedia
306	18079266	529	0.000047	com.adage
307	18079076	1192	0.000026	org.maven
308	18077604	1341	0.000023	gov.mo
309	18073450	43	0.000878	net.facebook
310	18070040	712	0.000039	gov.dot
311	18069866	164	0.000160	uk.co.google
312	18069354	89	0.000415	com.godaddy
313	18068256	172	0.000155	com.zendesk
314	18066418	222	0.000106	com.typepad
315	18066074	278	0.000087	com.usatoday
316	18062078	324	0.000074	com.mapquest
317	18057670	1423	0.000022	gov.ky
318	18052106	1687	0.000018	com.manta
319	18050286	426	0.000058	org.hbr
320	18050186	490	0.000050	net.researchgate
321	18049408	327	0.000073	com.getclicky
322	18049396	1398	0.000022	com.convertkit
323	18048286	2155	0.000014	it.justpaste
324	18048048	1348	0.000023	com.creativebloq
325	18047592	1544	0.000020	org.aclweb
326	18045082	889	0.000034	com.wordstream
327	18043064	1539	0.000020	ly.snip
328	18043014	227	0.000105	com.giphy
329	18040924	2014	0.000015	me.websta
330	18040570	356	0.000068	com.sxsw
331	18040058	840	0.000035	edu.psu
332	18037266	1365	0.000023	gov.maryland
333	18032816	1813	0.000017	ca.yelp
334	18031110	1187	0.000026	com.fastcodesign
335	18030678	1685	0.000018	io.material
336	18030052	1494	0.000021	org.amnesty
337	18027220	269	0.000089	org.python
338	18026066	522	0.000048	org.mediawiki
339	18026002	479	0.000051	com.buzzfeed
340	18024772	1150	0.000027	com.findlaw
341	18023106	783	0.000037	com.arstechnica
342	18022710	382	0.000064	com.oreilly
343	18022180	1667	0.000019	edu.toronto
344	18019428	1703	0.000018	com.healthgrades
345	18018232	2078	0.000015	tl.page
346	18016586	477	0.000051	edu.cornell
347	18013106	342	0.000071	com.springer
348	18008962	404	0.000062	it.placehold
349	18006368	1566	0.000020	com.raywenderlich
350	18005638	383	0.000064	com.nypost
351	18005080	937	0.000034	com.contentmarketinginstitute
352	18002754	325	0.000073	int.who
353	18001954	513	0.000048	org.nodejs
354	17999682	1524	0.000020	gov.mt
355	17997974	1345	0.000023	us.pa.state
356	17996652	329	0.000073	com.cnbc
357	17993466	1363	0.000023	gov.oregon
358	17990558	1066	0.000030	com.bandsintown
359	17988684	392	0.000063	com.gmail
360	17988550	1734	0.000018	com.wayfair
361	17984480	332	0.000072	fr.free
362	17980878	237	0.000102	org.drupal
363	17979326	910	0.000034	com.angieslist
364	17979146	450	0.000055	com.kickstarter
365	17979024	2154	0.000014	com.brandyourself
366	17978800	399	0.000062	uk.co.dailymail
367	17978276	1352	0.000023	com.quicksprout
368	17977606	260	0.000093	uk.org.ico
369	17977242	469	0.000052	gov.whitehouse
370	17974632	1014	0.000031	com.speakerdeck
371	17972780	240	0.000102	com.rawgit
372	17964500	659	0.000042	com.intel
373	17953938	899	0.000034	com.wikia
374	17951944	83	0.000482	com.googleadservices
375	17951496	506	0.000049	com.box
376	17945642	1418	0.000022	com.huffpost
377	17944878	1086	0.000029	net.leadpages
378	17943278	508	0.000049	com.cbsnews
379	17943212	323	0.000075	com.time
380	17942606	1976	0.000015	com.zynga
381	17940634	242	0.000101	com.getbootstrap
382	17940514	811	0.000036	com.superpages
383	17940204	1583	0.000019	com.impactbnd
384	17940144	177	0.000144	jp.co.yahoo
385	17937886	67	0.000548	net.jsfiddle
386	17936062	1271	0.000024	com.smallbiztrends
387	17936026	1511	0.000020	org.gnupg
388	17934886	1461	0.000021	co.leadpages
389	17934500	314	0.000076	com.staticflickr
390	17933798	1225	0.000025	com.googlegroups
391	17933250	1488	0.000021	com.thumbtack
392	17932302	367	0.000066	com.ft
393	17931464	2051	0.000015	com.ewtn
394	17929878	370	0.000066	com.office
395	17928686	1576	0.000019	com.kaggle
396	17926934	137	0.000200	com.wixsite
397	17922076	1996	0.000015	org.spie
398	17919510	1641	0.000019	com.thecut
399	17918918	1261	0.000025	com.ebayimg
400	17917840	1806	0.000017	com.googledrive
401	17917748	391	0.000063	com.aol
402	17917512	1573	0.000019	org.jenkins-ci
403	17915334	434	0.000056	com.fortune
404	17911152	2275	0.000014	net.organicfacts
405	17910680	357	0.000068	com.unsplash
406	17910196	2118	0.000015	it.polito
407	17902706	1753	0.000018	com.mindbodygreen
408	17899942	599	0.000043	com.proofpoint
409	17899484	1161	0.000027	edu.ucsd
410	17898182	2239	0.000014	net.furaffinity
411	17895926	766	0.000037	com.engadget
412	17895860	131	0.000218	com.weibo
413	17895616	229	0.000104	com.surveymonkey
414	17895124	1556	0.000020	com.crashlytics
415	17891646	1522	0.000020	com.toptal
416	17891064	298	0.000079	com.skype
417	17890434	1310	0.000024	com.avvo
418	17889754	2012	0.000015	com.doctoroz
419	17889278	1351	0.000023	io.fabric
420	17888404	1652	0.000019	com.thoughtworks
421	17888344	119	0.000253	com.jimdo
422	17884296	339	0.000071	com.w3schools
423	17882826	381	0.000064	org.un
424	17882360	1912	0.000016	com.mysanantonio
425	17880012	1426	0.000022	com.carto
426	17877832	1478	0.000021	com.grammarly
427	17876928	933	0.000034	com.pexels
428	17875774	1485	0.000021	org.sqlite
429	17875586	132	0.000214	com.youtube-nocookie
430	17873060	986	0.000032	com.gizmodo
431	17868426	1747	0.000018	gov.arts
432	17868138	992	0.000032	edu.upenn
433	17866058	1074	0.000030	org.vim
434	17864842	1812	0.000017	com.instapaper
435	17862354	690	0.000040	com.vice
436	17861898	674	0.000041	gov.nist
437	17857890	70	0.000536	org.reactjs
438	17857428	1877	0.000016	gov.la
439	17857390	1712	0.000018	com.politifact
440	17856846	826	0.000036	com.blackberry
441	17856382	1584	0.000019	com.ogilvy
442	17856122	719	0.000039	com.msdn
443	17855302	2249	0.000014	edu.utep
444	17855124	1578	0.000019	com.citysearch
445	17854150	893	0.000034	edu.umich
446	17852538	223	0.000106	net.behance
447	17850120	2156	0.000014	com.dynamics
448	17849038	390	0.000063	com.booking
449	17846000	2273	0.000014	com.asmallorange
450	17843806	1311	0.000024	com.curbed
451	17842712	478	0.000051	com.herokuapp
452	17842298	216	0.000111	com.automattic
453	17842098	1541	0.000020	org.aiga
454	17842080	923	0.000034	org.worldbank
455	17841888	147	0.000176	com.aspnetcdn
456	17841278	1850	0.000017	com.deepmind
457	17841020	1228	0.000025	com.sprinklr
458	17840682	1058	0.000030	com.thinkwithgoogle
459	17839270	2360	0.000013	it.clyp
460	17838612	1540	0.000020	com.instapage
461	17837952	272	0.000088	com.digg
462	17837540	1614	0.000019	com.cmswire
463	17836636	447	0.000055	com.goodreads
464	17835202	2033	0.000015	au.com.huffingtonpost
465	17834194	681	0.000041	com.symantec
466	17832794	385	0.000064	com.dailymotion
467	17832500	1606	0.000019	com.vendio
468	17832000	2204	0.000014	net.openreview
469	17831708	865	0.000035	net.openid
470	17830532	2206	0.000014	com.kvue
471	17830032	171	0.000155	com.feedburner
472	17829348	1479	0.000021	gov.wi
473	17826692	2056	0.000015	com.kudzu
474	17826526	2047	0.000015	com.stamen
475	17826378	1275	0.000024	com.merriam-webster
476	17824996	1510	0.000020	com.csoonline
477	17819228	1901	0.000016	it.binged
478	17818128	1394	0.000022	com.coschedule
479	17817378	2187	0.000014	com.writersdigest
480	17817048	699	0.000040	org.bitbucket
481	17815720	876	0.000035	edu.columbia
482	17815700	1897	0.000016	google.ai
483	17814606	1244	0.000025	com.auth0
484	17814108	1112	0.000029	edu.utexas
485	17813510	1103	0.000029	org.weforum
486	17811808	1757	0.000018	com.merchantcircle
487	17811776	2732	0.000013	com.bitballoon
488	17811618	2121	0.000015	edu.dukeupress
489	17810846	2018	0.000015	com.ingress
490	17808694	148	0.000175	com.tripadvisor
491	17808548	1858	0.000016	com.king5
492	17808180	307	0.000077	com.wiley
493	17803782	1958	0.000016	com.nngroup
494	17803738	1457	0.000021	com.vanityfair
495	17801028	337	0.000072	com.hp
496	17797994	125	0.000236	jp.co.google
497	17797558	320	0.000075	com.scribd
498	17795684	336	0.000072	com.tripod
499	17794742	701	0.000040	io.codepen
500	17794630	2162	0.000014	io.prototypr
501	17794528	446	0.000055	com.aliyuncs
502	17794052	972	0.000033	uk.co.guardian
503	17793662	566	0.000045	com.samsung
504	17793286	451	0.000055	com.slack
505	17793034	685	0.000041	org.eff
506	17791936	547	0.000046	com.webs
507	17789994	474	0.000052	com.atlassian
508	17789800	198	0.000119	de.amazon
509	17789764	2815	0.000012	edu.alamo
510	17788872	1520	0.000020	com.jeffbullas
511	17785572	1844	0.000017	ca.ubc
512	17783666	452	0.000054	com.newrelic
513	17778998	1776	0.000017	com.financialexpress
514	17777484	1051	0.000030	com.yellowpages
515	17777164	1647	0.000019	org.owasp
516	17776950	1209	0.000026	org.whatbrowser
517	17772704	1422	0.000022	org.tigris
518	17771794	1723	0.000018	com.thermofisher
519	17771104	429	0.000057	com.businesswire
520	17769374	1664	0.000019	org.wikidata
521	17769220	205	0.000113	com.bandcamp
522	17768404	195	0.000122	com.constantcontact
523	17767070	1444	0.000021	com.pcworld
524	17766282	861	0.000035	com.dropboxusercontent
525	17763526	1233	0.000025	edu.purdue
526	17762444	297	0.000080	com.wufoo
527	17762030	734	0.000038	com.createjs
528	17761810	396	0.000063	com.force
529	17759846	565	0.000045	in.co.google
530	17759466	364	0.000067	org.doi
531	17757706	2193	0.000014	com.hotfrog
532	17757234	863	0.000035	com.foxnews
533	17756726	1402	0.000022	org.letsencrypt
534	17755956	200	0.000117	org.icann
535	17755908	418	0.000060	com.inc
536	17755824	1528	0.000020	com.invisionapp
537	17755374	2322	0.000013	com.yellowbook
538	17755084	295	0.000081	gov.cdc
539	17752452	1135	0.000028	org.altervista
540	17749954	2167	0.000014	com.khou
541	17749580	2106	0.000015	com.quickanddirtytips
542	17749254	1416	0.000022	org.sonatype
543	17749176	2422	0.000013	es.iac
544	17749162	170	0.000156	ru.mail
545	17748102	1281	0.000024	com.storify
546	17745564	1185	0.000026	us.imageshack
547	17745434	2359	0.000013	org.hg
548	17743856	696	0.000040	com.psychologytoday
549	17743458	1251	0.000025	com.upwork
550	17743324	1052	0.000030	com.ycombinator
551	17742228	1646	0.000019	com.kinsta
552	17742204	1027	0.000031	com.hootsuite
553	17741772	1204	0.000026	ca.blogspot
554	17741450	2284	0.000014	com.theminimalists
555	17738328	1254	0.000025	com.ifttt
556	17732728	299	0.000079	com.prnewswire
557	17732640	2086	0.000015	jp.riken
558	17730736	1938	0.000016	at.tugraz
559	17730652	841	0.000035	com.docker
560	17730110	1337	0.000023	in.blogspot
561	17728102	2132	0.000014	com.theoutline
562	17727422	1172	0.000027	com.indiegogo
563	17724128	954	0.000033	com.alexa
564	17723922	3550	0.000012	com.twitpic
565	17723324	576	0.000044	com.windowsphone
566	17723084	1173	0.000027	com.homeadvisor
567	17722600	1694	0.000018	uk.co.metro
568	17720038	2745	0.000013	com.idt
569	17719404	2456	0.000013	com.23hq
570	17717984	1471	0.000021	org.khanacademy
571	17716196	1821	0.000017	org.elasticsearch
572	17715978	767	0.000037	com.indiatimes
573	17715462	1989	0.000015	com.shoutmeloud
574	17714586	480	0.000051	com.nature
575	17713698	414	0.000060	edu.cmu
576	17713112	1856	0.000017	com.city-data
577	17712356	2116	0.000015	com.kgw
578	17711724	1238	0.000025	org.pewresearch
579	17711380	975	0.000033	com.sfgate
580	17711188	1672	0.000018	gov.nh
581	17711070	1983	0.000015	google.design
582	17708460	457	0.000053	com.gitlab
583	17708384	507	0.000049	uk.co.independent
584	17708180	1678	0.000018	org.polymer-project
585	17708112	2129	0.000014	org.designmuseum
586	17708080	219	0.000108	jp.ne.hatena
587	17707174	224	0.000106	to.amzn
588	17704286	1143	0.000027	edu.wisc
589	17703708	527	0.000047	com.statista
590	17702676	802	0.000036	com.netflix
591	17702444	1208	0.000026	com.firefox
592	17701640	2944	0.000012	edu.brown
593	17700316	1777	0.000017	com.tutsplus
594	17699332	2197	0.000014	ca.uwaterloo
595	17697030	2189	0.000014	com.company
596	17696512	1609	0.000019	com.martechtoday
597	17696414	560	0.000045	org.pbs
598	17696178	1632	0.000019	com.fiverr
599	17693936	2353	0.000013	com.instructables
600	17693836	1226	0.000025	com.clicky
601	17693516	244	0.000101	com.wpengine
602	17693384	1009	0.000031	com.uservoice
603	17690248	3133	0.000012	net.digitalcongo
604	17688104	496	0.000049	us.icio
605	17688098	1724	0.000018	us.nm.state
606	17685732	2920	0.000012	com.wvec
607	17685630	2831	0.000012	com.growtix
608	17684812	1786	0.000017	us.ma.state
609	17684532	1116	0.000028	uk.ac.cam
610	17684330	2372	0.000013	com.warriorplus
611	17684134	957	0.000033	com.shutterstock
612	17683636	1314	0.000024	uk.co.theregister
613	17682188	1007	0.000032	es.agpd
614	17682158	2168	0.000014	com.what3words
615	17680382	1876	0.000016	com.itsnicethat
616	17679870	326	0.000073	org.joomla
617	17676374	2153	0.000014	com.dreamgrow
618	17675740	1212	0.000026	com.playstation
619	17674912	2325	0.000013	org.webpagetest
620	17674664	1815	0.000017	io.pantheon
621	17673952	3026	0.000012	org.nalip
622	17673070	1382	0.000022	com.digitaltrends
623	17672802	2256	0.000014	com.googlelabs
624	17672798	106	0.000299	net.2mdn
625	17671734	509	0.000049	tv.twitch
626	17671174	1168	0.000027	com.steamcommunity
627	17670820	2061	0.000015	com.targetmarketingmag
628	17670692	178	0.000144	me.line
629	17670586	2801	0.000013	co.edureka
630	17670360	2230	0.000014	eu.i-scoop
631	17670228	1929	0.000016	com.wral
632	17669872	1974	0.000015	us.wi.state
633	17668578	2229	0.000014	net.wrightflyer
634	17666822	2355	0.000013	gov.cabq
635	17666534	340	0.000071	com.bitly
636	17666208	368	0.000066	cn.com.sina
637	17665830	1376	0.000022	com.intuit
638	17665486	1339	0.000023	kr.or.kisa
639	17665464	1043	0.000030	com.newsweek
640	17665278	1152	0.000027	edu.northwestern
641	17664282	2384	0.000013	edu.uah
642	17663658	1816	0.000017	com.rabbitmq
643	17662888	2067	0.000015	com.wfaa
644	17662822	1312	0.000024	com.ning
645	17662498	1923	0.000016	ch.ethz
646	17661652	1622	0.000019	com.sharefile
647	17661252	1259	0.000025	com.pcmag
648	17660468	407	0.000061	edu.nyu
649	17659788	1029	0.000031	gov.fcc
650	17658992	348	0.000070	org.opensource
651	17658162	74	0.000531	me.ogp
652	17658048	2400	0.000013	com.wikidot
653	17657344	1610	0.000019	com.com
654	17657246	187	0.000133	com.eepurl
655	17657014	1227	0.000025	com.ssrn
656	17656988	694	0.000040	com.xinhuanet
657	17654064	1780	0.000017	org.scala-lang
658	17653188	1408	0.000022	edu.unc
659	17652568	1894	0.000016	org.iihs
660	17652104	771	0.000037	org.plos
661	17651732	1274	0.000024	tv.ustream
662	17651382	1068	0.000030	ly.ow
663	17650966	2194	0.000014	com.almanac
664	17650526	2055	0.000015	com.gamespot
665	17650220	2335	0.000013	com.bibliocommons
666	17649444	660	0.000042	com.feedly
667	17648694	797	0.000036	com.deloitte
668	17646576	959	0.000033	gov.senate
669	17646290	2218	0.000014	org.onegreenplanet
670	17645346	2125	0.000014	com.yourdomain
671	17645168	433	0.000057	com.squareup
672	17644982	1886	0.000016	com.mariadb
673	17643414	1548	0.000020	org.postimg
674	17642978	1291	0.000024	org.cambridge
675	17642506	2381	0.000013	com.marksdailyapple
676	17642072	261	0.000091	com.histats
677	17641504	1666	0.000019	com.digitaloceanspaces
678	17641474	1267	0.000024	com.canva
679	17641424	1392	0.000022	im.gitter
680	17641326	1198	0.000026	com.techrepublic
681	17640734	2749	0.000013	com.themonitor
682	17640688	1577	0.000019	uk.co.thesun
683	17640560	1618	0.000019	com.nba
684	17639544	2170	0.000014	com.winemag
685	17638276	1431	0.000022	com.mcafee
686	17638268	913	0.000034	gov.justice
687	17635694	722	0.000039	com.steampowered
688	17633614	886	0.000035	com.timeanddate
689	17633566	445	0.000055	com.adweek
690	17631684	834	0.000035	com.aliexpress
691	17630196	302	0.000078	com.netdna-ssl
692	17630110	1764	0.000017	us.oh.state
693	17629376	1241	0.000025	com.optinmonster
694	17628644	1389	0.000022	org.js
695	17628624	2240	0.000014	jp.ac.kobe-u
696	17627756	657	0.000042	gov.noaa
697	17626576	1782	0.000017	org.openweathermap
698	17625866	853	0.000035	com.marketwatch
699	17625214	2371	0.000013	com.winefolly
700	17624654	1585	0.000019	org.golang
701	17623884	343	0.000071	ca.google
702	17623882	1171	0.000027	com.hollywoodreporter
703	17623394	2642	0.000013	org.travelblog
704	17621812	2915	0.000012	me.pxlme
705	17621742	1718	0.000018	com.crunchbase
706	17621104	2417	0.000013	com.thedrinksbusiness
707	17620918	1253	0.000025	com.mlb
708	17620844	2183	0.000014	com.designobserver
709	17619798	1957	0.000016	com.whitepages
710	17618860	1308	0.000024	fr.lemonde
711	17617276	1650	0.000019	com.pastebin
712	17616020	2667	0.000013	com.backyardchickens
713	17615996	378	0.000065	com.themeisle
714	17615324	247	0.000099	io.polyfill
715	17614672	3674	0.000011	org.torproject
716	17614462	1196	0.000026	com.politico
717	17612598	965	0.000033	de.blogspot
718	17612468	2143	0.000014	com.programmableweb
719	17612380	777	0.000037	gov.house
720	17612350	2378	0.000013	uk.ac.hud
721	17612226	313	0.000076	com.fc2
722	17609572	351	0.000069	jp.co.rakuten
723	17609426	1284	0.000024	se.haxx
724	17609170	401	0.000062	com.smugmug
725	17609048	2191	0.000014	com.azfamily
726	17607352	126	0.000236	info.aboutads
727	17607032	5050	0.000007	com.formula1
728	17606320	2948	0.000012	com.locationrebel
729	17604020	252	0.000097	com.marriott
730	17603354	185	0.000134	com.xing
731	17603156	1543	0.000020	org.doxygen
732	17602956	491	0.000050	com.snapchat
733	17601902	2771	0.000013	com.trendland
734	17600640	1073	0.000030	com.americanexpress
735	17600636	1115	0.000028	com.redhat
736	17600606	2394	0.000013	com.sitejabber
737	17600436	2311	0.000014	com.galvanize
738	17600090	4285	0.000009	com.dreamstime
739	17599690	2035	0.000015	com.insiderpages
740	17599126	1419	0.000022	kr.flic
741	17599066	1110	0.000029	gov.uspto
742	17599060	837	0.000035	br.com.uol
743	17596014	530	0.000047	com.163
744	17595876	290	0.000082	gov.ftc
745	17595472	495	0.000049	com.nasdaq
746	17595126	2753	0.000013	com.lookuppage
747	17593550	1134	0.000028	fr.blogspot
748	17592570	1278	0.000024	com.prezi
749	17591712	2659	0.000013	com.avsforum
750	17591328	410	0.000061	mp.mailchi
751	17590608	2020	0.000015	edu.arizona
752	17590230	793	0.000036	com.nielsen
753	17589738	2364	0.000013	com.chamberofcommerce
754	17589414	2147	0.000014	com.towardsdatascience
755	17589090	1050	0.000030	com.sciencedaily
756	17588114	978	0.000033	io.readthedocs
757	17587844	283	0.000083	com.dedecms
758	17587504	1549	0.000020	uk.co.wired
759	17586578	1252	0.000025	com.dell
760	17585810	1435	0.000021	com.billboard
761	17585660	421	0.000059	com.criteo
762	17585524	2283	0.000014	org.zenit
763	17585188	1062	0.000030	org.change
764	17584840	1304	0.000024	edu.academia
765	17583818	588	0.000044	com.newyorker
766	17582200	3591	0.000012	com.sophos
767	17582180	1741	0.000018	de.welt
768	17581488	352	0.000069	net.themeforest
769	17581304	2293	0.000014	org.gwtproject
770	17580662	2788	0.000013	io.setosa
771	17580656	1276	0.000024	st.prom
772	17580614	1433	0.000021	fm.last
773	17580540	1730	0.000018	com.fifa
774	17580530	2687	0.000013	com.storeboard
775	17580282	2169	0.000014	au.com.truelocal
776	17580194	2297	0.000014	com.2findlocal
777	17580070	1093	0.000029	com.visualstudio
778	17579740	1111	0.000029	com.500px
779	17579538	250	0.000097	jp.co.amazon
780	17578580	2264	0.000014	net.webhostingsecretrevealed
781	17574976	1752	0.000018	org.rubyonrails
782	17574924	59	0.000607	com.messenger
783	17574886	1690	0.000018	com.mtv
784	17574662	2277	0.000014	com.newsbank
785	17573782	1095	0.000029	de.heise
786	17573384	1911	0.000016	com.ibtimes
787	17570610	1616	0.000019	com.problogger
788	17570126	1995	0.000015	com.ehow
789	17569784	1998	0.000015	mp.j
790	17568580	1036	0.000031	com.cbslocal
791	17568370	2398	0.000013	com.wcnc
792	17568244	1096	0.000029	com.investopedia
793	17567780	2930	0.000012	edu.unl
794	17567104	2317	0.000014	ly.cl
795	17566344	687	0.000041	com.caniuse
796	17566302	431	0.000057	com.verisign
797	17566120	1455	0.000021	com.hotmail
798	17565914	2181	0.000014	au.com.yellowpages
799	17565600	1453	0.000021	com.rollingstone
800	17565572	2115	0.000015	com.local
801	17564428	231	0.000104	fr.google
802	17563688	215	0.000111	it.google
803	17563288	2003	0.000015	com.smartblogger
804	17562868	1663	0.000019	org.coursera
805	17562472	2220	0.000014	gov.louisvilleky
806	17562082	1202	0.000026	com.domain
807	17560868	597	0.000043	com.nationalgeographic
808	17560104	2123	0.000015	com.theinnovationenterprise
809	17559640	2854	0.000012	ke.co.blogspot
810	17558802	2265	0.000014	io.kubernetes
811	17558702	2319	0.000014	net.brownbook
812	17558172	1859	0.000016	de.zeit
813	17558106	1344	0.000023	com.freepik
814	17557620	2205	0.000014	com.goinswriter
815	17557432	731	0.000039	com.tandfonline
816	17556946	1480	0.000021	edu.jhu
817	17556522	2071	0.000015	com.riddle
818	17556376	1256	0.000025	com.vox
819	17555602	1127	0.000028	com.smashingmagazine
820	17554660	1756	0.000018	edu.msu
821	17554422	838	0.000035	com.uk
822	17554304	2958	0.000012	org.dyndns
823	17553914	2403	0.000013	com.wsoctv
824	17553776	2406	0.000013	com.independent
825	17553776	1387	0.000022	com.nymag
826	17552988	1809	0.000017	com.posterous
827	17550834	1189	0.000026	com.digitalocean
828	17550516	883	0.000035	com.gofundme
829	17549804	255	0.000095	com.myshopify
830	17549356	2746	0.000013	com.spoke
831	17549122	2064	0.000015	com.chambermaster
832	17548302	1179	0.000027	de.spiegel
833	17548188	1784	0.000017	com.ikea
834	17548154	2263	0.000014	com.bizcommunity
835	17548094	2730	0.000013	com.communitywalk
836	17547516	2399	0.000013	com.ibmbigdatahub
837	17547486	1906	0.000016	com.thewritepractice
838	17546846	1599	0.000019	org.filezilla-project
839	17546810	1899	0.000016	com.techradar
840	17546678	1963	0.000015	com.visioncritical
841	17546154	1969	0.000015	com.brafton
842	17545852	1627	0.000019	com.codeplex
843	17545338	428	0.000057	com.sohu
844	17544316	335	0.000072	com.jotform
845	17543714	1779	0.000017	com.lawyers
846	17543442	2316	0.000014	edu.hbs
847	17543058	1401	0.000022	edu.usc
848	17542964	152	0.000169	com.addtoany
849	17542808	2705	0.000013	com.nation2
850	17542602	1317	0.000023	edu.uchicago
851	17542376	2182	0.000014	com.w3techs
852	17541496	1371	0.000023	sh.brew
853	17541272	1325	0.000023	com.strikingly
854	17540262	1988	0.000015	org.aclu
855	17540236	2579	0.000013	com.kens5
856	17539906	453	0.000054	jp.ne.sakura
857	17539894	1092	0.000029	com.prweb
858	17539810	2428	0.000013	com.tractorsupply
859	17539382	3540	0.000012	com.gyazo
860	17539240	2717	0.000013	com.yelloyello
861	17538820	1231	0.000025	com.elpais
862	17538662	3864	0.000010	com.rottentomatoes
863	17538296	2138	0.000014	net.hockeyapp
864	17537912	1697	0.000018	com.howstuffworks
865	17536686	2805	0.000012	com.lacartes
866	17536288	1628	0.000019	io.getmdl
867	17535392	2343	0.000013	com.citysquares
868	17534222	761	0.000037	net.daum
869	17533460	2407	0.000013	com.kmov
870	17530710	2816	0.000012	com.mothering
871	17530426	484	0.000051	com.iconfinder
872	17529686	2873	0.000012	org.rethinkingschools
873	17528810	1456	0.000021	org.wiktionary
874	17528532	707	0.000040	com.emarketer
875	17528512	259	0.000094	me.t
876	17528368	2885	0.000012	com.asus
877	17527494	1904	0.000016	com.rt
878	17527482	993	0.000032	com.oup
879	17527248	1383	0.000022	com.theglobeandmail
880	17524712	1270	0.000024	co.vine
881	17524420	2768	0.000013	org.foodrevolution
882	17524032	2365	0.000013	com.wpxi
883	17523232	973	0.000033	com.airbnb
884	17522954	970	0.000033	gov.usa
885	17522872	2702	0.000013	com.njmonthly
886	17522602	1011	0.000031	org.unesco
887	17522048	2800	0.000013	org.thebestschools
888	17521268	2309	0.000014	com.ezlocal
889	17521124	173	0.000153	com.bluehost
890	17521040	228	0.000105	com.maxcdn
891	17520736	2401	0.000013	com.cbs
892	17519970	1405	0.000022	org.example
893	17519746	2806	0.000012	com.calmclinic
894	17519654	964	0.000033	gov.copyright
895	17519130	2134	0.000014	edu.ncsu
896	17517626	3900	0.000010	com.domaintools
897	17517264	2799	0.000013	com.trepup
898	17517190	1849	0.000017	edu.indiana
899	17516236	1410	0.000022	org.unicode
900	17514928	2734	0.000013	com.mykaratestore
901	17514770	1860	0.000016	com.adespresso
902	17514742	487	0.000050	org.whatwg
903	17514450	3782	0.000011	gd.is
904	17513384	1837	0.000017	re.cli
905	17513134	3162	0.000012	com.000webhostapp
906	17512942	907	0.000034	com.alibaba
907	17512916	1624	0.000019	com.britannica
908	17512736	1210	0.000026	com.reverbnation
909	17512256	609	0.000043	com.patreon
910	17511530	4022	0.000010	edu.iu
911	17511124	794	0.000036	com.yandex
912	17511074	525	0.000047	com.outlook
913	17510210	1151	0.000027	org.fao
914	17509976	2756	0.000013	co.wanelo
915	17508946	1693	0.000018	com.udemy
916	17508814	1188	0.000026	gov.usgs
917	17508588	941	0.000034	com.ggpht
918	17508506	1309	0.000024	uk.co.mirror
919	17508422	1156	0.000027	edu.umn
920	17507730	318	0.000075	nl.google
921	17505054	258	0.000094	com.disqus
922	17504754	1313	0.000024	com.pwc
923	17504638	961	0.000033	com.pinimg
924	17504458	1800	0.000017	com.html5rocks
925	17504148	1124	0.000028	com.sun
926	17503468	1200	0.000026	com.uber
927	17501792	4643	0.000008	com.mysite
928	17501756	2308	0.000014	org.gimp
929	17501722	2066	0.000015	com.packtpub
930	17501690	2676	0.000013	com.pages10
931	17501468	2421	0.000013	com.tuck
932	17500992	2615	0.000013	org.swi-prolog
933	17500218	1835	0.000017	edu.virginia
934	17499886	2897	0.000012	be.brussels
935	17499784	1106	0.000029	au.net.abc
936	17499678	226	0.000105	com.googletagservices
937	17496996	3531	0.000012	ch.cern
938	17496282	2580	0.000013	com.ktvb
939	17496102	501	0.000049	com.bigcartel
940	17495798	1930	0.000016	com.nfl
941	17495794	1300	0.000024	com.showmelocal
942	17494518	1358	0.000023	org.pnas
943	17494014	2175	0.000014	uk.co.realbusiness
944	17494010	2728	0.000013	ly.visual
945	17492442	2109	0.000015	com.discovery
946	17492384	2622	0.000013	org.virginiadot
947	17491978	1069	0.000030	com.us
948	17491852	1829	0.000017	edu.cuny
949	17491298	1681	0.000018	com.podbean
950	17491172	1182	0.000026	com.accenture
951	17491140	2755	0.000013	com.pushwoosh
952	17490876	2588	0.000013	com.yellowbot
953	17490652	2903	0.000012	com.watchuseek
954	17490510	1451	0.000021	com.thehill
955	17490442	2834	0.000012	com.callupcontact
956	17490156	2397	0.000013	com.echelman
957	17490000	3853	0.000010	org.greenpeace
958	17489174	1783	0.000017	com.screencast
959	17488476	2043	0.000015	com.webnode
960	17488356	1119	0.000028	com.lifehacker
961	17487060	991	0.000032	org.iso
962	17487048	728	0.000039	com.gartner
963	17485804	1680	0.000018	com.hulu
964	17485784	1927	0.000016	co.gcdn
965	17484600	1525	0.000020	com.windows
966	17483866	2001	0.000015	com.birdeye
967	17483034	951	0.000034	ru.google
968	17482970	3909	0.000010	org.bitcoin
969	17481326	2413	0.000013	com.topsy
970	17481036	2429	0.000013	com.texasbar
971	17480454	882	0.000035	com.stitcher
972	17478940	2924	0.000012	com.talkbass
973	17478584	2278	0.000014	ca.ualberta
974	17478504	843	0.000035	gg.discord
975	17478450	2855	0.000012	com.cylex-usa
976	17477514	3673	0.000011	nl.xs4all
977	17477396	2925	0.000012	info.ufacity
978	17477382	104	0.000304	com.namecheap
979	17477122	2849	0.000012	com.louisville
980	17476316	2460	0.000013	uk.gov.westsussex
981	17475686	2668	0.000013	com.salespider
982	17475490	1568	0.000019	com.nokia
983	17475476	1034	0.000031	com.digiday
984	17475262	2889	0.000012	org.stnicholascenter
985	17475056	2850	0.000012	au.com.hotfrog
986	17474798	1257	0.000025	org.webkit
987	17474492	2430	0.000013	net.blog5
988	17474236	2271	0.000014	tv.periscope
989	17473902	877	0.000035	uk.co.tripadvisor
990	17473802	2900	0.000012	org.phys
991	17473188	1612	0.000019	edu.umd
992	17473104	884	0.000035	gov.ny
993	17472122	1981	0.000015	ru.narod
994	17471648	233	0.000104	jp.ameblo
995	17471606	5178	0.000007	net.minecraft
996	17470784	163	0.000162	com.youku
997	17470482	1677	0.000018	org.gnome
998	17470166	184	0.000137	com.nginx
999	17470156	1447	0.000021	com.splashthat
1000	17469878	2681	0.000013	com.bleacherreport

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

April 2019 crawl archive now available

April 30, 2019Sebastian Nagel

The crawl archive for April 2019 is now available! It contains 2.5 billion web pages or 198 TiB of uncompressed content, crawled between April 18th and 26th.

The April crawl contains page captures of 750 million URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks (harmonic centrality) published as part of the Nov/Dec/Jan 2018/2019 webgraph data set from the following sources:

sitemaps, RSS and Atom feeds
a breadth-first side crawl within a maximum of 3 links (“hops”) away from the homepages of the top 60 million hosts and domains and a random sample of 1 million human-readable sitemap pages (HTML format)
a random sample of 1 billion outlinks taken from WAT files of the March crawl

The following minor changes to the crawler configuration have been made:

the crawler now sends again an Accept-Language HTTP header, requesting English content
the configuration has been tweaked to include less non-HTML content

Archive Location and Download

The April crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-18/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-18/segment.paths.gz	100
WARC files	CC-MAIN-2019-18/warc.paths.gz	56000	44.86
WAT files	CC-MAIN-2019-18/wat.paths.gz	56000	16.32
WET files	CC-MAIN-2019-18/wet.paths.gz	56000	6.96
Robots.txt files	CC-MAIN-2019-18/robotstxt.paths.gz	56000	0.16
Non-200 responses files	CC-MAIN-2019-18/non200responses.paths.gz	56000	1.67
URL index files	CC-MAIN-2019-18/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-18/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

March 2019 crawl archive now available

April 1, 2019Sebastian Nagel

The crawl archive for March 2019 is now available! It contains 2.55 billion web pages or 210 TiB of uncompressed content, crawled between March 18th and 27th.

The March crawl contains page captures of 660 million URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks (harmonic centrality) published as part of the Nov/Dec/Jan 2018/2019 webgraph data set from the following sources:

sitemaps, RSS and Atom feeds
a breadth-first side crawl within a maximum of 6 links (“hops”) away from the homepages of the top 60 million hosts and domains
a random sample of outlinks taken from WAT files of the February crawl

Archive Location and Download

The March crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-13/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-13/segment.paths.gz	100
WARC files	CC-MAIN-2019-13/warc.paths.gz	56000	49.09
WAT files	CC-MAIN-2019-13/wat.paths.gz	56000	17.37
WET files	CC-MAIN-2019-13/wet.paths.gz	56000	7.47
Robots.txt files	CC-MAIN-2019-13/robotstxt.paths.gz	56000	0.17
Non-200 responses files	CC-MAIN-2019-13/non200responses.paths.gz	56000	1.63
URL index files	CC-MAIN-2019-13/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-13/. Also the columnar index has been updated to contain this crawl.

February 2019 crawl archive now available

March 1, 2019Sebastian Nagel

The crawl archive for February 2019 is now available! It contains 2.9 billion web pages or 225 TiB of uncompressed content, crawled between February 15th and 24th.

The February crawl contains page captures of 750 million URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks (harmonic centrality) published as part of the Nov/Dec/Jan 2018/2019 webgraph data set from the following sources:

sitemaps, RSS and Atom feeds
a breadth-first side crawl within a maximum of 5 links (“hops”) away from the homepages of the top 50 million hosts and domains
a random sample of outlinks taken from WAT files of the January crawl

The number of sampled URLs per domain depends on the domain’s harmonic centrality rank in the webgraph data set – higher ranking domain are allowed to “contribute” more URLs.

The way our crawler handles politeness limits per host and/or pay-level domain has been improved:
First, limits are now configurable and are based on the harmonic centrality rank of a domain.
Second, we now also put a limit on the number of hosts/subdomains per domain. This limit is also based on the domain rank and ranges from 500,000 subdomains for top-ranking domains (think of blogspot.com) to less than 100 for low-ranking domains. While the the number of hosts covered in the February crawl dropped to 50 millions from 60 millions in January, we see a positive impact on the total amount of pages crawled for large domains. Technically, every host requires a DNS lookup and a robots.txt fetch even if only a single page is fetched from this host and the performance of the crawler improves if resources are focused on few 100,000 subdomains and not spread over millions of hosts. We also hope that a limit on the number of hosts per domain makes the crawler more robust against link spam. The set of sampled subdomains for large domains will vary from month to month to guarantee a good overall coverage if multiple monthly crawls are combined.

Archive Location and Download

The February crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-09/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-09/segment.paths.gz	100
WARC files	CC-MAIN-2019-09/warc.paths.gz	64000	59.86
WAT files	CC-MAIN-2019-09/wat.paths.gz	64000	18.23
WET files	CC-MAIN-2019-09/wet.paths.gz	64000	7.62
Robots.txt files	CC-MAIN-2019-09/robotstxt.paths.gz	64000	0.17
Non-200 responses files	CC-MAIN-2019-09/non200responses.paths.gz	64000	1.79
URL index files	CC-MAIN-2019-09/cc-index.paths.gz	302	0.21

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-09/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs Nov/Dec/Jan 2018 – 2019

February 20, 2019Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of November, December 2018 and January 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

Host-level graph

The graph consists of 407 million nodes and 4.2 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 323 million dangling nodes (79%) and the largest strongly connected component contains 63 million (15%) nodes.

You can download the graph and the ranks of all 407 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2018-19-nov-dec-jan/host/. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2018-19-nov-dec-jan/host/ as prefix to access the files from everywhere.

Download files of the Common Crawl Nov/Dec/Jan 2018-19 host-level webgraph

Size	File	Description
2.90 GB	cc-main-2018-19-nov-dec-jan-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 42 vertices files
18.84 GB	cc-main-2018-19-nov-dec-jan-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 84 edges files

7.81 GB	cc-main-2018-19-nov-dec-jan-host.graph	graph in BVGraph format
2 kB	cc-main-2018-19-nov-dec-jan-host.properties

8.16 GB	cc-main-2018-19-nov-dec-jan-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2018-19-nov-dec-jan-host-t.properties

1 kB	cc-main-2018-19-nov-dec-jan-host.stats	WebGraph statistics

7.50 GB	cc-main-2018-19-nov-dec-jan-host-ranks.txt.gz	harmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 90 million nodes and 1.69 billion edges. 53% or 48 million nodes are dangling nodes, the largest strongly connected component covers 37 million or 41% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2018-19-nov-dec-jan/domain/ resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2018-19-nov-dec-jan/domain/.

Download files of the Common Crawl Nov/Dec/Jan 2018-19 domain-level webgraph

Size	File	Description
0.62 GB	cc-main-2018-19-nov-dec-jan-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
6.76 GB	cc-main-2018-19-nov-dec-jan-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.68 GB	cc-main-2018-19-nov-dec-jan-domain.graph	graph in BVGraph format
2 kB	cc-main-2018-19-nov-dec-jan-domain.properties

3.82 GB	cc-main-2018-19-nov-dec-jan-domain-t.graph	transpose of the graph
2 kB	cc-main-2018-19-nov-dec-jan-domain-t.properties

1 kB	cc-main-2018-19-nov-dec-jan-domain.stats	WebGraph statistics

1.96 GB	cc-main-2018-19-nov-dec-jan-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 90 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Nov/Dec/Jan 2018-2019)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	27203288	2	0.012818	com.facebook
2	27081816	1	0.017236	com.googleapis
3	25533108	3	0.010690	com.google
4	24267906	4	0.007625	com.twitter
5	24001384	5	0.006755	com.youtube
6	23187226	6	0.006532	org.w
7	21605786	8	0.003925	com.instagram
8	21386658	7	0.004753	org.gmpg
9	20954100	11	0.003053	com.linkedin
10	20252174	12	0.002871	org.wordpress
11	20166276	15	0.002217	com.wordpress
12	20071538	24	0.001532	com.gravatar
13	20054574	22	0.001673	com.pinterest
14	20035420	27	0.001366	org.wikipedia
15	19689680	21	0.001831	com.apple
16	19669598	13	0.002776	com.bootstrapcdn
17	19590352	36	0.000986	com.blogspot
18	19579602	28	0.001308	com.vimeo
19	19357866	41	0.000827	be.youtu
20	19345240	14	0.002221	com.cloudflare
21	19288382	37	0.000940	gl.goo
22	19267938	29	0.001236	com.microsoft
23	19181584	25	0.001444	com.adobe
24	19148714	42	0.000817	com.amazon
25	19143492	17	0.002000	com.googletagmanager
26	19087530	49	0.000656	com.tumblr
27	19040054	23	0.001572	com.macromedia
28	19024404	51	0.000647	com.wp
29	19009908	16	0.002181	com.flickr
30	18980982	71	0.000509	ly.bit
31	18864572	74	0.000479	com.yahoo
32	18847172	39	0.000879	com.amazonaws
33	18818456	38	0.000888	com.paypal
34	18798784	20	0.001840	com.github
35	18762366	65	0.000584	org.mozilla
36	18761330	26	0.001413	com.gstatic
37	18756286	64	0.000596	me.wp
38	18648308	97	0.000312	com.googleusercontent
39	18645512	40	0.000867	net.cloudfront
40	18636726	85	0.000364	com.soundcloud
41	18628624	109	0.000267	com.nytimes
42	18621690	81	0.000425	com.weebly
43	18599952	54	0.000633	net.doubleclick
44	18588240	44	0.000760	org.w3
45	18584910	87	0.000346	co.t
46	18575634	101	0.000303	com.reddit
47	18568330	68	0.000521	com.medium
48	18560552	150	0.000157	org.wikimedia
49	18552104	111	0.000257	com.dropbox
50	18520572	83	0.000402	org.creativecommons
51	18509376	134	0.000192	org.archive
52	18508730	33	0.001034	io.github
53	18458414	77	0.000443	com.bing
54	18443042	146	0.000176	net.slideshare
55	18442342	124	0.000215	com.imgur
56	18429066	31	0.001176	ru.yandex
57	18423958	82	0.000417	de.google
58	18405610	30	0.001179	net.fbcdn
59	18398944	158	0.000150	edu.stanford
60	18379294	241	0.000099	com.bbc
61	18375750	215	0.000111	com.tinyurl
62	18368156	34	0.001028	org.apache
63	18362114	94	0.000316	com.mailchimp
64	18338296	183	0.000127	com.wired
65	18322334	136	0.000190	com.blogger
66	18280996	63	0.000599	eu.europa
67	18277804	130	0.000200	com.issuu
68	18271824	219	0.000109	com.bloomberg
69	18257422	182	0.000127	com.myspace
70	18254210	80	0.000425	com.jquery
71	18249798	78	0.000433	com.baidu
72	18230364	347	0.000069	com.appspot
73	18220768	137	0.000188	com.eventbrite
74	18214820	125	0.000212	com.yelp
75	18209194	138	0.000185	com.spotify
76	18208644	143	0.000180	org.ietf
77	18202076	189	0.000125	com.oracle
78	18200628	172	0.000139	com.android
79	18196004	248	0.000095	org.npr
80	18194938	331	0.000072	com.theverge
81	18188710	32	0.001108	com.squarespace
82	18180064	307	0.000077	com.googleblog
83	18168696	173	0.000139	org.gnu
84	18168282	115	0.000241	com.youtube-nocookie
85	18166400	352	0.000068	com.quora
86	18166372	84	0.000388	com.statcounter
87	18162270	355	0.000068	com.deviantart
88	18147690	314	0.000076	com.buzzfeed
89	18132148	281	0.000083	org.python
90	18130544	284	0.000082	me.about
91	18122852	426	0.000057	com.slate
92	18120874	443	0.000055	org.ieee
93	18109888	357	0.000068	uk.co.independent
94	18104220	117	0.000228	com.whatsapp
95	18094292	279	0.000085	com.w3schools
96	18092538	72	0.000495	org.schema
97	18087066	448	0.000054	edu.upenn
98	18080120	45	0.000737	com.fontawesome
99	18076578	476	0.000051	edu.ucla
100	18072366	424	0.000057	edu.washington
101	18072078	641	0.000045	org.chromium
102	18068624	468	0.000052	uk.ac.ox
103	18067404	386	0.000063	com.newyorker
104	18066728	186	0.000125	net.behance
105	18057700	282	0.000083	com.example
106	18054090	400	0.000061	org.arxiv
107	18052930	104	0.000285	com.ytimg
108	18049852	192	0.000123	com.dribbble
109	18029132	222	0.000109	gov.ca
110	18026716	140	0.000184	com.forbes
111	18025030	374	0.000065	gov.loc
112	18013454	228	0.000103	com.fastcompany
113	18008156	253	0.000092	com.foursquare
114	18007062	380	0.000064	com.about
115	18005498	179	0.000132	com.cnn
116	18005262	157	0.000150	com.theguardian
117	18005254	466	0.000052	com.evernote
118	18002708	379	0.000064	com.git-scm
119	18001892	337	0.000071	au.com.google
120	18001316	490	0.000050	edu.princeton
121	17997576	247	0.000096	com.typeform
122	17995020	469	0.000052	com.withgoogle
123	17991120	648	0.000044	com.storify
124	17986952	525	0.000047	com.stackexchange
125	17985482	652	0.000044	google.blog
126	17982244	9	0.003675	com.godaddy
127	17976782	229	0.000103	com.nbcnews
128	17974972	161	0.000148	uk.co.bbc
129	17973294	332	0.000072	uk.co.blogspot
130	17971188	396	0.000061	com.tandfonline
131	17957502	408	0.000060	com.mysql
132	17946028	632	0.000045	ca.blogspot
133	17943522	479	0.000051	com.libsyn
134	17940278	196	0.000120	es.google
135	17934926	491	0.000050	com.tinypic
136	17933522	482	0.000051	com.ubuntu
137	17932534	748	0.000039	com.nike
138	17931294	402	0.000061	org.bitbucket
139	17930976	276	0.000085	org.doi
140	17929634	336	0.000072	com.getpocket
141	17927596	676	0.000043	com.jetbrains
142	17909710	278	0.000085	com.mozilla
143	17909040	697	0.000041	com.sap
144	17900594	449	0.000054	com.googlecode
145	17899774	73	0.000484	com.list-manage
146	17895240	185	0.000126	com.huffingtonpost
147	17894146	635	0.000045	tv.ustream
148	17893688	86	0.000351	com.paypalobjects
149	17890446	459	0.000053	com.trello
150	17886818	269	0.000086	edu.mit
151	17882064	152	0.000154	net.sourceforge
152	17879878	245	0.000096	com.githubusercontent
153	17877554	498	0.000049	com.chrome
154	17864368	953	0.000033	edu.gatech
155	17859670	447	0.000054	com.docker
156	17858794	719	0.000040	com.ssrn
157	17858574	597	0.000045	co.g
158	17857678	90	0.000329	com.wix
159	17856626	205	0.000116	com.washingtonpost
160	17849554	1082	0.000029	com.diigo
161	17847314	360	0.000067	gov.fda
162	17845128	127	0.000205	org.bbb
163	17840318	961	0.000033	com.flipboard
164	17839092	939	0.000034	it.scoop
165	17838238	819	0.000037	com.nvidia
166	17836702	297	0.000080	com.reuters
167	17836356	296	0.000080	com.mapquest
168	17830996	570	0.000046	com.pingdom
169	17830362	277	0.000085	com.go
170	17827980	199	0.000119	org.debian
171	17822238	198	0.000119	com.wsj
172	17822206	898	0.000035	com.fastcodesign
173	17819482	35	0.001004	com.fb
174	17815978	814	0.000037	site.business
175	17814932	234	0.000101	com.techcrunch
176	17807884	250	0.000094	com.usatoday
177	17807344	148	0.000171	gov.nih
178	17807156	153	0.000154	com.etsy
179	17804386	543	0.000047	org.eclipse
180	17796560	1025	0.000031	com.hbo
181	17791748	46	0.000708	net.akamaihd
182	17790986	200	0.000118	com.live
183	17790210	977	0.000032	ms.1drv
184	17789540	975	0.000033	nl.blogspot
185	17782296	216	0.000111	com.businessinsider
186	17779806	415	0.000059	com.unity3d
187	17772776	471	0.000052	com.cdbaby
188	17769846	710	0.000041	se.haxx
189	17768930	163	0.000142	org.iana
190	17766540	98	0.000311	com.shopify
191	17765610	283	0.000082	com.herokuapp
192	17762614	280	0.000084	edu.harvard
193	17760570	293	0.000080	net.windows
194	17760214	747	0.000039	org.unicode
195	17758178	110	0.000264	com.jimdo
196	17758168	344	0.000070	com.msn
197	17757864	287	0.000081	uk.co.telegraph
198	17756896	209	0.000112	com.typepad
199	17755326	147	0.000174	com.opera
200	17752324	1084	0.000029	com.creativebloq
201	17750792	852	0.000036	edu.rutgers
202	17750190	662	0.000043	gov.wa
203	17750042	944	0.000034	com.history
204	17748756	362	0.000066	gov.nasa
205	17748704	844	0.000037	edu.illinois
206	17743998	509	0.000049	au.gov.nsw
207	17737554	631	0.000045	gov.dot
208	17730560	1024	0.000031	edu.pitt
209	17730390	191	0.000124	com.imdb
210	17727110	95	0.000315	net.jsdelivr
211	17726578	373	0.000065	com.mashable
212	17721654	67	0.000526	com.vk
213	17719938	47	0.000677	net.facebook
214	17719050	195	0.000121	uk.co.amazon
215	17717626	105	0.000279	com.google-analytics
216	17715986	313	0.000076	com.cnet
217	17712616	1192	0.000027	org.wikibooks
218	17711346	238	0.000100	com.ibm
219	17708674	906	0.000035	ca.utoronto
220	17706048	372	0.000065	com.ted
221	17703164	930	0.000034	au.com.blogspot
222	17696634	809	0.000038	com.ecwid
223	17692804	422	0.000058	uk.co.pinterest
224	17688850	954	0.000033	com.theknot
225	17683380	971	0.000033	edu.osu
226	17676892	368	0.000066	com.latimes
227	17675948	231	0.000103	net.php
228	17674866	1023	0.000031	com.dw
229	17673722	972	0.000033	org.hrw
230	17669176	181	0.000128	com.stackoverflow
231	17666396	1063	0.000030	io.itch
232	17663236	262	0.000090	com.npmjs
233	17654604	917	0.000035	us.mn.state
234	17654482	387	0.000063	uk.co.dailymail
235	17654042	306	0.000077	com.time
236	17653078	175	0.000137	com.twimg
237	17651364	214	0.000112	com.surveymonkey
238	17648462	493	0.000050	net.researchgate
239	17639638	1287	0.000024	com.kinja
240	17636060	913	0.000035	gov.defense
241	17634608	423	0.000058	edu.cornell
242	17632872	804	0.000038	com.citrix
243	17631522	18	0.001984	com.wixstatic
244	17627972	1349	0.000023	com.instapaper
245	17627386	456	0.000053	io.readthedocs
246	17622738	903	0.000035	com.vogue
247	17622484	339	0.000071	me.telegram
248	17622274	738	0.000040	org.postgresql
249	17619732	1211	0.000026	com.dezeen
250	17619540	842	0.000037	com.citysearch
251	17617810	440	0.000056	com.ft
252	17616210	688	0.000042	org.kernel
253	17615932	969	0.000033	com.yellowpages
254	17615850	144	0.000179	uk.co.google
255	17615132	275	0.000085	org.acm
256	17611938	160	0.000148	com.zendesk
257	17608414	420	0.000058	com.kickstarter
258	17607082	1060	0.000030	com.strava
259	17606762	419	0.000058	edu.berkeley
260	17606252	1045	0.000030	gov.mo
261	17604126	333	0.000072	com.cnbc
262	17602550	52	0.000636	com.qq
263	17598790	670	0.000043	com.adjust
264	17596982	1026	0.000031	gov.oregon
265	17596684	299	0.000080	com.meetup
266	17594788	1016	0.000031	org.tensorflow
267	17594178	312	0.000077	com.mapbox
268	17592452	159	0.000150	com.salesforce
269	17586524	353	0.000068	com.gmail
270	17577594	1066	0.000030	com.googlesource
271	17574716	1176	0.000027	edu.kit
272	17574656	327	0.000073	com.springer
273	17574172	55	0.000629	net.jsfiddle
274	17571342	848	0.000037	com.wikia
275	17570210	1123	0.000028	gov.ky
276	17570030	685	0.000042	com.matterport
277	17569184	1055	0.000030	com.hackernoon
278	17569072	382	0.000064	com.fortune
279	17568196	397	0.000061	com.photobucket
280	17565868	376	0.000065	com.giphy
281	17561918	349	0.000069	com.nypost
282	17561528	664	0.000043	com.angieslist
283	17558588	1103	0.000029	gov.wi
284	17558450	908	0.000035	com.xrea
285	17558178	187	0.000125	com.ebay
286	17557786	870	0.000036	com.pixabay
287	17555784	1035	0.000031	org.wnyc
288	17554024	627	0.000045	com.economist
289	17553246	285	0.000082	com.hubspot
290	17552904	858	0.000036	edu.columbia
291	17552482	317	0.000076	org.un
292	17551250	394	0.000062	org.hbr
293	17547768	824	0.000037	com.arstechnica
294	17547252	521	0.000048	com.livechatinc
295	17544648	967	0.000033	com.missingkids
296	17542812	135	0.000191	com.feedburner
297	17542646	563	0.000046	com.nationalgeographic
298	17542210	839	0.000037	edu.yale
299	17541890	960	0.000033	org.ohchr
300	17539760	826	0.000037	org.aarp
301	17538968	550	0.000046	com.scribd
302	17536254	1037	0.000031	gov.maryland
303	17535552	987	0.000032	gov.michigan
304	17534878	1170	0.000027	gov.mt
305	17532728	354	0.000068	com.oreilly
306	17528914	116	0.000238	com.addthis
307	17524962	410	0.000060	com.theatlantic
308	17522894	1164	0.000027	org.amnesty
309	17522838	767	0.000039	com.engadget
310	17522742	1048	0.000030	us.pa.state
311	17522588	1446	0.000022	com.jigsy
312	17520734	1275	0.000025	com.healthgrades
313	17520216	679	0.000042	com.intel
314	17517294	404	0.000061	gov.whitehouse
315	17517106	1250	0.000025	com.manta
316	17515170	689	0.000042	com.vice
317	17515068	412	0.000059	com.unsplash
318	17507818	311	0.000077	com.wiley
319	17506494	128	0.000204	com.wixsite
320	17503220	637	0.000045	com.wikihow
321	17499836	1302	0.000024	com.merchantcircle
322	17496442	341	0.000070	com.livejournal
323	17494952	342	0.000070	com.booking
324	17494632	1395	0.000022	io.soup
325	17493230	370	0.000065	com.skype
326	17490618	518	0.000048	com.samsung
327	17490516	655	0.000044	com.zdnet
328	17487772	398	0.000061	com.entrepreneur
329	17485998	300	0.000080	com.staticflickr
330	17485468	343	0.000070	com.prnewswire
331	17484254	1306	0.000024	ca.yelp
332	17484254	1216	0.000026	com.contently
333	17483554	272	0.000085	int.who
334	17483044	828	0.000037	com.qz
335	17477120	359	0.000067	com.office
336	17476598	472	0.000052	com.cisco
337	17476580	1424	0.000022	com.gimletmedia
338	17476460	1540	0.000020	com.designobserver
339	17475042	294	0.000080	com.hp
340	17474806	260	0.000090	gov.cdc
341	17471496	236	0.000101	com.disqus
342	17470994	1376	0.000023	us.wi.state
343	17467786	640	0.000045	com.cbsnews
344	17467412	517	0.000048	com.statista
345	17467326	126	0.000208	com.weibo
346	17466370	729	0.000040	co.elastic
347	17465780	551	0.000046	ca.pinterest
348	17465738	832	0.000037	edu.psu
349	17462258	1212	0.000026	org.tigris
350	17460552	1296	0.000024	com.thoughtworks
351	17454102	407	0.000060	com.inc
352	17452694	492	0.000050	org.mediawiki
353	17450142	340	0.000071	com.dailymotion
354	17449322	389	0.000063	com.aol
355	17448426	976	0.000033	com.gizmodo
356	17447376	1278	0.000025	org.emojipedia
357	17445752	1081	0.000029	net.leadpages
358	17445540	500	0.000049	gov.nist
359	17442880	1459	0.000021	com.zynga
360	17442700	361	0.000067	org.ampproject
361	17442350	1218	0.000026	us.nm.state
362	17442298	1453	0.000021	com.activerain
363	17441802	1046	0.000030	com.bandsintown
364	17439472	484	0.000051	com.nature
365	17439328	520	0.000048	com.venturebeat
366	17438974	572	0.000046	com.box
367	17438964	178	0.000135	com.constantcontact
368	17438634	213	0.000112	to.amzn
369	17433964	970	0.000033	com.thenextweb
370	17433742	1322	0.000024	com.superpages
371	17432018	508	0.000049	com.symantec
372	17424908	552	0.000046	org.nodejs
373	17424538	242	0.000099	org.drupal
374	17423606	180	0.000131	com.tripadvisor
375	17423290	698	0.000041	com.deloitte
376	17422498	1044	0.000030	us.fl.state
377	17422248	251	0.000094	com.digg
378	17419896	991	0.000032	edu.utexas
379	17419420	959	0.000033	com.googlegroups
380	17418564	1093	0.000029	com.pexels
381	17418492	1329	0.000024	ly.snip
382	17418108	322	0.000075	fr.free
383	17417322	308	0.000077	com.sciencedirect
384	17413382	203	0.000117	com.bandcamp
385	17413228	633	0.000045	com.moz
386	17412704	1408	0.000022	com.whitepages
387	17410218	732	0.000040	com.psychologytoday
388	17407554	1480	0.000021	com.digitaltrends
389	17404092	1539	0.000020	edu.scad
390	17399826	1056	0.000030	org.weforum
391	17397762	330	0.000072	com.sxsw
392	17394976	202	0.000117	de.amazon
393	17394802	464	0.000052	com.goodreads
394	17393720	937	0.000034	org.eff
395	17392836	754	0.000039	com.indiatimes
396	17391108	1147	0.000028	com.thinkwithgoogle
397	17385920	1442	0.000022	org.khanacademy
398	17380096	901	0.000035	com.shutterstock
399	17379546	829	0.000037	edu.umich
400	17377974	658	0.000043	com.raywenderlich
401	17376058	375	0.000065	com.businesswire
402	17375914	1352	0.000023	edu.usc
403	17375836	270	0.000086	ca.google
404	17373678	226	0.000104	com.stumbleupon
405	17373002	1371	0.000023	com.mysanantonio
406	17368554	204	0.000116	com.automattic
407	17368054	891	0.000035	au.net.abc
408	17365624	864	0.000036	org.worldbank
409	17364686	1350	0.000023	edu.unc
410	17364370	1113	0.000028	org.example
411	17362738	1375	0.000023	it.eventbrite
412	17361690	1243	0.000025	com.merriam-webster
413	17360026	1550	0.000020	edu.hmc
414	17357560	912	0.000035	uk.co.guardian
415	17356764	871	0.000036	com.netflix
416	17354116	446	0.000055	com.slack
417	17352062	1438	0.000022	me.websta
418	17350742	1261	0.000025	com.kaggle
419	17350270	544	0.000047	org.pbs
420	17347142	506	0.000049	com.webs
421	17341612	1338	0.000023	com.ning
422	17341024	1339	0.000023	com.speakerdeck
423	17338712	1598	0.000020	au.com.yelp
424	17337250	1602	0.000020	org.themoth
425	17336832	1272	0.000025	com.canva
426	17336530	1384	0.000023	com.pcworld
427	17335622	1186	0.000027	com.indiegogo
428	17334616	1247	0.000025	edu.toronto
429	17333104	2537	0.000014	com.instructables
430	17331566	1517	0.000021	com.brandyourself
431	17331038	722	0.000040	org.unesco
432	17330476	1171	0.000027	com.pcmag
433	17330344	956	0.000033	com.marketwatch
434	17329390	945	0.000033	com.foxnews
435	17325772	526	0.000047	tv.twitch
436	17321778	1196	0.000026	org.mozillazine
437	17320920	1552	0.000020	org.owasp
438	17319708	1374	0.000023	com.googleapps
439	17319644	1146	0.000028	co.leadpages
440	17319502	1604	0.000020	com.yellowbook
441	17319460	1391	0.000022	org.coursera
442	17319074	1316	0.000024	edu.academia
443	17318194	318	0.000075	com.tripod
444	17318084	996	0.000032	edu.ucsd
445	17316988	763	0.000039	com.gartner
446	17316564	915	0.000035	com.sfgate
447	17315318	718	0.000040	com.blackberry
448	17314360	1117	0.000028	org.haskell
449	17314258	364	0.000066	it.placehold
450	17311942	1549	0.000020	edu.utep
451	17311156	1224	0.000026	gov.nh
452	17310230	1263	0.000025	edu.northwestern
453	17306310	1190	0.000027	de.spiegel
454	17303936	321	0.000075	com.getclicky
455	17302292	328	0.000073	com.rawgit
456	17301718	371	0.000065	edu.nyu
457	17300434	1087	0.000029	org.maven
458	17299754	348	0.000069	edu.cmu
459	17298408	1065	0.000030	edu.wisc
460	17294964	957	0.000033	com.dropboxusercontent
461	17294946	295	0.000080	com.smugmug
462	17290908	1335	0.000024	com.googledrive
463	17290086	923	0.000034	gov.fcc
464	17289604	534	0.000047	com.outlook
465	17288796	1277	0.000025	edu.uchicago
466	17287666	634	0.000045	com.windowsphone
467	17284804	1327	0.000024	gov.la
468	17284524	1689	0.000019	org.maximumfun
469	17284360	305	0.000078	net.datatables
470	17284320	571	0.000046	com.lifehacker
471	17283846	501	0.000049	in.co.google
472	17283788	580	0.000046	gov.noaa
473	17283492	1644	0.000020	edu.uah
474	17282770	802	0.000038	com.steampowered
475	17279092	1427	0.000022	com.invisionapp
476	17274466	704	0.000041	com.msdn
477	17274380	1088	0.000029	org.vim
478	17274242	169	0.000141	jp.co.yahoo
479	17274032	473	0.000052	com.cargocollective
480	17273984	1142	0.000028	com.ycombinator
481	17272438	235	0.000101	gov.ftc
482	17271196	1364	0.000023	org.iihs
483	17270984	896	0.000035	gov.census
484	17270534	1245	0.000025	com.upwork
485	17270494	2228	0.000017	com.ehow
486	17270410	100	0.000305	org.networkadvertising
487	17267670	696	0.000041	com.webmd
488	17267420	1422	0.000022	edu.purdue
489	17267352	335	0.000072	com.stripe
490	17266616	2427	0.000015	com.techradar
491	17266546	1052	0.000030	org.sciencemag
492	17264826	1127	0.000028	org.altervista
493	17264782	1151	0.000028	io.material
494	17263612	1482	0.000021	com.fifa
495	17262826	2161	0.000018	com.crunchbase
496	17262428	1451	0.000021	com.technologyreview
497	17261712	822	0.000037	gov.senate
498	17260998	993	0.000032	ly.ow
499	17260098	1169	0.000027	com.playstation
500	17259954	1237	0.000026	com.target
501	17258268	916	0.000035	com.clicky
502	17257832	1538	0.000020	uk.co.wired
503	17256742	391	0.000062	com.force
504	17256046	926	0.000034	com.java
505	17255234	1537	0.000020	com.gettyimages
506	17254874	1809	0.000019	us.countrystudies
507	17254832	1965	0.000018	com.semrush
508	17251558	1185	0.000027	org.gnupg
509	17250598	1122	0.000028	com.politico
510	17250452	1662	0.000019	com.womentechmakers
511	17250130	902	0.000035	gov.uspto
512	17247996	851	0.000036	org.whatbrowser
513	17247558	2155	0.000018	com.vanityfair
514	17245144	142	0.000180	ru.mail
515	17243458	435	0.000056	com.snapchat
516	17242522	1129	0.000028	com.istockphoto
517	17242096	217	0.000110	com.bitly
518	17241974	384	0.000064	com.adweek
519	17241808	1698	0.000019	com.ikea
520	17241278	268	0.000087	com.wufoo
521	17238236	162	0.000144	com.eepurl
522	17233098	1109	0.000029	org.archlinux
523	17232910	1334	0.000024	fr.lemonde
524	17231910	1331	0.000024	com.econsultancy
525	17231408	1138	0.000028	com.udemy
526	17231268	108	0.000268	jp.co.google
527	17230832	1389	0.000022	com.today
528	17228032	1717	0.000019	com.yellowbot
529	17227482	1227	0.000026	com.intuit
530	17227376	973	0.000033	org.iso
531	17226796	1567	0.000020	com.aliexpress
532	17226258	1468	0.000021	au.com.smh
533	17225468	1197	0.000026	co.vine
534	17225278	958	0.000033	com.hootsuite
535	17224354	1432	0.000022	com.underconsideration
536	17223030	1633	0.000020	uk.ac.hud
537	17222818	1429	0.000022	com.com
538	17221248	874	0.000036	com.nielsen
539	17219942	1755	0.000019	com.communitywalk
540	17219670	2806	0.000013	com.123rf
541	17217362	170	0.000141	com.xing
542	17216198	941	0.000034	com.livestream
543	17215352	950	0.000033	com.timeanddate
544	17214566	892	0.000035	de.blogspot
545	17214498	687	0.000042	com.proofpoint
546	17214342	316	0.000076	org.joomla
547	17214010	1303	0.000024	org.pnas
548	17213976	949	0.000033	com.americanexpress
549	17213226	1140	0.000028	org.fao
550	17211764	246	0.000096	com.wpengine
551	17211060	1011	0.000031	uk.ac.cam
552	17210928	1344	0.000023	com.snap
553	17210646	1270	0.000025	us.ma.state
554	17210158	430	0.000056	com.barnesandnoble
555	17210110	427	0.000057	com.squareup
556	17209608	772	0.000039	gov.justice
557	17207476	1341	0.000023	com.billboard
558	17207004	1020	0.000031	com.alibaba
559	17205496	1198	0.000026	net.noscript
560	17204520	1397	0.000022	org.letsencrypt
561	17203670	2386	0.000016	ca.uwaterloo
562	17203164	1711	0.000019	com.espn
563	17201262	1033	0.000031	io.fabric
564	17199364	2322	0.000016	ca.ubc
565	17198912	980	0.000032	com.variety
566	17195066	1143	0.000028	com.bostonglobe
567	17194856	1416	0.000022	com.homestars
568	17194608	2401	0.000015	com.tutsplus
569	17194034	2192	0.000018	edu.msu
570	17193846	1706	0.000019	com.bitballoon
571	17192748	668	0.000043	com.feedly
572	17192454	1292	0.000024	in.blogspot
573	17191504	1086	0.000029	fr.blogspot
574	17191464	2272	0.000017	com.fiverr
575	17189994	2226	0.000017	edu.indiana
576	17189122	1479	0.000021	uk.co.thesun
577	17187842	884	0.000036	gov.nps
578	17187810	1671	0.000019	com.mcafee
579	17186678	820	0.000037	com.gofundme
580	17186358	2859	0.000012	com.twitpic
581	17183206	1148	0.000028	com.dell
582	17182996	2892	0.000012	com.codecademy
583	17182726	1426	0.000022	com.city-data
584	17182686	1460	0.000021	io.bitbucket
585	17181510	708	0.000041	com.photoshelter
586	17181394	3334	0.000010	com.dreamstime
587	17181320	2493	0.000015	com.newscientist
588	17180454	1363	0.000023	com.nytco
589	17179308	463	0.000052	us.icio
590	17178552	1202	0.000026	com.yandex
591	17178234	206	0.000116	com.histats
592	17176076	2342	0.000016	uk.ac.ed
593	17175502	1040	0.000031	gov.fbi
594	17174634	846	0.000037	com.500px
595	17174572	431	0.000056	cn.com.sina
596	17173950	630	0.000045	com.mobirise
597	17173428	1217	0.000026	org.jenkins-ci
598	17172852	2448	0.000015	ca.ualberta
599	17170928	1618	0.000020	com.googlelabs
600	17170156	2164	0.000018	com.socialmediaexaminer
601	17170034	1059	0.000030	com.wayfair
602	17169444	1238	0.000026	uk.co.mirror
603	17169408	1300	0.000024	us.oh.state
604	17169406	806	0.000038	com.buffer
605	17169298	155	0.000152	it.google
606	17168866	529	0.000047	com.format
607	17168200	1346	0.000023	org.threejs
608	17167624	774	0.000039	com.uk
609	17167450	1448	0.000022	org.spie
610	17165660	1294	0.000024	kr.flic
611	17165516	1132	0.000028	edu.umn
612	17165214	1304	0.000024	com.iconarchive
613	17163904	233	0.000103	com.myshopify
614	17162886	434	0.000056	com.nasdaq
615	17162052	709	0.000041	com.uservoice
616	17161576	2353	0.000016	com.screencast
617	17161352	905	0.000035	br.com.uol
618	17160850	298	0.000080	nl.google
619	17160062	1219	0.000026	com.scientificamerican
620	17159442	2800	0.000013	ly.visual
621	17158882	964	0.000033	com.prweb
622	17158856	1178	0.000027	com.smashingmagazine
623	17158618	1366	0.000023	com.nymag
624	17157146	381	0.000064	com.dmca
625	17156996	1324	0.000024	com.hollywoodreporter
626	17155372	1273	0.000025	com.warnerbros
627	17155154	862	0.000036	net.openid
628	17154578	765	0.000039	gov.copyright
629	17153906	1249	0.000025	com.prezi
630	17152002	2439	0.000015	com.aljazeera
631	17151074	149	0.000170	gov.privacyshield
632	17150824	1042	0.000031	com.airbnb
633	17150340	924	0.000034	ca.cbc
634	17149820	1253	0.000025	com.gigaom
635	17148414	1101	0.000029	com.searchengineland
636	17148086	1320	0.000024	net.recode
637	17147394	2214	0.000017	com.searchenginejournal
638	17145414	1165	0.000027	com.reverbnation
639	17144640	1058	0.000030	com.redhat
640	17143504	273	0.000085	com.fc2
641	17142386	3438	0.000010	com.hubpages
642	17142298	1378	0.000023	com.freepik
643	17141892	1333	0.000024	com.nyt
644	17141868	699	0.000041	com.patreon
645	17141858	646	0.000044	gov.hhs
646	17141712	1357	0.000023	com.kissmetrics
647	17141112	1485	0.000021	com.rollingstone
648	17140398	1039	0.000031	org.apa
649	17139398	223	0.000108	fr.google
650	17138016	1191	0.000027	com.crashlytics
651	17137096	4071	0.000008	com.answers
652	17137056	1450	0.000022	com.autodesk
653	17136872	1348	0.000023	com.theglobeandmail
654	17136816	1210	0.000026	com.indeed
655	17136588	257	0.000091	com.getbootstrap
656	17135766	3377	0.000010	com.domaintools
657	17134556	1533	0.000020	edu.dukeupress
658	17134490	2474	0.000015	edu.bu
659	17133898	1297	0.000024	org.scala-lang
660	17133524	706	0.000041	com.alexa
661	17133178	1047	0.000030	com.sciencedaily
662	17131500	1323	0.000024	com.vox
663	17131458	1080	0.000029	gov.usgs
664	17130422	107	0.000269	com.googleadservices
665	17130056	1419	0.000022	com.elpais
666	17129938	1777	0.000019	edu.alamo
667	17129706	474	0.000052	br.com.google
668	17128186	2346	0.000016	edu.asu
669	17128168	390	0.000062	com.newrelic
670	17127826	1505	0.000021	com.nba
671	17127350	788	0.000038	gov.state
672	17127102	2778	0.000013	com.macrumors
673	17126992	2393	0.000016	edu.ncsu
674	17126312	1385	0.000023	edu.jhu
675	17125306	2759	0.000013	com.starwars
676	17124492	1225	0.000026	us.imageshack
677	17124350	320	0.000075	com.netdna-ssl
678	17123996	1675	0.000019	org.virginiadot
679	17122508	2334	0.000016	ch.ethz
680	17122024	2301	0.000016	com.msnbc
681	17121934	1571	0.000020	com.nokia
682	17121692	705	0.000041	com.mckinsey
683	17121268	1182	0.000027	org.gentoo
684	17120294	429	0.000057	gov.irs
685	17119582	2046	0.000018	com.css-tricks
686	17119430	417	0.000059	com.bigcartel
687	17118212	1354	0.000023	com.thehill
688	17117616	1799	0.000019	edu.virginia
689	17117124	53	0.000634	com.messenger
690	17116930	1753	0.000019	com.fixr
691	17116722	992	0.000032	io.codepen
692	17115848	2203	0.000018	com.zazzle
693	17115538	1399	0.000022	com.gallup
694	17115352	907	0.000035	com.adage
695	17115320	502	0.000049	fr.amazon
696	17114884	194	0.000121	com.youku
697	17114790	3323	0.000010	com.rottentomatoes
698	17114464	1269	0.000025	com.businessweek
699	17114350	1268	0.000025	com.uber
700	17112978	1307	0.000024	com.nydailynews
701	17112294	325	0.000073	com.bizjournals
702	17112210	1849	0.000018	com.smartguy
703	17111208	1553	0.000020	com.hotfrog
704	17110654	2888	0.000012	edu.brown
705	17110092	1530	0.000021	uk.co.lrb
706	17109686	1527	0.000021	edu.umd
707	17108562	2418	0.000015	tv.periscope
708	17107332	1206	0.000026	int.coe
709	17106560	982	0.000032	org.oecd
710	17106474	1004	0.000032	org.change
711	17104716	1493	0.000021	com.searchenginewatch
712	17104622	1390	0.000022	it.binged
713	17104560	1496	0.000021	io.prototypr
714	17104464	541	0.000047	gov.sec
715	17103876	139	0.000185	de.bund
716	17103676	2165	0.000018	com.posterous
717	17103630	672	0.000043	com.emarketer
718	17103156	2500	0.000015	au.com.news
719	17103002	1853	0.000018	edu.ucdavis
720	17102406	2329	0.000016	com.blogs
721	17101526	2312	0.000016	com.nfl
722	17101098	2478	0.000015	com.cbs
723	17100940	1822	0.000019	com.hulu
724	17099830	1328	0.000024	com.pwc
725	17099418	1358	0.000023	ly.plot
726	17098734	1180	0.000027	com.firebaseapp
727	17098528	326	0.000073	me.fb
728	17098266	1236	0.000026	org.cambridge
729	17097620	1359	0.000023	fm.last
730	17097536	1256	0.000025	uk.co.theregister
731	17097340	1430	0.000022	com.kudzu
732	17097214	2298	0.000016	org.aclu
733	17097148	1541	0.000020	org.ushistory
734	17096970	286	0.000082	com.naver
735	17095930	890	0.000035	gov.sba
736	17095924	2609	0.000014	com.wikidot
737	17095660	465	0.000052	gov.epa
738	17095576	1708	0.000019	com.akamai
739	17094822	1572	0.000020	org.jstor
740	17094514	255	0.000092	com.marriott
741	17094372	1149	0.000028	org.redcross
742	17093170	350	0.000068	net.themeforest
743	17091942	2491	0.000015	com.lonelyplanet
744	17091664	2441	0.000015	mp.j
745	17089358	1518	0.000021	au.com.truelocal
746	17089222	2333	0.000016	com.discovery
747	17089064	1488	0.000021	com.domain
748	17088086	1006	0.000031	com.cbslocal
749	17087402	2773	0.000013	org.phys
750	17085534	1295	0.000024	gov.nyc
751	17085312	1134	0.000028	io.bower
752	17085254	2227	0.000017	org.rubyonrails
753	17085212	677	0.000043	uk.co.tripadvisor
754	17083968	2286	0.000017	com.urbandictionary
755	17083722	3081	0.000011	com.fivethirtyeight
756	17083390	1463	0.000021	com.insiderpages
757	17082484	1694	0.000019	org.twinery
758	17081344	190	0.000124	jp.ne.hatena
759	17080300	1795	0.000019	org.milaap
760	17079140	1678	0.000019	es.iac
761	17078816	1168	0.000027	com.accenture
762	17077842	1690	0.000019	com.2findlocal
763	17077284	872	0.000036	com.att
764	17077178	2240	0.000017	de.zeit
765	17077136	653	0.000044	gov.ny
766	17075592	914	0.000035	com.chicagotribune
767	17075424	1745	0.000019	com.planetware
768	17075300	237	0.000100	jp.co.amazon
769	17075058	2274	0.000017	edu.umass
770	17074694	1070	0.000029	com.investopedia
771	17074424	1656	0.000020	com.wsoctv
772	17074178	1902	0.000018	org.postimg
773	17073772	2157	0.000018	uk.ac.ucl
774	17072414	1857	0.000018	com.linkcentre
775	17072314	1585	0.000020	edu.vassar
776	17071320	2332	0.000016	com.ibtimes
777	17071232	1908	0.000018	com.chron
778	17071194	2151	0.000018	edu.cuny
779	17070736	1043	0.000030	gov.va
780	17070650	1568	0.000020	com.zillow
781	17070606	3084	0.000011	com.lynda
782	17070176	1667	0.000019	com.phnompenhpost
783	17069034	1002	0.000032	com.formstack
784	17068194	1345	0.000023	re.cli
785	17067768	831	0.000037	com.sagepub
786	17067044	2842	0.000013	com.animoto
787	17066988	1461	0.000021	ca.kijiji
788	17066682	1264	0.000025	com.xkcd
789	17066212	1564	0.000020	com.warriorplus
790	17066032	1120	0.000028	com.business2community
791	17065974	1473	0.000021	org.sigcomm
792	17065804	673	0.000043	org.openstreetmap
793	17064674	1654	0.000020	com.tiki-toki
794	17063856	1554	0.000020	jp.ac.kobe-u
795	17063490	2674	0.000013	com.kaspersky
796	17062710	1750	0.000019	com.trendland
797	17062642	478	0.000051	com.atlassian
798	17061988	983	0.000032	com.zoho
799	17061278	1663	0.000019	fr.estrepublicain
800	17059814	451	0.000053	gov.usda
801	17058166	2986	0.000012	com.9to5mac
802	17057630	1477	0.000021	com.theoutline
803	17057298	811	0.000038	gov.usa
804	17055890	3123	0.000011	uk.bl
805	17055592	1372	0.000023	com.strikingly
806	17055276	1756	0.000019	edu.ufl
807	17054970	256	0.000091	com.elegantthemes
808	17054808	2321	0.000016	com.apnews
809	17054684	454	0.000053	com.pinimg
810	17054674	1555	0.000020	org.gwtproject
811	17054664	93	0.000317	com.namecheap
812	17054558	530	0.000047	com.gotowebinar
813	17054260	2345	0.000016	org.gimp
814	17054258	647	0.000044	gov.ed
815	17054118	176	0.000136	org.icann
816	17053764	1676	0.000019	ws.snack
817	17053588	1519	0.000021	com.hotmail
818	17053486	2514	0.000015	com.ifttt
819	17053234	1489	0.000021	net.hockeyapp
820	17051790	3465	0.000010	com.virustotal
821	17051586	369	0.000066	org.opensource
822	17051534	1513	0.000021	com.acninc
823	17050572	2950	0.000012	org.moma
824	17050542	684	0.000042	ca.amazon
825	17049542	1380	0.000023	com.stitcher
826	17048914	994	0.000032	org.plos
827	17048462	2791	0.000013	edu.unl
828	17048406	1310	0.000024	com.over-blog
829	17048000	1746	0.000019	com.mercurynews
830	17047454	2762	0.000013	com.topsy
831	17046932	1790	0.000019	com.khamsat
832	17046596	4389	0.000007	com.lmgtfy
833	17046156	2853	0.000012	com.sophos
834	17045274	1720	0.000019	com.ignimgs
835	17044996	1392	0.000022	us.zoom
836	17044350	274	0.000085	com.maxcdn
837	17043462	2676	0.000013	edu.gmu
838	17043266	1008	0.000031	com.oup
839	17043250	947	0.000033	com.accuweather
840	17042470	1855	0.000018	net.wrightflyer
841	17042238	2487	0.000015	edu.utah
842	17042178	1128	0.000028	com.mixcloud
843	17041944	1099	0.000029	org.doxygen
844	17041938	2467	0.000015	com.producthunt
845	17041168	2315	0.000016	com.thestar
846	17040806	2230	0.000017	edu.arizona
847	17040254	1491	0.000021	com.sky
848	17039272	2473	0.000015	org.openoffice
849	17038806	691	0.000042	com.163
850	17037800	1702	0.000019	com.howstuffworks
851	17036946	1551	0.000020	com.company
852	17036894	2201	0.000018	com.pastebin
853	17036498	2269	0.000017	ru.narod
854	17036430	1398	0.000022	io.pantheon
855	17036358	1635	0.000020	com.discordapp
856	17035370	3275	0.000010	org.greenpeace
857	17034618	2231	0.000017	com.deadline
858	17034446	1472	0.000021	com.local
859	17034088	2873	0.000012	com.campaignmonitor
860	17033592	193	0.000121	jp.ameblo
861	17032336	2889	0.000012	org.bitcoin
862	17031994	1351	0.000023	com.socialmediatoday
863	17031174	1589	0.000020	it.blogspot
864	17030976	1293	0.000024	edu.si
865	17030968	7175	0.000005	org.audacityteam
866	17030720	841	0.000037	com.yp
867	17030368	2242	0.000017	com.livestrong
868	17030334	2450	0.000015	com.bestbuy
869	17029458	1313	0.000024	com.globo
870	17029366	166	0.000142	me.line
871	17028546	1852	0.000018	tv.royanews
872	17027902	2535	0.000014	com.mentalfloss
873	17027090	1298	0.000024	com.gumroad
874	17026950	1863	0.000018	com.boston
875	17026888	2617	0.000014	com.getresponse
876	17024844	1435	0.000022	com.cafepress
877	17024728	1208	0.000026	com.forrester
878	17022092	703	0.000041	com.usnews
879	17021648	999	0.000032	com.walmart
880	17020694	1449	0.000022	org.wiktionary
881	17020672	437	0.000056	com.criteo
882	17020270	1631	0.000020	au.com.whitepages
883	17016310	1444	0.000022	ca.calgaryseocompany
884	17016272	421	0.000058	com.adroll
885	17015756	1019	0.000031	de.heise
886	17014878	1441	0.000022	com.technorati
887	17014632	1808	0.000019	de.welt
888	17014592	1565	0.000020	com.bizcommunity
889	17014084	1401	0.000022	mil.army
890	17012948	2825	0.000013	com.fox
891	17012222	1729	0.000019	com.contentmarketinginstitute
892	17011716	2561	0.000014	com.yolasite
893	17011618	512	0.000048	com.udacity
894	17011062	2170	0.000018	com.podbean
895	17011022	1481	0.000021	de.bundesverfassungsgericht
896	17010836	221	0.000109	me.t
897	17010598	112	0.000255	info.aboutads
898	17010538	3071	0.000011	com.googlepages
899	17009902	1738	0.000019	com.pushwoosh
900	17009370	701	0.000041	com.gitlab
901	17009104	1233	0.000026	org.sonatype
902	17008736	3493	0.000010	org.notepad-plus-plus
903	17008340	3301	0.000010	edu.uic
904	17008246	1668	0.000019	com.waze
905	17007154	808	0.000038	es.com.blogspot
906	17007064	1458	0.000021	com.tiddlywiki
907	17006900	1100	0.000029	com.digiday
908	17006648	1658	0.000020	com.lulu
909	17006086	807	0.000038	uk.co.eventbrite
910	17005352	2910	0.000012	com.ndtv
911	17005126	1683	0.000019	com.ssllabs
912	17004666	1583	0.000020	com.sproutsocial
913	17004224	1830	0.000019	me.pxlme
914	17004142	1528	0.000021	com.neilpatel
915	17003742	1407	0.000022	int.wipo
916	17003612	1502	0.000021	org.filezilla-project
917	17002472	452	0.000053	com.custhelp
918	17001786	2238	0.000017	org.raspberrypi
919	17000878	1685	0.000019	com.quandl
920	17000606	2883	0.000012	edu.tufts
921	17000112	2366	0.000016	com.salon
922	16999154	3279	0.000010	org.metmuseum
923	16998660	3480	0.000010	com.spreaker
924	16998546	2543	0.000014	com.fineartamerica
925	16996432	1652	0.000020	net.brownbook
926	16996258	1289	0.000024	com.bmj
927	16994812	2542	0.000014	uk.co.express
928	16994548	3268	0.000010	in.lnkd
929	16993498	1189	0.000027	com.techtarget
930	16991836	3027	0.000012	edu.hawaii
931	16991760	1254	0.000025	org.pewresearch
932	16991692	2876	0.000012	com.fitbit
933	16991658	3392	0.000010	org.edx
934	16991126	2154	0.000018	uk.co.huffingtonpost
935	16990656	1030	0.000031	com.fotolia
936	16990256	1311	0.000024	com.optimizely
937	16990212	727	0.000040	com.geocities
938	16989440	1410	0.000022	com.mariadb
939	16989388	1068	0.000030	com.infusionsoft
940	16988498	3210	0.000011	com.popsci
941	16987912	827	0.000037	gov.house
942	16987790	3467	0.000010	cc.tiny
943	16986628	1766	0.000019	com.spoke
944	16986466	2666	0.000014	nl.uva
945	16985926	1727	0.000019	org.unfe
946	16985880	1028	0.000031	es.amazon
947	16985718	1647	0.000020	uk.gov.westsussex
948	16985558	1681	0.000019	com.chamberofcommerce
949	16985386	3584	0.000009	gd.is
950	16985282	1308	0.000024	net.java
951	16985238	654	0.000044	com.houzz
952	16985156	1090	0.000029	gov.archives
953	16984162	3313	0.000010	com.avast
954	16983948	2216	0.000017	com.examiner
955	16983802	1645	0.000020	com.thefabricator
956	16983790	505	0.000049	com.redbubble
957	16983296	1824	0.000019	com.computerworld
958	16982204	3454	0.000010	com.klout
959	16981086	934	0.000034	com.delicious
960	16978942	3329	0.000010	org.kiva
961	16978376	453	0.000053	com.teamviewer
962	16978280	2048	0.000018	com.cio
963	16977838	2171	0.000018	com.thedailybeast
964	16977598	411	0.000059	mp.mailchi
965	16977372	2435	0.000015	br.com.blogspot
966	16976616	1166	0.000027	com.netdna-cdn
967	16976496	889	0.000036	com.arcgis
968	16976058	2844	0.000013	com.createspace
969	16976028	4182	0.000008	net.deviantart
970	16975580	1761	0.000019	com.yelloyello
971	16975516	1576	0.000020	gov.cabq
972	16975490	480	0.000051	com.iconfinder
973	16975074	1431	0.000022	au.com.yellowpages
974	16973724	1257	0.000025	io.getmdl
975	16972780	1228	0.000026	com.thedrum
976	16972204	1696	0.000019	com.us
977	16971756	2422	0.000015	org.linuxfoundation
978	16969112	5378	0.000006	com.depositphotos
979	16969008	2328	0.000016	com.ign
980	16967900	1563	0.000020	org.gmplib
981	16967888	2832	0.000013	edu.caltech
982	16967252	2895	0.000012	com.infoq
983	16966764	2202	0.000018	edu.uci
984	16966186	1332	0.000024	com.xbox
985	16966066	1425	0.000022	com.techrepublic
986	16966028	1262	0.000025	com.glassdoor
987	16965506	1599	0.000020	com.apachelounge
988	16965454	895	0.000035	org.unicef
989	16965116	2990	0.000012	com.discogs
990	16964500	2913	0.000012	es.abc
991	16963166	743	0.000039	com.biomedcentral
992	16962674	2863	0.000012	nl.xs4all
993	16962502	1423	0.000022	org.heart
994	16961832	2610	0.000014	org.olympic
995	16960736	252	0.000093	com.ssl-images-amazon
996	16959972	2680	0.000013	de.bild
997	16959790	2672	0.000013	com.nbc
998	16959298	1691	0.000019	com.realtytimes
999	16959228	1456	0.000021	com.mediafire
1000	16959080	1569	0.000020	com.galvanize

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

January 2019 crawl archive now available

January 28, 2019Sebastian Nagel

The crawl archive for January 2019 is now available! It contains 2.85 billion web pages or 240 TiB of uncompressed content, crawled between January 15th and 24th.

The January crawl contains page captures of 850 million URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks (harmonic centrality) published as part of the Aug/Sep/Oct 2018 webgraph data set from the following sources:

sitemaps, RSS and Atom feeds
a breadth-first side crawl within a maximum of 6 links (“hops”) away from the homepages of the top 50 million hosts and domains
a random sample of outlinks taken from WAT files of the December crawl

The number of sampled URLs per domain depends on the domain’s harmonic centrality rank in the webgraph data set – higher ranking domain are allowed to “contribute” more URLs.

Archive Location and Download

The January crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-04/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-04/segment.paths.gz	100
WARC files	CC-MAIN-2019-04/warc.paths.gz	64000	58.86
WAT files	CC-MAIN-2019-04/wat.paths.gz	64000	18.88
WET files	CC-MAIN-2019-04/wet.paths.gz	64000	7.98
Robots.txt files	CC-MAIN-2019-04/robotstxt.paths.gz	64000	0.18
Non-200 responses files	CC-MAIN-2019-04/non200responses.paths.gz	64000	1.65
URL index files	CC-MAIN-2019-04/cc-index.paths.gz	302	0.21

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-04/. Also the columnar index has been updated to contain this crawl.

December 2018 crawl archive now available

December 22, 2018Sebastian Nagel

The crawl archive for December 2018 is now available! It contains 3.1 billion web pages or 250 TiB of uncompressed content, crawled between December 9th and 19th.

The December crawl contains page captures of 735 million URLs not contained in any crawl archive before. New URLs stem from:

extracting and sampling URLs from sitemaps, RSS and Atom feeds if provided by hosts visited in prior crawls. Hosts are selected from the highest-ranking 60 million domains of the Aug/Sep/Oct 2018 webgraph data set
a breadth-first side crawl within a maximum of 6 links (“hops”) away from the home pages of the top 50 million domains of the webgraph dataset
a random sample of outlinks taken from WAT files of the November crawl
30 million external links sampled from Wikipedia data dumps

Archive Location and Download

The December crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2018-51/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2018-51/segment.paths.gz	100
WARC files	CC-MAIN-2018-51/warc.paths.gz	63840	65.31
WAT files	CC-MAIN-2018-51/wat.paths.gz	63840	20.01
WET files	CC-MAIN-2018-51/wet.paths.gz	63840	8.43
Robots.txt files	CC-MAIN-2018-51/robotstxt.paths.gz	63840	0.22
Non-200 responses files	CC-MAIN-2018-51/non200responses.paths.gz	63840	1.71
URL index files	CC-MAIN-2018-51/cc-index.paths.gz	302	0.24

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-51/. Also the columnar index has been updated to contain this crawl.

November 2018 crawl archive now available

November 29, 2018Sebastian Nagel

The crawl archive for November 2018 is now available! It contains 2.6 billion web pages or 220 TiB of uncompressed content, crawled between November 12th and 22nd.

The November crawl contains 640 million new URLs, not contained in any crawl archive before. New URLs stem from:

extracting and sampling URLs from sitemaps, RSS and Atom feeds if provided by hosts visited in prior crawls. Hosts are selected from the highest-ranking 60 million domains of the Aug/Sep/Oct 2018 webgraph data set
a breadth-first side crawl within a maximum of 10 links (“hops”) away from the home pages of the top 40 million domains of the webgraph dataset
a random sample of outlinks taken from WAT files of the October crawl
50 million external links sampled from Wikipedia data dumps

Archive Location and Download

The November crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2018-47/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2018-47/segment.paths.gz	100
WARC files	CC-MAIN-2018-47/warc.paths.gz	56000	54.16
WAT files	CC-MAIN-2018-47/wat.paths.gz	56000	17.36
WET files	CC-MAIN-2018-47/wet.paths.gz	56000	7.42
Robots.txt files	CC-MAIN-2018-47/robotstxt.paths.gz	56000	0.2
Non-200 responses files	CC-MAIN-2018-47/non200responses.paths.gz	56000	1.92
URL index files	CC-MAIN-2018-47/cc-index.paths.gz	302	0.2

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-47/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs Aug/Sep/Oct 2018

November 13, 2018Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of August, September and October 2018. Additional information about data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., the Feb/Mar/Apr 2017 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

Host-level graph

The graph consists of 903 million nodes and 5.25 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 819 million dangling nodes (91%) and the largest strongly connected component contains only 60 million (6.5%) nodes. The host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 903 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2018-aug-sep-oct/host/. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2018-aug-sep-oct/host/ as prefix to access the files from everywhere.

The following files and formats are provided:

Size	File	Description
5.66 GB	cc-main-2018-aug-sep-oct-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 42 vertices files
23.60 GB	cc-main-2018-aug-sep-oct-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 98 edges files

9.63 GB	cc-main-2018-aug-sep-oct-host.graph	graph in BVGraph format
2 kB	cc-main-2018-aug-sep-oct-host.properties

10.83 GB	cc-main-2018-aug-sep-oct-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2018-aug-sep-oct-host-t.properties

1 kB	cc-main-2018-aug-sep-oct-host.stats	WebGraph statistics

13.47 GB	cc-main-2018-aug-sep-oct-host-ranks.txt.gz	harmonic centrality and pagerank

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 87 million nodes and 1.48 billion edges. 56% or 49 million nodes are dangling nodes, the largest strongly connected component covers 33.5 million or 38% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2018-aug-sep-oct/domain/ resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2018-aug-sep-oct/domain/.

Download files of the Common Crawl Aug/Sep/Oct 2018 domain-level webgraph

Size	File	Description
0.60 GB	cc-main-2018-aug-sep-oct-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
5.95 GB	cc-main-2018-aug-sep-oct-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.24 GB	cc-main-2018-aug-sep-oct-domain.graph	graph in BVGraph format
2 kB	cc-main-2018-aug-sep-oct-domain.properties

3.39 GB	cc-main-2018-aug-sep-oct-domain-t.graph	transpose of the graph
2 kB	cc-main-2018-aug-sep-oct-domain-t.properties

1 kB	cc-main-2018-aug-sep-oct-domain.stats	WebGraph statistics

1.89 GB	cc-main-2018-aug-sep-oct-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 87 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Aug/Sept/Oct 2018)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	24993276	2	0.012750	com.facebook
2	24671056	1	0.017210	com.googleapis
3	23453366	3	0.010761	com.google
4	22371572	4	0.008252	com.twitter
5	22136836	5	0.006786	com.youtube
6	21115246	6	0.006404	org.w
7	19598338	9	0.003741	com.instagram
8	19499472	8	0.004616	org.gmpg
9	19210640	10	0.003396	com.linkedin
10	18557384	14	0.002133	com.wordpress
11	18475084	13	0.002729	org.wordpress
12	18409784	25	0.001383	org.wikipedia
13	18366100	24	0.001489	com.gravatar
14	18361154	21	0.001623	com.pinterest
15	17992020	12	0.002819	com.bootstrapcdn
16	17977192	19	0.001774	com.apple
17	17931956	32	0.001000	com.blogspot
18	17718492	41	0.000758	be.youtu
19	17629132	34	0.000918	gl.goo
20	17616074	26	0.001308	com.microsoft
21	17591076	16	0.001984	com.googletagmanager
22	17583026	37	0.000840	com.amazon
23	17567490	15	0.002013	com.cloudflare
24	17539566	46	0.000658	com.tumblr
25	17512444	23	0.001557	com.adobe
26	17478800	28	0.001256	com.vimeo
27	17351964	61	0.000482	com.yahoo
28	17328248	22	0.001618	com.macromedia
29	17262608	45	0.000669	com.wp
30	17247612	36	0.000867	com.paypal
31	17245380	20	0.001675	com.github
32	17243016	27	0.001272	com.gstatic
33	17237096	33	0.000928	com.amazonaws
34	17218638	47	0.000617	me.wp
35	17175034	52	0.000568	org.mozilla
36	17148704	98	0.000318	com.googleusercontent
37	17124438	40	0.000793	co.t
38	17114262	75	0.000425	com.weebly
39	17088226	115	0.000256	com.nytimes
40	17080324	39	0.000796	net.cloudfront
41	17058308	78	0.000424	org.creativecommons
42	17046746	51	0.000584	org.w3
43	17034446	150	0.000171	org.wikimedia
44	16978138	65	0.000465	com.medium
45	16977018	59	0.000490	com.flickr
46	16969818	60	0.000482	ly.bit
47	16934330	38	0.000830	io.github
48	16925564	139	0.000188	net.slideshare
49	16907918	149	0.000171	com.theguardian
50	16886648	71	0.000433	com.jquery
51	16848068	133	0.000196	com.imgur
52	16838070	165	0.000145	com.myspace
53	16825278	53	0.000551	eu.europa
54	16813968	159	0.000152	com.imdb
55	16800572	35	0.000899	net.fbcdn
56	16751537	127	0.000208	com.issuu
57	16731296	113	0.000267	org.apache
58	16730104	30	0.001101	net.doubleclick
59	16713084	188	0.000124	com.tinyurl
60	16690164	278	0.000085	com.theverge
61	16686224	105	0.000288	com.reddit
62	16681526	120	0.000229	com.yelp
63	16673945	17	0.001886	com.wixstatic
64	16666427	260	0.000091	com.appspot
65	16661248	274	0.000086	com.buzzfeed
66	16660287	143	0.000182	com.oracle
67	16653660	134	0.000191	com.spotify
68	16642850	241	0.000097	me.about
69	16641262	123	0.000216	com.android
70	16635142	335	0.000071	org.chromium
71	16619845	7	0.004633	com.godaddy
72	16619118	155	0.000164	com.tripadvisor
73	16611356	31	0.001041	com.squarespace
74	16609865	285	0.000083	com.mysql
75	16599943	308	0.000076	com.about
76	16593164	346	0.000067	org.arxiv
77	16592299	131	0.000204	org.ietf
78	16588645	95	0.000343	com.soundcloud
79	16583851	337	0.000071	edu.upenn
80	16581591	435	0.000057	edu.princeton
81	16574803	358	0.000066	org.ieee
82	16573729	156	0.000164	org.gnu
83	16567342	124	0.000215	com.dropbox
84	16567126	324	0.000073	com.deviantart
85	16543836	148	0.000174	com.forbes
86	16528334	130	0.000206	com.whatsapp
87	16521358	87	0.000400	com.statcounter
88	16521254	427	0.000058	google.blog
89	16520460	430	0.000057	com.ssrn
90	16519609	62	0.000481	org.schema
91	16516790	132	0.000199	org.archive
92	16511114	144	0.000182	net.sourceforge
93	16502431	180	0.000130	com.cnn
94	16499128	320	0.000074	gov.loc
95	16489333	191	0.000121	com.foursquare
96	16488380	189	0.000122	edu.stanford
97	16487861	88	0.000386	com.bing
98	16483708	371	0.000065	edu.ucla
99	16479852	389	0.000062	com.stackexchange
100	16478936	494	0.000050	edu.gatech
101	16478408	414	0.000059	org.sciencemag
102	16476180	183	0.000129	com.dribbble
103	16475471	203	0.000109	com.nbcnews
104	16470933	413	0.000059	com.withgoogle
105	16450054	255	0.000092	com.example
106	16447977	359	0.000066	com.googlecode
107	16443987	108	0.000278	com.ytimg
108	16438657	169	0.000142	uk.co.bbc
109	16436761	236	0.000098	edu.mit
110	16428294	230	0.000099	com.mozilla
111	16424352	224	0.000102	com.githubusercontent
112	16414627	458	0.000054	com.sap
113	16412253	565	0.000046	com.flipboard
114	16411958	205	0.000109	com.washingtonpost
115	16408402	146	0.000176	com.blogger
116	16406897	546	0.000047	com.chrome
117	16404143	49	0.000590	com.fb
118	16402707	571	0.000045	edu.utah
119	16398122	402	0.000060	com.jetbrains
120	16396163	492	0.000050	com.chron
121	16395449	322	0.000073	com.git-scm
122	16394655	186	0.000125	com.huffingtonpost
123	16393901	223	0.000102	com.businessinsider
124	16393651	97	0.000330	com.wix
125	16391015	89	0.000380	com.paypalobjects
126	16379847	122	0.000225	org.bbb
127	16379547	243	0.000097	com.live
128	16372670	287	0.000082	gov.fda
129	16372641	280	0.000084	au.com.google
130	16372405	54	0.000524	com.list-manage
131	16371002	286	0.000082	edu.harvard
132	16367016	532	0.000047	com.fastcodesign
133	16366146	365	0.000066	com.tinypic
134	16364937	194	0.000117	com.wsj
135	16347562	410	0.000059	tv.ustream
136	16344105	298	0.000080	com.cnet
137	16342765	334	0.000071	com.bbc
138	16339948	387	0.000062	com.variety
139	16339248	407	0.000060	org.eclipse
140	16338636	441	0.000056	co.g
141	16336819	304	0.000078	com.reuters
142	16333116	258	0.000091	org.doi
143	16326228	262	0.000090	com.ibm
144	16321163	266	0.000088	com.wired
145	16318957	312	0.000076	uk.co.telegraph
146	16317274	197	0.000112	com.typepad
147	16317262	349	0.000067	com.gmail
148	16316342	418	0.000058	org.iana
149	16309796	269	0.000087	com.bloomberg
150	16309363	248	0.000095	net.windows
151	16308394	104	0.000290	com.shopify
152	16304961	456	0.000054	co.ibb
153	16304891	181	0.000129	com.stackoverflow
154	16303280	240	0.000097	com.techcrunch
155	16297167	55	0.000519	net.akamaihd
156	16296675	272	0.000087	com.go
157	16296217	154	0.000166	gov.nih
158	16289519	394	0.000061	gov.nasa
159	16288695	339	0.000071	com.msn
160	16287525	352	0.000067	com.latimes
161	16285423	162	0.000147	com.etsy
162	16282310	109	0.000274	com.google-analytics
163	16282286	508	0.000049	edu.rutgers
164	16282232	545	0.000047	ca.utoronto
165	16275672	170	0.000142	com.twimg
166	16275582	103	0.000293	com.mailchimp
167	16274598	90	0.000378	de.google
168	16271240	265	0.000088	org.acm
169	16268707	366	0.000066	com.mashable
170	16267430	498	0.000050	com.quora
171	16264746	416	0.000058	au.gov.nsw
172	16264148	116	0.000242	com.jimdo
173	16261109	50	0.000589	com.fontawesome
174	16252668	550	0.000047	com.vogue
175	16251642	467	0.000053	com.zdnet
176	16250818	357	0.000067	uk.co.dailymail
177	16247730	663	0.000044	com.hbo
178	16247621	447	0.000055	com.googleblog
179	16245680	761	0.000039	com.dezeen
180	16244687	277	0.000085	com.usatoday
181	16244320	158	0.000162	com.eventbrite
182	16243163	582	0.000045	edu.osu
183	16239459	263	0.000090	com.meetup
184	16229462	429	0.000058	gov.archives
185	16224180	450	0.000055	edu.cornell
186	16223175	461	0.000053	edu.berkeley
187	16218763	396	0.000061	com.ted
188	16217936	151	0.000170	com.opera
189	16214980	581	0.000045	edu.washington
190	16211363	299	0.000080	com.udacity
191	16208363	580	0.000045	org.hrw
192	16197869	208	0.000107	com.surveymonkey
193	16195499	316	0.000075	com.time
194	16192449	486	0.000051	com.ecwid
195	16187438	409	0.000060	com.kickstarter
196	16187407	321	0.000074	org.npr
197	16187344	696	0.000042	com.discogs
198	16181108	700	0.000042	io.itch
199	16177807	496	0.000050	org.unicode
200	16177748	313	0.000076	com.springer
201	16176015	29	0.001149	ru.yandex
202	16174231	446	0.000055	org.kernel
203	16173199	370	0.000065	com.aol
204	16173059	701	0.000042	com.economist
205	16171404	290	0.000081	com.hp
206	16168983	231	0.000099	com.mapquest
207	16167485	48	0.000602	com.qq
208	16163785	758	0.000039	org.wikibooks
209	16160518	362	0.000066	com.cnbc
210	16154540	390	0.000062	org.un
211	16152769	333	0.000072	org.python
212	16152422	488	0.000051	com.ft
213	16151474	210	0.000107	org.drupal
214	16148981	401	0.000060	me.paypal
215	16148740	690	0.000042	com.strava
216	16148152	417	0.000058	com.angieslist
217	16144222	267	0.000088	com.hubspot
218	16142474	136	0.000191	com.zendesk
219	16141403	504	0.000049	org.aarp
220	16139927	364	0.000066	com.giphy
221	16138024	741	0.000040	org.amnesty
222	16136086	552	0.000046	com.yellowpages
223	16133616	343	0.000069	com.nypost
224	16132797	767	0.000038	com.wikia
225	16132223	714	0.000041	com.dropboxusercontent
226	16131615	419	0.000058	com.fortune
227	16128988	70	0.000439	net.jsfiddle
228	16128428	330	0.000072	com.wiley
229	16127117	91	0.000355	com.baidu
230	16126449	201	0.000110	uk.co.amazon
231	16124648	509	0.000049	com.unsplash
232	16123335	145	0.000179	uk.co.google
233	16122860	361	0.000066	com.prnewswire
234	16119093	821	0.000037	com.slate
235	16117800	482	0.000051	com.cisco
236	16114323	353	0.000067	com.photobucket
237	16112036	561	0.000046	com.venturebeat
238	16111166	873	0.000036	com.pixabay
239	16108436	976	0.000034	com.arstechnica
240	16105654	198	0.000111	org.purl
241	16102642	206	0.000108	com.ebay
242	16101242	798	0.000038	com.manta
243	16099223	137	0.000189	com.wixsite
244	16098318	702	0.000042	com.intel
245	16097598	685	0.000043	com.nationalgeographic
246	16096399	442	0.000056	com.entrepreneur
247	16090287	405	0.000060	gov.whitehouse
248	16090076	459	0.000054	com.nature
249	16089802	319	0.000074	com.oreilly
250	16088319	376	0.000064	com.office
251	16087148	576	0.000045	com.samsung
252	16084044	57	0.000494	com.vk
253	16082814	479	0.000052	com.matterport
254	16080553	475	0.000052	org.postgresql
255	16078046	601	0.000045	com.newyorker
256	16075255	297	0.000081	gov.cdc
257	16075119	173	0.000138	com.constantcontact
258	16073933	697	0.000042	com.vice
259	16073113	829	0.000037	edu.psu
260	16071402	1136	0.000031	com.gizmodo
261	16071402	551	0.000047	com.scribd
262	16070353	923	0.000035	com.qz
263	16068616	356	0.000067	org.ampproject
264	16068603	557	0.000046	gov.nist
265	16068356	294	0.000081	me.telegram
266	16067025	490	0.000051	com.wikihow
267	16066386	812	0.000037	ly.snip
268	16066142	220	0.000104	com.disqus
269	16065694	973	0.000034	edu.yale
270	16062922	474	0.000052	com.cbsnews
271	16062264	779	0.000038	edu.kit
272	16060241	607	0.000044	org.eff
273	16059786	583	0.000045	com.box
274	16059328	237	0.000098	net.php
275	16057917	126	0.000209	com.feedburner
276	16057179	476	0.000052	com.theatlantic
277	16055682	828	0.000037	com.engadget
278	16052013	264	0.000089	gov.ftc
279	16047507	791	0.000038	com.merchantcircle
280	16044901	252	0.000093	com.digg
281	16044141	448	0.000055	org.hbr
282	16042660	707	0.000041	org.nodejs
283	16042189	453	0.000055	com.inc
284	16040892	374	0.000064	com.images-amazon
285	16039745	379	0.000064	com.skype
286	16038801	212	0.000107	com.salesforce
287	16038291	716	0.000041	com.statista
288	16035364	1421	0.000027	edu.utexas
289	16034294	293	0.000081	com.staticflickr
290	16033600	291	0.000081	com.fastcompany
291	16033037	1439	0.000027	com.pexels
292	16030476	694	0.000042	edu.columbia
293	16028618	876	0.000036	com.marketwatch
294	16027153	600	0.000045	com.avvo
295	16024349	1436	0.000027	com.storify
296	16023986	340	0.000070	int.who
297	16023292	106	0.000284	com.addthis
298	16021320	775	0.000038	com.indiatimes
299	16016177	1445	0.000026	com.thinkwithgoogle
300	16015721	406	0.000060	org.maven
301	16014399	449	0.000055	com.w3schools
302	16013658	1408	0.000028	com.smashingmagazine
303	16012164	878	0.000036	com.mysanantonio
304	16011714	372	0.000064	co.elastic
305	16011705	215	0.000105	com.stumbleupon
306	16011229	226	0.000101	to.amzn
307	16008183	1492	0.000025	edu.purdue
308	16006545	195	0.000116	net.behance
309	16006038	560	0.000046	org.pbs
310	16005359	63	0.000476	me.fb
311	16003375	302	0.000079	com.googlesyndication
312	16002994	969	0.000034	au.net.abc
313	16002951	1601	0.000022	com.vanityfair
314	16002791	499	0.000050	com.slack
315	16001492	270	0.000087	gov.ca
316	15999972	311	0.000076	com.tripod
317	15995276	338	0.000071	com.sxsw
318	15993036	408	0.000060	uk.co.blogspot
319	15990251	141	0.000185	com.weibo
320	15987681	684	0.000043	net.researchgate
321	15985567	1415	0.000027	com.alexa
322	15984335	355	0.000067	com.dailymotion
323	15982835	1401	0.000028	edu.ucsd
324	15982138	686	0.000042	com.blackberry
325	15981891	1014	0.000033	org.worldbank
326	15979100	315	0.000075	fr.free
327	15978004	472	0.000052	net.leadpages
328	15976584	974	0.000034	com.thenextweb
329	15971688	554	0.000046	com.moz
330	15970463	1699	0.000020	org.owasp
331	15969404	360	0.000066	com.sciencedirect
332	15968776	762	0.000039	com.uservoice
333	15968498	1003	0.000033	com.shutterstock
334	15965774	375	0.000064	edu.cmu
335	15962210	176	0.000134	org.icann
336	15961179	732	0.000040	com.proofpoint
337	15958400	903	0.000035	edu.uark
338	15957323	1082	0.000032	com.evernote
339	15956890	373	0.000064	com.livejournal
340	15955233	691	0.000042	com.googlesource
341	15951480	1006	0.000033	ly.ow
342	15949779	589	0.000045	gov.sec
343	15946301	955	0.000034	com.speakerdeck
344	15944949	1351	0.000029	com.lifehacker
345	15941679	584	0.000045	com.citysearch
346	15941304	879	0.000035	org.unesco
347	15940525	814	0.000037	com.psychologytoday
348	15937813	1319	0.000031	com.trello
349	15937606	913	0.000035	com.sfgate
350	15936285	994	0.000033	com.designobserver
351	15934103	1536	0.000024	edu.northwestern
352	15933505	457	0.000054	com.snapchat
353	15932071	1320	0.000031	uk.ac.ox
354	15931696	671	0.000043	tv.twitch
355	15931551	1021	0.000032	gov.fcc
356	15930923	678	0.000043	org.bitbucket
357	15929820	1778	0.000019	com.fifa
358	15929724	412	0.000059	com.businesswire
359	15928928	803	0.000037	org.aiga
360	15926617	244	0.000096	com.wufoo
361	15926593	569	0.000045	com.atlassian
362	15926203	214	0.000106	de.amazon
363	15925496	327	0.000072	com.typeform
364	15924766	1603	0.000022	com.mcafee
365	15922998	1047	0.000032	com.libsyn
366	15922233	1878	0.000017	org.coursera
367	15921752	916	0.000035	com.zynga
368	15921549	961	0.000034	com.kudzu
369	15921529	1852	0.000018	com.semrush
370	15920541	674	0.000043	com.ubuntu
371	15920148	1673	0.000021	com.econsultancy
372	15918200	1440	0.000027	com.indiegogo
373	15917275	1383	0.000028	com.politico
374	15917261	295	0.000081	org.mediawiki
375	15916739	754	0.000039	org.aclweb
376	15916639	963	0.000034	com.deloitte
377	15914854	930	0.000034	org.spie
378	15914760	981	0.000033	com.livestream
379	15912203	1449	0.000026	co.vine
380	15910119	1553	0.000023	org.khanacademy
381	15908826	516	0.000048	com.goodreads
382	15908339	989	0.000033	gov.uspto
383	15907762	303	0.000079	org.joomla
384	15906992	1398	0.000028	com.zoho
385	15902686	908	0.000035	me.websta
386	15901516	825	0.000037	com.foxnews
387	15900302	350	0.000067	com.booking
388	15899160	1013	0.000033	io.codepen
389	15898382	129	0.000206	com.youtube-nocookie
390	15897026	163	0.000146	jp.co.yahoo
391	15896378	1613	0.000022	edu.unc
392	15896141	1652	0.000021	com.technologyreview
393	15894277	1709	0.000020	com.digitaltrends
394	15893383	1217	0.000031	org.iso
395	15893229	1569	0.000023	com.pingdom
396	15893117	914	0.000035	gov.senate
397	15892860	289	0.000082	com.smugmug
398	15890595	199	0.000111	com.bandcamp
399	15889383	975	0.000034	com.mckinsey
400	15888601	920	0.000035	it.binged
401	15888102	1389	0.000028	com.udemy
402	15885995	991	0.000033	com.what3words
403	15885247	2561	0.000012	com.sophos
404	15884644	1622	0.000022	org.weforum
405	15884560	380	0.000064	net.themeforest
406	15884272	626	0.000044	gov.noaa
407	15882777	1896	0.000017	com.ehow
408	15881215	718	0.000041	org.vim
409	15879594	1490	0.000025	com.elpais
410	15879197	1343	0.000030	com.sciencedaily
411	15879074	445	0.000056	com.squareup
412	15879052	816	0.000037	com.gartner
413	15877064	439	0.000056	com.netflix
414	15873982	470	0.000053	com.webs
415	15873498	271	0.000087	com.rawgit
416	15873485	1035	0.000032	edu.uah
417	15873257	1710	0.000020	uk.co.wired
418	15871503	463	0.000053	com.bizjournals
419	15871161	1002	0.000033	com.americanexpress
420	15870167	1568	0.000023	org.pnas
421	15869151	433	0.000057	com.monster
422	15869106	1024	0.000032	com.nielsen
423	15866670	1317	0.000031	com.redhat
424	15866441	667	0.000044	com.java
425	15865156	76	0.000425	org.reactjs
426	15864692	1949	0.000017	ch.ethz
427	15862191	400	0.000060	com.force
428	15861930	404	0.000060	com.herokuapp
429	15861377	1798	0.000019	com.socialmediaexaminer
430	15860803	1108	0.000031	com.adage
431	15860212	892	0.000035	com.googledrive
432	15859653	1899	0.000017	com.tutsplus
433	15858284	114	0.000263	jp.co.google
434	15857073	1696	0.000020	edu.usc
435	15856570	984	0.000033	com.prweb
436	15856191	760	0.000039	gov.justice
437	15855748	1481	0.000025	com.playstation
438	15855432	1734	0.000020	com.canva
439	15854961	514	0.000049	us.icio
440	15852813	172	0.000138	com.xing
441	15850197	866	0.000036	re.cli
442	15850114	1572	0.000023	edu.uchicago
443	15849567	1400	0.000028	com.bostonglobe
444	15848723	801	0.000038	com.steampowered
445	15844441	292	0.000081	ca.google
446	15844259	437	0.000057	com.bigcartel
447	15843423	2150	0.000015	com.urbandictionary
448	15842728	844	0.000036	io.material
449	15841425	481	0.000051	com.bigcommerce
450	15839196	1540	0.000024	com.caniuse
451	15838309	245	0.000096	com.getclicky
452	15834448	1384	0.000028	com.dell
453	15834375	808	0.000037	gov.state
454	15834214	1732	0.000020	com.hotmail
455	15833672	250	0.000094	es.google
456	15831338	1692	0.000021	au.com.smh
457	15830767	1632	0.000022	com.upwork
458	15830199	737	0.000040	org.gnupg
459	15829812	998	0.000033	edu.utep
460	15829095	354	0.000067	com.stripe
461	15828852	901	0.000035	com.msdn
462	15828183	422	0.000058	com.adweek
463	15826915	1701	0.000020	com.codeplex
464	15826257	2005	0.000016	ca.uwaterloo
465	15825896	107	0.000283	org.networkadvertising
466	15824996	2475	0.000013	com.twitpic
467	15823630	1375	0.000029	uk.ac.cam
468	15823242	225	0.000101	com.myshopify
469	15823088	1752	0.000019	com.nike
470	15822418	845	0.000036	com.outlook
471	15822314	1498	0.000025	com.gettyimages
472	15821233	1341	0.000030	com.istockphoto
473	15820921	1189	0.000031	de.heise
474	15819474	1600	0.000022	com.marketo
475	15818475	520	0.000048	com.cargocollective
476	15818334	1368	0.000029	ca.blogspot
477	15817231	1990	0.000016	com.norton
478	15815744	1459	0.000026	de.spiegel
479	15814626	846	0.000036	jp.co.fujixerox
480	15813630	997	0.000033	com.chicagotribune
481	15812887	1807	0.000018	com.ikea
482	15812477	1550	0.000023	com.ning
483	15812254	2052	0.000016	com.crunchbase
484	15811034	699	0.000042	com.webmd
485	15808826	202	0.000110	com.windowsphone
486	15808375	1521	0.000024	com.scientificamerican
487	15808267	239	0.000097	com.getbootstrap
488	15808212	2567	0.000012	com.codecademy
489	15807983	1099	0.000031	edu.alamo
490	15807399	507	0.000049	com.npmjs
491	15806866	1585	0.000022	com.billboard
492	15806166	1052	0.000032	com.theschooloflife
493	15805533	2014	0.000016	com.msnbc
494	15804337	2303	0.000014	com.instructables
495	15803920	725	0.000040	gov.copyright
496	15803640	1530	0.000024	uk.ac.ucl
497	15803572	1676	0.000021	fr.lemonde
498	15802334	925	0.000034	edu.umich
499	15800853	1378	0.000028	edu.wisc
500	15800747	140	0.000188	ru.mail
501	15800371	2358	0.000013	com.starwars
502	15797878	865	0.000036	de.blogspot
503	15797791	1508	0.000024	com.kissmetrics
504	15797047	1115	0.000031	com.beautifulpixels
505	15796947	1386	0.000028	com.airbnb
506	15796853	2451	0.000013	edu.hbs
507	15796325	166	0.000145	com.eepurl
508	15795637	768	0.000038	com.css-tricks
509	15795363	233	0.000098	com.bitly
510	15794822	1615	0.000022	edu.jhu
511	15793691	1362	0.000029	com.alibaba
512	15792654	1164	0.000031	com.sun
513	15792271	772	0.000038	com.tandfonline
514	15791593	893	0.000035	com.underconsideration
515	15790633	518	0.000048	in.co.google
516	15789087	793	0.000038	com.uber
517	15788507	704	0.000042	com.photoshelter
518	15787332	566	0.000046	com.symantec
519	15787196	2936	0.000010	uk.bl
520	15786855	683	0.000043	gov.hhs
521	15783209	807	0.000037	io.getmdl
522	15782831	1691	0.000021	com.irishtimes
523	15781874	2154	0.000015	edu.ncsu
524	15781333	1393	0.000028	com.searchenginejournal
525	15781203	67	0.000448	com.messenger
526	15780051	517	0.000048	org.sonatype
527	15778677	979	0.000033	ca.cbc
528	15778587	1592	0.000022	com.yandex
529	15777890	431	0.000057	com.clicky
530	15777427	1768	0.000019	com.hulu
531	15776625	1443	0.000026	com.accenture
532	15774420	1610	0.000022	edu.academia
533	15773314	528	0.000047	gov.epa
534	15772833	1403	0.000028	com.marketingland
535	15772472	972	0.000034	uk.co.guardian
536	15771759	2040	0.000016	tv.periscope
537	15769999	1616	0.000022	com.today
538	15768447	2453	0.000013	ly.visual
539	15766945	369	0.000065	edu.nyu
540	15766843	1409	0.000028	org.apa
541	15766561	2894	0.000011	com.girlswhocode
542	15766409	1546	0.000024	com.hollywoodreporter
543	15765504	549	0.000047	uk.co.independent
544	15764791	2378	0.000013	com.glamour
545	15764760	2131	0.000015	au.com.news
546	15764489	553	0.000046	gov.ed
547	15763374	1714	0.000020	com.invisionapp
548	15763321	2450	0.000013	org.gimp
549	15763104	709	0.000041	com.feedly
550	15763001	1322	0.000031	org.change
551	15761645	2072	0.000015	com.ibtimes
552	15761255	1598	0.000022	com.thomsonreuters
553	15760281	1517	0.000024	gov.nyc
554	15760200	1826	0.000018	com.posterous
555	15759019	943	0.000034	com.bravesites
556	15758123	3649	0.000008	com.space
557	15758105	1434	0.000027	gov.bls
558	15756979	443	0.000056	cn.com.sina
559	15756712	397	0.000061	com.custhelp
560	15755071	2389	0.000013	com.tesla
561	15753906	1476	0.000025	com.businessweek
562	15753434	774	0.000038	com.uk
563	15753186	1782	0.000019	com.zillow
564	15752235	1814	0.000018	com.zapier
565	15751997	2583	0.000012	com.dreamstime
566	15751546	3003	0.000010	com.klout
567	15750992	1623	0.000022	com.thehill
568	15750722	234	0.000098	com.wpengine
569	15750076	2978	0.000010	com.rottentomatoes
570	15749795	2693	0.000012	com.campaignmonitor
571	15749130	1629	0.000022	uk.ac.ed
572	15748626	2246	0.000014	com.wikidot
573	15748387	2252	0.000014	com.123rf
574	15748038	217	0.000105	fr.google
575	15747873	1611	0.000022	com.intuit
576	15747479	1641	0.000021	org.letsencrypt
577	15746782	875	0.000036	com.questionpro
578	15744807	664	0.000044	com.gotowebinar
579	15744526	1987	0.000016	com.nokia
580	15742939	2658	0.000012	edu.brown
581	15742494	3600	0.000008	com.formula1
582	15742364	2184	0.000014	com.mentalfloss
583	15742342	451	0.000055	gov.irs
584	15742266	491	0.000050	net.openid
585	15740664	1688	0.000021	com.nba
586	15739222	1593	0.000022	org.pewresearch
587	15738724	2222	0.000014	com.aljazeera
588	15738356	1058	0.000032	com.ezlocal
589	15737381	1437	0.000027	org.altervista
590	15737002	1478	0.000025	in.blogspot
591	15736320	279	0.000084	it.placehold
592	15735548	3233	0.000009	edu.uic
593	15735280	2251	0.000014	com.programmableweb
594	15735247	2153	0.000015	com.cbs
595	15734807	1153	0.000031	gov.sba
596	15734218	2198	0.000014	com.techradar
597	15734158	826	0.000037	gov.census
598	15733513	1747	0.000019	org.postimg
599	15732878	506	0.000049	gov.usda
600	15732533	1535	0.000024	com.target
601	15731343	721	0.000041	com.docker
602	15731122	1519	0.000024	com.gigaom
603	15731013	2800	0.000011	com.oxforddictionaries
604	15728172	1693	0.000021	net.daum
605	15727989	962	0.000034	com.gofundme
606	15727980	1639	0.000022	kr.flic
607	15726681	1156	0.000031	com.formstack
608	15726256	763	0.000039	org.sqlite
609	15725436	1661	0.000021	com.autodesk
610	15724812	1396	0.000028	com.techrepublic
611	15724806	817	0.000037	com.patreon
612	15721240	970	0.000034	com.insiderpages
613	15721227	1795	0.000019	com.us
614	15720198	1031	0.000032	com.hotfrog
615	15720166	966	0.000034	com.whitepages
616	15719605	1900	0.000017	edu.illinois
617	15719578	1642	0.000021	com.pwc
618	15718460	2039	0.000016	edu.asu
619	15718391	2486	0.000013	com.animoto
620	15717320	249	0.000094	com.fc2
621	15717052	1825	0.000018	org.rubyonrails
622	15716625	742	0.000040	com.wunderground
623	15716037	213	0.000106	org.debian
624	15715960	1124	0.000031	org.cmlibrary
625	15715829	1062	0.000032	com.idt
626	15715705	1359	0.000029	com.investopedia
627	15715452	1856	0.000018	com.howstuffworks
628	15714753	1326	0.000030	org.redcross
629	15714617	1493	0.000025	com.indeed
630	15713762	2101	0.000015	com.lonelyplanet
631	15713705	2054	0.000016	com.gamespot
632	15713431	910	0.000035	gov.nps
633	15713159	1084	0.000032	com.thesprintbook
634	15712729	1141	0.000031	com.smartguy
635	15711951	832	0.000037	com.att
636	15711660	2049	0.000016	com.refinery29
637	15709292	522	0.000048	com.vendio
638	15709144	2851	0.000011	com.domaintools
639	15708842	874	0.000036	com.itsnicethat
640	15707939	1801	0.000018	org.filezilla-project
641	15707760	1395	0.000028	com.vmware
642	15707005	171	0.000139	it.google
643	15706361	3994	0.000007	com.boredpanda
644	15705364	1391	0.000028	gov.va
645	15705335	849	0.000036	com.pinimg
646	15704965	1428	0.000027	com.reverbnation
647	15704604	2016	0.000016	ca.ubc
648	15704346	1995	0.000016	com.nfl
649	15703768	666	0.000044	com.houzz
650	15703700	1516	0.000024	com.prezi
651	15703074	1912	0.000017	edu.indiana
652	15702174	3049	0.000010	com.hubpages
653	15701522	436	0.000057	com.nasdaq
654	15701359	2734	0.000011	com.9to5mac
655	15701239	1579	0.000023	com.pcworld
656	15700785	1824	0.000018	edu.ucdavis
657	15700731	1416	0.000027	gov.usgs
658	15700075	886	0.000035	com.500px
659	15699652	1001	0.000033	com.acninc
660	15699448	2157	0.000015	com.livestrong
661	15699048	1328	0.000030	org.oecd
662	15698519	2267	0.000014	com.newscientist
663	15697206	1846	0.000018	com.espn
664	15697101	1484	0.000025	edu.umn
665	15697074	1703	0.000020	com.freepik
666	15696322	1902	0.000017	edu.virginia
667	15694887	1605	0.000022	com.vox
668	15694606	1858	0.000018	com.deadline
669	15693525	483	0.000051	org.whatbrowser
670	15692991	1499	0.000025	com.mixcloud
671	15691728	847	0.000036	com.emarketer
672	15691597	1360	0.000029	fr.blogspot
673	15691539	695	0.000042	com.flippa
674	15691203	256	0.000092	com.elegantthemes
675	15690356	1590	0.000022	com.newsweek
676	15689675	2170	0.000015	com.getresponse
677	15688589	460	0.000054	io.atom
678	15688584	1700	0.000020	com.gallup
679	15688318	2187	0.000014	edu.bu
680	15687369	2815	0.000011	org.moma
681	15686394	1888	0.000017	com.findlaw
682	15683775	1542	0.000024	edu.si
683	15683516	2094	0.000015	com.pastebin
684	15682917	1155	0.000031	dk.fcm
685	15682640	1547	0.000024	com.globo
686	15682617	368	0.000065	org.openstreetmap
687	15681889	1142	0.000031	org.writersleague
688	15680577	1884	0.000017	edu.cuny
689	15680551	1925	0.000017	com.starbucks
690	15680465	1447	0.000026	com.warnerbros
691	15679238	2075	0.000015	com.socialmediatoday
692	15678966	1150	0.000031	com.prosperent
693	15678673	1114	0.000031	org.grayarea
694	15678448	1984	0.000016	org.aclu
695	15677879	739	0.000040	org.jenkins-ci
696	15674592	2002	0.000016	com.mercurynews
697	15674443	1552	0.000023	com.business2community
698	15674402	1836	0.000018	mp.j
699	15674363	4368	0.000007	com.petapixel
700	15673782	2630	0.000012	com.googlepages
701	15673492	1894	0.000017	com.hostgator
702	15673279	745	0.000039	com.geocities
703	15672825	1336	0.000030	org.mayoclinic
704	15672261	167	0.000143	gov.privacyshield
705	15671234	1049	0.000032	com.ycombinator
706	15670571	1496	0.000025	net.java
707	15670017	1463	0.000026	us.imageshack
708	15669905	2432	0.000013	com.psychcentral
709	15669061	1624	0.000022	com.boston
710	15667800	1509	0.000024	org.fao
711	15666438	1980	0.000016	edu.arizona
712	15665639	1581	0.000023	com.nydailynews
713	15665192	1832	0.000018	de.welt
714	15665099	238	0.000098	com.youku
715	15664693	1915	0.000017	com.salon
716	15664554	2365	0.000013	edu.gmu
717	15663687	1017	0.000032	com.aweber
718	15663661	242	0.000097	jp.co.amazon
719	15663656	2299	0.000014	com.yourdomain
720	15662150	2021	0.000016	com.domain
721	15661511	2285	0.000014	com.ew
722	15659706	1149	0.000031	com.collegian
723	15659043	796	0.000038	org.elasticsearch
724	15658487	1380	0.000028	com.mlb
725	15658455	899	0.000035	com.delicious
726	15658257	2239	0.000014	ca.ualberta
727	15657830	3265	0.000009	org.edx
728	15655920	988	0.000033	google.design
729	15655666	2776	0.000011	org.kiva
730	15654526	1410	0.000028	com.weather
731	15654299	1837	0.000018	net.codecanyon
732	15654288	2743	0.000011	com.lynda
733	15654205	1503	0.000024	com.merriam-webster
734	15654147	1042	0.000032	com.womentechmakers
735	15654091	1065	0.000032	net.brownbook
736	15653789	986	0.000033	com.hootsuite
737	15653450	3979	0.000007	com.lmgtfy
738	15652070	426	0.000058	com.ea
739	15651918	1779	0.000019	edu.umd
740	15651858	1497	0.000025	com.thedrum
741	15650961	1735	0.000020	com.aliexpress
742	15650884	204	0.000109	com.automattic
743	15650243	1514	0.000024	int.coe
744	15650019	2247	0.000014	org.openoffice
745	15649988	1658	0.000021	com.firefox
746	15649772	1599	0.000022	com.searchenginewatch
747	15649620	1853	0.000018	com.zazzle
748	15648960	2027	0.000016	com.gq
749	15648656	1574	0.000023	org.cambridge
750	15648651	1904	0.000017	edu.msu
751	15647741	444	0.000056	com.barnesandnoble
752	15647388	2149	0.000015	com.azcentral
753	15647181	2429	0.000013	edu.wustl
754	15647110	2554	0.000012	org.semanticscholar
755	15646960	1887	0.000017	edu.umass
756	15646650	1555	0.000023	fm.last
757	15646167	2060	0.000016	au.com.blogspot
758	15645607	1191	0.000031	site.tenerifeforum
759	15645549	3140	0.000010	com.copyblogger
760	15645390	1102	0.000031	uk.gov.peterborough
761	15644870	2289	0.000014	com.topsy
762	15644590	897	0.000035	com.unity3d
763	15644472	1628	0.000022	com.over-blog
764	15643911	1501	0.000025	com.waze
765	15642164	2320	0.000014	com.gawker
766	15642103	2466	0.000013	ms.1drv
767	15641828	1370	0.000029	com.timeanddate
768	15641339	3477	0.000009	com.answers
769	15641169	1325	0.000030	com.arcgis
770	15640859	794	0.000038	com.clkmg
771	15639069	1080	0.000032	com.cbslocal
772	15638930	2576	0.000012	org.phys
773	15638348	1695	0.000021	com.stitcher
774	15637681	1663	0.000021	com.gumroad
775	15637217	1366	0.000029	gov.fbi
776	15637050	2334	0.000013	com.fiverr
777	15636227	1800	0.000019	com.lulu
778	15635676	1671	0.000021	com.rollingstone
779	15635417	1880	0.000017	com.nvidia
780	15635094	2702	0.000012	com.headspace
781	15634767	341	0.000070	org.opensource
782	15634400	1684	0.000021	com.neilpatel
783	15633999	1771	0.000019	uk.co.metro
784	15633208	990	0.000033	jp.ac.kobe-u
785	15632996	1741	0.000020	com.mtv
786	15632518	56	0.000499	net.facebook
787	15632516	2649	0.000012	edu.tufts
788	15632419	915	0.000035	br.com.uol
789	15631894	2562	0.000012	com.fox
790	15631850	1057	0.000032	com.brightcove
791	15631313	1505	0.000024	com.sky
792	15630776	2933	0.000010	com.popsci
793	15630301	3389	0.000009	com.wolfram
794	15627572	2434	0.000013	com.theonion
795	15627567	1348	0.000029	org.readthedocs
796	15627335	1927	0.000017	com.trendmicro
797	15626654	388	0.000062	com.marriott
798	15626569	342	0.000070	nl.google
799	15626305	2721	0.000011	edu.caltech
800	15625896	1060	0.000032	com.2findlocal
801	15625270	1472	0.000025	uk.co.theregister
802	15625157	891	0.000035	uk.co.eventbrite
803	15625156	1121	0.000031	com.fotolia
804	15624596	1849	0.000018	com.history
805	15624274	306	0.000077	com.naver
806	15623770	2985	0.000010	edu.dartmouth
807	15623624	1606	0.000022	com.bmj
808	15623028	2515	0.000012	ch.cern
809	15622919	1914	0.000017	it.scoop
810	15621936	1357	0.000029	com.walmart
811	15621746	1930	0.000017	org.kde
812	15621344	1898	0.000017	com.nrf
813	15619330	1649	0.000021	im.gitter
814	15619286	2379	0.000013	com.bestbuy
815	15619283	473	0.000052	com.iconfinder
816	15618356	1866	0.000018	org.jstor
817	15618109	1377	0.000028	com.searchengineland
818	15616272	184	0.000128	jp.ne.hatena
819	15615800	1543	0.000024	com.splashthat
820	15614563	3110	0.000010	org.notepad-plus-plus
821	15614110	1627	0.000022	com.com
822	15613738	1529	0.000024	org.heart
823	15612896	2529	0.000012	edu.uiuc
824	15612666	2730	0.000011	com.fitbit
825	15611859	1026	0.000032	com.company
826	15610954	2489	0.000012	com.wikispaces
827	15610875	1541	0.000024	com.cafepress
828	15610542	1738	0.000020	com.ssllabs
829	15610139	2352	0.000013	de.bild
830	15608795	69	0.000447	com.parallels
831	15608630	917	0.000035	gov.usa
832	15608624	1806	0.000018	com.buffer
833	15608543	1966	0.000016	com.discordapp
834	15607778	1206	0.000031	com.infusionsoft
835	15607523	2031	0.000016	edu.uci
836	15607224	838	0.000036	org.openweathermap
837	15606632	3159	0.000010	gd.is
838	15605502	182	0.000129	jp.ameblo
839	15604837	900	0.000035	com.cdbaby
840	15604598	1000	0.000033	com.newsbank
841	15604393	1815	0.000018	com.deezer
842	15603804	1822	0.000018	com.discovery
843	15602784	765	0.000038	org.doxygen
844	15602226	1030	0.000032	org.travelblog
845	15602213	1034	0.000032	org.tpr
846	15601034	428	0.000058	net.launchpad
847	15600189	777	0.000038	com.sagepub
848	15598493	1059	0.000032	com.chamberofcommerce
849	15598065	510	0.000049	com.cracked
850	15597590	749	0.000039	org.plos
851	15597251	4943	0.000006	com.checkpoint
852	15597031	1936	0.000017	uk.co.thesun
853	15597000	99	0.000302	com.namecheap
854	15596654	3162	0.000009	com.spreaker
855	15596199	1533	0.000024	com.xkcd
856	15593759	1331	0.000030	com.tableau
857	15593642	1488	0.000025	com.pcmag
858	15593497	1934	0.000017	edu.ufl
859	15591634	3453	0.000009	edu.buffalo
860	15591344	2771	0.000011	com.producthunt
861	15591279	3424	0.000009	org.lifehack
862	15591113	1977	0.000016	com.examiner
863	15591022	1073	0.000032	net.azurewebsites
864	15590091	2360	0.000013	com.bleacherreport
865	15589566	1016	0.000033	com.bizcommunity
866	15589420	996	0.000033	com.chambermaster
867	15589294	1147	0.000031	com.oup
868	15589126	1889	0.000017	com.thedailybeast
869	15588805	2640	0.000012	com.snopes
870	15588137	2084	0.000015	com.ign
871	15588059	3592	0.000008	com.appleinsider
872	15587571	1096	0.000031	com.lookuppage
873	15587541	2059	0.000016	com.mac
874	15587446	746	0.000039	com.usnews
875	15586928	669	0.000043	com.163
876	15586259	2966	0.000010	org.greenpeace
877	15586114	3672	0.000008	edu.temple
878	15586108	919	0.000035	com.tiddlywiki
879	15585180	1993	0.000016	de.zeit
880	15584327	1660	0.000021	com.strikingly
881	15584124	1854	0.000018	co.angel
882	15583419	2237	0.000014	com.yolasite
883	15583266	541	0.000047	com.1and1
884	15583228	1650	0.000021	com.windows
885	15583160	2454	0.000013	net.comcast
886	15582183	4541	0.000007	com.blog
887	15581911	1350	0.000029	com.shareasale
888	15581584	1103	0.000031	com.spoke
889	15580712	2676	0.000012	com.macrumors
890	15580235	2106	0.000015	com.si
891	15580055	2947	0.000010	com.avast
892	15579731	1104	0.000031	com.communitywalk
893	15579534	1039	0.000032	com.independent
894	15579484	1828	0.000018	it.blogspot
895	15578882	2235	0.000014	com.icloud
896	15578813	2227	0.000014	ca.sfu
897	15578331	1759	0.000019	edu.duke
898	15578134	1406	0.000028	gov.ny
899	15578058	3048	0.000010	edu.ucsc
900	15577482	1781	0.000019	com.lithium
901	15577447	3081	0.000010	com.marieclaire
902	15577247	890	0.000035	com.mariadb
903	15576962	3202	0.000009	com.brainyquote
904	15576768	2598	0.000012	ca.globalnews
905	15576034	2895	0.000011	edu.oregonstate
906	15575967	738	0.000040	es.com.blogspot
907	15575607	681	0.000043	fr.amazon
908	15575040	2570	0.000012	com.nintendo
909	15574989	153	0.000166	de.bund
910	15574629	2078	0.000015	com.popsugar
911	15574040	1116	0.000031	com.lacartes
912	15573641	1929	0.000017	com.angelfire
913	15573588	2067	0.000015	org.poynter
914	15573573	1071	0.000032	com.citysquares
915	15573417	2202	0.000014	com.movember
916	15573266	1882	0.000017	uk.ac.lse
917	15573017	1045	0.000032	com.thegreatdiscontent
918	15572917	2098	0.000015	org.wpmudev
919	15572164	2522	0.000012	com.fineartamerica
920	15572083	2493	0.000012	edu.vt
921	15571965	2833	0.000011	edu.hawaii
922	15571730	2171	0.000015	com.teenvogue
923	15571641	1531	0.000024	com.calendly
924	15571635	1558	0.000023	com.steamcommunity
925	15571429	3026	0.000010	org.thinkprogress
926	15571425	1435	0.000027	com.techtarget
927	15571248	2045	0.000016	com.blogtalkradio
928	15571188	687	0.000042	uk.co.tripadvisor
929	15571109	1526	0.000024	com.glassdoor
930	15570512	1544	0.000024	com.xbox
931	15570312	1381	0.000028	me.m
932	15569885	2218	0.000014	uk.co.express
933	15569288	1675	0.000021	uk.co.mirror
934	15568503	119	0.000232	info.aboutads
935	15568393	2508	0.000012	com.blogs
936	15568027	2115	0.000015	com.templatemonster
937	15567578	328	0.000072	com.netdna-ssl
938	15567289	1371	0.000029	gov.dol
939	15567230	1554	0.000023	org.unicef
940	15567188	1135	0.000031	com.netdna-cdn
941	15566390	411	0.000059	com.mapbox
942	15566061	1088	0.000032	com.americantowns
943	15565930	2456	0.000013	org.7-zip
944	15565290	3852	0.000008	com.thenation
945	15564772	853	0.000036	ca.amazon
946	15564624	4817	0.000006	com.depositphotos
947	15564555	2687	0.000012	edu.pitt
948	15564538	2603	0.000012	nl.uva
949	15564284	2111	0.000015	sg.com.google
950	15564236	1020	0.000032	com.galvanize
951	15563763	1043	0.000032	com.judysbook
952	15563330	1086	0.000032	org.twinery
953	15563027	1756	0.000019	com.timeout
954	15562787	1697	0.000020	com.mediafire
955	15561709	2086	0.000015	com.w3techs
956	15561676	1373	0.000029	com.ups
957	15561291	945	0.000034	gov.house
958	15560994	855	0.000036	io.pantheon
959	15560477	2395	0.000013	com.me
960	15559169	3207	0.000009	cc.tiny
961	15559098	1958	0.000016	com.apnews
962	15557883	3248	0.000009	org.code
963	15557727	537	0.000047	com.getpocket
964	15557527	672	0.000043	com.elsevier
965	15556848	729	0.000040	com.prestashop
966	15556833	2388	0.000013	com.homedepot
967	15556798	1446	0.000026	com.bufferapp
968	15556767	3416	0.000009	com.virustotal
969	15556551	1694	0.000021	com.outbrain
970	15556036	5022	0.000006	com.wechat
971	15555722	2340	0.000013	com.pandora
972	15555713	2301	0.000014	com.foxmovies
973	15555616	4140	0.000007	com.kpcb
974	15555485	2860	0.000011	com.lanyrd
975	15555314	724	0.000041	com.redbubble
976	15553924	3262	0.000009	org.catalyst
977	15553307	2096	0.000015	tech.ces
978	15553284	1591	0.000022	gov.wa
979	15553160	1477	0.000025	jp.blogspot
980	15552948	2110	0.000015	com.twilio
981	15552738	477	0.000052	mp.mailchi
982	15552671	3581	0.000008	com.biography
983	15552065	1932	0.000017	com.healthline
984	15551576	1085	0.000032	com.pacegallery
985	15551541	3002	0.000010	com.iconosquare
986	15550836	2000	0.000016	com.baltimoresun
987	15550166	2803	0.000011	com.imageshack
988	15549943	1945	0.000017	gov.uscourts
989	15549663	2875	0.000011	int.esa
990	15549147	2731	0.000011	com.virgin
991	15548392	4588	0.000006	com.diigo
992	15548390	1810	0.000018	com.people
993	15548386	1473	0.000025	se.haxx
994	15547759	1706	0.000020	com.visualstudio
995	15547737	3064	0.000010	com.freelancer
996	15547640	2307	0.000014	com.xerox
997	15547505	512	0.000049	com.myportfolio
998	15547438	1364	0.000029	es.amazon
999	15546424	3356	0.000009	com.complex
1000	15545977	469	0.000053	br.com.google

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

October 2018 crawl archive now available

October 30, 2018Sebastian Nagel

The crawl archive for October 2018 is now available! It contains 3.0 billion web pages and 240 TiB of uncompressed content, crawled between October 15th and 24th.

The October crawl contains 600 million new URLs, not contained in any crawl archive before. New URLs stem from:

extracting and sampling URLs from sitemaps, RSS and Atom feeds if provided by hosts visited in prior crawls. Hosts are selected from the highest-ranking 60 million domains of the May/June/July 2018 webgraph data set
a breadth-first side crawl within a maximum of 10 links (“hops”) away from the home pages of the top 40 million domains of the webgraph dataset
a random sample of outlinks taken from WAT files of the September crawl
15 million external links sampled from Wikipedia data dumps

Please note that the character set detection was not fully working for the first 13 segments of the October crawl – about 15% of the page captures in these segments have no charset and language assigned. More information is found in the bug report.

Archive Location and Download

The October crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2018-43/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2018-43/segment.paths.gz	100
WARC files	CC-MAIN-2018-43/warc.paths.gz	56000	58.84
WAT files	CC-MAIN-2018-43/wat.paths.gz	56000	19.34
WET files	CC-MAIN-2018-43/wet.paths.gz	56000	8.22
Robots.txt files	CC-MAIN-2018-43/robotstxt.paths.gz	56000	0.21
Non-200 responses files	CC-MAIN-2018-43/non200responses.paths.gz	56000	1.78
URL index files	CC-MAIN-2018-43/cc-index.paths.gz	302	0.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-43/. Also the columnar index has been updated to contain this crawl.