2018 – Common Crawl

December 2018 crawl archive now available

The crawl archive for December 2018 is now available! It contains 3.1 billion web pages or 250 TiB of uncompressed content, crawled between December 9th and 19th.

The December crawl contains page captures of 735 million URLs not contained in any crawl archive before. New URLs stem from:

extracting and sampling URLs from sitemaps, RSS and Atom feeds if provided by hosts visited in prior crawls. Hosts are selected from the highest-ranking 60 million domains of the Aug/Sep/Oct 2018 webgraph data set
a breadth-first side crawl within a maximum of 6 links (“hops”) away from the home pages of the top 50 million domains of the webgraph dataset
a random sample of outlinks taken from WAT files of the November crawl
30 million external links sampled from Wikipedia data dumps

Archive Location and Download

The December crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2018-51/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2018-51/segment.paths.gz	100
WARC files	CC-MAIN-2018-51/warc.paths.gz	63840	65.31
WAT files	CC-MAIN-2018-51/wat.paths.gz	63840	20.01
WET files	CC-MAIN-2018-51/wet.paths.gz	63840	8.43
Robots.txt files	CC-MAIN-2018-51/robotstxt.paths.gz	63840	0.22
Non-200 responses files	CC-MAIN-2018-51/non200responses.paths.gz	63840	1.71
URL index files	CC-MAIN-2018-51/cc-index.paths.gz	302	0.24

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-51/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

November 2018 crawl archive now available

The crawl archive for November 2018 is now available! It contains 2.6 billion web pages or 220 TiB of uncompressed content, crawled between November 12th and 22nd.

The November crawl contains 640 million new URLs, not contained in any crawl archive before. New URLs stem from:

extracting and sampling URLs from sitemaps, RSS and Atom feeds if provided by hosts visited in prior crawls. Hosts are selected from the highest-ranking 60 million domains of the Aug/Sep/Oct 2018 webgraph data set
a breadth-first side crawl within a maximum of 10 links (“hops”) away from the home pages of the top 40 million domains of the webgraph dataset
a random sample of outlinks taken from WAT files of the October crawl
50 million external links sampled from Wikipedia data dumps

Archive Location and Download

The November crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2018-47/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2018-47/segment.paths.gz	100
WARC files	CC-MAIN-2018-47/warc.paths.gz	56000	54.16
WAT files	CC-MAIN-2018-47/wat.paths.gz	56000	17.36
WET files	CC-MAIN-2018-47/wet.paths.gz	56000	7.42
Robots.txt files	CC-MAIN-2018-47/robotstxt.paths.gz	56000	0.2
Non-200 responses files	CC-MAIN-2018-47/non200responses.paths.gz	56000	1.92
URL index files	CC-MAIN-2018-47/cc-index.paths.gz	302	0.2

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-47/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs Aug/Sep/Oct 2018

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of August, September and October 2018. Additional information about data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., the Feb/Mar/Apr 2017 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

Host-level graph

The graph consists of 903 million nodes and 5.25 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 819 million dangling nodes (91%) and the largest strongly connected component contains only 60 million (6.5%) nodes. The host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 903 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2018-aug-sep-oct/host/. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2018-aug-sep-oct/host/ as prefix to access the files from everywhere.

The following files and formats are provided:

Size	File	Description
5.66 GB	cc-main-2018-aug-sep-oct-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 42 vertices files
23.60 GB	cc-main-2018-aug-sep-oct-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 98 edges files

9.63 GB	cc-main-2018-aug-sep-oct-host.graph	graph in BVGraph format
2 kB	cc-main-2018-aug-sep-oct-host.properties

10.83 GB	cc-main-2018-aug-sep-oct-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2018-aug-sep-oct-host-t.properties

1 kB	cc-main-2018-aug-sep-oct-host.stats	WebGraph statistics

13.47 GB	cc-main-2018-aug-sep-oct-host-ranks.txt.gz	harmonic centrality and pagerank

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 87 million nodes and 1.48 billion edges. 56% or 49 million nodes are dangling nodes, the largest strongly connected component covers 33.5 million or 38% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2018-aug-sep-oct/domain/ resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2018-aug-sep-oct/domain/.

Download files of the Common Crawl Aug/Sep/Oct 2018 domain-level webgraph

Size	File	Description
0.60 GB	cc-main-2018-aug-sep-oct-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
5.95 GB	cc-main-2018-aug-sep-oct-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.24 GB	cc-main-2018-aug-sep-oct-domain.graph	graph in BVGraph format
2 kB	cc-main-2018-aug-sep-oct-domain.properties

3.39 GB	cc-main-2018-aug-sep-oct-domain-t.graph	transpose of the graph
2 kB	cc-main-2018-aug-sep-oct-domain-t.properties

1 kB	cc-main-2018-aug-sep-oct-domain.stats	WebGraph statistics

1.89 GB	cc-main-2018-aug-sep-oct-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 87 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Aug/Sept/Oct 2018)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	24993276	2	0.012750	com.facebook
2	24671056	1	0.017210	com.googleapis
3	23453366	3	0.010761	com.google
4	22371572	4	0.008252	com.twitter
5	22136836	5	0.006786	com.youtube
6	21115246	6	0.006404	org.w
7	19598338	9	0.003741	com.instagram
8	19499472	8	0.004616	org.gmpg
9	19210640	10	0.003396	com.linkedin
10	18557384	14	0.002133	com.wordpress
11	18475084	13	0.002729	org.wordpress
12	18409784	25	0.001383	org.wikipedia
13	18366100	24	0.001489	com.gravatar
14	18361154	21	0.001623	com.pinterest
15	17992020	12	0.002819	com.bootstrapcdn
16	17977192	19	0.001774	com.apple
17	17931956	32	0.001000	com.blogspot
18	17718492	41	0.000758	be.youtu
19	17629132	34	0.000918	gl.goo
20	17616074	26	0.001308	com.microsoft
21	17591076	16	0.001984	com.googletagmanager
22	17583026	37	0.000840	com.amazon
23	17567490	15	0.002013	com.cloudflare
24	17539566	46	0.000658	com.tumblr
25	17512444	23	0.001557	com.adobe
26	17478800	28	0.001256	com.vimeo
27	17351964	61	0.000482	com.yahoo
28	17328248	22	0.001618	com.macromedia
29	17262608	45	0.000669	com.wp
30	17247612	36	0.000867	com.paypal
31	17245380	20	0.001675	com.github
32	17243016	27	0.001272	com.gstatic
33	17237096	33	0.000928	com.amazonaws
34	17218638	47	0.000617	me.wp
35	17175034	52	0.000568	org.mozilla
36	17148704	98	0.000318	com.googleusercontent
37	17124438	40	0.000793	co.t
38	17114262	75	0.000425	com.weebly
39	17088226	115	0.000256	com.nytimes
40	17080324	39	0.000796	net.cloudfront
41	17058308	78	0.000424	org.creativecommons
42	17046746	51	0.000584	org.w3
43	17034446	150	0.000171	org.wikimedia
44	16978138	65	0.000465	com.medium
45	16977018	59	0.000490	com.flickr
46	16969818	60	0.000482	ly.bit
47	16934330	38	0.000830	io.github
48	16925564	139	0.000188	net.slideshare
49	16907918	149	0.000171	com.theguardian
50	16886648	71	0.000433	com.jquery
51	16848068	133	0.000196	com.imgur
52	16838070	165	0.000145	com.myspace
53	16825278	53	0.000551	eu.europa
54	16813968	159	0.000152	com.imdb
55	16800572	35	0.000899	net.fbcdn
56	16751537	127	0.000208	com.issuu
57	16731296	113	0.000267	org.apache
58	16730104	30	0.001101	net.doubleclick
59	16713084	188	0.000124	com.tinyurl
60	16690164	278	0.000085	com.theverge
61	16686224	105	0.000288	com.reddit
62	16681526	120	0.000229	com.yelp
63	16673945	17	0.001886	com.wixstatic
64	16666427	260	0.000091	com.appspot
65	16661248	274	0.000086	com.buzzfeed
66	16660287	143	0.000182	com.oracle
67	16653660	134	0.000191	com.spotify
68	16642850	241	0.000097	me.about
69	16641262	123	0.000216	com.android
70	16635142	335	0.000071	org.chromium
71	16619845	7	0.004633	com.godaddy
72	16619118	155	0.000164	com.tripadvisor
73	16611356	31	0.001041	com.squarespace
74	16609865	285	0.000083	com.mysql
75	16599943	308	0.000076	com.about
76	16593164	346	0.000067	org.arxiv
77	16592299	131	0.000204	org.ietf
78	16588645	95	0.000343	com.soundcloud
79	16583851	337	0.000071	edu.upenn
80	16581591	435	0.000057	edu.princeton
81	16574803	358	0.000066	org.ieee
82	16573729	156	0.000164	org.gnu
83	16567342	124	0.000215	com.dropbox
84	16567126	324	0.000073	com.deviantart
85	16543836	148	0.000174	com.forbes
86	16528334	130	0.000206	com.whatsapp
87	16521358	87	0.000400	com.statcounter
88	16521254	427	0.000058	google.blog
89	16520460	430	0.000057	com.ssrn
90	16519609	62	0.000481	org.schema
91	16516790	132	0.000199	org.archive
92	16511114	144	0.000182	net.sourceforge
93	16502431	180	0.000130	com.cnn
94	16499128	320	0.000074	gov.loc
95	16489333	191	0.000121	com.foursquare
96	16488380	189	0.000122	edu.stanford
97	16487861	88	0.000386	com.bing
98	16483708	371	0.000065	edu.ucla
99	16479852	389	0.000062	com.stackexchange
100	16478936	494	0.000050	edu.gatech
101	16478408	414	0.000059	org.sciencemag
102	16476180	183	0.000129	com.dribbble
103	16475471	203	0.000109	com.nbcnews
104	16470933	413	0.000059	com.withgoogle
105	16450054	255	0.000092	com.example
106	16447977	359	0.000066	com.googlecode
107	16443987	108	0.000278	com.ytimg
108	16438657	169	0.000142	uk.co.bbc
109	16436761	236	0.000098	edu.mit
110	16428294	230	0.000099	com.mozilla
111	16424352	224	0.000102	com.githubusercontent
112	16414627	458	0.000054	com.sap
113	16412253	565	0.000046	com.flipboard
114	16411958	205	0.000109	com.washingtonpost
115	16408402	146	0.000176	com.blogger
116	16406897	546	0.000047	com.chrome
117	16404143	49	0.000590	com.fb
118	16402707	571	0.000045	edu.utah
119	16398122	402	0.000060	com.jetbrains
120	16396163	492	0.000050	com.chron
121	16395449	322	0.000073	com.git-scm
122	16394655	186	0.000125	com.huffingtonpost
123	16393901	223	0.000102	com.businessinsider
124	16393651	97	0.000330	com.wix
125	16391015	89	0.000380	com.paypalobjects
126	16379847	122	0.000225	org.bbb
127	16379547	243	0.000097	com.live
128	16372670	287	0.000082	gov.fda
129	16372641	280	0.000084	au.com.google
130	16372405	54	0.000524	com.list-manage
131	16371002	286	0.000082	edu.harvard
132	16367016	532	0.000047	com.fastcodesign
133	16366146	365	0.000066	com.tinypic
134	16364937	194	0.000117	com.wsj
135	16347562	410	0.000059	tv.ustream
136	16344105	298	0.000080	com.cnet
137	16342765	334	0.000071	com.bbc
138	16339948	387	0.000062	com.variety
139	16339248	407	0.000060	org.eclipse
140	16338636	441	0.000056	co.g
141	16336819	304	0.000078	com.reuters
142	16333116	258	0.000091	org.doi
143	16326228	262	0.000090	com.ibm
144	16321163	266	0.000088	com.wired
145	16318957	312	0.000076	uk.co.telegraph
146	16317274	197	0.000112	com.typepad
147	16317262	349	0.000067	com.gmail
148	16316342	418	0.000058	org.iana
149	16309796	269	0.000087	com.bloomberg
150	16309363	248	0.000095	net.windows
151	16308394	104	0.000290	com.shopify
152	16304961	456	0.000054	co.ibb
153	16304891	181	0.000129	com.stackoverflow
154	16303280	240	0.000097	com.techcrunch
155	16297167	55	0.000519	net.akamaihd
156	16296675	272	0.000087	com.go
157	16296217	154	0.000166	gov.nih
158	16289519	394	0.000061	gov.nasa
159	16288695	339	0.000071	com.msn
160	16287525	352	0.000067	com.latimes
161	16285423	162	0.000147	com.etsy
162	16282310	109	0.000274	com.google-analytics
163	16282286	508	0.000049	edu.rutgers
164	16282232	545	0.000047	ca.utoronto
165	16275672	170	0.000142	com.twimg
166	16275582	103	0.000293	com.mailchimp
167	16274598	90	0.000378	de.google
168	16271240	265	0.000088	org.acm
169	16268707	366	0.000066	com.mashable
170	16267430	498	0.000050	com.quora
171	16264746	416	0.000058	au.gov.nsw
172	16264148	116	0.000242	com.jimdo
173	16261109	50	0.000589	com.fontawesome
174	16252668	550	0.000047	com.vogue
175	16251642	467	0.000053	com.zdnet
176	16250818	357	0.000067	uk.co.dailymail
177	16247730	663	0.000044	com.hbo
178	16247621	447	0.000055	com.googleblog
179	16245680	761	0.000039	com.dezeen
180	16244687	277	0.000085	com.usatoday
181	16244320	158	0.000162	com.eventbrite
182	16243163	582	0.000045	edu.osu
183	16239459	263	0.000090	com.meetup
184	16229462	429	0.000058	gov.archives
185	16224180	450	0.000055	edu.cornell
186	16223175	461	0.000053	edu.berkeley
187	16218763	396	0.000061	com.ted
188	16217936	151	0.000170	com.opera
189	16214980	581	0.000045	edu.washington
190	16211363	299	0.000080	com.udacity
191	16208363	580	0.000045	org.hrw
192	16197869	208	0.000107	com.surveymonkey
193	16195499	316	0.000075	com.time
194	16192449	486	0.000051	com.ecwid
195	16187438	409	0.000060	com.kickstarter
196	16187407	321	0.000074	org.npr
197	16187344	696	0.000042	com.discogs
198	16181108	700	0.000042	io.itch
199	16177807	496	0.000050	org.unicode
200	16177748	313	0.000076	com.springer
201	16176015	29	0.001149	ru.yandex
202	16174231	446	0.000055	org.kernel
203	16173199	370	0.000065	com.aol
204	16173059	701	0.000042	com.economist
205	16171404	290	0.000081	com.hp
206	16168983	231	0.000099	com.mapquest
207	16167485	48	0.000602	com.qq
208	16163785	758	0.000039	org.wikibooks
209	16160518	362	0.000066	com.cnbc
210	16154540	390	0.000062	org.un
211	16152769	333	0.000072	org.python
212	16152422	488	0.000051	com.ft
213	16151474	210	0.000107	org.drupal
214	16148981	401	0.000060	me.paypal
215	16148740	690	0.000042	com.strava
216	16148152	417	0.000058	com.angieslist
217	16144222	267	0.000088	com.hubspot
218	16142474	136	0.000191	com.zendesk
219	16141403	504	0.000049	org.aarp
220	16139927	364	0.000066	com.giphy
221	16138024	741	0.000040	org.amnesty
222	16136086	552	0.000046	com.yellowpages
223	16133616	343	0.000069	com.nypost
224	16132797	767	0.000038	com.wikia
225	16132223	714	0.000041	com.dropboxusercontent
226	16131615	419	0.000058	com.fortune
227	16128988	70	0.000439	net.jsfiddle
228	16128428	330	0.000072	com.wiley
229	16127117	91	0.000355	com.baidu
230	16126449	201	0.000110	uk.co.amazon
231	16124648	509	0.000049	com.unsplash
232	16123335	145	0.000179	uk.co.google
233	16122860	361	0.000066	com.prnewswire
234	16119093	821	0.000037	com.slate
235	16117800	482	0.000051	com.cisco
236	16114323	353	0.000067	com.photobucket
237	16112036	561	0.000046	com.venturebeat
238	16111166	873	0.000036	com.pixabay
239	16108436	976	0.000034	com.arstechnica
240	16105654	198	0.000111	org.purl
241	16102642	206	0.000108	com.ebay
242	16101242	798	0.000038	com.manta
243	16099223	137	0.000189	com.wixsite
244	16098318	702	0.000042	com.intel
245	16097598	685	0.000043	com.nationalgeographic
246	16096399	442	0.000056	com.entrepreneur
247	16090287	405	0.000060	gov.whitehouse
248	16090076	459	0.000054	com.nature
249	16089802	319	0.000074	com.oreilly
250	16088319	376	0.000064	com.office
251	16087148	576	0.000045	com.samsung
252	16084044	57	0.000494	com.vk
253	16082814	479	0.000052	com.matterport
254	16080553	475	0.000052	org.postgresql
255	16078046	601	0.000045	com.newyorker
256	16075255	297	0.000081	gov.cdc
257	16075119	173	0.000138	com.constantcontact
258	16073933	697	0.000042	com.vice
259	16073113	829	0.000037	edu.psu
260	16071402	1136	0.000031	com.gizmodo
261	16071402	551	0.000047	com.scribd
262	16070353	923	0.000035	com.qz
263	16068616	356	0.000067	org.ampproject
264	16068603	557	0.000046	gov.nist
265	16068356	294	0.000081	me.telegram
266	16067025	490	0.000051	com.wikihow
267	16066386	812	0.000037	ly.snip
268	16066142	220	0.000104	com.disqus
269	16065694	973	0.000034	edu.yale
270	16062922	474	0.000052	com.cbsnews
271	16062264	779	0.000038	edu.kit
272	16060241	607	0.000044	org.eff
273	16059786	583	0.000045	com.box
274	16059328	237	0.000098	net.php
275	16057917	126	0.000209	com.feedburner
276	16057179	476	0.000052	com.theatlantic
277	16055682	828	0.000037	com.engadget
278	16052013	264	0.000089	gov.ftc
279	16047507	791	0.000038	com.merchantcircle
280	16044901	252	0.000093	com.digg
281	16044141	448	0.000055	org.hbr
282	16042660	707	0.000041	org.nodejs
283	16042189	453	0.000055	com.inc
284	16040892	374	0.000064	com.images-amazon
285	16039745	379	0.000064	com.skype
286	16038801	212	0.000107	com.salesforce
287	16038291	716	0.000041	com.statista
288	16035364	1421	0.000027	edu.utexas
289	16034294	293	0.000081	com.staticflickr
290	16033600	291	0.000081	com.fastcompany
291	16033037	1439	0.000027	com.pexels
292	16030476	694	0.000042	edu.columbia
293	16028618	876	0.000036	com.marketwatch
294	16027153	600	0.000045	com.avvo
295	16024349	1436	0.000027	com.storify
296	16023986	340	0.000070	int.who
297	16023292	106	0.000284	com.addthis
298	16021320	775	0.000038	com.indiatimes
299	16016177	1445	0.000026	com.thinkwithgoogle
300	16015721	406	0.000060	org.maven
301	16014399	449	0.000055	com.w3schools
302	16013658	1408	0.000028	com.smashingmagazine
303	16012164	878	0.000036	com.mysanantonio
304	16011714	372	0.000064	co.elastic
305	16011705	215	0.000105	com.stumbleupon
306	16011229	226	0.000101	to.amzn
307	16008183	1492	0.000025	edu.purdue
308	16006545	195	0.000116	net.behance
309	16006038	560	0.000046	org.pbs
310	16005359	63	0.000476	me.fb
311	16003375	302	0.000079	com.googlesyndication
312	16002994	969	0.000034	au.net.abc
313	16002951	1601	0.000022	com.vanityfair
314	16002791	499	0.000050	com.slack
315	16001492	270	0.000087	gov.ca
316	15999972	311	0.000076	com.tripod
317	15995276	338	0.000071	com.sxsw
318	15993036	408	0.000060	uk.co.blogspot
319	15990251	141	0.000185	com.weibo
320	15987681	684	0.000043	net.researchgate
321	15985567	1415	0.000027	com.alexa
322	15984335	355	0.000067	com.dailymotion
323	15982835	1401	0.000028	edu.ucsd
324	15982138	686	0.000042	com.blackberry
325	15981891	1014	0.000033	org.worldbank
326	15979100	315	0.000075	fr.free
327	15978004	472	0.000052	net.leadpages
328	15976584	974	0.000034	com.thenextweb
329	15971688	554	0.000046	com.moz
330	15970463	1699	0.000020	org.owasp
331	15969404	360	0.000066	com.sciencedirect
332	15968776	762	0.000039	com.uservoice
333	15968498	1003	0.000033	com.shutterstock
334	15965774	375	0.000064	edu.cmu
335	15962210	176	0.000134	org.icann
336	15961179	732	0.000040	com.proofpoint
337	15958400	903	0.000035	edu.uark
338	15957323	1082	0.000032	com.evernote
339	15956890	373	0.000064	com.livejournal
340	15955233	691	0.000042	com.googlesource
341	15951480	1006	0.000033	ly.ow
342	15949779	589	0.000045	gov.sec
343	15946301	955	0.000034	com.speakerdeck
344	15944949	1351	0.000029	com.lifehacker
345	15941679	584	0.000045	com.citysearch
346	15941304	879	0.000035	org.unesco
347	15940525	814	0.000037	com.psychologytoday
348	15937813	1319	0.000031	com.trello
349	15937606	913	0.000035	com.sfgate
350	15936285	994	0.000033	com.designobserver
351	15934103	1536	0.000024	edu.northwestern
352	15933505	457	0.000054	com.snapchat
353	15932071	1320	0.000031	uk.ac.ox
354	15931696	671	0.000043	tv.twitch
355	15931551	1021	0.000032	gov.fcc
356	15930923	678	0.000043	org.bitbucket
357	15929820	1778	0.000019	com.fifa
358	15929724	412	0.000059	com.businesswire
359	15928928	803	0.000037	org.aiga
360	15926617	244	0.000096	com.wufoo
361	15926593	569	0.000045	com.atlassian
362	15926203	214	0.000106	de.amazon
363	15925496	327	0.000072	com.typeform
364	15924766	1603	0.000022	com.mcafee
365	15922998	1047	0.000032	com.libsyn
366	15922233	1878	0.000017	org.coursera
367	15921752	916	0.000035	com.zynga
368	15921549	961	0.000034	com.kudzu
369	15921529	1852	0.000018	com.semrush
370	15920541	674	0.000043	com.ubuntu
371	15920148	1673	0.000021	com.econsultancy
372	15918200	1440	0.000027	com.indiegogo
373	15917275	1383	0.000028	com.politico
374	15917261	295	0.000081	org.mediawiki
375	15916739	754	0.000039	org.aclweb
376	15916639	963	0.000034	com.deloitte
377	15914854	930	0.000034	org.spie
378	15914760	981	0.000033	com.livestream
379	15912203	1449	0.000026	co.vine
380	15910119	1553	0.000023	org.khanacademy
381	15908826	516	0.000048	com.goodreads
382	15908339	989	0.000033	gov.uspto
383	15907762	303	0.000079	org.joomla
384	15906992	1398	0.000028	com.zoho
385	15902686	908	0.000035	me.websta
386	15901516	825	0.000037	com.foxnews
387	15900302	350	0.000067	com.booking
388	15899160	1013	0.000033	io.codepen
389	15898382	129	0.000206	com.youtube-nocookie
390	15897026	163	0.000146	jp.co.yahoo
391	15896378	1613	0.000022	edu.unc
392	15896141	1652	0.000021	com.technologyreview
393	15894277	1709	0.000020	com.digitaltrends
394	15893383	1217	0.000031	org.iso
395	15893229	1569	0.000023	com.pingdom
396	15893117	914	0.000035	gov.senate
397	15892860	289	0.000082	com.smugmug
398	15890595	199	0.000111	com.bandcamp
399	15889383	975	0.000034	com.mckinsey
400	15888601	920	0.000035	it.binged
401	15888102	1389	0.000028	com.udemy
402	15885995	991	0.000033	com.what3words
403	15885247	2561	0.000012	com.sophos
404	15884644	1622	0.000022	org.weforum
405	15884560	380	0.000064	net.themeforest
406	15884272	626	0.000044	gov.noaa
407	15882777	1896	0.000017	com.ehow
408	15881215	718	0.000041	org.vim
409	15879594	1490	0.000025	com.elpais
410	15879197	1343	0.000030	com.sciencedaily
411	15879074	445	0.000056	com.squareup
412	15879052	816	0.000037	com.gartner
413	15877064	439	0.000056	com.netflix
414	15873982	470	0.000053	com.webs
415	15873498	271	0.000087	com.rawgit
416	15873485	1035	0.000032	edu.uah
417	15873257	1710	0.000020	uk.co.wired
418	15871503	463	0.000053	com.bizjournals
419	15871161	1002	0.000033	com.americanexpress
420	15870167	1568	0.000023	org.pnas
421	15869151	433	0.000057	com.monster
422	15869106	1024	0.000032	com.nielsen
423	15866670	1317	0.000031	com.redhat
424	15866441	667	0.000044	com.java
425	15865156	76	0.000425	org.reactjs
426	15864692	1949	0.000017	ch.ethz
427	15862191	400	0.000060	com.force
428	15861930	404	0.000060	com.herokuapp
429	15861377	1798	0.000019	com.socialmediaexaminer
430	15860803	1108	0.000031	com.adage
431	15860212	892	0.000035	com.googledrive
432	15859653	1899	0.000017	com.tutsplus
433	15858284	114	0.000263	jp.co.google
434	15857073	1696	0.000020	edu.usc
435	15856570	984	0.000033	com.prweb
436	15856191	760	0.000039	gov.justice
437	15855748	1481	0.000025	com.playstation
438	15855432	1734	0.000020	com.canva
439	15854961	514	0.000049	us.icio
440	15852813	172	0.000138	com.xing
441	15850197	866	0.000036	re.cli
442	15850114	1572	0.000023	edu.uchicago
443	15849567	1400	0.000028	com.bostonglobe
444	15848723	801	0.000038	com.steampowered
445	15844441	292	0.000081	ca.google
446	15844259	437	0.000057	com.bigcartel
447	15843423	2150	0.000015	com.urbandictionary
448	15842728	844	0.000036	io.material
449	15841425	481	0.000051	com.bigcommerce
450	15839196	1540	0.000024	com.caniuse
451	15838309	245	0.000096	com.getclicky
452	15834448	1384	0.000028	com.dell
453	15834375	808	0.000037	gov.state
454	15834214	1732	0.000020	com.hotmail
455	15833672	250	0.000094	es.google
456	15831338	1692	0.000021	au.com.smh
457	15830767	1632	0.000022	com.upwork
458	15830199	737	0.000040	org.gnupg
459	15829812	998	0.000033	edu.utep
460	15829095	354	0.000067	com.stripe
461	15828852	901	0.000035	com.msdn
462	15828183	422	0.000058	com.adweek
463	15826915	1701	0.000020	com.codeplex
464	15826257	2005	0.000016	ca.uwaterloo
465	15825896	107	0.000283	org.networkadvertising
466	15824996	2475	0.000013	com.twitpic
467	15823630	1375	0.000029	uk.ac.cam
468	15823242	225	0.000101	com.myshopify
469	15823088	1752	0.000019	com.nike
470	15822418	845	0.000036	com.outlook
471	15822314	1498	0.000025	com.gettyimages
472	15821233	1341	0.000030	com.istockphoto
473	15820921	1189	0.000031	de.heise
474	15819474	1600	0.000022	com.marketo
475	15818475	520	0.000048	com.cargocollective
476	15818334	1368	0.000029	ca.blogspot
477	15817231	1990	0.000016	com.norton
478	15815744	1459	0.000026	de.spiegel
479	15814626	846	0.000036	jp.co.fujixerox
480	15813630	997	0.000033	com.chicagotribune
481	15812887	1807	0.000018	com.ikea
482	15812477	1550	0.000023	com.ning
483	15812254	2052	0.000016	com.crunchbase
484	15811034	699	0.000042	com.webmd
485	15808826	202	0.000110	com.windowsphone
486	15808375	1521	0.000024	com.scientificamerican
487	15808267	239	0.000097	com.getbootstrap
488	15808212	2567	0.000012	com.codecademy
489	15807983	1099	0.000031	edu.alamo
490	15807399	507	0.000049	com.npmjs
491	15806866	1585	0.000022	com.billboard
492	15806166	1052	0.000032	com.theschooloflife
493	15805533	2014	0.000016	com.msnbc
494	15804337	2303	0.000014	com.instructables
495	15803920	725	0.000040	gov.copyright
496	15803640	1530	0.000024	uk.ac.ucl
497	15803572	1676	0.000021	fr.lemonde
498	15802334	925	0.000034	edu.umich
499	15800853	1378	0.000028	edu.wisc
500	15800747	140	0.000188	ru.mail
501	15800371	2358	0.000013	com.starwars
502	15797878	865	0.000036	de.blogspot
503	15797791	1508	0.000024	com.kissmetrics
504	15797047	1115	0.000031	com.beautifulpixels
505	15796947	1386	0.000028	com.airbnb
506	15796853	2451	0.000013	edu.hbs
507	15796325	166	0.000145	com.eepurl
508	15795637	768	0.000038	com.css-tricks
509	15795363	233	0.000098	com.bitly
510	15794822	1615	0.000022	edu.jhu
511	15793691	1362	0.000029	com.alibaba
512	15792654	1164	0.000031	com.sun
513	15792271	772	0.000038	com.tandfonline
514	15791593	893	0.000035	com.underconsideration
515	15790633	518	0.000048	in.co.google
516	15789087	793	0.000038	com.uber
517	15788507	704	0.000042	com.photoshelter
518	15787332	566	0.000046	com.symantec
519	15787196	2936	0.000010	uk.bl
520	15786855	683	0.000043	gov.hhs
521	15783209	807	0.000037	io.getmdl
522	15782831	1691	0.000021	com.irishtimes
523	15781874	2154	0.000015	edu.ncsu
524	15781333	1393	0.000028	com.searchenginejournal
525	15781203	67	0.000448	com.messenger
526	15780051	517	0.000048	org.sonatype
527	15778677	979	0.000033	ca.cbc
528	15778587	1592	0.000022	com.yandex
529	15777890	431	0.000057	com.clicky
530	15777427	1768	0.000019	com.hulu
531	15776625	1443	0.000026	com.accenture
532	15774420	1610	0.000022	edu.academia
533	15773314	528	0.000047	gov.epa
534	15772833	1403	0.000028	com.marketingland
535	15772472	972	0.000034	uk.co.guardian
536	15771759	2040	0.000016	tv.periscope
537	15769999	1616	0.000022	com.today
538	15768447	2453	0.000013	ly.visual
539	15766945	369	0.000065	edu.nyu
540	15766843	1409	0.000028	org.apa
541	15766561	2894	0.000011	com.girlswhocode
542	15766409	1546	0.000024	com.hollywoodreporter
543	15765504	549	0.000047	uk.co.independent
544	15764791	2378	0.000013	com.glamour
545	15764760	2131	0.000015	au.com.news
546	15764489	553	0.000046	gov.ed
547	15763374	1714	0.000020	com.invisionapp
548	15763321	2450	0.000013	org.gimp
549	15763104	709	0.000041	com.feedly
550	15763001	1322	0.000031	org.change
551	15761645	2072	0.000015	com.ibtimes
552	15761255	1598	0.000022	com.thomsonreuters
553	15760281	1517	0.000024	gov.nyc
554	15760200	1826	0.000018	com.posterous
555	15759019	943	0.000034	com.bravesites
556	15758123	3649	0.000008	com.space
557	15758105	1434	0.000027	gov.bls
558	15756979	443	0.000056	cn.com.sina
559	15756712	397	0.000061	com.custhelp
560	15755071	2389	0.000013	com.tesla
561	15753906	1476	0.000025	com.businessweek
562	15753434	774	0.000038	com.uk
563	15753186	1782	0.000019	com.zillow
564	15752235	1814	0.000018	com.zapier
565	15751997	2583	0.000012	com.dreamstime
566	15751546	3003	0.000010	com.klout
567	15750992	1623	0.000022	com.thehill
568	15750722	234	0.000098	com.wpengine
569	15750076	2978	0.000010	com.rottentomatoes
570	15749795	2693	0.000012	com.campaignmonitor
571	15749130	1629	0.000022	uk.ac.ed
572	15748626	2246	0.000014	com.wikidot
573	15748387	2252	0.000014	com.123rf
574	15748038	217	0.000105	fr.google
575	15747873	1611	0.000022	com.intuit
576	15747479	1641	0.000021	org.letsencrypt
577	15746782	875	0.000036	com.questionpro
578	15744807	664	0.000044	com.gotowebinar
579	15744526	1987	0.000016	com.nokia
580	15742939	2658	0.000012	edu.brown
581	15742494	3600	0.000008	com.formula1
582	15742364	2184	0.000014	com.mentalfloss
583	15742342	451	0.000055	gov.irs
584	15742266	491	0.000050	net.openid
585	15740664	1688	0.000021	com.nba
586	15739222	1593	0.000022	org.pewresearch
587	15738724	2222	0.000014	com.aljazeera
588	15738356	1058	0.000032	com.ezlocal
589	15737381	1437	0.000027	org.altervista
590	15737002	1478	0.000025	in.blogspot
591	15736320	279	0.000084	it.placehold
592	15735548	3233	0.000009	edu.uic
593	15735280	2251	0.000014	com.programmableweb
594	15735247	2153	0.000015	com.cbs
595	15734807	1153	0.000031	gov.sba
596	15734218	2198	0.000014	com.techradar
597	15734158	826	0.000037	gov.census
598	15733513	1747	0.000019	org.postimg
599	15732878	506	0.000049	gov.usda
600	15732533	1535	0.000024	com.target
601	15731343	721	0.000041	com.docker
602	15731122	1519	0.000024	com.gigaom
603	15731013	2800	0.000011	com.oxforddictionaries
604	15728172	1693	0.000021	net.daum
605	15727989	962	0.000034	com.gofundme
606	15727980	1639	0.000022	kr.flic
607	15726681	1156	0.000031	com.formstack
608	15726256	763	0.000039	org.sqlite
609	15725436	1661	0.000021	com.autodesk
610	15724812	1396	0.000028	com.techrepublic
611	15724806	817	0.000037	com.patreon
612	15721240	970	0.000034	com.insiderpages
613	15721227	1795	0.000019	com.us
614	15720198	1031	0.000032	com.hotfrog
615	15720166	966	0.000034	com.whitepages
616	15719605	1900	0.000017	edu.illinois
617	15719578	1642	0.000021	com.pwc
618	15718460	2039	0.000016	edu.asu
619	15718391	2486	0.000013	com.animoto
620	15717320	249	0.000094	com.fc2
621	15717052	1825	0.000018	org.rubyonrails
622	15716625	742	0.000040	com.wunderground
623	15716037	213	0.000106	org.debian
624	15715960	1124	0.000031	org.cmlibrary
625	15715829	1062	0.000032	com.idt
626	15715705	1359	0.000029	com.investopedia
627	15715452	1856	0.000018	com.howstuffworks
628	15714753	1326	0.000030	org.redcross
629	15714617	1493	0.000025	com.indeed
630	15713762	2101	0.000015	com.lonelyplanet
631	15713705	2054	0.000016	com.gamespot
632	15713431	910	0.000035	gov.nps
633	15713159	1084	0.000032	com.thesprintbook
634	15712729	1141	0.000031	com.smartguy
635	15711951	832	0.000037	com.att
636	15711660	2049	0.000016	com.refinery29
637	15709292	522	0.000048	com.vendio
638	15709144	2851	0.000011	com.domaintools
639	15708842	874	0.000036	com.itsnicethat
640	15707939	1801	0.000018	org.filezilla-project
641	15707760	1395	0.000028	com.vmware
642	15707005	171	0.000139	it.google
643	15706361	3994	0.000007	com.boredpanda
644	15705364	1391	0.000028	gov.va
645	15705335	849	0.000036	com.pinimg
646	15704965	1428	0.000027	com.reverbnation
647	15704604	2016	0.000016	ca.ubc
648	15704346	1995	0.000016	com.nfl
649	15703768	666	0.000044	com.houzz
650	15703700	1516	0.000024	com.prezi
651	15703074	1912	0.000017	edu.indiana
652	15702174	3049	0.000010	com.hubpages
653	15701522	436	0.000057	com.nasdaq
654	15701359	2734	0.000011	com.9to5mac
655	15701239	1579	0.000023	com.pcworld
656	15700785	1824	0.000018	edu.ucdavis
657	15700731	1416	0.000027	gov.usgs
658	15700075	886	0.000035	com.500px
659	15699652	1001	0.000033	com.acninc
660	15699448	2157	0.000015	com.livestrong
661	15699048	1328	0.000030	org.oecd
662	15698519	2267	0.000014	com.newscientist
663	15697206	1846	0.000018	com.espn
664	15697101	1484	0.000025	edu.umn
665	15697074	1703	0.000020	com.freepik
666	15696322	1902	0.000017	edu.virginia
667	15694887	1605	0.000022	com.vox
668	15694606	1858	0.000018	com.deadline
669	15693525	483	0.000051	org.whatbrowser
670	15692991	1499	0.000025	com.mixcloud
671	15691728	847	0.000036	com.emarketer
672	15691597	1360	0.000029	fr.blogspot
673	15691539	695	0.000042	com.flippa
674	15691203	256	0.000092	com.elegantthemes
675	15690356	1590	0.000022	com.newsweek
676	15689675	2170	0.000015	com.getresponse
677	15688589	460	0.000054	io.atom
678	15688584	1700	0.000020	com.gallup
679	15688318	2187	0.000014	edu.bu
680	15687369	2815	0.000011	org.moma
681	15686394	1888	0.000017	com.findlaw
682	15683775	1542	0.000024	edu.si
683	15683516	2094	0.000015	com.pastebin
684	15682917	1155	0.000031	dk.fcm
685	15682640	1547	0.000024	com.globo
686	15682617	368	0.000065	org.openstreetmap
687	15681889	1142	0.000031	org.writersleague
688	15680577	1884	0.000017	edu.cuny
689	15680551	1925	0.000017	com.starbucks
690	15680465	1447	0.000026	com.warnerbros
691	15679238	2075	0.000015	com.socialmediatoday
692	15678966	1150	0.000031	com.prosperent
693	15678673	1114	0.000031	org.grayarea
694	15678448	1984	0.000016	org.aclu
695	15677879	739	0.000040	org.jenkins-ci
696	15674592	2002	0.000016	com.mercurynews
697	15674443	1552	0.000023	com.business2community
698	15674402	1836	0.000018	mp.j
699	15674363	4368	0.000007	com.petapixel
700	15673782	2630	0.000012	com.googlepages
701	15673492	1894	0.000017	com.hostgator
702	15673279	745	0.000039	com.geocities
703	15672825	1336	0.000030	org.mayoclinic
704	15672261	167	0.000143	gov.privacyshield
705	15671234	1049	0.000032	com.ycombinator
706	15670571	1496	0.000025	net.java
707	15670017	1463	0.000026	us.imageshack
708	15669905	2432	0.000013	com.psychcentral
709	15669061	1624	0.000022	com.boston
710	15667800	1509	0.000024	org.fao
711	15666438	1980	0.000016	edu.arizona
712	15665639	1581	0.000023	com.nydailynews
713	15665192	1832	0.000018	de.welt
714	15665099	238	0.000098	com.youku
715	15664693	1915	0.000017	com.salon
716	15664554	2365	0.000013	edu.gmu
717	15663687	1017	0.000032	com.aweber
718	15663661	242	0.000097	jp.co.amazon
719	15663656	2299	0.000014	com.yourdomain
720	15662150	2021	0.000016	com.domain
721	15661511	2285	0.000014	com.ew
722	15659706	1149	0.000031	com.collegian
723	15659043	796	0.000038	org.elasticsearch
724	15658487	1380	0.000028	com.mlb
725	15658455	899	0.000035	com.delicious
726	15658257	2239	0.000014	ca.ualberta
727	15657830	3265	0.000009	org.edx
728	15655920	988	0.000033	google.design
729	15655666	2776	0.000011	org.kiva
730	15654526	1410	0.000028	com.weather
731	15654299	1837	0.000018	net.codecanyon
732	15654288	2743	0.000011	com.lynda
733	15654205	1503	0.000024	com.merriam-webster
734	15654147	1042	0.000032	com.womentechmakers
735	15654091	1065	0.000032	net.brownbook
736	15653789	986	0.000033	com.hootsuite
737	15653450	3979	0.000007	com.lmgtfy
738	15652070	426	0.000058	com.ea
739	15651918	1779	0.000019	edu.umd
740	15651858	1497	0.000025	com.thedrum
741	15650961	1735	0.000020	com.aliexpress
742	15650884	204	0.000109	com.automattic
743	15650243	1514	0.000024	int.coe
744	15650019	2247	0.000014	org.openoffice
745	15649988	1658	0.000021	com.firefox
746	15649772	1599	0.000022	com.searchenginewatch
747	15649620	1853	0.000018	com.zazzle
748	15648960	2027	0.000016	com.gq
749	15648656	1574	0.000023	org.cambridge
750	15648651	1904	0.000017	edu.msu
751	15647741	444	0.000056	com.barnesandnoble
752	15647388	2149	0.000015	com.azcentral
753	15647181	2429	0.000013	edu.wustl
754	15647110	2554	0.000012	org.semanticscholar
755	15646960	1887	0.000017	edu.umass
756	15646650	1555	0.000023	fm.last
757	15646167	2060	0.000016	au.com.blogspot
758	15645607	1191	0.000031	site.tenerifeforum
759	15645549	3140	0.000010	com.copyblogger
760	15645390	1102	0.000031	uk.gov.peterborough
761	15644870	2289	0.000014	com.topsy
762	15644590	897	0.000035	com.unity3d
763	15644472	1628	0.000022	com.over-blog
764	15643911	1501	0.000025	com.waze
765	15642164	2320	0.000014	com.gawker
766	15642103	2466	0.000013	ms.1drv
767	15641828	1370	0.000029	com.timeanddate
768	15641339	3477	0.000009	com.answers
769	15641169	1325	0.000030	com.arcgis
770	15640859	794	0.000038	com.clkmg
771	15639069	1080	0.000032	com.cbslocal
772	15638930	2576	0.000012	org.phys
773	15638348	1695	0.000021	com.stitcher
774	15637681	1663	0.000021	com.gumroad
775	15637217	1366	0.000029	gov.fbi
776	15637050	2334	0.000013	com.fiverr
777	15636227	1800	0.000019	com.lulu
778	15635676	1671	0.000021	com.rollingstone
779	15635417	1880	0.000017	com.nvidia
780	15635094	2702	0.000012	com.headspace
781	15634767	341	0.000070	org.opensource
782	15634400	1684	0.000021	com.neilpatel
783	15633999	1771	0.000019	uk.co.metro
784	15633208	990	0.000033	jp.ac.kobe-u
785	15632996	1741	0.000020	com.mtv
786	15632518	56	0.000499	net.facebook
787	15632516	2649	0.000012	edu.tufts
788	15632419	915	0.000035	br.com.uol
789	15631894	2562	0.000012	com.fox
790	15631850	1057	0.000032	com.brightcove
791	15631313	1505	0.000024	com.sky
792	15630776	2933	0.000010	com.popsci
793	15630301	3389	0.000009	com.wolfram
794	15627572	2434	0.000013	com.theonion
795	15627567	1348	0.000029	org.readthedocs
796	15627335	1927	0.000017	com.trendmicro
797	15626654	388	0.000062	com.marriott
798	15626569	342	0.000070	nl.google
799	15626305	2721	0.000011	edu.caltech
800	15625896	1060	0.000032	com.2findlocal
801	15625270	1472	0.000025	uk.co.theregister
802	15625157	891	0.000035	uk.co.eventbrite
803	15625156	1121	0.000031	com.fotolia
804	15624596	1849	0.000018	com.history
805	15624274	306	0.000077	com.naver
806	15623770	2985	0.000010	edu.dartmouth
807	15623624	1606	0.000022	com.bmj
808	15623028	2515	0.000012	ch.cern
809	15622919	1914	0.000017	it.scoop
810	15621936	1357	0.000029	com.walmart
811	15621746	1930	0.000017	org.kde
812	15621344	1898	0.000017	com.nrf
813	15619330	1649	0.000021	im.gitter
814	15619286	2379	0.000013	com.bestbuy
815	15619283	473	0.000052	com.iconfinder
816	15618356	1866	0.000018	org.jstor
817	15618109	1377	0.000028	com.searchengineland
818	15616272	184	0.000128	jp.ne.hatena
819	15615800	1543	0.000024	com.splashthat
820	15614563	3110	0.000010	org.notepad-plus-plus
821	15614110	1627	0.000022	com.com
822	15613738	1529	0.000024	org.heart
823	15612896	2529	0.000012	edu.uiuc
824	15612666	2730	0.000011	com.fitbit
825	15611859	1026	0.000032	com.company
826	15610954	2489	0.000012	com.wikispaces
827	15610875	1541	0.000024	com.cafepress
828	15610542	1738	0.000020	com.ssllabs
829	15610139	2352	0.000013	de.bild
830	15608795	69	0.000447	com.parallels
831	15608630	917	0.000035	gov.usa
832	15608624	1806	0.000018	com.buffer
833	15608543	1966	0.000016	com.discordapp
834	15607778	1206	0.000031	com.infusionsoft
835	15607523	2031	0.000016	edu.uci
836	15607224	838	0.000036	org.openweathermap
837	15606632	3159	0.000010	gd.is
838	15605502	182	0.000129	jp.ameblo
839	15604837	900	0.000035	com.cdbaby
840	15604598	1000	0.000033	com.newsbank
841	15604393	1815	0.000018	com.deezer
842	15603804	1822	0.000018	com.discovery
843	15602784	765	0.000038	org.doxygen
844	15602226	1030	0.000032	org.travelblog
845	15602213	1034	0.000032	org.tpr
846	15601034	428	0.000058	net.launchpad
847	15600189	777	0.000038	com.sagepub
848	15598493	1059	0.000032	com.chamberofcommerce
849	15598065	510	0.000049	com.cracked
850	15597590	749	0.000039	org.plos
851	15597251	4943	0.000006	com.checkpoint
852	15597031	1936	0.000017	uk.co.thesun
853	15597000	99	0.000302	com.namecheap
854	15596654	3162	0.000009	com.spreaker
855	15596199	1533	0.000024	com.xkcd
856	15593759	1331	0.000030	com.tableau
857	15593642	1488	0.000025	com.pcmag
858	15593497	1934	0.000017	edu.ufl
859	15591634	3453	0.000009	edu.buffalo
860	15591344	2771	0.000011	com.producthunt
861	15591279	3424	0.000009	org.lifehack
862	15591113	1977	0.000016	com.examiner
863	15591022	1073	0.000032	net.azurewebsites
864	15590091	2360	0.000013	com.bleacherreport
865	15589566	1016	0.000033	com.bizcommunity
866	15589420	996	0.000033	com.chambermaster
867	15589294	1147	0.000031	com.oup
868	15589126	1889	0.000017	com.thedailybeast
869	15588805	2640	0.000012	com.snopes
870	15588137	2084	0.000015	com.ign
871	15588059	3592	0.000008	com.appleinsider
872	15587571	1096	0.000031	com.lookuppage
873	15587541	2059	0.000016	com.mac
874	15587446	746	0.000039	com.usnews
875	15586928	669	0.000043	com.163
876	15586259	2966	0.000010	org.greenpeace
877	15586114	3672	0.000008	edu.temple
878	15586108	919	0.000035	com.tiddlywiki
879	15585180	1993	0.000016	de.zeit
880	15584327	1660	0.000021	com.strikingly
881	15584124	1854	0.000018	co.angel
882	15583419	2237	0.000014	com.yolasite
883	15583266	541	0.000047	com.1and1
884	15583228	1650	0.000021	com.windows
885	15583160	2454	0.000013	net.comcast
886	15582183	4541	0.000007	com.blog
887	15581911	1350	0.000029	com.shareasale
888	15581584	1103	0.000031	com.spoke
889	15580712	2676	0.000012	com.macrumors
890	15580235	2106	0.000015	com.si
891	15580055	2947	0.000010	com.avast
892	15579731	1104	0.000031	com.communitywalk
893	15579534	1039	0.000032	com.independent
894	15579484	1828	0.000018	it.blogspot
895	15578882	2235	0.000014	com.icloud
896	15578813	2227	0.000014	ca.sfu
897	15578331	1759	0.000019	edu.duke
898	15578134	1406	0.000028	gov.ny
899	15578058	3048	0.000010	edu.ucsc
900	15577482	1781	0.000019	com.lithium
901	15577447	3081	0.000010	com.marieclaire
902	15577247	890	0.000035	com.mariadb
903	15576962	3202	0.000009	com.brainyquote
904	15576768	2598	0.000012	ca.globalnews
905	15576034	2895	0.000011	edu.oregonstate
906	15575967	738	0.000040	es.com.blogspot
907	15575607	681	0.000043	fr.amazon
908	15575040	2570	0.000012	com.nintendo
909	15574989	153	0.000166	de.bund
910	15574629	2078	0.000015	com.popsugar
911	15574040	1116	0.000031	com.lacartes
912	15573641	1929	0.000017	com.angelfire
913	15573588	2067	0.000015	org.poynter
914	15573573	1071	0.000032	com.citysquares
915	15573417	2202	0.000014	com.movember
916	15573266	1882	0.000017	uk.ac.lse
917	15573017	1045	0.000032	com.thegreatdiscontent
918	15572917	2098	0.000015	org.wpmudev
919	15572164	2522	0.000012	com.fineartamerica
920	15572083	2493	0.000012	edu.vt
921	15571965	2833	0.000011	edu.hawaii
922	15571730	2171	0.000015	com.teenvogue
923	15571641	1531	0.000024	com.calendly
924	15571635	1558	0.000023	com.steamcommunity
925	15571429	3026	0.000010	org.thinkprogress
926	15571425	1435	0.000027	com.techtarget
927	15571248	2045	0.000016	com.blogtalkradio
928	15571188	687	0.000042	uk.co.tripadvisor
929	15571109	1526	0.000024	com.glassdoor
930	15570512	1544	0.000024	com.xbox
931	15570312	1381	0.000028	me.m
932	15569885	2218	0.000014	uk.co.express
933	15569288	1675	0.000021	uk.co.mirror
934	15568503	119	0.000232	info.aboutads
935	15568393	2508	0.000012	com.blogs
936	15568027	2115	0.000015	com.templatemonster
937	15567578	328	0.000072	com.netdna-ssl
938	15567289	1371	0.000029	gov.dol
939	15567230	1554	0.000023	org.unicef
940	15567188	1135	0.000031	com.netdna-cdn
941	15566390	411	0.000059	com.mapbox
942	15566061	1088	0.000032	com.americantowns
943	15565930	2456	0.000013	org.7-zip
944	15565290	3852	0.000008	com.thenation
945	15564772	853	0.000036	ca.amazon
946	15564624	4817	0.000006	com.depositphotos
947	15564555	2687	0.000012	edu.pitt
948	15564538	2603	0.000012	nl.uva
949	15564284	2111	0.000015	sg.com.google
950	15564236	1020	0.000032	com.galvanize
951	15563763	1043	0.000032	com.judysbook
952	15563330	1086	0.000032	org.twinery
953	15563027	1756	0.000019	com.timeout
954	15562787	1697	0.000020	com.mediafire
955	15561709	2086	0.000015	com.w3techs
956	15561676	1373	0.000029	com.ups
957	15561291	945	0.000034	gov.house
958	15560994	855	0.000036	io.pantheon
959	15560477	2395	0.000013	com.me
960	15559169	3207	0.000009	cc.tiny
961	15559098	1958	0.000016	com.apnews
962	15557883	3248	0.000009	org.code
963	15557727	537	0.000047	com.getpocket
964	15557527	672	0.000043	com.elsevier
965	15556848	729	0.000040	com.prestashop
966	15556833	2388	0.000013	com.homedepot
967	15556798	1446	0.000026	com.bufferapp
968	15556767	3416	0.000009	com.virustotal
969	15556551	1694	0.000021	com.outbrain
970	15556036	5022	0.000006	com.wechat
971	15555722	2340	0.000013	com.pandora
972	15555713	2301	0.000014	com.foxmovies
973	15555616	4140	0.000007	com.kpcb
974	15555485	2860	0.000011	com.lanyrd
975	15555314	724	0.000041	com.redbubble
976	15553924	3262	0.000009	org.catalyst
977	15553307	2096	0.000015	tech.ces
978	15553284	1591	0.000022	gov.wa
979	15553160	1477	0.000025	jp.blogspot
980	15552948	2110	0.000015	com.twilio
981	15552738	477	0.000052	mp.mailchi
982	15552671	3581	0.000008	com.biography
983	15552065	1932	0.000017	com.healthline
984	15551576	1085	0.000032	com.pacegallery
985	15551541	3002	0.000010	com.iconosquare
986	15550836	2000	0.000016	com.baltimoresun
987	15550166	2803	0.000011	com.imageshack
988	15549943	1945	0.000017	gov.uscourts
989	15549663	2875	0.000011	int.esa
990	15549147	2731	0.000011	com.virgin
991	15548392	4588	0.000006	com.diigo
992	15548390	1810	0.000018	com.people
993	15548386	1473	0.000025	se.haxx
994	15547759	1706	0.000020	com.visualstudio
995	15547737	3064	0.000010	com.freelancer
996	15547640	2307	0.000014	com.xerox
997	15547505	512	0.000049	com.myportfolio
998	15547438	1364	0.000029	es.amazon
999	15546424	3356	0.000009	com.complex
1000	15545977	469	0.000053	br.com.google

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

October 2018 crawl archive now available

The crawl archive for October 2018 is now available! It contains 3.0 billion web pages and 240 TiB of uncompressed content, crawled between October 15th and 24th.

The October crawl contains 600 million new URLs, not contained in any crawl archive before. New URLs stem from:

extracting and sampling URLs from sitemaps, RSS and Atom feeds if provided by hosts visited in prior crawls. Hosts are selected from the highest-ranking 60 million domains of the May/June/July 2018 webgraph data set
a breadth-first side crawl within a maximum of 10 links (“hops”) away from the home pages of the top 40 million domains of the webgraph dataset
a random sample of outlinks taken from WAT files of the September crawl
15 million external links sampled from Wikipedia data dumps

Please note that the character set detection was not fully working for the first 13 segments of the October crawl – about 15% of the page captures in these segments have no charset and language assigned. More information is found in the bug report.

Archive Location and Download

The October crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2018-43/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2018-43/segment.paths.gz	100
WARC files	CC-MAIN-2018-43/warc.paths.gz	56000	58.84
WAT files	CC-MAIN-2018-43/wat.paths.gz	56000	19.34
WET files	CC-MAIN-2018-43/wet.paths.gz	56000	8.22
Robots.txt files	CC-MAIN-2018-43/robotstxt.paths.gz	56000	0.21
Non-200 responses files	CC-MAIN-2018-43/non200responses.paths.gz	56000	1.78
URL index files	CC-MAIN-2018-43/cc-index.paths.gz	302	0.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-43/. Also the columnar index has been updated to contain this crawl.

September 2018 crawl archive now available

The crawl archive for September 2018 is now available! It contains 2.8 billion web pages and 220 TiB of uncompressed content, crawled between September 17th and 26th.

The following improvements and fixes to the data formats have been made:

the columnar index contains the content language of a web page as a new field. Please read the instructions below how to upgrade your tools to read newly added fields.
WARC revisit records (HTTP status 304) in the URL indexes do not include a field for the payload "digest" anymore. The corresponding column "content_digest" in the columnar index now contains null values.
we’ve fixed a bug in the WARC writer which added an extra line break (\r\n) between HTTP header and payload in WARC response record. See the announcement on our Google group for details. Thanks again to Greg Lindahl for discovering this bug!

The September crawl contains 500 million new URLs, not contained in any crawl archive before. New URLs stem from

the continued seed donation of URLs from mixnode.com
extracting and sampling URLs from sitemaps, RSS and Atom feeds if provided by hosts visited in prior crawls. Hosts are selected from the highest-ranking 60 million domains of the May/June/July 2018 webgraph data set
a breadth-first side crawl within a maximum of 6 links (“hops”) away from the home pages of the top 25 million domains of the webgraph dataset
a random sample taken from WAT files of the August crawl

New Fields in the Columnar URL Index

The columnar index has been updated to contain two new fields added to WARC and CDX files starting with the August crawl:

content_charset: the character encoding used by the HTML page
content_languages: a comma-separated list of ISO-639-3 language codes detected identified by the Compact Language Detector 2 (CLD2)

In addition, the column content_digest now contains null values.

The table schema in the cc-index-table project on github has been updated to reflect these changes.

Please follow the instructions below to upgrade to the new schema for Spark, Athena/Presto or Hive. If you do not want to use the new fields, no action is required, the tools should continue to work with the old schema.

Spark

The property spark.sql.parquet.mergeSchema must be set to true, e.g. by running the Spark job with the command

spark-submit --conf spark.sql.parquet.mergeSchema=true ...

Note that enabling schema merging has a negative impact on the performance of Spark jobs,
you may want to enable it only in case the new fields are required for your task.

Athena / Presto

Please create a new table using the updated schema. The old schema will continue to work but the new fields cannot be used. Further information can be found in the chapter about schema updates in the Athena documentation.

Hive

Schema evolution is supported since version 0.13. The procedure is essentially the same as for Athena – you need to drop and re-create the table with the updated schema only in case the new fields are used.

Archive Location and Download

The September crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2018-39/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2018-39/segment.paths.gz	100
WARC files	CC-MAIN-2018-39/warc.paths.gz	56320	47.9
WAT files	CC-MAIN-2018-39/wat.paths.gz	56320	18.5
WET files	CC-MAIN-2018-39/wet.paths.gz	56320	7.87
Robots.txt files	CC-MAIN-2018-39/robotstxt.paths.gz	56320	0.19
Non-200 responses files	CC-MAIN-2018-39/non200responses.paths.gz	56320	1.83
URL index files	CC-MAIN-2018-39/cc-index.paths.gz	302	0.22

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-39/. Also the columnar index has been updated to contain this crawl.

We are grateful to our friends at mixnode for donating a seed list of 200 Million URLs to enhance the Common Crawl.

August Crawl Archive Introduces Language Annotations

The crawl archive for August 2018 is now available! It contains 2.65 billion web pages and 220 TiB of uncompressed content, crawled between August 14th and 22th. Together with an upgrade of the crawler software we’ve plugged in a language detector and now provide as annotation the language a web page is written in.

Please note that the WARC files of August 2018 (CC-MAIN-2018-34) are affected by a WARC format error and contain an extra \r\n between HTTP header and payload content. Also the given "Content-Length" is off by 2 bytes. For more information about this bug see this post on our user forum.

Language Annotations

We now run the Compact Language Detector 2 (CLD2) on HTML pages to identify the language of a document. CLD2 is able to identify 160 different languages and up to 3 languages per document. The detected languages resp. the ISO-639-3 code are shown in the URL index as a new field, e.g., "languages": "zho,eng". The WARC metadata records contain the full CLD2 response including scores and text coverage:

languages-cld2: {"reliable":true,"text-bytes":3783,"languages":[{"code":"zh","code-iso-639-3":"zho","text-covered":0.93,"score":1943.0,"name":"Chinese"},{"code":"en","code-iso-639-3":"eng","text-covered":0.05,"score":523.0,"name":"ENGLISH"}]}

On github you’ll find the Java bindings to the CLD2 native library and the distribution of the primary document languages as part of our crawl statistics.

Please note that the columnar index does not contain the detected languages for now. This requires a change of the table schema. We plan to add the new fields later after we’ve verified that an update of the schema does not break common tools (e.g., Spark or Presto/Athena) used to process the table.

Crawler Software Upgrade and Minor Changes to WARC Files

Our crawler has been upgraded and is now based on the most recent version of Apache Nutch (1.15). The source code can be found on github in our Nutch fork.

In conjunction with the crawler upgrade we made the following minor changes affecting the WARC record format of the crawl archives:

"HTTP 304 notmodified" responses are now stored as WARC revisit records in the "crawldiagnostics" subset along with 404s, redirects and other non-200 responses. For now the revisit records contain a payload digest although there is no payload sent together with HTTP 304 responses. The stupid reason is that the columnar index requires the digest field and we want to make sure that all tools continue to work as expected. The SHA-1 digest of an empty payload (zero bytes) is used for the revisit records.
All HTTP response headers are now preserved. As before, if the page content is truncated or was compressed or chunked during transfer, the headers "Content-Encoding", "Transfer-Encoding" and "Content-Length" need to be rewritten, otherwise WARC readers may fail reading the record payload. E.g., a page compressed on the HTTP protocol layer may have the following headers – the original headers are prefixed with X-Crawler-:
```
X-Crawler-Content-Encoding: gzip
X-Crawler-Content-Length: 2010
Content-Length: 16125
```
The crawler may now also store pages fetched partially because of a network disconnect. These captures are marked as WARC-Truncated: disconnect in the WARC record header. Note that the crawler may also truncate the page payload because of a content limit (we store only 1 MB per page) or a time limit (after 10 minutes a page download is canceled).
the WARC record headers indicate still "WARC/1.0" although we follow the WARC specification, v1.1. While testing various WARC reader libraries we’ve found that at least two of them fail on records with a "WARC/1.1" header.

Please note that due to a bug the first two crawled segments are without robots.txt captures.

Archive Location and Download

The August crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2018-34/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2018-34/segment.paths.gz	100
WARC files	CC-MAIN-2018-34/warc.paths.gz	71520	67.79
WAT files	CC-MAIN-2018-34/wat.paths.gz	71520	16.76
WET files	CC-MAIN-2018-34/wet.paths.gz	71520	6.92
Robots.txt files	CC-MAIN-2018-34/robotstxt.paths.gz	71520	0.19
Non-200 responses files	CC-MAIN-2018-34/non200responses.paths.gz	71520	1.79
URL index files	CC-MAIN-2018-34/cc-index.paths.gz	302	0.21

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-34/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs May/June/July 2018

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of May, June and July 2018. Additional information about data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs).

Host-level graph

The graph consists of 886 million nodes and 5.4 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 793 million dangling nodes (89.5%) and the largest strongly connected component contains only 67 million (7.5%) nodes. The host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 886 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2018-may-jun-jul/host/. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2018-may-jun-jul/host/ as prefix to access the files from everywhere.

The following files and formats are provided:

Download files of the Common Crawl May/June/July 2018 host-level webgraph

Size	File	Description
5.60 GB	cc-main-2018-may-jun-jul-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 98 vertices files
25.12 GB	cc-main-2018-may-jun-jul-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 196 edges files

9.99 GB	cc-main-2018-may-jun-jul-host.graph	graph in BVGraph format
2 kB	cc-main-2018-may-jun-jul-host.properties

11.30 GB	cc-main-2018-may-jun-jul-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2018-may-jun-jul-host-t.properties

1 kB	cc-main-2018-may-jun-jul-host.stats	WebGraph statistics

13.35 GB	cc-main-2018-may-jun-jul-host-ranks.txt.gz	harmonic centrality and pagerank

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 92 million nodes and 1.45 billion edges. 57% or 53 million nodes are dangling nodes, the largest strongly connected component covers 34 million or 37% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2018-may-jun-jul/domain/ resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2018-may-jun-jul/domain/.

Download files of the Common Crawl May/June/July 2018 domain-level webgraph

Size	File	Description
0.64 GB	cc-main-2018-may-jun-jul-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
5.85 GB	cc-main-2018-may-jun-jul-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.21 GB	cc-main-2018-may-jun-jul-domain.graph	graph in BVGraph format
2 kB	cc-main-2018-may-jun-jul-domain.properties

3.43 GB	cc-main-2018-may-jun-jul-domain-t.graph	transpose of the graph
2 kB	cc-main-2018-may-jun-jul-domain-t.properties

1 kB	cc-main-2018-may-jun-jul-domain.stats	WebGraph statistics

1.96 GB	cc-main-2018-may-jun-jul-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 92 million domains is available for download.

Top 1000 domains ranked by harmonic centrality (May/June/July 2018)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	25381622	2	0.013272	com.facebook
2	24767500	1	0.016429	com.googleapis
3	23574566	3	0.009596	com.google
4	22826384	4	0.008408	com.twitter
5	22398676	5	0.007043	com.youtube
6	21446850	6	0.006211	org.w
7	20000170	7	0.004495	org.gmpg
8	19917792	9	0.003686	com.instagram
9	19565892	11	0.003123	com.linkedin
10	18904142	25	0.001434	com.gravatar
11	18886866	14	0.002009	com.wordpress
12	18791656	26	0.001378	org.wikipedia
13	18683474	23	0.001591	com.pinterest
14	18605644	13	0.002616	org.wordpress
15	18523062	21	0.001661	com.apple
16	18506550	12	0.002795	com.bootstrapcdn
17	18299454	33	0.000893	com.blogspot
18	18261082	24	0.001454	com.vimeo
19	18104372	37	0.000799	com.amazon
20	18101742	34	0.000875	gl.goo
21	18052860	38	0.000756	be.youtu
22	18015066	28	0.001162	com.microsoft
23	17990876	16	0.001783	com.googletagmanager
24	17945844	19	0.001702	com.adobe
25	17942552	44	0.000652	com.tumblr
26	17901968	15	0.001947	com.cloudflare
27	17853502	20	0.001684	com.macromedia
28	17832868	45	0.000641	com.wp
29	17823530	61	0.000483	com.yahoo
30	17781678	40	0.000719	com.flickr
31	17733538	46	0.000626	ly.bit
32	17680810	48	0.000606	me.wp
33	17674312	35	0.000857	com.paypal
34	17654864	32	0.000904	com.amazonaws
35	17602138	22	0.001598	com.github
36	17588460	104	0.000250	com.nytimes
37	17550714	54	0.000545	org.mozilla
38	17517872	70	0.000400	com.weebly
39	17506702	89	0.000291	com.googleusercontent
40	17435634	41	0.000714	io.github
41	17400536	184	0.000140	com.wsj
42	17353638	144	0.000209	com.dropbox
43	17341062	166	0.000161	org.wikimedia
44	17336710	141	0.000217	com.imgur
45	17319686	57	0.000497	com.medium
46	17318280	68	0.000411	org.creativecommons
47	17316806	65	0.000434	com.bing
48	17274976	147	0.000198	com.blogger
49	17261700	29	0.001115	com.gstatic
50	17257470	66	0.000422	com.jquery
51	17236104	211	0.000119	com.businessinsider
52	17211698	155	0.000182	net.slideshare
53	17211430	207	0.000120	com.wired
54	17203902	53	0.000577	co.t
55	17197252	56	0.000520	eu.europa
56	17187720	182	0.000142	com.myspace
57	17184156	92	0.000278	com.mailchimp
58	17153616	36	0.000843	org.apache
59	17131630	31	0.000912	net.doubleclick
60	17127558	69	0.000402	com.statcounter
61	17120814	63	0.000477	com.list-manage
62	17112850	246	0.000100	org.npr
63	17105220	145	0.000203	com.issuu
64	17099450	27	0.001250	ru.yandex
65	17089826	314	0.000078	com.theverge
66	17089012	321	0.000077	com.appspot
67	17080688	168	0.000159	org.gnu
68	17059280	142	0.000216	com.yelp
69	17056948	52	0.000581	org.w3
70	17053438	331	0.000075	com.about
71	17048062	267	0.000090	me.about
72	17035726	176	0.000148	com.oracle
73	17031278	8	0.004428	com.godaddy
74	17016312	175	0.000148	org.ietf
75	17014942	377	0.000065	com.slate
76	16988198	42	0.000702	net.cloudfront
77	16987550	301	0.000082	com.buzzfeed
78	16986236	226	0.000111	com.tinyurl
79	16984038	436	0.000056	edu.princeton
80	16970148	336	0.000074	com.deviantart
81	16945098	206	0.000122	com.cnn
82	16943242	366	0.000066	edu.washington
83	16941304	105	0.000250	com.reddit
84	16922914	393	0.000062	edu.ucla
85	16917390	85	0.000302	com.soundcloud
86	16917338	449	0.000055	com.nike
87	16909478	193	0.000136	uk.co.bbc
88	16901808	60	0.000485	org.schema
89	16899732	378	0.000064	org.arxiv
90	16896852	397	0.000060	org.chromium
91	16894404	181	0.000142	com.theguardian
92	16888586	165	0.000163	com.forbes
93	16887802	380	0.000064	com.stackexchange
94	16886350	161	0.000172	com.android
95	16885880	343	0.000070	gov.loc
96	16881304	437	0.000056	com.qz
97	16878064	333	0.000074	com.foursquare
98	16871092	242	0.000102	com.nbcnews
99	16862884	312	0.000079	gov.fda
100	16856244	389	0.000063	org.ieee
101	16855606	30	0.000991	com.squarespace
102	16849916	443	0.000055	org.sciencemag
103	16828640	82	0.000323	net.akamaihd
104	16825050	284	0.000085	com.example
105	16821922	423	0.000057	com.trello
106	16815472	172	0.000152	com.whatsapp
107	16812836	215	0.000119	es.google
108	16812598	298	0.000082	com.typeform
109	16793418	616	0.000043	com.flipboard
110	16787856	59	0.000493	net.fbcdn
111	16782836	151	0.000190	org.bbb
112	16781822	218	0.000118	edu.stanford
113	16781438	427	0.000057	com.libsyn
114	16779444	502	0.000049	google.blog
115	16777134	283	0.000085	com.go
116	16774636	419	0.000057	com.withgoogle
117	16763315	610	0.000043	edu.utah
118	16762919	64	0.000437	com.ytimg
119	16761512	328	0.000076	com.reuters
120	16756209	251	0.000097	com.live
121	16749983	164	0.000163	org.archive
122	16747795	518	0.000048	edu.gatech
123	16741185	75	0.000357	com.fb
124	16739169	106	0.000250	edu.utexas
125	16738538	208	0.000120	com.huffingtonpost
126	16737695	286	0.000084	com.bloomberg
127	16732629	241	0.000103	com.techcrunch
128	16726768	317	0.000078	edu.harvard
129	16724125	205	0.000123	com.dribbble
130	16723935	318	0.000078	com.git-scm
131	16719142	169	0.000159	gov.nih
132	16707450	146	0.000199	net.sourceforge
133	16706716	340	0.000072	com.msn
134	16705396	77	0.000351	com.wix
135	16697236	294	0.000083	uk.co.blogspot
136	16694665	316	0.000078	com.googlecode
137	16688264	369	0.000066	com.bbc
138	16685043	225	0.000111	com.typepad
139	16684281	234	0.000106	com.washingtonpost
140	16683086	213	0.000119	com.imdb
141	16679284	538	0.000047	com.chron
142	16671633	706	0.000042	com.hbo
143	16668347	364	0.000067	com.mashable
144	16665099	87	0.000294	com.shopify
145	16661523	76	0.000351	com.paypalobjects
146	16659375	262	0.000092	edu.mit
147	16650967	394	0.000062	com.tinypic
148	16650908	293	0.000083	au.com.google
149	16647149	308	0.000080	com.cnet
150	16641164	300	0.000082	com.usatoday
151	16640112	296	0.000083	net.windows
152	16637416	416	0.000058	au.gov.nsw
153	16626638	299	0.000082	com.ibm
154	16622823	371	0.000065	uk.co.dailymail
155	16622125	339	0.000073	uk.co.telegraph
156	16619319	362	0.000067	com.gmail
157	16615887	173	0.000151	com.eventbrite
158	16612204	227	0.000110	net.php
159	16609048	523	0.000048	com.fastcodesign
160	16596577	334	0.000074	com.time
161	16594204	421	0.000057	com.ted
162	16593660	73	0.000366	de.google
163	16591616	531	0.000047	org.rubyonrails
164	16575402	271	0.000088	com.mapquest
165	16575133	561	0.000045	edu.illinois
166	16571670	170	0.000154	com.opera
167	16568994	395	0.000061	com.latimes
168	16565169	819	0.000037	com.dezeen
169	16560426	306	0.000081	com.hp
170	16556998	180	0.000143	com.stackoverflow
171	16554184	452	0.000055	org.eclipse
172	16549466	236	0.000105	com.ebay
173	16543706	440	0.000055	com.kickstarter
174	16540301	425	0.000057	gov.nasa
175	16538974	231	0.000106	uk.co.amazon
176	16533349	486	0.000051	edu.cornell
177	16531917	188	0.000139	com.etsy
178	16529057	410	0.000058	com.aol
179	16526090	542	0.000046	com.quora
180	16525042	288	0.000084	com.meetup
181	16520185	520	0.000048	com.googleblog
182	16518640	761	0.000039	io.itch
183	16517972	414	0.000058	com.variety
184	16517219	499	0.000050	edu.berkeley
185	16508733	622	0.000043	uk.co.pinterest
186	16506112	100	0.000257	com.livestream
187	16502945	514	0.000049	com.ft
188	16501376	466	0.000053	co.g
189	16499713	439	0.000056	com.theatlantic
190	16498131	519	0.000048	com.zdnet
191	16492014	244	0.000101	com.surveymonkey
192	16488627	199	0.000130	com.tripadvisor
193	16485175	390	0.000063	com.cnbc
194	16483162	1037	0.000031	com.engadget
195	16481038	463	0.000054	com.mixcloud
196	16476002	605	0.000044	com.vogue
197	16470277	752	0.000039	com.nationalgeographic
198	16468657	750	0.000040	com.creativebloq
199	16467451	565	0.000045	com.yellowpages
200	16467125	90	0.000290	com.addthis
201	16466147	239	0.000103	org.drupal
202	16464114	311	0.000079	com.udacity
203	16463605	898	0.000035	com.sfgate
204	16456191	748	0.000040	com.discogs
205	16455549	255	0.000095	com.digg
206	16453381	817	0.000037	com.wikia
207	16449510	528	0.000047	com.nature
208	16448944	185	0.000140	com.spotify
209	16448054	560	0.000045	org.pbs
210	16448046	86	0.000300	com.twimg
211	16444439	431	0.000056	com.angieslist
212	16437315	384	0.000063	com.skype
213	16435380	469	0.000053	com.fortune
214	16434501	102	0.000255	net.jsfiddle
215	16433743	602	0.000044	com.newyorker
216	16431871	492	0.000051	com.cbsnews
217	16430182	405	0.000059	gov.whitehouse
218	16427414	354	0.000069	org.python
219	16425704	280	0.000086	com.hubspot
220	16424605	324	0.000076	gov.cdc
221	16421283	547	0.000046	org.aarp
222	16420623	541	0.000046	com.findlaw
223	16419061	194	0.000135	com.zendesk
224	16416565	879	0.000036	com.arstechnica
225	16415324	493	0.000051	org.hbr
226	16415039	160	0.000173	com.wixsite
227	16414264	513	0.000049	com.cisco
228	16414231	58	0.000495	com.vk
229	16414007	361	0.000067	com.photobucket
230	16408388	357	0.000069	com.springer
231	16407667	509	0.000049	com.superpages
232	16406036	728	0.000041	com.intel
233	16405431	407	0.000058	com.giphy
234	16404519	253	0.000096	to.amzn
235	16404240	800	0.000038	com.manta
236	16403321	47	0.000608	com.qq
237	16403260	1188	0.000029	com.gizmodo
238	16403076	498	0.000050	com.entrepreneur
239	16402950	608	0.000043	com.venturebeat
240	16402108	1047	0.000031	edu.upenn
241	16401145	360	0.000068	com.nypost
242	16401094	406	0.000059	org.un
243	16395874	1405	0.000028	uk.ac.ox
244	16395078	626	0.000043	com.scribd
245	16394056	961	0.000034	com.thenextweb
246	16393237	587	0.000044	com.unsplash
247	16392164	470	0.000053	com.xrea
248	16390829	764	0.000039	com.hackernoon
249	16390735	1013	0.000032	edu.columbia
250	16385680	713	0.000041	com.box
251	16383953	230	0.000108	com.stumbleupon
252	16383638	1551	0.000024	edu.purdue
253	16381649	593	0.000044	com.vice
254	16381302	913	0.000035	ly.snip
255	16380414	229	0.000109	net.behance
256	16380111	600	0.000044	com.symantec
257	16379592	111	0.000236	com.jimdo
258	16378318	959	0.000034	com.googledrive
259	16377172	257	0.000094	com.salesforce
260	16375969	387	0.000063	com.images-amazon
261	16374033	504	0.000049	org.unicode
262	16372683	445	0.000055	com.office
263	16370954	635	0.000043	com.citysearch
264	16365786	876	0.000036	com.healthgrades
265	16363957	303	0.000081	org.acm
266	16363446	240	0.000103	com.disqus
267	16363144	338	0.000073	com.tripod
268	16356913	992	0.000033	com.pixabay
269	16354234	370	0.000066	com.oreilly
270	16343963	1048	0.000031	com.indiegogo
271	16342521	1040	0.000031	com.evernote
272	16337476	735	0.000040	gov.noaa
273	16337226	785	0.000038	com.spreadshirt
274	16332902	1364	0.000029	com.searchengineland
275	16330466	1597	0.000023	uk.co.theregister
276	16327360	620	0.000043	com.avvo
277	16325045	189	0.000138	com.constantcontact
278	16324354	489	0.000051	com.inc
279	16322948	920	0.000035	com.naturalnews
280	16321702	376	0.000065	org.ampproject
281	16321578	450	0.000055	me.paypal
282	16320188	342	0.000071	com.livejournal
283	16319952	422	0.000057	com.businesswire
284	16319387	974	0.000033	au.net.abc
285	16319144	259	0.000093	org.joomla
286	16314140	885	0.000035	com.dropboxusercontent
287	16312103	853	0.000036	com.statista
288	16308378	558	0.000045	com.goodreads
289	16308361	1398	0.000028	com.sciencedaily
290	16307378	1377	0.000029	com.storify
291	16302792	796	0.000038	com.curbed
292	16302729	186	0.000139	com.feedburner
293	16300939	1548	0.000024	com.pcmag
294	16300342	644	0.000042	gov.defense
295	16300060	726	0.000041	org.eff
296	16298671	383	0.000064	com.sxsw
297	16298398	1594	0.000023	com.mcafee
298	16296843	471	0.000052	com.snapchat
299	16295047	1002	0.000032	com.shutterstock
300	16294087	700	0.000042	com.moz
301	16293401	158	0.000175	uk.co.google
302	16291354	433	0.000056	com.adweek
303	16289733	337	0.000073	gov.ca
304	16288869	237	0.000105	com.bandcamp
305	16285197	233	0.000106	de.amazon
306	16281279	909	0.000035	gov.census
307	16279095	627	0.000043	site.business
308	16278411	767	0.000039	com.economist
309	16273403	346	0.000070	com.wiley
310	16271595	157	0.000177	com.weibo
311	16271436	911	0.000035	gov.uspto
312	16270078	93	0.000273	me.fb
313	16266403	990	0.000033	gov.fcc
314	16263211	1624	0.000022	com.pcworld
315	16263070	1366	0.000029	org.worldbank
316	16262404	1695	0.000021	com.fifa
317	16261893	892	0.000035	com.merchantcircle
318	16261024	749	0.000040	tv.twitch
319	16257455	1719	0.000020	edu.unc
320	16255699	954	0.000034	com.steampowered
321	16255672	1637	0.000022	org.khanacademy
322	16255397	880	0.000036	com.indiatimes
323	16255076	309	0.000080	com.smugmug
324	16254667	564	0.000045	com.wikihow
325	16253704	949	0.000034	org.unesco
326	16253384	1590	0.000023	edu.northwestern
327	16252481	1056	0.000031	com.redhat
328	16250050	1599	0.000023	com.scientificamerican
329	16247225	808	0.000037	gov.nist
330	16246611	1563	0.000024	com.smashingmagazine
331	16241341	822	0.000037	com.deloitte
332	16240554	1388	0.000028	com.politico
333	16239514	325	0.000076	com.googlesyndication
334	16238580	787	0.000038	org.tigris
335	16237777	367	0.000066	com.prnewswire
336	16235711	1017	0.000032	edu.yale
337	16235543	925	0.000035	com.ubuntu
338	16233613	742	0.000040	org.aiga
339	16232678	1487	0.000026	com.pexels
340	16230507	1493	0.000026	com.thinkwithgoogle
341	16229621	1440	0.000027	com.alibaba
342	16226635	304	0.000081	ca.google
343	16226232	359	0.000068	com.dailymotion
344	16226122	1703	0.000021	com.vanityfair
345	16223766	1794	0.000019	com.udemy
346	16223352	277	0.000086	com.windowsphone
347	16222378	621	0.000043	com.slack
348	16219936	1409	0.000028	ca.blogspot
349	16219473	332	0.000074	com.bitly
350	16214229	962	0.000034	gov.nps
351	16213218	279	0.000086	com.wufoo
352	16213088	755	0.000039	com.webmd
353	16212554	794	0.000038	de.blogspot
354	16209004	1019	0.000032	com.prweb
355	16204931	1766	0.000019	edu.usc
356	16204520	546	0.000046	com.homeadvisor
357	16203590	991	0.000033	com.deepmind
358	16203266	322	0.000077	com.mozilla
359	16201535	1652	0.000022	org.weforum
360	16200784	1912	0.000017	com.ehow
361	16199879	586	0.000044	com.netflix
362	16199442	724	0.000041	com.samsung
363	16197387	477	0.000052	com.webs
364	16197196	1833	0.000018	com.ikea
365	16196228	183	0.000141	jp.co.yahoo
366	16195807	2548	0.000012	com.sophos
367	16195059	813	0.000037	org.amnesty
368	16191501	996	0.000033	org.spie
369	16191482	1635	0.000022	com.billboard
370	16191120	1374	0.000029	com.hootsuite
371	16188221	1001	0.000032	com.whitepages
372	16188111	319	0.000078	fr.free
373	16187195	187	0.000139	com.xing
374	16186531	815	0.000037	com.java
375	16186252	1827	0.000018	org.coursera
376	16184139	1381	0.000029	com.speakerdeck
377	16182369	152	0.000190	com.youtube-nocookie
378	16178099	1908	0.000017	com.tutsplus
379	16177836	899	0.000035	com.marketwatch
380	16175545	995	0.000033	edu.psu
381	16174492	1688	0.000021	com.chrome
382	16173930	1077	0.000031	com.airbnb
383	16173485	1740	0.000020	au.com.smh
384	16173008	952	0.000034	gov.senate
385	16172258	276	0.000087	com.getbootstrap
386	16171414	1443	0.000027	com.marketingland
387	16169883	1104	0.000030	com.ycombinator
388	16168541	413	0.000058	int.who
389	16168403	1021	0.000032	edu.umich
390	16167647	1511	0.000025	com.xkcd
391	16165208	1617	0.000022	com.merriam-webster
392	16164062	986	0.000033	it.binged
393	16162437	1052	0.000031	com.sun
394	16162150	769	0.000039	com.googlesource
395	16161882	1439	0.000027	edu.ucsd
396	16160824	399	0.000060	com.mysql
397	16156698	418	0.000057	com.bigcartel
398	16154861	846	0.000036	gov.state
399	16153725	981	0.000033	com.itsnicethat
400	16153693	1459	0.000027	uk.ac.cam
401	16152535	260	0.000093	com.myshopify
402	16152053	1482	0.000026	co.vine
403	16150830	540	0.000047	gov.usda
404	16147936	1934	0.000017	edu.ucdavis
405	16147539	1840	0.000018	com.autodesk
406	16147195	795	0.000038	org.aclweb
407	16146633	1403	0.000028	com.css-tricks
408	16144670	2154	0.000014	edu.ncsu
409	16143655	1646	0.000022	com.playstation
410	16143349	946	0.000034	io.material
411	16143314	1105	0.000030	org.iso
412	16142931	836	0.000036	gov.justice
413	16141858	827	0.000037	com.foxnews
414	16141188	829	0.000037	com.gartner
415	16140967	1721	0.000020	uk.ac.ucl
416	16137787	382	0.000064	com.booking
417	16136856	945	0.000034	com.psychologytoday
418	16136353	81	0.000337	com.baidu
419	16135804	786	0.000038	gov.copyright
420	16135574	1593	0.000023	com.target
421	16135083	2021	0.000016	edu.arizona
422	16131304	1450	0.000027	io.codepen
423	16130058	415	0.000058	com.monster
424	16126494	482	0.000052	gov.irs
425	16126161	1732	0.000020	com.freepik
426	16123562	1430	0.000027	com.gumroad
427	16122500	1519	0.000025	de.spiegel
428	16120242	285	0.000085	gov.ftc
429	16117722	1660	0.000021	com.com
430	16116716	363	0.000067	com.githubusercontent
431	16115063	1977	0.000016	com.msnbc
432	16113873	572	0.000045	in.co.google
433	16110022	1722	0.000020	com.gigaom
434	16109670	1438	0.000027	com.dell
435	16108948	907	0.000035	com.tandfonline
436	16108826	396	0.000060	net.themeforest
437	16108552	1442	0.000027	com.businessweek
438	16107032	533	0.000047	gov.epa
439	16106841	1039	0.000031	com.gofundme
440	16106821	326	0.000076	com.rawgit
441	16106609	1875	0.000018	com.angelfire
442	16105593	1824	0.000018	com.yoast
443	16104950	2525	0.000012	com.fiverr
444	16104946	1623	0.000022	com.nymag
445	16103599	1616	0.000022	com.hollywoodreporter
446	16103457	1027	0.000032	ca.cbc
447	16103234	1648	0.000022	com.sap
448	16103036	1098	0.000030	com.nielsen
449	16102998	426	0.000057	org.nodejs
450	16102709	2510	0.000012	edu.hbs
451	16102701	195	0.000135	com.eepurl
452	16102688	741	0.000040	com.blackberry
453	16102121	2618	0.000012	edu.caltech
454	16101515	1576	0.000024	com.ning
455	16100948	582	0.000044	uk.co.independent
456	16099080	967	0.000033	com.underconsideration
457	16098787	1904	0.000017	com.semrush
458	16096514	2562	0.000012	com.popsci
459	16096221	1903	0.000017	com.howstuffworks
460	16096111	734	0.000040	gov.hhs
461	16094981	792	0.000038	com.usnews
462	16093468	17	0.001739	com.wixstatic
463	16093240	1591	0.000023	org.fao
464	16091056	1985	0.000016	tv.periscope
465	16090419	2152	0.000014	com.cbs
466	16089586	1457	0.000027	org.altervista
467	16088656	480	0.000052	us.icio
468	16087093	451	0.000055	com.force
469	16086649	1023	0.000032	com.500px
470	16085221	2114	0.000015	uk.ac.ed
471	16083763	2228	0.000014	com.instructables
472	16083658	1936	0.000017	org.filezilla-project
473	16082249	1675	0.000021	com.nba
474	16081782	2599	0.000012	com.codecademy
475	16081087	1531	0.000025	com.elpais
476	16080942	1091	0.000030	es.iac
477	16078592	120	0.000226	com.google-analytics
478	16078096	305	0.000081	com.staticflickr
479	16077806	861	0.000036	uk.co.guardian
480	16076776	1517	0.000025	com.warnerbros
481	16076399	496	0.000050	com.cargocollective
482	16076336	1868	0.000018	com.canva
483	16076212	2072	0.000015	com.gamespot
484	16075519	1665	0.000021	edu.jhu
485	16074907	1437	0.000027	edu.wisc
486	16072844	976	0.000033	com.uservoice
487	16070548	963	0.000033	net.researchgate
488	16070276	1420	0.000028	com.istockphoto
489	16069880	1022	0.000032	com.insiderpages
490	16068798	790	0.000038	tv.ustream
491	16068337	2191	0.000014	au.com.news
492	16068265	4005	0.000007	com.space
493	16067417	881	0.000036	gov.arts
494	16067121	281	0.000085	com.fc2
495	16067028	444	0.000055	com.sciencedirect
496	16066807	1726	0.000020	com.hulu
497	16066412	1434	0.000027	gov.usgs
498	16064058	1804	0.000019	com.fedex
499	16063290	1460	0.000027	com.forrester
500	16061979	1897	0.000017	org.pnas
501	16061256	799	0.000038	com.feedly
502	16060176	2986	0.000010	com.hubpages
503	16060072	2107	0.000015	com.crunchbase
504	16059447	1725	0.000020	com.mercurynews
505	16057745	1415	0.000028	com.reverbnation
506	16057012	1004	0.000032	com.lighthouseapp
507	16056491	1609	0.000023	com.indeed
508	16056336	2396	0.000013	com.programmableweb
509	16055614	746	0.000040	com.gotowebinar
510	16055125	1234	0.000029	com.mlb
511	16054055	1005	0.000032	com.timeanddate
512	16053957	1713	0.000020	kr.flic
513	16053898	159	0.000174	com.googleadservices
514	16053749	1600	0.000023	edu.si
515	16052692	247	0.000099	com.getclicky
516	16052033	222	0.000114	jp.co.amazon
517	16051826	1749	0.000020	com.today
518	16051338	1376	0.000029	ly.ow
519	16050686	441	0.000055	edu.cmu
520	16050498	1375	0.000029	org.redcross
521	16050390	601	0.000044	com.squareup
522	16048735	1809	0.000019	com.domain
523	16048179	1692	0.000021	edu.uchicago
524	16047498	1385	0.000028	de.heise
525	16046589	1042	0.000031	com.googlelabs
526	16046497	935	0.000034	com.patreon
527	16045843	2144	0.000015	com.ibtimes
528	16045451	401	0.000059	com.clicky
529	16043594	1826	0.000018	com.socialmediaexaminer
530	16042787	1416	0.000028	com.americanexpress
531	16042447	411	0.000058	com.w3schools
532	16042132	2179	0.000014	org.gimp
533	16041800	727	0.000041	com.photoshelter
534	16041645	386	0.000063	edu.nyu
535	16039704	910	0.000035	org.scala-lang
536	16038413	2877	0.000010	com.oxforddictionaries
537	16038164	888	0.000035	ca.amazon
538	16037904	1761	0.000019	com.upwork
539	16036849	1604	0.000023	org.apa
540	16036736	1549	0.000024	com.accenture
541	16036020	2320	0.000013	com.csmonitor
542	16035521	2772	0.000011	com.lynda
543	16034570	2092	0.000015	com.bestbuy
544	16034530	943	0.000034	com.emarketer
545	16032865	403	0.000059	com.herokuapp
546	16032606	998	0.000033	au.com.yellowpages
547	16032334	581	0.000045	com.houzz
548	16031562	1657	0.000022	com.codeplex
549	16027393	108	0.000243	jp.co.google
550	16027146	1654	0.000022	com.theglobeandmail
551	16027065	1817	0.000019	com.zillow
552	16026139	3127	0.000009	org.notepad-plus-plus
553	16025586	1035	0.000031	com.uber
554	16025512	2158	0.000014	com.aljazeera
555	16025193	481	0.000052	org.doi
556	16024531	1402	0.000028	gov.fbi
557	16024228	278	0.000086	com.youku
558	16024147	1135	0.000030	edu.alamo
559	16023519	1774	0.000019	org.letsencrypt
560	16023515	1847	0.000018	com.lulu
561	16022000	1418	0.000028	com.unity3d
562	16021724	472	0.000052	com.iconfinder
563	16019791	198	0.000133	com.histats
564	16019669	1878	0.000017	com.norton
565	16019565	703	0.000042	uk.co.tripadvisor
566	16019111	1386	0.000028	com.walmart
567	16018429	2180	0.000014	edu.asu
568	16017954	1556	0.000024	com.prezi
569	16016889	989	0.000033	gov.usa
570	16015420	1760	0.000020	com.thehill
571	16013857	2053	0.000015	com.thestar
572	16013612	1561	0.000024	in.blogspot
573	16012265	1041	0.000031	jp.co.fujixerox
574	16011010	1980	0.000016	com.trendmicro
575	16010772	1533	0.000025	com.bufferapp
576	16009533	1579	0.000024	com.intuit
577	16008583	1507	0.000025	edu.umn
578	16007713	2512	0.000012	edu.wustl
579	16006379	1029	0.000032	com.chamberofcommerce
580	16005982	1119	0.000030	net.brownbook
581	16005447	1679	0.000021	com.hotmail
582	16004096	430	0.000056	cn.com.sina
583	16003516	1392	0.000028	com.techrepublic
584	16002880	1666	0.000021	com.econsultancy
585	16002231	4188	0.000007	com.boredpanda
586	16001968	107	0.000247	com.messenger
587	16001601	2167	0.000014	com.icloud
588	16001177	957	0.000034	com.outlook
589	16000754	2412	0.000013	com.twitpic
590	16000697	2045	0.000015	com.ifttt
591	16000659	2134	0.000015	com.lonelyplanet
592	16000307	1851	0.000018	edu.virginia
593	16000043	329	0.000075	com.naver
594	15999112	2103	0.000015	com.mentalfloss
595	15998718	2260	0.000014	com.refinery29
596	15997519	3501	0.000008	net.minecraft
597	15997019	235	0.000106	fr.google
598	15996815	1554	0.000024	com.jetbrains
599	15996044	1399	0.000028	com.aweber
600	15995815	2389	0.000013	com.animoto
601	15995749	1395	0.000028	us.imageshack
602	15995631	1839	0.000018	com.zazzle
603	15995483	1111	0.000030	com.ezlocal
604	15995033	400	0.000060	com.newrelic
605	15994874	1754	0.000020	com.posterous
606	15994709	1986	0.000016	org.aclu
607	15994590	633	0.000043	gov.sec
608	15994083	980	0.000033	uk.co.eventbrite
609	15992997	2533	0.000012	edu.unl
610	15991024	3390	0.000009	com.fitbit
611	15990563	3562	0.000008	com.wolfram
612	15990135	1049	0.000031	edu.utep
613	15989699	1765	0.000019	org.owasp
614	15989560	1914	0.000017	com.people
615	15989353	1700	0.000021	com.irishtimes
616	15987982	1658	0.000021	org.cambridge
617	15987488	1714	0.000020	com.aliexpress
618	15986613	2745	0.000011	org.kiva
619	15986099	2201	0.000014	com.getresponse
620	15985328	870	0.000036	ca.yelp
621	15984766	2650	0.000011	com.klout
622	15984420	1724	0.000020	edu.academia
623	15984370	3826	0.000008	edu.byu
624	15984363	1918	0.000017	edu.cuny
625	15983934	3152	0.000009	edu.dartmouth
626	15983757	3907	0.000007	com.lmgtfy
627	15982277	1397	0.000028	com.alexa
628	15982117	2841	0.000011	com.lastpass
629	15981087	1479	0.000026	com.mckinsey
630	15980938	2078	0.000015	it.scoop
631	15980757	98	0.000261	org.reactjs
632	15979441	88	0.000292	net.facebook
633	15979247	2766	0.000011	com.campaignmonitor
634	15978259	3392	0.000009	edu.uic
635	15977194	2122	0.000015	ch.ethz
636	15976084	148	0.000197	ru.mail
637	15975453	2328	0.000013	com.glamour
638	15975183	197	0.000135	it.google
639	15974182	1128	0.000030	fr.blogspot
640	15973189	1773	0.000019	com.foxbusiness
641	15972517	1993	0.000016	edu.msu
642	15972470	2206	0.000014	ca.ualberta
643	15972287	956	0.000034	com.city-data
644	15970326	2029	0.000016	edu.uci
645	15970066	1729	0.000020	com.newsweek
646	15969815	797	0.000038	org.jenkins-ci
647	15969741	1676	0.000021	com.marketo
648	15969666	1012	0.000032	com.cdbaby
649	15969585	1848	0.000018	com.hostgator
650	15969244	2216	0.000014	com.softpedia
651	15969035	4712	0.000006	com.diigo
652	15968736	997	0.000033	au.com.truelocal
653	15968524	1705	0.000021	com.yandex
654	15968467	3507	0.000008	com.starwars
655	15967022	3277	0.000009	com.softonic
656	15966584	1387	0.000028	com.lifehacker
657	15964769	417	0.000057	com.stripe
658	15964236	1681	0.000021	com.thomsonreuters
659	15964012	1967	0.000016	com.nfl
660	15962731	780	0.000038	com.uk
661	15962641	1451	0.000027	com.weather
662	15962045	2262	0.000014	edu.bu
663	15960841	177	0.000146	org.icann
664	15960580	2896	0.000010	org.ala
665	15960416	835	0.000036	org.openstreetmap
666	15957618	1647	0.000022	mp.j
667	15957395	289	0.000084	com.maxcdn
668	15957230	91	0.000290	org.networkadvertising
669	15957144	3168	0.000009	com.avast
670	15955925	2664	0.000011	org.virtualbox
671	15955293	2008	0.000016	edu.umass
672	15954737	1829	0.000018	gov.nyc
673	15954491	2343	0.000013	com.homedepot
674	15953595	1942	0.000017	edu.ufl
675	15952668	1691	0.000021	com.nokia
676	15952333	2161	0.000014	com.livestrong
677	15951184	2124	0.000015	com.history
678	15948690	409	0.000058	com.fastcompany
679	15947723	2289	0.000013	com.newscientist
680	15946716	1706	0.000021	com.vox
681	15944352	313	0.000078	com.taobao
682	15944010	516	0.000048	net.openid
683	15943715	1584	0.000023	fm.last
684	15943623	2190	0.000014	org.craigslist
685	15943310	878	0.000036	br.com.uol
686	15942512	2950	0.000010	ca.uwaterloo
687	15940080	374	0.000065	com.netdna-ssl
688	15938438	1730	0.000020	com.pwc
689	15935620	988	0.000033	gov.sba
690	15935207	458	0.000054	com.barnesandnoble
691	15935045	2653	0.000011	org.moma
692	15934238	2744	0.000011	org.phys
693	15933975	708	0.000041	com.docker
694	15933262	1055	0.000031	com.adage
695	15933136	1114	0.000030	com.formstack
696	15932731	3680	0.000008	cc.co
697	15932160	969	0.000033	com.pinimg
698	15932109	1670	0.000021	com.xbox
699	15931176	500	0.000050	com.cracked
700	15930661	349	0.000070	nl.google
701	15930196	204	0.000123	jp.ameblo
702	15929700	2800	0.000011	edu.hawaii
703	15929305	2102	0.000015	com.blogtalkradio
704	15929173	857	0.000036	com.delicious
705	15928584	2993	0.000010	com.123rf
706	15928210	2163	0.000014	com.britannica
707	15927984	2946	0.000010	org.greenpeace
708	15927067	1859	0.000018	com.stitcher
709	15926910	1831	0.000018	com.marketwired
710	15926723	1514	0.000025	gov.ny
711	15926477	3027	0.000010	uk.bl
712	15926226	1998	0.000016	net.boingboing
713	15926205	352	0.000070	org.opensource
714	15925493	697	0.000042	fr.amazon
715	15925417	2059	0.000015	com.templatemonster
716	15925384	1928	0.000017	com.networkworld
717	15925363	1038	0.000031	com.infusionsoft
718	15925035	1424	0.000028	com.shareasale
719	15924932	1020	0.000032	au.com.yelp
720	15924102	1025	0.000032	org.designmuseum
721	15924071	4109	0.000007	org.libreoffice
722	15922360	3072	0.000010	com.wikidot
723	15921986	1536	0.000024	com.globo
724	15921710	2960	0.000010	ca.globalnews
725	15921257	2464	0.000012	com.fox
726	15920771	771	0.000039	com.163
727	15920695	3399	0.000009	org.edx
728	15919256	1961	0.000016	com.mac
729	15918811	1842	0.000018	gov.treasury
730	15918386	2085	0.000015	com.urbandictionary
731	15917993	1528	0.000025	gov.bls
732	15917781	202	0.000126	jp.ne.hatena
733	15916788	1400	0.000028	com.arcgis
734	15915656	543	0.000046	com.technologyreview
735	15915609	1709	0.000021	com.gettyimages
736	15915236	872	0.000036	com.msdn
737	15915195	1704	0.000021	com.windows
738	15914569	1094	0.000030	com.mtnonline
739	15912776	2802	0.000011	com.knowyourmeme
740	15912472	223	0.000111	com.automattic
741	15912083	783	0.000038	com.discordapp
742	15912041	1213	0.000029	com.gloworld
743	15910756	4260	0.000007	com.trulia
744	15910422	948	0.000034	com.mysanantonio
745	15910398	238	0.000104	com.parallels
746	15910326	1092	0.000030	com.cbslocal
747	15909788	454	0.000055	com.mapbox
748	15909109	1694	0.000021	com.mtv
749	15909076	2695	0.000011	com.imageshack
750	15908308	1855	0.000018	edu.duke
751	15907269	1406	0.000028	com.accuweather
752	15907201	4130	0.000007	com.techsmith
753	15907143	1742	0.000020	uk.co.wired
754	15907097	3594	0.000008	com.makezine
755	15906461	2759	0.000011	edu.pitt
756	15906416	2077	0.000015	edu.indiana
757	15905735	1085	0.000030	edu.uah
758	15905631	1496	0.000025	me.m
759	15905303	1083	0.000030	com.judysbook
760	15905116	1494	0.000026	com.buffer
761	15904923	1612	0.000023	com.searchenginewatch
762	15904458	4619	0.000006	org.edublogs
763	15902765	1428	0.000028	com.ups
764	15902175	777	0.000038	gov.ed
765	15902033	960	0.000034	au.com.whitepages
766	15901135	2288	0.000013	uk.co.metro
767	15900981	2136	0.000015	com.ign
768	15900568	1940	0.000017	net.codecanyon
769	15899526	1987	0.000016	com.pastebin
770	15898425	2015	0.000016	com.nvidia
771	15898288	1068	0.000031	com.womentechmakers
772	15898118	3296	0.000009	org.code
773	15898010	2976	0.000010	edu.oregonstate
774	15897795	1963	0.000016	com.espn
775	15897477	1723	0.000020	org.gnome
776	15896920	833	0.000036	com.proofpoint
777	15896661	1463	0.000027	gov.dot
778	15896612	1067	0.000031	com.zoho
779	15895176	2534	0.000012	com.producthunt
780	15895150	751	0.000039	com.atlassian
781	15894496	2188	0.000014	ca.ubc
782	15894439	576	0.000045	com.us
783	15894285	1815	0.000019	com.contentmarketinginstitute
784	15894081	1501	0.000025	com.investopedia
785	15893267	2508	0.000012	com.bankofamerica
786	15893187	1685	0.000021	gov.wa
787	15892999	1921	0.000017	com.deadline
788	15892117	2544	0.000012	com.nhl
789	15892001	3657	0.000008	org.lifehack
790	15891790	1683	0.000021	com.vmware
791	15891718	2451	0.000012	com.starbucks
792	15891455	2362	0.000013	ly.visual
793	15891399	1204	0.000029	org.change
794	15891348	1923	0.000017	uk.ac.lse
795	15891194	2115	0.000015	com.magentocommerce
796	15890881	248	0.000098	org.iana
797	15890376	2631	0.000012	com.lifewire
798	15889967	1015	0.000032	com.visualstudio
799	15889397	1559	0.000024	jp.blogspot
800	15889184	1606	0.000023	com.sky
801	15889094	1601	0.000023	com.gotomeeting
802	15889052	1060	0.000031	com.bizcommunity
803	15888940	3281	0.000009	com.smashwords
804	15887168	1651	0.000022	com.mediafire
805	15886599	1578	0.000024	com.ssrn
806	15886095	1686	0.000021	net.recode
807	15885832	2593	0.000012	com.asus
808	15884393	1636	0.000022	se.haxx
809	15884138	1370	0.000029	es.amazon
810	15883786	474	0.000052	com.teamviewer
811	15883621	1787	0.000019	com.outbrain
812	15882577	699	0.000042	com.getpocket
813	15881184	3053	0.000010	com.macrumors
814	15880772	3268	0.000009	net.battle
815	15880362	1581	0.000024	com.nydailynews
816	15879641	1797	0.000019	edu.vanderbilt
817	15879231	2417	0.000013	com.thestreet
818	15878673	941	0.000034	net.azurewebsites
819	15878499	1641	0.000022	fr.lemonde
820	15878493	1620	0.000022	org.postimg
821	15877702	3393	0.000009	com.formula1
822	15877565	1455	0.000027	com.oup
823	15876621	2624	0.000012	gov.cia
824	15876304	2710	0.000011	org.olympic
825	15875628	2576	0.000012	org.7-zip
826	15875577	3906	0.000007	uk.ac.warwick
827	15875481	2573	0.000012	com.tesla
828	15873726	2350	0.000013	hk.com.google
829	15873629	1872	0.000018	com.ecwid
830	15873120	1003	0.000032	com.mlstatic
831	15872169	1663	0.000021	com.glassdoor
832	15872163	2076	0.000015	ca.utoronto
833	15871583	2303	0.000013	net.comcast
834	15870416	2118	0.000015	com.readwrite
835	15870130	2185	0.000014	ca.qc.gouv
836	15869916	1785	0.000019	gov.congress
837	15867724	947	0.000034	com.att
838	15867327	1801	0.000019	uk.co.mirror
839	15867299	495	0.000050	com.marriott
840	15866239	2956	0.000010	com.coinbase
841	15866134	2024	0.000016	com.me
842	15866035	3256	0.000009	gd.is
843	15865519	1421	0.000028	org.plos
844	15864929	1518	0.000025	com.business2community
845	15863777	1079	0.000030	com.sagepub
846	15863406	2876	0.000010	com.fineartamerica
847	15863024	1177	0.000029	me.pxlme
848	15862995	1569	0.000024	com.over-blog
849	15862909	1520	0.000025	com.techtarget
850	15862687	1970	0.000016	ru.narod
851	15862496	1770	0.000019	com.ssllabs
852	15861835	2423	0.000013	com.ge
853	15861301	1838	0.000018	org.unicef
854	15860987	2032	0.000016	int.wipo
855	15860621	179	0.000143	de.bund
856	15859232	982	0.000033	gov.house
857	15858772	2112	0.000015	uk.co.thesun
858	15858732	3078	0.000010	net.sucuri
859	15858417	2184	0.000014	com.yolasite
860	15858413	2567	0.000012	ms.1drv
861	15858271	2123	0.000015	au.com.blogspot
862	15857854	2686	0.000011	com.fool
863	15857834	3950	0.000007	com.thenation
864	15857610	4418	0.000007	edu.temple
865	15857583	3341	0.000009	com.makeuseof
866	15857018	1832	0.000018	edu.umd
867	15856943	782	0.000038	es.com.blogspot
868	15855364	1583	0.000023	com.pingdom
869	15855334	2292	0.000013	com.macworld
870	15855290	455	0.000054	jp.ne.sakura
871	15855088	2563	0.000012	com.webnode
872	15854840	3044	0.000010	com.freelancer
873	15854000	2297	0.000013	gov.nsf
874	15853872	2623	0.000012	edu.brown
875	15852601	4914	0.000006	ca.gc.statcan
876	15852595	1997	0.000016	com.getfirebug
877	15851933	2282	0.000013	com.wikispaces
878	15850737	2009	0.000016	org.jstor
879	15850691	1856	0.000018	co.angel
880	15850665	2912	0.000010	edu.tufts
881	15850001	743	0.000040	org.bitbucket
882	15849764	2243	0.000014	edu.osu
883	15849214	2121	0.000015	edu.tamu
884	15849065	2127	0.000015	org.wpmudev
885	15847636	834	0.000036	net.noscript
886	15847232	3735	0.000008	com.appleinsider
887	15847068	149	0.000195	com.ggpht
888	15846532	1763	0.000019	it.blogspot
889	15846485	2490	0.000012	org.documentcloud
890	15845902	1926	0.000017	com.cc
891	15844588	2207	0.000014	us.zoom
892	15844032	1755	0.000020	com.rollingstone
893	15843496	4205	0.000007	li.paper
894	15843171	2052	0.000015	edu.rutgers
895	15842669	2365	0.000013	com.theonion
896	15842292	723	0.000041	com.geocities
897	15842157	3328	0.000009	com.indiewire
898	15841702	2978	0.000010	int.esa
899	15840954	939	0.000034	com.netdna-cdn
900	15840667	2604	0.000012	ly.generalassemb
901	15840602	3571	0.000008	edu.buffalo
902	15840094	485	0.000051	br.com.google
903	15840038	1078	0.000030	com.bitballoon
904	15839703	484	0.000051	com.1and1
905	15838970	2571	0.000012	com.sony
906	15838770	674	0.000042	com.trustpilot
907	15837379	1200	0.000029	org.oecd
908	15837034	2064	0.000015	com.azcentral
909	15836081	1142	0.000030	com.communitywalk
910	15835821	2197	0.000014	org.videolan
911	15834907	2546	0.000012	com.pandora
912	15833287	5062	0.000006	org.anitaborg
913	15833284	2388	0.000013	gov.in
914	15833081	3017	0.000010	com.4shared
915	15832793	3058	0.000010	org.metmuseum
916	15832533	2149	0.000015	com.theknot
917	15832479	897	0.000035	org.osgeo
918	15830114	201	0.000127	me.line
919	15827610	372	0.000065	com.bizjournals
920	15826874	2584	0.000012	com.fujitsu
921	15826746	2674	0.000011	com.blogs
922	15826244	272	0.000087	org.debian
923	15824893	7770	0.000004	edu.du
924	15823769	2337	0.000013	com.bleacherreport
925	15823410	566	0.000045	com.quantcast
926	15822917	2409	0.000013	uk.co.express
927	15822479	2148	0.000015	com.redbubble
928	15822413	2856	0.000010	com.cosmopolitan
929	15822210	1803	0.000019	org.cancer
930	15821817	1136	0.000030	com.graphis
931	15821068	2058	0.000015	de.zeit
932	15821053	2346	0.000013	ca.sfu
933	15820729	732	0.000040	com.wunderground
934	15820675	2311	0.000013	com.convinceandconvert
935	15820372	2888	0.000010	org.bitcoin
936	15820192	1464	0.000027	com.usps
937	15819872	4545	0.000006	com.blog
938	15819590	2067	0.000015	com.salon
939	15817277	2779	0.000011	com.technet
940	15817261	1799	0.000019	net.daringfireball
941	15817189	2560	0.000012	com.googlepages
942	15816647	115	0.000229	com.bluehost
943	15816557	2089	0.000015	com.w3techs
944	15816260	1771	0.000019	com.calendly
945	15815469	3050	0.000010	com.rottentomatoes
946	15814999	7268	0.000004	com.elance
947	15814532	2529	0.000012	com.createspace
948	15813943	855	0.000036	com.comscore
949	15813610	2404	0.000013	edu.colorado
950	15813599	1089	0.000030	com.2findlocal
951	15813565	1057	0.000031	org.tpr
952	15812908	1643	0.000022	com.bt
953	15812841	365	0.000067	com.rackcdn
954	15811234	3410	0.000009	com.kotaku
955	15811008	3783	0.000008	edu.syr
956	15810497	729	0.000041	com.verisign
957	15809902	1009	0.000032	com.tiddlywiki
958	15808957	1736	0.000020	com.strikingly
959	15808867	6197	0.000005	com.mercedes-benz
960	15808860	2736	0.000011	com.oprah
961	15808851	1707	0.000021	com.bmj
962	15808367	2140	0.000015	com.popsugar
963	15807408	2175	0.000014	org.hrw
964	15807355	2034	0.000016	com.shareholder
965	15807316	1888	0.000017	com.digicert
966	15807087	1645	0.000022	com.steamcommunity
967	15807036	4057	0.000007	com.pastemagazine
968	15806668	3099	0.000010	com.voanews
969	15806626	1044	0.000031	org.travelblog
970	15806470	1441	0.000027	org.heart
971	15806290	2963	0.000010	com.thrillist
972	15805772	4164	0.000007	com.youcaring
973	15805729	1075	0.000031	com.independent
974	15805411	2177	0.000014	net.atlassian
975	15804971	4529	0.000006	com.secondlife
976	15804311	1546	0.000024	int.coe
977	15804252	2363	0.000013	com.xerox
978	15802080	1907	0.000017	com.computerworld
979	15801861	2608	0.000012	com.groupon
980	15801256	3809	0.000008	edu.rochester
981	15800709	3428	0.000009	com.sas
982	15799663	2037	0.000016	com.getsatisfaction
983	15799467	476	0.000052	com.aliyuncs
984	15798626	6525	0.000005	com.threatpost
985	15798271	1825	0.000018	ru.spb
986	15797944	2224	0.000014	com.gawker
987	15797809	2468	0.000012	me.flavors
988	15797730	4197	0.000007	com.slides
989	15797641	2794	0.000011	com.madmimi
990	15797509	2411	0.000013	com.hindustantimes
991	15796978	4338	0.000007	org.teamusa
992	15796570	1467	0.000026	gov.va
993	15795672	1870	0.000018	mil.navy
994	15795478	350	0.000070	jp.co.rakuten
995	15795389	720	0.000041	com.hilton
996	15795036	1046	0.000031	com.chicagotribune
997	15795024	1486	0.000026	com.cafepress
998	15794283	2336	0.000013	org.dyndns
999	15794113	2271	0.000014	com.teenvogue
1000	15793698	896	0.000035	gov.export

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

3.25 Billion Pages Crawled in July 2018

The crawl archive for July 2018 is now available! The archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2018-30/. It contains 3.25 billion web pages and 255 TiB of uncompressed content, crawled between July 15th and 23th.

The July crawl contains 625 million new URLs, not contained in any crawl archive before. New URLs are “mined” by

extracting and sampling URLs from sitemaps, RSS and Atom feeds if provided by hosts visited in prior crawls. Hosts are selected from the highest-ranking 60 million domains of the Feb/Mar/Apr 2018 webgraph data set
a breadth-first side crawl within a maximum of 6 links (“hops”) away from the home pages of the top 25 million hosts or top 25 million domains of the webgraph dataset
a random sample taken from WAT files of the June crawl

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2018-30/segment.paths.gz	100
WARC files	CC-MAIN-2018-30/warc.paths.gz	64000	61.07
WAT files	CC-MAIN-2018-30/wat.paths.gz	64000	20.93
WET files	CC-MAIN-2018-30/wet.paths.gz	64000	9.08
Robots.txt files	CC-MAIN-2018-30/robotstxt.paths.gz	64000	0.21
Non-200 responses files	CC-MAIN-2018-30/non200responses.paths.gz	64000	2.15
URL index files	CC-MAIN-2018-30/cc-index.paths.gz	302	0.25

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-30/. Also the columnar index has been updated to contain this crawl.

June 2018 Crawl Archive Now Available

The crawl archive for June 2018 is now available! The archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2018-26/. It contains 3.05 billion web pages and 235 TiB of uncompressed content, crawled between June 18th and 25th.

The June crawl contains 700 million new URLs, not contained in any crawl archive before. New URLs are “mined” by

extracting and sampling URLs from sitemaps, RSS and Atom feeds if provided by hosts visited in prior crawls. Hosts are selected from the highest-ranking 60 million domains of the Feb/Mar/Apr 2018 webgraph data set
a breadth-first side crawl within a maximum of 4 links (“hops”) away from the home pages of the top 25 million hosts or top 25 million domains of the webgraph dataset
a random sample taken from WAT files of the May crawl

The remaining URLs (more than 2 billion) have already been included in one of the previous monthly crawl archives and have been stored in our URL database for a later re-fetch – if not marked as duplicates, classified as spam, etc. This huge “collection of bookmarks” dates back multiple years, even back to 2012 when we first received seed donations from Blekko. This month we started to remove old “bookmarks” from our URL database. In the future we’ll remember a URL for only 12 months after seen last as a seed or outlink. On the one hand, we hope to increase the dynamic of the crawls, on the other hand a smaller URL database will save resources.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2018-26/segment.paths.gz	100
WARC files	CC-MAIN-2018-26/warc.paths.gz	64000	56.58
WAT files	CC-MAIN-2018-26/wat.paths.gz	64000	19.19
WET files	CC-MAIN-2018-26/wet.paths.gz	64000	8.37
Robots.txt files	CC-MAIN-2018-26/robotstxt.paths.gz	64000	0.2
Non-200 responses files	CC-MAIN-2018-26/non200responses.paths.gz	64000	1.73
URL index files	CC-MAIN-2018-26/cc-index.paths.gz	302	0.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-26/. Also the columnar index has been updated to contain this crawl.

May 2018 Crawl Archive Now Available

The crawl archive for May 2018 is now available! The archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2018-22/. It contains 2.75 billion web pages and 215 TiB of uncompressed content, crawled between May 20th and 28th.

The May crawl contains 550 million new URLs, not contained in any crawl archive before. New URLs are “mined” by

extracting and sampling URLs from sitemaps, RSS and Atom feeds if provided by hosts visited in prior crawls. Hosts are selected from the highest-ranking 60 million domains of the Feb/Mar/Apr 2018 webgraph data set
a breadth-first side crawl within a maximum of 4 links (“hops”) away from the home pages of the top 40 million hosts or top 40 million domains of the webgraph dataset
a random sample taken from WAT files of the April crawl

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2018-22/segment.paths.gz	100
WARC files	CC-MAIN-2018-22/warc.paths.gz	64000	51.61
WAT files	CC-MAIN-2018-22/wat.paths.gz	64000	17.61
WET files	CC-MAIN-2018-22/wet.paths.gz	64000	7.59
Robots.txt files	CC-MAIN-2018-22/robotstxt.paths.gz	64000	0.19
Non-200 responses files	CC-MAIN-2018-22/non200responses.paths.gz	64000	1.65
URL index files	CC-MAIN-2018-22/cc-index.paths.gz	302	0.21

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-22/. Also the columnar index has been updated to contain this crawl.