2017 – Common Crawl

December 2017 Crawl Archive Now Available

The crawl archive for December 2017 is now available! The archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2017-51/. It contains 2.9 billion web pages and over 240 TiB of uncompressed content.

To improve coverage and freshness we added 650 million new URLs (not contained in any crawl archive before)

sampled from sitemaps if provided by any of the top 80 million hosts taken from the Aug/Sept/Oct 2017 webgraph data set
found by a side crawl within a maximum of 4 links (“hops”) away from the home pages of the top 25 million hosts and domains
a random sample take from WAT files of the November crawl
and the continued donation of URLs from mixnode.com

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2017-51/segment.paths.gz	100
WARC files	CC-MAIN-2017-51/warc.paths.gz	80000	61.2
WAT files	CC-MAIN-2017-51/wat.paths.gz	80000	19.38
WET files	CC-MAIN-2017-51/wet.paths.gz	80000	8.41
Robots.txt files	CC-MAIN-2017-51/robotstxt.paths.gz	80000	0.13
Non-200 responses files	CC-MAIN-2017-51/non200responses.paths.gz	80000	1.47
URL index files	CC-MAIN-2017-51/cc-index.paths.gz	302	0.21

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2017-51/. For more information on working with the URL index, please refer to the previous blog post or the Index Server API. There is also a command-line tool client for common use cases of the URL index.

We are grateful to our friends at mixnode for donating a seed list of 300+ Million URLs to enhance the Common Crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

November 2017 Crawl Archive Now Available

The crawl archive for November 2017 is now available! The archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2017-47/. It contains 3.2 billion web pages and 260 TiB of uncompressed content.

To improve coverage and freshness we added 750 million new URLs (not contained in any crawl archive before)

sampled from sitemaps if provided by any of the top 80 million hosts taken from the Aug/Sept/Oct 2017 webgraph data set
found by a side crawl within a maximum of 4 links (“hops”) away from the home pages of the top 10 million hosts and domains
a random sample take from WAT files of the October crawl
and the continued donation of URLs from mixnode.com

For the first time, during the November crawl, we took measures to actively fight link spam. In the past our policy was to direct the crawl to relevant content, a strategy which avoids spam but does not exclude it. Spam is a valid object of research, and thus spammy content is included in our crawl archives. Spam should not bear on other use cases (mining data for natural language processing) as long as it represents a very low percentage of all documents. However, during this crawl, we faced significant technical challenges caused by link spam:

sitemap spam: every host of one spam cluster announced 20,000 sitemap URLs in its robots.txt (e.g. http://large-enamal-camp-coffee-pot.fyzi.info/robots.txt). The robots.txt files of 125,000 hosts referenced, in total, 2.5 billion sitemaps.
this and a few more clusters caused the unexpectedly large size of the latest host-level web graph

Penalizing spam domains is the easiest way for us to avoid further issues and also to ensure that these spam clusters do not start to dominate future crawls.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2017-47/segment.paths.gz	100
WARC files	CC-MAIN-2017-47/warc.paths.gz	80000	66.17
WAT files	CC-MAIN-2017-47/wat.paths.gz	80000	20.71
WET files	CC-MAIN-2017-47/wet.paths.gz	80000	9.06
Robots.txt files	CC-MAIN-2017-47/robotstxt.paths.gz	80000	0.16
Non-200 responses files	CC-MAIN-2017-47/non200responses.paths.gz	80000	2.33
URL index files	CC-MAIN-2017-47/cc-index.paths.gz	302	0.24

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2017-47/. For more information on working with the URL index, please refer to the previous blog post or the Index Server API. There is also a command-line tool client for common use cases of the URL index.

We are grateful to our friends at mixnode for donating a seed list of 300+ Million URLs to enhance the Common Crawl.

Host- and Domain-Level Web Graphs Aug/Sept/Oct 2017

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of August, September, and October 2017. These graphs, along with ranked lists of hosts and domains, follow the first (February, March, April 2017) and second (May, June, July 2017) web graph releases. Additional information about data formats, the processing pipeline, our objectives, and credits can be found in a prior announcement.

What’s new?

Here is a summary of notable aspects of this web graph release:

Tools and scripts to produce the web graph and rank the graph vertices are released as part of the project “cc-webgraph” on GitHub.
As compared to prior web graphs, two changes are caused by the large size of this host-level graph (5.1 billion hosts):
- the text dump of the graph is split into multiple files;
- there is no page rank calculation at this time. At present, we provide ranking by harmonic centrality, and hope to add page rank values in the upcoming weeks.
- Update Feb 7, 2018: the host-level ranks file now also contains the page ranks. Thanks to Sebastiano Vigna, one of the authors of the WebGraph framework, for the kind support!
For the domain-level graph, we provide ranking by both harmonic centrality and page rank.
The host-level graph contains a significant portion of hosts related to link spam clusters (possibly 50% or more of the hosts). This data set, therefore, is a useful tool for the study of link spam; from it, we have identified 300,000 spam domains. 2.25 billion hosts in the host-level webgraph belong to these domains. However, in the October crawl archive, these domains comprise less than 2% of the crawled HTML pages (56 million pages out of 3.6 billion) and less than 0.3% of the crawled domains (70,000 out of 26 million). We will start to penalize pages from these domains going forward.

Host-level graph

The graph consists of 5.1 billion nodes and 18.8 billion edges. The graph includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. The host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 5.1 billion hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2017-aug-sep-oct/hostgraph/. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2017-aug-sep-oct/hostgraph/ as prefix to access the files from everywhere.

The following files and formats are provided:

Size	File	Description
27.9 GB	vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 48 vertices files
95.2 GB	edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 72 edge files

37.9 GB	bvgraph.graph	graph in BVGraph format
2.0 GB	bvgraph.offsets
2 kB	bvgraph.properties

56.9 GB	bvgraph-t.graph	transpose of the graph (outlinks mapped to inlinks)
6.7 GB	bvgraph-t.offsets
2 kB	bvgraph-t.properties

1 kB	bvgraph.stats	WebGraph statistics

74 GB	ranks.txt.gz	hosts ranked by harmonic centrality and pagerank

To download the graph in text format, you need to download all files listed in the two path listings.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs). The extraction of PLDs is based on the public suffix list from publicsuffix.org. Only “ICANN” domains are accepted; “private” domains are not accepted (cf. section “divisions” in the documentation on publicsuffix.org). For example, foo.blogspot.com and data.commoncrawl.org are not accepted as pay-level domains, they are aggregated, respectively, as the domains blogspot.com, amazonaws.com.

The domain-level graph has 93 million nodes and 1,258 million edges. 60% or 56 million nodes are dangling nodes, the largest strongly connected component covers 31 million or 33% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2017-aug-sep-oct/domaingraph/ resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2017-aug-sep-oct/domaingraph/.

Download files of the Common Crawl Aug/Sept/Oct 2017 domain-level webgraph

Size	File	Description
0.65 GB	vertices.txt.gz	nodes ⟨id, rev host⟩
5.0 GB	edges.txt.gz	edges ⟨from_id, to_id⟩

2.7 GB	bvgraph.graph	graph in BVGraph format
0.09 GB	bvgraph.offsets
2 kB	bvgraph.properties

3.0 GB	bvgraph-t.graph	transpose of the graph (outlinks mapped to inlinks)
0.14 GB	bvgraph-t.offsets
2 kB	bvgraph-t.properties

1 kB	bvgraph.stats	WebGraph statistics

1.9 GB	ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 93 million domains is available for download.

Top 1000 domains ranked by harmonic centrality (Aug/Sept/Oct 2017)

harmonic centrality rank	hc value	page rank	page rank value	reversed domain name
1	24624394	1	0.016162	com.facebook
2	22489142	3	0.009682	com.google
3	22230666	4	0.009431	com.twitter
4	21920250	2	0.011812	com.googleapis
5	21446462	5	0.008148	com.youtube
6	19158272	6	0.005169	org.gmpg
7	18918596	7	0.003893	com.instagram
8	18780074	9	0.003043	com.linkedin
9	18322072	10	0.002469	org.wordpress
10	18163650	20	0.001295	org.wikipedia
11	17977714	16	0.001665	com.pinterest
12	17934400	23	0.001146	com.wordpress
13	17769666	30	0.000940	com.blogspot
14	17754502	15	0.001705	com.apple
15	17668520	19	0.001310	com.gravatar
16	17559336	21	0.001278	com.vimeo
17	17418572	11	0.002239	com.adobe
18	17384530	26	0.001022	com.microsoft
19	17371196	36	0.000819	com.amazon
20	17338218	14	0.001979	com.macromedia
21	17291274	56	0.000617	com.flickr
22	17174674	48	0.000707	com.tumblr
23	17140484	46	0.000716	be.youtu
24	17134992	39	0.000792	gl.goo
25	17053582	27	0.001006	com.paypal
26	16948294	72	0.000500	com.yahoo
27	16940548	73	0.000498	ly.bit
28	16907622	25	0.001066	com.amazonaws
29	16905114	35	0.000836	me.wp
30	16860996	97	0.000263	com.nytimes
31	16822716	38	0.000794	com.github
32	16802262	32	0.000884	io.github
33	16747451	80	0.000376	org.creativecommons
34	16713617	107	0.000223	com.googleusercontent
35	16708091	86	0.000325	com.weebly
36	16691336	110	0.000213	com.blogger
37	16675803	62	0.000602	net.cloudfront
38	16652535	42	0.000756	com.huffingtonpost
39	16626251	78	0.000398	eu.europa
40	16602228	77	0.000421	org.mozilla
41	16600054	154	0.000144	org.wikimedia
42	16598199	130	0.000165	net.slideshare
43	16569992	129	0.000166	com.myspace
44	16564424	22	0.001153	net.fbcdn
45	16558860	79	0.000378	com.medium
46	16542265	31	0.000902	com.cloudflare
47	16520668	66	0.000558	org.w3
48	16520419	44	0.000732	com.cnn
49	16509535	17	0.001657	com.bootstrapcdn
50	16488554	102	0.000239	com.android
51	16467545	192	0.000118	com.photobucket
52	16418457	85	0.000342	com.soundcloud
53	16413233	165	0.000135	com.ebay
54	16409927	214	0.000103	com.about
55	16407381	127	0.000169	org.apache
56	16404812	70	0.000505	com.wp
57	16404114	274	0.000081	gov.nasa
58	16362115	105	0.000230	com.yelp
59	16346871	88	0.000315	co.t
60	16344031	248	0.000090	com.livejournal
61	16332901	145	0.000151	uk.co.bbc
62	16319548	112	0.000198	com.issuu
63	16317299	285	0.000078	com.cnbc
64	16312613	132	0.000164	org.ietf
65	16312538	106	0.000224	com.dropbox
66	16303111	243	0.000091	uk.co.telegraph
67	16302093	300	0.000075	com.appspot
68	16302037	135	0.000158	com.forbes
69	16294126	133	0.000162	net.sourceforge
70	16285841	71	0.000502	com.gstatic
71	16256975	234	0.000093	org.npr
72	16240414	84	0.000342	com.wix
73	16232633	180	0.000126	com.live
74	16224880	143	0.000152	com.spotify
75	16220531	358	0.000061	com.alexa
76	16220429	40	0.000787	com.statcounter
77	16219025	29	0.000952	com.squarespace
78	16216103	52	0.000667	com.mashable
79	16206276	288	0.000077	edu.mit
80	16197041	185	0.000120	com.oracle
81	16196376	50	0.000691	com.bing
82	16184193	146	0.000150	com.imgur
83	16181303	331	0.000067	gov.loc
84	16166139	181	0.000126	com.disqus
85	16162488	47	0.000714	net.akamaihd
86	16160177	184	0.000124	com.typepad
87	16156797	226	0.000096	com.mozilla
88	16149031	266	0.000083	me.about
89	16147987	115	0.000196	com.baidu
90	16144928	104	0.000233	com.reddit
91	16139722	152	0.000145	com.theguardian
92	16139491	431	0.000050	edu.ucla
93	16137233	403	0.000054	com.evernote
94	16133815	117	0.000192	org.archive
95	16129264	153	0.000144	org.gnu
96	16123931	286	0.000078	com.foursquare
97	16123753	325	0.000069	com.buzzfeed
98	16119679	219	0.000099	com.techcrunch
99	16111561	176	0.000128	com.imdb
100	16095828	376	0.000058	com.slate
101	16084632	189	0.000119	edu.stanford
102	16083049	8	0.003658	com.godaddy
103	16081934	348	0.000063	com.mysql
104	16081543	186	0.000120	com.wsj
105	16081479	329	0.000068	com.w3schools
106	16077599	149	0.000148	com.etsy
107	16075216	471	0.000047	uk.ac.ox
108	16070390	235	0.000093	com.nbcnews
109	16067264	271	0.000082	com.wired
110	16058778	223	0.000097	com.tinyurl
111	16052972	241	0.000091	com.cnet
112	16047962	292	0.000077	com.reuters
113	16037078	37	0.000802	com.fb
114	16025773	554	0.000042	edu.princeton
115	16021981	76	0.000429	com.paypalobjects
116	16009160	291	0.000077	com.meetup
117	16007189	228	0.000096	com.businessinsider
118	16007189	169	0.000132	com.digg
119	16000273	290	0.000077	edu.harvard
120	15992746	415	0.000052	org.nodejs
121	15987529	224	0.000097	gov.ca
122	15981478	122	0.000186	com.feedburner
123	15981363	158	0.000142	com.opera
124	15978615	125	0.000171	com.twimg
125	15978398	412	0.000053	edu.berkeley
126	15975429	168	0.000132	com.dribbble
127	15967402	268	0.000082	com.bloomberg
128	15967162	310	0.000071	uk.co.dailymail
129	15962655	284	0.000078	com.msn
130	15962653	445	0.000049	org.worldbank
131	15959796	116	0.000196	com.jquery
132	15959089	81	0.000376	net.doubleclick
133	15952785	68	0.000520	org.schema
134	15946127	308	0.000072	com.bbc
135	15945993	252	0.000090	com.aol
136	15943161	212	0.000105	com.go
137	15941345	254	0.000089	com.usatoday
138	15933198	492	0.000046	com.theverge
139	15931703	277	0.000080	com.ibm
140	15926311	90	0.000284	com.addthis
141	15926283	128	0.000166	com.eventbrite
142	15924888	504	0.000045	co.g
143	15924857	231	0.000095	com.staticflickr
144	15921145	302	0.000074	com.time
145	15919014	210	0.000105	com.surveymonkey
146	15918521	208	0.000106	com.washingtonpost
147	15916491	414	0.000053	org.pbs
148	15916267	332	0.000067	uk.co.blogspot
149	15915021	164	0.000138	com.zendesk
150	15912031	350	0.000062	com.images-amazon
151	15911961	141	0.000153	gov.nih
152	15911136	295	0.000076	com.latimes
153	15907787	270	0.000082	au.com.google
154	15904282	460	0.000047	gov.wa
155	15904089	526	0.000044	org.eclipse
156	15900204	430	0.000050	uk.co.guardian
157	15897748	45	0.000720	me.fb
158	15897738	318	0.000070	com.gmail
159	15891803	505	0.000045	com.pixabay
160	15888886	375	0.000058	org.un
161	15884396	420	0.000051	com.variety
162	15882443	120	0.000187	com.list-manage
163	15881628	560	0.000041	edu.washington
164	15880410	113	0.000198	uk.co.google
165	15877859	642	0.000039	org.chromium
166	15866419	557	0.000042	org.sciencemag
167	15861689	591	0.000040	com.chron
168	15858537	429	0.000050	org.python
169	15856599	813	0.000036	it.scoop
170	15851608	298	0.000076	com.example
171	15844579	888	0.000033	edu.gatech
172	15833612	800	0.000037	com.arstechnica
173	15829201	1091	0.000026	com.panoramio
174	15826638	845	0.000035	edu.illinois
175	15822824	213	0.000104	com.rackcdn
176	15821250	225	0.000096	net.php
177	15819582	313	0.000071	org.acm
178	15816637	461	0.000047	com.scribd
179	15815448	468	0.000047	com.dropboxusercontent
180	15812590	368	0.000059	com.kickstarter
181	15811866	551	0.000042	com.quora
182	15811831	118	0.000188	com.jimdo
183	15810448	215	0.000102	gov.ftc
184	15807293	303	0.000074	com.stackoverflow
185	15801389	194	0.000117	org.drupal
186	15801000	101	0.000239	org.bbb
187	15799715	644	0.000039	com.withgoogle
188	15798347	69	0.000516	com.vk
189	15797645	919	0.000032	edu.utah
190	15795447	395	0.000055	com.theatlantic
191	15792110	406	0.000054	edu.cornell
192	15789477	886	0.000033	com.flipboard
193	15789159	433	0.000050	com.cisco
194	15782967	276	0.000080	fr.free
195	15780935	456	0.000048	com.getpocket
196	15774946	28	0.000982	ru.yandex
197	15774174	202	0.000110	uk.co.amazon
198	15773176	346	0.000063	gov.whitehouse
199	15769731	807	0.000036	edu.columbia
200	15769040	459	0.000047	com.venturebeat
201	15768429	363	0.000060	com.webs
202	15762156	848	0.000035	edu.yale
203	15760325	493	0.000046	com.zdnet
204	15759707	632	0.000039	org.kernel
205	15755584	895	0.000032	com.businessweek
206	15754511	656	0.000038	com.economist
207	15751466	916	0.000032	com.jetbrains
208	15744171	938	0.000031	uk.org.tate
209	15743842	606	0.000040	com.libsyn
210	15742882	142	0.000152	com.windowsphone
211	15742594	447	0.000049	au.gov.nsw
212	15734592	407	0.000053	com.inc
213	15733632	991	0.000029	com.googlecode
214	15732803	187	0.000120	com.mailchimp
215	15729191	55	0.000619	com.people
216	15728871	200	0.000112	net.behance
217	15727994	343	0.000064	com.wiley
218	15726723	359	0.000061	com.wikihow
219	15725173	49	0.000698	com.googletagmanager
220	15724381	95	0.000279	de.google
221	15722940	87	0.000324	com.qq
222	15720539	1011	0.000028	com.storify
223	15720127	612	0.000039	com.box
224	15714779	175	0.000129	jp.co.yahoo
225	15710657	835	0.000035	org.unicode
226	15710342	694	0.000038	com.vogue
227	15709514	532	0.000043	com.samsung
228	15708779	240	0.000092	com.salesforce
229	15704117	453	0.000049	com.deviantart
230	15703085	870	0.000034	com.ecwid
231	15701297	314	0.000071	com.barnesandnoble
232	15699018	372	0.000058	com.oreilly
233	15698383	900	0.000032	org.arxiv
234	15698294	1012	0.000028	edu.rice
235	15697335	111	0.000208	com.shopify
236	15694572	317	0.000070	com.tripod
237	15693750	506	0.000045	com.wikia
238	15692548	327	0.000068	com.dailymotion
239	15692268	1119	0.000025	com.diigo
240	15691299	590	0.000040	com.nationalgeographic
241	15690673	140	0.000153	com.tripadvisor
242	15690554	379	0.000057	com.office
243	15689753	147	0.000149	com.stumbleupon
244	15687628	57	0.000615	com.bleacherreport
245	15686291	263	0.000085	gov.cdc
246	15682833	978	0.000029	com.discogs
247	15679965	960	0.000030	ms.1drv
248	15679700	966	0.000030	com.hbo
249	15678267	868	0.000034	org.eff
250	15678215	470	0.000047	gov.dot
251	15674964	660	0.000038	gov.fcc
252	15674650	705	0.000038	com.tinypic
253	15673888	566	0.000041	com.vice
254	15673605	330	0.000068	com.skype
255	15669074	387	0.000056	com.cbsnews
256	15667839	518	0.000044	com.blackberry
257	15664466	926	0.000031	org.ieee
258	15663927	802	0.000037	com.googleblog
259	15662453	898	0.000032	gov.ky
260	15661561	342	0.000064	int.who
261	15658654	735	0.000037	com.unsplash
262	15658214	737	0.000037	com.indiatimes
263	15655416	482	0.000046	com.git-scm
264	15647888	366	0.000060	com.ted
265	15647883	227	0.000096	com.mapquest
266	15645919	1073	0.000026	com.sublimetext
267	15644112	864	0.000034	gov.mo
268	15643731	570	0.000041	com.foxnews
269	15643086	805	0.000036	com.livestream
270	15635833	360	0.000061	com.springer
271	15635534	811	0.000036	gov.michigan
272	15633180	474	0.000046	com.npmjs
273	15631559	967	0.000030	com.ning
274	15630137	578	0.000041	com.java
275	15630040	546	0.000043	de.blogspot
276	15629832	933	0.000031	gov.mt
277	15626210	972	0.000029	in.blogspot
278	15626170	669	0.000038	com.sfgate
279	15621493	1101	0.000026	com.trello
280	15621434	1076	0.000026	org.amnesty
281	15620756	963	0.000030	com.hatenablog
282	15618941	499	0.000045	com.ft
283	15617683	834	0.000035	com.marketwatch
284	15615909	150	0.000145	com.ytimg
285	15615850	832	0.000036	com.yellowpages
286	15614805	1042	0.000027	edu.psu
287	15614789	862	0.000034	gov.oregon
288	15613749	1049	0.000027	edu.ucsd
289	15612913	1273	0.000022	com.codeplex
290	15612778	922	0.000031	com.ubuntu
291	15611819	1161	0.000024	edu.purdue
292	15610229	405	0.000054	com.goodreads
293	15609712	1896	0.000017	com.fifa
294	15608683	218	0.000099	com.wufoo
295	15607880	275	0.000080	com.hubspot
296	15607085	894	0.000033	gov.nist
297	15605771	345	0.000063	com.sxsw
298	15605337	530	0.000043	gov.state
299	15603425	964	0.000030	edu.upenn
300	15602070	1084	0.000026	com.alibaba
301	15601468	1025	0.000028	com.boston
302	15599982	851	0.000035	com.timeanddate
303	15598270	675	0.000038	org.aarp
304	15597733	195	0.000117	de.amazon
305	15595192	344	0.000063	com.prnewswire
306	15594528	1143	0.000025	com.posterous
307	15590525	795	0.000037	com.atlassian
308	15588435	1298	0.000022	net.comcast
309	15587626	1029	0.000028	edu.wisc
310	15587161	1067	0.000027	com.qz
311	15586347	481	0.000046	com.intel
312	15585861	1188	0.000024	com.instapaper
313	15585841	935	0.000031	com.politico
314	15585103	1385	0.000020	com.ehow
315	15584724	962	0.000030	com.pcmag
316	15584332	1068	0.000026	uk.ac.cam
317	15584130	1210	0.000023	com.vox
318	15579785	399	0.000054	edu.cmu
319	15574628	386	0.000056	com.symantec
320	15573666	385	0.000056	com.snapchat
321	15573517	438	0.000049	com.entrepreneur
322	15573187	497	0.000045	com.nature
323	15571901	792	0.000037	com.weather
324	15569480	968	0.000030	com.gizmodo
325	15566198	974	0.000029	com.nintendo
326	15565922	901	0.000032	com.lifehacker
327	15564424	487	0.000046	com.xrea
328	15563896	159	0.000141	com.weibo
329	15562578	979	0.000029	edu.utexas
330	15562258	264	0.000084	com.getbootstrap
331	15560672	390	0.000056	com.businesswire
332	15557969	1097	0.000026	com.hotmail
333	15554890	821	0.000036	us.imageshack
334	15552184	294	0.000076	net.themeforest
335	15550866	338	0.000065	org.debian
336	15550791	131	0.000164	com.bandcamp
337	15549249	822	0.000036	net.daringfireball
338	15547805	1230	0.000023	edu.northwestern
339	15545647	1134	0.000025	com.discovery
340	15544046	2268	0.000014	com.wikidot
341	15540991	990	0.000029	com.indiegogo
342	15539810	997	0.000028	co.vine
343	15539129	880	0.000033	com.engadget
344	15538263	934	0.000031	com.stackexchange
345	15537892	256	0.000088	com.smugmug
346	15537882	548	0.000043	com.newyorker
347	15536239	1907	0.000017	edu.cuny
348	15535776	1028	0.000028	id.co.blogspot
349	15533870	197	0.000116	com.wixsite
350	15533376	846	0.000035	tv.ustream
351	15533354	882	0.000033	fr.blogspot
352	15532951	119	0.000188	com.constantcontact
353	15532138	629	0.000039	us.mn.state
354	15532035	2187	0.000014	com.twitpic
355	15531885	495	0.000046	com.moz
356	15530153	1513	0.000019	org.khanacademy
357	15529512	333	0.000067	mp.j
358	15529460	952	0.000030	gov.fbi
359	15528428	512	0.000044	com.giphy
360	15527665	1005	0.000028	au.net.abc
361	15526603	1297	0.000022	ie.thejournal
362	15525234	599	0.000040	com.uk
363	15524467	925	0.000031	gov.usgs
364	15524258	876	0.000033	edu.umich
365	15523358	906	0.000032	org.change
366	15522235	625	0.000039	org.redcross
367	15520950	258	0.000087	to.amzn
368	15518905	883	0.000033	gov.maryland
369	15518473	364	0.000060	com.fastcompany
370	15516021	13	0.002020	com.wixstatic
371	15515303	1378	0.000020	org.owasp
372	15514772	1206	0.000023	com.googledrive
373	15513946	986	0.000029	org.plos
374	15513913	1525	0.000019	org.cambridge
375	15513359	869	0.000034	ca.cbc
376	15510150	871	0.000033	com.mlb
377	15509159	1043	0.000027	com.dell
378	15509019	1153	0.000024	com.nba
379	15508147	980	0.000029	com.pingdom
380	15507985	872	0.000033	com.slack
381	15507614	661	0.000038	com.chicagotribune
382	15506774	1826	0.000018	org.gnome
383	15505375	171	0.000131	fr.google
384	15504823	229	0.000095	com.scorecardresearch
385	15504231	874	0.000033	com.searchengineland
386	15501664	1199	0.000023	com.mtv
387	15500750	2182	0.000014	ca.uwaterloo
388	15499743	347	0.000063	org.whatbrowser
389	15498496	1324	0.000021	com.nike
390	15497371	1719	0.000019	net.boingboing
391	15496711	1328	0.000021	edu.jhu
392	15496579	1322	0.000021	edu.academia
393	15494830	853	0.000035	com.sun
394	15493312	652	0.000038	br.com.uol
395	15492698	58	0.000615	me.m
396	15492117	289	0.000077	com.hp
397	15491322	1095	0.000026	com.manta
398	15490293	1831	0.000018	com.blogs
399	15489363	713	0.000037	com.sciencedaily
400	15487527	545	0.000043	com.geocities
401	15485237	816	0.000036	gov.census
402	15484388	33	0.000876	com.messenger
403	15480933	890	0.000033	ca.blogspot
404	15480347	1268	0.000022	com.jigsy
405	15480054	312	0.000071	us.icio
406	15480048	392	0.000056	com.force
407	15478407	829	0.000036	us.pa.state
408	15477943	1149	0.000024	com.target
409	15476319	555	0.000042	uk.co.independent
410	15474849	441	0.000049	com.squareup
411	15474804	1135	0.000025	com.prezi
412	15474620	440	0.000049	gov.noaa
413	15474277	178	0.000127	org.icann
414	15474275	1278	0.000022	com.elpais
415	15473471	995	0.000029	org.altervista
416	15473455	1141	0.000025	edu.si
417	15472680	1394	0.000020	com.gawker
418	15471696	335	0.000066	com.bitly
419	15470693	918	0.000032	fm.last
420	15470552	1181	0.000024	com.hulu
421	15470341	1501	0.000019	tv.periscope
422	15469316	1074	0.000026	edu.umn
423	15468872	1917	0.000017	com.sky
424	15468611	193	0.000117	it.google
425	15465583	1175	0.000024	com.pcworld
426	15462509	1168	0.000024	com.teespring
427	15460362	341	0.000065	com.booking
428	15458683	1272	0.000022	com.upwork
429	15458633	1096	0.000026	de.spiegel
430	15456464	457	0.000047	org.doi
431	15453451	987	0.000029	com.vimeopro
432	15451494	299	0.000075	ly.ow
433	15450701	830	0.000036	com.fastcodesign
434	15450422	137	0.000155	com.eepurl
435	15449841	1001	0.000028	net.researchgate
436	15449833	1302	0.000022	com.salon
437	15449395	961	0.000030	org.unesco
438	15448777	2017	0.000016	com.ikea
439	15447906	1015	0.000028	com.airbnb
440	15447624	2263	0.000014	com.wikispaces
441	15447260	1275	0.000022	edu.uchicago
442	15446865	1169	0.000024	gov.nyc
443	15446299	610	0.000039	es.com.blogspot
444	15445625	849	0.000035	com.bandsintown
445	15443204	689	0.000038	com.cbslocal
446	15442794	2080	0.000015	com.speakerdeck
447	15442432	1815	0.000018	edu.virginia
448	15442282	1813	0.000018	com.csmonitor
449	15441838	1448	0.000020	com.vanityfair
450	15440110	1239	0.000023	com.scientificamerican
451	15439458	936	0.000031	com.thenextweb
452	15439164	1955	0.000017	edu.msu
453	15438677	1326	0.000021	com.freewebs
454	15438547	850	0.000035	com.shutterstock
455	15435909	2395	0.000013	org.nypl
456	15434875	439	0.000049	com.delicious
457	15434850	454	0.000048	com.sciencedirect
458	15433820	913	0.000032	gov.wi
459	15433080	1292	0.000022	edu.unc
460	15432813	336	0.000066	com.technorati
461	15431314	237	0.000093	ca.google
462	15428443	1281	0.000022	com.lulu
463	15428051	1038	0.000028	com.over-blog
464	15427519	1395	0.000020	edu.usc
465	15427022	1189	0.000024	uk.co.theregister
466	15426856	1852	0.000018	uk.ac.ed
467	15426682	520	0.000044	com.githubusercontent
468	15425947	521	0.000044	org.hbr
469	15425391	1066	0.000027	com.istockphoto
470	15425349	653	0.000038	gov.copyright
471	15424494	859	0.000034	com.msdn
472	15424044	2140	0.000015	fr.lefigaro
473	15424014	1362	0.000020	au.com.smh
474	15421339	607	0.000040	org.bitbucket
475	15421040	53	0.000656	com.atdmt
476	15419617	860	0.000034	com.ooyala
477	15419493	1329	0.000021	com.searchenginewatch
478	15418920	136	0.000156	com.xing
479	15418361	843	0.000035	com.psychologytoday
480	15417558	1250	0.000023	com.thinkwithgoogle
481	15416891	2112	0.000015	au.com.theaustralian
482	15416295	2276	0.000014	ca.ualberta
483	15415618	2241	0.000014	com.softpedia
484	15415548	2180	0.000015	de.bild
485	15413194	2053	0.000016	org.moma
486	15412912	1184	0.000024	com.cbs
487	15412881	930	0.000031	com.ggpht
488	15410659	637	0.000039	gov.nps
489	15408840	2212	0.000014	edu.gmu
490	15407643	253	0.000090	com.fc2
491	15407570	2142	0.000015	edu.asu
492	15407046	1402	0.000020	edu.duke
493	15405694	2120	0.000015	com.gamespot
494	15405561	1527	0.000019	com.nfl
495	15403868	1008	0.000028	com.zoho
496	15402737	529	0.000043	cn.com.sina
497	15398565	1858	0.000018	edu.umd
498	15398497	2107	0.000015	com.yfrog
499	15398058	1124	0.000025	com.globo
500	15397979	367	0.000060	com.photoshelter
501	15396257	2252	0.000014	com.mentalfloss
502	15395215	565	0.000041	fr.amazon
503	15394332	2103	0.000015	com.yolasite
504	15393973	1490	0.000019	au.com.blogspot
505	15392266	1257	0.000022	com.billboard
506	15391272	937	0.000031	us.fl.state
507	15390893	465	0.000047	com.fortune
508	15389726	1167	0.000024	com.forrester
509	15388867	174	0.000129	com.youtube-nocookie
510	15387526	1151	0.000024	org.filezilla-project
511	15386771	2191	0.000014	com.pastebin
512	15386754	1265	0.000022	it.blogspot
513	15385485	41	0.000777	org.networkadvertising
514	15384939	1870	0.000018	ru.narod
515	15384906	743	0.000037	com.att
516	15384404	1930	0.000017	com.computerworld
517	15383297	803	0.000037	gov.justice
518	15383264	3100	0.000010	com.xanga
519	15380743	2580	0.000012	org.kiva
520	15379900	2001	0.000016	nl.blogspot
521	15379898	2284	0.000014	com.googlepages
522	15379595	3058	0.000010	cc.co
523	15379486	1479	0.000019	com.colourlovers
524	15379092	183	0.000125	com.nielsen
525	15378656	1427	0.000020	fr.lemonde
526	15378212	1036	0.000028	org.postimg
527	15378043	242	0.000091	es.google
528	15377701	923	0.000031	com.gofundme
529	15377000	2554	0.000012	edu.unl
530	15376832	297	0.000076	nl.google
531	15376563	921	0.000031	org.postgresql
532	15376372	216	0.000102	com.myshopify
533	15375315	602	0.000040	gov.senate
534	15375181	63	0.000602	com.shareaholic
535	15375078	1063	0.000027	com.gigaom
536	15374432	863	0.000034	com.steampowered
537	15373201	361	0.000061	edu.nyu
538	15372956	1841	0.000018	com.techradar
539	15372552	917	0.000032	com.sagepub
540	15371944	638	0.000039	com.quantcast
541	15371504	2453	0.000013	com.citysearch
542	15370774	1360	0.000021	com.semrush
543	15370511	1376	0.000020	com.deezer
544	15370278	485	0.000046	br.com.google
545	15370278	1017	0.000028	com.mckinsey
546	15368963	910	0.000032	com.redhat
547	15368455	958	0.000030	net.azurewebsites
548	15367444	854	0.000034	com.prweb
549	15366697	355	0.000062	io.atom
550	15366349	1125	0.000025	gov.uspto
551	15366186	1643	0.000019	com.ibtimes
552	15366092	2384	0.000013	ly.visual
553	15365287	1356	0.000021	de.zeit
554	15365040	1116	0.000025	com.ycombinator
555	15362746	162	0.000139	org.joomla
556	15362001	373	0.000058	com.naver
557	15361416	1462	0.000020	com.pexels
558	15359899	569	0.000041	com.webmd
559	15358171	1477	0.000019	com.nbc
560	15357605	887	0.000033	net.leadpages
561	15357271	126	0.000170	ru.mail
562	15357180	617	0.000039	org.openstreetmap
563	15356987	1887	0.000017	edu.ufl
564	15356792	1050	0.000027	org.tigris
565	15355908	1177	0.000024	com.smashingmagazine
566	15355445	1355	0.000021	com.ssllabs
567	15354953	1361	0.000021	se.haxx
568	15354144	1307	0.000021	com.econsultancy
569	15353069	98	0.000256	jp.co.google
570	15351292	458	0.000047	gov.fda
571	15349006	1289	0.000022	com.timeout
572	15348715	971	0.000029	gov.nh
573	15348496	2203	0.000014	com.bestbuy
574	15346338	2238	0.000014	com.codecademy
575	15346329	1256	0.000022	com.whitepages
576	15346261	1483	0.000019	com.philly
577	15346115	2130	0.000015	edu.caltech
578	15344481	1918	0.000017	com.deadline
579	15343922	977	0.000029	org.iana
580	15341679	2157	0.000015	com.gq
581	15341443	1201	0.000023	com.bostonglobe
582	15340856	1935	0.000017	com.starwars
583	15339524	2249	0.000014	com.instructables
584	15338407	1359	0.000021	com.xkcd
585	15338034	2459	0.000013	edu.bu
586	15336970	1995	0.000016	edu.indiana
587	15336662	1301	0.000022	com.animoto
588	15336583	1382	0.000020	com.vmware
589	15335607	1270	0.000022	com.zazzle
590	15334096	549	0.000042	gov.hhs
591	15333272	444	0.000049	com.marriott
592	15332387	3068	0.000010	cc.tiny
593	15331948	2716	0.000011	com.sophos
594	15331939	1833	0.000018	com.angelfire
595	15331003	3404	0.000009	com.blog
596	15330787	1152	0.000024	com.com
597	15329429	1051	0.000027	com.superpages
598	15327531	1285	0.000022	com.nokia
599	15326681	1884	0.000018	com.me
600	15325341	2195	0.000014	uk.ac.ucl
601	15324942	196	0.000116	com.googleadservices
602	15324850	239	0.000092	jp.co.amazon
603	15324725	293	0.000077	com.list-manage1
604	15324671	989	0.000029	com.hootsuite
605	15324436	2073	0.000015	edu.umass
606	15320812	2989	0.000010	org.icrc
607	15320731	897	0.000032	com.formstack
608	15320554	1062	0.000027	com.nydailynews
609	15320137	2458	0.000013	org.gimp
610	15320032	2341	0.000013	edu.uiuc
611	15319217	2702	0.000011	com.klout
612	15318501	1919	0.000017	org.aclu
613	15318473	170	0.000131	jp.ameblo
614	15318003	2375	0.000013	com.canalblog
615	15317657	1006	0.000028	org.unicef
616	15317145	388	0.000056	in.co.google
617	15316269	2654	0.000012	com.technet
618	15316214	1881	0.000018	edu.rutgers
619	15314472	1837	0.000018	com.gettyimages
620	15313957	1946	0.000017	com.thedailybeast
621	15313931	2498	0.000012	uk.co.metro
622	15313538	550	0.000042	com.cargocollective
623	15313474	2478	0.000012	gov.cia
624	15313434	875	0.000033	com.pinimg
625	15313153	1333	0.000021	com.brandyourself
626	15313042	488	0.000046	com.nwsource
627	15312657	2682	0.000011	com.nabble
628	15311990	2410	0.000013	com.fiverr
629	15311590	1366	0.000020	com.reference
630	15311208	2243	0.000014	edu.uci
631	15310681	2169	0.000015	com.denverpost
632	15310451	519	0.000044	gov.usa
633	15310254	946	0.000031	org.iso
634	15309261	354	0.000062	com.newrelic
635	15308400	573	0.000041	com.herokuapp
636	15308142	836	0.000035	tv.twitch
637	15308029	1552	0.000019	com.mac
638	15307787	374	0.000058	com.bizjournals
639	15307496	1924	0.000017	org.rubyonrails
640	15307485	727	0.000037	com.usnews
641	15306050	902	0.000032	com.fotolia
642	15305652	3165	0.000009	com.laughingsquid
643	15305451	349	0.000063	com.bigcartel
644	15304790	2010	0.000016	com.ign
645	15304773	3586	0.000008	edu.syr
646	15302826	1960	0.000017	edu.ucdavis
647	15301597	1271	0.000022	com.playstation
648	15301507	369	0.000059	gov.irs
649	15300774	3377	0.000009	com.answers
650	15300201	2642	0.000012	edu.hawaii
651	15300118	2064	0.000016	com.topsy
652	15300099	2316	0.000014	org.videolan
653	15299454	1311	0.000021	com.underconsideration
654	15298943	1941	0.000017	com.investopedia
655	15298571	2725	0.000011	com.hubpages
656	15298437	2699	0.000011	org.greenpeace
657	15298405	2299	0.000014	org.webkit
658	15298109	1304	0.000022	com.accenture
659	15297707	1882	0.000018	com.howstuffworks
660	15297485	992	0.000029	us.ma.state
661	15296995	1089	0.000026	com.netflix
662	15296641	3260	0.000009	com.bigthink
663	15296190	1232	0.000023	com.firefox
664	15296189	449	0.000049	com.angieslist
665	15295815	1218	0.000023	com.techrepublic
666	15295556	3520	0.000008	edu.ucsc
667	15295331	2044	0.000016	ca.utoronto
668	15294445	424	0.000051	com.houzz
669	15293381	4486	0.000007	com.skyrock
670	15293286	2247	0.000014	com.urbandictionary
671	15292886	2815	0.000011	org.donorschoose
672	15292847	1306	0.000021	com.wfaa
673	15292368	1018	0.000028	com.justgiving
674	15292016	1913	0.000017	com.getfirebug
675	15291955	1160	0.000024	com.king5
676	15291413	1877	0.000018	com.us
677	15291199	1225	0.000023	uk.co.mirror
678	15290829	852	0.000035	com.reverbnation
679	15290632	842	0.000035	gov.sba
680	15289434	2049	0.000016	com.ndtv
681	15288265	2970	0.000010	edu.brown
682	15288029	2379	0.000013	org.coursera
683	15287961	2190	0.000014	com.readwriteweb
684	15287447	1611	0.000019	ru.spb
685	15286885	1238	0.000023	ly.snip
686	15285464	1349	0.000021	br.com.blogspot
687	15285356	1947	0.000017	com.ey
688	15285088	416	0.000052	gov.usda
689	15284567	334	0.000066	kr.flic
690	15284507	1856	0.000018	uk.co.thesun
691	15282524	563	0.000041	gov.sec
692	15281588	1140	0.000025	com.merchantcircle
693	15281481	1873	0.000018	edu.tamu
694	15281105	2795	0.000011	com.techsmith
695	15279280	1224	0.000023	net.vnexpress
696	15279179	2269	0.000014	edu.osu
697	15278501	1166	0.000024	com.mixcloud
698	15278297	3369	0.000009	edu.rochester
699	15277931	156	0.000143	jp.ne.hatena
700	15277896	3307	0.000009	org.oxfam
701	15277882	1236	0.000023	com.examiner
702	15277496	1832	0.000018	int.wipo
703	15277085	2087	0.000015	edu.arizona
704	15276954	808	0.000036	com.hostgator
705	15276571	1139	0.000025	jp.blogspot
706	15276267	3446	0.000009	com.allbusiness
707	15275991	1895	0.000017	com.msnbc
708	15275573	469	0.000047	com.wunderground
709	15274401	1438	0.000020	org.craigslist
710	15273961	483	0.000046	gov.ny
711	15273916	1984	0.000016	com.udemy
712	15272770	1007	0.000028	com.cafepress
713	15271907	1325	0.000021	ca.kijiji
714	15270985	2469	0.000013	com.tutsplus
715	15269961	2591	0.000012	com.wolfram
716	15269647	1818	0.000018	de.welt
717	15268436	2468	0.000013	com.makezine
718	15267469	2907	0.000010	com.friendfeed
719	15267315	847	0.000035	net.openid
720	15267116	2491	0.000012	it.repubblica
721	15266908	1380	0.000020	it.justpaste
722	15264794	1039	0.000028	com.500px
723	15264333	1369	0.000020	com.bizcommunity
724	15264318	2172	0.000015	com.pbworks
725	15263255	1277	0.000022	com.gumroad
726	15262891	1196	0.000023	com.mysanantonio
727	15262769	1389	0.000020	com.foxbusiness
728	15262355	1524	0.000019	com.canva
729	15261885	2337	0.000013	com.glamour
730	15260928	3282	0.000009	com.avast
731	15260868	1129	0.000025	de.heise
732	15260080	3333	0.000009	com.news
733	15260065	3269	0.000009	gd.is
734	15258443	2043	0.000016	uk.ac.lse
735	15258381	2461	0.000013	com.tv
736	15258369	428	0.000050	com.typeform
737	15257788	2295	0.000014	com.theonion
738	15257354	1455	0.000020	io.codepen
739	15257242	370	0.000059	com.adweek
740	15257204	957	0.000030	org.mediawiki
741	15256000	2255	0.000014	org.acs
742	15255931	1902	0.000017	com.getsatisfaction
743	15254860	2733	0.000011	com.expedia
744	15254608	260	0.000086	com.windows
745	15254582	496	0.000046	gov.ed
746	15253723	1295	0.000022	com.sohu
747	15252995	1058	0.000027	org.oecd
748	15252244	1555	0.000019	org.ap
749	15252208	1241	0.000023	com.city-data
750	15251628	443	0.000049	gov.epa
751	15250980	3583	0.000008	edu.missouri
752	15249746	2189	0.000014	com.ask
753	15249300	1880	0.000018	com.norton
754	15249295	2164	0.000015	edu.ncsu
755	15248729	817	0.000036	net.launchpad
756	15248611	3534	0.000008	com.scobleizer
757	15248299	912	0.000032	com.redbubble
758	15245757	1547	0.000019	com.newsweek
759	15245385	1521	0.000019	com.blogtalkradio
760	15245282	1287	0.000022	com.garmin
761	15245212	1027	0.000028	com.hollywoodreporter
762	15245183	1637	0.000019	com.yandex
763	15245175	2278	0.000014	com.foodnetwork
764	15243922	2092	0.000015	com.si
765	15243100	2194	0.000014	com.mediabistro
766	15242393	1183	0.000024	com.mediafire
767	15242360	134	0.000161	com.parallels
768	15242173	2110	0.000015	com.delta
769	15241483	337	0.000066	org.microformats
770	15241345	1442	0.000020	th.co.lazada
771	15241235	2123	0.000015	com.ford
772	15240434	1865	0.000018	com.allthingsd
773	15240082	1299	0.000022	com.technologyreview
774	15239583	2193	0.000014	com.freep
775	15239251	356	0.000062	com.teamviewer
776	15239065	2652	0.000012	edu.pitt
777	15238897	207	0.000107	org.purl
778	15238532	1850	0.000018	org.openoffice
779	15237565	2457	0.000013	com.podbean
780	15237162	371	0.000059	com.nypost
781	15237103	2429	0.000013	uk.co.timesonline
782	15235925	2074	0.000015	com.real
783	15235282	2363	0.000013	com.oxforddictionaries
784	15234189	2331	0.000013	ch.ethz
785	15233592	3334	0.000009	org.notepad-plus-plus
786	15233550	2292	0.000014	com.nhl
787	15233542	2449	0.000013	net.jsfiddle
788	15232983	3327	0.000009	net.battle
789	15232781	2555	0.000012	com.chrome
790	15232780	1209	0.000023	org.iihs
791	15232749	1180	0.000024	org.fao
792	15232078	2280	0.000014	org.gutenberg
793	15231865	1939	0.000017	com.ssrn
794	15231276	3438	0.000009	net.wordle
795	15231085	1441	0.000020	net.brownbook
796	15230217	3648	0.000008	com.dreamstime
797	15229969	3435	0.000009	be.blogspot
798	15229857	953	0.000030	us.nm.state
799	15229697	1901	0.000017	org.weforum
800	15229356	2346	0.000013	org.lifehack
801	15229182	3444	0.000009	edu.umaryland
802	15228293	528	0.000043	com.list-manage2
803	15227693	382	0.000057	net.windows
804	15227013	2496	0.000012	edu.vt
805	15226905	3239	0.000009	com.voanews
806	15225763	2425	0.000013	com.shutterfly
807	15225225	2762	0.000011	edu.ucsf
808	15225176	1962	0.000017	com.starwoodhotels
809	15224438	435	0.000050	com.ea
810	15223961	911	0.000032	com.gartner
811	15223450	4115	0.000007	it.libero
812	15222361	826	0.000036	org.w
813	15222226	1879	0.000018	org.slashdot
814	15222185	1467	0.000019	org.cancer
815	15221037	396	0.000055	jp.ne.sakura
816	15220922	1358	0.000021	org.json
817	15220813	1093	0.000026	com.bufferapp
818	15219986	1357	0.000021	com.unity3d
819	15219634	2215	0.000014	net.earthlink
820	15219035	2059	0.000016	com.digitaltrends
821	15218571	2158	0.000015	uk.co.huffingtonpost
822	15217732	2950	0.000010	edu.tufts
823	15217533	1420	0.000020	com.wsoctv
824	15217060	2435	0.000013	com.thefreedictionary
825	15216593	353	0.000062	net.freenode
826	15216469	1337	0.000021	com.kgw
827	15215867	1871	0.000018	gov.uscourts
828	15215804	1108	0.000025	com.steamcommunity
829	15214737	2050	0.000016	com.kaspersky
830	15214400	1428	0.000020	com.ripple
831	15214306	1413	0.000020	mil.navy
832	15212636	3281	0.000009	edu.emory
833	15212571	858	0.000034	com.linksynergy
834	15212046	3884	0.000008	com.4shared
835	15211666	1392	0.000020	com.chamberofcommerce
836	15211429	2271	0.000014	com.chronicle
837	15211268	1071	0.000026	be.google
838	15211197	3057	0.000010	com.squidoo
839	15211139	3033	0.000010	com.esquire
840	15211059	2122	0.000015	com.azcentral
841	15210567	1418	0.000020	ly.list
842	15210307	2364	0.000013	com.sony
843	15210129	909	0.000032	com.deloitte
844	15209068	628	0.000039	com.webnode
845	15208892	1367	0.000020	net.yahoo
846	15208875	2952	0.000010	com.threadless
847	15208572	1411	0.000020	org.whatwg
848	15208528	4053	0.000007	edu.udel
849	15208357	381	0.000057	ru.vkontakte
850	15208257	3102	0.000010	ca.mcgill
851	15208136	1261	0.000022	com.zillow
852	15207337	2744	0.000011	edu.uga
853	15207332	985	0.000029	com.163
854	15206915	1150	0.000024	com.techtarget
855	15206493	2281	0.000014	org.wnyc
856	15206087	1106	0.000025	com.yoast
857	15205825	3036	0.000010	pt.sapo
858	15205659	1992	0.000016	org.jstor
859	15205644	1158	0.000024	com.fedex
860	15204700	1352	0.000021	org.oxfordjournals
861	15204575	1347	0.000021	com.thomsonreuters
862	15204520	1182	0.000024	org.gnupg
863	15204413	1965	0.000017	org.ampproject
864	15204277	2262	0.000014	com.css-tricks
865	15203728	389	0.000056	com.monster
866	15203287	633	0.000039	com.cdbaby
867	15202918	1194	0.000023	com.business2community
868	15202339	7097	0.000005	com.zimbio
869	15202259	3395	0.000009	com.macrumors
870	15201640	3010	0.000010	edu.dartmouth
871	15201627	4114	0.000007	gr.blogspot
872	15201276	589	0.000040	com.friendster
873	15201171	2536	0.000012	org.computer
874	15200999	1053	0.000027	gov.dhs
875	15200758	1365	0.000020	com.bmj
876	15200264	1228	0.000023	com.nymag
877	15200256	383	0.000057	com.youku
878	15200090	2089	0.000015	org.npmjs
879	15199967	2282	0.000014	ca.ubc
880	15199583	1305	0.000021	com.oup
881	15199125	3791	0.000008	org.laptop
882	15198697	3862	0.000008	org.wikiquote
883	15198650	2147	0.000015	com.gopro
884	15198375	2014	0.000016	me.flavors
885	15197141	1387	0.000020	com.hotfrog
886	15196701	2635	0.000012	com.aviary
887	15196573	1170	0.000024	com.googleapps
888	15196261	2217	0.000014	com.popsugar
889	15196007	2051	0.000016	com.patch
890	15195975	1451	0.000020	com.communitywalk
891	15195769	4424	0.000007	com.pbase
892	15195511	1048	0.000027	us.wi.state
893	15195187	3640	0.000008	tv.wat
894	15194595	2038	0.000016	com.autodesk
895	15194091	3050	0.000010	edu.oregonstate
896	15193999	160	0.000141	info.aboutads
897	15193540	1331	0.000021	gov.uscis
898	15193209	2352	0.000013	com.harpercollins
899	15193130	3237	0.000009	com.blackgirlscode
900	15192484	4626	0.000006	com.boredpanda
901	15192375	583	0.000040	com.hilton
902	15192287	6829	0.000005	com.weheartit
903	15191839	2162	0.000015	com.ifttt
904	15191738	5780	0.000005	com.rebelmouse
905	15191009	3197	0.000009	int.esa
906	15190294	2155	0.000015	org.raspberrypi
907	15190293	3941	0.000008	com.pixlr
908	15190078	799	0.000037	com.adage
909	15189887	2100	0.000015	com.netvibes
910	15189887	3173	0.000009	edu.iastate
911	15189680	480	0.000046	com.taobao
912	15189297	1336	0.000021	com.today
913	15188664	3488	0.000009	edu.usf
914	15188658	2009	0.000016	com.thestar
915	15188260	2205	0.000014	org.r-project
916	15188008	1406	0.000020	com.sap
917	15187715	3040	0.000010	org.charitywater
918	15187204	2483	0.000012	tv.blip
919	15187084	2117	0.000015	com.livestrong
920	15187046	2321	0.000013	com.britannica
921	15185988	3408	0.000009	au.com.theage
922	15185981	1561	0.000019	com.mercurynews
923	15185466	1897	0.000017	com.foxsports
924	15185012	1296	0.000022	org.apa
925	15183779	3348	0.000009	tv.arte
926	15183521	3302	0.000009	com.bhphotovideo
927	15183491	1266	0.000022	com.comodo
928	15183363	801	0.000037	com.brightcove
929	15182962	3483	0.000009	uk.org.bhf
930	15182523	4155	0.000007	mx.unam
931	15181553	1111	0.000025	com.convertkit
932	15181533	2207	0.000014	com.geekwire
933	15181414	2613	0.000012	nl.xs4all
934	15181217	2852	0.000011	com.newsvine
935	15180790	2472	0.000013	com.w3techs
936	15180377	2192	0.000014	com.newscientist
937	15179981	3151	0.000009	com.popsci
938	15179907	1835	0.000018	gov.in
939	15179435	1195	0.000023	com.gotomeeting
940	15179161	2424	0.000013	com.fox
941	15178849	1340	0.000021	com.rollingstone
942	15177895	1346	0.000021	co.angel
943	15177525	2544	0.000012	com.avg
944	15177088	109	0.000213	com.yimg
945	15173343	464	0.000047	com.zenfolio
946	15172342	2366	0.000013	org.kde
947	15172015	939	0.000031	uk.co.eventbrite
948	15171872	4082	0.000007	net.minecraft
949	15171718	6627	0.000005	fr.unblog
950	15171446	2616	0.000012	com.ezinearticles
951	15171116	1956	0.000017	com.macworld
952	15171092	1026	0.000028	com.uservoice
953	15170472	3519	0.000008	cc.arduino
954	15170318	2436	0.000013	net.oauth
955	15170061	446	0.000049	com.gotowebinar
956	15170030	2607	0.000012	us.zoom
957	15169703	1972	0.000016	com.wetransfer
958	15169221	597	0.000040	ru.google
959	15169001	450	0.000049	com.meyerweb
960	15168876	3420	0.000009	org.britishcouncil
961	15168688	3394	0.000009	net.deviantart
962	15168623	1172	0.000024	com.intuit
963	15168438	1088	0.000026	com.americanexpress
964	15168426	2008	0.000016	com.amzn
965	15167776	2060	0.000016	net.speedtest
966	15167505	3206	0.000009	se.blogspot
967	15167365	409	0.000053	com.mapbox
968	15167024	3694	0.000008	edu.uky
969	15166707	1022	0.000028	gov.va
970	15166663	4038	0.000007	com.secondlife
971	15166551	2061	0.000016	com.comcast
972	15165934	1496	0.000019	com.espn
973	15165734	1054	0.000027	com.walmart
974	15165621	4169	0.000007	org.edublogs
975	15165053	2128	0.000015	com.mydomain
976	15164578	982	0.000029	us.tx.state
977	15164372	4160	0.000007	ca.yorku
978	15164294	1056	0.000027	jp.ne.goo
979	15164103	812	0.000036	com.emarketer
980	15163086	1822	0.000018	uk.gov.legislation
981	15162903	3668	0.000008	tt.db
982	15162643	5940	0.000005	edu.ua
983	15162357	3391	0.000009	com.mlive
984	15162205	1348	0.000021	com.arabianbusiness
985	15162134	1942	0.000017	org.c-span
986	15161315	1916	0.000017	uk.co.wired
987	15161076	2482	0.000012	org.worldcat
988	15159608	2314	0.000014	net.daum
989	15159215	2631	0.000012	com.chow
990	15159182	3301	0.000009	org.ibiblio
991	15159041	1223	0.000023	gov.archives
992	15158591	2539	0.000012	com.univision
993	15158148	1938	0.000017	com.aliexpress
994	15157040	4377	0.000007	com.vodpod
995	15156983	1853	0.000018	com.merriam-webster
996	15156908	2427	0.000013	org.hrc
997	15156319	2516	0.000012	com.crunchbase
998	15156162	3351	0.000009	fr.cnrs
999	15155947	1431	0.000020	com.ktvb
1000	15155558	326	0.000068	pl.google

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

October 2017 Crawl Archive Now Available

The crawl archive for October 2017 is now available! The archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2017-43/. It contains 3.65 billion web pages and over 300 TiB of uncompressed content.

To improve coverage and freshness we added over 900 million new URLs (not contained in any crawl archive before):

350 million URLs are a random sample extracted from sitemaps if provided by any of the top 80 million hosts taken from the May/June/July 2017 webgraph data set
250 million URLs were found by a side crawl within a maximum of 3 links (“hops”) away from the home pages of the top 80 million hosts
150 million URLs are randomly chosen from WAT files of the September crawl
180 million URLs are links donated by mixnode.com

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2017-43/segment.paths.gz	100
WARC files	CC-MAIN-2017-43/warc.paths.gz	89100	84.9
WAT files	CC-MAIN-2017-43/wat.paths.gz	89100	24.33
WET files	CC-MAIN-2017-43/wet.paths.gz	89100	10.58
Robots.txt files	CC-MAIN-2017-43/robotstxt.paths.gz	89100	0.17
Non-200 responses files	CC-MAIN-2017-43/non200responses.paths.gz	89100	2.62
URL index files	CC-MAIN-2017-43/cc-index.paths.gz	302	0.28

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2017-43/. For more information on working with the URL index, please refer to the previous blog post or the Index Server API. There is also a command-line tool client for common use cases of the URL index.

We are grateful to our friends at mixnode for donating a seed list of 200+ Million URLs to enhance the Common Crawl.

September 2017 Crawl Archive Now Available

The crawl archive for September 2017 is now available! The archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2017-39/. It contains 3.01 billion web pages and over 250 TiB of uncompressed content.

To improve coverage and freshness we added one billion new URLs (not contained in any crawl archive before):

300 million URLs are a random sample extracted from sitemaps if provided by any of the top 60 million hosts taken from the May/June/July 2017 webgraph data set
500 million URLs were found by a side crawl within a maximum of 3 links (“hops”) away from the home pages of the top 60 million hosts and from a list of university domains collected by a Common Crawl user
200 million URLs are randomly chosen from WAT files of the August crawl

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2017-39/segment.paths.gz	100
WARC files	CC-MAIN-2017-39/warc.paths.gz	72000	59.13
WAT files	CC-MAIN-2017-39/wat.paths.gz	72000	20.1
WET files	CC-MAIN-2017-39/wet.paths.gz	72000	8.86
Robots.txt files	CC-MAIN-2017-39/robotstxt.paths.gz	72000	0.16
Non-200 responses files	CC-MAIN-2017-39/non200responses.paths.gz	72000	2.07
URL index files	CC-MAIN-2017-39/cc-index.paths.gz	302	0.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2017-39/. For more information on working with the URL index, please refer to the previous blog post or the Index Server API. There is also a command-line tool client for common use cases of the URL index.

August 2017 Crawl Archive Now Available

The crawl archive for August 2017 is now available! The archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2017-34/. It contains 3.28 billion+ web pages and over 280 TiB of uncompressed content.

To improve coverage and freshness we used the top 50 million ranked hosts from the May/June/July 2017 webgraph data set and added over 800 million new URLs (not contained in any crawl archive before), of which

300 million URLs were found by a side crawl within a maximum of 3 links (“hops”) away from the home pages of the top 40 million hosts;
525 million URLs are a random sample extracted from sitemaps (if provided by any of the top 50 million hosts).

The following improvements affect the WAT and WET extraction:

improved spacing / word segmentation in WET extracts, see issue #13
extract URLs from JavaScript code in onClick attributes (issue #8)

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files. By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2017-34/segment.paths.gz	100
WARC files	CC-MAIN-2017-34/warc.paths.gz	72000	65.14
WAT files	CC-MAIN-2017-34/wat.paths.gz	72000	22.18
WET files	CC-MAIN-2017-34/wet.paths.gz	72000	9.81
Robots.txt files	CC-MAIN-2017-34/robotstxt.paths.gz	72000	0.12
Non-200 responses files	CC-MAIN-2017-34/non200responses.paths.gz	72000	1.49
URL index files	CC-MAIN-2017-34/cc-index.paths.gz	302	0.24

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2017-34/. For more information on working with the URL index, please refer to the previous blog post or the Index Server API. There is also a command-line tool client for common use cases of the URL index.

Now Available: Host- and Domain-Level Web Graphs

We are pleased to announce the release of host-level and domain-level web graphs based on the published crawls of May, June, and July 2017. These graphs, along with ranked lists of hosts and domains, follow on our first host-level web graph (February, March, April 2017). Detailed information about the data formats, the processing pipeline, our objectives, and credits can be found in the prior announcement.

Host-level graph

The graph consists of 1.3 billion nodes and 5.25 billion edges. The graph includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. The host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 1.3 billion hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2017-may-jun-jul/hostgraph/. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2017-may-jun-jul/hostgraph/ as prefix to access the files from everywhere.

The following files and formats are provided:

Size	File	Description
7.8 GB	vertices.txt.gz	nodes ⟨id, rev host⟩
20.4 GB	edges.txt.gz	edges ⟨from_id, to_id⟩

10.0 GB	bvgraph.graph	graph in BVGraph format
0.59 GB	bvgraph.offsets
2 kB	bvgraph.properties

14.5 GB	bvgraph-t.graph	transpose of the graph (outlinks mapped to inlinks)
1.6 GB	bvgraph-t.offsets
2 kB	bvgraph-t.properties

1 kB	bvgraph.stats	WebGraph statistics

18.9 GB	ranks.txt.gz	harmonic centrality and pagerank

Note that differences in the rankings and the structure of the web graph are due our objective to make monthly crawls more diverse and to reduce the overlap between consecutive crawls. During both February/March/April and May/June/July we crawled about 9 billion pages. As an indicator of less overlap the number of unique URLs increased from 5.0 to 6.2 billion and the number of unique hosts went up from 70 to 90 million (or from 65 to 82 million with leading “www.” removed). The largest strongly connected component contains now 59 million nodes/hosts (45 million in the February/March/April graph). However, the May/June/July host-level graph has doubled its size in terms of edges and more than tripled in terms of nodes. This growth is caused by a significant increase in the number of dangling nodes. 1.2 billion dangling nodes provide a solid foundation to extend the next crawls but we need a closer look at the distribution of these hosts among domains and TLDs.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs). The extraction of PLDs is based on the public suffix list from publicsuffix.org. Only “ICANN” domains are accepted, “private” domains are not (cf. section “divisions” in the documentation on publicsuffix.org). In short, foo.blogspot.com and data.commoncrawl.org are not accepted as pay-level domains, they are aggregated as domains blogspot.com resp. amazonaws.com.

The domain-level graph has 91 million nodes and 1,071 million edges. 55% are dangling nodes, the largest strongly connected component covers 30 million or 33% of the nodes.

All files related to the domain graph are placed on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2017-may-jun-jul/domaingraph/ resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2017-may-jun-jul/domaingraph/.

Download files of the Common Crawl May/June/July 2017 domain-level webgraph

Size	File	Description
0.63 GB	vertices.txt.gz	nodes ⟨id, rev host⟩
4.3 GB	edges.txt.gz	edges ⟨from_id, to_id⟩

2.4 GB	bvgraph.graph	graph in BVGraph format
0.09 GB	bvgraph.offsets
2 kB	bvgraph.properties

2.6 GB	bvgraph-t.graph	transpose of the graph (outlinks mapped to inlinks)
0.13 GB	bvgraph-t.offsets
2 kB	bvgraph-t.properties

1 kB	bvgraph.stats	WebGraph statistics

1.8 GB	ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 91 million domains is available for download.

Top 1000 domains ranked by harmonic centrality (May/June/July 2017)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	24989952	1	0.0155264576161686	com.facebook
2	22460880	3	0.00866038900847366	com.twitter
3	22097514	2	0.0128827315785546	com.googleapis
4	21301616	4	0.00741843063139714	com.youtube
5	21234402	5	0.00706371663388533	com.google
7	18990882	6	0.00545683515949591	org.gmpg
8	18904882	10	0.00280210492427644	com.linkedin
9	18843450	9	0.00307727244449465	com.instagram
10	18259830	25	0.00126167740795298	org.wikipedia
11	18053094	38	0.00101731276962237	com.blogspot
12	17970588	34	0.00109405966554137	com.wordpress
13	17919956	22	0.00141960022500196	com.pinterest
14	17835014	16	0.0020636325734346	com.apple
15	17697268	14	0.00222054582589118	org.wordpress
17	17561416	12	0.00240805753831021	com.macromedia
18	17508598	32	0.00111771803587573	com.gravatar
19	17446398	55	0.000661340427978196	be.youtu
20	17384778	52	0.000701194639286359	com.amazon
21	17353492	17	0.00187094862083697	com.adobe
22	17343286	56	0.000637906122981968	com.flickr
23	17323658	47	0.000801430611613196	com.vimeo
24	17279624	49	0.000729089638101731	gl.goo
25	17250688	42	0.000933859058158277	com.paypal
26	17229298	46	0.000826236952032279	com.microsoft
28	17103198	70	0.000474840687818822	ly.bit
29	17089526	66	0.000507632417477531	com.tumblr
30	17033012	50	0.000719273960345051	com.github
31	16996118	43	0.000881255954914393	com.amazonaws
32	16946912	77	0.000406272523681585	org.creativecommons
34	16863562	67	0.000501297307358693	com.yahoo
35	16862378	93	0.000299154154651788	org.mozilla
37	16812384	83	0.00037282011840036	org.w3
38	16752483	125	0.000207624449137057	com.weebly
39	16729464	115	0.000221622522886096	com.googleusercontent
40	16717669	118	0.000215230097627401	com.blogger
41	16697624	153	0.000171757229153436	com.myspace
44	16646934	185	0.000141243073946399	org.wikimedia
46	16644900	79	0.000396589141351638	com.medium
47	16631223	75	0.000433568676537	io.github
48	16612200	99	0.000268805299302926	com.android
49	16590272	18	0.00162368845735663	com.bootstrapcdn
50	16558434	166	0.000157623382042305	org.apache
51	16525428	229	0.000115801379415941	com.photobucket
52	16504426	195	0.000131254056913804	com.businessinsider
55	16476553	304	8.18438859138264e-05	com.ebay
56	16469721	137	0.000187664601325017	com.issuu
57	16460853	37	0.00102123707091179	com.bing
58	16452729	182	0.000142450905972814	com.imdb
59	16427950	82	0.000374838485018944	eu.europa
60	16411357	48	0.000746154314145809	com.statcounter
61	16390338	245	0.000101622215034005	com.appspot
62	16379952	209	0.000124338192701991	com.about
63	16373742	140	0.00018454904883009	com.nytimes
64	16363821	352	7.06416235739108e-05	gov.nasa
65	16353372	59	0.000571194421273779	com.cloudflare
66	16346221	13	0.00239759215021479	net.fbcdn
67	16338312	44	0.000857327043073336	me.wp
68	16326390	71	0.000465848580411827	com.gstatic
69	16320041	347	7.16053406333361e-05	com.theverge
70	16319414	122	0.000209662262430235	com.yelp
72	16279524	286	8.69073948101929e-05	org.npr
73	16272083	325	7.56332706724679e-05	com.googleblog
74	16256304	290	8.48972545255239e-05	me.about
75	16248199	199	0.000128879026140158	org.ietf
76	16241255	272	9.24645494623427e-05	com.mozilla
77	16240479	156	0.00016832163499487	org.gnu
78	16239722	92	0.000305401671067505	co.t
79	16239387	216	0.000120371938835806	com.oracle
80	16232526	228	0.000116114540948403	com.typepad
81	16231084	453	5.44944627802185e-05	edu.ucla
82	16227427	27	0.00121558207856905	net.akamaihd
84	16214309	232	0.000114818535982162	uk.co.bbc
85	16214235	136	0.000189983194885236	com.jquery
86	16211814	221	0.000118993796467497	com.imgur
87	16197420	472	5.2567183006594e-05	edu.princeton
88	16196855	336	7.32728429381573e-05	gov.noaa
89	16191891	74	0.000440473851712633	org.schema
90	16190145	178	0.000144231760365021	net.slideshare
91	16190073	62	0.000564081957805812	net.cloudfront
92	16183730	284	8.92999121057943e-05	com.wsj
93	16175963	188	0.000140208643596679	com.forbes
94	16172502	76	0.00041917417746326	com.paypalobjects
95	16170294	121	0.000212087858171349	com.soundcloud
96	16165048	200	0.000128726042287979	com.spotify
97	16161367	213	0.000122378879977989	edu.stanford
98	16158114	202	0.000127454157734194	com.disqus
99	16156608	481	5.14517415590491e-05	com.mysql
100	16148691	167	0.000157054492258209	com.dropbox
101	16146672	276	9.11987772125017e-05	com.tinyurl
102	16141698	139	0.000184631238862252	com.constantcontact
103	16141420	331	7.44922001669146e-05	edu.mit
104	16139995	273	9.20851285715325e-05	com.nbcnews
106	16134891	103	0.000249718966343057	com.wix
107	16131390	440	5.63178297481015e-05	com.googlecode
108	16126524	187	0.00014024908309338	com.theguardian
109	16123314	95	0.000285005386439374	com.huffingtonpost
111	16112941	353	7.01996718671122e-05	gov.loc
112	16105005	155	0.000170093991134689	net.sourceforge
113	16101600	401	6.14791022911939e-05	org.python
114	16088860	379	6.60560437018202e-05	com.alexa
115	16088645	410	5.97561569096326e-05	co.g
116	16079028	252	9.88994061572069e-05	com.go
118	16077453	364	6.82295063450322e-05	com.foursquare
121	16066757	45	0.000829281063065997	com.squarespace
122	16061946	393	6.23091312037812e-05	com.sun
124	16058079	442	5.61406400406246e-05	com.withgoogle
125	16057273	260	9.56090724221574e-05	com.washingtonpost
127	16035956	63	0.000537473792465926	com.vk
128	16035411	86	0.000352598947727924	net.doubleclick
129	16035239	489	5.06581788427578e-05	org.chromium
130	16028498	584	4.26316314674836e-05	edu.gatech
132	16027022	416	5.92122336053588e-05	au.gov.nsw
133	16025684	126	0.000207293393080606	com.reddit
134	16025106	509	4.91710961371229e-05	org.sciencemag
135	16024041	389	6.27321618308479e-05	org.nodejs
136	16023948	605	4.19017591329544e-05	edu.cuny
138	16012899	133	0.000193040982037554	com.feedburner
140	16000132	538	4.57515027163085e-05	edu.illinois
141	15999778	499	4.96494716668815e-05	edu.berkeley
142	15998912	399	6.16482775444537e-05	au.com.google
144	15987228	506	4.93463645755921e-05	com.pixabay
145	15976013	258	9.60743680804667e-05	uk.co.amazon
146	15972907	111	0.000230813335083292	com.cnn
148	15967558	544	4.54457479041577e-05	com.libsyn
149	15952405	371	6.7120499269601e-05	com.wired
150	15949618	259	9.59425814050923e-05	com.surveymonkey
151	15946748	383	6.53396206082701e-05	uk.co.dailymail
152	15946483	452	5.45761550958247e-05	com.variety
153	15930068	24	0.0013474030796477	com.fb
154	15925842	215	0.000120638726611947	com.etsy
155	15924889	391	6.24685427611247e-05	uk.co.blogspot
156	15912395	311	7.90671790542201e-05	com.dailymotion
157	15906905	211	0.000123239102782434	com.digg
158	15901097	181	0.000142453252268271	gov.nih
159	15894206	370	6.75180807177585e-05	com.reuters
160	15886820	701	3.89256402003242e-05	edu.columbia
162	15872957	787	3.751174278726e-05	com.nike
163	15868055	227	0.000116596591295635	com.live
165	15864752	354	7.01156504511129e-05	com.livejournal
166	15864738	657	3.96129599599408e-05	com.sap
167	15862665	664	3.95013568833938e-05	com.discogs
168	15857975	282	8.9964923414358e-05	com.aol
169	15855750	445	5.5748966776129e-05	org.mediawiki
170	15845942	29	0.00114281853454816	me.fb
171	15842947	351	7.07813033957103e-05	com.usatoday
172	15842283	633	4.05686573540759e-05	edu.utah
173	15837138	549	4.51699741643719e-05	net.daringfireball
174	15836674	496	4.9999960509278e-05	org.eclipse
175	15834873	368	6.76642625755714e-05	com.npmjs
176	15833638	550	4.49681211898885e-05	com.netflix
177	15832429	337	7.30215403346833e-05	com.tripod
178	15831038	878	3.3656387101054e-05	com.diigo
180	15817871	804	3.67639700948407e-05	uk.org.tate
181	15816200	369	6.75600758631612e-05	edu.harvard
182	15815422	469	5.3001847136341e-05	com.time
183	15810344	823	3.59360502579382e-05	id.co.blogspot
184	15809026	114	0.000221797473311586	com.mashable
185	15807460	807	3.6447534726932e-05	com.hbo
187	15802457	769	3.84109849525509e-05	com.chron
188	15801557	778	3.79843092102341e-05	edu.washington
189	15795011	426	5.81220523939956e-05	com.gmail
190	15793121	113	0.000225135156397275	com.jimdo
191	15792847	355	7.00169347047329e-05	com.meetup
192	15792500	629	4.06926572016734e-05	org.ampproject
193	15791399	596	4.21539576611305e-05	com.jetbrains
194	15790948	322	7.68171654135211e-05	com.bloomberg
195	15789362	152	0.000173965702149204	de.google
196	15787832	288	8.59046288833472e-05	to.amzn
197	15786152	637	4.04886202931351e-05	org.virtualbox
198	15780286	624	4.10339488441819e-05	uk.co.guardian
199	15777717	313	7.90555572975389e-05	com.techcrunch
200	15776975	411	5.95014769939008e-05	uk.co.telegraph
201	15776477	548	4.52609403123979e-05	com.cnbc
202	15775169	250	9.94157457855262e-05	uk.co.google
204	15771603	468	5.30605297385559e-05	com.goodreads
205	15766175	397	6.18619648011229e-05	com.msn
206	15761598	392	6.23915895238641e-05	com.kickstarter
207	15761370	171	0.000149517430613129	com.twimg
208	15757899	958	3.10644934362954e-05	in.blogspot
209	15756903	301	8.3087741732879e-05	com.images-amazon
210	15755871	21	0.00144087639474043	com.wixstatic
211	15740120	360	6.91064104263346e-05	io.codepen
212	15734160	225	0.000117332550512604	com.eventbrite
213	15733130	413	5.93919728037878e-05	com.bbc
214	15731621	889	3.33158516532838e-05	com.newyorker
215	15730889	396	6.2001076998058e-05	com.latimes
216	15729693	447	5.53136069293824e-05	com.webs
219	15727119	174	0.000147912563107984	org.archive
220	15723275	407	6.02624636349819e-05	com.bizjournals
221	15722493	648	3.98018712364008e-05	com.wikia
222	15722063	223	0.000118348799632694	org.drupal
223	15716740	296	8.4089596881278e-05	net.php
224	15716703	643	4.02154313123456e-05	com.venturebeat
225	15716576	619	4.12625555463031e-05	org.pbs
226	15712519	914	3.26392230464033e-05	com.marketwatch
227	15712144	770	3.83803578491251e-05	com.exacttarget
229	15706093	833	3.55961710127199e-05	com.sublimetext
230	15703601	198	0.000129275694487554	com.stumbleupon
232	15697321	271	9.26344464771574e-05	fr.free
233	15694373	465	5.327297946833e-05	gov.whitehouse
234	15693760	968	3.09182306870607e-05	com.gizmodo
235	15692378	796	3.71950102005915e-05	com.chrome
236	15686384	977	3.06408953569782e-05	com.instapaper
237	15682234	308	8.07156735886469e-05	gov.cdc
238	15682010	1034	2.90083132906254e-05	com.ning
239	15679191	476	5.17732111584554e-05	com.office
240	15677383	419	5.86742754402436e-05	com.xrea
242	15666975	632	4.05994257035936e-05	com.ted
243	15666829	1467	2.32876139970216e-05	edu.usc
244	15666264	1085	2.82676357406144e-05	edu.umn
245	15665846	1072	2.84598053412361e-05	com.arstechnica
246	15664243	535	4.58793673858625e-05	de.blogspot
247	15663366	883	3.35673118772062e-05	com.chicagotribune
248	15663034	28	0.0011763022119838	com.atdmt
249	15662537	261	9.51380930957369e-05	com.baidu
250	15660953	150	0.000174749380463391	com.list-manage
251	15660326	613	4.14680955973278e-05	com.scribd
252	15659073	1430	2.40444714698486e-05	edu.uchicago
253	15657783	575	4.32697253018267e-05	com.inc
254	15655666	1005	2.98439224149668e-05	com.evernote
255	15654762	69	0.000490371412441887	com.wp
256	15654252	339	7.26444892171607e-05	gov.ca
257	15651499	591	4.22531065519201e-05	com.dropboxusercontent
258	15651280	409	5.98380198468418e-05	com.cnet
259	15648867	285	8.75060881994027e-05	gov.ftc
260	15645937	880	3.36512290531188e-05	com.nationalgeographic
261	15641291	651	3.97372609578435e-05	uk.co.independent
262	15641175	927	3.20977639678945e-05	edu.yale
263	15640389	1152	2.76178324420893e-05	edu.wisc
264	15640290	842	3.50248903988879e-05	com.nintendo
265	15638043	659	3.96001518252408e-05	us.imageshack
266	15636823	907	3.27312169013512e-05	com.indiatimes
267	15635520	800	3.69358708077931e-05	com.cbslocal
268	15635163	1700	1.8722596335099e-05	edu.msu
269	15634089	944	3.15472671606861e-05	com.naturalnews
270	15632793	398	6.16851659136072e-05	com.ibm
271	15630971	1332	2.62606272746041e-05	org.ieee
272	15629452	257	9.65661295436327e-05	net.behance
274	15629172	875	3.37107150206004e-05	com.slate
275	15626020	480	5.14543056738778e-05	edu.cornell
276	15624467	976	3.07073787371343e-05	org.altervista
277	15622049	30	0.00112494056494761	me.m
278	15620427	23	0.00135395152637574	com.messenger
280	15617113	926	3.21020959728217e-05	com.mysanantonio
281	15614327	298	8.37226642358646e-05	com.opera
283	15612189	1544	2.13674763862979e-05	edu.virginia
285	15610650	235	0.000109224765834484	de.amazon
286	15609900	597	4.20971145241691e-05	com.bandsintown
287	15609689	268	9.38629534487172e-05	com.fc2
289	15606613	307	8.08602374785154e-05	com.staticflickr
290	15606205	503	4.94721115017751e-05	com.zdnet
291	15603206	924	3.21430290644091e-05	com.googledrive
292	15596851	586	4.24190156027991e-05	com.theatlantic
293	15595738	35	0.00106767518499836	ru.yandex
294	15591053	564	4.39081259808879e-05	com.fortune
295	15588997	861	3.43835713600411e-05	org.iso
296	15588791	947	3.15155421032288e-05	com.redhat
298	15587828	487	5.06700299908967e-05	com.w3schools
299	15587438	1345	2.59590471932613e-05	it.blogspot
300	15586623	786	3.75300358233661e-05	com.nature
301	15584911	431	5.74012367083115e-05	com.git-scm
302	15578116	691	3.89761753170653e-05	com.ggpht
303	15576627	193	0.000132159977180025	com.ytimg
305	15575464	106	0.000239116241712999	jp.co.google
306	15574856	891	3.33060387457206e-05	org.caringbridge
307	15570844	666	3.94027427723104e-05	com.buzzfeed
308	15567017	920	3.2305470209502e-05	com.vagrantup
309	15566713	1423	2.41810395096844e-05	com.bostonglobe
310	15566432	570	4.36715547457013e-05	com.cisco
312	15561846	773	3.82831940182663e-05	org.imagemagick
313	15561571	1318	2.67799890705726e-05	edu.si
314	15560481	1000	3.00286031998172e-05	com.economist
315	15559161	403	6.07417952206027e-05	edu.nyu
317	15557711	816	3.60704732755116e-05	com.engadget
318	15557648	207	0.000124999566076152	com.youtube-nocookie
320	15555868	97	0.000277346301246524	com.qq
321	15553042	921	3.22905898022667e-05	com.deepmind
322	15548476	776	3.8040845245793e-05	com.uk
323	15546635	532	4.65404973239769e-05	com.cbsnews
324	15546418	912	3.26484283331076e-05	com.sfgate
325	15546261	868	3.40855431726984e-05	edu.umich
326	15545867	237	0.000108426094503134	com.zendesk
327	15541664	1059	2.86845309523356e-05	edu.upenn
328	15541417	784	3.76122537114407e-05	com.tinypic
329	15541348	300	8.31191255523891e-05	uk.gov
330	15539380	795	3.72790288125804e-05	gov.nps
331	15538763	768	3.84226974381513e-05	es.com.blogspot
333	15534785	1027	2.91722456460812e-05	org.amnesty
334	15533640	630	4.06382514774634e-05	org.un
335	15533524	1028	2.91342711800371e-05	com.businessweek
336	15530590	841	3.51095688807082e-05	com.prweb
337	15530339	372	6.6861234129418e-05	com.hp
338	15529094	316	7.85334205313627e-05	com.smugmug
339	15526892	365	6.80643039930844e-05	io.atom
340	15526016	1864	1.66783616152529e-05	com.wikispaces
341	15525436	561	4.43833209085508e-05	com.deviantart
342	15524543	1348	2.59183160819967e-05	com.quora
343	15523699	312	7.90560957094403e-05	com.stackoverflow
344	15523357	11	0.00266026928788743	com.godaddy
345	15523265	867	3.41028601949806e-05	fr.blogspot
346	15521467	2177	1.43407953172429e-05	edu.hawaii
348	15520800	230	0.000115300591313306	com.googleadservices
349	15515626	100	0.000260210965158318	com.addthis
350	15511731	214	0.00012109708512216	com.weibo
351	15510271	189	0.000134991236934387	com.tripadvisor
352	15510056	466	5.31867573470599e-05	edu.cmu
353	15508137	1654	1.94505252040299e-05	edu.academia
354	15507099	546	4.54186649903759e-05	com.samsung
355	15506715	522	4.73185887798753e-05	com.delicious
356	15506306	310	7.96162415466004e-05	com.salesforce
357	15504623	1389	2.47638521688997e-05	edu.psu
358	15501767	1062	2.8590454491023e-05	edu.utexas
359	15500355	460	5.37255274210099e-05	com.businesswire
360	15499671	328	7.50250431392317e-05	fr.google
361	15499498	1054	2.87241689600589e-05	com.thenextweb
362	15499343	567	4.37356148766312e-05	com.skype
363	15499317	1018	2.94461794513264e-05	com.thedenverchannel
364	15499031	418	5.87309989696444e-05	com.booking
365	15498260	902	3.30560434754304e-05	com.ft
366	15497563	1445	2.37713091163388e-05	com.storify
367	15497320	138	0.0001870649963529	org.bbb
368	15496677	162	0.000163189952970138	com.bandcamp
369	15494308	794	3.73092071789939e-05	com.giphy
370	15494284	172	0.000149178020840425	com.eepurl
371	15492950	752	3.86710102667409e-05	org.rubyonrails
372	15492593	980	3.05867053615036e-05	ly.ow
373	15491779	350	7.09328555378064e-05	com.technorati
375	15490311	1440	2.39101606822674e-05	uk.ac.ox
376	15489287	1753	1.80336311984752e-05	edu.ucdavis
377	15488936	306	8.09041634071766e-05	mp.j
378	15488438	1669	1.92668234479678e-05	edu.umd
380	15486978	196	0.000130600081415146	org.joomla
381	15486803	521	4.74575675113086e-05	ca.google
382	15486012	1300	2.71672578237322e-05	com.foxnews
383	15484935	1138	2.77285208514453e-05	com.ubuntu
384	15482294	1518	2.19662629504982e-05	com.hootsuite
385	15481981	385	6.4295300561396e-05	com.barnesandnoble
386	15481043	1939	1.59034639194696e-05	edu.bu
387	15481011	849	3.47893789322901e-05	br.com.uol
388	15480413	1052	2.87377660159276e-05	org.change
389	15479814	1390	2.47562201385043e-05	net.researchgate
390	15479732	1046	2.87858158612307e-05	ca.cbc
391	15477862	278	9.10597679558766e-05	com.myshopify
392	15477604	1189	2.73974389493186e-05	gov.uspto
393	15473925	1662	1.93123862526904e-05	au.com.blogspot
394	15471584	1558	2.11443170255096e-05	com.ibtimes
395	15468867	127	0.000201494587730564	com.windowsphone
396	15468553	1670	1.92511697630888e-05	au.com.smh
397	15467171	863	3.43166520951593e-05	tv.ustream
398	15464178	1764	1.78996714682684e-05	edu.asu
399	15463055	1372	2.53582202622928e-05	com.vice
400	15462257	935	3.19137383273714e-05	ca.blogspot
401	15462107	764	3.85046143189563e-05	com.msdn
402	15461554	1487	2.27938995549172e-05	com.gigaom
403	15456424	1818	1.70999687690856e-05	fr.lemonde
404	15455464	886	3.3487653580818e-05	gov.arts
405	15452785	1419	2.42221672388526e-05	au.net.abc
406	15452669	1008	2.97794424811164e-05	com.pcworld
407	15452549	606	4.18331091885497e-05	com.geocities
408	15452088	1103	2.8074916778745e-05	com.over-blog
410	15451699	1790	1.74633853037583e-05	edu.arizona
411	15450350	1380	2.49714498779882e-05	uk.co.theregister
412	15450005	557	4.45739926153101e-05	com.symantec
413	15447649	955	3.10938947847226e-05	com.intel
414	15446867	279	9.09687835769951e-05	com.wufoo
415	15446683	1725	1.83123804765686e-05	it.scoop
416	15445503	41	0.00094307691355208	com.fbsbx
417	15444645	1355	2.57197109861248e-05	com.indiegogo
418	15443795	897	3.32227150997552e-05	fm.last
419	15443680	1011	2.96862133554023e-05	com.searchengineland
420	15443404	638	4.03884046551975e-05	org.tigris
421	15441943	120	0.000213563106056296	com.xing
422	15441722	906	3.28011039522532e-05	com.steampowered
423	15440081	265	9.44501307131196e-05	com.wixsite
424	15437622	893	3.32776293807694e-05	in.co.google
426	15435634	696	3.89468179483598e-05	org.kernel
427	15435243	1579	2.08114538791667e-05	edu.rutgers
428	15434410	390	6.26848449332716e-05	com.hubspot
429	15432056	945	3.1541523599974e-05	ly.snip
431	15429997	1656	1.94377049562573e-05	com.qz
432	15429003	437	5.67323209565391e-05	com.example
433	15427966	1033	2.90523344374742e-05	com.pinimg
434	15424166	2786	1.10264831868485e-05	org.moma
435	15423777	1522	2.18888107535011e-05	com.nymag
436	15423182	194	0.00013209008750952	org.icann
437	15422064	1470	2.32323539012823e-05	com.examiner
438	15421520	1066	2.85530482795166e-05	com.communitywalk
439	15418061	644	4.01838728589206e-05	com.googlegroups
440	15417796	505	4.94289151194946e-05	com.wiley
441	15417576	362	6.8485542721816e-05	us.icio
442	15416742	971	3.07993617237965e-05	org.wish
443	15415984	827	3.57195346814776e-05	com.brightcove
444	15415033	464	5.3283954538054e-05	com.fastcompany
445	15414280	1599	2.03032488006835e-05	br.com.blogspot
446	15413662	1315	2.68653930493553e-05	jp.blogspot
447	15413140	1513	2.21687586861463e-05	edu.duke
448	15412816	938	3.17991908431134e-05	com.tqn
449	15411542	2073	1.50117732512574e-05	com.denverpost
450	15411487	1890	1.64147347916477e-05	edu.ufl
451	15409424	950	3.13211107727205e-05	com.jsbin
452	15408909	959	3.10548447850379e-05	com.bibliocommons
453	15408700	177	0.000144292920751828	jp.ne.hatena
454	15407603	1567	2.10091681230242e-05	edu.ucsd
455	15407141	1269	2.72490186529387e-05	com.shutterstock
456	15406563	520	4.74694841820413e-05	com.nwsource
457	15404198	916	3.25322651787121e-05	org.lls
458	15402911	486	5.06747050151017e-05	com.sxsw
459	15402581	1313	2.6876013908712e-05	com.box
460	15401873	1798	1.73668495255569e-05	edu.tamu
461	15400807	1363	2.55335729572364e-05	com.lifehacker
462	15400234	677	3.91846367320029e-05	org.bitbucket
463	15400214	890	3.33135211093082e-05	com.reverbnation
464	15399047	1038	2.89493063164984e-05	com.intuit
466	15397689	1426	2.41475123446898e-05	com.theglobeandmail
467	15395311	1527	2.17995803147687e-05	com.boston
468	15394935	2639	1.16533507648765e-05	edu.wsu
469	15394864	2508	1.2433445453334e-05	com.codecademy
470	15394856	303	8.18811211827657e-05	com.mailchimp
471	15394689	3599	8.40481697313659e-06	edu.case
473	15394158	854	3.4663109492416e-05	org.vim
474	15393227	1361	2.55650930767044e-05	gov.usgs
475	15392918	2938	1.03733739575562e-05	edu.udel
476	15391640	1588	2.06360756674507e-05	com.lulu
478	15388924	571	4.35997856907777e-05	com.herokuapp
479	15388579	2833	1.07911954379239e-05	edu.usf
480	15387385	1414	2.42826197261023e-05	org.arxiv
481	15386491	1458	2.3550335270772e-05	com.hollywoodreporter
482	15385502	829	3.56957822588539e-05	com.squareup
483	15384701	560	4.45055236043777e-05	com.rawgit
484	15384598	994	3.01587098357822e-05	int.wipo
485	15384544	513	4.89234029936009e-05	int.who
486	15384253	1479	2.30586570568822e-05	com.mlb
487	15380758	423	5.84115059504074e-05	nl.google
488	15379754	1514	2.21563978466527e-05	uk.ac.cam
489	15379070	435	5.69413663780201e-05	it.google
490	15377833	2457	1.27170779759503e-05	edu.rochester
491	15377085	583	4.26581830659939e-05	com.sciencedirect
492	15376952	540	4.5590233520292e-05	es.google
493	15375930	492	5.04565655944046e-05	com.prnewswire
496	15374063	512	4.89486762586997e-05	com.netdna-cdn
497	15373640	895	3.32748831426672e-05	gov.census
498	15372707	526	4.70109683913423e-05	org.acm
499	15372236	997	3.00968502339631e-05	uk.co.eventbrite
500	15371625	264	9.45864958803737e-05	com.dribbble
502	15369552	438	5.66258761382857e-05	com.bigcartel
503	15369206	777	3.80078034824024e-05	com.blackberry
504	15368955	323	7.62773882788658e-05	it.placehold
505	15368028	2523	1.23363276509568e-05	com.freewebs
507	15365341	847	3.491720143606e-05	gov.house
508	15365035	542	4.5495955558977e-05	com.moz
509	15364679	1037	2.89595958340174e-05	com.pcmag
510	15363450	1090	2.82004663551755e-05	th.co.lazada
511	15363266	595	4.21623397499087e-05	com.java
512	15363220	2446	1.27866233856021e-05	edu.brown
513	15362506	1490	2.27546025150466e-05	org.unesco
514	15357119	1039	2.89434233736272e-05	com.timeanddate
516	15355214	2062	1.50466348954934e-05	com.me
517	15354656	556	4.45769549760707e-05	com.wunderground
518	15354608	478	5.15403521918125e-05	com.naver
519	15354297	1020	2.93180988863568e-05	com.googlelabs
520	15354200	2545	1.21756108703223e-05	edu.dartmouth
521	15354116	1069	2.84725561625317e-05	com.cafepress
522	15353246	1658	1.93811871032173e-05	nl.blogspot
523	15352221	3361	9.06439332694793e-06	com.blog
524	15349206	566	4.37902161051386e-05	net.openid
525	15348907	1181	2.74418894971913e-05	gov.state
526	15348783	587	4.2359471534077e-05	com.campaign-archive2
527	15348394	502	4.95388543949701e-05	com.snapchat
528	15347563	3530	8.63918778341324e-06	com.answers
529	15347241	3126	9.74923014967996e-06	com.panoramio
530	15346426	762	3.85484763743062e-05	org.doi
532	15343527	519	4.77202432693482e-05	gov.usda
533	15343427	995	3.01325901940305e-05	com.nydailynews
534	15343011	2678	1.14107241890089e-05	com.fiverr
535	15342904	404	6.05683558445266e-05	gov.irs
536	15340458	1586	2.06527553264387e-05	com.mediafire
537	15338766	2049	1.51263773021797e-05	com.yolasite
538	15338739	1793	1.74163462409128e-05	edu.northwestern
539	15338645	3048	9.99423133900803e-06	au.edu.unimelb
540	15338264	2955	1.02945568869793e-05	pt.sapo
541	15337836	2252	1.3890799881422e-05	edu.uci
542	15337304	2156	1.45372776850819e-05	com.vox
543	15337112	1343	2.60056262188797e-05	de.spiegel
544	15336552	1831	1.6951961616819e-05	edu.indiana
545	15336407	1499	2.24037156386934e-05	com.mixcloud
546	15336092	402	6.13443915079197e-05	com.gotowebinar
547	15335961	1486	2.27988200023977e-05	com.politico
548	15335080	1928	1.60404649507357e-05	uk.ac.ed
549	15334549	2453	1.27409920227876e-05	org.coursera
550	15333737	2728	1.12118871484912e-05	edu.iastate
551	15333326	1320	2.67149480639566e-05	com.gofundme
552	15331375	1311	2.69569796358108e-05	com.sciencedaily
553	15328641	2561	1.20420514958202e-05	edu.wustl
554	15326663	1385	2.48050571742303e-05	com.com
555	15324920	627	4.09333910649588e-05	org.postgresql
556	15324038	2742	1.11586867111723e-05	com.aljazeera
557	15322328	3388	8.99605195071346e-06	cc.tiny
558	15322264	2022	1.52880655359817e-05	edu.jhu
559	15321808	869	3.39056838936811e-05	tv.twitch
560	15320933	4059	7.40539131056082e-06	org.edublogs
561	15320472	1606	2.01774638677572e-05	com.scientificamerican
562	15320227	2253	1.38879432187941e-05	edu.georgetown
563	15320215	831	3.56813161908985e-05	fr.amazon
564	15318380	1503	2.23612984764625e-05	org.eff
565	15317401	3437	8.86702554218844e-06	com.metacafe
566	15317100	191	0.000133568759135424	jp.ameblo
567	15317067	1693	1.8888419168614e-05	edu.purdue
568	15315866	1341	2.60218077289035e-05	com.techtarget
569	15315501	1796	1.7382847608717e-05	com.computerworld
570	15315302	358	6.92385615991618e-05	com.list-manage1
571	15314942	2630	1.16835618607635e-05	com.voanews
573	15313159	810	3.63928853611669e-05	org.filezilla-project
575	15312739	1584	2.07088116839661e-05	com.globo
576	15312531	1695	1.88078663306853e-05	com.espn
577	15312290	61	0.000565420241560954	com.googletagmanager
579	15310359	1953	1.57895689797867e-05	com.econsultancy
580	15309983	2199	1.42058821135541e-05	edu.caltech
581	15309668	4006	7.50369095396379e-06	ca.yorku
584	15308817	1493	2.26547145145621e-05	uk.co.mirror
585	15307236	1405	2.44901711292864e-05	gov.wa
586	15306671	149	0.000177379963368227	com.bleacherreport
587	15306314	112	0.000229383265424507	ru.mail
588	15306144	555	4.46855914261337e-05	com.wikihow
589	15305890	2552	1.20966873341379e-05	com.posterous
590	15305715	3450	8.82359793937048e-06	edu.getty
591	15305261	3153	9.66215408282903e-06	edu.iu
592	15305198	1802	1.73123902085213e-05	com.networkworld
593	15303934	2990	1.01379864079042e-05	edu.rice
594	15303667	493	5.03591727892366e-05	com.force
595	15302835	3165	9.62217258755941e-06	com.popsci
596	15302205	2057	1.50768807284092e-05	com.blogs
597	15301680	966	3.09598932353572e-05	ca.amazon
598	15301018	1492	2.26668010052997e-05	com.prezi
599	15300885	675	3.92213914102238e-05	org.openstreetmap
600	15300438	1433	2.39976511418591e-05	org.videolan
601	15300390	2692	1.13711151099246e-05	edu.oregonstate
602	15300263	905	3.28686767582425e-05	com.mckinsey
603	15299663	1374	2.53508968618347e-05	co.vine
604	15297995	2171	1.44041517200603e-05	com.udemy
605	15296726	1690	1.90166303335655e-05	com.indeed
606	15296477	1303	2.71367812776429e-05	com.500px
607	15295228	1770	1.77233586297493e-05	com.airbnb
608	15294719	1756	1.79860216972368e-05	com.us
609	15294676	2896	1.05399264773873e-05	edu.buffalo
610	15293634	1684	1.90790330006778e-05	com.codeplex
611	15292934	253	9.81881014385652e-05	jp.co.amazon
612	15292288	1542	2.14433652955457e-05	mil.navy
613	15291760	357	6.9641067198322e-05	net.themeforest
614	15291748	1875	1.65847048348261e-05	com.searchenginewatch
615	15291136	1001	2.9937094462355e-05	com.weather
616	15290802	2498	1.24886423946348e-05	com.instructables
617	15289988	1598	2.04147744275967e-05	com.infoworld
618	15288865	1338	2.60730524379888e-05	org.postimg
619	15288655	1014	2.96274588622905e-05	gov.nist
620	15287654	1655	1.94498068426432e-05	me.flavors
621	15287153	2189	1.42494598478046e-05	hk.com.google
622	15285623	1569	2.09919187442072e-05	org.worldbank
623	15285448	446	5.57049729509609e-05	jp.ne.sakura
624	15285211	919	3.24107060440747e-05	gov.senate
625	15284629	1330	2.62832178308331e-05	com.dell
626	15283827	1137	2.77346639122896e-05	com.alternion
627	15283594	1858	1.67084293512309e-05	org.weforum
628	15282288	2578	1.19657096495962e-05	edu.vanderbilt
629	15281800	1048	2.87643509027424e-05	com.istockphoto
630	15281782	1319	2.67520978971456e-05	org.unicef
631	15281371	1819	1.70846042948261e-05	com.blogtalkradio
632	15281301	1368	2.54329071778898e-05	com.psychologytoday
633	15280641	1604	2.02133971936181e-05	com.digitaltrends
634	15279721	1107	2.80258341333577e-05	com.bitballoon
635	15279522	201	0.000128219206443624	org.purl
636	15279264	36	0.00104599466571899	com.parallels
637	15279200	2880	1.0610438448301e-05	au.edu.anu
638	15279147	1379	2.4993271977574e-05	com.walmart
642	15278238	1917	1.61582921968463e-05	com.howstuffworks
643	15278227	2410	1.30002357541342e-05	au.com.theaustralian
644	15277966	2019	1.52926412575938e-05	ca.utoronto
645	15277900	3108	9.82306713797465e-06	com.seekingalpha
646	15277858	767	3.84259493608188e-05	com.cargocollective
647	15277312	1036	2.89853194713678e-05	com.yarnpkg
649	15275046	2546	1.21728936988854e-05	com.patch
650	15273144	530	4.65890979908163e-05	com.marriott
651	15272960	2564	1.20339218653302e-05	nz.co.stuff
652	15272836	1461	2.34742927248467e-05	com.bufferapp
653	15272534	233	0.000113347706063239	com.shopify
654	15272287	2861	1.0673622554462e-05	br.com.abril
655	15272172	579	4.31163726460518e-05	gov.fda
656	15272111	2654	1.15546716330409e-05	org.metmuseum
657	15270748	1771	1.7720478702479e-05	com.mac
658	15270052	1921	1.61418125507527e-05	edu.unc
659	15268572	885	3.35402797098504e-05	com.webmd
661	15268283	2121	1.47329526454372e-05	com.nokia
662	15268033	1483	2.2878848034952e-05	org.plos
663	15267754	4046	7.42719003848336e-06	edu.ua
664	15267001	2238	1.3967856594105e-05	org.wiktionary
665	15264534	436	5.67508362709212e-05	com.bitly
666	15264521	1739	1.8211369785099e-05	com.xkcd
667	15264460	1930	1.59995476208952e-05	com.amzn
668	15264453	2469	1.26617143551831e-05	com.allthingsd
669	15262825	2856	1.06933839212956e-05	edu.ucr
670	15262636	1511	2.21802765343256e-05	gov.fbi
671	15262608	1101	2.81053814869449e-05	com.oreilly
672	15260639	1975	1.56605882146644e-05	net.comcast
674	15259591	2391	1.30968680707361e-05	com.pastebin
675	15259139	2476	1.26118695308562e-05	net.boingboing
676	15258733	1784	1.75043751947912e-05	com.playstation
677	15258666	2398	1.30425871141071e-05	com.canalblog
678	15258256	695	3.89580808757773e-05	net.launchpad
679	15257933	477	5.1625634539935e-05	com.youku
680	15256999	3412	8.92246050318757e-06	cc.co
681	15256828	3140	9.69399959688503e-06	com.twitpic
683	15256704	1613	2.00790892379684e-05	com.today
684	15256213	1116	2.79029401808774e-05	org.cmlibrary
686	15255446	1222	2.73108581082248e-05	de.heise
687	15255118	2228	1.40200068711249e-05	com.googlepages
688	15254918	1672	1.92155195280816e-05	com.technet
689	15254068	302	8.22175508543349e-05	com.getbootstrap
690	15253812	3433	8.87454964811042e-06	edu.temple
691	15252999	2556	1.20687755115901e-05	com.ign
693	15252474	3151	9.66350917742778e-06	com.upi
694	15252276	2074	1.50071783868689e-05	com.nba
695	15252014	848	3.48867990264086e-05	cn.com.sina
696	15251586	1870	1.66343745503257e-05	uk.co.thesun
697	15250778	2105	1.48179911042675e-05	com.discovery
698	15250169	1304	2.71164863688636e-05	com.getpocket
699	15249530	1932	1.5985973562373e-05	com.thedailybeast
700	15249456	894	3.32758542409795e-05	org.gnupg
701	15249338	986	3.03560667284895e-05	net.azurewebsites
702	15249295	1359	2.56231659003632e-05	gov.fcc
703	15249233	1325	2.6533125002868e-05	ru.google
704	15249091	809	3.64185428454994e-05	com.hilton
705	15248094	1121	2.78672834461891e-05	gov.ny
706	15246873	4155	7.2064158065949e-06	edu.syr
707	15246346	1448	2.37502348483709e-05	br.com.google
708	15246265	1523	2.18628778111356e-05	com.zazzle
710	15245258	604	4.19341270866239e-05	com.entrepreneur
711	15245122	2348	1.34109723726176e-05	ch.ethz
712	15244639	4244	7.05352140883181e-06	com.space
713	15244618	3062	9.9552002600227e-06	com.indiewire
714	15243268	3689	8.20683368467091e-06	it.repubblica
715	15242487	1159	2.75783380257729e-05	jp.exblog
716	15241707	1627	1.97344047444238e-05	com.vmware
717	15241153	1772	1.77159273715622e-05	gov.nyc
718	15240597	4469	6.70690376470189e-06	edu.fiu
719	15240265	585	4.24602948328796e-05	com.typeform
720	15239779	1119	2.7880890311812e-05	edu.alamo
721	15239357	576	4.32670162572188e-05	com.metafilter
722	15239100	1874	1.66039378221903e-05	gov.utah
723	15238566	2756	1.10983574160883e-05	edu.pitt
724	15238213	837	3.54670748199058e-05	com.technologyreview
725	15237455	2743	1.11564239851983e-05	ms.1drv
726	15236971	2001	1.54095292080742e-05	com.mercurynews
727	15235587	2528	1.23026764563188e-05	com.urbandictionary
728	15235218	4812	6.26441533285652e-06	uk.ac.st-andrews
731	15234240	500	4.95978355797237e-05	org.freecsstemplates
732	15234119	2310	1.36176106587469e-05	edu.ncsu
733	15233298	774	3.81419726467888e-05	com.newsweek
734	15233113	2515	1.23694068280388e-05	edu.vt
735	15232928	2242	1.39511923347534e-05	org.gimp
736	15232535	2125	1.47205731999793e-05	com.ehow
737	15232422	2766	1.10740320522725e-05	org.greenpeace
739	15231780	1386	2.4798751939996e-05	com.steamcommunity
740	15231759	1681	1.90934768631241e-05	com.mcafee
741	15230059	2579	1.19584044094577e-05	com.irishtimes
742	15229095	1597	2.04331098882185e-05	com.livestream
743	15228828	1364	2.55057605268654e-05	com.stackexchange
744	15228452	1377	2.51087702775784e-05	com.feedly
745	15228080	4305	6.93390584748876e-06	edu.pdx
746	15227949	244	0.000102784590967636	jp.co.yahoo
747	15227896	1879	1.65447602576108e-05	com.zillow
748	15227135	2399	1.30401089444467e-05	com.foxsports
749	15226937	1217	2.73204489809514e-05	com.redbubble
751	15225915	1571	2.09669925267844e-05	com.reference
752	15225808	1775	1.76300650079329e-05	se.google
753	15225166	518	4.78475650600298e-05	gov.copyright
754	15224935	2046	1.51404543018914e-05	uk.co.wired
755	15224708	1053	2.87281963287335e-05	com.templatemonster
756	15223786	2603	1.18487163717953e-05	com.tutsplus
757	15223148	2275	1.38238195338617e-05	edu.uiuc
758	15221995	5134	5.8713899338997e-06	edu.brandeis
759	15221724	2563	1.20373170102433e-05	com.observer
760	15220659	1444	2.37776101286514e-05	com.justgiving
761	15220635	3017	1.00492793873175e-05	gd.is
762	15220238	143	0.000182177431084415	com.people
764	15218272	454	5.43767285380984e-05	com.nypost
765	15217792	2017	1.52947227103117e-05	org.openoffice
766	15217664	1521	2.19029938843643e-05	net.yahoo
767	15217239	515	4.86242565462417e-05	com.photoshelter
768	15216199	1105	2.80631360388941e-05	org.stopidentityfraud
769	15215956	2941	1.03557602124572e-05	edu.nd
770	15215827	1076	2.83930562753374e-05	com.sagepub
771	15215619	792	3.73306683068506e-05	com.cdbaby
772	15215146	3080	9.89422877978489e-06	com.pbworks
773	15215078	3131	9.73273115724507e-06	ch.epfl
774	15214730	1016	2.95913720075838e-05	net.docdroid
775	15214344	3538	8.59892849115483e-06	com.nationalpost
776	15214306	1836	1.6890525363002e-05	nl.xs4all
777	15213130	2531	1.22807841365904e-05	edu.osu
778	15212390	1754	1.80225680175084e-05	com.target
779	15211733	2996	1.01271424887745e-05	ca.ualberta
780	15210252	2264	1.38516035718976e-05	com.britannica
781	15208855	3521	8.66915050464592e-06	com.readwriteweb
782	15208756	2362	1.32853529783148e-05	tv.blip
783	15208449	662	3.95683024324097e-05	com.webnode
784	15207643	533	4.61857724018269e-05	com.informit
785	15207576	588	4.23206675622496e-05	com.houzz
786	15206681	917	3.25050735019443e-05	com.atlassian
787	15206417	703	3.88738057972146e-05	gov.epa
788	15205305	1436	2.39702540982123e-05	com.patreon
789	15204875	1913	1.62263884951756e-05	org.pnas
790	15204526	4602	6.54795414698615e-06	com.plurk
791	15204495	2565	1.20309774714031e-05	com.deadline
792	15204117	2461	1.2694191531719e-05	com.csmonitor
793	15203245	2558	1.20590026406928e-05	com.thefreedictionary
794	15202856	1482	2.29792000644675e-05	com.techradar
795	15201654	366	6.79713039346072e-05	gov.export
796	15201546	1302	2.71523901829089e-05	jp.jugem
797	15201171	3914	7.66665907938012e-06	com.patheos
798	15200965	797	3.70188361564265e-05	com.springer
799	15200633	3161	9.63918862375184e-06	com.fifa
801	15199882	2910	1.04940988571428e-05	uk.co.metro
802	15199828	2796	1.09820826242647e-05	com.podbean
803	15199347	2842	1.07590520070566e-05	com.eu
804	15198865	2341	1.34651543708415e-05	com.9to5mac
805	15198766	3893	7.72829568187162e-06	edu.ucsc
806	15198366	1366	2.54906334920141e-05	gov.sba
807	15197638	2529	1.22968407272032e-05	org.khanacademy
808	15196883	3320	9.15893198278876e-06	com.discovermagazine
809	15196046	1630	1.97102959583915e-05	com.bloglovin
810	15195823	1312	2.68820751313068e-05	gov.dot
811	15194939	2479	1.26030313566866e-05	gov.cia
812	15194836	3913	7.66754130260334e-06	edu.missouri
813	15194521	741	3.86928597648577e-05	gov.sec
814	15194106	462	5.34455013327675e-05	com.adage
816	15194059	1748	1.81076747100426e-05	org.owasp
818	15193361	814	3.62833594255235e-05	org.maven
820	15193182	590	4.22817895742876e-05	com.clicky
822	15192587	1986	1.5564837127422e-05	com.netvibes
823	15191503	1701	1.86858940694352e-05	uk.org.greenend
824	15191192	3079	9.8973806614334e-06	edu.uic
825	15190493	1556	2.11535012115229e-05	net.seesaa
827	15189584	1871	1.66248944998599e-05	com.livescience
828	15189058	2329	1.3520097935398e-05	com.newscientist
829	15189011	2434	1.28219860937364e-05	edu.umass
830	15188283	4481	6.69072560608157e-06	edu.byu
831	15188205	2343	1.34543166872523e-05	edu.gwu
832	15187440	2505	1.24388641430322e-05	com.getsatisfaction
833	15186750	3563	8.49826407908942e-06	edu.ucar
834	15186042	1967	1.57161284089379e-05	com.bestbuy
835	15185601	2618	1.17677918287584e-05	org.slashdot
836	15185560	3302	9.20889985322655e-06	com.lynda
837	15185025	1552	2.12374312638794e-05	com.itunes
838	15183956	898	3.31541720640316e-05	com.linksynergy
839	15182044	220	0.000119236860219388	com.elegantthemes
840	15181895	471	5.27435047435429e-05	com.zenfolio
841	15181614	299	8.36816972799617e-05	kr.flic
842	15181486	4787	6.29442653768555e-06	com.treehugger
843	15181407	4906	6.14399747187076e-06	com.4shared
844	15181277	2775	1.10589713260247e-05	sg.com.google
845	15181256	870	3.38511869472674e-05	com.uservoice
846	15180717	2553	1.20874453751689e-05	au.com.news
847	15180230	2959	1.02691773243877e-05	edu.uoregon
849	15179697	39	0.000963859187439238	com.atlassolutions
850	15179276	1415	2.42376980846539e-05	com.calameo
851	15178765	694	3.89619798369146e-05	gov.ed
853	15178015	1441	2.38849164666484e-05	net.edgesuite
854	15176822	1810	1.72249794008617e-05	com.angelfire
855	15176641	2864	1.06660801729435e-05	com.friendfeed
856	15176459	3346	9.09973283027648e-06	com.squidoo
857	15176231	4362	6.86729655781333e-06	edu.sjsu
858	15175482	534	4.59747379453658e-05	com.list-manage2
859	15175282	639	4.03760068683785e-05	org.w
860	15175276	1976	1.56554998174229e-05	com.billboard
861	15175270	888	3.33229976472426e-05	org.jenkins-ci
862	15174784	1815	1.71639406507681e-05	org.craigslist
863	15173098	1413	2.43275295847814e-05	ch.google
864	15172912	1624	1.98193414781252e-05	com.norton
865	15172759	2499	1.24788748858098e-05	org.gutenberg
866	15172598	3542	8.58529237980699e-06	edu.gmu
867	15172079	4332	6.90179812828641e-06	com.urbanoutfitters
869	15170249	963	3.0982348094362e-05	com.warriorplus
870	15170065	2892	1.05536267729891e-05	gov.lbl
871	15169915	2430	1.28272703930632e-05	ca.ubc
872	15169647	3670	8.24518633348136e-06	edu.emory
873	15169635	1960	1.57546904235563e-05	com.madmimi
874	15169591	2777	1.10579126365468e-05	com.flipboard
875	15168793	2649	1.158087404906e-05	uk.co.express
876	15168561	3044	9.9973459042887e-06	com.groupon
877	15168529	6596	4.61066717637004e-06	fm.ask
878	15168472	1758	1.79602590370848e-05	org.dyndns
879	15168121	940	3.16738878962008e-05	org.osgeo
880	15167971	3402	8.95368354470453e-06	edu.unh
881	15167568	3702	8.16829795347617e-06	com.dreamstime
882	15167447	2174	1.43790430825757e-05	de.mpg
883	15167217	2352	1.33549876043573e-05	com.timeout
884	15167075	2291	1.37930561320996e-05	com.trello
885	15166885	4434	6.75650883193862e-06	net.inquirer
887	15163788	3397	8.97470907356599e-06	tt.db
888	15163403	2568	1.2028676812975e-05	com.smashwords
889	15163045	4009	7.5009851913861e-06	org.lifehack
890	15162790	611	4.15506626046456e-05	com.mapbox
892	15162627	4088	7.33799753523127e-06	com.theta360
894	15162508	2650	1.15685409144789e-05	com.elpais
895	15162189	3984	7.5452462328069e-06	mx.unam
897	15162074	1715	1.84504055261042e-05	com.hyatt
898	15161589	2299	1.37457361006392e-05	ca.qc.gouv
899	15161293	2915	1.04695633262643e-05	edu.uga
900	15160912	2890	1.05582698397468e-05	tv.periscope
901	15160846	2239	1.39665182554598e-05	com.ikea
902	15160700	3231	9.40684172196742e-06	com.rt
903	15160131	1508	2.22312003369202e-05	com.netscape
904	15159916	4084	7.3412350461331e-06	com.dpreview
905	15159557	2969	1.02451694615442e-05	com.waze
906	15159470	3174	9.58306422626019e-06	edu.tufts
908	15158248	1465	2.32953650777979e-05	com.usps
909	15158207	2633	1.16729003842566e-05	org.wnyc
910	15156889	4620	6.52677225898427e-06	com.pbase
911	15156880	2327	1.35354017520418e-05	uk.co.huffingtonpost
912	15156876	2204	1.41810128439695e-05	com.socialmediaexaminer
913	15156485	2452	1.27547105359095e-05	com.miamiherald
914	15156351	1677	1.91454793535634e-05	com.163
915	15155262	1929	1.60291831429757e-05	com.nfl
916	15153991	1525	2.18455988672043e-05	com.fedex
917	15153959	1834	1.69239070862306e-05	com.suntimes
918	15153907	3435	8.87050857955755e-06	net.uk
919	15153765	2846	1.07327970998889e-05	net.earthlink
920	15153389	2575	1.19824685546719e-05	org.eu
921	15151835	1043	2.88478957939967e-05	jp.ac.kobe-u
922	15151429	4014	7.48637493537742e-06	com.macrumors
923	15150887	622	4.12161691159013e-05	com.friendster
925	15150658	4938	6.09241171910296e-06	li.paper
926	15150593	1017	2.94689613731383e-05	org.aclweb
929	15147788	4248	7.04318153363729e-06	org.nypl
930	15146250	3661	8.25885549831049e-06	com.seattlepi
931	15146224	932	3.19355693878378e-05	edu.utep
932	15146129	4091	7.33275777840132e-06	com.esquire
933	15145615	1651	1.94907985965323e-05	com.alibaba
934	15145254	1946	1.58403729551741e-05	com.gallup
935	15145191	975	3.07094414631654e-05	com.gartner
936	15144687	3032	1.00243908229847e-05	org.ibiblio
937	15144604	2206	1.41652445483039e-05	ru.habrahabr
938	15144355	569	4.36758910514297e-05	com.orkut
939	15144010	3765	8.05161795367949e-06	com.psychcentral
941	15142711	999	3.00349367316673e-05	com.tandfonline
942	15142296	4075	7.3622724322498e-06	be.blogspot
943	15141826	1636	1.96036095572849e-05	ru.spb
944	15140942	1191	2.73934146878194e-05	jp.ne.goo
945	15140702	5199	5.81304350399957e-06	io.soup
946	15140439	6368	4.80285781528238e-06	com.autoblog
947	15140096	4769	6.31731333379226e-06	gr.blogspot
948	15140030	1731	1.82779854197245e-05	com.marketingland
949	15139772	3355	9.0739118458068e-06	org.propublica
950	15139731	1382	2.49154931051116e-05	com.unsplash
951	15139224	3672	8.24261668907916e-06	uk.ac.vam
952	15138529	54	0.000677034331007159	net.ovh
953	15138391	1853	1.67490179527703e-05	org.ap
954	15138382	1747	1.81116122309211e-05	de.bayern
955	15138317	2614	1.1780491106467e-05	com.ezinearticles
956	15138275	3532	8.63374503998574e-06	at.ac.univie
957	15138076	1902	1.62814600358615e-05	com.lww
958	15137904	826	3.58116025428148e-05	org.whatbrowser
959	15137802	1909	1.62391225244165e-05	ru.narod
960	15137495	1799	1.73626417117781e-05	de.welt
961	15137128	2527	1.23125015700162e-05	com.vogue
963	15136768	2667	1.14903880876818e-05	com.sacbee
964	15136495	2223	1.40422839026896e-05	com.salon
966	15135193	3370	9.04444445804001e-06	com.howtogeek
967	15135056	3877	7.75786443249133e-06	com.scienceblogs
968	15134830	2576	1.19753053105799e-05	com.speakerdeck
969	15134635	3740	8.0925036532909e-06	edu.utk
970	15134622	2115	1.47665516324275e-05	edu.colorado
971	15134362	2663	1.15263452785689e-05	ca.ctvnews
972	15133333	3216	9.46266143255436e-06	edu.rit
973	15133149	5021	5.99011002035218e-06	com.spreaker
974	15132872	3172	9.58906484578881e-06	com.sbnation
975	15132829	4474	6.70057653000476e-06	com.ravelry
976	15132461	9119	3.36255538243955e-06	com.purevolume
977	15132213	1429	2.40986597355896e-05	org.redcross
978	15131951	1012	2.96536226245788e-05	it.amazon
979	15130530	3830	7.87841169754985e-06	jp.co.japantimes
980	15130523	3947	7.60521847167761e-06	edu.neu
981	15129963	2984	1.01806930403748e-05	org.notepad-plus-plus
982	15129854	1249	2.72621515187053e-05	us.tx.state
983	15129838	5806	5.24730127406629e-06	edu.stonybrook
984	15129190	879	3.36537032047091e-05	com.githubusercontent
985	15128753	3356	9.07349412732171e-06	nz.co.nzherald
986	15128682	4466	6.71188460059978e-06	au.edu.uq
987	15128202	1743	1.81424145083871e-05	com.hulu
989	15127370	3722	8.11380321181256e-06	com.kotaku
990	15127197	2803	1.09596426379084e-05	com.pixlr
991	15126832	2536	1.22403614813801e-05	org.plosone
992	15126819	2319	1.35869429391614e-05	be.skynet
993	15126628	5285	5.73383523576963e-06	org.kiva
994	15126245	5889	5.19218457452368e-06	com.dailykos
995	15125565	1947	1.58394285740344e-05	com.windows
997	15124648	4380	6.83359980472686e-06	com.appleinsider
998	15124445	7272	4.16137009681623e-06	com.zimbio
999	15124312	3493	8.72400390257565e-06	com.sony
1000	15124231	1516	2.20203522611855e-05	com.usnews
1001	15124118	2213	1.41046155978645e-05	org.jstor
1002	15124063	3578	8.44147236834851e-06	br.usp
1003	15123962	1692	1.89005659232104e-05	com.hotmail
1004	15123945	3591	8.42266358295749e-06	org.publicradio
1005	15123923	4299	6.94773919350873e-06	ca.queensu
1007	15122919	3094	9.85256523885283e-06	com.techsmith
1008	15122846	1392	2.47451015821843e-05	com.starwoodhotels
1009	15122669	2332	1.35082248225311e-05	int.itu
1010	15122441	4721	6.38885267967483e-06	ru.msk
1011	15122321	1918	1.61560935000156e-05	com.slack
1012	15122267	3518	8.67216939620092e-06	com.ocregister
1013	15121528	1561	2.11146320845391e-05	com.att
1014	15119594	2419	1.29223390974961e-05	ru.com
1016	15118761	2726	1.12330945363111e-05	fr.lefigaro
1017	15118270	1936	1.59672085303341e-05	com.merriam-webster
1018	15117908	815	3.62635751772531e-05	org.sonatype
1019	15117906	1576	2.08815868514927e-05	com.domain
1020	15117709	819	3.60499120933233e-05	jp.livedoor
1021	15117533	2047	1.51327792586241e-05	com.gettyimages
1022	15117448	2780	1.10566824510243e-05	org.pewresearch
1023	15117248	2146	1.46293466334101e-05	com.xbox
1024	15117204	2336	1.34965990334014e-05	com.philly
1025	15117135	1998	1.54386617149377e-05	com.macworld
1026	15116770	2935	1.03852509028486e-05	com.pandora
1027	15116005	2235	1.39818040735583e-05	com.screencast
1028	15115778	2913	1.04753710838801e-05	cn.com.chinadaily
1029	15115238	1781	1.75423937882377e-05	co.angel
1030	15114938	3611	8.37665878835035e-06	com.softpedia
1031	15114818	3438	8.86482430240979e-06	com.sun-sentinel
1032	15114562	3761	8.0617698796768e-06	com.gq
1033	15113975	4760	6.33100426632562e-06	uk.ac.lse
1034	15113456	5622	5.41405793070476e-06	ie.dcu
1035	15113066	4242	7.05731974728055e-06	cz.cvut
1036	15112227	1905	1.62743170924513e-05	us.mn.state
1037	15111997	4000	7.52083022965871e-06	com.gamespot
1038	15111439	1680	1.90947838704124e-05	net.battle
1039	15111082	4326	6.90589418244931e-06	org.kqed
1040	15110600	2513	1.23787698420637e-05	com.deezer
1041	15110590	6308	4.84743641510709e-06	com.weheartit
1042	15110406	2436	1.28163659354189e-05	us.fed.fs
1043	15110005	6060	5.04386130379262e-06	edu.boisestate
1044	15109716	3658	8.26486260361324e-06	com.itv
1045	15109619	3575	8.44807075839288e-06	com.chromeexperiments
1046	15109222	2427	1.28446958304232e-05	com.ecwid
1047	15109222	1817	1.71096481543741e-05	com.rollingstone
1048	15109090	2162	1.45052881285571e-05	com.lg
1049	15109034	649	3.97925746370024e-05	gov.usa
1050	15108729	1384	2.48508143917158e-05	com.mediapost
1051	15108642	1894	1.63687473890889e-05	org.7-zip
1053	15107528	3451	8.82152744851336e-06	uk.ac.ucl
1056	15106039	1418	2.42315485369215e-05	org.oecd
1057	15105171	4587	6.56637741524037e-06	edu.uconn
1058	15104892	1992	1.54943353581812e-05	com.oup
1059	15104097	2187	1.42705509195955e-05	com.fastcodesign
1060	15104069	1761	1.7949821530063e-05	org.fao
1061	15103689	1966	1.57209150085303e-05	com.ssrn
1062	15103023	612	4.15348394643097e-05	com.ea
1063	15102701	1197	2.73652354802987e-05	site.tenerifeforum
1064	15102234	2456	1.27338596207715e-05	org.gnome
1065	15102132	2812	1.09082842009225e-05	com.dallasnews
1066	15101892	4703	6.41996854624355e-06	edu.baylor
1067	15100957	2625	1.17066663780132e-05	com.upwork
1068	15100938	568	4.3688041709467e-05	com.udacity
1069	15100803	3810	7.92556546534424e-06	com.marthastewart
1071	15099903	108	0.000233897006385183	com.scorecardresearch
1072	15099840	7227	4.18439407525323e-06	com.smore
1073	15099764	2185	1.42753327167622e-05	gov.uscourts
1074	15099487	2331	1.35150828730417e-05	com.thestar
1075	15099386	3796	7.95835809109849e-06	com.mediabistro
1076	15098556	2467	1.26718746433665e-05	com.washingtontimes
1077	15098287	2173	1.43821141052024e-05	gov.fws
1078	15098247	2698	1.13376477961827e-05	de.fu-berlin
1079	15096938	3394	8.97977597433521e-06	org.thinkprogress
1080	15096415	2378	1.31988303928714e-05	uk.co.ebay
1081	15096031	1534	2.16177620913569e-05	org.cancer
1082	15095930	2192	1.42362632550556e-05	com.tunein
1084	15095382	4528	6.63269521044456e-06	net.fusion
1085	15095323	3483	8.75212816606324e-06	com.shutterfly
1086	15094901	2845	1.07360892958791e-05	edu.unl
1087	15094502	1633	1.96647199416434e-05	gov.archives
1088	15094481	3555	8.53632335646553e-06	ca.uwaterloo
1089	15094330	4195	7.13316306266006e-06	com.rebelmouse
1090	15094229	7095	4.27197287772721e-06	nr.co
1091	15094084	1073	2.84596850945921e-05	com.wsoctv
1092	15093234	2440	1.28051282516683e-05	ch.cern
1093	15092967	1376	2.51450016917556e-05	com.magentocommerce
1094	15092862	3005	1.00907350023168e-05	com.blurb
1095	15092691	2730	1.12020560201465e-05	com.medscape
1098	15091627	4891	6.17105152283906e-06	com.thingiverse
1099	15091515	1612	2.00799886698044e-05	com.biblegateway
1100	15091174	3187	9.55435977880778e-06	com.carbonmade
1101	15090152	2328	1.35223797685066e-05	gov.weather
1102	15089995	2202	1.41915895238769e-05	com.canva
1103	15089724	1663	1.93009906939172e-05	com.thehill
1104	15089040	2883	1.06020285008986e-05	com.twilio
1105	15088768	3281	9.2762882038917e-06	tv.arte
1106	15088645	6971	4.35419930787964e-06	sg.blogspot
1107	15087852	4080	7.35568049360429e-06	com.gawker
1108	15087391	7507	4.03564477881975e-06	fr.online
1109	15087176	3396	8.97494140584996e-06	edu.colostate
1110	15086946	5259	5.75702090283342e-06	edu.nau
1111	15086937	3132	9.73185560248581e-06	net.jsfiddle

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link SPAM detection, etc. Let us know about your results via Common Crawl’s Google Group!

July 2017 Crawl Archive Now Available

The crawl archive for July 2017 is now available! The archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2017-30/. It contains 2.89 billion+ web pages and over 240 TiB of uncompressed content.

To improve coverage and freshness we used the top 50 million ranked hosts from the Feb/Mar/Apr 2017 webgraph data set and added over 550 million new URLs (not contained in any crawl archive before), of which:

300 million URLs were found by a side crawl within a maximum of 4 links (“hops”) away from the home pages of the top 50 million hosts;
250 million URLs are a random sample extracted from sitemaps (if provided by any of these 50 million hosts).

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files. By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2017-30/segment.paths.gz	100
WARC files	CC-MAIN-2017-30/warc.paths.gz	72000	57.62
WAT files	CC-MAIN-2017-30/wat.paths.gz	72000	18.58
WET files	CC-MAIN-2017-30/wet.paths.gz	72000	8.19
Robots.txt files	CC-MAIN-2017-30/robotstxt.paths.gz	72000	0.16
Non-200 responses files	CC-MAIN-2017-30/non200responses.paths.gz	72000	5.03
URL index files	CC-MAIN-2017-30/cc-index.paths.gz	302	0.25

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2017-30/. For more information on working with the URL index, please refer to the previous blog post or the Index Server API. There is also a command-line tool client for common use cases of the URL index.

June 2017 Crawl Archive Now Available

The crawl archive for June 2017 is now available! The archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2017-26/. It contains 3.16 billion+ web pages and over 260 TiB of uncompressed content.

To improve coverage and freshness we used the top 40 million ranked hosts from the Feb/Mar/Apr 2017 webgraph data set and added almost 800 million new URLs (not contained in any crawl archive before), of which:

500 million URLs were found by a side crawl within a maximum of 3 links (“hops”) away from the home pages of the top 40 million hosts;
300 million URLs are a random sample extracted from sitemaps (if provided by any of these 40 million hosts).

About 33% of the crawl archive’s 3.16 billion URLs overlap with the preceding May 2017 crawl.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files. By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2017-26/segment.paths.gz	100
WARC files	CC-MAIN-2017-26/warc.paths.gz	71839	63.33
WAT files	CC-MAIN-2017-26/wat.paths.gz	71840	21.06
WET files	CC-MAIN-2017-26/wet.paths.gz	71839	9.44
Robots.txt files	CC-MAIN-2017-26/robotstxt.paths.gz	71840	0.13
Non-200 responses files	CC-MAIN-2017-26/non200responses.paths.gz	71840	1.77
URL index files	CC-MAIN-2017-26/cc-index.paths.gz	302	0.24

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2017-26/. For more information on working with the URL index, please refer to the previous blog post or the Index Server API. There is also a command-line tool client for common use cases of the URL index.

May 2017 Crawl Archive Now Available

The crawl archive for May 2017 is now available! The archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2017-22/. It contains 2.96 billion+ web pages and over 250 TiB of uncompressed content.

To improve coverage and freshness we used the top 25 million ranked hosts from the Feb/Mar/Apr 2017 webgraph data set and added about 500 million new URLs (not contained in any crawl archive before), of which:

330 million URLs were found by a side crawl within a maximum of 3 links (“hops”) away from the home pages of the top 25 million hosts;
160 million URLs are a random sample extracted from sitemaps (if provided by any of these 25 million hosts).

About 40% of the crawl archive’s 2.96 billion URLs overlap with the preceding April 2017 crawl.

The following changes have been made to WARC (also WAT and WET) files:

the timestamp in WARC filenames now indicates the capture time (fetch time) of the WARC content (see details)
WARC files and the URL index now contain the detected MIME type (based on the actual content) in addition to the “Content-Type” sent in the HTTP response (details).

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files. By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2017-22/segment.paths.gz	100
WARC files	CC-MAIN-2017-22/warc.paths.gz	56788	57.98
WAT files	CC-MAIN-2017-22/wat.paths.gz	56788	19.87
WET files	CC-MAIN-2017-22/wet.paths.gz	56788	8.95
Robots.txt files	CC-MAIN-2017-22/robotstxt.paths.gz	56788	0.11
Non-200 responses	CC-MAIN-2017-22/non200responses.paths.gz	56788	1.38

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2017-22/. For more information on working with the URL index, please refer to the previous blog post or the Index Server API. There is also a command-line tool client for common use cases of the URL index.