Blog – Page 3 – Common Crawl

October 2019 crawl archive now available

October 29, 2019Sebastian Nagel

The crawl archive for October 2019 is now available! It contains 3.0 billion web pages or 280 TiB of uncompressed content, crawled between October 13th and 24th. It includes page captures of 1.1 billion URLs not contained in any crawl archive before.

Archive Location and Download

The October crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-43/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line, you end up with the S3 and HTTP paths respectively.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-43/segment.paths.gz	100
WARC files	CC-MAIN-2019-43/warc.paths.gz	56000	59.56
WAT files	CC-MAIN-2019-43/wat.paths.gz	56000	21.7
WET files	CC-MAIN-2019-43/wet.paths.gz	56000	9.94
Robots.txt files	CC-MAIN-2019-43/robotstxt.paths.gz	56000	0.15
Non-200 responses files	CC-MAIN-2019-43/non200responses.paths.gz	56000	1.69
URL index files	CC-MAIN-2019-43/cc-index.paths.gz	302	0.22

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-43/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

September 2019 crawl archive now available

September 28, 2019Sebastian Nagel

The crawl archive for September 2019 is now available! It contains 2.55 billion web pages or 240 TiB of uncompressed content, crawled between September 15th and 24th. It includes page captures of 1.0 billion URLs not contained in any crawl archive before. The other 1.5 billion pages have been already captured in prior crawls and are now revisited.

Archive Location and Download

The September crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-39/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-39/segment.paths.gz	100
WARC files	CC-MAIN-2019-39/warc.paths.gz	56000	55.99
WAT files	CC-MAIN-2019-39/wat.paths.gz	56000	18.01
WET files	CC-MAIN-2019-39/wet.paths.gz	56000	8.16
Robots.txt files	CC-MAIN-2019-39/robotstxt.paths.gz	56000	0.14
Non-200 responses files	CC-MAIN-2019-39/non200responses.paths.gz	56000	1.63
URL index files	CC-MAIN-2019-39/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-39/. Also the columnar index has been updated to contain this crawl.

August 2019 crawl archive now available

August 30, 2019Sebastian Nagel

The crawl archive for August 2019 is now available! It contains 2.95 billion web pages or 260 TiB of uncompressed content, crawled between August 17th and 26th.

The August crawl contains page captures of 1.1 billion URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks (harmonic centrality) published as part of the May/Jun/Jul 2019 webgraph data set from the following sources:

a random sample of 2.1 billion outlinks extracted from July crawl WAT files
1.8 billion URLs mined in a breadth-first side crawl within a maximum of 6 links (“hops”), started from
- the homepages of the top 60 million hosts and domains and randomly selected samples of
- 2 million human-readable sitemap pages (HTML format)
- 3 million URLs of pages written in 130 less-represented languages (cf. language distributions)
1 billion URLs extracted and sampled from 20 million sitemaps, RSS and Atom feeds

Starting with this crawl the following fixes and improvements are applied to the provided data formats:

reliable marking of WARC records with truncated payload, see issue report “WARC-Truncated header”
improved decoding of XML/HTML character entities in WAT and WET files

Archive Location and Download

The August crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-35/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-35/segment.paths.gz	100
WARC files	CC-MAIN-2019-35/warc.paths.gz	56000	53.53
WAT files	CC-MAIN-2019-35/wat.paths.gz	56000	20.85
WET files	CC-MAIN-2019-35/wet.paths.gz	56000	9.29
Robots.txt files	CC-MAIN-2019-35/robotstxt.paths.gz	56000	0.18
Non-200 responses files	CC-MAIN-2019-35/non200responses.paths.gz	56000	1.79
URL index files	CC-MAIN-2019-35/cc-index.paths.gz	302	0.22

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-35/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs May/June/July 2019

August 8, 2019Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of May, June and July 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark on GitHub which host all scripts and tools required to construct the graphs.

What’s new?

Links from Content-Location and Link HTTP headers are now also used to span up the web graphs. This is in accordance with RFC 5988 which defines the Link HTTP header as semantically equivalent to the element in HTML. It also fits previous web graph releases which used to include all kinds of links including technical ones and redirects.

Host-level graph

The graph consists of 445 million nodes and 3.14 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 382 million dangling nodes (86%) and the largest strongly connected component contains 48 million (11%) nodes.

You can download the graph and the ranks of all 445 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-may-jun-jul/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-may-jun-jul/host/ as prefix to access the files from everywhere.

Download files of the Common Crawl May/June/July 2019 host-level webgraph

Size	File	Description
3.02 GB	cc-main-2019-may-jun-jul-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 28 vertices files
14.75 GB	cc-main-2019-may-jun-jul-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 56 edges files

6.42 GB	cc-main-2019-may-jun-jul-host.graph	graph in BVGraph format
2 kB	cc-main-2019-may-jun-jul-host.properties

7.01 GB	cc-main-2019-may-jun-jul-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2019-may-jun-jul-host-t.properties

1 kB	cc-main-2019-may-jun-jul-host.stats	WebGraph statistics

7.22 GB	cc-main-2019-may-jun-jul-host-ranks.txt.gz	harmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 88 million nodes and 1.9 billion edges. 52% or 46 million nodes are dangling nodes, the largest strongly connected component covers 35 million or 40% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-may-jun-jul/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-may-jun-jul/domain/.

Download files of the Common Crawl May/June/July 2019 domain-level webgraph

Size	File	Description
0.61 GB	cc-main-2019-may-jun-jul-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
7.50 GB	cc-main-2019-may-jun-jul-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

4.06 GB	cc-main-2019-may-jun-jul-domain.graph	graph in BVGraph format
2 kB	cc-main-2019-may-jun-jul-domain.properties

3.99 GB	cc-main-2019-may-jun-jul-domain-t.graph	transpose of the graph
2 kB	cc-main-2019-may-jun-jul-domain-t.properties

1 kB	cc-main-2019-may-jun-jul-domain.stats	WebGraph statistics

1.91 GB	cc-main-2019-may-jun-jul-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 90 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (May/June/July 2019)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	29977668	1	0.020841	com.googleapis
2	27867704	3	0.011812	com.facebook
3	27419980	2	0.012857	com.google
4	25196030	4	0.007273	com.twitter
5	24558836	5	0.006439	org.w
6	24533702	6	0.005984	com.youtube
7	22592098	9	0.003799	com.instagram
8	22060650	7	0.004857	org.gmpg
9	21829028	13	0.002863	com.linkedin
10	21595446	8	0.004481	com.googletagmanager
11	20930920	22	0.001704	com.gravatar
12	20912076	24	0.001531	com.pinterest
13	20730700	11	0.003384	com.cloudflare
14	20698732	17	0.002180	com.wordpress
15	20613210	12	0.003087	org.wordpress
16	20607942	26	0.001241	org.wikipedia
17	20408594	14	0.002452	com.bootstrapcdn
18	20351540	20	0.001823	com.apple
19	20148418	41	0.000904	com.blogspot
20	20103846	30	0.001124	com.vimeo
21	20036764	21	0.001719	com.jquery
22	19874716	50	0.000673	com.wp
23	19870332	29	0.001130	com.microsoft
24	19839912	43	0.000816	gl.goo
25	19828406	45	0.000769	com.amazon
26	19793040	18	0.002021	com.gstatic
27	19790998	19	0.002015	com.adobe
28	19788744	57	0.000573	com.tumblr
29	19754126	31	0.001104	com.amazonaws
30	19619798	25	0.001407	com.macromedia
31	19616602	34	0.001057	com.googlesyndication
32	19585788	47	0.000744	be.youtu
33	19585670	39	0.000937	com.google-analytics
34	19583342	62	0.000531	ly.bit
35	19572994	68	0.000440	com.yahoo
36	19549710	33	0.001080	com.flickr
37	19526876	35	0.001023	net.cloudfront
38	19526762	23	0.001676	com.github
39	19503814	60	0.000553	me.wp
40	19467672	27	0.001170	ru.yandex
41	19467424	58	0.000568	org.mozilla
42	19454890	106	0.000305	com.googleusercontent
43	19411724	49	0.000725	net.doubleclick
44	19374766	52	0.000658	co.t
45	19366860	44	0.000776	com.baidu
46	19322188	70	0.000401	com.weebly
47	19321754	105	0.000310	com.reddit
48	19317094	123	0.000234	com.nytimes
49	19313908	46	0.000749	com.paypal
50	19308094	104	0.000312	com.soundcloud
51	19278436	67	0.000448	com.medium
52	19268558	66	0.000451	io.github
53	19266970	63	0.000517	org.w3
54	19255616	80	0.000379	org.creativecommons
55	19228034	184	0.000143	uk.co.bbc
56	19219470	175	0.000151	com.imgur
57	19191484	137	0.000194	com.forbes
58	19184320	168	0.000154	net.slideshare
59	19169524	56	0.000588	org.schema
60	19166214	162	0.000162	com.bing
61	19163388	180	0.000144	net.sourceforge
62	19155882	182	0.000143	org.wikimedia
63	19145618	48	0.000738	com.googleadservices
64	19143840	215	0.000109	com.businessinsider
65	19136040	233	0.000104	com.techcrunch
66	19125198	273	0.000089	com.reuters
67	19112730	152	0.000169	com.theguardian
68	19091708	177	0.000147	com.imdb
69	19081148	64	0.000496	net.jsdelivr
70	19076642	145	0.000177	org.apache
71	19068912	202	0.000120	org.gnu
72	19067720	250	0.000097	com.ibm
73	19065904	274	0.000089	com.cnet
74	19060402	194	0.000124	com.washingtonpost
75	19056162	164	0.000159	com.blogger
76	19049622	336	0.000073	gov.nasa
77	19043406	271	0.000090	com.android
78	19038878	32	0.001080	com.fontawesome
79	19030824	196	0.000123	com.huffingtonpost
80	19022764	243	0.000100	com.oracle
81	19022114	99	0.000323	com.shopify
82	19010092	178	0.000147	com.stackoverflow
83	19008866	264	0.000092	com.bbc
84	18991504	138	0.000194	com.wixsite
85	18979794	193	0.000128	org.ampproject
86	18979606	331	0.000074	com.latimes
87	18966924	334	0.000073	com.livejournal
88	18954352	148	0.000171	com.eventbrite
89	18952914	406	0.000061	com.zdnet
90	18951470	38	0.000950	com.addthis
91	18941168	260	0.000093	com.usatoday
92	18930306	261	0.000093	com.wired
93	18929948	473	0.000052	com.economist
94	18924894	122	0.000237	com.ytimg
95	18915820	295	0.000083	com.prnewswire
96	18907784	107	0.000304	com.whatsapp
97	18905562	241	0.000101	com.appspot
98	18903750	289	0.000086	org.npr
99	18899826	605	0.000046	com.thenextweb
100	18898732	139	0.000192	com.issuu
101	18897130	198	0.000122	org.ietf
102	18893188	181	0.000143	jp.co.yahoo
103	18889096	142	0.000183	com.spotify
104	18888760	449	0.000055	com.venturebeat
105	18888186	55	0.000590	eu.europa
106	18886240	382	0.000064	com.goodreads
107	18880882	37	0.000994	com.qq
108	18880880	601	0.000046	org.ieee
109	18876988	209	0.000114	com.bandcamp
110	18874448	359	0.000068	com.quora
111	18872666	426	0.000058	com.cisco
112	18869640	211	0.000112	net.behance
113	18866560	474	0.000052	org.arxiv
114	18852080	394	0.000062	com.buzzfeed
115	18844806	95	0.000330	com.sharethis
116	18834502	427	0.000058	com.deviantart
117	18834166	899	0.000031	com.ibtimes
118	18829762	185	0.000141	com.giphy
119	18828960	96	0.000328	com.statcounter
120	18825074	649	0.000043	com.stackexchange
121	18823624	170	0.000152	uk.co.google
122	18818848	283	0.000087	com.cnbc
123	18817384	825	0.000034	org.eclipse
124	18814566	333	0.000074	com.aol
125	18814392	485	0.000051	com.pixabay
126	18806944	206	0.000117	com.disqus
127	18800912	458	0.000054	com.about
128	18793968	42	0.000849	com.squarespace
129	18793572	522	0.000048	com.mysql
130	18792740	144	0.000180	com.yelp
131	18790794	355	0.000068	com.theatlantic
132	18787424	417	0.000059	me.about
133	18787006	317	0.000077	com.skype
134	18782636	476	0.000052	com.visualstudio
135	18780538	232	0.000104	me.t
136	18772666	948	0.000030	com.nvidia
137	18772560	468	0.000053	com.wikihow
138	18768358	276	0.000089	com.sciencedirect
139	18767822	224	0.000106	com.dribbble
140	18762266	324	0.000075	com.scribd
141	18759236	712	0.000039	google.blog
142	18756886	183	0.000143	com.salesforce
143	18756236	551	0.000048	com.slate
144	18753968	131	0.000208	com.dropbox
145	18751696	407	0.000061	uk.co.independent
146	18751242	299	0.000081	com.fastcompany
147	18746590	257	0.000094	com.googlecode
148	18746142	213	0.000111	com.hubspot
149	18744470	440	0.000057	com.newyorker
150	18744452	430	0.000058	com.box
151	18743332	120	0.000249	org.networkadvertising
152	18736956	667	0.000042	org.chromium
153	18735918	463	0.000053	gov.loc
154	18734190	297	0.000082	com.example
155	18733762	200	0.000121	com.cnn
156	18731428	671	0.000041	com.tinypic
157	18728160	269	0.000090	com.fc2
158	18726104	790	0.000035	com.nymag
159	18723184	707	0.000039	com.smashingmagazine
160	18719704	616	0.000045	com.evernote
161	18718576	272	0.000090	com.nbcnews
162	18716396	548	0.000048	net.azurewebsites
163	18710606	219	0.000108	com.npmjs
164	18709770	155	0.000167	org.archive
165	18708768	306	0.000079	com.w3schools
166	18705090	1024	0.000028	ca.utoronto
167	18703880	191	0.000130	jp.ne.hatena
168	18699974	477	0.000052	io.codepen
169	18699212	61	0.000544	com.vk
170	18699016	969	0.000029	com.ign
171	18694692	703	0.000039	com.speakerdeck
172	18694256	853	0.000033	com.mediafire
173	18691628	506	0.000049	com.foursquare
174	18686102	894	0.000031	com.nike
175	18684136	608	0.000046	com.trello
176	18679302	119	0.000251	info.aboutads
177	18676168	376	0.000066	com.mozilla
178	18670790	53	0.000604	com.wix
179	18669780	639	0.000044	uk.ac.ox
180	18664264	146	0.000174	com.amazon-adsystem
181	18661162	103	0.000317	com.paypalobjects
182	18658320	84	0.000366	com.bizjournals
183	18653438	342	0.000072	com.getpocket
184	18639078	316	0.000077	ca.google
185	18636324	498	0.000050	com.indiatimes
186	18628316	596	0.000047	com.pinimg
187	18626162	624	0.000045	com.cbslocal
188	18624278	311	0.000078	edu.mit
189	18623878	942	0.000030	com.chron
190	18622420	114	0.000272	net.windows
191	18618786	1158	0.000025	org.tensorflow
192	18618326	726	0.000038	ca.blogspot
193	18617602	842	0.000033	com.sap
194	18615678	841	0.000033	com.css-tricks
195	18612144	360	0.000068	com.entrepreneur
196	18606050	623	0.000045	com.libsyn
197	18603340	134	0.000205	com.unpkg
198	18602302	117	0.000253	com.stripe
199	18600352	308	0.000079	edu.harvard
200	18597464	226	0.000106	com.wsj
201	18595214	1070	0.000026	com.hackernoon
202	18594174	836	0.000033	com.thehill
203	18592786	59	0.000557	com.fb
204	18590510	625	0.000045	ca.cbc
205	18590172	912	0.000031	org.unicode
206	18586610	792	0.000035	com.buffer
207	18585880	369	0.000067	com.elsevier
208	18581126	794	0.000035	com.theglobeandmail
209	18580570	15	0.002238	com.wixstatic
210	18579326	363	0.000068	me.telegram
211	18578858	662	0.000042	com.searchengineland
212	18576402	179	0.000147	org.bbb
213	18574900	656	0.000043	site.business
214	18574476	481	0.000051	com.withgoogle
215	18574346	252	0.000097	es.google
216	18572616	874	0.000032	org.kernel
217	18572498	644	0.000044	com.flipboard
218	18571106	725	0.000038	co.ibb
219	18565014	658	0.000042	com.huffpost
220	18563588	1005	0.000028	edu.rutgers
221	18562788	848	0.000033	uk.co.wired
222	18560744	759	0.000036	com.ssrn
223	18560606	113	0.000272	com.weibo
224	18557646	1039	0.000027	com.aljazeera
225	18555860	736	0.000037	gov.archives
226	18554338	346	0.000071	com.mapbox
227	18554008	637	0.000044	org.d3js
228	18553278	151	0.000170	com.yimg
229	18551098	1093	0.000026	org.hrw
230	18549104	603	0.000046	gg.discord
231	18546834	1468	0.000020	com.hm
232	18546708	1146	0.000025	ly.visual
233	18545968	985	0.000029	com.geekwire
234	18545464	201	0.000120	com.optimizely
235	18544754	1251	0.000023	ca.huffingtonpost
236	18544114	212	0.000111	edu.stanford
237	18542860	943	0.000030	uk.co.huffingtonpost
238	18542334	618	0.000045	co.elastic
239	18539242	1877	0.000017	com.pearltrees
240	18535880	1829	0.000017	cn.people
241	18530690	1402	0.000021	com.diigo
242	18528804	296	0.000082	com.tinyurl
243	18528034	455	0.000054	com.mapquest
244	18525550	979	0.000029	org.slashdot
245	18524276	1106	0.000025	edu.osu
246	18523478	65	0.000473	net.akamaihd
247	18522316	882	0.000032	com.theconversation
248	18517436	278	0.000089	org.purl
249	18517362	375	0.000066	com.mashable
250	18514272	1097	0.000026	com.dw
251	18513770	934	0.000030	com.bt
252	18511512	844	0.000033	com.today
253	18511490	877	0.000032	com.marketwired
254	18510324	1267	0.000023	jp.co.ntv
255	18509842	1099	0.000026	com.mentalfloss
256	18509588	986	0.000029	com.computerworld
257	18505276	1376	0.000021	jp.ac.u-tokyo
258	18505224	840	0.000033	co.g
259	18503072	847	0.000033	com.healthline
260	18502704	782	0.000035	com.ecwid
261	18501610	1428	0.000021	com.sas
262	18501526	465	0.000053	com.yoast
263	18499006	1101	0.000025	edu.gatech
264	18490078	454	0.000054	com.moz
265	18487976	1636	0.000019	com.kaggle
266	18487772	1371	0.000021	com.makeuseof
267	18487400	501	0.000050	me.m
268	18487164	280	0.000088	com.bloomberg
269	18487154	771	0.000036	com.econsultancy
270	18486838	880	0.000032	uk.parliament
271	18483770	820	0.000034	com.newsweek
272	18483730	1300	0.000022	com.googlesource
273	18480640	1426	0.000021	blog.home
274	18480254	761	0.000036	com.outbrain
275	18478306	1077	0.000026	com.sfchronicle
276	18475570	204	0.000119	org.iana
277	18473642	313	0.000077	com.scorecardresearch
278	18471466	154	0.000169	gov.nih
279	18461762	1279	0.000022	com.avg
280	18459822	460	0.000054	com.theverge
281	18457732	636	0.000044	jp.shinobi
282	18455036	1009	0.000028	org.postgresql
283	18452056	1415	0.000021	com.dailydot
284	18449572	940	0.000030	com.foxbusiness
285	18448330	814	0.000034	com.adjust
286	18447054	764	0.000036	edu.brookings
287	18446074	767	0.000036	com.business2community
288	18437928	1861	0.000017	com.uniqlo
289	18435866	1446	0.000020	com.dezeen
290	18433310	347	0.000071	com.trustpilot
291	18432722	805	0.000035	com.contentmarketinginstitute
292	18431968	1036	0.000027	com.trendmicro
293	18431222	909	0.000031	org.aarp
294	18430922	850	0.000033	com.searchenginewatch
295	18428042	309	0.000078	org.python
296	18427984	160	0.000163	com.twimg
297	18427454	508	0.000049	edu.berkeley
298	18425200	768	0.000036	uk.co.pinterest
299	18423396	446	0.000055	com.bigcommerce
300	18418266	1358	0.000021	edu.iastate
301	18417172	1307	0.000022	com.motherjones
302	18416930	715	0.000039	com.techtarget
303	18414068	281	0.000087	com.myspace
304	18413522	1154	0.000025	com.hostgator
305	18411314	1046	0.000027	com.medicalnewstoday
306	18410696	1025	0.000028	com.bustle
307	18410084	69	0.000420	com.list-manage
308	18409562	323	0.000076	uk.co.telegraph
309	18409362	330	0.000074	com.meetup
310	18408690	1168	0.000024	org.openoffice
311	18406012	1296	0.000022	com.contently
312	18403532	720	0.000038	com.cdbaby
313	18402004	514	0.000049	com.adage
314	18401404	1337	0.000022	org.wnyc
315	18400286	714	0.000039	com.neilpatel
316	18398610	1527	0.000020	com.mathworks
317	18397518	478	0.000052	net.researchgate
318	18394598	938	0.000030	co.apple
319	18394586	329	0.000074	com.go
320	18393232	94	0.000347	com.godaddy
321	18392026	411	0.000060	com.msn
322	18391674	404	0.000061	com.ted
323	18390148	1484	0.000020	io.material
324	18389994	817	0.000034	com.arstechnica
325	18389508	860	0.000033	com.wikia
326	18388188	970	0.000029	com.vogue
327	18384912	377	0.000066	me.wa
328	18382020	1475	0.000020	se.blogspot
329	18380670	789	0.000035	edu.washington
330	18380362	165	0.000158	com.opera
331	18377106	248	0.000098	com.rawgit
332	18376938	819	0.000034	com.bandsintown
333	18374558	1066	0.000026	com.convinceandconvert
334	18374154	1238	0.000023	com.convertkit
335	18373850	1871	0.000017	io.soup
336	18370310	1438	0.000020	com.secondlife
337	18366152	1721	0.000018	com.zara
338	18364438	287	0.000086	com.live
339	18362628	238	0.000102	com.surveymonkey
340	18358654	188	0.000132	com.etsy
341	18356388	169	0.000153	com.feedburner
342	18356094	1944	0.000016	edu.uark
343	18355048	1911	0.000017	com.mysanantonio
344	18354726	266	0.000092	uk.org.ico
345	18352630	429	0.000058	org.hbr
346	18352406	602	0.000046	com.livechatinc
347	18352058	1493	0.000020	com.thenation
348	18351586	750	0.000037	com.yellowpages
349	18349922	112	0.000281	com.mailchimp
350	18349126	815	0.000034	com.wordstream
351	18349062	1506	0.000020	com.toptal
352	18347036	1216	0.000024	io.itch
353	18342626	494	0.000050	com.kickstarter
354	18341572	235	0.000104	com.typepad
355	18340608	420	0.000059	com.googleblog
356	18338046	366	0.000068	com.aliyuncs
357	18337760	1670	0.000018	com.manta
358	18337600	1463	0.000020	com.amcharts
359	18336652	1419	0.000021	com.indiewire
360	18335378	456	0.000054	com.fortune
361	18333310	51	0.000663	net.fbcdn
362	18333286	231	0.000105	uk.co.amazon
363	18331092	1587	0.000019	ly.adobe
364	18329760	922	0.000030	com.searchenginejournal
365	18328602	1569	0.000019	ms.nyti
366	18325744	374	0.000066	com.ft
367	18323516	2004	0.000016	com.zoominfo
368	18323442	1179	0.000024	com.grammarly
369	18321780	1620	0.000019	li.paper
370	18321750	1210	0.000024	com.csmonitor
371	18321512	2148	0.000015	com.brandyourself
372	18320716	2076	0.000015	me.websta
373	18310842	340	0.000072	com.getclicky
374	18310400	911	0.000031	uk.gov.nationalarchives
375	18306846	863	0.000033	com.engadget
376	18304066	159	0.000165	com.zendesk
377	18300980	962	0.000029	com.cio
378	18300896	87	0.000360	de.google
379	18300852	1625	0.000019	id.co.blogspot
380	18300752	1735	0.000018	org.unfpa
381	18299478	686	0.000040	com.intel
382	18297716	431	0.000058	com.nationalgeographic
383	18297684	1942	0.000016	com.cinemablend
384	18295492	1939	0.000016	com.wral
385	18295372	663	0.000042	com.vice
386	18294668	443	0.000056	com.oreilly
387	18294554	1020	0.000028	com.weddingwire
388	18293034	461	0.000053	com.nature
389	18292998	1440	0.000020	com.harpercollins
390	18291570	290	0.000085	gov.cdc
391	18290658	364	0.000068	com.githubusercontent
392	18290368	520	0.000048	com.photobucket
393	18290364	926	0.000030	com.socialmediaexaminer
394	18290020	998	0.000028	com.firebaseapp
395	18289150	875	0.000032	com.angieslist
396	18288842	901	0.000031	com.sendpulse
397	18288628	822	0.000034	edu.columbia
398	18287750	823	0.000034	com.pexels
399	18286600	1541	0.000019	com.mindbodygreen
400	18279452	1516	0.000020	com.mailjet
401	18278356	149	0.000171	com.tripadvisor
402	18278168	319	0.000077	com.wiley
403	18276930	1850	0.000017	com.merchantcircle
404	18276454	268	0.000090	com.digg
405	18276088	1890	0.000017	fr.huffingtonpost
406	18275746	1695	0.000018	com.thoughtworks
407	18273760	1014	0.000028	org.ocks
408	18273220	2062	0.000015	jp.pinterest
409	18272768	484	0.000051	com.cbsnews
410	18271878	352	0.000069	int.who
411	18270528	816	0.000034	com.format
412	18270108	255	0.000096	net.php
413	18269924	1464	0.000020	com.thecut
414	18268658	2055	0.000015	org.spie
415	18264554	214	0.000110	org.aboutcookies
416	18263300	1231	0.000023	com.mynewsdesk
417	18261732	409	0.000060	com.office
418	18261624	1071	0.000026	com.fastcodesign
419	18260856	1452	0.000020	fr.liberation
420	18260774	335	0.000073	com.time
421	18260366	444	0.000056	org.freecodecamp
422	18260020	1606	0.000019	com.dummies
423	18259400	1778	0.000018	com.instapaper
424	18258930	755	0.000036	com.mediapost
425	18255842	630	0.000044	com.proofpoint
426	18254118	1878	0.000017	it.binged
427	18254086	1321	0.000022	ly.snip
428	18252858	416	0.000059	uk.co.dailymail
429	18249260	604	0.000046	org.nodejs
430	18248590	392	0.000062	fr.free
431	18248492	464	0.000053	com.statista
432	18247356	879	0.000032	com.gizmodo
433	18246646	315	0.000077	com.st-hatena
434	18245388	1660	0.000018	com.superpages
435	18244078	1120	0.000025	com.theknot
436	18243678	357	0.000068	com.unsplash
437	18241494	1397	0.000021	com.jeffbullas
438	18236208	1522	0.000020	com.biography
439	18235946	2146	0.000015	de.huffingtonpost
440	18234820	1432	0.000021	com.csoonline
441	18234726	1486	0.000020	com.louisvuitton
442	18233512	121	0.000246	com.jimdo
443	18232920	1040	0.000027	uk.ac.cam
444	18232348	1338	0.000022	google.ai
445	18231586	2190	0.000014	com.mango
446	18230902	1227	0.000023	com.activecampaign
447	18226336	964	0.000029	com.netlify
448	18226172	953	0.000030	com.eater
449	18223984	1004	0.000028	com.smallbiztrends
450	18223564	2105	0.000015	site.negocio
451	18223100	277	0.000089	com.ebay
452	18221778	1301	0.000022	ca.yellowpages
453	18220422	689	0.000040	com.windowsphone
454	18220366	775	0.000035	com.marketwatch
455	18219714	1147	0.000025	com.redhat
456	18217972	2170	0.000015	edu.scad
457	18217660	1290	0.000022	com.digitaltrends
458	18217318	1123	0.000025	org.mathjax
459	18216670	1658	0.000019	com.politifact
460	18215546	2225	0.000014	com.dexknows
461	18214790	490	0.000050	gov.whitehouse
462	18210044	1225	0.000023	com.quicksprout
463	18207494	176	0.000150	com.slack
464	18205208	1655	0.000019	uk.co.bbci
465	18203194	1336	0.000022	com.cmswire
466	18202308	79	0.000382	net.jsfiddle
467	18199674	1683	0.000018	com.nyt
468	18198490	1928	0.000016	com.itsnicethat
469	18197492	835	0.000034	edu.psu
470	18196856	354	0.000068	com.booking
471	18196796	688	0.000040	com.webs
472	18195852	960	0.000030	edu.ucla
473	18191364	701	0.000039	gov.nist
474	18191138	945	0.000030	com.sprinklr
475	18191102	307	0.000079	gov.ca
476	18188332	76	0.000389	com.livestream
477	18186908	1375	0.000021	net.openid
478	18186750	1131	0.000025	gov.fbi
479	18185834	475	0.000052	tv.twitch
480	18183498	1982	0.000016	google.design
481	18176950	1790	0.000017	com.psmag
482	18175788	774	0.000036	com.oath
483	18173816	1498	0.000020	org.gnupg
484	18172144	351	0.000069	com.hp
485	18171630	291	0.000085	org.acm
486	18167296	2488	0.000013	org.travelblog
487	18167032	2243	0.000014	com.ingress
488	18165578	1264	0.000023	com.coschedule
489	18164766	1746	0.000018	com.financialexpress
490	18164648	1868	0.000017	com.allafrica
491	18164360	1110	0.000025	edu.princeton
492	18163672	2305	0.000014	com.tommy
493	18163376	1629	0.000019	org.whatbrowser
494	18162344	1299	0.000022	com.kinsta
495	18161312	2441	0.000014	com.algorithmia
496	18161140	530	0.000048	net.brightcove
497	18158432	2024	0.000016	jp.riken
498	18157666	831	0.000034	com.msdn
499	18157640	511	0.000049	edu.cornell
500	18156568	2279	0.000014	com.theminimalists
501	18153940	236	0.000103	to.amzn
502	18153840	1460	0.000020	net.noscript
503	18147056	305	0.000079	com.typeform
504	18146838	1727	0.000018	com.iconarchive
505	18145360	928	0.000030	org.weforum
506	18144622	838	0.000033	com.git-scm
507	18143540	2201	0.000014	net.organicfacts
508	18140242	1724	0.000018	com.gap
509	18138900	709	0.000039	org.bitbucket
510	18136630	403	0.000061	com.dailymotion
511	18134648	393	0.000062	com.nypost
512	18134424	2244	0.000014	com.bonfire
513	18133832	2095	0.000015	it.polito
514	18132572	903	0.000031	com.sfgate
515	18130544	239	0.000101	com.stumbleupon
516	18130386	2273	0.000014	net.brownbook
517	18129594	2073	0.000015	com.zynga
518	18127296	523	0.000048	edu.yale
519	18126134	1656	0.000019	com.wayfair
520	18125998	254	0.000096	org.drupal
521	18125926	381	0.000065	org.un
522	18123812	2300	0.000014	com.23hq
523	18122836	614	0.000045	gov.sec
524	18120156	415	0.000059	com.gmail
525	18119684	1196	0.000024	com.playstation
526	18118234	1717	0.000018	org.polymer-project
527	18113328	1772	0.000018	za.co.iol
528	18112548	1994	0.000016	au.com.huffingtonpost
529	18110922	2344	0.000014	com.marksdailyapple
530	18110894	1507	0.000020	com.impactbnd
531	18109740	648	0.000043	com.jwplatform
532	18109236	1513	0.000020	com.instapage
533	18107292	1374	0.000021	com.ning
534	18106990	2035	0.000015	com.dreamgrow
535	18106122	292	0.000085	cn.com.sina
536	18105010	2188	0.000014	net.openreview
537	18104806	1604	0.000019	com.aolcdn
538	18104004	210	0.000113	com.constantcontact
539	18103874	1889	0.000017	uk.ac.jisc
540	18103656	1993	0.000016	com.towardsdatascience
541	18101830	1669	0.000018	com.thermofisher
542	18100270	1924	0.000016	com.city-data
543	18099900	941	0.000030	uk.co.guardian
544	18099826	2136	0.000015	com.whitepages
545	18099500	1891	0.000017	com.deepmind
546	18098496	611	0.000046	com.mobirise
547	18097440	356	0.000068	com.springer
548	18096278	1929	0.000016	org.elasticsearch
549	18094390	743	0.000037	com.steampowered
550	18092048	1091	0.000026	com.auth0
551	18092008	192	0.000128	com.eepurl
552	18091694	1214	0.000024	kr.or.kisa
553	18090800	832	0.000034	gov.senate
554	18090404	71	0.000398	me.fb
555	18090188	3930	0.000010	com.artstation
556	18090110	684	0.000040	org.eff
557	18088506	2075	0.000015	com.quickanddirtytips
558	18088220	1856	0.000017	com.googledrive
559	18087890	2267	0.000014	lb.com.dailystar
560	18087370	1083	0.000026	de.spiegel
561	18087184	2189	0.000014	com.oilprice
562	18086598	1377	0.000021	io.bower
563	18086586	1997	0.000016	com.batchgeo
564	18086360	1060	0.000027	com.clicky
565	18085990	1172	0.000024	com.merriam-webster
566	18084746	1927	0.000016	com.nytco
567	18084272	284	0.000087	com.histats
568	18083856	1613	0.000019	org.jenkins-ci
569	18083580	1966	0.000016	com.underconsideration
570	18083090	2221	0.000014	com.swatch
571	18081868	617	0.000045	uk.co.blogspot
572	18078936	343	0.000071	com.sxsw
573	18078574	619	0.000045	com.patreon
574	18077388	1471	0.000020	io.getmdl
575	18076506	1081	0.000026	com.hollywoodreporter
576	18075394	610	0.000046	com.163
577	18075348	156	0.000166	ru.mail
578	18074940	1845	0.000017	com.rabbitmq
579	18074636	1783	0.000017	com.lexology
580	18074550	1665	0.000018	com.invisionapp
581	18074272	1987	0.000016	com.lightreading
582	18073906	1351	0.000021	edu.northwestern
583	18073556	996	0.000028	com.ubuntu
584	18073454	2111	0.000015	edu.dukeupress
585	18071852	2173	0.000015	org.onegreenplanet
586	18071480	2141	0.000015	com.hotfrog
587	18070592	2361	0.000014	edu.uah
588	18068872	1369	0.000021	org.khanacademy
589	18068438	1461	0.000020	uk.co.thesun
590	18066904	3487	0.000012	com.wikidot
591	18066370	1614	0.000019	com.digitaloceanspaces
592	18065602	2432	0.000014	net.sott
593	18065500	1547	0.000019	com.technologyreview
594	18065282	349	0.000070	com.staticflickr
595	18063590	78	0.000383	org.reactjs
596	18061954	660	0.000042	com.xinhuanet
597	18061776	2555	0.000013	com.idt
598	18061678	247	0.000098	de.amazon
599	18061268	739	0.000037	com.qz
600	18057936	1931	0.000016	com.googleapps
601	18057884	1753	0.000018	io.pantheon
602	18057768	2132	0.000015	net.eenews
603	18057116	779	0.000035	com.deloitte
604	18057038	1651	0.000019	com.checkatrade
605	18054622	657	0.000043	com.psychologytoday
606	18054596	900	0.000031	gov.nps
607	18051078	1973	0.000016	com.shoutmeloud
608	18049174	2761	0.000013	ca.411
609	18048620	1496	0.000020	com.citysearch
610	18048184	1748	0.000018	com.tutsplus
611	18044990	2126	0.000015	io.flutter
612	18044036	2064	0.000015	com.vanguardngr
613	18043242	1473	0.000020	edu.unc
614	18043174	1934	0.000016	com.gimletmedia
615	18042626	1627	0.000019	com.fifa
616	18041436	2463	0.000013	org.simile-widgets
617	18041230	932	0.000030	edu.upenn
618	18040930	2098	0.000015	com.designobserver
619	18040834	661	0.000042	org.pbs
620	18040552	2410	0.000014	com.ubu
621	18040422	1118	0.000025	net.recode
622	18039566	1208	0.000024	jobs.amazon
623	18038446	345	0.000071	com.tripod
624	18036562	1315	0.000022	edu.purdue
625	18035196	921	0.000030	com.variety
626	18034502	980	0.000029	com.alexa
627	18034150	1211	0.000024	us.imageshack
628	18033174	2198	0.000014	edu.arizona
629	18032862	2008	0.000016	in.huffingtonpost
630	18030734	1592	0.000019	com.yell
631	18030278	974	0.000029	org.sciencemag
632	18029728	1320	0.000022	uk.co.theregister
633	18028246	1679	0.000018	com.verywellmind
634	18025954	852	0.000033	org.worldbank
635	18025638	865	0.000033	io.readthedocs
636	18025104	130	0.000208	com.youku
637	18024714	2178	0.000015	com.epochtimes
638	18024306	2186	0.000015	info.bem
639	18023398	221	0.000107	com.taobao
640	18022288	1144	0.000025	com.elpais
641	18021800	1963	0.000016	org.dartlang
642	18021566	1088	0.000026	org.altervista
643	18021294	358	0.000068	org.debian
644	18020992	445	0.000056	com.force
645	18020940	1275	0.000023	com.ifttt
646	18020466	2209	0.000014	com.youm7
647	18019640	1073	0.000026	com.vox
648	18019568	1933	0.000016	com.hulu
649	18019032	2256	0.000014	au.com.yellowpages
650	18018980	2505	0.000013	com.pushwoosh
651	18016612	1177	0.000024	com.nydailynews
652	18016130	698	0.000039	gov.noaa
653	18014600	1657	0.000019	com.yext
654	18014022	958	0.000030	com.shutterstock
655	18013628	2320	0.000014	com.gifyu
656	18013320	1262	0.000023	com.storify
657	18013256	676	0.000041	com.samsung
658	18012944	1095	0.000026	edu.ucsd
659	18011978	422	0.000058	edu.nyu
660	18009736	696	0.000040	com.tandfonline
661	18009582	447	0.000055	com.atlassian
662	18009246	896	0.000031	com.geocities
663	18008812	439	0.000057	edu.cmu
664	18008746	2433	0.000014	com.yelloyello
665	18008602	780	0.000035	com.netflix
666	18007440	1291	0.000022	tv.ustream
667	18007104	620	0.000045	us.icio
668	18006812	1138	0.000025	edu.utexas
669	18005924	448	0.000055	com.gitlab
670	18005790	2093	0.000015	com.targetmarketingmag
671	18004306	2166	0.000015	com.cargurus
672	18004206	886	0.000032	com.docker
673	18002932	1191	0.000024	com.trustedshops
674	18002218	2479	0.000013	com.analyticsvidhya
675	18001434	2445	0.000013	com.2findlocal
676	17998520	845	0.000033	com.foxnews
677	17997146	2080	0.000015	jp.huffingtonpost
678	17995736	2637	0.000013	com.instructables
679	17995238	1945	0.000016	com.nokia
680	17995100	1197	0.000024	edu.academia
681	17992664	756	0.000036	com.gettyimages
682	17991230	245	0.000099	com.wpengine
683	17991084	2212	0.000014	ca.uwaterloo
684	17988686	2547	0.000013	com.cmgdigital
685	17987090	866	0.000033	edu.umich
686	17986974	693	0.000040	com.symantec
687	17986634	810	0.000034	net.2mdn
688	17986626	2129	0.000015	com.mondaq
689	17986164	952	0.000030	com.ycombinator
690	17985794	2206	0.000014	com.keepersecurity
691	17985096	388	0.000063	com.newrelic
692	17984746	2054	0.000015	com.doctoroz
693	17984534	908	0.000031	com.uservoice
694	17983862	207	0.000115	com.naver
695	17982684	1557	0.000019	com.pastebin
696	17980416	189	0.000132	com.xing
697	17978736	1857	0.000017	com.duckduckgo
698	17978312	956	0.000030	com.thinkwithgoogle
699	17978128	1907	0.000017	se.haxx
700	17976984	2007	0.000016	com.thecvf
701	17975926	2255	0.000014	au.com.truelocal
702	17974640	3534	0.000012	com.9to5mac
703	17974534	2130	0.000015	uk.co.yelp
704	17974162	1444	0.000020	fm.last
705	17974086	891	0.000032	com.dropboxusercontent
706	17973354	1554	0.000019	com.sankei
707	17972924	2205	0.000014	com.tiddlywiki
708	17971858	2315	0.000014	com.galvanize
709	17971240	2149	0.000015	es.huffingtonpost
710	17971108	249	0.000098	com.automattic
711	17969728	920	0.000031	com.investopedia
712	17967994	2235	0.000014	com.bizcommunity
713	17967458	1156	0.000025	org.cambridge
714	17967296	1220	0.000023	com.freeprivacypolicy
715	17967286	917	0.000031	org.change
716	17966542	2145	0.000015	com.winemag
717	17966324	2444	0.000014	com.maritime-executive
718	17965424	1052	0.000027	gov.uspto
719	17964464	2556	0.000013	com.alternion
720	17963358	1834	0.000017	com.autodesk
721	17963124	2411	0.000014	com.communitywalk
722	17962726	1839	0.000017	org.coursera
723	17962202	1255	0.000023	com.upwork
724	17960682	2341	0.000014	net.futurecdn
725	17959974	2089	0.000015	com.kudzu
726	17959858	2352	0.000014	com.ericsson
727	17958320	1832	0.000017	com.adespresso
728	17956922	2527	0.000013	edu.alamo
729	17956784	1260	0.000023	com.irishtimes
730	17956778	2342	0.000014	com.filedn
731	17956660	1353	0.000021	edu.usc
732	17956158	1041	0.000027	com.wunderground
733	17955722	864	0.000033	br.com.uol
734	17955718	697	0.000039	com.gartner
735	17955384	2254	0.000014	com.gamespot
736	17955254	2074	0.000015	com.btplc
737	17954432	2058	0.000015	com.showmelocal
738	17954256	2386	0.000014	com.massimodutti
739	17953774	2020	0.000016	edu.virginia
740	17953600	1731	0.000018	com.ikea
741	17953476	2260	0.000014	com.insiderpages
742	17953416	1274	0.000023	com.indiegogo
743	17952676	2030	0.000016	com.goinswriter
744	17949694	2440	0.000014	com.bershka
745	17949350	2184	0.000015	com.almanac
746	17949160	770	0.000036	gov.census
747	17946880	1233	0.000023	com.intuit
748	17945914	413	0.000060	com.inc
749	17944532	4347	0.000009	com.programmableweb
750	17943268	1132	0.000025	com.pcmag
751	17942666	2194	0.000014	com.writersdigest
752	17942134	2283	0.000014	com.citysquares
753	17941646	1535	0.000020	com.fiverr
754	17941316	1872	0.000017	com.csswizardry
755	17941296	1373	0.000021	com.vanityfair
756	17941172	1903	0.000017	jp.sankeibiz
757	17940796	2456	0.000013	com.live5news
758	17939234	1117	0.000025	gov.usgs
759	17938916	914	0.000031	com.zoho
760	17938282	3400	0.000012	com.freep
761	17937408	830	0.000034	com.blackberry
762	17937256	2163	0.000015	jp.booklog
763	17936844	2351	0.000014	com.thedrinksbusiness
764	17935426	1057	0.000027	com.politico
765	17935388	2197	0.000014	com.winefolly
766	17934768	687	0.000040	com.alibaba
767	17934366	2970	0.000013	com.jeeran
768	17934080	2494	0.000013	io.stackedit
769	17933862	1958	0.000016	ca.ubc
770	17933764	161	0.000163	me.line
771	17933538	3940	0.000010	org.greenpeace
772	17933062	2381	0.000014	com.yellowbook
773	17932336	2459	0.000013	za.co.bdlive
774	17932212	2396	0.000014	com.asianage
775	17932090	1631	0.000019	com.udemy
776	17932058	3415	0.000012	com.glamour
777	17931830	1635	0.000019	com.chrome
778	17931642	1194	0.000024	com.techrepublic
779	17931614	849	0.000033	com.unity3d
780	17931590	2033	0.000015	mp.j
781	17931536	598	0.000047	gov.usda
782	17931004	2365	0.000014	net.islamweb
783	17929334	808	0.000034	int.wipo
784	17928466	2355	0.000014	com.wsoctv
785	17927592	665	0.000042	com.marketo
786	17927048	1049	0.000027	edu.umn
787	17926932	414	0.000060	mp.mailchi
788	17926354	968	0.000029	com.aliexpress
789	17925900	2608	0.000013	org.torproject
790	17925360	2322	0.000014	com.utah
791	17925228	954	0.000030	com.sciencedaily
792	17924502	1932	0.000016	org.ap
793	17924098	724	0.000038	gov.house
794	17923976	2208	0.000014	com.chamberofcommerce
795	17923594	1953	0.000016	com.urbandictionary
796	17923558	2570	0.000013	com.spoke
797	17922794	2807	0.000013	com.salespider
798	17921176	2369	0.000014	com.ibmbigdatahub
799	17921126	981	0.000029	au.net.abc
800	17921074	1565	0.000019	com.problogger
801	17921008	533	0.000048	com.snapchat
802	17920922	1425	0.000021	fr.lemonde
803	17919346	141	0.000185	jp.co.google
804	17917592	5539	0.000006	cc.co
805	17917020	2017	0.000016	com.posterous
806	17916882	1129	0.000025	com.canva
807	17916104	1638	0.000019	com.britannica
808	17915782	2379	0.000014	com.wpxi
809	17915468	2040	0.000015	edu.cuny
810	17915450	2519	0.000013	com.americantowns
811	17914364	675	0.000041	gov.hhs
812	17913996	2287	0.000014	org.themoth
813	17913328	1420	0.000021	com.rollingstone
814	17913022	1245	0.000023	com.xkcd
815	17912984	3582	0.000011	edu.brown
816	17912880	634	0.000044	com.feedly
817	17912458	2447	0.000013	com.hdnux
818	17912396	2609	0.000013	com.zionsbank
819	17912316	2515	0.000013	com.pacegallery
820	17911434	2645	0.000013	com.tupalo
821	17911136	640	0.000044	au.com.google
822	17911060	843	0.000033	com.uk
823	17909056	128	0.000215	com.youtube-nocookie
824	17908862	1322	0.000022	com.vmware
825	17908568	2094	0.000015	org.semanticscholar
826	17908342	1970	0.000016	com.sanspo
827	17908248	1013	0.000028	com.java
828	17908178	2239	0.000014	it.scoop
829	17907842	470	0.000053	com.adweek
830	17907550	2301	0.000014	uk.co.dennis
831	17907474	2108	0.000015	jp.co.sankei
832	17906950	2576	0.000013	za.co.sowetanlive
833	17906748	504	0.000049	gov.copyright
834	17906250	362	0.000068	com.wufoo
835	17905562	2310	0.000014	edu.uci
836	17904908	2052	0.000015	jp.ne.iza
837	17904810	2543	0.000013	org.foodrevolution
838	17904456	2491	0.000013	com.thewritepractice
839	17904454	2147	0.000015	com.parksassociates
840	17904410	1111	0.000025	fr.blogspot
841	17904034	2636	0.000013	au.com.whitepages
842	17903878	1329	0.000022	com.billboard
843	17903360	1272	0.000023	com.prezi
844	17902216	2172	0.000015	com.local
845	17901414	320	0.000076	gov.ftc
846	17899970	1831	0.000017	edu.illinois
847	17899586	975	0.000029	com.indeed
848	17899066	811	0.000034	org.unesco
849	17898986	1852	0.000017	com.hatenablog
850	17898184	2497	0.000013	dk.brics
851	17898006	2118	0.000015	uk.ac.ed
852	17897618	1173	0.000024	org.unicef
853	17897232	425	0.000058	com.criteo
854	17896398	2151	0.000015	org.linuxfoundation
855	17896068	2215	0.000014	com.vendio
856	17895718	1981	0.000016	uk.ac.ucl
857	17894996	279	0.000089	com.marriott
858	17894196	6753	0.000005	com.blog
859	17893762	1187	0.000024	com.steamcommunity
860	17893582	834	0.000034	com.gofundme
861	17893354	2022	0.000016	net.privacypolicytemplate
862	17893038	4067	0.000009	com.virustotal
863	17892018	467	0.000053	com.iconfinder
864	17891616	2541	0.000013	com.lacartes
865	17891274	2269	0.000014	ai.fast
866	17891232	1796	0.000017	com.howstuffworks
867	17888984	1222	0.000023	com.dell
868	17888866	2473	0.000013	com.ibegin
869	17888166	1470	0.000020	com.over-blog
870	17887404	350	0.000069	net.themeforest
871	17887376	302	0.000080	com.netdna-ssl
872	17887272	3576	0.000011	edu.tufts
873	17886572	2487	0.000013	za.co.moneyweb
874	17886136	1689	0.000018	com.twilio
875	17886042	887	0.000032	com.hootsuite
876	17884576	1230	0.000023	com.gallup
877	17884332	2394	0.000014	com.machinelearningmastery
878	17882906	2389	0.000014	io.dropwizard
879	17882512	992	0.000028	com.att
880	17881416	2308	0.000014	com.ehow
881	17880788	3660	0.000011	com.discogs
882	17880724	2333	0.000014	com.blogs
883	17880688	2139	0.000015	com.dandb
884	17879696	486	0.000051	com.squareup
885	17879532	1037	0.000027	gov.bls
886	17879478	401	0.000061	com.bitly
887	17878540	3665	0.000011	com.twitpic
888	17878358	3064	0.000013	com.invoicesherpa
889	17877578	650	0.000043	com.herokuapp
890	17877102	2265	0.000014	ru.narod
891	17876570	1875	0.000017	com.tunein
892	17875252	1570	0.000019	com.com
893	17874650	1980	0.000016	jp.co.zakzak
894	17874052	613	0.000046	com.airbnb
895	17873488	2123	0.000015	uk.co.realbusiness
896	17872208	837	0.000033	gov.justice
897	17871838	1951	0.000016	co.gcdn
898	17871618	267	0.000091	com.myshopify
899	17870844	3504	0.000012	de.bild
900	17870478	234	0.000104	jp.co.amazon
901	17869976	1905	0.000017	org.filezilla-project
902	17869722	2574	0.000013	com.growtix
903	17869710	1922	0.000016	com.newsfactor
904	17868626	2775	0.000013	org.earthmagazine
905	17868240	3595	0.000011	cc.tiny
906	17868102	339	0.000072	org.opensource
907	17867628	1710	0.000018	org.owasp
908	17867434	1678	0.000018	org.cancer
909	17865220	370	0.000067	org.doi
910	17864752	1215	0.000024	ly.ow
911	17864458	2820	0.000013	co.iglobal
912	17863556	1330	0.000022	edu.uchicago
913	17863262	133	0.000206	de.bund
914	17862854	259	0.000094	com.getbootstrap
915	17862186	499	0.000050	com.nasdaq
916	17861824	1000	0.000028	com.lifehacker
917	17861748	1271	0.000023	org.pnas
918	17861644	395	0.000062	io.atom
919	17861324	1458	0.000020	in.blogspot
920	17860306	2644	0.000013	ai.becominghuman
921	17860248	2680	0.000013	com.googlemaps
922	17858530	2009	0.000016	net.nend
923	17856868	4730	0.000008	com.colourlovers
924	17856798	1413	0.000021	com.splashthat
925	17856676	982	0.000029	com.jetbrains
926	17856176	915	0.000031	jp.livedoor
927	17856152	303	0.000080	com.ssl-images-amazon
928	17854224	2643	0.000013	nl.zeelandnet
929	17853620	869	0.000032	com.pingdom
930	17853534	3078	0.000013	com.sophos
931	17852908	2525	0.000013	gr.huffingtonpost
932	17852142	1002	0.000028	de.blogspot
933	17850682	2760	0.000013	com.fox13memphis
934	17850488	2114	0.000015	com.richmediagallery
935	17850378	1821	0.000017	com.hotmail
936	17850366	72	0.000395	com.messenger
937	17850276	2231	0.000014	edu.asu
938	17850052	995	0.000028	org.iso
939	17849976	1389	0.000021	com.imimg
940	17849372	1145	0.000025	com.uber
941	17849120	2356	0.000014	com.tuck
942	17848556	1726	0.000018	com.nba
943	17848232	2404	0.000014	jp.news24
944	17847732	1559	0.000019	com.ogilvy
945	17847318	2772	0.000013	com.addustour
946	17846800	2831	0.000013	org.grayarea
947	17846714	2103	0.000015	com.homestars
948	17846650	1136	0.000025	com.seattletimes
949	17846580	265	0.000092	ru.rambler
950	17845988	2362	0.000014	edu.utah
951	17845868	3862	0.000010	com.starwars
952	17845640	479	0.000051	jp.ne.sakura
953	17844718	1063	0.000027	gov.congress
954	17843102	1410	0.000021	dk.datatilsynet
955	17842932	859	0.000033	com.stitcher
956	17842798	2971	0.000013	com.oilandgas360
957	17842486	1785	0.000017	edu.umd
958	17842430	758	0.000036	com.yandex
959	17840100	1885	0.000017	com.wetransfer
960	17839628	2457	0.000013	ms.1drv
961	17838212	977	0.000029	com.prweb
962	17838086	423	0.000058	com.smugmug
963	17837702	2414	0.000014	com.delta
964	17836356	2306	0.000014	edu.bu
965	17836156	1141	0.000025	com.500px
966	17834668	2796	0.000013	org.cmlibrary
967	17834248	2565	0.000013	com.fixr
968	17833764	1312	0.000022	com.firefox
969	17833368	2050	0.000015	edu.ufl
970	17831610	2409	0.000014	ca.ualberta
971	17831386	3977	0.000010	com.thingiverse
972	17830888	400	0.000061	com.discordapp
973	17830714	3817	0.000010	edu.unl
974	17829748	2746	0.000013	tw.com.ibon
975	17829134	2490	0.000013	au.com.hotfrog
976	17828966	2350	0.000014	de.mpg
977	17828928	1160	0.000025	com.timeanddate
978	17828580	2495	0.000013	com.figure-eight
979	17828574	2370	0.000014	com.codecademy
980	17827964	890	0.000032	gov.usa
981	17827518	256	0.000096	it.google
982	17827064	2427	0.000014	com.outboundengine
983	17826874	1308	0.000022	com.strikingly
984	17826840	1243	0.000023	com.target
985	17825758	2455	0.000013	com.theblogpress
986	17825300	2585	0.000013	com.expressbusinessdirectory
987	17825288	2216	0.000014	com.nfl
988	17825192	2607	0.000013	com.elocal
989	17825120	2628	0.000013	au.com.news
990	17824314	1116	0.000025	com.scientificamerican
991	17824168	1325	0.000022	co.vine
992	17823710	747	0.000037	com.cargocollective
993	17823530	691	0.000040	com.caniuse
994	17821930	2107	0.000015	com.angelfire
995	17820788	3005	0.000013	com.hbo
996	17820648	1639	0.000019	uk.co.screamingfrog
997	17820304	2617	0.000013	com.ovoenergy
998	17820010	737	0.000037	uk.co.eventbrite
999	17819726	2691	0.000013	com.normacomics
1000	17819576	752	0.000037	com.sagepub

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

July 2019 crawl archive now available

July 30, 2019Sebastian Nagel

The crawl archive for July 2019 is now available! It contains 2.6 billion web pages or 220 TiB of uncompressed content, crawled between July 15th and 24th.

The July crawl contains page captures of 810 million URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks (harmonic centrality) published as part of the Feb/Mar/Apr 2019 webgraph data set from the following sources:

a random sample of 2.0 billion outlinks taken from June crawl WAT files
1.8 billion URLs mined in a breadth-first side crawl within a maximum of 6 links (“hops”), started from
- the homepages of the top 60 million hosts and domains and randomly selected samples of
- 2 million human-readable sitemap pages (HTML format)
- 2 million URLs of pages written in 130 less-represented languages (cf. language distributions)
900 million URLs extracted and sampled from 20 million sitemaps, RSS and Atom feeds

Archive Location and Download

The July crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-30/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-30/segment.paths.gz	100
WARC files	CC-MAIN-2019-30/warc.paths.gz	56000	46.10
WAT files	CC-MAIN-2019-30/wat.paths.gz	56000	17.62
WET files	CC-MAIN-2019-30/wet.paths.gz	56000	7.80
Robots.txt files	CC-MAIN-2019-30/robotstxt.paths.gz	56000	0.14
Non-200 responses files	CC-MAIN-2019-30/non200responses.paths.gz	56000	1.63
URL index files	CC-MAIN-2019-30/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-30/. Also the columnar index has been updated to contain this crawl.

June 2019 crawl archive now available

July 2, 2019Sebastian Nagel

The crawl archive for June 2019 is now available! It contains 2.6 billion web pages or 220 TiB of uncompressed content, crawled between June 16th and 27th with an operational break from 21st to 24th.

The June crawl contains page captures of 880 million URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks (harmonic centrality) published as part of the Feb/Mar/Apr 2019 webgraph data set from the following sources:

sitemaps, RSS and Atom feeds
a breadth-first side crawl within a maximum of 6 links (“hops”) away from the homepages of the top 60 million hosts and domains and a random sample of 1 million human-readable sitemap pages (HTML format)
a random sample of 2.0 billion outlinks taken from May crawl WAT files

Starting with this crawl the WAT extraction has been improved by properly decoding HTML character entities in URLs and strings. For details, please see the issue report “WAT: unescape XML/HTML character entities”.

Archive Location and Download

The June crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-26/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-26/segment.paths.gz	100
WARC files	CC-MAIN-2019-26/warc.paths.gz	56000	49.42
WAT files	CC-MAIN-2019-26/wat.paths.gz	56000	17.24
WET files	CC-MAIN-2019-26/wet.paths.gz	56000	7.59
Robots.txt files	CC-MAIN-2019-26/robotstxt.paths.gz	56000	0.14
Non-200 responses files	CC-MAIN-2019-26/non200responses.paths.gz	56000	1.52
URL index files	CC-MAIN-2019-26/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-26/. Also the columnar index has been updated to contain this crawl.

May 2019 crawl archive now available

May 31, 2019Sebastian Nagel

The crawl archive for May 2019 is now available! It contains 2.65 billion web pages or 220 TiB of uncompressed content, crawled between May 19th and 27th.

The May crawl contains page captures of 825 million URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks (harmonic centrality) published as part of the Feb/Mar/Apr 2019 webgraph data set from the following sources:

sitemaps, RSS and Atom feeds
a breadth-first side crawl within a maximum of 4 links (“hops”) away from the homepages of the top 60 million hosts and domains and a random sample of 1 million human-readable sitemap pages (HTML format)
a random sample of 1.6 billion outlinks taken from WAT files of the April crawl

Archive Location and Download

The May crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-22/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-22/segment.paths.gz	100
WARC files	CC-MAIN-2019-22/warc.paths.gz	56000	51.48
WAT files	CC-MAIN-2019-22/wat.paths.gz	56000	17.75
WET files	CC-MAIN-2019-22/wet.paths.gz	56000	7.7
Robots.txt files	CC-MAIN-2019-22/robotstxt.paths.gz	56000	0.17
Non-200 responses files	CC-MAIN-2019-22/non200responses.paths.gz	56000	1.84
URL index files	CC-MAIN-2019-22/cc-index.paths.gz	302	0.2

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-22/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs Feb/Mar/Apr 2019

May 9, 2019Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of February, March and April 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

What’s new?

The software which builds the graph from WAT and WARC files has been extended to extract more links from the HTML <head> element:

more links are taken from <metadata> elements, e.g, the thumbnail meta name, Open Graph or twitter:* properties
links from <script> elements are now included

Note that previous web graph releases already include all kinds of links: not only <a href="..."> but also links to images and multi-media content, links from <form> elements, canonical links, and many more.

While the domain-level graph shows almost the same size and metrics as the previous one released three months ago, the host-level graph has increased in size by 85 million nodes but is less densely connected. The growth in the number of nodes is mainly caused by a link spam cluster of 190 million hosts distributed over 15k domains. Thanks to the webgraph these domains (e.g., 24340.tw) are detected and the crawler is advised not to visit them again.

Host-level graph

The graph consists of 492 million nodes and 3.0 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 426 million dangling nodes (87%) and the largest strongly connected component contains 52 million (10.5%) nodes.

You can download the graph and the ranks of all 492 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-feb-mar-apr/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-feb-mar-apr/host/ as prefix to access the files from everywhere.

Download files of the Common Crawl Feb/Mar/Apr 2019 host-level webgraph

Size	File	Description
3.36 GB	cc-main-2019-feb-mar-apr-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 28 vertices files
14.40 GB	cc-main-2019-feb-mar-apr-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 56 edges files

6.33 GB	cc-main-2019-feb-mar-apr-host.graph	graph in BVGraph format
2 kB	cc-main-2019-feb-mar-apr-host.properties

7.02 GB	cc-main-2019-feb-mar-apr-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2019-feb-mar-apr-host-t.properties

1 kB	cc-main-2019-feb-mar-apr-host.stats	WebGraph statistics

7.85 GB	cc-main-2019-feb-mar-apr-host-ranks.txt.gz	harmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 91 million nodes and 1.89 billion edges. 51% or 46 million nodes are dangling nodes, the largest strongly connected component covers 38 million or 42% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-feb-mar-apr/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-feb-mar-apr/domain/.

Download files of the Common Crawl Feb/Mar/Apr 2019 domain-level webgraph

Size	File	Description
0.63 GB	cc-main-2019-feb-mar-apr-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
7.48 GB	cc-main-2019-feb-mar-apr-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

4.01 GB	cc-main-2019-feb-mar-apr-domain.graph	graph in BVGraph format
2 kB	cc-main-2019-feb-mar-apr-domain.properties

4.02 GB	cc-main-2019-feb-mar-apr-domain-t.graph	transpose of the graph
2 kB	cc-main-2019-feb-mar-apr-domain-t.properties

1 kB	cc-main-2019-feb-mar-apr-domain.stats	WebGraph statistics

1.98 GB	cc-main-2019-feb-mar-apr-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 90 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Feb/Mar/Apr 2019)

harmonic centrality rank	hc value	page rank	page rank value	reversed hostname
1	29096444	1	0.020470	com.googleapis
2	28007982	3	0.012308	com.facebook
3	26801494	2	0.013202	com.google
4	24862042	4	0.006929	com.twitter
5	24696428	5	0.006794	com.youtube
6	24174048	6	0.006211	org.w
7	22325176	9	0.003651	com.instagram
8	22277546	7	0.004565	org.gmpg
9	21663616	13	0.002903	com.linkedin
10	21253740	8	0.003880	com.googletagmanager
11	20994008	22	0.001629	com.gravatar
12	20770844	11	0.003144	com.cloudflare
13	20763426	12	0.002915	org.wordpress
14	20723350	15	0.002103	com.wordpress
15	20597380	19	0.001856	com.pinterest
16	20589106	26	0.001344	org.wikipedia
17	20538196	14	0.002455	com.bootstrapcdn
18	20340120	18	0.001857	com.apple
19	20168332	28	0.001208	com.vimeo
20	20100244	40	0.000945	com.blogspot
21	20076842	21	0.001721	com.jquery
22	19900496	44	0.000861	gl.goo
23	19874514	49	0.000756	be.youtu
24	19845858	24	0.001528	com.adobe
25	19808478	27	0.001240	com.microsoft
26	19798758	50	0.000749	com.amazon
27	19710384	60	0.000607	com.tumblr
28	19689216	53	0.000667	com.wp
29	19584236	34	0.001016	com.amazonaws
30	19567898	25	0.001456	com.macromedia
31	19563206	88	0.000433	com.yahoo
32	19557094	51	0.000734	com.flickr
33	19526652	42	0.000880	com.google-analytics
34	19499686	80	0.000502	ly.bit
35	19489648	32	0.001034	com.googlesyndication
36	19472580	62	0.000580	org.mozilla
37	19466578	23	0.001557	com.gstatic
38	19459466	31	0.001093	net.cloudfront
39	19428390	20	0.001823	com.github
40	19302390	66	0.000558	me.wp
41	19278286	39	0.000949	net.doubleclick
42	19253848	46	0.000802	com.paypal
43	19222312	99	0.000316	com.googleusercontent
44	19214440	82	0.000487	com.medium
45	19194374	41	0.000882	com.squarespace
46	19181944	85	0.000440	com.weebly
47	19164390	79	0.000520	org.w3
48	19162880	127	0.000234	com.nytimes
49	19140860	86	0.000440	io.github
50	19138696	102	0.000307	com.reddit
51	19125448	92	0.000375	org.creativecommons
52	19052088	154	0.000166	net.slideshare
53	19050126	162	0.000162	com.theguardian
54	19047964	139	0.000189	com.imgur
55	19010700	57	0.000626	com.bing
56	19007804	136	0.000202	com.forbes
57	18975024	166	0.000158	net.sourceforge
58	18969344	217	0.000110	com.businessinsider
59	18964518	64	0.000566	org.schema
60	18930562	202	0.000115	com.myspace
61	18929412	161	0.000162	com.blogger
62	18929098	206	0.000113	com.techcrunch
63	18929096	188	0.000132	com.android
64	18907716	101	0.000313	com.mailchimp
65	18887038	251	0.000097	com.tinyurl
66	18886912	54	0.000649	com.baidu
67	18881598	249	0.000098	com.wired
68	18879314	91	0.000411	de.google
69	18872146	354	0.000068	com.photobucket
70	18870082	182	0.000140	com.stackoverflow
71	18844718	100	0.000316	org.ampproject
72	18842202	38	0.000953	org.apache
73	18829206	266	0.000090	com.bbc
74	18824202	103	0.000307	com.shopify
75	18821492	361	0.000068	com.quora
76	18818304	315	0.000076	com.appspot
77	18801506	37	0.000974	com.fontawesome
78	18797872	113	0.000275	com.ytimg
79	18796778	36	0.000976	com.addthis
80	18776040	209	0.000112	com.oracle
81	18766484	558	0.000045	org.chromium
82	18761466	353	0.000069	com.googleblog
83	18753718	380	0.000064	com.theverge
84	18728722	526	0.000047	org.ieee
85	18726166	510	0.000048	edu.washington
86	18724898	462	0.000053	com.economist
87	18724374	96	0.000330	com.statcounter
88	18720414	98	0.000317	com.soundcloud
89	18717340	151	0.000171	org.ietf
90	18714060	553	0.000046	edu.yale
91	18706362	319	0.000075	com.githubusercontent
92	18703214	300	0.000078	com.ted
93	18695606	61	0.000589	eu.europa
94	18694532	441	0.000056	com.venturebeat
95	18691888	235	0.000103	com.hubspot
96	18688542	655	0.000043	com.tinypic
97	18680252	144	0.000180	com.spotify
98	18673738	141	0.000185	com.yelp
99	18671830	133	0.000213	com.issuu
100	18662516	395	0.000063	com.cisco
101	18657986	93	0.000354	co.t
102	18652360	95	0.000340	com.sharethis
103	18648440	438	0.000056	com.deviantart
104	18644944	702	0.000040	edu.princeton
105	18644036	265	0.000090	com.sciencedirect
106	18635732	360	0.000068	me.about
107	18631710	460	0.000053	org.arxiv
108	18627650	279	0.000086	org.npr
109	18615844	179	0.000141	org.wikimedia
110	18608530	751	0.000038	google.blog
111	18606402	341	0.000071	com.theatlantic
112	18596892	345	0.000071	com.mozilla
113	18592502	572	0.000044	edu.ucla
114	18587904	454	0.000054	com.mysql
115	18584416	134	0.000211	com.dropbox
116	18581890	963	0.000033	com.jetbrains
117	18579386	121	0.000250	com.whatsapp
118	18576676	294	0.000081	com.example
119	18575790	81	0.000499	net.jsdelivr
120	18574812	271	0.000089	com.fastcompany
121	18567024	331	0.000072	com.typeform
122	18560472	400	0.000062	com.zdnet
123	18556406	468	0.000052	com.wikihow
124	18554544	30	0.001112	ru.yandex
125	18552784	540	0.000046	com.thenextweb
126	18551678	502	0.000049	com.git-scm
127	18550824	1063	0.000030	com.chrome
128	18546650	169	0.000156	com.salesforce
129	18542982	375	0.000065	uk.co.blogspot
130	18537770	443	0.000055	com.about
131	18535658	117	0.000263	org.networkadvertising
132	18535608	488	0.000050	com.pixabay
133	18526084	212	0.000112	com.dribbble
134	18525790	201	0.000116	com.stumbleupon
135	18524658	1434	0.000021	com.diigo
136	18509334	499	0.000049	com.ubuntu
137	18501930	741	0.000038	org.eclipse
138	18497596	505	0.000049	com.slate
139	18497252	208	0.000112	com.googlecode
140	18490168	58	0.000611	com.wix
141	18486604	425	0.000058	com.moz
142	18481186	191	0.000127	com.cnn
143	18475472	122	0.000242	com.stripe
144	18475424	181	0.000141	uk.co.bbc
145	18464340	652	0.000043	com.stackexchange
146	18462864	369	0.000066	com.entrepreneur
147	18458960	275	0.000087	com.nbcnews
148	18453376	253	0.000095	gov.ca
149	18444504	595	0.000044	com.withgoogle
150	18440430	518	0.000048	com.qz
151	18439414	542	0.000046	com.trello
152	18429892	214	0.000111	edu.stanford
153	18426238	1071	0.000030	edu.illinois
154	18424824	1040	0.000030	edu.gatech
155	18424290	293	0.000081	com.foursquare
156	18421718	1509	0.000020	org.wikibooks
157	18418178	573	0.000044	com.searchengineland
158	18417420	516	0.000048	com.unity3d
159	18414768	670	0.000042	org.sciencemag
160	18412040	267	0.000090	com.npmjs
161	18401668	463	0.000053	gov.loc
162	18397856	926	0.000034	com.sap
163	18397328	16	0.002068	com.wixstatic
164	18396224	1097	0.000029	edu.rutgers
165	18394270	156	0.000165	org.bbb
166	18392954	213	0.000111	es.google
167	18392682	661	0.000042	com.variety
168	18391296	155	0.000166	com.twimg
169	18381936	538	0.000046	com.libsyn
170	18380340	546	0.000046	com.evernote
171	18380064	174	0.000152	com.imdb
172	18378896	211	0.000112	com.wsj
173	18377090	33	0.001018	net.fbcdn
174	18373124	142	0.000185	gov.privacyshield
175	18369136	705	0.000040	com.techtarget
176	18368040	45	0.000851	com.fb
177	18365978	1190	0.000026	edu.utah
178	18365960	146	0.000180	org.archive
179	18365486	377	0.000065	com.getpocket
180	18358592	439	0.000056	gov.fda
181	18357482	194	0.000125	com.optimizely
182	18350542	419	0.000060	au.com.google
183	18346246	904	0.000034	com.econsultancy
184	18346130	210	0.000112	net.windows
185	18345860	1315	0.000024	com.douban
186	18345148	449	0.000055	org.freecodecamp
187	18338274	1321	0.000023	com.discogs
188	18338174	613	0.000043	uk.ac.ox
189	18335780	1019	0.000031	com.nike
190	18332678	1207	0.000026	org.tensorflow
191	18325660	75	0.000531	com.vk
192	18324804	287	0.000082	edu.mit
193	18323908	953	0.000033	com.buffer
194	18323654	1203	0.000026	com.aljazeera
195	18321184	1133	0.000028	ca.utoronto
196	18317982	813	0.000036	com.netlify
197	18315894	1079	0.000030	com.nvidia
198	18314254	531	0.000047	net.azurewebsites
199	18311612	350	0.000069	com.msn
200	18310748	984	0.000032	org.kernel
201	18307064	1123	0.000028	it.scoop
202	18305946	94	0.000346	com.paypalobjects
203	18299178	723	0.000039	com.indeed
204	18298870	807	0.000036	com.mixcloud
205	18296584	236	0.000103	com.live
206	18291116	1121	0.000028	org.postgresql
207	18289064	810	0.000036	com.neilpatel
208	18280540	365	0.000067	com.discordapp
209	18269300	1217	0.000026	ms.1drv
210	18269146	942	0.000034	com.business2community
211	18267622	303	0.000078	com.reuters
212	18266166	389	0.000064	gov.nasa
213	18263344	1452	0.000021	com.makeuseof
214	18261880	153	0.000168	gov.nih
215	18261660	444	0.000055	com.udacity
216	18257262	1289	0.000024	com.hostgator
217	18253994	996	0.000032	com.chron
218	18252416	264	0.000091	com.ibm
219	18243366	1024	0.000031	com.socialmediaexaminer
220	18240594	1126	0.000028	com.trendmicro
221	18238016	203	0.000114	com.washingtonpost
222	18236948	1117	0.000028	com.computerworld
223	18235154	568	0.000045	com.images-amazon
224	18232544	180	0.000141	com.etsy
225	18231292	1340	0.000023	io.itch
226	18226226	773	0.000037	co.g
227	18218104	1206	0.000026	edu.osu
228	18215060	968	0.000033	com.yoast
229	18211670	1277	0.000024	com.hbo
230	18210326	190	0.000130	com.ebay
231	18208488	304	0.000077	com.cnet
232	18207542	291	0.000082	edu.harvard
233	18207312	1766	0.000017	com.pearltrees
234	18206394	956	0.000033	com.mediafire
235	18206036	715	0.000039	site.business
236	18201166	56	0.000629	net.akamaihd
237	18200054	1055	0.000030	com.healthline
238	18198490	483	0.000051	com.usnews
239	18196662	196	0.000120	com.huffingtonpost
240	18192554	1144	0.000027	com.bustle
241	18190242	1001	0.000032	com.me
242	18188000	557	0.000045	org.d3js
243	18186074	165	0.000159	com.eventbrite
244	18185850	87	0.000437	com.list-manage
245	18185730	1737	0.000018	com.panoramio
246	18184450	384	0.000064	com.mashable
247	18181712	465	0.000053	edu.berkeley
248	18181104	803	0.000036	co.ibb
249	18180796	277	0.000087	com.bloomberg
250	18177922	760	0.000037	com.adjust
251	18177008	822	0.000036	com.ecwid
252	18174004	349	0.000070	com.mapbox
253	18172194	891	0.000034	gov.wa
254	18172152	1028	0.000031	org.aarp
255	18170224	1158	0.000027	edu.brookings
256	18169502	192	0.000127	org.iana
257	18165772	1250	0.000025	com.dw
258	18165654	1245	0.000025	com.medicalnewstoday
259	18163584	218	0.000109	net.php
260	18163574	420	0.000059	me.telegram
261	18163048	296	0.000081	org.acm
262	18162354	207	0.000112	org.gnu
263	18158198	1476	0.000021	com.sas
264	18152116	611	0.000043	me.paypal
265	18150248	1561	0.000020	com.dezeen
266	18150072	1088	0.000029	com.cio
267	18149100	799	0.000036	co.elastic
268	18148524	1293	0.000024	uk.org.tate
269	18147684	402	0.000062	com.latimes
270	18146412	199	0.000118	uk.co.amazon
271	18146172	455	0.000054	com.bigcommerce
272	18145158	1565	0.000020	be.blogspot
273	18144468	1260	0.000025	com.hackernoon
274	18143080	316	0.000076	uk.co.telegraph
275	18141808	1428	0.000022	com.googlesource
276	18141690	1448	0.000021	edu.iastate
277	18138884	1698	0.000018	org.edublogs
278	18137392	1559	0.000020	com.mathworks
279	18134024	1234	0.000025	gov.michigan
280	18132964	372	0.000066	com.livejournal
281	18132828	1140	0.000028	com.xrea
282	18132092	1623	0.000019	li.paper
283	18128158	47	0.000767	com.qq
284	18127880	1679	0.000018	com.dummies
285	18126586	143	0.000183	com.unpkg
286	18123856	1015	0.000031	com.searchenginejournal
287	18122824	939	0.000034	com.searchenginewatch
288	18120806	1802	0.000017	fr.unblog
289	18119164	286	0.000083	com.go
290	18114598	658	0.000042	com.livechatinc
291	18113078	149	0.000173	com.opera
292	18112360	757	0.000037	au.gov.nsw
293	18110326	1162	0.000027	va.vatican
294	18104616	1474	0.000021	jp.ac.u-tokyo
295	18104360	1016	0.000031	uk.co.pinterest
296	18103034	374	0.000065	com.elsevier
297	18097712	1412	0.000022	com.activecampaign
298	18097614	311	0.000076	com.meetup
299	18097106	1925	0.000016	com.jigsy
300	18094268	1084	0.000029	uk.gov.nationalarchives
301	18092632	1219	0.000026	us.mn.state
302	18091718	1246	0.000025	com.firebaseapp
303	18091416	1247	0.000025	com.convinceandconvert
304	18090158	1367	0.000023	us.fl.state
305	18083166	1700	0.000018	org.emojipedia
306	18079266	529	0.000047	com.adage
307	18079076	1192	0.000026	org.maven
308	18077604	1341	0.000023	gov.mo
309	18073450	43	0.000878	net.facebook
310	18070040	712	0.000039	gov.dot
311	18069866	164	0.000160	uk.co.google
312	18069354	89	0.000415	com.godaddy
313	18068256	172	0.000155	com.zendesk
314	18066418	222	0.000106	com.typepad
315	18066074	278	0.000087	com.usatoday
316	18062078	324	0.000074	com.mapquest
317	18057670	1423	0.000022	gov.ky
318	18052106	1687	0.000018	com.manta
319	18050286	426	0.000058	org.hbr
320	18050186	490	0.000050	net.researchgate
321	18049408	327	0.000073	com.getclicky
322	18049396	1398	0.000022	com.convertkit
323	18048286	2155	0.000014	it.justpaste
324	18048048	1348	0.000023	com.creativebloq
325	18047592	1544	0.000020	org.aclweb
326	18045082	889	0.000034	com.wordstream
327	18043064	1539	0.000020	ly.snip
328	18043014	227	0.000105	com.giphy
329	18040924	2014	0.000015	me.websta
330	18040570	356	0.000068	com.sxsw
331	18040058	840	0.000035	edu.psu
332	18037266	1365	0.000023	gov.maryland
333	18032816	1813	0.000017	ca.yelp
334	18031110	1187	0.000026	com.fastcodesign
335	18030678	1685	0.000018	io.material
336	18030052	1494	0.000021	org.amnesty
337	18027220	269	0.000089	org.python
338	18026066	522	0.000048	org.mediawiki
339	18026002	479	0.000051	com.buzzfeed
340	18024772	1150	0.000027	com.findlaw
341	18023106	783	0.000037	com.arstechnica
342	18022710	382	0.000064	com.oreilly
343	18022180	1667	0.000019	edu.toronto
344	18019428	1703	0.000018	com.healthgrades
345	18018232	2078	0.000015	tl.page
346	18016586	477	0.000051	edu.cornell
347	18013106	342	0.000071	com.springer
348	18008962	404	0.000062	it.placehold
349	18006368	1566	0.000020	com.raywenderlich
350	18005638	383	0.000064	com.nypost
351	18005080	937	0.000034	com.contentmarketinginstitute
352	18002754	325	0.000073	int.who
353	18001954	513	0.000048	org.nodejs
354	17999682	1524	0.000020	gov.mt
355	17997974	1345	0.000023	us.pa.state
356	17996652	329	0.000073	com.cnbc
357	17993466	1363	0.000023	gov.oregon
358	17990558	1066	0.000030	com.bandsintown
359	17988684	392	0.000063	com.gmail
360	17988550	1734	0.000018	com.wayfair
361	17984480	332	0.000072	fr.free
362	17980878	237	0.000102	org.drupal
363	17979326	910	0.000034	com.angieslist
364	17979146	450	0.000055	com.kickstarter
365	17979024	2154	0.000014	com.brandyourself
366	17978800	399	0.000062	uk.co.dailymail
367	17978276	1352	0.000023	com.quicksprout
368	17977606	260	0.000093	uk.org.ico
369	17977242	469	0.000052	gov.whitehouse
370	17974632	1014	0.000031	com.speakerdeck
371	17972780	240	0.000102	com.rawgit
372	17964500	659	0.000042	com.intel
373	17953938	899	0.000034	com.wikia
374	17951944	83	0.000482	com.googleadservices
375	17951496	506	0.000049	com.box
376	17945642	1418	0.000022	com.huffpost
377	17944878	1086	0.000029	net.leadpages
378	17943278	508	0.000049	com.cbsnews
379	17943212	323	0.000075	com.time
380	17942606	1976	0.000015	com.zynga
381	17940634	242	0.000101	com.getbootstrap
382	17940514	811	0.000036	com.superpages
383	17940204	1583	0.000019	com.impactbnd
384	17940144	177	0.000144	jp.co.yahoo
385	17937886	67	0.000548	net.jsfiddle
386	17936062	1271	0.000024	com.smallbiztrends
387	17936026	1511	0.000020	org.gnupg
388	17934886	1461	0.000021	co.leadpages
389	17934500	314	0.000076	com.staticflickr
390	17933798	1225	0.000025	com.googlegroups
391	17933250	1488	0.000021	com.thumbtack
392	17932302	367	0.000066	com.ft
393	17931464	2051	0.000015	com.ewtn
394	17929878	370	0.000066	com.office
395	17928686	1576	0.000019	com.kaggle
396	17926934	137	0.000200	com.wixsite
397	17922076	1996	0.000015	org.spie
398	17919510	1641	0.000019	com.thecut
399	17918918	1261	0.000025	com.ebayimg
400	17917840	1806	0.000017	com.googledrive
401	17917748	391	0.000063	com.aol
402	17917512	1573	0.000019	org.jenkins-ci
403	17915334	434	0.000056	com.fortune
404	17911152	2275	0.000014	net.organicfacts
405	17910680	357	0.000068	com.unsplash
406	17910196	2118	0.000015	it.polito
407	17902706	1753	0.000018	com.mindbodygreen
408	17899942	599	0.000043	com.proofpoint
409	17899484	1161	0.000027	edu.ucsd
410	17898182	2239	0.000014	net.furaffinity
411	17895926	766	0.000037	com.engadget
412	17895860	131	0.000218	com.weibo
413	17895616	229	0.000104	com.surveymonkey
414	17895124	1556	0.000020	com.crashlytics
415	17891646	1522	0.000020	com.toptal
416	17891064	298	0.000079	com.skype
417	17890434	1310	0.000024	com.avvo
418	17889754	2012	0.000015	com.doctoroz
419	17889278	1351	0.000023	io.fabric
420	17888404	1652	0.000019	com.thoughtworks
421	17888344	119	0.000253	com.jimdo
422	17884296	339	0.000071	com.w3schools
423	17882826	381	0.000064	org.un
424	17882360	1912	0.000016	com.mysanantonio
425	17880012	1426	0.000022	com.carto
426	17877832	1478	0.000021	com.grammarly
427	17876928	933	0.000034	com.pexels
428	17875774	1485	0.000021	org.sqlite
429	17875586	132	0.000214	com.youtube-nocookie
430	17873060	986	0.000032	com.gizmodo
431	17868426	1747	0.000018	gov.arts
432	17868138	992	0.000032	edu.upenn
433	17866058	1074	0.000030	org.vim
434	17864842	1812	0.000017	com.instapaper
435	17862354	690	0.000040	com.vice
436	17861898	674	0.000041	gov.nist
437	17857890	70	0.000536	org.reactjs
438	17857428	1877	0.000016	gov.la
439	17857390	1712	0.000018	com.politifact
440	17856846	826	0.000036	com.blackberry
441	17856382	1584	0.000019	com.ogilvy
442	17856122	719	0.000039	com.msdn
443	17855302	2249	0.000014	edu.utep
444	17855124	1578	0.000019	com.citysearch
445	17854150	893	0.000034	edu.umich
446	17852538	223	0.000106	net.behance
447	17850120	2156	0.000014	com.dynamics
448	17849038	390	0.000063	com.booking
449	17846000	2273	0.000014	com.asmallorange
450	17843806	1311	0.000024	com.curbed
451	17842712	478	0.000051	com.herokuapp
452	17842298	216	0.000111	com.automattic
453	17842098	1541	0.000020	org.aiga
454	17842080	923	0.000034	org.worldbank
455	17841888	147	0.000176	com.aspnetcdn
456	17841278	1850	0.000017	com.deepmind
457	17841020	1228	0.000025	com.sprinklr
458	17840682	1058	0.000030	com.thinkwithgoogle
459	17839270	2360	0.000013	it.clyp
460	17838612	1540	0.000020	com.instapage
461	17837952	272	0.000088	com.digg
462	17837540	1614	0.000019	com.cmswire
463	17836636	447	0.000055	com.goodreads
464	17835202	2033	0.000015	au.com.huffingtonpost
465	17834194	681	0.000041	com.symantec
466	17832794	385	0.000064	com.dailymotion
467	17832500	1606	0.000019	com.vendio
468	17832000	2204	0.000014	net.openreview
469	17831708	865	0.000035	net.openid
470	17830532	2206	0.000014	com.kvue
471	17830032	171	0.000155	com.feedburner
472	17829348	1479	0.000021	gov.wi
473	17826692	2056	0.000015	com.kudzu
474	17826526	2047	0.000015	com.stamen
475	17826378	1275	0.000024	com.merriam-webster
476	17824996	1510	0.000020	com.csoonline
477	17819228	1901	0.000016	it.binged
478	17818128	1394	0.000022	com.coschedule
479	17817378	2187	0.000014	com.writersdigest
480	17817048	699	0.000040	org.bitbucket
481	17815720	876	0.000035	edu.columbia
482	17815700	1897	0.000016	google.ai
483	17814606	1244	0.000025	com.auth0
484	17814108	1112	0.000029	edu.utexas
485	17813510	1103	0.000029	org.weforum
486	17811808	1757	0.000018	com.merchantcircle
487	17811776	2732	0.000013	com.bitballoon
488	17811618	2121	0.000015	edu.dukeupress
489	17810846	2018	0.000015	com.ingress
490	17808694	148	0.000175	com.tripadvisor
491	17808548	1858	0.000016	com.king5
492	17808180	307	0.000077	com.wiley
493	17803782	1958	0.000016	com.nngroup
494	17803738	1457	0.000021	com.vanityfair
495	17801028	337	0.000072	com.hp
496	17797994	125	0.000236	jp.co.google
497	17797558	320	0.000075	com.scribd
498	17795684	336	0.000072	com.tripod
499	17794742	701	0.000040	io.codepen
500	17794630	2162	0.000014	io.prototypr
501	17794528	446	0.000055	com.aliyuncs
502	17794052	972	0.000033	uk.co.guardian
503	17793662	566	0.000045	com.samsung
504	17793286	451	0.000055	com.slack
505	17793034	685	0.000041	org.eff
506	17791936	547	0.000046	com.webs
507	17789994	474	0.000052	com.atlassian
508	17789800	198	0.000119	de.amazon
509	17789764	2815	0.000012	edu.alamo
510	17788872	1520	0.000020	com.jeffbullas
511	17785572	1844	0.000017	ca.ubc
512	17783666	452	0.000054	com.newrelic
513	17778998	1776	0.000017	com.financialexpress
514	17777484	1051	0.000030	com.yellowpages
515	17777164	1647	0.000019	org.owasp
516	17776950	1209	0.000026	org.whatbrowser
517	17772704	1422	0.000022	org.tigris
518	17771794	1723	0.000018	com.thermofisher
519	17771104	429	0.000057	com.businesswire
520	17769374	1664	0.000019	org.wikidata
521	17769220	205	0.000113	com.bandcamp
522	17768404	195	0.000122	com.constantcontact
523	17767070	1444	0.000021	com.pcworld
524	17766282	861	0.000035	com.dropboxusercontent
525	17763526	1233	0.000025	edu.purdue
526	17762444	297	0.000080	com.wufoo
527	17762030	734	0.000038	com.createjs
528	17761810	396	0.000063	com.force
529	17759846	565	0.000045	in.co.google
530	17759466	364	0.000067	org.doi
531	17757706	2193	0.000014	com.hotfrog
532	17757234	863	0.000035	com.foxnews
533	17756726	1402	0.000022	org.letsencrypt
534	17755956	200	0.000117	org.icann
535	17755908	418	0.000060	com.inc
536	17755824	1528	0.000020	com.invisionapp
537	17755374	2322	0.000013	com.yellowbook
538	17755084	295	0.000081	gov.cdc
539	17752452	1135	0.000028	org.altervista
540	17749954	2167	0.000014	com.khou
541	17749580	2106	0.000015	com.quickanddirtytips
542	17749254	1416	0.000022	org.sonatype
543	17749176	2422	0.000013	es.iac
544	17749162	170	0.000156	ru.mail
545	17748102	1281	0.000024	com.storify
546	17745564	1185	0.000026	us.imageshack
547	17745434	2359	0.000013	org.hg
548	17743856	696	0.000040	com.psychologytoday
549	17743458	1251	0.000025	com.upwork
550	17743324	1052	0.000030	com.ycombinator
551	17742228	1646	0.000019	com.kinsta
552	17742204	1027	0.000031	com.hootsuite
553	17741772	1204	0.000026	ca.blogspot
554	17741450	2284	0.000014	com.theminimalists
555	17738328	1254	0.000025	com.ifttt
556	17732728	299	0.000079	com.prnewswire
557	17732640	2086	0.000015	jp.riken
558	17730736	1938	0.000016	at.tugraz
559	17730652	841	0.000035	com.docker
560	17730110	1337	0.000023	in.blogspot
561	17728102	2132	0.000014	com.theoutline
562	17727422	1172	0.000027	com.indiegogo
563	17724128	954	0.000033	com.alexa
564	17723922	3550	0.000012	com.twitpic
565	17723324	576	0.000044	com.windowsphone
566	17723084	1173	0.000027	com.homeadvisor
567	17722600	1694	0.000018	uk.co.metro
568	17720038	2745	0.000013	com.idt
569	17719404	2456	0.000013	com.23hq
570	17717984	1471	0.000021	org.khanacademy
571	17716196	1821	0.000017	org.elasticsearch
572	17715978	767	0.000037	com.indiatimes
573	17715462	1989	0.000015	com.shoutmeloud
574	17714586	480	0.000051	com.nature
575	17713698	414	0.000060	edu.cmu
576	17713112	1856	0.000017	com.city-data
577	17712356	2116	0.000015	com.kgw
578	17711724	1238	0.000025	org.pewresearch
579	17711380	975	0.000033	com.sfgate
580	17711188	1672	0.000018	gov.nh
581	17711070	1983	0.000015	google.design
582	17708460	457	0.000053	com.gitlab
583	17708384	507	0.000049	uk.co.independent
584	17708180	1678	0.000018	org.polymer-project
585	17708112	2129	0.000014	org.designmuseum
586	17708080	219	0.000108	jp.ne.hatena
587	17707174	224	0.000106	to.amzn
588	17704286	1143	0.000027	edu.wisc
589	17703708	527	0.000047	com.statista
590	17702676	802	0.000036	com.netflix
591	17702444	1208	0.000026	com.firefox
592	17701640	2944	0.000012	edu.brown
593	17700316	1777	0.000017	com.tutsplus
594	17699332	2197	0.000014	ca.uwaterloo
595	17697030	2189	0.000014	com.company
596	17696512	1609	0.000019	com.martechtoday
597	17696414	560	0.000045	org.pbs
598	17696178	1632	0.000019	com.fiverr
599	17693936	2353	0.000013	com.instructables
600	17693836	1226	0.000025	com.clicky
601	17693516	244	0.000101	com.wpengine
602	17693384	1009	0.000031	com.uservoice
603	17690248	3133	0.000012	net.digitalcongo
604	17688104	496	0.000049	us.icio
605	17688098	1724	0.000018	us.nm.state
606	17685732	2920	0.000012	com.wvec
607	17685630	2831	0.000012	com.growtix
608	17684812	1786	0.000017	us.ma.state
609	17684532	1116	0.000028	uk.ac.cam
610	17684330	2372	0.000013	com.warriorplus
611	17684134	957	0.000033	com.shutterstock
612	17683636	1314	0.000024	uk.co.theregister
613	17682188	1007	0.000032	es.agpd
614	17682158	2168	0.000014	com.what3words
615	17680382	1876	0.000016	com.itsnicethat
616	17679870	326	0.000073	org.joomla
617	17676374	2153	0.000014	com.dreamgrow
618	17675740	1212	0.000026	com.playstation
619	17674912	2325	0.000013	org.webpagetest
620	17674664	1815	0.000017	io.pantheon
621	17673952	3026	0.000012	org.nalip
622	17673070	1382	0.000022	com.digitaltrends
623	17672802	2256	0.000014	com.googlelabs
624	17672798	106	0.000299	net.2mdn
625	17671734	509	0.000049	tv.twitch
626	17671174	1168	0.000027	com.steamcommunity
627	17670820	2061	0.000015	com.targetmarketingmag
628	17670692	178	0.000144	me.line
629	17670586	2801	0.000013	co.edureka
630	17670360	2230	0.000014	eu.i-scoop
631	17670228	1929	0.000016	com.wral
632	17669872	1974	0.000015	us.wi.state
633	17668578	2229	0.000014	net.wrightflyer
634	17666822	2355	0.000013	gov.cabq
635	17666534	340	0.000071	com.bitly
636	17666208	368	0.000066	cn.com.sina
637	17665830	1376	0.000022	com.intuit
638	17665486	1339	0.000023	kr.or.kisa
639	17665464	1043	0.000030	com.newsweek
640	17665278	1152	0.000027	edu.northwestern
641	17664282	2384	0.000013	edu.uah
642	17663658	1816	0.000017	com.rabbitmq
643	17662888	2067	0.000015	com.wfaa
644	17662822	1312	0.000024	com.ning
645	17662498	1923	0.000016	ch.ethz
646	17661652	1622	0.000019	com.sharefile
647	17661252	1259	0.000025	com.pcmag
648	17660468	407	0.000061	edu.nyu
649	17659788	1029	0.000031	gov.fcc
650	17658992	348	0.000070	org.opensource
651	17658162	74	0.000531	me.ogp
652	17658048	2400	0.000013	com.wikidot
653	17657344	1610	0.000019	com.com
654	17657246	187	0.000133	com.eepurl
655	17657014	1227	0.000025	com.ssrn
656	17656988	694	0.000040	com.xinhuanet
657	17654064	1780	0.000017	org.scala-lang
658	17653188	1408	0.000022	edu.unc
659	17652568	1894	0.000016	org.iihs
660	17652104	771	0.000037	org.plos
661	17651732	1274	0.000024	tv.ustream
662	17651382	1068	0.000030	ly.ow
663	17650966	2194	0.000014	com.almanac
664	17650526	2055	0.000015	com.gamespot
665	17650220	2335	0.000013	com.bibliocommons
666	17649444	660	0.000042	com.feedly
667	17648694	797	0.000036	com.deloitte
668	17646576	959	0.000033	gov.senate
669	17646290	2218	0.000014	org.onegreenplanet
670	17645346	2125	0.000014	com.yourdomain
671	17645168	433	0.000057	com.squareup
672	17644982	1886	0.000016	com.mariadb
673	17643414	1548	0.000020	org.postimg
674	17642978	1291	0.000024	org.cambridge
675	17642506	2381	0.000013	com.marksdailyapple
676	17642072	261	0.000091	com.histats
677	17641504	1666	0.000019	com.digitaloceanspaces
678	17641474	1267	0.000024	com.canva
679	17641424	1392	0.000022	im.gitter
680	17641326	1198	0.000026	com.techrepublic
681	17640734	2749	0.000013	com.themonitor
682	17640688	1577	0.000019	uk.co.thesun
683	17640560	1618	0.000019	com.nba
684	17639544	2170	0.000014	com.winemag
685	17638276	1431	0.000022	com.mcafee
686	17638268	913	0.000034	gov.justice
687	17635694	722	0.000039	com.steampowered
688	17633614	886	0.000035	com.timeanddate
689	17633566	445	0.000055	com.adweek
690	17631684	834	0.000035	com.aliexpress
691	17630196	302	0.000078	com.netdna-ssl
692	17630110	1764	0.000017	us.oh.state
693	17629376	1241	0.000025	com.optinmonster
694	17628644	1389	0.000022	org.js
695	17628624	2240	0.000014	jp.ac.kobe-u
696	17627756	657	0.000042	gov.noaa
697	17626576	1782	0.000017	org.openweathermap
698	17625866	853	0.000035	com.marketwatch
699	17625214	2371	0.000013	com.winefolly
700	17624654	1585	0.000019	org.golang
701	17623884	343	0.000071	ca.google
702	17623882	1171	0.000027	com.hollywoodreporter
703	17623394	2642	0.000013	org.travelblog
704	17621812	2915	0.000012	me.pxlme
705	17621742	1718	0.000018	com.crunchbase
706	17621104	2417	0.000013	com.thedrinksbusiness
707	17620918	1253	0.000025	com.mlb
708	17620844	2183	0.000014	com.designobserver
709	17619798	1957	0.000016	com.whitepages
710	17618860	1308	0.000024	fr.lemonde
711	17617276	1650	0.000019	com.pastebin
712	17616020	2667	0.000013	com.backyardchickens
713	17615996	378	0.000065	com.themeisle
714	17615324	247	0.000099	io.polyfill
715	17614672	3674	0.000011	org.torproject
716	17614462	1196	0.000026	com.politico
717	17612598	965	0.000033	de.blogspot
718	17612468	2143	0.000014	com.programmableweb
719	17612380	777	0.000037	gov.house
720	17612350	2378	0.000013	uk.ac.hud
721	17612226	313	0.000076	com.fc2
722	17609572	351	0.000069	jp.co.rakuten
723	17609426	1284	0.000024	se.haxx
724	17609170	401	0.000062	com.smugmug
725	17609048	2191	0.000014	com.azfamily
726	17607352	126	0.000236	info.aboutads
727	17607032	5050	0.000007	com.formula1
728	17606320	2948	0.000012	com.locationrebel
729	17604020	252	0.000097	com.marriott
730	17603354	185	0.000134	com.xing
731	17603156	1543	0.000020	org.doxygen
732	17602956	491	0.000050	com.snapchat
733	17601902	2771	0.000013	com.trendland
734	17600640	1073	0.000030	com.americanexpress
735	17600636	1115	0.000028	com.redhat
736	17600606	2394	0.000013	com.sitejabber
737	17600436	2311	0.000014	com.galvanize
738	17600090	4285	0.000009	com.dreamstime
739	17599690	2035	0.000015	com.insiderpages
740	17599126	1419	0.000022	kr.flic
741	17599066	1110	0.000029	gov.uspto
742	17599060	837	0.000035	br.com.uol
743	17596014	530	0.000047	com.163
744	17595876	290	0.000082	gov.ftc
745	17595472	495	0.000049	com.nasdaq
746	17595126	2753	0.000013	com.lookuppage
747	17593550	1134	0.000028	fr.blogspot
748	17592570	1278	0.000024	com.prezi
749	17591712	2659	0.000013	com.avsforum
750	17591328	410	0.000061	mp.mailchi
751	17590608	2020	0.000015	edu.arizona
752	17590230	793	0.000036	com.nielsen
753	17589738	2364	0.000013	com.chamberofcommerce
754	17589414	2147	0.000014	com.towardsdatascience
755	17589090	1050	0.000030	com.sciencedaily
756	17588114	978	0.000033	io.readthedocs
757	17587844	283	0.000083	com.dedecms
758	17587504	1549	0.000020	uk.co.wired
759	17586578	1252	0.000025	com.dell
760	17585810	1435	0.000021	com.billboard
761	17585660	421	0.000059	com.criteo
762	17585524	2283	0.000014	org.zenit
763	17585188	1062	0.000030	org.change
764	17584840	1304	0.000024	edu.academia
765	17583818	588	0.000044	com.newyorker
766	17582200	3591	0.000012	com.sophos
767	17582180	1741	0.000018	de.welt
768	17581488	352	0.000069	net.themeforest
769	17581304	2293	0.000014	org.gwtproject
770	17580662	2788	0.000013	io.setosa
771	17580656	1276	0.000024	st.prom
772	17580614	1433	0.000021	fm.last
773	17580540	1730	0.000018	com.fifa
774	17580530	2687	0.000013	com.storeboard
775	17580282	2169	0.000014	au.com.truelocal
776	17580194	2297	0.000014	com.2findlocal
777	17580070	1093	0.000029	com.visualstudio
778	17579740	1111	0.000029	com.500px
779	17579538	250	0.000097	jp.co.amazon
780	17578580	2264	0.000014	net.webhostingsecretrevealed
781	17574976	1752	0.000018	org.rubyonrails
782	17574924	59	0.000607	com.messenger
783	17574886	1690	0.000018	com.mtv
784	17574662	2277	0.000014	com.newsbank
785	17573782	1095	0.000029	de.heise
786	17573384	1911	0.000016	com.ibtimes
787	17570610	1616	0.000019	com.problogger
788	17570126	1995	0.000015	com.ehow
789	17569784	1998	0.000015	mp.j
790	17568580	1036	0.000031	com.cbslocal
791	17568370	2398	0.000013	com.wcnc
792	17568244	1096	0.000029	com.investopedia
793	17567780	2930	0.000012	edu.unl
794	17567104	2317	0.000014	ly.cl
795	17566344	687	0.000041	com.caniuse
796	17566302	431	0.000057	com.verisign
797	17566120	1455	0.000021	com.hotmail
798	17565914	2181	0.000014	au.com.yellowpages
799	17565600	1453	0.000021	com.rollingstone
800	17565572	2115	0.000015	com.local
801	17564428	231	0.000104	fr.google
802	17563688	215	0.000111	it.google
803	17563288	2003	0.000015	com.smartblogger
804	17562868	1663	0.000019	org.coursera
805	17562472	2220	0.000014	gov.louisvilleky
806	17562082	1202	0.000026	com.domain
807	17560868	597	0.000043	com.nationalgeographic
808	17560104	2123	0.000015	com.theinnovationenterprise
809	17559640	2854	0.000012	ke.co.blogspot
810	17558802	2265	0.000014	io.kubernetes
811	17558702	2319	0.000014	net.brownbook
812	17558172	1859	0.000016	de.zeit
813	17558106	1344	0.000023	com.freepik
814	17557620	2205	0.000014	com.goinswriter
815	17557432	731	0.000039	com.tandfonline
816	17556946	1480	0.000021	edu.jhu
817	17556522	2071	0.000015	com.riddle
818	17556376	1256	0.000025	com.vox
819	17555602	1127	0.000028	com.smashingmagazine
820	17554660	1756	0.000018	edu.msu
821	17554422	838	0.000035	com.uk
822	17554304	2958	0.000012	org.dyndns
823	17553914	2403	0.000013	com.wsoctv
824	17553776	2406	0.000013	com.independent
825	17553776	1387	0.000022	com.nymag
826	17552988	1809	0.000017	com.posterous
827	17550834	1189	0.000026	com.digitalocean
828	17550516	883	0.000035	com.gofundme
829	17549804	255	0.000095	com.myshopify
830	17549356	2746	0.000013	com.spoke
831	17549122	2064	0.000015	com.chambermaster
832	17548302	1179	0.000027	de.spiegel
833	17548188	1784	0.000017	com.ikea
834	17548154	2263	0.000014	com.bizcommunity
835	17548094	2730	0.000013	com.communitywalk
836	17547516	2399	0.000013	com.ibmbigdatahub
837	17547486	1906	0.000016	com.thewritepractice
838	17546846	1599	0.000019	org.filezilla-project
839	17546810	1899	0.000016	com.techradar
840	17546678	1963	0.000015	com.visioncritical
841	17546154	1969	0.000015	com.brafton
842	17545852	1627	0.000019	com.codeplex
843	17545338	428	0.000057	com.sohu
844	17544316	335	0.000072	com.jotform
845	17543714	1779	0.000017	com.lawyers
846	17543442	2316	0.000014	edu.hbs
847	17543058	1401	0.000022	edu.usc
848	17542964	152	0.000169	com.addtoany
849	17542808	2705	0.000013	com.nation2
850	17542602	1317	0.000023	edu.uchicago
851	17542376	2182	0.000014	com.w3techs
852	17541496	1371	0.000023	sh.brew
853	17541272	1325	0.000023	com.strikingly
854	17540262	1988	0.000015	org.aclu
855	17540236	2579	0.000013	com.kens5
856	17539906	453	0.000054	jp.ne.sakura
857	17539894	1092	0.000029	com.prweb
858	17539810	2428	0.000013	com.tractorsupply
859	17539382	3540	0.000012	com.gyazo
860	17539240	2717	0.000013	com.yelloyello
861	17538820	1231	0.000025	com.elpais
862	17538662	3864	0.000010	com.rottentomatoes
863	17538296	2138	0.000014	net.hockeyapp
864	17537912	1697	0.000018	com.howstuffworks
865	17536686	2805	0.000012	com.lacartes
866	17536288	1628	0.000019	io.getmdl
867	17535392	2343	0.000013	com.citysquares
868	17534222	761	0.000037	net.daum
869	17533460	2407	0.000013	com.kmov
870	17530710	2816	0.000012	com.mothering
871	17530426	484	0.000051	com.iconfinder
872	17529686	2873	0.000012	org.rethinkingschools
873	17528810	1456	0.000021	org.wiktionary
874	17528532	707	0.000040	com.emarketer
875	17528512	259	0.000094	me.t
876	17528368	2885	0.000012	com.asus
877	17527494	1904	0.000016	com.rt
878	17527482	993	0.000032	com.oup
879	17527248	1383	0.000022	com.theglobeandmail
880	17524712	1270	0.000024	co.vine
881	17524420	2768	0.000013	org.foodrevolution
882	17524032	2365	0.000013	com.wpxi
883	17523232	973	0.000033	com.airbnb
884	17522954	970	0.000033	gov.usa
885	17522872	2702	0.000013	com.njmonthly
886	17522602	1011	0.000031	org.unesco
887	17522048	2800	0.000013	org.thebestschools
888	17521268	2309	0.000014	com.ezlocal
889	17521124	173	0.000153	com.bluehost
890	17521040	228	0.000105	com.maxcdn
891	17520736	2401	0.000013	com.cbs
892	17519970	1405	0.000022	org.example
893	17519746	2806	0.000012	com.calmclinic
894	17519654	964	0.000033	gov.copyright
895	17519130	2134	0.000014	edu.ncsu
896	17517626	3900	0.000010	com.domaintools
897	17517264	2799	0.000013	com.trepup
898	17517190	1849	0.000017	edu.indiana
899	17516236	1410	0.000022	org.unicode
900	17514928	2734	0.000013	com.mykaratestore
901	17514770	1860	0.000016	com.adespresso
902	17514742	487	0.000050	org.whatwg
903	17514450	3782	0.000011	gd.is
904	17513384	1837	0.000017	re.cli
905	17513134	3162	0.000012	com.000webhostapp
906	17512942	907	0.000034	com.alibaba
907	17512916	1624	0.000019	com.britannica
908	17512736	1210	0.000026	com.reverbnation
909	17512256	609	0.000043	com.patreon
910	17511530	4022	0.000010	edu.iu
911	17511124	794	0.000036	com.yandex
912	17511074	525	0.000047	com.outlook
913	17510210	1151	0.000027	org.fao
914	17509976	2756	0.000013	co.wanelo
915	17508946	1693	0.000018	com.udemy
916	17508814	1188	0.000026	gov.usgs
917	17508588	941	0.000034	com.ggpht
918	17508506	1309	0.000024	uk.co.mirror
919	17508422	1156	0.000027	edu.umn
920	17507730	318	0.000075	nl.google
921	17505054	258	0.000094	com.disqus
922	17504754	1313	0.000024	com.pwc
923	17504638	961	0.000033	com.pinimg
924	17504458	1800	0.000017	com.html5rocks
925	17504148	1124	0.000028	com.sun
926	17503468	1200	0.000026	com.uber
927	17501792	4643	0.000008	com.mysite
928	17501756	2308	0.000014	org.gimp
929	17501722	2066	0.000015	com.packtpub
930	17501690	2676	0.000013	com.pages10
931	17501468	2421	0.000013	com.tuck
932	17500992	2615	0.000013	org.swi-prolog
933	17500218	1835	0.000017	edu.virginia
934	17499886	2897	0.000012	be.brussels
935	17499784	1106	0.000029	au.net.abc
936	17499678	226	0.000105	com.googletagservices
937	17496996	3531	0.000012	ch.cern
938	17496282	2580	0.000013	com.ktvb
939	17496102	501	0.000049	com.bigcartel
940	17495798	1930	0.000016	com.nfl
941	17495794	1300	0.000024	com.showmelocal
942	17494518	1358	0.000023	org.pnas
943	17494014	2175	0.000014	uk.co.realbusiness
944	17494010	2728	0.000013	ly.visual
945	17492442	2109	0.000015	com.discovery
946	17492384	2622	0.000013	org.virginiadot
947	17491978	1069	0.000030	com.us
948	17491852	1829	0.000017	edu.cuny
949	17491298	1681	0.000018	com.podbean
950	17491172	1182	0.000026	com.accenture
951	17491140	2755	0.000013	com.pushwoosh
952	17490876	2588	0.000013	com.yellowbot
953	17490652	2903	0.000012	com.watchuseek
954	17490510	1451	0.000021	com.thehill
955	17490442	2834	0.000012	com.callupcontact
956	17490156	2397	0.000013	com.echelman
957	17490000	3853	0.000010	org.greenpeace
958	17489174	1783	0.000017	com.screencast
959	17488476	2043	0.000015	com.webnode
960	17488356	1119	0.000028	com.lifehacker
961	17487060	991	0.000032	org.iso
962	17487048	728	0.000039	com.gartner
963	17485804	1680	0.000018	com.hulu
964	17485784	1927	0.000016	co.gcdn
965	17484600	1525	0.000020	com.windows
966	17483866	2001	0.000015	com.birdeye
967	17483034	951	0.000034	ru.google
968	17482970	3909	0.000010	org.bitcoin
969	17481326	2413	0.000013	com.topsy
970	17481036	2429	0.000013	com.texasbar
971	17480454	882	0.000035	com.stitcher
972	17478940	2924	0.000012	com.talkbass
973	17478584	2278	0.000014	ca.ualberta
974	17478504	843	0.000035	gg.discord
975	17478450	2855	0.000012	com.cylex-usa
976	17477514	3673	0.000011	nl.xs4all
977	17477396	2925	0.000012	info.ufacity
978	17477382	104	0.000304	com.namecheap
979	17477122	2849	0.000012	com.louisville
980	17476316	2460	0.000013	uk.gov.westsussex
981	17475686	2668	0.000013	com.salespider
982	17475490	1568	0.000019	com.nokia
983	17475476	1034	0.000031	com.digiday
984	17475262	2889	0.000012	org.stnicholascenter
985	17475056	2850	0.000012	au.com.hotfrog
986	17474798	1257	0.000025	org.webkit
987	17474492	2430	0.000013	net.blog5
988	17474236	2271	0.000014	tv.periscope
989	17473902	877	0.000035	uk.co.tripadvisor
990	17473802	2900	0.000012	org.phys
991	17473188	1612	0.000019	edu.umd
992	17473104	884	0.000035	gov.ny
993	17472122	1981	0.000015	ru.narod
994	17471648	233	0.000104	jp.ameblo
995	17471606	5178	0.000007	net.minecraft
996	17470784	163	0.000162	com.youku
997	17470482	1677	0.000018	org.gnome
998	17470166	184	0.000137	com.nginx
999	17470156	1447	0.000021	com.splashthat
1000	17469878	2681	0.000013	com.bleacherreport

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

April 2019 crawl archive now available

April 30, 2019Sebastian Nagel

The crawl archive for April 2019 is now available! It contains 2.5 billion web pages or 198 TiB of uncompressed content, crawled between April 18th and 26th.

The April crawl contains page captures of 750 million URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks (harmonic centrality) published as part of the Nov/Dec/Jan 2018/2019 webgraph data set from the following sources:

sitemaps, RSS and Atom feeds
a breadth-first side crawl within a maximum of 3 links (“hops”) away from the homepages of the top 60 million hosts and domains and a random sample of 1 million human-readable sitemap pages (HTML format)
a random sample of 1 billion outlinks taken from WAT files of the March crawl

The following minor changes to the crawler configuration have been made:

the crawler now sends again an Accept-Language HTTP header, requesting English content
the configuration has been tweaked to include less non-HTML content

Archive Location and Download

The April crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-18/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-18/segment.paths.gz	100
WARC files	CC-MAIN-2019-18/warc.paths.gz	56000	44.86
WAT files	CC-MAIN-2019-18/wat.paths.gz	56000	16.32
WET files	CC-MAIN-2019-18/wet.paths.gz	56000	6.96
Robots.txt files	CC-MAIN-2019-18/robotstxt.paths.gz	56000	0.16
Non-200 responses files	CC-MAIN-2019-18/non200responses.paths.gz	56000	1.67
URL index files	CC-MAIN-2019-18/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-18/. Also the columnar index has been updated to contain this crawl.

March 2019 crawl archive now available

April 1, 2019Sebastian Nagel

The crawl archive for March 2019 is now available! It contains 2.55 billion web pages or 210 TiB of uncompressed content, crawled between March 18th and 27th.

The March crawl contains page captures of 660 million URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks (harmonic centrality) published as part of the Nov/Dec/Jan 2018/2019 webgraph data set from the following sources:

sitemaps, RSS and Atom feeds
a breadth-first side crawl within a maximum of 6 links (“hops”) away from the homepages of the top 60 million hosts and domains
a random sample of outlinks taken from WAT files of the February crawl

Archive Location and Download

The March crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-13/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2019-13/segment.paths.gz	100
WARC files	CC-MAIN-2019-13/warc.paths.gz	56000	49.09
WAT files	CC-MAIN-2019-13/wat.paths.gz	56000	17.37
WET files	CC-MAIN-2019-13/wet.paths.gz	56000	7.47
Robots.txt files	CC-MAIN-2019-13/robotstxt.paths.gz	56000	0.17
Non-200 responses files	CC-MAIN-2019-13/non200responses.paths.gz	56000	1.63
URL index files	CC-MAIN-2019-13/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-13/. Also the columnar index has been updated to contain this crawl.